Update, review, tidy and fix across s/scripts and Docker setup

AGENTS.md

@@ -0,0 +1,152 @@
+# AGENTS.md - Guidance for AI/LLM Development
+
+This document provides context and guidance for AI agents, LLMs, and automated tools working on the rcpchgrowth-python repository.
+
+## Project Overview
+
+**rcpchgrowth-python** is a Python library for calculating children's growth measurements against UK and international growth references. The project is currently transitioning from UK-WHO reference data to pure WHO reference data.
+
+### Active Branch: `who-validation`
+
+The current development branch is `who-validation`, which:
+
+- Replaces UK-WHO reference data with WHO reference data
+- Maintains backward compatibility at the API level
+- Uses test fixtures generated from WHO's published `anthro` and `anthroplus` R packages (RCPCH forks with enhanced precision)
+- All new tests pass; the branch is ready for validation
+
+## Development Workflow
+
+### Container-Based Development
+
+The project uses Docker for a consistent development environment. Always use the convenience scripts in the `s/` folder:
+
+```bash
+s/up          # Start container
+s/notebook    # Launch JupyterLab
+s/test        # Run pytest (auto-starts container if needed)
+s/test --running  # Run pytest in already-running container
+s/shell       # Interactive bash in container
+s/down        # Stop container
+```
+
+**Key Point**: The container runs JupyterLab in the background, enabling both interactive notebook development AND command-line test execution simultaneously.
+
+## Testing Strategy
+
+### Test Fixtures
+
+**Standard fixture** (new):
+
+- `rcpchgrowth/tests/sds_age_validation_2021_refactored_2026.json` - 3984 test cases generated from WHO data
+- All tests pass against this fixture
+- This is the current/target state
+
+**Deprecated fixture** (old):
+
+- `rcpchgrowth/tests/sds_age_validation_2021_deprecated.json` - 4002 test cases from live branch
+- 18 test cases removed during transition (see docs/LIVE_DATASET_FAILED_TESTS_SUMMARY.md)
+- Kept for regression testing if needed
+
+WHO dataset details:
+
+- `rcpchgrowth/tests/who_test_data/README.md` - Inventory of WHO test files and rationale for `who_under2_gold_192.csv`
+
+### Running Tests
+
+```bash
+# Run the UK-WHO integration suite
+s/test rcpchgrowth/tests/test_uk_who.py -v
+
+# Run specific test case
+s/test rcpchgrowth/tests/test_uk_who.py::test_uk_who_reference_integration -v
+
+# Run all tests
+s/test
+
+# In an already-running container
+s/test --running rcpchgrowth/tests/ -v
+```
+
+## Key Code Locations
+
+| Component | Location |
+|-----------|----------|
+| Main library | `rcpchgrowth/` |
+| Measurement calculation | `rcpchgrowth/measurement.py` |
+| Tests | `rcpchgrowth/tests/` |
+| Test data | `rcpchgrowth/tests/sds_age_validation_2021_refactored_2026.json` |
+| Reference data | `rcpchgrowth/data_tables/` |
+| Documentation | `docs/` |
+
+## Important Considerations for LLM Development
+
+### Test Fixture Strategy
+
+The test fixture is **fixed and finite** (3984 cases). When modifying calculation logic:
+
+1. Run tests to identify failures
+2. Analyze whether failures are expected (intentional algorithm changes) or bugs
+3. Do NOT modify test fixtures without explicit user direction
+4. Document any intentional test failures in PR comments
+
+### Preterm/Early Infant Focus
+
+The 18 removed test cases (from live → who-validation transition) were concentrated in:
+
+- Very early infancy (mostly <0.5 years)
+- Preterm/late preterm births (27+2 to 44+0 weeks gestation)
+- 61% female cases with duplicate age-measurement combinations
+
+This indicates **largest numerical divergence between UK-WHO and WHO occurs in preterm/early infant assessment**. When debugging calculation differences, prioritize these scenarios.
+
+### Reference Data
+
+WHO reference data is loaded from `rcpchgrowth/data_tables/`:
+
+- LMS values (Lambda, Mu, Sigma) stored as JSON
+- Age-corrected calculations for preterm infants (up to 2 years)
+- Different handling vs. UK-WHO; be cautious with age correction logic
+
+### Git Branch Context
+
+- **live** (main production branch) - uses UK-WHO data
+- **who-validation** (active development) - uses WHO data, all new tests passing
+- New changes should target `who-validation` for PR review
+- Do not push directly to `live`
+
+## Documentation for Developers
+
+- [LIVE_DATASET_FAILED_TESTS_SUMMARY.md](docs/LIVE_DATASET_FAILED_TESTS_SUMMARY.md) - Analysis of 18 removed test cases, including demographics and patterns
+- [README.md](README.md) - Installation and quick-start (human-focused, but useful context)
+
+## Common Tasks
+
+### Adding a New Feature
+
+1. Write test cases in `rcpchgrowth/tests/`
+2. Implement feature in appropriate module
+3. Run `s/test` to validate
+4. If test fixture needs updating, document justification
+
+### Debugging a Test Failure
+
+1. Check if it's a known issue in `docs/LIVE_DATASET_FAILED_TESTS_SUMMARY.md`
+2. Run specific test with `-v` flag for full output
+3. Inspect test fixture data for the failing case
+4. Compare old vs. new calculation if transitioning between reference systems
+
+### Environment Issues
+
+If container fails to start or tests don't run:
+
+1. `s/down` then `s/up` to restart
+2. Check `docker compose logs` for errors
+3. Verify pytest installed: `s/test --running --version`
+4. See docker-compose.yml for setup command
+
+## Contact & Issues
+
+- Issues: https://github.com/rcpch/rcpchgrowth-python/issues
+- Repository: https://github.com/rcpch/rcpchgrowth-python
+- Documentation: https://growth.rcpch.ac.uk/products/python-library/

Dockerfile

@@ -25,17 +25,13 @@ COPY requirements.txt ./
 # Copy the full source (needed for editable install)
 COPY rcpchgrowth rcpchgrowth
 
-# Upgrade pip/setuptools/wheel first
-RUN pip install --upgrade pip setuptools wheel
-
-# Install runtime + notebook deps first (ensures wheels pulled before editable wiring)
-RUN pip install python-dateutil scipy pandas matplotlib jupyterlab ipykernel
-
-# Install dev/test tools
-RUN pip install -r requirements.txt
-
-# Finally perform editable install (should be fast now; minimal build isolation work)
-RUN pip install -e . || (echo '--- Editable install failed; running pip debug ---' && python -m pip debug && exit 1)
+# Upgrade build tooling, install the package (with notebook extras) editable,
+# and install dev/test deps. Notebook extras (jupyterlab, ipykernel, pandas,
+# matplotlib) and runtime deps (python-dateutil, scipy) come from pyproject.toml.
+RUN pip install --upgrade pip setuptools wheel \
+ && pip install -e .[notebook] \
+ && pip install -r requirements.txt \
+ && python -m ipykernel install --name rcpchgrowth --display-name 'rcpchgrowth' --sys-prefix
 
 # Bring in notebooks only after package install so they are never considered for package discovery
 COPY notebooks notebooks

README.md

@@ -7,6 +7,8 @@
 [![Binder](https://img.shields.io/badge/Binder-Binder?style=flat-square&labelColor=%2311a7f2&color=%230d0d58&logo=jupyter)](https://mybinder.org/v2/gh/rcpch/rcpchgrowth-python/live?urlpath=lab/tree/notebooks/Quickstart.ipynb)
 [![Codespaces](https://img.shields.io/badge/Codespaces-Open_in_Cloud?style=flat-square&labelColor=%2311a7f2&color=%230d0d58&logo=github&logoColor=white)](https://codespaces.new/rcpch/rcpchgrowth-python?quickstart=1)
 
+**For AI/LLM agents working on this repository:** Please read [AGENTS.md](AGENTS.md) for project context, development workflow, and testing strategy.
+
 Please go to <https://growth.rcpch.ac.uk/products/python-library/> for full documentation.
 
 Issues can be raised here <https://github.com/rcpch/rcpchgrowth-python/issues>
@@ -19,10 +21,42 @@ Issues can be raised here <https://github.com/rcpch/rcpchgrowth-python/issues>
 
 If you want to avoid setting up docker environments, there are shortcut scripts the create a dockerized environment with RCPCHGrowth already installed.
 
-This will generate a container which will launch some Jupyter notebooks in a browser and allow local dev ( with hot reload).
+This will generate a container which will launch some Jupyter notebooks in a browser and allow local dev (with hot reload).
+
+#### Convenience Scripts
+
+The `s/` folder contains helper scripts for common development tasks:
+
+| Script | Purpose |
+|--------|---------|
+| `s/up` | Start the development container |
+| `s/down` | Stop the development container |
+| `s/test` | Run pytest (auto-starts container if needed; use `--running` flag for already-running container) |
+| `s/notebook` | Launch JupyterLab in your browser |
+| `s/shell` | Open an interactive bash shell in the container |
+| `s/python` | Launch Python REPL in the container |
+
+**Quick start:**
 
 ```bash
-s/up
+# Start the container and launch notebooks
+s/notebook
+
+# Run tests (in a separate terminal)
+s/test
+
+# Or run tests in an already-running container
+s/test --running
+
+# Run the UK-WHO integration suite
+s/test rcpchgrowth/tests/test_uk_who.py -v
+
+# Reference WHO test datasets and under-2 gold-standard rationale
+# (192 deterministic anthro-generated cases)
+# See rcpchgrowth/tests/who_test_data/README.md
+
+# Stop when done
+s/down
 ```
 
 ### Minimal installation (without docker) assuming you have a python virtual env setup

docker-compose.yml

@@ -1,26 +1,23 @@
 services:
-  dev:
+  rcpchgrowth-python:
     build: .
-    container_name: rcpchgrowth-dev
+    container_name: rcpchgrowth-python
     volumes:
+      # Mount the entire project directory for live development
       - .:/app
+      # Prevent overwriting installed packages and cache directories
+      - /usr/local/lib/python3.12/site-packages
+      - /app/.pytest_cache
     working_dir: /app
     environment:
       - PYTHONPATH=/app
     ports:
       - "8888:8888"
-    command:
-      - bash
-      - -lc
-      - |
-        set -e
-        mkdir -p /app/notebooks
-        echo '[DEV] Editable install'
-        pip install -e /app[notebook]
-        echo '[DEV] Sanity import'
-        echo '[DEV] Register kernel'
-        python -m ipykernel install --name rcpchgrowth --display-name 'rcpchgrowth' --sys-prefix
-        echo '[DEV] Launching JupyterLab'
-        exec jupyter lab --ip=0.0.0.0 --no-browser --allow-root \
-          --NotebookApp.token='' --NotebookApp.password='' \
-          --notebook-dir=/app/notebooks
+    # Re-wire the editable install against the bind-mounted source (creates
+    # rcpchgrowth.egg-info on the host on first run), then exec JupyterLab.
+    # All deps and the Jupyter kernel are baked into the image at build time.
+    command: >
+      bash -lc "pip install -e . -q &&
+      exec jupyter lab --ip=0.0.0.0 --no-browser --allow-root
+      --NotebookApp.token='' --NotebookApp.password=''
+      --notebook-dir=/app/notebooks"