Examples workflow broken

[mattwthompson]

Since late April, seemingly: https://github.com/openforcefield/openff-toolkit/actions/workflows/examples.yml?query=branch%3Amain

[j-wags]

Diffing the installed conda env from May 15 1:44 PM PDT Conda Latest CI run (which passed) against the May 16 1:27 PM PDT Conda Latest CI run (which failed) does not appear to indicate any difference in installed conda packages.

[mattwthompson]

Diffing the entire logs doesn't provide much value, either

The traceback is long, since there are several Interchange methods the AM1-BCC call gets routed through, and the toolkit reports mostly noise from the antechamber error:

< ValueError: No registered toolkits can provide the capability "assign_partial_charges" for args "()" and kwargs "{'molecule': Molecule with name '163' and SMILES '[H][O][C@@]1([H])[c]2[c]([c]([O][c]3[c]([H])[c]([F])[c]([H])[c]([C]#[N])[c]3[H])[c]([H])[c]([H])[c]2[S](=[O])(=[O])[C]([H])([H])[H])[C]([H])([H])[C]1([F])[F]', 'partial_charge_method': 'am1bcc'}"
< Available toolkits are: [ToolkitWrapper around OpenFF NAGL version 0.5.5, ToolkitWrapper around The RDKit version 2024.03.5, ToolkitWrapper around AmberTools version 23.6, ToolkitWrapper around Built-in Toolkit version None]
<  ToolkitWrapper around OpenFF NAGL version 0.5.5 <class 'openff.nagl_models._dynamic_fetch.BadFileSuffixError'> : OpenFF NAGL models are based on PyTorch files and expect a `.pt` suffix. Found an unrecognized file path extension on filename='am1bcc'
<  ToolkitWrapper around The RDKit version 2024.03.5 <class 'openff.toolkit.utils.exceptions.ChargeMethodUnavailableError'> : partial_charge_method 'am1bcc' is not available from RDKitToolkitWrapper. Available charge methods are {'mmff94': {}, 'gasteiger': {}}
<  ToolkitWrapper around AmberTools version 23.6 <class 'subprocess.CalledProcessError'> : Command '['antechamber', '-i', 'molecule.sdf', '-fi', 'sdf', '-o', 'charged.mol2', '-fo', 'mol2', '-pf', 'yes', '-dr', 'n', '-c', 'bcc', '-nc', '0.0']' returned non-zero exit status 1.
<  ToolkitWrapper around Built-in Toolkit version None <class 'openff.toolkit.utils.exceptions.ChargeMethodUnavailableError'> : Partial charge method "am1bcc"" is not supported by the Built-in toolkit. Available charge methods are {'zeros': {'rec_confs': 0, 'min_confs': 0, 'max_confs': 0}, 'formal_charge': {'rec_confs': 0, 'min_confs': 0, 'max_confs': 0}}

[j-wags]

It's also not a runner image change. The info is identical for both (except the worker id):

Current runner version: '2.334.0'
Runner Image Provisioner
  Hosted Compute Agent
  Version: 20260213.493
  Commit: 5c115507f6dd24b8de37d8bbe0bb4509d0cc0fa3
  Build Date: 2026-02-13T00:28:41Z
  Worker ID: {425cabad-cb45-48bc-9c67-cbe1cc649d11} (NOTE: this does change between the two runs)
  Azure Region: eastus
Operating System
  Ubuntu
  24.04.4
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20260513.135.3
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20260513.135/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20260513.135
GITHUB_TOKEN Permissions
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'actions/checkout@v6' (SHA:de0fac2e4500dabe0009e67214ff5f5447ce83dd)
Download action repository 'mamba-org/setup-micromamba@v3' (SHA:d7c9bd84e824b79d2af72a2d4196c7f4300d3476)
Complete job name: Test on ubuntu-latest, Python 3.12, OpenEye=false

[mattwthompson]

The environment solution falls back to AmberTools 23, which forces some other packages¹ down to older versions. Removing it from example environments (conda-forge/openff-toolkit-feedstock#120) might fix this

Footnotes

1. It's built with Boost 1.84, which was released in December 2023. 1.85 was released in April 2024, so many compiled dependencies are stuck around that time period of releases ↩

[mattwthompson]

Aha, well ... our examples use openmmforcefields so that doesn't really get AmberTools out of our environments

[j-wags]

The environment solution falls back to AmberTools 23,

Interesting. It looks like our linux builds are pulling AT 23.6 while mac builds on the same day pull AT 24.8

[j-wags]

Just recording here that, in the process of trying to reproduce linux conda envs selecting ambertools 23, I've been thus far unable to actually get past the env solving step, in either a linux docker container on my macbook, or in a github codespace.

micromamba create -n temp --dry-run -c conda-forge openff-toolkit

fails after 90ish seconds with

Resolving Environment                                                 ⧖ StartingKilled

I'm thinking this is probably a memory issue. I'd experienced this while helping a user a few weeks ago and thought it was something weird with their setup. However this is probably an increasingly large problem for our linux-based users at this point.

[j-wags]

I've found that specifying python=3.13 makes the solve complete pretty quickly without getting killed. But I don't like that that's needed.

[j-wags]

Ok, it appears to be the nomkl pin that's causing linux builds to select ambertools 23.6. The nomkl pin in the conda env references this closed PR from two years ago #1821, and the pin was added to the test envs about 5 months ago: #2137 and #2139

micromamba env create --dry-run -f conda.yaml "python=3.12"

yields

  Updating specs:

   - python=3.12
   - openff-toolkit-examples
   - pytest=7.4
   - pytest-rerunfailures
   - nbval
   - numpy<2.3
   - xmltodict<=1.0.2
   - nomkl
   - pip


  Package                               Version  Build                              Channel          Size
───────────────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────────────────────────────

  + _openmp_mutex                           4.5  20_gnu                             conda-forge      29kB
  + _python_abi3_support                    1.0  hd8ed1ab_2                         conda-forge       8kB
  + ambertools                             23.6  cuda_None_nompi_py312h41aabf6_105  conda-forge      95MB

But removing the nomkl pin from conda.yaml yields:

specs:

   - python=3.12
   - openff-toolkit-examples
   - pytest=7.4
   - pytest-rerunfailures
   - nbval
   - numpy<2.3
   - xmltodict<=1.0.2
   - pip


  Package                               Version  Build                              Channel          Size
───────────────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────────────────────────────

  + _openmp_mutex                           4.5  7_kmp_llvm                         conda-forge       8kB
  + _python_abi3_support                    1.0  hd8ed1ab_2                         conda-forge       8kB
  + ambertools                             24.8  cuda_None_nompi_py312hbe1986e_101  conda-forge     102MB