Skip to content

Read files xml#509

Merged
svadams (svadams) merged 5 commits into
MetOffice:mainfrom
mo-marqh:readFilesXML
Jun 3, 2026
Merged

Read files xml#509
svadams (svadams) merged 5 commits into
MetOffice:mainfrom
mo-marqh:readFilesXML

Conversation

@mo-marqh
Copy link
Copy Markdown
Member

@mo-marqh mo-marqh commented May 20, 2026 •
edited by svadams
Loading

PR Summary

Sci/Tech Reviewer: Harry Shepherd (@harry-shepherd)
Code Reviewer: svadams (@svadams)

introduce field-group naming convention for read fields to enable better management by lfric_core lfric_xios interface

Code Quality Checklist

  • I have performed a self-review of my own code
  • My code follows the project's style guidelines
  • Comments have been included that aid understanding and enhance the readability of the code
  • My changes generate no new warnings
  • All automated checks in the CI pipeline have completed successfully

Testing

  • I have tested this change locally, using the LFRic Apps rose-stem suite
  • If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
  • I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
  • Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - readFilesXML/run7

Suite Information

Item Value
Suite Name readFilesXML/run7
Suite User mark.hedley
Workflow Start 2026-05-20T14:56:16
Groups Run all', 'lfric_atm_ex1a_performance_runstats
Dependency Reference Main Like
casim MetOffice/casim@2026.03.2 True
jules MetOffice/jules@2026.03.2 True
lfric_apps mo-marqh/lfric_apps@readFilesXML False
lfric_core mo-marqh/lfric_core@EdHone-351-file-field_group False
moci MetOffice/moci@2026.03.2 True
SimSys_Scripts MetOffice/SimSys_Scripts@cab3315 True
socrates MetOffice/socrates@2026.03.2 True
socrates-spectral MetOffice/socrates-spectral@2026.03.2 True
ukca MetOffice/ukca@1cdb9c2 True

Task Information

✅ succeeded tasks - 1553

Security Considerations

  • I have reviewed my changes for potential security issues
  • Sensitive data is properly handled (if applicable)
  • Authentication and authorisation are properly implemented (if applicable)

Performance Impact

  • Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

  • Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

  • Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

  • If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

  • I understand this area of code and the changes being added
  • The proposed changes correspond to the pull request description
  • Documentation is sufficient (do documentation papers need updating)
  • Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

  • All dependencies have been resolved
  • Related Issues have been properly linked and addressed
  • CLA compliance has been confirmed
  • Code quality standards have been met
  • Tests are adequate and have passed
  • Documentation is complete and accurate
  • Security considerations have been addressed
  • Performance impact is acceptable

mo-marqh added 2 commits May 20, 2026 08:47
@mo-marqh
prefer _field_group naming convention for single field_groups for rea…
69d32e8 
…d files; enable setting of read_access=".true." programatically on enable
@mo-marqh
linked PR dependency
02fd080
@mo-marqh mo-marqh added the Linked Core This PR is linked to a MetOffice/lfric_core PR label May 20, 2026
@mo-marqh mo-marqh mentioned this pull request May 20, 2026
Closed
28 tasks
@mo-marqh
Copy link
Copy Markdown
Member Author

mo-marqh commented May 20, 2026

this PR supersedes #337

@mo-marqh
Copy link
Copy Markdown
Member Author

mo-marqh commented May 20, 2026

on performance testing, then C896 performance runs were compared for baseline and this linked changeset using XIOS2 on Genoa

#337 identified a slowdown during initialisation, due to far more data than is needed being loaded. This PR 7 linked core PR address this, as evidenced by:

find ~/cylc-run/readFilesXML/run3/log/job/1/run_lfric_atm_nwp_gal9-C896_MG_720t_perfRun* -name timer.txt -exec grep -h -e lfric_atm -e gungho_driver.initialise {} +
||                       lfric_atm||              1675.48||              1675.51||              1675.55||                    1||             100.0000||            1675.5066||
||        gungho_driver.initialise||                47.05||                47.08||                47.11||                    1||               2.8101||              47.0826||
||                       lfric_atm||              1701.58||              1701.61||              1701.66||                    1||             100.0000||            1701.6069||
||        gungho_driver.initialise||                41.78||                41.82||                41.85||                    1||               2.4576||              41.8183||
||                       lfric_atm||              1669.10||              1669.13||              1671.69||                    1||             100.0000||            1669.1332||
||        gungho_driver.initialise||                40.09||                40.13||                40.17||                    1||               2.4043||              40.1315||
||                       lfric_atm||              1679.26||              1679.29||              1679.33||                    1||             100.0000||            1679.2943||
||        gungho_driver.initialise||                40.03||                40.06||                40.08||                    1||               2.3854||              40.0583||

find ~/cylc-run/readFilesXML_test_baseline/run1/log/job/1/run_lfric_atm_nwp_gal9-C896_MG_720t_perfRun* -name timer.txt -exec grep -h -e lfric_atm -e gungho_driver.initialise {} +
||                       lfric_atm||              1668.39||              1668.46||              1670.44||                    1||             100.0000||            1668.4554||
||        gungho_driver.initialise||                46.59||                46.63||                46.65||                    1||               2.7945||              46.6258||
||                       lfric_atm||              1706.54||              1706.57||              1706.60||                    1||             100.0000||            1706.5711||
||        gungho_driver.initialise||                42.16||                42.19||                42.28||                    1||               2.4724||              42.1939||
||                       lfric_atm||              1668.97||              1668.99||              1669.02||                    1||             100.0000||            1668.9911||
||        gungho_driver.initialise||                43.26||                43.29||                43.32||                    1||               2.5939||              43.2926||
||                       lfric_atm||              1668.86||              1668.89||              1668.93||                    1||             100.0000||            1668.8870||
||        gungho_driver.initialise||                43.64||                43.67||                43.70||                    1||               2.6168||              43.6721||

showing comparable results for this change at scale

DanStoneMO
DanStoneMO approved these changes May 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@DanStoneMO DanStoneMO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JEDI works fine with this change as-is. No linked JEDI PR will be needed.

thomasmelvin
thomasmelvin approved these changes May 21, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@thomasmelvin thomasmelvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me

@harry-shepherd
Copy link
Copy Markdown
Contributor

Harry Shepherd (harry-shepherd) commented May 26, 2026

Hi mo-marqh , do we want to add the field groupings to all the XML files that contain reads? I have a summary of the xml files and their respective file id's below if that is something to be desired. If it's not appropriate, I'm happy to approve the change as it is,

Files that are reading fields without `field_group`

This doesn't include any file definitions that don't include any fields

{'applications/gungho_model/example/file_def_ancil.xml': ['id="orography_mean_ancil"'],
 'applications/jedi_lfric_tests/example_id_tlm_tests/iodef.xml': ['id="lfric_checkpoint_read"',
                                                                  'id="read_lfric_fd_dump"',
                                                                  'id="orography_mean_ancil"',
                                                                  'id="ls"'],
 'applications/jedi_lfric_tests/example_tlm_forecast_tl/iodef.xml': ['id="lfric_checkpoint_read"',
                                                                     'id="read_lfric_fd_dump"',
                                                                     'id="orography_mean_ancil"',
                                                                     'id="ls"'],
 'applications/jedi_lfric_tests/example_tlm_tests/iodef.xml': ['id="lfric_checkpoint_read"',
                                                               'id="read_lfric_fd_dump"',
                                                               'id="orography_mean_ancil"',
                                                               'id="ls"'],
 'applications/ngarch/example/metadata/file_def_ancil_gal.xml': ['id="orography_subgrid_ancil"',
                                                                 'id="land_area_ancil"',
                                                                 'id="hydtop_ancil"',
                                                                 'id="surface_frac_ancil"'],
 'applications/ngarch/example/metadata/file_def_ancil_hadisst.xml': ['id="sst_ancil"',
                                                                     'id="sea_ice_ancil"'],
 'applications/ngarch/example/metadata/file_def_ancil_idealised.xml': ['id="gas_mmr_ancil"'],
 'applications/ngarch/example/metadata/file_def_ancil_main.xml': ['id="orography_mean_ancil"',
                                                                  'id="sea_ancil"',
                                                                  'id="albedo_vis_ancil"',
                                                                  'id="albedo_nir_ancil"',
                                                                  'id="soil_ancil"',
                                                                  'id="soil_rough_ancil"',
                                                                  'id="urban_ancil"',
                                                                  'id="plant_func_ancil"',
                                                                  'id="ozone_ancil"',
                                                                  'id="aerosols_ancil"',
                                                                  'id="emiss_c2h6_ancil"',
                                                                  'id="emiss_c3h8_ancil"',
                                                                  'id="emiss_c5h8_ancil"',
                                                                  'id="emiss_ch4_ancil"',
                                                                  'id="emiss_co_ancil"',
                                                                  'id="emiss_hcho_ancil"',
                                                                  'id="emiss_me2co_ancil"',
                                                                  'id="emiss_mecho_ancil"',
                                                                  'id="emiss_nh3_ancil"',
                                                                  'id="emiss_no_ancil"',
                                                                  'id="emiss_meoh_ancil"',
                                                                  'id="emiss_no_aircrft_ancil"',
                                                                  'id="emiss_bc_biofuel_ancil"',
                                                                  'id="emiss_bc_fossil_ancil"',
                                                                  'id="emiss_bc_biomass_ancil"',
                                                                  'id="emiss_bc_biomass_hi_ancil"',
                                                                  'id="emiss_bc_biomass_lo_ancil"',
                                                                  'id="emiss_dms_land_ancil"',
                                                                  'id="dms_conc_ocean_ancil"',
                                                                  'id="emiss_monoterp_ancil"',
                                                                  'id="emiss_om_biofuel_ancil"',
                                                                  'id="emiss_om_fossil_ancil"',
                                                                  'id="emiss_om_biomass_ancil"',
                                                                  'id="emiss_om_biomass_hi_ancil"',
                                                                  'id="emiss_om_biomass_lo_ancil"',
                                                                  'id="emiss_so2_low_ancil"',
                                                                  'id="emiss_so2_high_ancil"',
                                                                  'id="emiss_so2_nat_ancil"',
                                                                  'id="soil_dust_ancil"',
                                                                  'id="emiss_murk_ancil"',
                                                                  'id="h2o2_limit_ancil"',
                                                                  'id="ho2_ancil"',
                                                                  'id="no3_ancil"',
                                                                  'id="o3_ancil"',
                                                                  'id="oh_ancil"',
                                                                  'id="snow_analysis_ancil"',
                                                                  'id="cloud_drop_no_conc_ancil"',
                                                                  'id="easy_asymmetry_sw_ancil"',
                                                                  'id="easy_asymmetry_lw_ancil"',
                                                                  'id="easy_absorption_sw_ancil"',
                                                                  'id="easy_absorption_lw_ancil"',
                                                                  'id="easy_extinction_sw_ancil"',
                                                                  'id="easy_extinction_lw_ancil"'],
 'rose-stem/app/adjoint_tests/file/file_def_ancil.xml': ['id="orography_mean_ancil"'],
 'rose-stem/app/adjoint_tests/file/file_def_ls.xml': ['id="ls"'],
 'rose-stem/app/adjoint_tests/file/file_def_read.xml': ['id="read_lfric_fd_dump"'],
 'rose-stem/app/jedi_id_tlm_tests/file/iodef.xml': ['id="lfric_checkpoint_read"',
                                                    'id="read_lfric_fd_dump"'],
 'rose-stem/app/jedi_tlm_forecast_tl/file/iodef.xml': ['id="lfric_checkpoint_read"',
                                                       'id="read_lfric_fd_dump"'],
 'rose-stem/app/jedi_tlm_tests/file/iodef.xml': ['id="lfric_checkpoint_read"',
                                                 'id="read_lfric_fd_dump"'],
 'rose-stem/app/lfric2lfric/file/iodef_lbc.xml': ['id="lfric_lbc_read"',
                                                  'id="src_orography_mean_ancil"',
                                                  'id="dst_orography_mean_ancil"'],
 'rose-stem/app/lfric2lfric/file/iodef_orography.xml': ['id="dst_orography_mean_ancil"'],
 'rose-stem/app/lfric_atm/file/iodef_gal_nwp_oper.xml': ['id="read_lfric_fd_dump"'],
 'rose-stem/app/lfric_atm/file/iodef_gal_nwp_oper_hres.xml': ['id="read_lfric_fd_dump"'],
 'rose-stem/app/lfric_coupled/file/mydef.xml': ['id="lfric_checkpoint_read"',
                                                'id="read_lfric_fd_dump"',
                                                'id="orography_mean_ancil"',
                                                'id="orography_subgrid_ancil"',
                                                'id="land_area_ancil"',
                                                'id="sea_ancil"',
                                                'id="sst_ancil"',
                                                'id="albedo_vis_ancil"',
                                                'id="albedo_nir_ancil"',
                                                'id="hydtop_ancil"',
                                                'id="soil_ancil"',
                                                'id="surface_frac_ancil"',
                                                'id="plant_func_ancil"',
                                                'id="ozone_ancil"',
                                                'id="aerosols_ancil"',
                                                'id="emiss_bc_biofuel_ancil"',
                                                'id="emiss_bc_fossil_ancil"',
                                                'id="emiss_bc_biomass_ancil"',
                                                'id="emiss_dms_land_ancil"',
                                                'id="dms_conc_ocean_ancil"',
                                                'id="emiss_monoterp_ancil"',
                                                'id="emiss_om_biofuel_ancil"',
                                                'id="emiss_om_fossil_ancil"',
                                                'id="emiss_om_biomass_ancil"',
                                                'id="emiss_so2_low_ancil"',
                                                'id="emiss_so2_high_ancil"',
                                                'id="emiss_so2_nat_ancil"',
                                                'id="soil_dust_ancil"',
                                                'id="h2o2_limit_ancil"',
                                                'id="ho2_ancil"',
                                                'id="no3_ancil"',
                                                'id="o3_ancil"',
                                                'id="oh_ancil"'],
 'rose-stem/app/linear_model/file/file_def_ancil.xml': ['id="orography_mean_ancil"'],
 'rose-stem/app/linear_model/file/file_def_ls.xml': ['id="ls"'],
 'rose-stem/app/linear_model/file/file_def_read.xml': ['id="read_lfric_fd_dump"'],
 'science/linear/integration-test/nwp_gal9/iodef.xml': ['id="read_lfric_fd_dump"',
                                                        'id="orography_mean_ancil"',
                                                        'id="ls"']}

@github-actions github-actions Bot added the cla-modified The CLA has been modified as part of this PR - added by GA label May 28, 2026
@mo-marqh
Copy link
Copy Markdown
Member Author

mo-marqh commented May 28, 2026

Hi mo-marqh , do we want to add the field groupings to all the XML files that contain reads? I have a summary of the xml files and their respective file id's below if that is something to be desired. If it's not appropriate, I'm happy to approve the change as it is,

thanks Harry Shepherd (@harry-shepherd)
5d49535
follows your helpfully provided list and adds named field_group definitions in each case

svadams
svadams (svadams) approved these changes Jun 1, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@svadams svadams (svadams) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are understood and look good to me so approving for merge

@github-actions github-actions Bot removed the cla-modified The CLA has been modified as part of this PR - added by GA label Jun 2, 2026
DrTVockerodtMO
DrTVockerodtMO approved these changes Jun 2, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@DrTVockerodtMO DrTVockerodtMO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look fine on the adjoint side of things, code owner review approved!

@svadams svadams (svadams) mentioned this pull request Jun 3, 2026
Merged
29 tasks
@svadams
Copy link
Copy Markdown
Contributor

svadams (svadams) commented Jun 3, 2026

Developer tests pass and this branch has been tested with corresponding lfric_core branch from MetOffice/lfric_core#352

@svadams svadams (svadams) merged commit da3ef63 into MetOffice:main Jun 3, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@svadams svadams (svadams) svadams approved these changes

@harry-shepherd Harry Shepherd (harry-shepherd) harry-shepherd approved these changes

@thomasmelvin thomasmelvin thomasmelvin approved these changes

@cjohnson-pi cjohnson-pi cjohnson-pi approved these changes

@DrTVockerodtMO DrTVockerodtMO DrTVockerodtMO approved these changes

@DanStoneMO DanStoneMO DanStoneMO approved these changes

@tommbendall Thomas Bendall (tommbendall) Awaiting requested review from tommbendall tommbendall is a code owner

@jameskent-metoffice James Kent (jameskent-metoffice) Awaiting requested review from jameskent-metoffice jameskent-metoffice is a code owner

@christophermaynard Hacka Fett (christophermaynard) Awaiting requested review from christophermaynard christophermaynard is a code owner

@HengistPodd Richard Hill (HengistPodd) Awaiting requested review from HengistPodd HengistPodd is a code owner

@mo-lottieturner Lottie Turner (mo-lottieturner) Awaiting requested review from mo-lottieturner mo-lottieturner is a code owner

@ss421 Steven Sandbach (ss421) Awaiting requested review from ss421 ss421 is a code owner

@matthewrmshin Matt Shin (matthewrmshin) Awaiting requested review from matthewrmshin matthewrmshin is a code owner

Assignees

@mo-marqh mo-marqh

Labels

Linked Core This PR is linked to a MetOffice/lfric_core PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@mo-marqh @harry-shepherd @svadams @thomasmelvin @cjohnson-pi @DrTVockerodtMO @DanStoneMO @mo-ssd-bot