Trouble with drv_flds_in namelists for some CAM cases using ctsm5.2.027

[ekluzek]

Brief summary of bug

Some compsets with CAM and CLM versions other than CAM7*CLM60 fail with a namelist error in the drv_flds_in namelist.

General bug information

CTSM version you are using: ctsm5.2.027
Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: With CAM compsets other than CAM7 with CLM60

Details of bug

An example test that fails is:

SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4

Important details of your setup / configuration so we can reproduce the bug

Originally discussed by @fvitt here:

/glade/work/fvitt/camdev/dust_emissions

Which includes the entire lists of tests that fail.

Important output or errors that show the problem

Create namelist for component clm
   Calling /glade/work/fvitt/camdev/dust_emissions/components/clm//cime_config/buildnml
ERROR: Command /glade/work/fvitt/camdev/dust_emissions/components/clm/bld/build-namelist failed rc=255
out=
err=ERROR : CLM build-namelist::CLMBuildNamelist::process_namelist_infile() : Invalid namelist variable in '-infile' /glade/derecho/scratch/fvitt/aux_cam_intel_20240903232235/ERP_Ld3.f09_f09_mg17.FWHIST.derecho_intel.cam-reduced_hist1d.GC.aux_cam_intel_20240903232235/Buildconf/camconf/drv_flds_in.
 ERROR: in _validate_pair (package Build::NamelistDefinition): Variable name ndep_list not found in /glade/work/fvitt/camdev/dust_emissions/components/clm/bld/namelist_files/namelist_definition_drv.xml, /glade/work/fvitt/camdev/dust_emissions/components/clm/bld/namelist_files/namelist_definition_drv_flds.xml, /glade/work/fvitt/camdev/dust_emissions/components/clm/bld/namelist_files/namelist_definition_ctsm.xml'
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/test_scheduler.py", line 1125, in _run_catch_exceptions
    return run(test)
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/test_scheduler.py", line 1014, in _setup_phase
    expect(
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/utils.py", line 176, in expect
    raise exc_type(msg)

[ekluzek]

Note a different problem for the above test includes this error regarding megan specifiers:

Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
Key: megan_specifier, 
 Value 1: 'CH2O = formaldehyde', 'CO = carbon_monoxide', 
 Value 2: 'CH2O = formaldehyde','CO = carbon_monoxide'
ERROR: incompatible settings in drv_flds_in from 
 /glade/derecho/scratch/fvitt/aux_cam_intel_20240903232235/SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4.GC.aux_cam_intel_20240903232235/Buildconf/camconf/drv_flds_in 
 and 
 /glade/derecho/scratch/fvitt/aux_cam_intel_20240903232235/SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4.GC.aux_cam_intel_20240903232235/Buildconf/clmconf/drv_flds_in'
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/test_scheduler.py", line 1125, in _run_catch_exceptions
    return run(test)
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/test_scheduler.py", line 1014, in _setup_phase
    expect(
  File "/glade/work/fvitt/camdev/dust_emissions/cime/CIME/utils.py", line 176, in expect
    raise exc_type(msg)

This possibly should be it's own issue. I'll look into it first.

[ekluzek]

Running this test:

SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4

with LND_SETS_DUST_EMIS_DRV_FLDS=FALSE

The problem I see is this:

Create namelist for component drv
Calling /glade/derecho/scratch/erik/cam_dust_emis/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
Key: megan_specifier,
Value 1: 'CH2O = formaldehyde', 'CO = carbon_monoxide',
Value 2: 'CH2O = formaldehyde','CO = carbon_monoxide'
ERROR: incompatible settings in drv_flds_in from
/glade/derecho/scratch/erik/cam_dust_emis/cime/scripts/SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4.20240904_134310_17s4aa/Buildconf/camconf/drv_flds_in
and
/glade/derecho/scratch/erik/cam_dust_emis/cime/scripts/SMS_D_Ln9.f19_f19_mg17.FWma2000climo.derecho_intel.cam-outfrq9s_waccm_ma_mam4.20240904_134310_17s4aa/Buildconf/clmconf/drv_flds_in

Which is actually a bug in CMEPS for comparing drv_flds_in namelists.

[ekluzek]

On the CTSM side the problem is mentioned in #2713 as we need to make sure --no-megan is set and the other drv_flds_in build-namelist command line options are NOT turned on.

[ekluzek]

The namelist merge issue in CMEPS is now documented here:

ESCOMP/CMEPS#509

It'll require a cmeps update in CTSM. So I'll include it with the externals update issue #2640.

[wwieder]

@ekluzek will check on this an move it to todo or done columns

[ekluzek]

I thought this might be done, but it's not. We just had a workaround to prevent it from happening in the cases we needed. But, we still want at least a simple fix in for usability. The fix is really in CMEPS, and then to update the CMEPS version in CTSM. It can be a lower priority though.