2008/08/15 esmp tag: 8 model yrs/day (2.5 hrs/model yr) ------------------------------------------------------------------------------------------------------------------------------- Job Name Status Notes ------------------------------------------------------------------------------------------------------------------------------- ESM2.1 ****** ESM2.1U_Control-1860_perth_mmmddyyyy pending perth + mods for co2, o2, topaz diags .... ESM2.1U_Control-1990_lm2_jun182008 crashed Equivalent to CM2.4 hi-res + topaz. passed regression tests (after fixing bug in topaz). production run failed: sat vap pressure overflow ESM2.1U_Control-1860_O_2008_03_stocks_test1 to be redone in Perth ESM2.1U_Control-1860_dec192007_accelerate_samesign crashed 0119 start from year 50. crashed year 60 - no space on device. Resubmitted. system failover stopped model at year 120. Try to resubmit from year 119. check script ESM2.1U_Control-1860_dec192007_accelerate_linear crashed 0090 start from year 50. crashed in year 90. restarted from 0089 - crashed in year 90 again. ............................................................................................................................................................... ESM2M ***** ESM2M_Control-1860_generic (aug142008) 0008 Perth + opacity bugfix ESM2M_Control-1860_non_generic (aug142008) 0006 Perth + opacity bugfix ESM2M_Control-1860_generic_preP_jul212008 prePerth + updates: failed restart test ESM2M_Control-1860_non_generic_preP_jul212008 prePerth + updates: failed restart test ESM2M_Control-1860_preP_generic_jun232008 crashed passed regression tests (irr_mem restart quirk) unable to run 1 month (try fixing diag table) changed domains_stack_size to 5000000, but did not help problem is total SW < 0 -> fvis < 0 -> neg sq root in ocean_shortwave. temporary fix from Niki - need fix from Mike W in cie model. ESM2M-coarse check with Eric moved to testing tag. compilation failed. start from omsk_2008_03: compiled ok. regression tests ok. try to make debug executable: compile failed. got updated mkmf debug template - compile ok. ............................................................................................................................................................... ESM2G ***** ESM2G_realepi_generic_no_diffuse 0002 1860. Perth + realepi. undef diffuse_ml_to_interior ESM2G_realepi_non_generic_no_diffuse 1860. Perth + realepi. undef diffuse_ml_to_interior ESM2G_generic_chl_true 0007 1860. prePerth + updates. read chl from file. jul152008 ESM2G_non_generic_chl_true 0009 1860. prePerth + updates. read chl from file. jul152008 ............................................................................................................................................................... OCEAN_ICE ********* ............................................................................................................................................................... ESM2.4 ****** CM2.4SQ_A_Control-1990_B5_O_2008_03_topaz_3months crashed bscip OBGC IC created. ran 1 month, but no output ?? crashed (sat vap pres overflow) with dt_cpld etc = 1800. changed to 1200 and submitted 3 year run. (if crash, check obgc IC - glodap) (1 month run ran successfully with dt_cpld etc = 1200). Model blew up in TOPAZ in Sept of year 3. Save files and restart from July. Run 2 month segments. 2008/06/23 Model did not show problem in Sept. Still running ============================================================================================================================================ COMPLETED/STOPPED JOBS ESM2.1 ------ ESM2.1U_Control-1860_perth (jul312008) done test perth release. regression tests passed ESM2.1U_Control-1860_preP_jul182008 done prePerth + updates (lm3v update from 11/07 + other k_fw nml change) regression tests passed ESM2.1U_Control-1860_jun162008 done test co2_dry mods and minor change to topaz. fixed bug in topaz: regression tests passed. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008 1000 done stopped at year 200, restarted with equilibrated soil C obgc changed at year 263. stopped at year 357 to recreate year 330 history file. ESM2.1U_Control-1860_dec192007_free_co2_v2_fixed2 done 0401 started from year 400. Run 1 year only with prescribed co2. Use year 400 restart from dec 19 spinup with no changes. compare to year 401 of dec19 spinup: identical ESM2.1U_Control-1860_dec192007_free_co2_v2_fixed done 0401 started from year 400. Run 1 year only with prescribed co2. Use year 400 restart from dec 19 spinup with changes for free co2 run (no qco2, atm co2 = 286/(1+q)) compare to year 401 of dec19 spinup: co2 and qco2 different. ESM2.1U_Control-1860_dec192007_free_co2_v2 done 0705 started from year 400. changed atmco2 IC = 286e-6/(1+q), added co2_dry diagnostic year 414 history file is bad. recreated from raw cpio year 518 history file is bad. could not recreate from raw cpio may 16: stopped at year 525 to recreate year 518 history file. may 22: crashed at year 530 - bad temperatures (sat vap press) year 528 history bad. restarted from beginning of year 528. year 529 history/restart did not compare. 2008/06/10 stopped run at year 0544. restarted from year 0518. ESM2.1U_Control-1860_dec192007_free_co2_plus_rad done 0427 started from year 400. stopped at year 427. Check atmco2 IC. co2 units likely incorrect (dry vs moist) ESM2.1U_Control-1860_dec192007_free_co2_qco2_diag done 0481 started from year 400. stopped at year 481. Check atmco2 IC. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_free_co2 done 0416 started from year 400. stopped (atmos_co2 was not reset). ESM2.1U_Control-1860_dec192007_dup_qco2_diag done regression tests passed compared to dec19 run. 1 year production run: restart and hist files identical to dec19 run. ESM2.1U_Control-1860_apr242008 done identical to mar202008 - (updated obgc). passed regression tests. restart files identical to mar202008 tests. ESM2.1U_Control-1860_mar202008 done updated obgc. scaling test failed. scaling test resubmitted - passed. NOTE: repeat regression tests after upgrade. ESM2.1U_Control-1860_O_2007_10_historical_co2_phot_p done 1885 started from year 1860. (version1 stopped at year 1865) version2 stopped at year 1885 - check atmco2 IC. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb142008 done 0024 stopped at year 25. Start new spinup. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007 done 0504 stopped at year 504 ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_ap10_af5 crashed 0110 started from year 50. crashed at start of year 110 fix pending - try different acceleration instead ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_test4 done 0059 started from year 50. stopped at year 0059. ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_accelerate_test3 done 0139 started from year 50. Stopped at year 139. ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_anderson95 done 0100 stopped at year 100 ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_chl_true done 0100 stopped at year 100 ESM2M (ESM2.1M) --------------- ESM2M_Control-1860_preP_generic_jun232008_IC2 done use same IC as non_generic case, corrected diag table this was a check to see if IC was cause of crash ESM2M_Control-1860_preP_non_generic_jun232008 done passed regression tests (irr_mem restart quirk) can run for 3 months ESM2M_Control-1860_P_jun112008 crashed esmd test ESM2M generic topaz + cfc modify TOPAZ IC to include new tracers (eg sidet) om3_ecosystem_mdfl_generic crashed esmd test generic topaz. Nan in year 1, month 4. Nan in 1990/04/07. try changing ice_dhdt ESM2.1Mc_om1p7h_1860_O_2008_03_lm2_mar212008 failed restart test, but checksums okay. ESM2.1Mc_om1p7h_1860_O_2008_03_lm3v_mar212008 not run ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p1_jan172008 stopped 0008 (Suspended with frepriority 01/22/08) ESM2G ----- ESM2G_generic done 0020 1860. land cold start. prePerth + updates fixed xml/runscript to pick up HIM.res.nc ESM2G_non_generic done 0020 1860. land cold start. prePerth + updates fixed xml/runscript to pick up HIM.res.nc ESM2G_Control-1860_preP_non_generic_jun242008 passed restart test. scaling test passed Note: TOPAZ fields incorrect problem is that HIM.res.nc IC is not picked up and model initializes internally. ESM2G_Control-1860_preP_generic_jun242008 passed restart test. scaling test failed. Note: TOPAZ fields incorrect problem is that HIM.res.nc IC is not picked up and model initializes internally. ESM2G_Control-1860_P_jun112008 crashed test ESM2G generic topaz + cfc OCEAN_ICE --------- mom4p0_sis_ciafv2_O_2007_10_diurnal_true_dec192007_extend done 2235 extend control run, use new ciaf forcing started from year 2281. mom4p0_sis_ciafv2_O_2007_10_diurnal_true_dec192007_pertco2_extend done 2235 extend perturbed run, use new ciaf forcing started from year 2281. mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2 done 2282 started from year 2136, stop at year 2280 problems with co2_history file. (time attributes) - fixed ice_dhdt=40 did not work. changed to 260, then 280. NOTE: time_interp crash - check on this. resubmitted job - ran despite time_interp crash model ran to year 2152, but system crashed 2008/04/17 model started over from year 2136 (see stdout file). mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_half done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_2x done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_5x done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_fed_flux_new done 2188 started from year 2097, ice_dhdt=260, stop at year 2188 mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_p2nstatic_true done 2320 stop at year 2320 crash in year 2321 - did not change dhdt to run longer mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007 done 2320 1959-2004, 2050, 2096, 2142, 2188, 2234, 2280, 2326 stopped Feb 11 at year 2321 (increase dhdt > 400 to keep going) om3_ecosystem1_jan142008 stopped 0100 8 model yrs/day stopped Feb 13 at year 100 mom4p1_cpld_om1p7_spinup ---- ============================================================================================================================================ CIAF model runs 46 year loops (skip 1958) sfc diags for last 7 years of loop (eg 1998-2004) - modify pp section of xml as needed ocean_topaz_sfc = yrend-6 ice_dhdt changes after crashes loop 1 1959-2004 1998 start at 160, change to 200 at 2004/06/30 loop 2 2005-2050 2044 change to 240 after 2045/06/30 loop 3 2051-2096 2090 change to 260 after 2091/06/30 loop 4 2097-2142 2136 change to 280 after 2137/06/30 loop 5 2143-2188 2182 change to 300 after 2183/06/30 loop 6 2189-2234 2228 change to 310 after 2229/06/30 loop 7 2235-2280 2274 change to 350 after 2238/10/28, change to 400 after 2275/06/30 loop 8 2281-2326 2320 crash in year 2321 - did not change dhdt to run longer. ============================================================================================================================================ lst $archive/omsk/ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008/history/ lst $archive/omsk/ESM2.1U_Control-1860_dec192007_free_co2_v2/history lst $archive/omsk/mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2/history/ myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_feb192008.xml ESM2.1U_Control-1860_O_2007_12_lm3v_mom4p0_spinup_feb192008 myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007_free_co2.xml ESM2.1U_Control-1860_dec192007_free_co2_v2 myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007.xml ESM2.1U_Control-1860_dec192007_accelerate_samesign myfreppcheck -A -x ~/omsk/rts/ESM2.1U_Control-1860_O_2007_10_lm3v_mom4p0_dec192007.xml ESM2.1U_Control-1860_dec192007_accelerate_linear myfreppcheck -A -x ~/omsk/rts/mom4p0_sis_topaz_ciaf_diurnal_O_2007_10_dec192007.xml mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_pertco2 myfreppcheck -A -x ~/omsk/rts/mom4p0_sis_topaz_ciaf_diurnal_O_2007_10_dec192007.xml mom4p0_sis_ciaf_O_2007_10_diurnal_true_dec192007_p2nstatic_true myfreppcheck -A -x mom4p1_cpld_jan142008_v2.xml om3_ecosystem1_jan142008 ============================================================================================================================================