Minutes of the code management group meeting 2018-09-05

Teleconference Details

Wednesday 5th September 2018, 10am - 12noon BST

UK Landline: +44 (0)1223 761376 (x61376 from an internal University of Cambridge phone)
Participant PIN: 323180

Present

Met Office: John Hemmings (JH), Mohit Dalvi (MD), Fiona O’Connor (FOC), Lucy Neal (LN), Alan Hewitt (AJH), Marc Stringer (MS), Adrian Hill (AH), Jane Mulcahy (JM), Maff Glover (MG)

Cambridge: Luke Abraham (LA), Alex Archibald (ATA)

Leeds: Chris Dearden (CD), Kirsty Pringle (KP), Mark Richardson (MR)

Reading:

Lancaster: Ryan Hossaini (RH), Ewa Bednarz (EB)

Edinburgh:

UEA:

NIWA: Olaf Morgenstern (OM), Guang Zeng (GZ)

CSIRO:

Apologies: GM, NB, JK, SAN

Actions from this meeting

One-off actions

(ongoing) MY to check GLOMAP-mode code and report any missing bug fixes on the bugfixes page.
(ongoing) LA to liaise with MY, and to chase MY and GM as to when this would be done.
(ongoing) LA to migrate PUMA UM tickets to MOSRS - see table below.
(ongoing) LA to open tickets for all changes that still require a ticket.
(ongoing) FOC to recommend at next Exec meeting a policy of allowing only CMG to (certify and) distribute any UKCA configurations for science purpose.
(ongoing) MD to collate all UKCA parameters into ukca_constants in advance of removing duplicates already available in the UM and circulate details.
(ongoing) LA to put C_SPECIES definitions in the Redundant Code table (short-term), and switch to use M_SPECIES instead. Talk with JH about AQUM tests and requirements. Note that there will be problems with exoplanets and >100km top configs where M_air not constant.
(new) AJH to open ticket and assign to himself regarding loops over imode+n and imode-n.
(ongoing) MD/LA to talk about releases offline, and set up free-running and nudged GA7.1+StratTrop TS2000ish timeslice, with instructions to make it transient.
(new) NLA/MD make script for perpetual nudging (symbolic linking to existing files).
(new) NLA to port suite to ARCHER.
(ongoing) LA to create Wiki page on UKCA website for users to add details of configurations and input files.
(ongoing) ALL to review output from vn11.0 release candidate(s).
(new) MD to provide vn11.0 release candidate output again.
(ongoing) FOC to circulate doodle poll for further emissions discussion to CMG for all to fill-in. That meeting to report to CMG
(ongoing) MR to discuss with KP implications for GLOMAP in regards to a separate UKCA repository
(new) NLA to contact CEJ (cc JM) about trace-gas convective plume scavenging and how this will be impacted (or impact) the repository work
(ongoing) LA to contact FOC, AH, MR, KP, ATA, Ken Carslaw, Philip Stier regarding taking the proposal for a separate code repository forward.
(ongoing) FOC to see what action can be taken from MASS, then put out an email for POCs from institutes.
(new) FOC/AH/MD to develop a strategy on MASS for discussion at next CMG
(new) FOC to contact Garry Hayman to highlight bug #35.
(new) JM to take ownership of redundant code item #6 (Wet oxidation of SO2 in GLOMAP).
(new) LN to document redundant code item #9 (Deprecated UKCA netcdf attribute options).
(new) JM to do a catch-call ticket for GMED
(new) LN to l_fix_nh4no3_equilibrium in rose-stem and jobs and remove if necessary.
(new) NLA to do doodle poll for 90 minute meeting in first 2 weeks in November.

Standing actions

(ongoing - standing item) CMG to monitor list of bugs and circulate between group for owners.
(ongoing - standing item) At CMG future meetings there should be included an appendix listing the open bugs. If a ticket is included it is not an open bug (as the owner of ticket is person doing lodging).
(ongoing - standing item) ALL with access to keep monitoring ukca_bugs email list.
(ongoing - standing item) ALL to think about if code changes will change KGO and let systems team know in good time, and remember to include the UKCA & SC0138 keywords to all UKCA tickets. This should be a standard part of our working practice. Note also the required keywords.
(ongoing - standing item) ALL to note, review, and remove obsolete/redundant code
(ongoing - standing item) ALL closed ticket owners to produce a summary of their change for the UKCA newsletter
(ongoing - standing item) Code Owner to produce draft newsletter for CMG to approve.

Minutes

Actions from the Last Meeting (LA)

Minutes of the code management group meeting 2018-06-19

One-off actions

(ongoing) MY to check GLOMAP-mode code and report any missing bug fixes on the bugfixes page.
Keep ongoing. This should be done later in the year.
(ongoing) LA to liaise with MY.
Keep ongoing. LA to chase MY and GM as to when this would be done.
(ongoing) LA to migrate PUMA UM tickets to MOSRS - see table below.
Keep ongoing.
(ongoing) LA to open tickets for all changes that still require a ticket.
Keep ongoing.
(ongoing) FOC to recommend at next Exec meeting a policy of allowing only CMG to (certify and) distribute any UKCA configurations for science purpose.
Keep ongoing.
(ongoing) MD to collate all UKCA parameters into ukca_constants in advance of removing duplicates already available in the UM and circulate details.
The UM is moving its constants to SHUMlib, but this changes results. Keep ongoing.
(new) LA to put C_SPECIES definitions in the Redundant Code table (short-term), and switch to use M_SPECIES instead. Talk with JH about AQUM tests and requirements.
Note that there will be problems with exoplanets and >100km top configs where M_air not constant. Keep as ongoing.
(ongoing) AJH to scope out reducing repetition of mode definitions in GLOMAP. See UM:#4097
Close. Ticket under review.

AJH has since found some parts of code with loops around imode-n and imode+n, and this will probably need to be done in a more clever way. (e.g. mask with soluble modes etc.).

ACTION: AJH to open ticket and assign to himself.
(ongoing) MD/LA to talk about releases offline, and set up free-running and nudged GA7.1+StratTrop TS2000ish timeslice, with instructions to make it transient.
Keep ongoing.

MD has a nudged job on long run at the moment, but it is not possible to define perpetual year in the nudging within Rose.

ACTION: NLA/MD make script for perpetual nudging (symbolic linking to existing files).

ACTION: NLA to port suite to ARCHER.

MD has a ticket details page and will move to UKCA website. Costs estimates for ARCHER and Monsoon2/NEXCS should be include as done for vn8.4.

Have ready by start of October.
(ongoing) LA to create Wiki page on UKCA website for users to add details of configurations and input files.
Keep ongoing.
(ongoing) ALL to review output from vn11.0 release candidate(s).
Keep ongoing.

ATA highlighted that nothing is out of the ordinary, but STE is low.

AOD diagnostics and auto-assess output is missing.

ACTION: MD to provide output
(ongoing) MG to progress ticket UM:#3909 after doing further tests.
Close

This will be superseded by new Cray compiler - MG to link to ticket for new compiler.

OM highlighted that the NIWA XC50 has to use ifort as the CCE on XC50 doesn’t compile the UM.

It was noted that NCAS-CMS are having similar issues on ARCHER.
(new) ALL to feedback to AJH by 2nd July regarding UKCA Rose panel changes
Close
(new) FOC to ask Paul Agnew to send AQUM representative to CMG
Close (LN)
(new) FOC to circulate doodle poll for further emissions discussion to CMG for all to fill-in. That meeting to report to CMG
Keep ongoing.
(new) MR to discuss with KP implications for GLOMAP.
Keep ongoing - this is in regards to a separate UKCA repository

ACTION: NLA to contact CEJ about trace-gas convective plume scavenging, cc JM
(new) LA to contact FOC, AH, MR, KP, ATA, Ken Carslaw, Philip Stier regarding taking the proposal for a separate code repository forward.
ongoing, NLA to circulate again. Feedback has been positive from those contacted.
(new) FOC to see what action can be taken from MASS, then put out an email for POCs from institutes.
Keep ongoing.

FOC and AH have agreed to be joint owners for users outside of MO and will split institutes between them. They need to talk to MASS team.

ACTION: FOC/AH/MD to develop a strategy on MASS for discussion at next CMG
(new) LA to do doodle poll for weeks 1-2 of September 2018.
Close

Standing actions

(ongoing - standing item) CMG to monitor list of bugs and circulate between group for owners.
(ongoing - standing item) At CMG future meetings there should be included an appendix listing the open bugs. If a ticket is included it is not an open bug (as the owner of ticket is person doing lodging).
(ongoing - standing item) ALL with access to keep monitoring ukca_bugs email list.
(ongoing - standing item) ALL to think about if code changes will change KGO and let systems team know in good time, and remember to include the UKCA & SC0138 keywords to all UKCA tickets. This should be a standard part of our working practice. Note also the required keywords.
(ongoing - standing item) ALL to note, review, and remove obsolete/redundant code
(ongoing - standing item) ALL closed ticket owners to produce a summary of their change for the UKCA newsletter
(ongoing - standing item) Code Owner to produce draft newsletter for CMG to approve.

Bugs/Bugfixes (LA)

Open Bugs

#	Description	Ticket	Owner	Status/ Updates	Comments from previous meeting	Comments from this meeting
35	Error in labelling: Surface resistance <--> stomatal resistance ?	UM:#1409	Fiona O'Connor		Leave open. Garry Hayman started work in offline JULES.	Still ongoing. See action.
74	Bug in ukca_tropopause with definition of PV tropopause	UM:#1283 New ticket UM:#1367	Mohit Dalvi	Fix to be combined with Bug 75 below MOSRS: vn10.6_ukca_tropopause_amendment		Should make 11.2
75	Bug in STE fluxes	UM:#1367	Mohit Dalvi	Fix branch: vn11.1_ukca_fix_o3_ste will also include tropopause fix		Should make 11.2
87	Possible issues with YSTORE in UKCA_CHEMISTRY_CTL.	UM:#3245	Colin Johnson / Luke Abraham		Leave open.	Leave open. NLA to have another look. MS may have changed things with OMP changes.
96	Output drydiam diagnostics where l_ukca_arg_act but not l_ukca_radaer	UM:#3962	Alan J Hewitt	The beginnings of a solution are found in vn11.0_ukca_drydiam_activate_wo_radaer.	Leave open. AJH to take ownership - may be affected by reorganisation.	Leave open.
98	Incorrect usage of `npft` integer when `npft == 9 OR 13`	UM:#4157	Alan J Hewitt / Garry Hayman	To be lodged at vn11.2 Follow-on ticket to remove temporary logical (l_fix_ukca_h2dd_x) UM:#4245	NA	Under SciTech review. Logical will be removed after GA9.

ACTION: FOC to contact Garry Hayman to highlight bug #35.

Redundant Code (LA)

Code for deletion

PLEASE REMEMBER TO EDIT THE ORIGINAL REDUNDANT CODE TABLE AND NOT THIS PAGE

#	Description	Ticket	Remarks	Status	Timeframe	Owner	Comments from this meeting
2	Old dry deposition scheme	UM:#2356	None	PROPOSED	Medium-term	Nick Savage
4	ukca_strat_update.F90 - relax_ozone	UM:#1505	None	PROPOSED	Medium-term	Luke Abraham
5	Backward Euler Standard Tropospheric Chemistry	UM:#1507	None	PROPOSED	Medium-term	Fiona O'Connor
6	Wet oxidation of SO2 in GLOMAP	None	None	PROPOSED	Medium-term	Colin Johnson	JM to look at this re: UKESM1 c.f. offline GLOMAP. JM to take ownership.
7	Updraft velocity - l_conv_tke	UM:#3668	Proposed in December 2017. Approval being sought for GA series (likely GA8).	PROPOSED	Medium-term	Alan J Hewitt / Ian Boutle
8	All backward Euler chemistry schemes	None	Note that this is only for the “hard-coded” BE schemes. Quite a lot of work would be involved, but good as a longer-term aim. Note that problems for AQUM would need to be addressed first.	PROPOSED	Long-term	Luke Abraham
9	Deprecated UKCA netcdf attribute options	UM:#2513	'emission_type', string type attributes no longer recommeneded. Need some time for most new code to adopt these. Still used in AQUM files.	DEPRECATED	Medium-term	Nick Savage	AQUM are addressing the files. Have gone from operational jobs, have ticket to remove from rose-stem. LN to document.
10	ASAD families chemistry	None	Families chemistry has not been used for many years (since at least UM6.1. It is likely that it no longer works if turned on, and it complicates the code making other developments harder	DEPRECATED	Short-term	Luke Abraham
11	Dry deposition 9 PFT option	None	The 13 PFT option has been adopted for UKESM1. The 9 PFT option is possibly not being used at all. Proposed in June 2018.	PROPOSED	Long-term	Alan J Hewitt
12	Remove explicit `C_SPECIES` constants and calculate these from `M_SPECIES` and the mass of air	None	Will have wide-ranging rose-stem KGO breakages	PROPOSED	Short-term	Luke Abraham
13	Remove temporary logical l_fix_nacl_density Introduced at UM:#2405 by Colin Johnson	UM:#3490	Requires AMIP testing and a GMED ticket Could be accepted to GA9	PROPOSED	Medium-term	None	See JM GMED action
14	Remove temporary logical l_fix_nh4no3_equilibrium Introduced at UM:#646 by Carlos Ordonez	None	Requires AMIP testing and a GMED ticket Could be accepted to GA9	PROPOSED	Medium-term	None	See LN temporary logical action
15	Remove temporary logical l_fix_ukca_impscav Introduced at UM:#1421	UM:#4145	Approval being sought for GA series (likely GA9) GMED:#402	PROPOSED	Medium-term	Alan J Hewitt	See JM GMED action
16	Remove temporary logical l_fix_improve_drydep Introduced at UM:#1250	UM:#3997	Approval being sought for GA series (likely GA9) GMED:#393	PROPOSED	Medium-term	Alan J Hewitt	See JM GMED action
17	Remove temporary logical l_fix_ukca_h2dd_x Introduced at UM:#4157	UM:#4245	Approval being sought for GA series (likely GA9) GMED:#410	PROPOSED	Medium-term	Alan J Hewitt	See JM GMED action

ACTION: JM to take ownership of redundant code item #6 (Wet oxidation of SO2 in GLOMAP).

ACTION: LN to document redundant code item #9 (Deprecated UKCA netcdf attribute options).

ACTION: JM to do a catch-call ticket for GMED

ACTION: LN to l_fix_nh4no3_equilibrium in rose-stem and jobs and remove if necessary.

UKCA Release Job (LA/MD)

See previous comments under actions from the last meeting.

Targets for vn11.2 (and beyond) (LA)

PLEASE REMEMBER TO INCLUDE THE KEYWORDS: UKCA & SC0138

Tickets targeting vn11.2

UM vn11.2 code review submission deadline is 10th Oct 2018. SOCRATES/JULES/GCOM/KGO by 3rd Oct 2018.

#	Description
UM:#1367	Fix bug in Stratosphere-Troposphere Exchange fluxes of UKCA tracers
UM:#2689	Enable coupling of GLOMAP-MODE Aerosol to FAST-JX photolysis.
UM:#3127	Investigate KGO differences in um_n48_ukca_eg app using GNU on the Cray
UM:#3292	Consolidate constant variables used by UKCA
UM:#3342	Update of UKCA heterogeneous chemistry routine
UM:#3367	Removing transport of peroxy radical tracers in Strattrop (CHeST)
UM:#3673	Fail-if tests for CH4 and N2O in radiation panel for certain configurations of UKCA
UM:#3769	Time updating for glomap climatology
UM:#3779	Improvements to the inline comments in ukca_option_mod
UM:#3959	RO2-Permutation Chemistry
UM:#3970	Tidy the UKCA gui panel
UM:#4001	Magic numbers in ukca_setup_chem_mod.F90
UM:#4020	Improved calculation of ozone dry deposition to ocean - surface resistance term
UM:#4032	Addition of new diagnostics for the wet and dry deposition flux of oxidised nitrogen
UM:#4097	Remove repetition of mode definitions and some magic numbers
UM:#4131	Enable use of ASAD Backward-Euler solver with UKCA offline oxidants chemistry
UM:#4157	Bug in surface resistance on tiles (H2) for npft/=5 in ukca_surfddr
UM:#4202	Introduce UKCA variable 'nrsteps' as a Rose namelist variable
UM:#4212	Add new local variables to prevent memory striding in call to asad_cdrive

Tickets targeting vn11.3

UM vn11.3 code review submission deadline is 7th Feb 2019. SOCRATES/JULES/GCOM/CASIM/KGO by 31st Jan 2019.

#	Description
UM:#3962	Output dry diameter diagnostic in niche case where l_ukca_arg_act but not l_ukca_radaer
UM:#3997	Retire temporary logical l_fix_improve_drydep
UM:#4024	Triggering of i_ukca_dms_flux by l_sulpc_so2
UM:#4145	Retire temporary logical l_fix_ukca_impscav
UM:#4201	Verify use of qsmin in UKCA plume scavenging diagnostics
UM:#4221	Optimisation of GLOMAP_CLIM
UM:#4231	Adding Common Representatives Intermediate (CRI) mechanism
UM:#4245	Retire temporary logical l_fix_ukca_h2dd_x

"Work-stealing" between day and night points (MS)

Michele Guidolin from the HPC Optimisation team at the MO is implementing work stealing into the SW radiation in the UM, and I think it's something which might work for UKCA. From memory, I think Michele's work on SW radiation has produced around a 5% speed-up to an AMIP job.

As I understand for the SW radiation, the following is done for work-stealing

Pair up PEs on opposite sides of the globe, so you always have a ‘PE in light’ and a ‘PE in dark’
The ‘PE in light’ has more segments of SW radiation work to do than the ‘PE in dark’ and so you determine a number of segments of work to be sent from the ‘PE in light’ to the ‘PE in dark’ (this is the ‘work-stealing’).
The segments are passed between the ‘PE in light’ and ‘PE in dark’ using MPI

Now for the chemistry, I have in my head that it’s a general rule that much more chemistry is done in the light than in the dark – as is the case for the SW radiation. However, it if turns out that there’s always a lot of chemistry in say the south pole then we might have to re-think in what direction the work stealing is done. We would need some better analysis of the imbalance in the chemistry work, than I currently have, before starting anything like this. But for now let’s assume that I’m broadly right that there’s more chemistry done in the light than in the dark.

Unlike the SW radiation, where it seems that the ‘PE in light’ has more segments of work to do than the ‘PE in dark’, the segments of work for ‘PE in light’ and ‘PE in dark’ are the same. The difference is that the work for ‘PE in dark’ can be done much quicker, because the chemistry reactions turn out to be simpler and quicker.

The work stealing might work for both chemistry and aerosol, but I’ll continue with the chemistry solver for now as it’s more expensive and I know more about it. The top routine for the chemistry solver, which UKESM uses, is UKCA_CHEMISTRY_CTL. This applies OpenMP over the vertical levels. Here we might consider sending say 10 levels from the ‘PE in light’ to the ‘PE in dark’ to calculate, so that the ‘PE in light’ would calculate 75 levels and the ‘PE in dark’ would calculate 95 levels. However, sending and receiving all the information from the ‘PE in light’ to the ‘PE in dark’ in this routine is probably going to be very complicated and require a lot of code. It might be better to use the alternative to UKCA_CHEMISTRY_CTL which is UKCA_CHEMISTRY_CTL_COL - especially if Chris adds segments into this.

Discussion

FOC highlighted that a similar scheme to this was done for FastJ on the old NEC machine by a consultant. This code could be revived.

MR believes that following CDs work on chunking a lot of these numbers could be quantified.

It was agreed for CD to continue with chunking, and re-visit work stealing when that work is completed and do an assessment then.

AOB/DNM (LA)

Date of next meeting (LA)

ACTION: NLA to do doodle poll for 90 minute meeting in first 2 weeks in November.