Difference between revisions of "Testing Requirements for UKCA Code Changes"

From UKCA
 
(5 intermediate revisions by 3 users not shown)
Line 23: Line 23:
 
This suite allows you to test your changes with logicals on and includes thread and bit-comparability tests, as well as a rigorous test. These jobs only run for 2 model days (6 hours for rigorous) and are useful for making initial developments.
 
This suite allows you to test your changes with logicals on and includes thread and bit-comparability tests, as well as a rigorous test. These jobs only run for 2 model days (6 hours for rigorous) and are useful for making initial developments.
   
'''As there are no StratTrop or CRI-Strat rigorous tests under rose-stem, so it is important that this is done using this suite as it fulfils the requirement to test changes under rigorous'''.
+
'''As there are no CRI-Strat rigorous tests under rose-stem, so it is important that this is done using this suite as it fulfils the requirement to test new changes under rigorous'''.
   
 
You can link to the Rose Bush output on your ticket, and newer suites produce a trac.log file.
 
You can link to the Rose Bush output on your ticket, and newer suites produce a trac.log file.
Line 40: Line 40:
 
You may also want to make use of valnote/autoassess or Maverick to produce a set of standard plots as well.
 
You may also want to make use of valnote/autoassess or Maverick to produce a set of standard plots as well.
   
It is best to be in close contact with your SciTech Reviewer as they may request additional diagnostics/plots for the ticket.
+
It is best to be in close contact with your SciTech Reviewer as they may request additional diagnostics/plots for the ticket.
   
 
== AQUM ==
 
== AQUM ==
  +
A rose suite using the [https://code.metoffice.gov.uk/trac/nwpscience/wiki/ResearchSuites/AQUM [[AQUM standard job]]] is maintained at each UM version. The operational model is designated "aqeur" and the latest suite can be found at the top of the table on the page. The testing process for AQUM consists of the following steps:
  +
  +
1. Copy the latest aqeur rose suite.
  +
  +
2. Select the period you wish to run for (see below) in the suite.
  +
  +
3. Carry out a control run.
  +
  +
4. Edit the suite to include your UM branch in the UM compilation.
  +
  +
5. Verify the results, comparing the test and control runs. Standard verification rose suites are maintained on the same link given earlier. For guidance on using the suite and interpreting the output please contact the AQUM CMG representative.
  +
  +
'''Case study period selection''' Depending on the change being made it may be desirable to choose a case study period which coincides with an episode of elevated pollution. Please discuss your change with the AQUM CMG representative for advice on this.
  +
  +
== Specific testing when adding new Jules surface tiles in the dry deposition ==
  +
  +
New surface tiles were added as an urgent ticket for UKESM1 work in [https://code.metoffice.gov.uk/trac/um/ticket/1255 UM:#1255] and [https://code.metoffice.gov.uk/trac/um/ticket/1761 UM:#1761].
  +
  +
However, there are some less common ways of setting up suites that were not fully tested at the time.
  +
  +
Bug fixes behind temporary logical switches were made in tickets [https://code.metoffice.gov.uk/trac/um/ticket/4157 UM:#4157] and [https://code.metoffice.gov.uk/trac/um/ticket/4312 UM:#4312] and at least one more fix will be required in [https://code.metoffice.gov.uk/trac/um/ticket/4913 UM:#4913].
  +
  +
Existing bugs were also fixed in ticket [https://code.metoffice.gov.uk/trac/um/ticket/1250 UM:#1250].
  +
  +
We would like to avoid a repeat of the bugs added in UKESM1 for future UKESM releases.
  +
  +
This is quite a niche task, so there are not currently specific tests for this. Please get in touch with Alan Hewitt and Gerd Folberth before working on a ticket to extend surface tiles in the dry deposition scheme. Also cc both of us into the ticket and add keyword dry_deposition please.
   
 
== Temporary Logicals ==
 
== Temporary Logicals ==

Latest revision as of 09:30, 1 September 2020

When making changes to UKCA code, additional testing will be required compared to the usual UM developments.

Please also see:

All changes will require a ticket on the UM trac pages on the Met Office Science Repository Service.

rose-stem

All UM developments require rose-stem to be used on a committed branch, using the developer group. For UKCA changes you should also use the ukca keyword, like this:

rose-stem --group=developer,ukca

Climate Configurations

If you are making changes to climate configurations two types of suites are provided for you:

Short testing suite

Details about the UKCA short testing suites are maintained here:

This suite allows you to test your changes with logicals on and includes thread and bit-comparability tests, as well as a rigorous test. These jobs only run for 2 model days (6 hours for rigorous) and are useful for making initial developments.

As there are no CRI-Strat rigorous tests under rose-stem, so it is important that this is done using this suite as it fulfils the requirement to test new changes under rigorous.

You can link to the Rose Bush output on your ticket, and newer suites produce a trac.log file.

20-year AMIP suite

If you are introducing new code (either a science addition or a bugfix) that changes results in some way, you should also test using a 20-year AMIP simulation and assess whether or not your change produces large changes compared to the existing configuration. These changes may be absolutely fine, and significant differences from the control shouldn't prevent code being committed (unless these are unexpected).

Details of available suites can be found here:

20-year controls are run at the Met Office and archived to MASS.

Once your suite has finished you should use the UKCA Evaluation Suite to produce a series of standard plots from your suite and compare these to equivalent plots from the control. A combined PDF document with the suites interleaved and viewed under 2-page view is usually best here.

You may also want to make use of valnote/autoassess or Maverick to produce a set of standard plots as well.

It is best to be in close contact with your SciTech Reviewer as they may request additional diagnostics/plots for the ticket.

AQUM

A rose suite using the AQUM standard job is maintained at each UM version. The operational model is designated "aqeur" and the latest suite can be found at the top of the table on the page. The testing process for AQUM consists of the following steps:

1. Copy the latest aqeur rose suite.

2. Select the period you wish to run for (see below) in the suite.

3. Carry out a control run.

4. Edit the suite to include your UM branch in the UM compilation.

5. Verify the results, comparing the test and control runs. Standard verification rose suites are maintained on the same link given earlier. For guidance on using the suite and interpreting the output please contact the AQUM CMG representative.

Case study period selection Depending on the change being made it may be desirable to choose a case study period which coincides with an episode of elevated pollution. Please discuss your change with the AQUM CMG representative for advice on this.

Specific testing when adding new Jules surface tiles in the dry deposition

New surface tiles were added as an urgent ticket for UKESM1 work in UM:#1255 and UM:#1761.

However, there are some less common ways of setting up suites that were not fully tested at the time.

Bug fixes behind temporary logical switches were made in tickets UM:#4157 and UM:#4312 and at least one more fix will be required in UM:#4913.

Existing bugs were also fixed in ticket UM:#1250.

We would like to avoid a repeat of the bugs added in UKESM1 for future UKESM releases.

This is quite a niche task, so there are not currently specific tests for this. Please get in touch with Alan Hewitt and Gerd Folberth before working on a ticket to extend surface tiles in the dry deposition scheme. Also cc both of us into the ticket and add keyword dry_deposition please.

Temporary Logicals

If you need to make use of a temporary logical, the testing requirements are discussed here:

Note that you should AVOID using temporary logicals unless it is absolutely necessary.