...
No significant difference is a bit tricky to define here because most of these metrics are not accompanied by uncertainty estimates; to estimate them we'll use the RMS of the meas_mosaic metric (henceforth σ) results over different tracts of HSC Wide. Differences less than 0.1 mmag for photometry or 0.1 mas for astrometry (even if larger than the cross-tract RMS, though I doubt the RMS across tracts will be that small) can also be ignored.
We will run the metric comparison independently on each of at least 20 19 tracts of HSC-SSP Wide data, as well as the (single tract) HSC-SSP UDeep COSMOS dataset. Any tracts that that have metrics that are more than one sigma worse than meas_mosaic should be investigated more carefully to understand what (if anything went wrong); this does not necessarily preclude acceptance of jointcal as a replacement for meas_mosaic, but it's a sign that we'd like to understand the difference No more than 1 of these 20 tracts shall have a regression from the meas_mosaic performance in that tract by more than 2σ, and none may deviate by more than 3σ. The number of tracts may be extended if initial failure to achieve these rates appears to be due to statistical fluctuations.
The WCS mappings and photometric scalings output by meas_mosaic and jointcal should also be manually inspected for at least 5 visits in each of 5 tracts (including COSMOS) to look for unusual differences.
Doing the Work
Scheduling work is obviously a T/CAM responsibility, but it may be best to try to share the generation of the input data repositories for these tests with re-run of the full HSC-SSP PDR1 (NCSA may have already scheduled some time for this later this cycle).
Final sign-off on Scientific validation of the results is the responsibility of the tests is up to the Pipeline Scientist, with the understanding that (pending this test plan's review and approval) he'll only ask for significantly more testing than the above if the results are significantly worse/weirder than we expect.