DRP Metrics Monitoring 2021-11-01

Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Agenda:

• Announcements:
• Review Action items from last week
• Processing Status
• Review HSC RC2 rerun w_2021_42
  • What was noteworthy in this rerun?
    • Gen2 ( DM-32419 - Getting issue details... STATUS )
      • This is getting underway and is also a w42+ run where I will be setting up the  DM-30284 - Getting issue details... STATUS branches (see below).

• • • Gen3 ( DM-32248 - Getting issue details... STATUS )
  • What changed? Annotate
  • What do we expect from w_2021_48?
    
    • Changes toward Gen2-Gen3 parity:
    • What new metric can we expect next time 
• Review DC2
  • w_2021_40
  • • gen3:
    • gen2 ( DM-32071 - Getting issue details... STATUS ): this processing was done having set up w_2021_40 + the ticket branches from DM-30284 - Getting issue details... STATUS  in efforts to achieve gen2/gen3 parity.  In that ticket, various gen2 updates are made to mimic the gen3 behavior of the setting of exposureIds.  These are important as they are typically used for random number seeing, thus will have an effect on the results.  Those issues should all be solved in this run, but there is one lingering difference which is due to gen3 order of inputs not being deterministic.  I missed it at first as the inputs are sorted by visit in the code, but it turns out the detector order also matters.  In gen2 it seems to be systematic (and always in increasing id order), but it can vary in gen3 (and I think can vary with how the pipeline is run → it seems to be the same for the gen3/bps/DRP vs. gen2, but in some one-off testing, I ended up with a different ordering and was freaking out just a little since thus far the warps had been comparing identical!)  Here is and example of the difference in the warps just from a different detector input ordering:So, with all the exposureId fixes, we are only left with a slight difference in the PSFs (for which the detector ordering in gen3 just doesn't seem to ever match that of gen2:At this stage I was left with differences appearing in the deblender phase.  Since scarlet is expected to be rather unstable in certain (numerical) circumstances, I had attributed these differences to the above tiny differences in the PSF – all other inputs are identical as far as I can glean.  However, with the ordering of the detectors in gen3, I ran the coadd stages again and indeed got identical coaddPsfs...but I still get differences at the deblender stage.  So, paging Fred Moolekamp as to whether he can think of any reason gen2 vs. gen3 scarlet runs could differ (again, as far as I can tell, all inputs going in with all the fixes on DM-30284 are identical).
      • Ok, so where are we with the current run which only suffers the above small PSF differences going in?  Even better than before and at difference levels so small as to potentially be scientifically uninteresting.  Direct gen2/gen3 comparison plot can be perused at https://lsst.ncsa.illinois.edu/~lauren/DC2.2i_gen2/w_2021_40/vsGen3/plots and here are a few examples (all r-band):
      • I still feel slightly uncomfortable not knowing the root cause of these differences (and the possibility that the deblender is sensitive to some as yet unknown cause that may lead to non-reproducability), but it may be time to start considering the cost/benefit of chasing these lingering differences down.
    • Gen3 Pipeline for DP0.2 (backport-v23 label)
• AOB
  
  

Action Items