DRP Metrics Monitoring 2021-09-13

Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees: 

• Yusra AlSayyad 
• Simon Krughoff 
• Leanne Guy 

Agenda:

• Announcements:
• Review Action items from last week
  • Should we include debiased moments in the object tables talk on Wednesday. 
• Review w_2021_34
  • What was noteworthy in this rerun?
    • Gen2
      • log outputs were being swallowed ( DM-31530 - Getting issue details... STATUS  fixed this, so won't be a problem for next run)
      • new LinAlgError coming from scarlet observed (see  DM-31477 - Getting issue details... STATUS ) and Fred Moolekamp alerted.
      • gen2/gen3 parity through SFM (no external calibrations) maintained

• • • Gen3
      • Fairly OK. Couple dataset lookup errors.  On step4. 13000 multi-tract not supported. But there are no overlaps in RC2.
      • issue with step3 submission (tract ids were omitted). Resolved? Yes. 
  • What changed? Annotate
  • What do we expect from w_2021_38? 
    • Changes toward Gen2-Gen3 parity:
      • Eli Rykoff has put in a significant fix towards gen2/gen3 parity ( DM-31505 - Getting issue details... STATUS )...fingers crossed.
      • It may be that we can't hope for parity with jointcal/fgcm without imposing a deterministic ordering of inputs in gen3 middleware, so  DM-31701 - Getting issue details... STATUS  was created.
    • What new metric can we expect next time
      • Need Cross-band astrometry
      • Why do AM1/PA1 look better in Gen3 between w22 and w26 in HSC RC2?
        • Change in the faro algorithm that selects bright stars. 
        • DM-31673, DM-25789
      • Why is gen2to3 astrometry/photometry metrics so terrible? Could be the afterburner or faro?  
        • Eli Rykoff Jeffrey Carlin will look into which is causing problems. 
• Review DC2
  • w_2021_36
    • Gen2:  
      • back to SFM parity   Can now use this run to probe any gen2/gen3 differences in vanilla coaddition
      • only processed tract 3829
      • an ArithmeticError was observed in scarlet,  DM-31713 - Getting issue details... STATUS  created and assigned to Fred Moolekamp to investigate.
    • Gen3:
      • On step4. for w36 when running step4 you need DRP.yaml#step4,getTemplate
      • Needed 120 GB RAM to measure wPerp in faro. 
      • faro is not using parquet tables. This should help the memory usage.  
      • Yusra: Look into Source Tables for DC2 Gen3. Update: this was indeed a gen2-only issue as the config overrides in mind are for pipe_drivers task (Lauren MacArthur apologizes for the noise…I may have mislead myself looking into a gen3 repo that was actually converted from gen2, so also did not have the visit-level parquet files).

• • Gen3 Pipeline for DP0.2
• AOB
  • Dashboard extravaganza?!  Dan Taranu and Lauren MacArthur have made updates to the dashboards and with the help of Unknown User (brendal4) and Simon Krughoff, we have dispatched with all the proper variables/timestamps set, so they actually seem to "make sense" (let's have a look https://chronograf-demo.lsst.codes/sources/2/dashboards?  Of interest are those titled:
    • DRP metrics monthly DC2 (Gen3/2to3, test-med-1)
    • DRP metrics monthly for DC2 (Gen2)
    • DRP metrics monthly Gen 2 (HSC RC2)
  • SR:  What are Star/Galaxy Sep "unknowns in pipe analysis" 
  • LM: "unknown" = NaN
  • JB: We do want to plot objects where the measurements that went into the classification failed. 
  • DT: We have a nightly ci on hsc, Simon's working on an intermediate dataset. Should we use ci_imsim to check nightly metrics. 
    • rc2_subset? results in full coverage in all bands. end product is coadd with at least 1-visits coverage on every pixel in one patch. This is the same dataset on the getting started guide. We are planning on using for regression tests nightly. WIll push that out soon. 
    • ci_imsim gets run nightly. We should look at faro output from that to see if it is useful. 
  • KSK: Changes to dispatch_verify coming. It'll report which squash jobid your dispatch requested. Will be useful for which jobs get where. Drop NaNs when generating the jsons. Angelo is working on the part that drops the NaNs. You won't have to downgrade your version anymore. Hoping to get there by w38. 

Plots

9813

9615

9697

Action Items