Page History

(postponed from 2021-10-04)

Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra, Arun, Nate, Jim, Leanne, Eli, Lauren, Jeff, Brock, Lee, Hsin-Fang, Chris (Waters), Clare, Dan, Merlin, Simon, Robert, Josh, Sophie

Agenda:

• Announcements: Monica will be back for 0.15 FTE. Brock will continue indefinitely.
• Review Action items from last week
• Review w_2021_38
  • What was noteworthy in this rerun?
    • Gen2 (DM-31815)
      • LAM: lost some steam on this one with the more pressing DP0.2 issues.  Processing is ongoing and should be done with metrics dispatched in the next couple of days.
      • One change I will look into soon is Eli Rykoff 's fix on 

        Jira
        server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31462

         which should get us (close to) gen2/gen3 parity for fgcm

• • • Gen3 (DM-31795)
      • most issues were database connection issues. 
      • Some errors in image different in step4. 
      • bps report claimed that the execution butler failed, but exited with success? Michelle Gower looked into it. Are we worried about it?
      • BB: Metrics produced and may be dispatched. Ping SK and ask why they aren't showing up on the dashboard.
      • Sophie's analysis_drp plots are produced for w34
  • What changed? Annotate
  • What do we expect from w_2021_42?
    • Post column renaming (DM-31825). 
    • faro might be run as part of the steps if tickets gets in before Wed DM-32029
    • ER & LAM: On jointcal, Gen2-Gen3 config differences have to do with column names. Gen2 uses source table. If there are ordering differences, there can be some stochasticity.
    • Changes toward Gen2-Gen3 parity:
      • We also expect some changes (towards parity) on the next run.  I will hopefully get the coadd level exposure and source IDs in sync between the two middlewares on 

        Jira
        server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-30284

         (thanks again to Jim Bosch for his help on that one!), so we won't face random seeding difference issues. 
      • One question I have relates to jointcal: I've seen some chatter about obs_subaru gen3 configs moving around...are these configs being kept in sync with gen2?
    • What new metric can we expect next time 
      • YA's plan: raw number counts for DIA sources per unit area. I hear John added jointcal metrics a couple weeks ago
      • LG: New SRD metrics. Can take a wishlist of non-SRD metrics.
      • YA: offset with refcat and matches between band please
      • LK: Sky object metrics. 
• Review DC2
  • w_2021_36
  • Gen3: Boring
  • Gen2: calexps(?) were same between Gen2 & Gen3
  •  w_2021_40
    • Gen2:  running with all 

      Jira
      server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-30284

       branches set up.  Have parity up to mergeDetections, but... see that ticket for details about some nagging coadd PSF differences between gen2 & gen3.
      LAM: Jim says Scarlet can be sensitive to numerical differences. Everything except coadd PSFs are identical.
      JB: If we can't get identical results, and if we can assert that Gen3 results are good enough, we can still retire Gen2.
    • Gen3: w40 - full focal plane. Especially good for astrometric matches. 300,000ish jobs - finished all jobs except `calibrate`. `step2` and `step3` were both fine. On `step4` now.
  • Gen3 Pipeline for DP0.2 (backport-v23 label)
    • LAM: would like eyes on 

      Jira
      server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-32128

       and 

      Jira
      server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-32129

    • JC: nightly ci dispatching metrics
• AOB
  - DT showed pull (chi) plots for CModel in DC2 w26. Will show more plots on Wednesday team meeting Oct 20.

Action Items