Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Invited:

Attendees:

Regrets:

Agenda:

  • Announcements
  • Action items from last time:
    • All,  reduce the plugin warnings after the Logplosion.  (Thank you Lee for coordinating) We did it! 
      • NB: now replaced by the Plotplosion in w_2024_22 from fgcmFitCycle (see notes below) - a task which produces 11490 outputs for LATISS!
    • Look into who is converting NaNs to 0 in the metrics that appear on sasquatch. 
    • Look into why the shape size fractional diff scatter plots have not useful ylims and diverged from analysis_drp version. Jame's looking into it on DM-44451 says he'll get back to it June 8th.  Fixed on  DM-44333 (I believe the fundamental problem was DM-44179 - Getting issue details... STATUS which was fixed by DM-43333.  Those "masked partition" warnings were real!)
    • Check visit-level astrometry metrics on Sasquatch for w18. Confirmed step6 was not run with the sasquatch butler. w22 was though!
    • Mystery of the source tables getting generated for only detectors overlapping tracts solved on DM-44484 - Getting issue details... STATUS
    • Look into w18 failure:  DM-44333 - Getting issue details... STATUS
    • Eli: wPerp between w14 and w18, cModel 9813 changed, PSF did not, 9613 both changed yusra: looks like a shift Eli: has a theory involving aperture corrections Eli: aperture correction code is not good at the sub percent level, "I am very vexed", will dig more.  May be helped with DM-38632 - Getting issue details... STATUS which has some (hacky) improvements to aperture correction normalization.
    • We don't have a good sense right now if the plots we're looking at are showing data near the "danger zone" or not;
      • We don't know if we're approaching "danger" with the current metrics, as we're not comparing vs. thresholds.
      • JB: should have a policy that metrics are born with those thresholds (and grandfather that in for existing metrics - import from verify_metrics)
        • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
  • Processing Status
    • Processing of w_2024_22 RC2  ( DM-44526 - Getting issue details... STATUS )
      • Problems:
        • DM-44625 - Getting issue details... STATUS  
        • New <=4 day streak begins! Brian's wall-clock report: End-to-end wallclock time was about 3 days (68 hours), with step2cde (12 hours wallclock) and step3 (20 hours wallclock) taking the bulk of the time (along with faro_matched which took about 10 hours wallclock).
  • Review plots and metrics
  • Recall what we expected for w22:
    • Nope → w26: Tanaka-san plots?
    • AA1 to appear on the dashboard (with pilots remembering to use sasquatch butler always for CI runs)
    • Nope → w26: DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details). 
    • Nope → w26: Compensated Top Hats in w22?
    • Nope → w26: Multiband gbdes, doing some more QA
    • Nope → w26: DM-44452 - Getting issue details... STATUS
    • Lauren: two new metrics for calibration fidelity, needs to process lots of data for this?
    • FGCM plots will be butlerised, can be plot browsered. Eli Rykoff Would you post the collection name here? and to the w22 ticket?
      • FGCM ticket was DM-43942 - Getting issue details... STATUS and pickle fix was DM-44625 - Getting issue details... STATUS .  I ran the w22 run with the pickle fix into u/erykoff/RC2/DM-44625/fgcmcal/run1.
      • These plots are available in the Plot Navigator with the Global selection.  See, e.g, fgcm_Cycle3_SigmaFgcmReservedStarsCrunched  for fgcm repeatability metrics.
      • The introduction of this ticket has led to the Plotplosion in w_2024_22 from fgcmFitCycle - a single task which produces 11490 outputs for LATISS!
      • DM-44647 - Getting issue details... STATUS will attempt to help in the production of PDF pipeline graphs, but perhaps worth looking at this task again.
    • Yes w22:  Arun: shapeSize fractional diff binned by focal plane position? DM-39363 merged w20. 
    • Recall 2023-2024 history:
      • w03: Issues with using finalize characterize PSFs downstream. 
      • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
      • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
      • w15: quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
      • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
      • w23: RA/Dec columns renamed
      • w27: No major pipeline changes
      • w32: Use GAIA DR3 refcat in GBDES
      • w35: 
      • w40: scarlet_lite as a standalone package. 
      • w47: shapeHSM wraps Galsim
      • w48: shapeHSM wraps Galsim without slow down. Scarlet_lite standalone package bug fix for suspected cause of wPerp/PSF-metric regression. 
      • w50:
      • w02: 
      • w06: 
      • w10: moments-based star selector
      • w14: the astrometry repeatability metrics are now done faster. Thus might change the metrics a little bit. Eli: the changes are much smaller than the noise on the metrics in the tests.
      • w16: DM-43885 (capping the number stars going into PSF determination) – this should have zero effects on RC2 because there are zero detectors on RC2 subset. No payload errors (our-problem errors) for this weekly (and w14).
    • New/updated plots:
  • What pipeline changes do we expect for the w_2024_26(RC2) or w_2024_24 (DC2)?
    • Nope → w26: DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details). 
    • Nope → w26: Compensated Top Hats in w22? DM-38632
    • Multiband gbdes. TBD pending some more analysis on runtime. 
    • New warping task. Expecting to NOT affect metrics. 
  • What additional plots or metrics do we expect for the  w_2024_26(RC2) or w_2024_24 (DC2)?
    • Per-visit shapeFractionalDiff 
    • Tanaka-san plots:  DM-44452 - Getting issue details... STATUS Feedback requested! Which metrics to include. Default proposal is everything that was on chronograph as of last week. 

Ops Rehearsal 4 Pipeline

  • w_2024_23 tagged this Wednesday will be used to build templates for OR4. 
    • Does anyone have any reservations about the pipeline in its current state?
    • Are folks happy with skymap="ops_rehersal_prep_2k_v1") instead of skymap='DC2_cells_v1'"

Show and tell?

  • Actions
  • AOB:


Minutes (Trys)

Announcements:

  • Eli Rykoff : Accidental change to shape size fractional diff scatter plots has been fixed (needs follow up?)
  • w18 failure DM-44333 fixed

Processing status:

  • New <=4 day RC2 streak! (68 hours on this one)

Action Items:

  • Jeffrey Carlin Sophie Reed Continue moving plots to analysis_tools and retire old ones. Assign thresholds to existing metrics.
  • Move nightly rc2_subset from Lee Kelvin to somewhere more central. Get a slack bot for status updates.
  • Jim Bosch Eli Rykoff Lee Kelvin at fix the dimension discrepancy between filter/band to fix plot explosions ("plotsplosion")

Review plots/metrics

  • visit level AA1s are up
  • coadd level AA1s are present in plot browser but not on chronograf dashboard (but maybe they are? could be a loading issue)
  • Eli: FGCM plots:
    • fgcm_Cycle3_SigmaFgcmReservedStarsCrunched – g band has relatively worse performance at the red end. Discussions are planned at the next relevant science meeting.
    • fgcm_Cycle4_SigmaRef_{band}_Plot – color biased → median near 0
  • Plot navigator needs a speed boost
    • bug: summaries stored for collections actually the list every dataset type in the system

OR4 template building

  • Any reservations for building templates Thursday? None reported.
  • Jim: just reviewed code fix for OR3 longer runtimes → probably expect OR4 to be faster
  • smaller patch size reduced forcedPhotCoadd runtime

Pipeline changes for w26

  • Eli: compensated top hats are coming
  • Trys: tanaka-san plots
  • Clare: multiband gbdes TBD
  • Lauren: new per-detector metrics
  • Arun: new warping task. not expected to affect metrics




Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-07-01
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-06-03
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!Sophie: I haven't made progress
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04