DRP Metrics Monitoring 2023-06-26

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

• Yusra AlSayyad
•  Lee Kelvin
•  Erfan Nourbakhsh
•  Fred Moolekamp
•  Clare Saunders
•  Orion Eiger
•  Eli Rykoff
• Colin Slater
•  Hsin-Fang Chiang
•  Jim Bosch
•  Arun Kannawadi
• Christopher Waters 

Regrets:

•  Lauren MacArthur  road tripping
• Jeffrey Carlin 

Agenda:

• Meeting recorder - (last 6 meetings were:  Lee, Robert, Arun, Clare, Lee, Colin) 

• Announcements
  • SLAC is down this week - metrics meeting plots available on tiger for this week only
  • 2 meetings ago, open actions to investigate stellar locus jumps in RC2_subset (Eli)
    • Eli update: not entirely necessary to run everything by hand
    • tracking wperp - very sensitive to RC2_subset (noise? local minima?)
    • connects to aperture corrections on stars in single frame
    • band mis-registering can impact coadd-level photometry
    • we were also not handling primary detections properly in single-frame
    • as of 2 nights ago: still at 2.49 millimags (aka, a little better), so good news.
  • Clare w19 report:
    • fixed issue with meas_algorithms not propagating proper motions correctly
    • expected little difference - actually has some impact, reducing scatter, so more good news
    • does not impact gbdes directly, but improvements will filter through to analysis_tools
  • Jim: plan for future of RC2_subset is to run on a cronjob, rather than Jenkins
    • Eli: you /can/ download the outputs from Jenkins if you click the correct buttons!
• Processing Status
  • Processing of w_2023_23 RC2 (Orion) DM-39610 - Getting issue details... STATUS
    • expected changes: 
      • Clare: astrometric repeatability metrics - now in analysis_tools; more work required to push to Sasquatch (maybe selection?)
        
        • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
      • Eli: three detectors in RC2 that were formerly crashing now fixed
      • Orion: cm_tools changes:
        • when one step is complete, the next will now auto-begin (DM-39610)
      • RA/Dec columns were renamed
    • report:
      • very little to report on the pipelines side
      • Orion & Eli chased down aperture correction warnings incorrectly being reported as errors
      • encountered many frustrating pandas issues
      • broke faro into three different subsets - now implemented in drp_pipe
        • hard to run faro en masse; 
        • for the next CM run, this is how we will run faro
      • encountered step 4 errors as usual
      • subtract images still encountering issues as in previous months
        • Orion: look into location to store resource usage plots
    • chronograf report:
      • Yusra/Lee: we can look into whether we want to keep tracking both robust and non-robust sky object flux metrics
      • Nate: ellipticity residuals look somewhat volatile
        • Jim: perhaps a "meta-problem" (plotting issue on Chronograf) - if we can add a threshold onto the Chronograf, may indicate that we shouldn't pay too much attention to relatively small-scale fluctuations
    • memory / runtimes (w15 → w23)
      • consolidateObjectTable memory usage increase (from ~17 to ~20 GB)
      • a single patch is using a significant amount of memory for deblend
      • finalizeCharacterization now taking ~order of magnitude longer to finalize - can be refactored, but not a priority now
      • forcedPhotCcd runtime down a little
      • fgcmOutputProducts now up by order of magnitude
        • this task is all I/O bound, so probably just cluster issues
      • Clare: issue noticed when running RC2 with BPS/HTCondor
        • cores not always relinquishing memory from a prior task, e.g., going from a high memory task to a low memory task
        • end up not being able to run as many tasks at a time as you would expect
        • may not impact mem-usage plots (it's the cap on available memory per-core)
        • are cm_tools impacted by this issue?
        • Orion: cm_tools does little with memory requests - uses either default mem-request, or manual memory request override
          • cm_tools/Orion had a lot of trouble with memory requests recently
          • panda used to retry failed jobs with more memory, but now we retry with the same memory (loss of functionality)
          • much of this hidden inside panda black-box, requiring manual resubmit with memory request overrides
          • unknown how much cm_tools impacted by this right now - will talk more offline
• Review the w_2023_23 rerun:
  • Recall 2023 history:
    • w03:  Issues with using finalize characterize PSFs downstream. 
    • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
    • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
    • w15: pseudo- quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
    • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
    • w23: DM-39141 AMx/ADx/AFx
  •  Do the metrics on Sasquatch look OK?  (see below) 
  • Any changes in the resource usages? (see below) 
• What do we expect for w_2023_27? 
  • CS: re reinterpreted AFx. So we expect the baseline AF values to the higher than we were used to before.
  • Dan has some new plots in analysis_tools (cmodel bulge/disk) in extended so they don't get run automatically
  • repeatability metrics (both astrom/photom) on sasquatch)
  • those 3 errors in step 1 gone!
•  AOB:
  • Jim: any analysis_drp plots than can be retired due to replacement in analysis_tools?
    • Sophie: we have a list, and we can start to prioritize that
      • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim

Sasquatch:

Memory changes: 

Runtime changes:  runtime_w23vs15.png

Action Items