Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra

1.1.1. Attendees: 

1.1.2. Agenda:

  • Meeting recorder - (last 6 meetings were:  Jeff, Clare, Arun, Leanne, Sophie, Lee) 
  • Attendees - Nate, Yusra, Sophie, lee, Orion, Arun, Keith, Dan, Eli, Hsin Fang, Colin, Lauren,. Chris W., Jim, Leanne, Fred, Josh, jeff
  • Announcements
  • Review Action items from last month
  • Processing Status
  • What's going on with the DC2 reruns?
    • Nikolay is running on the IDF.
    • To do: check with him on metrics and plots.
  • Review w_2022_32(PREOPS-1225) and w_2022_36 RC2 (DM-36066) reruns: 
    • 32 first full RC2 on USDF
      • temporary HSC REPO
      • built by Fritz
      • Find problems w/ butler repo and pands
      • lots of out of memory systems, cant do automatic retries on panda
      • DRP Pipe has updated memory quests
    • w36
      • Ran with temp butler and pandas
      • most errors also in w32
      • Errors in subtract images, ticket filed
      • Memory issues solved w/ rescues
      • IT WORKS!
    • logs are going into google cloud (https://data-int.lsst.cloud/production-tools/) (https://console.cloud.google.com/logs Google Log Explorer can take queries to search logs. It needs IDF account login) 
    • Some discussion on log searching
    • follow up on USDF logging plans
    • lets look at parsl logging, seems to do the same thing, but it a bit different than HTCondor
    • Recall history:
      • w12: psfex
      • w16: piff  (bad size residuals)
      • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
      • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
      • w24: PIFF kernelSize to 25. new scarlet lite storage.
      • w28: attempt at fixing measure failures:  DM-35722
    • Open tickets for the errors:
      • characterize: Unable to measure aperture correction for required algorithm 'modelfit_CModel'
      • calibrate: DM-35552 (Arun is going to look at this, and detection issue)
      • detection:
      • measure = fixed. DM-35722 Thank you Fred!
      • analysis_drp plots don't show up. Plots are there! It is a bug in the plot browser. 
      • get analysis tools working on HSC data: DM-36305
    • Chronograf, plot-navigator 
      • Astromentric and photometric repeatability no change
      • WPERP creeping upwards, very evident in cosmos field
      • Why is WPERP much larger in cosmos
        • star galaxy separation?
        • metalicity?
        • look at greater depths
        • photoimetric repeatability very different as well
        • Denser field
        • look if things change if cut at magnitude vs SNR
        • can we run with 1/10 the amount of data?
        • Sophie is going to look at this
    • Plots:
      • main analysis_drp 
        • suspect this failed because some try block catching everything?
      • analysis_tools
        • only these are showing up
        • Need a plan for getting the rest of the plots ported over
        • Get configs setup for HSC data and not just imsim
  • New plots and metrics to expect?
  • AOB
  • Get ellipses from cmodel into object catalogs


DM-35752 - Getting issue details... STATUS


1.1.3. Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
 DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
 DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
 DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
 DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
 Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04