DRP Metrics Monitoring 2023-11-06

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Regrets:

Agenda:

• Meeting recorder -  Orion (last 6 meetings were:  Jim, Clare, Dan, Erfan, Lee, Robert, Arun) 

• Announcements
• Action items from last time:
  • Scaled-size scatter cuts  (LM says not merged)
  • Reference-catalog matching plots need to get closed out (not on main, yet, but unblocked)
  • Manifest checker: counts of quanta that failed and succeeded (merged but not run)
  • scarlet lite as a standalone package  (on 39)
  • FGCM logic change not lookup table (on 39)
  • Some mysteries from last week
• Any other changes past or expected?
  • DM-39970 (Dan) -- changes the default max to peak for SDSS centroid

• Processing Status
  • Review the w_2023_41 DC2 (Orion): 
    • wPerp from faro_tract ran out of memory multiple times. There was some talk of retiring this version of wPerp, since it also exists in analysis_tools? 
  • Processing of w_2023_43 RC2 (Jen/Orion)
    • About to proceed to Step 5
    • skyObjectMean has errors 

      ValueError: Column unknown_ap09Flux specified in parameters not available in parquet file.
      

      but it looks like the data id is 

      dataId={band: 'unknown', skymap: 'hsc_rings_v1', tract: 9813}

      so recommend re-run or rescue with proper band constraints.
    • skyObjectStd failed with error 

      ValueError: Column ~_pixelFlags_edge specified in parameters not available in parquet file. 
      
      it looks like a data id is 

      dataId={band: 'white~nd', skymap: 'hsc_rings_v1', tract: 9615}
      
      This doesn't look normal to me (Orion). Could this also be because of the missing band constraints ?

• Review the w_2023_41 rerun:
  • https://usdf-rsp.slac.stanford.edu/plot-navigator/dashboard_gen3
  • https://usdf-rsp-dev.slac.stanford.edu/chronograf/
  • Lauren: big jumps in some metrics is expected!
    • DM-41554 - Getting issue details... STATUS  
    • Stamp sizes for piff vs. psfex reveals extremely high psf sizes
    • Huge spike ~0
    • Something is going seriously wrong with piff and maybe it has to do with the stamp size
    • Luckily the thresholding metrics caught on
    • Size scatter ~0 but we have NaN for psfTraceRadiusDelta – NaN's are skipped so then other values should not  be close to zero
    • Many plots available on the ticket
    • Lauren re-ran these and found no issue on HSC-RC2-- as opposed to the DC2.
      • We are probably running this very wrong on our DC2's!
      • tract constraints need to be gone so the CM team needs to re-do their submission files.
      • Selecting on tract means we don't get full visits, so there's not much for finalizeCharacterization to work with.
      • We should be comparing the initial and final PSF measurements if possible.
  • Photometric repeatability plots:
    • very weird holes!
    • shape doesn't make sense!
    • jagged edges in the healsparse maps
  • Yusra: Tract constraints on step 1 and step 2 were actually on purpose because the tracts are a lot smaller and to our knowledge we don't run any full focal plane tasks
  • Eli: some tasks maybe do? Given the holes we see, none of the piff psf's should work
  • "Strange things are afoot"
    • Lauren: The tract constraints are having an affect (if they shouldn't make a difference, what are we seeing?)
  • Jim: cannot have the tract constraints; Yusra: everything is WAY faster if we have them
  • Jim: isolated star catalogs expect to tile the sky
    • If you have a detector that is not one of your target tracts, isolated star creation will break if there is a tract constraint
    • unless we expand the list of tracts to include adjacent tracts
  • Eli: isolated stars in Lauren's run actually look fine
  • Lauren to Eli: it seems like she got consolidatedMaps but without the tract constraints they were not consolidated. Does that make sense?
    • Eli: ?? no.
    • TBD
  • Big mystery #2, unclear if connected: Jump in all coadd psf metrics for the u-band. 
    • possible connection to scarlet_lite standalone package?
    • Yusra: "10^-2 is objectively bad"
    • Jim: does this metric only look at isolated stars? It probably should. (although Fred confirms that this should have no effect due to weighting of blended stars)
      • Possible solution: remove blended stars from the metric.
• Review the w_2023_43 rerun:
  • Recall what we expected:
  • Recall 2023 history:
    • w03: Issues with using finalize characterize PSFs downstream. 
    • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
    • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
    • w15: quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
    • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
    • w23: RA/Dec columns renamed
    • w27: No major pipeline changes
    • w32: Use GAIA DR3 refcat in GBDES
    • w35: 
    • w39: scarlet_lite as a standalone package. 
• What pipeline changes do we expect for w_2023_47?
  • "Rashly predict" that AB and AA will get into metrics – Clare
  • Sophie: new analysis tools potentially
  • Eli: finish in putting defaults in from LATISS for scaled size thing
  • No errors in step 4!!!!! !!!!! !!!!
• Show and tell:
• AOB:
  • DM-41554 - Getting issue details... STATUS needs discussing
  • Tanaka-san plot!
    • AA1: Clare: since writing, after discussion with Robert, decided to implement differently
      
      • look at the median of the absolute value of the difference between the RA and Dec positions of the reference catalog
      • we might actually want to not take an absolute value; take the actual difference
      • Clare thinks that with more visits, you can get a better fit of the astrometry and get a better fit between the science measurements and the reference catalog
      • Way below science requirements so that's "always a comfort"  

Action Items