12pm - 12:50pm EDT

Yusra's Zoom

Attendees:

Yusra, Monika, Hsin-Fang, John, Eli,  Chris, Sophie, Keith, Jeff, Dan, Lee, Nate, Andy, Merlin, Bryce, Robert Lupton, Leanne, Jim, Fred, Lauren, Simon, Clare

Agenda:

  • Review Action items from last week
  • Review w_2020_30:
    • What was noteworthy in this rerun?  
      • Nothing changed. Had to restart FGCM. Needed to run with special configs to use FGCM 
      •  
    • What changed? Annotate.
      • We don’t have to debug live, if we spend more than 3 minutes on a question we’ll assign someone or group to look into it for next time.
    • What do we expect from w_2020_34?
    • What new metric can we expect next time?
      • Rho stats
      • Fakes statistics
  • AOB


Next rerun will be weekly 34 which is tagged at 11:59 PDT August 21. Any changes must be in by then. 

  • Add FGCM-application configs to obs_subaru to make them "official"  Eli Rykoff   
  • Send Monika recipe for running a DRP with fakes through pipe_analysis Sophie Reed 
  • Rerun tract=9813 with fakes as part of regular RC2 processing Monika Adamow  
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
  • Send RC3 wishes to Jeff and Lauren  (Everyone)  
  • Get NCSA's size constraints for RC3  Yusra AlSayyad   
  • Start regularly running matchedVisitMetrics.py on the DC2 reruns. Needed by w36. Unknown User (emorganson)  (originally Monika Adamow updated Nov 2) 

Minutes:

Review action items from last week

Last time. Turned on FGCM and photometry repeatability got worse. Eli dug into this. Two things going on.

(1) Local background subtraction. Background is systematically off and this affects the fainter stars more than the brighter stars when using aperture flux (which is used for calibration). To get around this, used local background estimate. This made the internal FGCM metrics better, but also caused shift at bright end. See Ticket DM-26085. Eli shows example plot that shows curvature of local background subtraction. There is 1-2% offset at bright end. Depends on seeing and what the wings of PSF are doing. Wasn't making it to the photo-calib objects. This ticket is now in review. This seems to be main effect.

(2) Mean chromatic correction. With fewer observation, the chromatic corrections may not cancel each other out. Assumes mean star colors for now. Re-ran one tract and made a coadd, and ran pipe_analysis. Plots are available on the ticket. Not in w_2020_30 (currently under review). Pan-STARRS is reference at the moment. It was just a bookkeeping bug. For HSC wide field, we will have mix of different filters for i and r. In LSSTCam, the different chips will require bookkeeping to get chromatic correction right. u band may be more challenging.

Arun working on adding rho statistics. On vacation today. Didn't make it Stack for w_2020_30 but will for w_2020_34.

Lauren and Jeff discussing contents of RC3 dataset. This hasn't started yet. Will create ticket and confluence page today to start getting feedback. Want to set a date for input. Want a limit from NCSA on compute size and time. Yusra will get compute size limit. Proposed timeline for input on the order of ~2 weeks. Let's say 2 weeks from today, and agree on priorities at next meeting in this series.

What's new in w_2020_30?

Had some problems running FGCM. Lost connection to server.

Astrometry did not change. Ellipticity residuals did not change.

PA1 is not deterministic. This should change in "new framework"

Stellar Locus: Added tract number to the metric so can organize results by tract.

What do we expect to change next time?

Primary change is FGCM ticket above

Expect Arun's rho statistics

Full focal plane sky estimation ("sky corr"). News from NOAJ this week, the original author suggests changes to configuration. First round: 8K. In second round: Intent was to switch to 1K, but actually got set to 256. Need to confirm. There are two rounds of background subtraction. See ticket DM-17426 for details. All contained within skycorr. This will not change for w_2020_34. In general, would like to have one major behavior change per month.

Minimize special configs. Move FGCM configs into obs_subaru. Eli to do this.

Sophie working on metrics for fakes. This is work in progress. Monika did coadd fakes. Sophie to send Monika the details of fakes.

For w_2020_38, we should be ready to start doing validate_drp on DC2. Will this be a new dashboard? YES.


Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
 DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
 DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
 DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
 Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-01-04