AP Pipeline Meeting, 2021-04-19

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Krzysztof Findeisen
• Unknown User (gkovacs)
• Kenneth Herner
• Meredith Rawls

regrets:

Topics for discussion:

• Gen 3 conversion (all):
  • Eric suggests that we retire this bullet item!

• Pipelines Infrastructure (Krzysztof Findeisen ):
  • we now have HiTS running in Gen 3!
  • also working on improved metrics
  • discussion gen3 future of ap_verify, running with ctrl_bps, getting metrics from larger runs, etc.:
    • Krzysztof: right now, ap_verify does three things:
      • collects data in machine-readable format, incl config files
      • script for turning that data into a gen3 repo
      • script for running pipeline on the data and making metrics
    • problems in gen3:
      • third step needs pipetask, which is hard to run from python
      • we're duplicating ctrl_bps
      • ap_verify used to manage metrics, but get it for free in gen3
    • so when we can retire gen 2, should we remove ap_verify.py, just have it deal with ingest for CI, and then use standard cmd line task interface or ctrl_bps.
      • Eric: could we remove the gen2 capabilities from ap_verify early?  would need to get the CCB to sign off on this, and presumably need to deprecate for a full release?
      • kind of a question of whether there are external users?  KF says he does get questions from external users.  Seems like people find the clear documentation and get into it.  CM: are people doing anything serious here?  only runs on the fixed datasets, so probably they are getting rapidly stuck thereafter
      • should we RFC deprecation after the next stable?  (+ community post)
        • Krzysztof FindeisenRFC inserting deprecation warning for gen2 ap_verify for after next stable release 10 May 2021 
    • Chris: I use ap_verify a lot for integration tests, so as long as the overall ingest-run workflow
    • Eric: what do we need to do to get metrics in larger-scale reprocessing?
      • KF: just need to include the metricTasks in the gen3 pipeline–so if Meredith ran the pipeline in ap_verify on her monthly rerun, it would produce metrics.  (modulo some recently-appearing small bugs around instrument handling).  not sure about how they would scale, and upload to squash would require special handling.
      • Eric and Meredith agree that this month is too soon to try this, but we will aim for it next month
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith Rawls : great week of vacation!
  • Kenneth Herner : has been looking at convolve science image ticket, still getting up to speed on it

• Difference imaging algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Unknown User (gkovacs) : finished spatially varying ZOGY, in review once test run finishes.  then back to debugging
  • Ian Sullivan : also vacation last week
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al): 
  • out today
• ap_association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • pair coded full-focal plane distortion with JP
  • HSC is using its distortion model and feeding it to astrometry.py
  • affine-only solve then seems to give results comparable to full solution, from the log-level metrics
  • running affine-only solution on HiTS with no distortion model shows more scatter as expected, which confirms it's doing
  • so now try bigger runs with HSC to tune/tweak/test quality
  • also spent some time looking at off-CCD centroid failures–looks similar to previous instance
    • very rare (~1 source in HiTS), always marked as EDGE
  • working on removing map_ap_data, has gen2 and gen3 transform tasks in testing.  will create a few more columns of diffim flux in the apdb
• real-bogus (Eric Bellm, John Parejko ):
  • started outlining workflow for RB infrastructure with JP (Real/Bogus Data Flow)
• Review CI (https://chronograf-demo.lsst.codes/):
  • nothing new
  • HiTS ratio of DIAsources to science sources is essentially the same, HSC is different
    • CM says the DIASource catalog lengths differ in COSMOS CI gen2 vs gen3
    • speculation that this may be due to variance rescaling not being configured in our cosmos Gen3  (Meredith has this ticket)
• Review outstanding action items
• AOB

Action Items