AP Pipeline Meeting, 2023-09-18

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Ari Heinze 
• Kurlander Jake 
• John Parejko 
• Erin Howard 
• Bruno Sanchez 
• Brianna Smart 
• Krzysztof Findeisen 
• Hsin-Fang Chiang 
• Kenneth Herner 
• Meredith Rawls 
• Nima Sedaghat 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • DM no-meeting week next week
  • schedule workshop
  • updates to DMTN-199
• Auxtel processing status (all):
  • HF: no prompt processing done during last week's run: no imaging data first two nights; third night PP could not deserialize nextVisit events at USDF due to Kafka schema registry breakages at the summit.  traced back to Ceph storage problem at the summit.  still not totally fixed
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • John: waiting to hear on final approvals of config changes.  pipelines_check and ci_hsc now don't change 
  • template generation with DRP pipelines (future of APTemplate)
    • JP: why can't we just run ordinary drp_pipeline?
    • MR: adds a lot of additional complexity that we don't need just to make templates
    • KF: is it true even if we subset the pipeline?
    • JP: why don't we want the intermediate catalogs?  they will help with vetting coadds
    • MR: we want a good seeing selection, DRP tends to create deep coadds
    • HF: step 4 of DRP pipeline already makes goodSeeing and deep coadds
    • IS: advantage of dogfooding DRP pipeline now helps us identify issues of things we need to turn off rather than build up from scratch
    • MR: anticipates hiccups dealing with DECam, but we should probably just push through it
    • EB: what was the concern about extracting needed templates from an external run?  KF: ap_verify dataset only wants to include a small set of tracts and patches, filtering is a bit clumsy with current tooling.  KF's preferred solution would be to do our own drp runs with only those tracts/patches.  JP: why can't we just get that list?  KF, MR: butler export is fragile when there are multiple datastores.  butler transfer_datasets is the canonical way to do this; does support filtering by dimensions.
    • BS: do we have tools to investigate when a template is bad (poor seeing, noisy, bad coverage)
      • JP: we should use DRP analysis_tools (or add to them)
      • IS: we could output some metrics from getTemplateTask
      • EB: agrees but we should also make some metrics for acceptance of (incremental) templates
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Ken: working on DM-40458 to debug a subtraction failure–was able to reproduce failure of PSF matching (claim of no kernel)
  • Erin:
    • more work trying to run ap_pipe with cm_tools and panda, lots of hiccups
    • setting up a DC2 template run, trying to resurrect script from last year with Meredith
    • haven't touched Saha Bulge recently, but hopes to bring to pair coding
  • how frequently should we update our templates?
    • depends on if we use DRP-produced templates or run drp pipeline ourselves
    • DECam will have to be run by us regardless
• Satellites (Meredith Rawls ):
  • make some slides for Bob. 
  • Still mired in getting some DECam test data
  • Eric: keep an eye out for glint data
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno: still digging in details of efficiency computations.
    • also preparing for source_injection migration.  Eric: any blockers you see?  Bruno: it's complex.  mainly solves the "I have a DRP pipeline, I want to easily add fakes on top of it".  But not very relevant for ap pipeline needs.  Main value is unifying how we interact with Galsim.  Eric: source_injection didn't have much AP input into its design; if we need something that it doesn't support, lets make sure we get it.
  • Ian: trading review with Bruno
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: alert broker was working Thursday, testing adding new topic, died on Friday–cert error again?  Still fighting with it
• TransiNet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: still trying to spin up DC2 repo.  is DC2 at USDF still half-restored version of NCSA?  Maybe we should wait on new templates before we decide if it should be fixed
    • HF: DRP reprocesses one tract monthly, can you use the templates from that?
      • #dm-hsc-reprocessing (Orion or Jen)
    • KF: what's missing?  probably just old data?  NS: some tarballs not unpacked, per K-T.  dataIDs changed as well at USDF.  So changed to IDF
    • HF thinks whatever's at IDF is the most trustworthy, more so than what was transferred from NCSA
  • Harshit was out sick last week, hoping to make progress this week
• Solar-system processing (Ari Heinze ):
  • Ari: been working on isolated tracket file.  several assumptions obliterated by real data (four-tracklet linkages can be false! and good astrometry residuals don't mean it's good.)  So we must invest effort into precovery and recovery to confidently discover asteroids
    • could we get DECam to follow LSST around for
  • Eric: with Mario, started work on mirroring MPC database to USDF for use with known-object association with AuxTel
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • HSC metrics jumped after changing templates–would be good to apply Ken's compare runs tool
  • calibrate runtime jumped?
• Review outstanding action items
• QA meeting October 2:
  • Ian: potential switch to preconvolution by default - confirmed
  • streak masking/long trail rejection?  If visuals ready (Bri & Meredith)–probably defer to October --maybe
• AOB

Action Items