AP Pipeline Meeting, 2022-10-31

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Brianna Smart 
• Krzysztof Findeisen 
• Kenneth Herner 
• Erin Howard 
• Nima Sedaghat 
• John Parejko
• Ari Heinze 
• Meredith Rawls 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • no news has filtered to us
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko , Brianna Smart ): 
  • discussion of RFC-844 - Getting issue details... STATUS and DM-34874 - Getting issue details... STATUS
  • see https://lsstc.slack.com/archives/C2JPMCF5X/p1666912327564839
  • e.g., DM-36199 - Getting issue details... STATUS   : DIAForcedSource flags fields turning into a float due to a concat that creates nans since the flags field don't exist DM-36706 - Getting issue details... STATUS
    
    • (a separate question is what the DIAForcedSource flags should be https://jira.lsstcorp.org/browse/DM-36706
    • JP thinks instead of pandas.concat, use np.vstack with careful type checking
    • or just create those flags at zero
    • JP convinces us that we're not roundtripping the bits during a pandas (numpy) type conversion depending on what type
    • broader question: can we remove more of the pandas infrastructure?  should we?
      • KF: fine in principle with removing pandas, but thinks we should probably wait until this arrow support is more mature.  may replace one set of hard-to-debug issues
      • JP agrees that that is reasonable
      • JP hopes/wonders if we can do more of these things in numpy
      • Eric thinks pandas is baked pretty deeply in ap_association (including in-memory joins)–not convinced it's easy to replace.  (but maybe we should look more carefully)  So let's wait for this mature (instead of discovering a bunch of new bugs in astropy.table ourselves, for instance).  In the meantime, look to at least check column types explicitly, and warn/error on type errors
    • (side quest: define DIAObject flags)
  • Krzysztof Findeisen : nothing to report from Prompt Processing
• Precursor processing ( Kenneth Herner , Erin Howard , Meredith Rawls ):
  • Erin Howard : DC2 is still stuck–will investigate with Ken.  Template building is failing
  • Brianna Smart : HSC processed, but there are some visits with strange corners
    • got it working through SLURM, do not use "local"
    • Meredith says this came up a few years ago, we shouldn't have odd-numbered visits.  why aren't these being excluded?  We should
      • John found a 2 year old slack comment from Meredith.  DM-25990
      • resolution seems to have been "ask HSC".  Bri will ask
      • g-band failed also–maybe need to rebuild templates?
        • Ken wondered if maybe this was related to the DRP comment about astrometry failures?  unclear why this would only affect one band
  • Kenneth Herner : for saha bulge we decided just try try rebuilding from scratch rather than waiting for USDF unpacking
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • ticket in review with Ken for small bugfixes on convolve science.  No huge improvements
• Real/Bogus and transinet (Nima Sedaghat, Eric Bellm, John Parejko ):
  • Nima Sedaghat :
    • transinet, running lots of evaluation, finding some dropoff in performance if you train too long
    • on real-bogus, Arif is trying to reproduce braii work (ZTF classifier).  their training data not available
      • seems to overfit since the # of examples is too small, but paper reported better result.  they could only reproduce with exactly the same train/test split as in ZTF
      • Nima will go to Caltech for a week end
• alert distribution ( Brianna Smart , Eric Bellm ):
  • Brianna Smart got the system started in ArgoCD again, but it's not working.  Next step is try updating Kafka.  Read the changelogs, hoping it will "just work" to bump the version number
• Solar-system processing (Ari Heinze ):
  • still working on orbit-fitting; making good progress although it's challenging
• Circuit-breaking failures discussion (John Parejko ):
  • DM-35645 - Getting issue details... STATUS
  • came up in AuxTel context–telescope was physically moving during exposures.
  • could we make unit tests (or test data) to test our failure modes?  and add new pathological data as commissioning finds it?
  • we are worrying that DRP is making their pipelines more robust to e.g. missing WCS or PSF instead of failing noisily as we want it to
  • Ian: if we put hard stops in we should list these in a troubleshooting guide for commissioning folks
  • what's next?  JP: more brainstorming on failure modes?  either pair-coding, or a dedicated one-off-meeting
• Review CI (https://chronograf-demo.lsst.codes/):
  • still haven't found a ticket number for that little blip a few weeks ago
    • update: Ian Sullivan tracked it down to DM-36280
• Review outstanding action items
• QA meeting in November:
  • show results from HiTS/COSMOS reprocessing.
    
    • Erin has a HiTS notebook but hasn't been to the meeting
      • comfortable to present, Eric, Ian, and others to respond to questions
  • Ian hopes to show some results from convolve science
    • can at least discuss bugfixes, patch boundary bug
    • won't have ellipticity fix in, but could at least discussion early investigations
  • Eric on detection thresholds/fakes forced photometry?  not going to make it
• AOB

Action Items