AP Pipeline Meeting, 2021-02-22

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Unknown User (cmorrison)
• Unknown User (gkovacs)
• Ian Sullivan
• Krzysztof Findeisen
• Meredith Rawls
• Kenneth Herner

regrets:

Topics for discussion:

• Gen 3 conversion (all):
  • Gen3 squash upload is in review
  • decam linearity is in late stages of review
  • decam crosstalk is close(?) but not yet in review
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • opened RFC to get rid of formal names for ap_verify datasets
  • will change the names in squash, which will break continuity
    • could ask Angelo how hard it is to rename

• Difference imaging sprint (Ian Sullivan ):
  • lots of effort processing AP data in Gen3
  • Ken looked at templates made with goodSeeing, middleSeeing, etc.  They look pretty reasonable, with the quality dropping off with the seeing
  • Meredith tried making own calexp from raws using gen3 + bps, got lots of help from Michelle Gower.
    • 5 tasks: isr, characterize, calibrate, ..., diaPipe, 80k quanta
    • only scientifically interesting problems: a small number of characterize failures; diffim needed a slight bump in memory available
    • diaPipe had an error in almost every quantum.  AP HTM constructor arguments wrong?  Failure may have been after DIASources/Objects, though, so we do seem to have an APDB with at least a lot of data in it
      • ticket for the error still to come (not NULL error)
    • results are pretty patchy–but all of the filters have similar structure??
      • problem is in the DIAObjects–DIASources are everywhere
      • Chris thinks this is passing duplicate DIAObjects, they are probably there but have NULL values–will try to create appropriate catches around db queries
    • DECam flag filters don't cut out many of the sources
  • Gabor looked into metadata–only useful in older ip_diffim tasks, e.g. the A&L kernel solutions.  No metadata generated or stored during subtraction itself.
    • would be straightforward to add new metadata if we wanted to
  • Eric: made contact with Zooniverse dev, they fixed some bugs and pointed out some problems–hope to get this working soon.
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al): 
• out
• ap_association/SDM/fakes (Unknown User (cmorrison)):
  • have run SDM functors in diaPipe, seem to match old results.  going to go ahead and put this in production.
• Review CI (https://chronograf-demo.lsst.codes/):
  • two changes in the ratio of DIASources/Science sources
    • change on 2/9 is DM-27659, update to DECam templates
    • change on 2/11 is update to A&L afterburner, DM-28442.  HSC sees fewer DIASources, DECam a mix–which might be expected due to noise properties?
• Review outstanding action items
• AOB
  • QA meeting next week, 4pm Pacific

Action Items