AP Pipeline Meeting, 2021-04-26

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Unknown User (cmorrison)
• Krzysztof Findeisen
• Unknown User (gkovacs)
• Kenneth Herner
• Meredith Rawls

regrets:

Topics for discussion:

• Pipelines Infrastructure (Krzysztof Findeisen ):
  • discuss RFC-775 - Getting issue details... STATUS
    • Meredith has already been getting lots of comments from confused users about where pipelines live
    • concerns:
      • design is operations-focused (and positive in that vein), doesn't capture ad-hoc science users, and maybe development/construction process
      • concern about config sharing between AP & DRP
        • Eric has some hope this will be eased by greater focus on single cameras/datasets in commissioning, and in ops DRP configs will be updated on very definite timescales
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • not much from Meredith; Ken fighting some postgres schema issues–defaulting to write into the public schema rather than the user schema.
  • ken will ticket this to raise the point rather than silently fixing it

• Difference imaging algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Gabor working on debugging decorrelation afterburner
    • three orders of magnitude difference in noise between science and template.
    • having trouble isolating decorrelation afterburner effect because we also ran with doRescaleVariance=True–so needs to rerun
    • we're also not getting a good test here because we ran vs. the best seeing template–so this is the wrong case to convolve the science image
    • so multiple things to disentangle
    • also getting new and unusual crashes from dipole fitting
  • discussing mask handling with Robert–wants to discuss again with RHL
  • status of Gen3 DECam repo?
    • Meredith: Jim planning to make a DECam repo in the shared repo, but it is not there yet–Meredith can give access to her private repo in the meantime (no linearity or crosstalk, though)
  • Ian: will be scheduling more diffim sprint followup tickets in May sprint planning
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al): 
• ap_association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • three tickets in review
  • going to start pulling validate_drp data to look at AM metrics with distortion-model fitter
  • preparing to gut mapApData once SDM merges
  • reviews
• real-bogus (Eric Bellm, John Parejko ):
  • finished writeup of data flow, draft tasks to write for RB infrastructure
  • next to talk to some domain experts
• Review CI (https://chronograf-demo.lsst.codes/):
  • lots of confusing movement in the ratio of DIASources/science sources in both datasets and gen2 vs gen3
  • we think  that doScaleVariance was false for all combinations prior to April 21; at that time HiTS was ~identical in gen2 vs gen3 but COSMOS was discrepant
  • changes April 21 moved gen3 in both datasets (scale variance on)–both datasets now discrepant
  • ticket on April 22 turned on gen2 ScaleVariance for COSMOS but it doesn't match gen 3
  • so currently neither agree, and gen2 HiTS is the only dataset without scaleVariance
  • Chris can look at source catalogs to see if SFM is different
  • Meredith to continue investigation ticket.
• Review outstanding action items
• AOB
  • QA meeting next week.
  • please fill out sprint planning

Action Items