AP Pipeline Meeting, 2021-09-27

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan 
• Krzysztof Findeisen 
• Kenneth Herner 
• Unknown User (cmorrison) 
• Spencer Nelson 
• Meredith Rawls 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • JSR next week
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • RFC-807 - Getting issue details... STATUS naming
    • this is DRP focused, but we should expect to have to propagate changes back to AP–e.g., psFlux → psfFlux
    • bitmasks packed or not or will depend on implementation, doesn't affect us
    • Schema translator is the same between AP and DRP
      • KF: Do we need to keep the APDB schema as is?
        • CM: That is a question for DAX. This could change their timing metrics.
      • CM: Does not consider this too bad of a problem, since CM wrote the DRP and AP code. The two do not share code after the transform step.
        • The pipeline on our end would get caught by CI pretty quick if DRP changes things that effect AP.
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith
    • at present can't do both fakes and multi-tract:
      • multiple tracts trying to write the same dataset type with the same dataID
      • https://lsstc.slack.com/archives/C01FBUGM2CV/p1632769640247100
      • some discussion of how this may or may not differ from other products
      • Chris thinks there may be some workarounds with loadMulti–they will chat during pair coding
    • easiest would be to drop fakes
    • or use a single tract–but that would require too much time, still has to rebuilt calibs
    • Eric suggests: don't push on being ready for QA meeting, let's resolve the fakes loading before running HiTS (either by fixing fakes loading for multi-tract, or defining a single tract as a fallback)
  • Diffim sprint reprocessing (Ken)
    • Lots of extra DiaSources showing up in patch-like regions around the outside of the diffims using the best seeing-selected coadds
    • The DiaSources look good (and the same as before) using the middle seeing selected coadds
    • Suggestion is that there is a possibility that the wrong coadds are being grabbed for the best seeing case, and these might have few constituent images
• Image differencing algorithms  ( Ian Sullivan ):
  • DM-31884 - Getting issue details... STATUS  filed to test equivalence of template vs science image convolution.
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
• real-bogus (Eric Bellm, John Parejko ):
  • no updates
• alert distribution/performance tuning (Spencer Nelson):
• Preops DP0.2 preparation (all):
• Review CI (https://chronograf-demo.lsst.codes/):
  • There is a change in the DiaObjects metric for both Cosmos and HiTS starting 9/26.
  • Ian merged DM-30465 - Getting issue details... STATUS  that was a clean-up of some of Gabor's unfinished work on 9/25 . This should not have caused a change since it should have only affected image differencing using preconvolution, but it should be investigated.
• Review outstanding action items
• QA meeting in October (next week):
  • Ken/Ian: diffim sprint recap. 
  • Eric: followup from comments at last meeting
    • DM-31341 conclusions on edge and nondetections
  • Meredith: HiTS run–probably have to defer to november to get fakes
  • Ken: saha bulge?  hopefully november
• AOB
  • none

Action Items