AP Pipeline QA Meeting, 2021-11-01

4pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Krzysztof Findeisen 
• Unknown User (cmorrison) 
• Yusra AlSayyad 
• Andrew Connolly 
• Kenneth Herner 
• Colin Slater 
• Meredith Rawls 
• Krzysztof Suberlak 
• Zeljko Ivezic 
• Jeffrey Carlin 

regrets:

• Spencer Nelson (out sick)

Topics for discussion:

• Precursor processing (AP team):
  • Spencer Nelson on pipelines profiling (out sick)
    • ran profiling on DC2 imsim data in DM-31653 - Getting issue details... STATUS , in ~215 seconds.
    • (very little association, so an under-estimate of performance)
    • discussion at Pipelines team meeting identified some DRP-specific measurements during ISR that we can remove.  Otherwise no single big slow pieces.  Think about minimizing IO for calibration frames/preload.
    • Colin: what about using multiple cores?  Eric: good point, future work
  • DECam HiTS with fakes and Saha Bulge reprocessing both got held up by middleware changes ( DM-31769 - Getting issue details... STATUS , DM-32407 - Getting issue details... STATUS , others.).  Close to resumption now.
• SITCOM team updates (Andrew Connolly  et al.):
  • working on closed-loop comcam simulated out of focus images–mirror support loop now converges.  now waiting for T&S to give some simulated pointings.  hope to wrap this up with crowded fields, ready to integrate into T&S code.  Next look at rotational dependence
  • working on analyzing AuxTel data.  Translating zernikes into moving the secondary mirror for collimation
    • Krzysztof Suberlak : series of out of focus and in-focus images.  trying to understand reproducability run-to-run.
    • trying to look at differential zernike as a function of hexapod movement, since don't have a good AT model
    • Eric: how big are the physical shifts that these zernike changes represent?  Andy: about a mm shifts
    • Chris Stubbs asked about dependences on free seeing
    • Colin: how to think about the units of the zernike coefficients?  Andy: for AT, we don't know how stable it is.  don't know if scatter is mechanical hysteresis, or mirror shape, etc.
    • Eric: anything DM can help with?  Andy: DM astrometry is working well.  AT pointing is only good to 40" or so, so blind crossmatch is tough.  Not using butler exactly as DM would like–trying to pre-calculate catalogs needed prior to observations.  Next is masking & deblending donuts, unclear how much existing DM helps.  Want to move T&S zernike code to galsim, but requires a version bump.
• PreOps V&V (Colin Slater ): 
  • DP0.2 pilot runs are running, lots of data on disk–getting the V&V team geared up to look at it, prepare for full-scale processing.  No results to share yet.  Hopes to get Tim Morton's plot viewer onto the IDF
  • Eric: we do "QA" and V&V lots of ways–any sense of how this is going to proceed from a tooling process?  Colin: using this as forcing function to explore/consolidate.
• Review CI (https://chronograf-demo.lsst.codes/):
  • Jump in HiTS (but not COSMOS) DIAObject counts on 10/26–annotated as same time as DM-31491 - Getting issue details... STATUS
    • Unknown User (cmorrison) investigate HiTS objects counts job on 10/26 08 Nov 2021
    • Chris thinks this is a process ordering issue (HiTS has more CCDs than HSC so more like to block), thinks due to DM-32131 - Getting issue details... STATUS
    • fakes haven't changed

• Next meetings (all at 4pm Pacific):
  • December 6
  • Jan 3?
• AOB

Action Items