AP Pipeline Meeting, 2021-03-08

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Unknown User (cmorrison)
• Unknown User (gkovacs)
• Kenneth Herner
• Meredith Rawls
• Krzysztof Findeisen

regrets:

Topics for discussion:

• Gen 3 conversion (all):
  • Krzysztof: Gen3 CI is uploading to squash!
  • DECam linearity got merged
  • DECam crosstalk still in review with Meredith, should be close.
  • We put DM-21888 (Gen3 HiTS CI) in April for Krzysztof–after which point we think we are at gen3 parity

• Pipelines Infrastructure (Krzysztof Findeisen ):
  • nothing else

• Difference imaging sprint (Ian Sullivan ):
  • Ken: looking at good/middle/worse seeing coadds
  • Meredith: updating plotting tools in ap_pipe-notebooks
  • Ian: worked some on diffim unit tests for simulated data, will be ready for metrics soon
  • Eric: with Meredith, got cutout→Zooniverse pipeline working on Postgres + Gen3, also did some prototype calcuations/metrics on the cutouts
  • Gabor: put metadata exploration notebook on Trello board (

    diffimTests/tickets/DM-28201_diffim_metadata/inspect_diff_metadata.ipynb).  Doesn't seem to be super useful, though, since the metadata system hasn't been extensively used in later pipeline development–might be useful place for us to put new entries going forward.  Also thinking some about number of sources detected w/ decorr. afterburner–doesn't think it's a simple increase/decrease.  Also working on technote, thinking about preconvolution.

• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al): 
  • out
• ap_association/SDM/fakes (Unknown User (cmorrison)):
  • working on functors, doesn't have reviewers yet
  • Lee Kelvin asking about AP fakes
• Review CI (https://chronograf-demo.lsst.codes/):
  •  COSMOS shows a small bump in the DIAsrc/source ratio on Mar 1
  • gen2 and gen3 COSMOS don't agree on object counts–we will ticket this and pick it up in the next sprint planning once we have a bit more history to look at
    • Lauren MacArthur just popping in here...I'm not entirely sure of the context for these object counts, but if they rely on Gen2/Gen3 parity through singe frame processing we are not quite there yet, but I am working on it (and getting close!) See  DM-28858 - Getting issue details... STATUS  and  DM-28936 - Getting issue details... STATUS .  Ignore me if this is entirely irrelevant for this issue!
  • Gen3 COSMOS runtime looks a little bit better, but this may be because Gen3 doesn't account for I/O time the same way (it's not a walltime calculation)
  • todos:
    • Eric Bellm write ticket for Meredith Rawls ( TBC) to investigate differences in object counts between Gen2 and Gen3 COSMOS CI 15 Mar 2021 
    • Krzysztof Findeisen put together a gen2 vs gen3 dashboard on squash ( DM-29113 - Getting issue details... STATUS ) 29 Mar 2021 
• Review outstanding action items
  • Ian: Saha bulge catalogs: group has just released a paper on this dataset.  Catalog is for their internal use right now, but Ian is going to ask for access for our testing.
  • External catalog of the DECam galactic bulge dataset will be released soon. Details: https://academic.oup.com/mnras/article/499/2/2357/5900983
• AOB
  • Meredith: gen3 inter-chip crosstalk may require some extra work since it's going to require two pipetask run calls

Action Items