...
- Project updates (Eric Bellm , Ian Sullivan ):
- RHL visit last week
- likely to have meeting-free weeks in June (summer AAS?) & September
- sprint planning next week
- Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
- DM-33734 follow-on:
- ran a pipeline with no processing (just loaded images and saved them again) and profiled it
- Jim and K-T did not agree about which parts of the processing needed worrying about
- code spent time in four places:
- package imports (variable). Jim thinks can reduce healpy impact on daf.butler; K-T thinks it will only happen on first run
- in pre-execution step, gathers conda package info–takes 8 seconds! Jim & Tim discussed caching, K-T thinks it only happens on once
- Ian: is this dealt with on startup? KF: no: on first pipeline run (per distinct)
- overheads on reading raws specifically
- we call lsstcam.getCamera()–very slow generation of Python object). Again, Jim has an idea for improved amplifier handling; K-T again says it's a one-time penalty
- Eric: could imagine a warm-up task during preload (ditto the conda info above)
- IERS validation, takes several seconds of time per raw, could happen on every run of the pipeline
- Ian: is this still the case where astropy is trying to download the table?
- JP: can configure astropy to use an offline cache: https://docs.astropy.org/en/stable/utils/iers.html#working-offline
- we call lsstcam.getCamera()–very slow generation of Python object). Again, Jim has an idea for improved amplifier handling; K-T again says it's a one-time penalty
- Ian:
- we should figure out what can go into preload, and generate tickets for middleware to implement
- potential "preload" sprint in the summer?
- explore in-memory datastore for the butler?
- we should figure out what can go into preload, and generate tickets for middleware to implement
- "ap_verify in Prompt Processing": there is a successor to Squash planned for operations ("Sasquatch", still influxDB/Chronograf). Let's discuss what it would take to push metrics from PP.
- (deferred for two weeks)
- Eric Bellm discuss "ap_verify in Prompt Processing"
- Eric Bellm discuss "ap_verify in Prompt Processing"
- (deferred for two weeks)
- DM-33734 follow-on:
- Precursor processing (Meredith Rawls, Kenneth Herner ):
- Meredith Rawls : starting reprocessing with decorrelation bug fix, aiming for next week results
- Kenneth Herner : Processed tract 4431 with weekly 16 and 17, two patches (42 and 50) and two bands (g and r)
- Just running step 4 and step 5
- Variance plane currently looks exactly the same. Ian Sullivan : This is unexpected, since the variance plane should have changed
- g-band shows ~4% reduction in sources, r-band ~90% increase in sources. We expected a slight to moderate reduction in sources in all cases, so this is unexpected.
- Plot of the source distribution shows many sources outside of the patch boundary.
- Image differencing algorithms, DCR ( Ian Sullivan ):
- real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
- Working on training networks on simplified task of supernova detection
- Low signal to noise SN are labeled as SN, but model cannot learn them
- Eric: this is using too much the "god's eye view" from the truth labels. Random noise will generate increasing numbers of objects at lower thresholds.
- alert distribution (Eric Bellm ):
- Cosmic rays (Karlo Mrakovcic ):
- No major differences in CR algorithms that were evaluated
- Curated a "true" sample of CRs that are labeled as CRs and not CRs
- Every algorithm is above 99.1% completeness
- Out of 18,000 stamps, ~600 are not labeled identically between all of the algorithms
- LSST stack on science images appears to be algorithmically fine. Should still test true completeness by using LSST darks with CRs
- Current stack appears to miss 0.3% of CRs that appear to be true CRs
- Review CI (https://chronograf-demo.lsst.codes/)
- Review outstanding action items
- QA meeting in May:
- Diffim changes (Ian, DM-34364)
- New reprocessing run of HiTS using new decorrelation (Meredith, DM-34475)
- Possibly discuss Karlo's report on CRs (he will be on a plane)
- (Ian: DCR: defer to May+)
- AOB
- none
...