BlueJeans: https://bluejeans.com/426716450
attending:
- Eric Bellm
- Unknown User (cmorrison)
- Spencer Nelson
- John Swinbank
- Unknown User (gkovacs)
- Ian Sullivan
- Unknown User (emorganson)
- Meredith Rawls
Topics for discussion:
- Gen 3 conversion (
):
- AP tickets remaining are mostly on Krzysztof Findeisen who is on holiday
- Precursor processing (Meredith Rawls ):
- has been busy with Starlink paper
- report from NCSA (Unknown User (emorganson) + Unknown User (manninga) et al):
- nothing major; Andrew has taken down some firewalls etc. for Kafka
- Alert Distribution (Spencer Nelson ):
- has been working on alert stream simulator
- trying to get rate up; now serializing into a reservoir to get out of the python serialization bottleneck
- now hitting Kafka limits on the laptop—1000/sec probably good enough for now
- Eric Bellm to investigate why HSC stamps are so big
ap_association/SDM/fakes/alerts (Unknown User (cmorrison)):- alert serialization is running on CI dataset; getting a schema mismatch from DIAObjects, chasing down type issues
- looking for ideas about AlertID (also filenames: ccdVisitID in the repo?)
- is this even needed? alerts should be 1-1 with triggering DIASource, could it just be the DIASourceID?
- Eric imagines there could be circumstances where revised alerts could be issued; if alertID = DIASourceID it's hard to capture that.
- Spencer suggests looking at RFC-4122 about generic UUID ideas (v4)
- thinks that having data embedded in the identifier is somewhat brittle (Meredith: this practice is common in astronomy, even if it's not a best practice)
- leads to people parsing identifiers, so you can't change things downstream–effectively an API that has to be kept stable
- most other ids are CCDvisitID + a count; do we want to preserve that information?
- if we keep in the astronomy mode this speaks to DIASourceID + a timestamp
- is this even needed? alerts should be 1-1 with triggering DIASource, could it just be the DIASourceID?
- let's leave it in Chris's hands for now; other ideas
- DCR (Ian Sullivan ):
- review on crowded fields with Eric
- running into some artifacts and problems with masking that makes him think we need some unit tests for assembleCoadd (dcrAssembleCoadd)–will discuss with John and Eric in sprint plannin
- Image differencing (Unknown User (gkovacs) ):
- decorrelation afterburner merged last week–caused a breakage. Unit tests passed, needed to do one last test run of ap_verify but didn't have an easy way to do it
- We had another breakage on Friday: flagged by K-T and Eric, thanks to Gabor and Ian for getting it fixed
- John Swinbank to see if we can get the review of DM-24438 - Getting issue details... STATUS expedited
- We had another breakage on Friday: flagged by K-T and Eric, thanks to Gabor and Ian for getting it fixed
- on to ZOGY code
- writing up some noise-whitening info
- decorrelation afterburner merged last week–caused a breakage. Unit tests passed, needed to do one last test run of ap_verify but didn't have an easy way to do it
- Review CI (https://chronograf-demo.lsst.codes/):
- number of DIAsources dropped sharply in one CCD, went up in others
- should be decorrelation AB fix
- we'll ticket Ian to look at the images
- but there is also a processCCD/calibrate runtime increase at the same time
- likely other changes–probably the change in 3rd party libs from Conda?
- John Swinbankwill ask Arch if this is expected
- likely other changes–probably the change in 3rd party libs from Conda?
- memory usage increases because decorrelation afterburner has to hold several images
- number of DIAsources dropped sharply in one CCD, went up in others
- AOB
- tracking to have v2.1 avro schema is first "official release"–is that what we want?
- Eric Bellmto review the schemas
- Unknown User (cmorrison) to suggest any other schema changes
- tracking to have v2.1 avro schema is first "official release"–is that what we want?