Mondays 12pm - 12:50pm ET
Yusra's Zoom: https://princeton.zoom.us/my/yusra
Attendees:
Agenda:
- Announcements:
- Review Action items from last week
- Should we include debiased moments in the object tables talk on Wednesday.
- Review w_2021_34
- What was noteworthy in this rerun?
- Gen2
- log outputs were being swallowed ( - DM-31530Getting issue details... STATUS fixed this, so won't be a problem for next run)
- new LinAlgError coming from scarlet observed (see - DM-31477Getting issue details... STATUS ) and Fred Moolekamp alerted.
- gen2/gen3 parity through SFM (no external calibrations) maintained
- Gen2
- What was noteworthy in this rerun?
- Gen3
- Fairly OK. Couple dataset lookup errors. On step4. 13000 multi-tract not supported. But there are no overlaps in RC2.
- issue with step3 submission (tract ids were omitted). Resolved? Yes.
- Gen3
- What changed? Annotate
- What do we expect from w_2021_38?
- Changes toward Gen2-Gen3 parity:
- Eli Rykoff has put in a significant fix towards gen2/gen3 parity ( - DM-31505Getting issue details... STATUS )...fingers crossed.
- It may be that we can't hope for parity with jointcal/fgcm without imposing a deterministic ordering of inputs in gen3 middleware, so - DM-31701Getting issue details... STATUS was created.
- What new metric can we expect next time
- Need Cross-band astrometry
- Why do AM1/PA1 look better in Gen3 between w22 and w26 in HSC RC2?
- Change in the faro algorithm that selects bright stars.
- DM-31673, DM-25789
- Why is gen2to3 astrometry/photometry metrics so terrible? Could be the afterburner or faro?
- Eli Rykoff Jeffrey Carlin will look into which is causing problems.
- Changes toward Gen2-Gen3 parity:
- Review DC2
- w_2021_36
- Gen2:
- back to SFM parity
Can now use this run to probe any gen2/gen3 differences in vanilla coaddition
- only processed tract 3829
- an ArithmeticError was observed in scarlet, - DM-31713Getting issue details... STATUS created and assigned to Fred Moolekamp to investigate.
- back to SFM parity
- Gen3:
- On step4. for w36 when running step4 you need DRP.yaml#step4,getTemplate
- Needed 120 GB RAM to measure wPerp in faro.
- faro is not using parquet tables. This should help the memory usage.
- Yusra: Look into Source Tables for DC2 Gen3. Update: this was indeed a gen2-only issue as the config overrides in mind are for pipe_drivers task (Lauren MacArthur apologizes for the noise…I may have mislead myself looking into a gen3 repo that was actually converted from gen2, so also did not have the visit-level parquet files).
- Gen2:
- w_2021_36
- AOB
- Dashboard extravaganza?! Dan Taranu and Lauren MacArthur have made updates to the dashboards and with the help of Unknown User (brendal4) and Simon Krughoff, we have dispatched with all the proper variables/timestamps set, so they actually seem to "make sense" (let's have a look https://chronograf-demo.lsst.codes/sources/2/dashboards? Of interest are those titled:
- DRP metrics monthly DC2 (Gen3/2to3, test-med-1)
- DRP metrics monthly for DC2 (Gen2)
- DRP metrics monthly Gen 2 (HSC RC2)
- SR: What are Star/Galaxy Sep "unknowns in pipe analysis"
- LM: "unknown" = NaN
- JB: We do want to plot objects where the measurements that went into the classification failed.
- DT: We have a nightly ci on hsc, Simon's working on an intermediate dataset. Should we use ci_imsim to check nightly metrics.
- rc2_subset? results in full coverage in all bands. end product is coadd with at least 1-visits coverage on every pixel in one patch. This is the same dataset on the getting started guide. We are planning on using for regression tests nightly. WIll push that out soon.
- ci_imsim gets run nightly. We should look at faro output from that to see if it is useful.
- KSK: Changes to dispatch_verify coming. It'll report which squash jobid your dispatch requested. Will be useful for which jobs get where. Drop NaNs when generating the jsons. Angelo is working on the part that drops the NaNs. You won't have to downgrade your version anymore. Hoping to get there by w38.
- Dashboard extravaganza?! Dan Taranu and Lauren MacArthur have made updates to the dashboards and with the help of Unknown User (brendal4) and Simon Krughoff, we have dispatched with all the proper variables/timestamps set, so they actually seem to "make sense" (let's have a look https://chronograf-demo.lsst.codes/sources/2/dashboards? Of interest are those titled:
Plots
9813
9615
9697
Action Items
Description | Due date | Assignee | Task appears on |
---|---|---|---|
| 04 Sep 2020 | Sophie Reed | DRP Metrics Monitoring 2020-08-07 |
| 16 Mar 2022 | Jeffrey Carlin | DRP Metrics Monitoring 2022-02-28 |
| DRP Metrics Monitoring 2022-10-31 | ||
| Yusra AlSayyad | DRP Metrics Monitoring 2021-06-14 | |
| Arun Kannawadi | DRP Metrics Monitoring 2021-04-19 | |
| Arun Kannawadi | DRP Metrics Monitoring 2021-03-01 | |
| Yusra AlSayyad | DRP Metrics Monitoring 2021-01-04 | |
| Jeffrey Carlin | DRP Metrics Monitoring 2021-01-04 |