All
CI status, bugs, issues of the week
- Discussion on slack about wPerp requiring a large amount of memory: https://lsstc.slack.com/archives/C011E3ZUZAB/p1643131727008000
Review of action items from last week:
- Filtering before matching. This has been waiting on . All activity has been killed on NCSA machines.
- Failure mode. Introducing new matched catalogs made CI jobs break. If jenkins job completes today, we should be able to merge.
- wPerp switching over to using parquet tables. Jeff has started working on a ticket branch. Needs the data selectors to be merged first.
- Discussed with Sophie Reed. Would like to adopt faro as the place where algorithms for calculating metrics live. Port functions into faro. Will need to confirm with Yusra. Don't expect problems. Is there an issue with duplicating selectors? The ones we have implemented so far in faro are similar to analysis_drp, but more general.
What is the order of operations:
Matching before filtering in FITS input is separate and trying to finish ASAP.
Selectors is a per-requisite for the wPerp fix.
Asking Sophie to review the selectors ticket. Keith will also take a look.
Jeff noticed that metric values shift by about 5% (in both directions) after the filtering before matching. We are keeping more data because we are throwing out fewer objects due to a single problematic measurement.
We should make sure that we have the capability to diagnose the behavior when
- Fall 2021 epic
- Spring 2022 epic
- Backlog epic:
- Progress on conversion to parquet
v23 will be cut this afternoon. Would not be unexpected to cut of v23.0.2. We still want wPerp by step 5. French group wants this and we want to be able to have it available as an option to backport.
"freeze-drying bugs". Capture enough information to be able to do full interactive exploration.
ATLAS REFCAT 2 suited for shallower AuxTel data
DELVE. Is this our plan going into ComCam commissioning?
Who is responsible for
- identifying needed refcats
- processing commissioning data
- maintaining refcats
SIT-Com will drive the definition of needs, and may need to take on some of these tasks in practice. It will also need to have an operations role and name if these are activities that will be continued into operations.
Here is the current delve coverage, if having full grip coverage is vital we could try an prioritize that over the next few months https://data.darkenergysurvey.org/fnalmisc/decam_coverage.png
Data processing campaigns
- RC2/DC2 reprocessing epic :
- w_2021_40 RC2:
- w_2021_44 DC2 :
- w_2021_40 RC2:
- AuxTel
- AP/DiffIm
Faro for DP0.2
Potential co-working session ideas here
Next meeting
