Checklist: Gen3 ready for general use (DM-DAX-12 Feature Parity)
We are aiming for November 1st.
Planned Work
Tickets indicate planned work. Checkboxes without tickets are (hopefully) all statements about functionality that is available.
- Schema changes (changes requiring wholesale re-ingest could occur but sparingly... migration scripts may be possible)
- - DM-26407Getting issue details... STATUS
- - DM-21231Getting issue details... STATUS
- - DM-24432Getting issue details... STATUS
- - DM-26476Getting issue details... STATUS
- - DM-26692Getting issue details... STATUS
- - DM-26630Getting issue details... STATUS
- - DM-26738Getting issue details... STATUS
- - DM-26978Getting issue details... STATUS
- General usability
- list of needed Gen3 Butler functions/command-line tools
- - DM-26684Getting issue details... STATUS : simple command-line tool (prune-collection), high priority
- - DM-26685Getting issue details... STATUS : more complex command-line tool (query-datasets), unblocks many others, only slightly lower priority
- - DM-26690Getting issue details... STATUS : more complex command-line tool (query-data-ids), may not unblock as much
- - DM-26856Getting issue details... STATUS : butler method and command-line tool (prune/unregister/delete/remove-dataset-type)
- - DM-26874Getting issue details... STATUS : rename pipetask2 to pipetask, remove old pipetask
- BPS Features needed for developers (single/multi-node)
- Documentation
Open Questions
- Raw Ingest works (caveat:
-
DM-21706Getting issue details...
STATUS
) but:
- still an open question as when to run define-visits.
- Handling of photo diodes for BOT will need special ingest script and associated formatter infrastructure.
- We are still dependent on the Filter singleton being removed if we wish to combine multiple instruments in a repository.
Acceptance Tests
We regularly run small test pipeline with Postgres. ci_hsc_gen3 runs entirely with gen3 on a single node with multiprocessing and sqlite/local file. Google POC has demonstrated we can run an RC2 tract with condor using S3+Postgres (DMTN-157). These tests all assume shared everything where multiple users are using a shared datastore and a shared postgres database. These tests can not be complete given the current status of cp_pipe with Gen3 and the lack of jointcal.
- Run a pipeline on a single node using pipetask
- Access data products of that run from a notebook
- Run a 3 tract RC2 in under a week on NCSA Condor system using BPS
- cp_pipe equivalent available (recipe + example)
- run RC2-like processing (not necessarily full-scale) (https://github.com/lsst-dm/gen2-hsc-rc2 & https://paper.dropbox.com/doc/Using-HSC-RC2-to-test-algorithms-on-tract-scale-LIXxHweXU2ZOnuJB6cMZ0)
- run DC2-like processing (not necessarily full-scale)
- can run CPP - corresponding to appropriate above scales
- Ingest all the raw datasets that are currently hosted at NCSA for supported instruments.
Below this point is open for discussion but pertains more to Gen2 deprecation (list of appropriate tickets)
- DM-26173 Write plan for Gen2 deprecation
Remaining Checklist: DM-DAX-13 (Gen3 ready for Gen2 Deprecation )
Select any items below that are required for Gen2/3 parity and hence need to be completed by Nov 1st.
- All Schema changes are accompanied but a migration mechanism (or have appropriate CCB approval).
- Additional command-line tooling (functionality already accessible from Python, which we think is adequate for declaring feature parity in these cases)
- Gen3 continued development (would be on-going and even supplemented/aided by general DM users/developement)
- - DM-21333Getting issue details... STATUS
- - DM-21832Getting issue details... STATUS
- - DM-23985Getting issue details... STATUS
- - DM-26277Getting issue details... STATUS
- - DM-20695Getting issue details... STATUS
- - DM-21871Getting issue details... STATUS
- - DM-19470Getting issue details... STATUS
- - DM-21904Getting issue details... STATUS
- - DM-21872Getting issue details... STATUS
- - DM-26483Getting issue details... STATUS
- - DM-15257Getting issue details... STATUS
- - DM-25013Getting issue details... STATUS
- Support for core testing/development
- BPS Continue/Parallel development needed
- Full Gen3 (weekly-scale) runs of RC2 and DC2 occur and are now the source of truth weekly regressions.
HSC RC2 Data Processing Parity Checklist (Tasks for Science Pipelines)
- - DM-20695Getting issue details... STATUS : PsfWcsSelectImagesTask: Task to select images for coaddition
- - DM-19470Getting issue details... STATUS : Fit astrometry via Jointcal
- - DM-26791Getting issue details... STATUS : Fit photometry via FGCM
- - DM-17062Getting issue details... STATUS : Apply Jointcal and FGCM calibrations during coaddition
- - DM-22277Getting issue details... STATUS : Write|Transform|Consolidate Source Table Task
- - DM-24638Getting issue details... STATUS : Write|Transform|Consolidate Object Table Task
- Calibration Products Generation
QA tools intentionally not on this list.