Checklist: Gen3 ready for general use within DM and encouraged for advanced users (DM-DAX-12 Feature Parity)

List of potential 

We are aiming for November 1st.

Planned Work

Tickets indicate planned work.  Checkboxes without tickets are (hopefully) all statements about functionality that is available.

  • Schema changes (changes requiring wholesale re-ingest could occur but sparingly... migration scripts may be possible)
  •  General usability
  • list of needed Gen3 Butler functions/command-line tools
    • DM-26684 - Getting issue details... STATUS : simple command-line tool (prune-collection), high priority
    • DM-26685 - Getting issue details... STATUS : more complex command-line tool (query-datasets), unblocks many others, only slightly lower priority
    • DM-26690 - Getting issue details... STATUS : more complex command-line tool (query-data-ids), may not unblock as much
    • DM-26856 - Getting issue details... STATUS : butler method and command-line tool (prune/unregister/delete/remove-dataset-type)
    • DM-26874 - Getting issue details... STATUS : rename pipetask2 to pipetask, remove old pipetask
  • BPS Features needed for developers (single/multi-node)
    • DM-26458 - Getting issue details... STATUS
    • DM-27039 - Getting issue details... STATUS
    • DM-26402 - Getting issue details... STATUS
    • DM-26734 - Getting issue details... STATUS
    • DM-26408 - Getting issue details... STATUS  (Initial unit tests are not explicitly a requirement for Gen2 deprecation)
    • DM-26385 - Getting issue details... STATUS  (Again not explicitly a requirement for Gen2 deprecation, but should be a small change that will be less confusing for users and remove code.)
  • Documentation

Open Questions

  • Raw Ingest works (caveat:   DM-21706 - Getting issue details... STATUS ) but:
    • still an open question as when to run define-visits.
    • Handling of photo diodes for BOT will need special ingest script and associated formatter infrastructure.
  • We are still dependent on the Filter singleton being removed if we wish to combine multiple instruments in a repository.

Acceptance Tests

We regularly run small test pipeline with Postgres. ci_hsc_gen3 runs entirely with gen3 on a single node with multiprocessing and sqlite/local file.  Google POC has demonstrated we can run an RC2 tract with condor using S3+Postgres (DMTN-157). These tests all assume shared everything where multiple users are using a shared datastore and a shared postgres database.  These tests can not be complete given the current status of cp_pipe with Gen3 and the lack of jointcal.


Below this point is open for discussion but pertains more to Gen2 deprecation (list of appropriate tickets)

  • DM-26173 Write plan for Gen2 deprecation

Remaining Checklist:  DM-DAX-13 (Gen3 ready for Gen2 Deprecation )

Select any items below that are required for Gen2/3 parity and hence need to be completed by Nov 1st.

  • All Schema changes are accompanied but a migration mechanism (or have appropriate CCB approval). 
  • Additional command-line tooling (functionality already accessible from Python, which we think is adequate for declaring feature parity in these cases)
  • Gen3 continued development (would be on-going and even supplemented/aided by general DM users/developement) 
  • Support for core testing/development
  • BPS Continue/Parallel development needed
  • Full Gen3 (weekly-scale) runs of RC2 and DC2 occur and are now the source of truth weekly regressions.
  •  


HSC RC2 Data Processing Parity Checklist (Tasks for Science Pipelines)

  • DM-20695 - Getting issue details... STATUS : PsfWcsSelectImagesTask: Task to select images for coaddition
  • DM-19470 - Getting issue details... STATUS : Fit astrometry via Jointcal
  • DM-26791 - Getting issue details... STATUS :  Fit photometry via FGCM
  • DM-17062 - Getting issue details... STATUS : Apply Jointcal and FGCM calibrations during coaddition
  • DM-22277 - Getting issue details... STATUS : Write|Transform|Consolidate Source Table Task
  • DM-24638 - Getting issue details... STATUS : Write|Transform|Consolidate Object Table Task 
  • Calibration Products Generation
    • DM-21786 - Getting issue details... STATUS : Convert PTC code for Gen3.
      • DM-24704 - Getting issue details... STATUS : Convert Brighter-Fatter Kernel code for Gen3.
    • DM-25424 - Getting issue details... STATUS : Convert Defects for Gen3.
    • DM-25425 - Getting issue details... STATUS : Convert Curve for Gen3.

QA tools intentionally not on this list.


  • No labels