Date: Thu, 28 Mar 2024 21:18:51 +0000 (UTC)
Message-ID: <419821708.21481.1711660736160@confluence>
Subject: Exported From Confluence
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_21480_1251175336.1711660731117"
------=_Part_21480_1251175336.1711660731117
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-Location: file:///C:/exported.html
Checklist: Gen3 ready for general use within DM and encouraged for=
advanced users (DM-DAX-12 Feature Par=
ity)
List of potential
We are aiming for November 1st.
Planned Wor=
k
Tickets indicate planned work. Checkboxes without tickets =
are (hopefully) all statements about functionality that is available.
- Sche=
ma changes (changes requiring wholesale re-ingest could occur but sp=
aringly... migration scripts may be possible)
- DM-26407 - Jira project doesn't exist or you don=
't have permission to view it.
- <=
span class=3D"icon-in-pdf"> DM-21231 - Jira project doesn't exist or =
you don't have permission to view it.
- DM-24432 - Jira project doesn't exist or you don=
't have permission to view it.
-
DM-26476 - Jira project doesn't exist or y=
ou don't have permission to view it.
- DM-27033 - Jira project doesn't exist or you don=
't have permission to view it.
-
DM-26630 - Jira project doesn't exist or y=
ou don't have permission to view it.
-
DM-26738 - Jira project doesn't exist or y=
ou don't have permission to view it.
-
DM-26978 - Jira project doesn't exist or y=
ou don't have permission to view it.
- General usability
-
DM-26600 - Jira project doesn't exist or y=
ou don't have permission to view it.
- DM-26726 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26371 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-24414 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26739 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-27140 - Jira project doesn't exist or you don=
't have permission to view it.
- list of needed Ge=
n3 Butler functions/command-line tools
- =
DM-26684 - Jira project doesn't exist or=
you don't have permission to view it. : simple command-line tool (=
prune-collection), high priority
- =
DM-26685 - Jira project doesn't exist or=
you don't have permission to view it. : more complex command-line =
tool (query-datasets), unblocks many others, only slightly lower priority=
strong>
- =
DM-26690 - Jira project doesn't exist or=
you don't have permission to view it. : more complex command-line =
tool (query-data-ids), may not unblock as much
- =
DM-26856 - Jira project doesn't exist or=
you don't have permission to view it. : butler method and command-=
line tool (prune/unregister/delete/remove-dataset-type)
- DM-26874 - Jira project doesn't exist or you don=
't have permission to view it. : rename pipetask2 to p=
ipetask, remove old pipetask
- BPS<=
/span> Features needed for developers (single/multi-node)
- DM-26458 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-27039 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26402 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26734 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26408 - Jira project doesn't exist or you don=
't have permission to view it. (Initial unit tests are not ex=
plicitly a requirement for Gen2 deprecation)
- DM-26385 - Jira project doesn't exist or you don=
't have permission to view it. (Again not explicitly a requir=
ement for Gen2 deprecation, but should be a small change that will be less =
confusing for users and remove code.)
- Documentation
- DM-21687 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26724 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-26406 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-24329 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26736 - Jira project doesn't exist or you don=
't have permission to view it.
Open Questions
- Raw Ingest works (caveat: =
; DM-21706=
- Jira project doesn't exist or you don't have permission to view it. ) but:
- still an open question a=
s when to run define-visits.
- Handling of photo diodes=
for BOT will need special ingest script and associated formatter infrastru=
cture.
- We are still dependent o=
n the Filter singleton being removed if we wish to combine multiple instrum=
ents in a repository.
Acceptance Test=
s
We regularly run small test pipeline with Postgres. ci_hsc_gen3 runs ent=
irely with gen3 on a single node with multiprocessing and sqlite/local file=
. Google POC has demonstrated we can run an RC2 tract with condor usi=
ng S3+Postgres (DMTN-157). These tests all assume shared everything =
where multiple users are using a shared datastore and a shared postgres dat=
abase. These tests can not be complete given the current status of cp=
_pipe with Gen3 and the lack of jointcal.
Below this point is open for discussion but pertains more to Gen2 deprec=
ation (list of appropriate tickets)
- DM-26173 Write=
plan for Gen2 deprecation
Remaining Checklist: DM-DAX-1=
3 (Gen3 ready for Gen2 Deprecation )
Select any items below that are required for Gen2/3 parity and h=
ence need to be completed by Nov 1st.
- All Schema changes are accompanied but a migration=
mechanism (or have appropriate CCB approval).
- Additional command-line tooling (fu=
nctionality already accessible from Python, which we think is adequate for =
declaring feature parity in these cases)
- DM-26686 - Jira project doesn't exist or you don't have p=
ermission to view it.
- =
DM-26688 - Jira project doesn't exist or=
you don't have permission to view it.
- =
DM-26689 - Jira project doesn't exist or=
you don't have permission to view it.
- Gen3 continued development (would b=
e on-going and even supplemented/aided by general DM users/developement)&nb=
sp;
- DM-21333 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-21832 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-23985 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-26277 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-20695 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-21871 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-19470 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-21904 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-21872 - Jira project doesn't exist or you don't have permission=
to view it.
- <=
span class=3D"icon-in-pdf"> DM-26483 - Jira project doesn't exist or =
you don't have permission to view it.
- DM-15257 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-25013 - Jira project doesn't exist or you don=
't have permission to view it.
- Support for core testing/developmen=
t
- <=
span class=3D"icon-in-pdf"> DM-23576 - Jira project doesn't exist or =
you don't have permission to view it.
- DM-21865 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-22038 - Jira project doesn't exist or you don't have permission=
to view it.
- DM-22663 - Jira project doesn't exist or you don=
't have permission to view it.
- <=
span class=3D"icon-in-pdf"> DM-21888 - Jira project doesn't exist or =
you don't have permission to view it.
- DM-21916 - Jira project doesn't exist or you don=
't have permission to view it.
- BPS Continue/Para=
llel development
- DM-26385 - Jira project doesn't exist or you don=
't have permission to view it.
- DM-20406 - Jira project doesn't exist or you don=
't have permission to view it.
- <=
span class=3D"icon-in-pdf"> DM-26494 - Jira project doesn't exist or =
you don't have permission to view it.
- Full Gen3 (weekly-s=
cale) runs of RC2 and DC2 occur and are now the source of truth weekly regr=
essions.
-
HSC RC2 Data Processing=
Parity Checklist (Tasks for Science Pipelines)
- DM-20695 - Jira project doesn't exist or you don=
't have permission to view it. : PsfWcsSelectImagesTask: Task to se=
lect images for coaddition
- DM-19470 - Jira project doesn't exist or you don=
't have permission to view it. : Fit astrometry via Jointcal
- DM-26791 - Jira project doesn't exist or you don=
't have permission to view it. : Fit photometry via FGCM=
- DM-17062 - Jira project doesn't exist or you don=
't have permission to view it. : Apply Jointcal and FGCM calibratio=
ns during coaddition
- DM-22277 - Jira project doesn't exist or you don=
't have permission to view it. : Write|Transform|Consolidate Source=
Table Task
- DM-24638 - Jira project doesn't exist or you don=
't have permission to view it. : Write|Transform|Consolidate Object=
Table Task
- Calibration Products Gene=
ration
- DM-21786 - Jira project doesn't exist or you don=
't have permission to view it. : Convert PTC code for Gen3.
- DM-24704 - Jira project doesn't exist or you don=
't have permission to view it. : Convert Brighter-Fatter Kernel cod=
e for Gen3.
- DM-25424 - Jira project doesn't exist or you don=
't have permission to view it. : Convert Defects for Gen3.
- DM-25425 - Jira project doesn't exist or you don=
't have permission to view it. : Convert Curve for Gen3.
QA tools intentionally not on this list.
------=_Part_21480_1251175336.1711660731117--