Running RC2 via Gen3 + Pegasus + Oracle
Not as a production operator*, but as a special developer who shares repo with other developers. Not delivering a polished, efficient service, but rather delivering the starting point for study and more work. Besides the presentation, trying to have something usable/maintainable for the following 3 months or so while next set of work is ongoing.
Results of running Gen3 DRP pipeline on RC2 HSC data on lsst-dev
Using Oracle as backend for registry of shared butler repository
Pegasus for workflow
Hsin-Fang can run Gen3 as part of the monthly HSC-RC2 reprocessing runs
Incorporate into procedures with lower expectations than Gen2 (failure rate, usefulness of outputs, operability, etc)
Instructions for friendly-user developers using Gen3 Butler with Oracle
Including accessing weekly ci_hsc repo and RC outputs
Assuming running in Pegasus would still be too rough for most friendly-user developers
Maintainable Registry code that does not require updating a copy of large portions of the code for each RDBMS for every code change
Larger scale than ci_hsc
Oracle admins can start seeing data flow and more easily provide feedback
Increases visibility of Gen3 results
Gen2 RC running is one of the blockers to deprecating Gen2
Unblocks multi-registry work required for separation of production data from user data
Provides insight for the Batch Processing Service design doc
DMLT: June 4-6, 2019
Missing separation of production data from user data (requires user write access to production schema)
The outputs of a production pipeline should not be directly written to the production Data Backbone (or central database in general) to allow the Batch Production Service to:
Minimize database connections
Use various methods for retries and restarts
Many missing Batch Production Service production features, some of which are blocked by not-yet-implemented Gen3 features.
Thurs Gen3 meetings | ci_hsc/RC2 Running | NCSA - BPS | NCSA - Oracle | Gen3 | DRP |
2/21/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (sqlite3) in pegasus to provide feedback if things are no longer working. | Jim Bosch Oracle account | Must provide updated weekly Gen3 science configs prior to NCSA run | ||
2/28/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (sqlite3) in pegasus to provide feedback if things are no longer working. | Completed: Init Oracle accounts+wallets (Nate - 03/01), nightly DB backups (03/04), weekly ingest of ci_hsc, install Oracle client and cx_Oracle on lsst-dev (03/01) | |||
3/7/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle) , ci_hsc gen3 run (sqlite3) in pegasus to provide feedback if things are no longer working. | Completed Filename template checking script | |||
3/14/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), | Decision about how to support multiple RDBMSs. Completed code changes for sqlite side. Code ready to start making Oracle changes. | |||
3/21/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), | Completed BPS v0.1 exec config, allocateNodes (partitionable slots),Completed unique filename templates ci_hsc (where only requires config file change) | Completed: Easy way to initialize dev butler schema in Oracle | ||
3/28/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (sqlite3) in pegasus to provide feedback if things are no longer working. | ||||
4/4/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (sqlite3) in pegasus to provide feedback if things are no longer working. | Completed: Oracle Butler works (no efficiency checks, just doesn't abort). Selecting Oracle schema , Table and view names case-insensitive on DB side. | |||
4/11/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (sqlite) in pegasus to provide feedback if things are no longer working. | Completed BPS v0.1 status/history scripts | Completed: Scripts to initialize ci_hsc repo for a Gen3 run without latest weekly Gen2 outputs. | ||
4/18/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (Oracle) in pegasus to provide feedback if things are no longer working. | Completed: Mechanisms to create RC2 init repo | |||
4/25/2019 | Completed ci_hsc gen2 run (sqlite to load into Oracle), ci_hsc gen3 run (Oracle) in pegasus to provide feedback if things are no longer working. | Completed: RC2 init repo avail in Oracle | Complete RC2 DRP pipeline includes always write output config options where needed. | ||
4/25/2019 | Freeze: features, API, schema | ||||
5/2/2019 | Start running RC2 and reporting problems | ||||
5/9/2019 | |||||
5/16/2019 | |||||
5/23/2019 | |||||
5/30/2019 | Completed: Can access Oracle Registry + GPFS DataStore from NCSA LSP | ||||
6/06/2019 | Milestone completed. Presentation during DMLT meeting June 04-06. Includes instructions, any software installs, etc |