Delivery Date

Presumably just prior to Aug 2019 PCW meeting.

Misc work proposals for this milestone

Needs to have all input and then prioritized against available effort

What is required for pipeline hack-a-thon at Aug PCW, for production use cases through end FY2019, and getting rid of remaining Gen2 pieces?  

  • Placeholder for fixing problems/improving performance running RC2 
  • Unique ids (DM tech note dmtn-099)
  • Multi-registry (a substantial prototype using Oracle must exist before Gen3 )
    • Depends upon unique ids
  • Ability to manually run unit tests against non-sqlite backends
    • Whether this ever gets added 
  • Config rework (initial discussion started here)
  • Any work to support ingestion of auxTel (ComCam, ImSim) raw files?
  • Remaining work from  Production Gen3 Middleware Requests (Google Doc)
    • Subset, merge (more discussions here and here)
    • Depends upon unique ids
  • Get v1 of stable schema
  • Quantum Graph pickle portability (paths baked in?)
  • Pipeline schema files are treated as datasets/files in QuantumGraph
  • LSST software stack changes:
    • upgrade SQLAlchemy
    • include Oracle client and cx_Oracle
    • ctrl_bps move from lsst-dm to lsst Git (no objections to it staying in lsst-dm, but must exist in lsst before any official production runs which would be when?)
  • BPS have own job activator 
    • set Oracle action before each runQuantum so can tie Oracle traces to at least a particular pipelineTask (if not particular quantum)
    • call runQuantum method instead of subprocess call to pipetask
    • Not clear if this is only BPS work or some reorg in pipetask is required
  • BPS call QG generator directly (no subprocess)
    • BPS work + some reorg in pipetask
  • BPS database tables 
    • Operator and runtime information 
  • Datastore work in particular caching and in-memory (info avail Datastore Questions)
  • Gen3 version of ci_hsc (automatically run like Gen2 version) using single-node activator (pipetask) and sqlite (Nate is already working on this)
  • Butler code reorg/cleanup
    • 2 different places for table creation (Registry uses one and Datastore the other)
    • Can some of the Oracle registry bits be pulled out to a more generically named Registry that can be reused by other DB (e.g., PostgreSQL)?
  • (Posix) Datastore programmed differently than Registry wrt database.   Can it be more like Registry?
    • Table definition not in config/yaml file
    • Always checks whether table exists or not whereas Registry only does that in codes like makeButlerRepo.py
    • ??? 
  • Milestone report
    • Go through data access requirements to list potential remaining work
  • No labels