In the original vision for the Rubin middleware, it was envisioned as something usable outside the project (and indeed even outside astronomy).  In the end what we've ended up with in Gen3 is something which is conceptually fairly closely tied to astronomy but is still free of intrinsic Rubin dependencies.  (It has many incidental Rubin dependencies because of the history of its development and in particular because of the residual need for keeping Gen2 and Gen3 working at the same time.)

The SPHEREx project is seriously evaluating the use of the Gen3 middleware for its Level 1-3 pipelines.  (Ideally the resulting software would also be of interest to the SPHEREx science community, as well as used in the long-term support of calculational tools for SPHEREx when its data are released to the IRSA archive, but these are not requirements.)

Gregory Dubois-Felsmann is the "pipeline system designer" for SPHEREx and thus the instigator of this attempt at actual code re-use.  Tatiana Goldina, previously a member of the Rubin "SUIT" group at IPAC, is now a SPHEREx developer, and is familiar with the Rubin stack environment and some of its idiosyncracies.


SPHEREx is not planning to use afw in its pipelines.  Instead, the pipelines will be dominantly Python-based, using components from Astropy, and likely using Tractor, or an evolution of it, for photometry.  The project would like to avoid dependencies on afw and on the associated C++ build infrastructure of the Rubin project.  It appears that there's nothing intrinsic to the Gen3 design that prevents this, though as of today (Spring 2021) there are in practice quite a few dependency paths from Gen3 into the Rubin C++ world.

We are working on refactoring the Gen3 software to avoid these dependencies.

daf_butler  itself has already been designed to avoid Rubin C++/afw dependencies.

This work is tangentially entangled with a Rubin-motivated refactoring of the packages above afw  that contain the high-level definitions of the Rubin pipelines:  RFC-775 - Getting issue details... STATUS

Steps in this process:

1. Create a new package task_base

This new package will carry Task and its immediate requirements, and free of dependencies on either Butler or C++.  The creation of this package is part of RFC-775.  This has the following motivations:

  1. Project-specific Task code containing key algorithms can be written in a way that facilitates reuse even by people not interested in using the Butler infrastructure for I/O
  2. This can be done quickly and allows SPHEREx work on Tasks to begin now, before the rest of the refactoring is completed (which, among other things, awaits the retirement of Gen2)

This work has raised the following two smaller-scale RFCs for the elimination of C++ dependencies:  RFC-782 - Getting issue details... STATUS  and  RFC-783 - Getting issue details... STATUS .  These RFC envision the replacement, in their use in Task, of C++ logging and the C++-base PropertyList and PropertySet, and the creation of compatibility interfaces that avoid forcing an immediate "big bang" migration on the Rubin pipelines.

Initially task_base will be briefly deployed as a stand-alone package, included in repos.yaml but not depended on by anything else.  

This step is sufficient to allow development on Task code in SPHEREx to begin.

Then...

2. Modify pipe_base to be based on task_base

This step, intended to be transparent to Rubin stack code, modifies pipe_base to obtain Task and its relatives from task_base while leaving forwarding code in place so that other parts of the Rubin stack can still successfully import, e.g., Task, from pipe_base  instead of its new home.  This forwarding code is again intended to allow a soft migration of higher-level code.

Then...

3. Remove C++ and Rubin-specific dependencies from pipe_base and ctrl_mpexec


  • No labels