In the original vision for the Rubin middleware, it was envisioned as something usable outside the project (and indeed even outside astronomy). In the end what we've ended up with in Gen3 is something which is conceptually fairly closely tied to astronomy but is still free of intrinsic Rubin dependencies. (It has many incidental Rubin dependencies because of the history of its development and in particular because of the residual need for keeping Gen2 and Gen3 working at the same time.)
The SPHEREx project is seriously evaluating the use of the Gen3 middleware for its Level 1-3 pipelines. (Ideally the resulting software would also be of interest to the SPHEREx science community, as well as used in the long-term support of calculational tools for SPHEREx when its data are released to the IRSA archive, but these are not requirements.)
Gregory Dubois-Felsmann is the "pipeline system designer" for SPHEREx and thus the instigator of this attempt at actual code re-use. Tatiana Goldina, previously a member of the Rubin "SUIT" group at IPAC, is now a SPHEREx developer, and is familiar with the Rubin stack environment and some of its idiosyncracies.
SPHEREx is not planning to use afw
in its pipelines. Instead, the pipelines will be dominantly Python-based, using components from Astropy, and likely using Tractor, or an evolution of it, for photometry. The project would like to avoid dependencies on afw
and on the associated C++ build infrastructure of the Rubin project. It appears that there's nothing intrinsic to the Gen3 design that prevents this, though as of today (Spring 2021) there are in practice quite a few dependency paths from Gen3 into the Rubin C++ world.
We are working on refactoring the Gen3 software to avoid these dependencies.
daf_butler
itself has already been designed to avoid Rubin C++/afw
dependencies.
This work is tangentially entangled with a Rubin-motivated refactoring of the packages above afw
that contain the high-level definitions of the Rubin pipelines:
-
RFC-775Getting issue details...
STATUS
Steps in this process:
1. Create a new package task_base
This new package will carry Task
and its immediate requirements, and free of dependencies on either Butler or C++. The creation of this package is part of RFC-775. This has the following motivations:
- Project-specific
Task
code containing key algorithms can be written in a way that facilitates reuse even by people not interested in using the Butler infrastructure for I/O - This can be done quickly and allows SPHEREx work on
Task
s to begin now, before the rest of the refactoring is completed (which, among other things, awaits the retirement of Gen2)
This work has raised the following two smaller-scale RFCs for the elimination of C++ dependencies:
-
RFC-782Getting issue details...
STATUS
and
-
RFC-783Getting issue details...
STATUS
. These RFC envision the replacement, in their use in Task
, of C++ logging and the C++-base PropertyList
and PropertySet
, and the creation of compatibility interfaces that avoid forcing an immediate "big bang" migration on the Rubin pipelines.
Initially task_base
will be briefly deployed as a stand-alone package, included in repos.yaml
but not depended on by anything else.
This step is sufficient to allow development on Task
code in SPHEREx to begin.
Then...
2. Modify pipe_base to be based on task_base
This step, intended to be transparent to Rubin stack code, modifies pipe_base
to obtain Task
and its relatives from task_base
while leaving forwarding code in place so that other parts of the Rubin stack can still successfully import, e.g., Task
, from pipe_base
instead of its new home. This forwarding code is again intended to allow a soft migration of higher-level code.
Then...