In the original vision for the Rubin middleware, it was envisioned as something usable outside the project (and indeed even outside astronomy). In the end what we've ended up with in Gen3 is something which is conceptually fairly closely tied to astronomy but is still free of intrinsic Rubin dependencies. (It has many incidental Rubin dependencies because of the history of its development and in particular because of the residual need for keeping Gen2 and Gen3 working at the same time.)
The SPHEREx project is seriously evaluating the use of the Gen3 middleware for its Level 1-3 pipelines. (Ideally the resulting software would also be of interest to the SPHEREx science community, as well as used in the long-term support of calculational tools for SPHEREx when its data are released to the IRSA archive, but these are not requirements.)
Gregory Dubois-Felsmann is the "pipeline system designer" for SPHEREx and thus the instigator of this attempt at actual code re-use. Tatiana Goldina, previously a member of the Rubin "SUIT" group at IPAC, is now a SPHEREx developer, and is familiar with the Rubin stack environment and some of its idiosyncracies.
SPHEREx is not planning to use
afw in its pipelines. Instead, the pipelines will be dominantly Python-based, using components from Astropy, and likely using Tractor, or an evolution of it, for photometry. The project would like to avoid dependencies on
afw and on the associated C++ build infrastructure of the Rubin project. It appears that there's nothing intrinsic to the Gen3 design that prevents this, though as of today (Spring 2021) there are in practice quite a few dependency paths from Gen3 into the Rubin C++ world.
We are working on refactoring the Gen3 software to avoid these dependencies.
daf_butler itself has already been designed to avoid Rubin C++/
This work is tangentially entangled with a Rubin-motivated refactoring of the packages above
afw that contain the high-level definitions of the Rubin pipelines:
RFC-775Getting issue details...
Steps in this process:
1. Create a new package task_base
This new package will carry
Task and its immediate requirements, and free of dependencies on either Butler or C++. The creation of this package is part of RFC-775. This has the following motivations:
Taskcode containing key algorithms can be written in a way that facilitates reuse even by people not interested in using the Butler infrastructure for I/O
- This can be done quickly and allows SPHEREx work on
Tasks to begin now, before the rest of the refactoring is completed (which, among other things, awaits the retirement of Gen2)
This work has raised the following two smaller-scale RFCs for the elimination of C++ dependencies:
RFC-782Getting issue details...
RFC-783Getting issue details...
. These RFC envision the replacement, in their use in
Task, of C++ logging and the C++-base
PropertySet, and the creation of compatibility interfaces that avoid forcing an immediate "big bang" migration on the Rubin pipelines.
task_base will be briefly deployed as a stand-alone package, included in
repos.yaml but not depended on by anything else.
This step is sufficient to allow development on
Task code in SPHEREx to begin.
2. Modify pipe_base to be based on task_base
This step, intended to be transparent to Rubin stack code, modifies
pipe_base to obtain
Task and its relatives from
task_base while leaving forwarding code in place so that other parts of the Rubin stack can still successfully import, e.g.,
pipe_base instead of its new home. This forwarding code is again intended to allow a soft migration of higher-level code.