"Like Winter2015 Package Reorganization Planning, only not impossible!"
Bubbles of Like-Minded Packages
old packages (not complete) | purpose | |
---|---|---|
basic | base, daf_base, utils, pex_exceptions | low-level packages used by nearly all higher-level code (selectively used by qserv) |
butler | daf_butlerUtils | butler and data access middleware |
sci | meas_*, ip_*, coadd_*, pipe_tasks | high-level science pipelines |
task | pipe_base, pex_config | pipeline definition base classes and configuration |
afw | afw | low-level science data structures and algorithms |
ctrl | ctrl_* | execution middleware |
qserv | qserv_* | distributed database |
qa | testing_pipeQA, testing_displayQA | quality analysis and visualization tools |
ui | science user interface | |
production | ||
legacy | pex_policy, pex_logging, daf_persistence | code we have retired or want to retire soon |
Motivations for Splitting and Merging Packages
When We Split Packages
- Low-level components are stable, but high-level components are not.
- Some components are needed by a higher-level package, but other components are not.
- Components are not related to each other conceptually.
When We Merge Packages
- Users of any of the components typically want all of the components.
- Interfaces are tightly coupled and cannot be modified separately.
- Components have circular dependencies.
- Components are conceptually similar.
What Reorganization Won't Fix
- Build times: we need to refactor how we handle intrapackage Swig to make a difference here.
- Organization-as-documentation vs. organization-for-dependencies: we can't simultaneously optimize both of these metrics, and sometimes they conflict.
- Names: some of them will always be terrible. Let's not get distracted talking about them now (save that for the individual RFCs!)
Specific Proposals
- base, utils, pex_exceptions, daf_base: merge all of these?
- meas_algorithms: move Jarvis/shapelet code to separate, legacy package.
- afw: move Wcs from afw::image to afw::coord
- afw: move Kernel and warping from afw::math, Psf from afw::detection to (e.g.) afw::convolution
- meas_*, coadd_*, pipe_tasks (all of these end up with sci_ prefixes?)
- one package focused on single-frame/forced processing interfaces and utilities:
- subtask interfaces (for CalibrateTask + subtasks, measurement, deblending detection)
- mid-level driver code - not CmdLineTasks (currently split across pipe_tasks and meas_algorithms)
- miscellaneous utilities (CRs, backgrounds, repair - mostly in meas_algorithms)
- one package for standard PSF implementations (currently a subset of meas_algorithms)
- one package for standard deblender implementations (current meas_deblender)
- one package for standard astrometry implementations (current meas_astrom + AstrometryTask from pipe_tasks)
- one package for standard measurement plugins (current meas_base, probably meas_extensions_photometryKron too)
- one package for coaddition; includes command-line drivers (from coadd_* and pipe_tasks)
- one package for single-frame/forced processing command-line drivers
- one package for image differencing
- one package for source model fitting (meas_modelfit) - separate from other measurement plugins due to dependencies, build time
- one package for HSM shape measurements (meas_extensions_shapeHSM) - separate from other measurements due to dependencies, external origin
- one package focused on single-frame/forced processing interfaces and utilities:
- shapelet, skymap: just add a sci_ (or maybe afw_?) prefix; there will be more libraries like this in the future: astronomy-specific utility (not plugin or algorithm) code that isn't as broadly useful as what's in the current afw
To Legacy
- pex_policy
- pex_logging
- daf_persistence
- afw::formatters
- ap
- datarel