Science Pipelines and SQuaRE
Eliminate boilerplate in source files
Jonathan is working on this. Will be done after the HSC fork is merged. Will be an automatic transition managed by SQuaRE.
Switching to NumPy docstring format
Waiting for the new documentation build system, "LSST the Docs". Should be in the next week or two. Will be discussed in the "decision making" session tomorrow.
Still work to be done in determining the best way to document tasks and how tutorials and examples should be published
Continuous Integration
What does it mean to have "enough" CI to perform major changes? Suggestion that we need all the supported cameras processed through to measurement on coadds every week. This should include regression tests as well as tests for absolute values and how they relate to the science requirements.
What is a supported camera? SQuaRE emphasize that we should not guarantee to support in perpetuity every obs_
package that is submitted. Agreed that they should have designated "sponsors" who will maintain them. However, if a change to the stack causes a test to break for a specific camera, our first assumption should be that it's an algorithmic error rather than a bug in obs_foo
.
Plans for nightly QA runs that will run large scale (~hours) tests. Regular CI will run shorter (minutes) tests.
Limited developer support in Spring 16 and lsst-dev
shared stack
SQuaRE will provide only limited developer support during the short Summer 16 issue. Science Pipelines identify one issue deserving of immediate support as the shared stack on lsst-dev
. JDS will act as point of contact for SQuaRE on this.
Process Control and Data Access
Issues to discuss in details
- Abstracting access to OCS through butler
- Fetching data from staging are to disk (e.g., templates, exposures for forced dia photometry)
- provisioning hardware for L1: when, what
- batch processing related needs for L2/L3
- data backbone for L2/L3
- cross-site failure recovery
- cross-site upgrades (are all sites required to be on the same release etc)
- replicating L3 across different sites / L3 user mydb synchronization across multiple sites
- distributing DRP products - via network or physical shipping?
- storage technology for large files (object store)
- capturing provenance (Cooper starting to think about it)
- gray area: who captures provenance about OS/hardware
how much containerization should we be doing?
does it simplify provenance capture?
- NCSA is writing docs about L2
- Margaret will expose to Data Access when the docs are in reasonable shape
- need to bring IN2P3 to Data Access <-> NCSA into discussions (via JCC)
SUI/T and Architecture
Science Pipelines and Architecture
Previous coding standard decisions
Python 3, idiomatic Python, etc.
Almost all of this work has been blocking on the HSC merge. However, the onus is then on Science Pipelines to decide if and when to schedule the work: Architecture will not make this decision. Should assess:
- Is work done as a big bang or incrementally?
- If work is incremental, how does it impact ongoing tickets?
(KTL adds: Prioritization needs to happen through the usual mechanisms with input from Science Pipelines and Architecture. Science Pipelines does not get to make this decision unilaterally either.)
Rough estimate that the effort to make our Python code "idiomatic" would be two weeks of work for three people. Important to have enough test coverage in place to make sure this doesn't break anything; running an HSC data release candidate should be sufficient.
The plan is to carry out the Python 3 work in a "big bang" hack session at the August All Hands.
AstroPy
There will be a meeting with the AstroPy developers at UW in ~1 month. Where we draw the line in terms of stack integration has a huge impact on the amount of effort required. Aim to have a rough idea of this before the meeting, but expect to refine it based on input from the AstroPy folks. An ideal outcome might be to produce AstroPy affiliated packages which provide C++ APIs that we can use in the stack.
Many-core architectures
It is likely that future architectures, as we might reasonably expect to run on in operations, will skew towards large numbers of cores per system, with relatively small memory and storage per core. This suggests we should move towards threading in our high-performance C++ code rather than relying on multiprocessing at a higher level. The likely, but not definitive, technology choice is OpenMP. We do not regard Science Pipelines as responsible for "blue skies" research in this area, but it is reasonable to expect that they will provide examples and requirements. We do not think this decision can be driven purely "bottom-up" but will ultimately require input from the Architecture team.
Process Control and SQuaRE
SUI/T and Data Access
We all would like to exercise the SUI to Data Access system in a basic way regularly, deploy nightly out of NCSA. The idea is to get this going with a very small repository. Then, when we receive the PanSTARRS data, SUI can start to use that data to write a robust set of regression tests.
Jacek would like to get the small set of available around July-Augus 2016. Before then SUI will deliver example queries so we can verify we’re prepared to run the tests.
Senior management should prioritize how important it is to deliver Pan-STARRS data through Qserv/SUIT to our users, and define timeline. Two months ago the top two priorities were: end-to-end system, and serving Pan-STARRS data
Once the Pan-STARRS data arrives it should take Data Access about a month to get the pan stars data ready. Note, we don’t know what format it’s coming in; we will need to think about how to load it. After loaded, keep it available "for ever". Ok to limit access for users during scheduled stress DB tests.
Remote Butler - SUI really wants a Java client to the remote Butler, for the Firefly service. We need to discuss it with the architecture group, but perhaps we should consider *not* doing the python butler client for now, and doing the Java one instead. Java client useful for SUIT server side (performance reasons, does not want to fork python process). Python access still needed for SUIT user access.
Would be nice to have simple prototype in Fall 2016 that demonstrates credentials acquiring/passing
Science Pipelines and Process Control
Computing resources for developers
There's a modest cluster hanging off the back of lsst-dev
which is available for developers to use. Use the ctrl_orca
middleware to access it. Documentation is available on Confluence.
This cluster is not addressable by the ctrl_pool
middleware recently ported from HSC. Adding this functionality to ctrl_pool
should be a relatively modest task for Science Pipelines developers, although it probably falls strictly outside their remit per WBS. Getting this working will be a requirement in the intermediate future.
Future middleware development
Some discussion of the SuperTask concept and how it might relate to Science Pipelines. Nobody in the room is confident in describing either when it will be available or what impact it might have on existing tasks. We hope this will be clarified soon.
Hsin-Fang has volunteered to be a contact at NCSA for fielding short term maintenance issues with the existing middleware (pipe_base
, pex_config
etc). The Science Pipelines group will be proactive about putting larger requests to the Middleware Group well in advance of future cycles so that they can be included in the plan.
The group at NCSA which will be working on middleware is gradually ramping up in terms of staffing and experience.
Many-core architectures
Agreement that there will be effort available at NCSA during Summer 2016 to support an investigation into future stack parallelization strategies.
Feedback on L1 presentation
Simon is concerned that the authors/owners of the various boxes in Don's presentation from this morning should be clearly stated, and that the interfaces where alert data passes from Science Pipelines to NCSA, or vice versa, should be documented.
Architecture and Data Access
Public Interfaces to DAX
- VO
- Architecture team owns decision which subset. The plan is to pick protocols that are in wide spread use
- Currently on NCSA WBS. Arch team and Mario wants it to be in Data Access WBS. This would increase scope of Data Access (need to estimate, will need ~1FTE for some time).
- This work also includes discussing with VO community, pushing back when protocols are insufficient for us.
- Global DM-wide VO expert was considered, we decided it is not a good idea
- RESTful webservices
Internal interfaces
- butler interfaces
- SQL / mysql. Note, that there is strong desire to NOT expose SQL / mysql interface to public, because once we do, we will be stuck with supporting mysql interface for ever
Metadata / butler
- metadata depends on butler. It can use remote access to get data via butler
- butler depends on metadata. Usecase: metadata can manage repository configurations (versions), the versions are used by butler
Saving queries for published papers
- will save query text, that is easy
- reproducibility is harder to guarantee
- we could push results to cloud (especially if requestor pays)
- side note: need to think about sharing Qserv across multiple data releases and how we would upgrade UDFs without sacrificing reproducibility
Global catalog
need more definitions of interfaces to global catalog. This needs to be discussed with NCSA. Not sure when we will know
- Since we need it soon, prototyping is useful
- but keep an eye on existing systems, like Fermi DataCat
SUI/T and SQuaRE
Science Pipelines and SUI/T
Large scale visualization
It was agreed that the large scale visualization of HSC data as presented by Robert was a compelling use case for SUI. There was some discussion about what it would take to achieve this technically. It was agreed that it would be technically feasilble to generate PNG (or equivalent) images which would be used in such a visualization as part of the data release processing (rather than post-processing the data release by the SUI group).
Interfacing Firefly with Science Pipelines
Following some work in 2015 to integrate Firefly with the afw.display
system, this effort has been languishing. The SUI group are keen to see Science Pipelines developers using, and providing feedback on, Firefly; Science Pipelines developers are keen to have access to better visualization and debugging tools. However, the barriers to entry for getting Firefly running on individual laptops are -- arguably! -- offputting.
It was agreed that we will aim to establish a Firefly service on lsst-dev
which will enable developers to visualize data stored on the filesystem. We hope this will help build a critical mass of Firefly users.
A compelling user case for future development would be to enable visualization of afw.table
s through an interface similar to afw.display
.
SQuaRE and Data Access
SQuaRE wants
- TAP interface
- write access via TAP (or RESTful), but not direct SQL
- does not care about authorization in the short term
remote access to butler (python client), read and write
S3 plugin backend for butler
Unknown User (npease) to visit them for ~ a week
Data Access wants
- integrating Qserv integration tests with CI