PST recieved a note from DESC on "Note from DESC on pipelines". Read and discuss on #dm-sst-coord
Data security discussion with Livermore. Main remaining open issue is dealing with potential impactors. For objects that exceed 30deg/day and are determined by Livermore to not be sensitive assets, JPL/Earth impact people need to be alerted ASAP to run their own analysis for impactors.
Cybersecurity, NSF subcontracted the JASONS on security. Wil gave a report to the PST on the JASONS cybersecurity meeting at the ned of June. Discussion on implications for the summit - we need remote access. Will probably have a VPN.
Wil will propose to treat the Camera as a black box for security and ringfence it. Any implementation of security in the Camera will lead to a lot of delays.
September for a PST SciCollab talk on time sereis features? Works for Eric, (except 1 Sept)
Need to provide users with the capability to run ppl task based processing over images.
For catalogs - will be available in parquet format now so a DASK/Spark- like f/w and s/w to assist users to run over the collection of our catalogs
Variance quotas for different people - resource allocation mechanism to apply for more space beyond the baseline – Users Committee. Possibly automatically assign a small increment to avoid allocating space that is not used. 1 → 5 but to 20 has to go to an allocation committee. Limit up to which more could be allocated but beyond, might have to comtrinute money for resources.
Proposal process can take sci. merit and relase of results to teh whole community available as criteria. Promose not emforceable.
E.g new pz measure - publish as a table to federate would get extra points - have said this publicly.
In public, we have said:
used the word batch and have said their would be a batch system. i.e that users could submit batch jobs from a nb to slum, etc .....
said we would not constrain people to run entirely in our f/w. MCMC, forward modelling of catalogs that might not fit in our f/w. Implied they could run those
Democratize access to sky and replace the need to compete for telescope time. This had to come with compute resources for people with little access to resoources. These resources were not designed for DESC. They would saturate.
RHL: moset DESC questions about about can they do their science with our s/w - 10% is not enough
Batch is close to h/w - involves the USDF people.
Not standard in the RSP in a portable way.
GPDF: conventional batch + DASK/Spark system + support for m/w
CADC 2nd gen canfar - container based distributed batch-like system
RHL: classical codes - do not forclose on supporting a posix files system, e.g s-xtracter
GPDF: Difficult to escape providing posix if we are going to let people provide their own code. To not do so would not be in conflict with requirements but there isa nexpectation, we have said this, be a descope
CS: Posix file system. does the env have to match, type of data access, is data access via APIs or posix itself?
DMTN shoudl address these vague requirements in a quasi-technical manner.
GPDF: Reluctant to say that our files should be available via posix, e.g all image files would be available via a filesystem. But have a posix like fs for outputs is sustainable.
Use cases: time dependent lenses, high-redshift quasars, Andromeda - can download the data, bullk operations on catalogs.
Will write a DMTN (not just an RFC). Quasi technical language, others to provide some use cases to support this and make clear what is out of scope.
This can evolve over operations. Would be good to have for DP1. Not for DP0.2 to get feedback. Will we have user batch in the IDF at all? This will constrain the technical base.
HFC: Might use different tools to fulfill all the requirements.
Chase up whether DP1/2 will be on the IDF. What will be run at USDF/IDF? Update: Yes, DP1/2 will be on the IDF.
Eli Rykoff , Leanne Guy Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning