2021-04-14 Build Workshop

Pip v conda in nublado and user environments

• We should ask conda upstream to symlink instead of hardlink (based on https://github.com/conda/conda/issues/7867, the answer is a hard no)
• We need a plan to address or to waive off the requirement for pins (KT/FE)

• We have to clarify with Product Owners the backward compatibility guarantee in nublado; can we limit it to eg. commandline
• We need a technote to explain how that guarantee is going to be met (TJ/KT/FE)

• SQuaRE will create a process/configuration to cleanly separate service machinery packages from user-requested packages
• SQuaRE will propose a process to DM-CCB that maintains provenance of package installation requests (and consider what testing can be added for 3rd party requested packages)
• SQuaRE/KT to produce a technote describing how to manage 3rd parties technically, eg. with a stack→3rd party→JL cake
• JL extensions by users for now have to be owned by SQuaRE. In the future pre-built extensions will be available in JL3 and may be a partial or complete solution to that problem

• We are required to offer popular common astronomical packages in the nublado environment
• Right now nublado machinery and user requested packages are comingled ← problem
• Process for dealing with user requests is ad-hoc

• lsst_sims support in nublado will wait until sims builds with conda
• at which point it will be treated as a third party (see above)
• meanwhile users who have asked for lsst_sims in nublado have access to lsst-dev and are not blocked
• for summit use, for now, people can install in their home directory using eups
• the new build engineer could be made available to help the Sims team transition to their conda model if they can't get there themselves

SQR-054 review

We've done most of sqr-054 already.

• There's only one Python 3 on the system, and it comes from the stack.
• We use conda for everything we can.
• There's only one kernel, and it is based on the stack installation.
• There are currently no pinned builds.

Walk-on issues

• Naming/version convention for stack releases
• multiple lab containers per stack container naming convention 

• document naming convention for official releases to be specified so it can be testable (KT)
• SQuaRE will write a test class on the python side
• each time we build a new version of a container we should change a version by upping a docker tag
• the above is done for technical reasons and should not be visible to the user - newest is best
• we should not depend on metadata in Docker Hub that is not available in the protocol through eg Nexus

• SQuaRE will implement a way for the telescope deployments to have a different recommended per telescope deployment

• Tel deployments need to have different recommendeds than Sci deployments

• try using the stack→t&s container that t&s builds and layer lab on. top - may need multi-usermode work done 

• Currently the cake is stack→lab→t&s
• This is bad, how can we do better
• t&s more integrated with dm than anticipated (this is good, but there is entanglement)

• there is going to be a daily nublado-tel container build that consists of: stack daily → t& sanctioned → lab-tel
• we need to establish a time-budget that results in the above being produced and available in < 3hrs so quick fix turnaround before next night's observing is possible
• we should consult widely and revisit the times for the daily builds of services to prioritize start of data taking
• during-night hotfixes are possible using eups within the fixer's container, and can be upstreamed for integration into the daily build for next night
• similar approach could be taken for non-Science Platform containerized services (eg. header service, OODS)

• Time budget for fixing something reported overnight and having it available that night
• Higher frequency dm builds on slower cadence t&s builds

• change SQR-056 to specify that NTS is available for non-maintenance window integration work if you ask #ncsa-integ-teststand (but avoid night-time in Chile)
• when the base teststand is available, then the tucson test stand can be used an integration environment

• Maint windows working well for Tel
• NTS is both considered prod and int ←  this is a problem

• versioning for lab-tel containers has to reflect both the stack build and the t&s sanctioned version (user-visible)
• SQuaRE to figure out how to make sure spawner page does not display containers built on obsolete sanctioned versions