John asks about LSST-dev which can probably be fixed using lsstinstall. Are there other depencancies on newinstall? Seems not (most use containers which embed newinstall; changing container build fixes all of these).
RHL asks about Telescope and Site - they build on top of our containers so should be ok
GPDF asks about long term stability/availability of conda-forge ? KT thinks it has a multi year horizon, conda has interesting history and future partially supported by commercial company. Conda-forge is much more community based and has a big community.
John - who is the product owner for the build system ? - Unfortunately it's KT owning and Managing. Does KT understand who all the stakeholders are ? KT is confident he knows the people with Jenkins jobs and who the user base for lsstsw and newinstall - will go to community in any case.
How are validity ranges stored : Tim - uses the directory structure and filename. QE curves come from Camera directly and are imported. Jim - big wall in gen 3 be tween certified and those not yet certified. Export and import deserve the ??? we need more research on that.
RHL not sure squash in there for e.g. images. CamGeom deprecation was slipped in the document .. though Jim and TIm want to do this but surprised to see it in here. Otherwise happy with Document.
John - if there are technical comments it does not need all DMLT but then we are back to the outstanding action.
Colin - found it difficult to get a feel for what its describing - KTs diagram is a huge help. This may be partially why DMLT have not commented in detail. John agrees on the contend gave similar feedback to Chris - but nothing from DMLT was taken as all ok not befuddlement. If the latter we should include diagram and update.
Tim - defects easy to handle perhaps its worth having a worked example. Jim asks if KT diagram works for defects .. Tim says yes but there may be other approaches.
Jim - technote is good for the products which are fairly automatic (human yes/no) not the merged by human ones. John - we need write down we do not know when that is the case. This is somewhat the case in this doc
GPDF crosstalk corrections are handled ? Tim - yes. In a given CDB3 instance when you replace a calibration is it replaced (is it bi-temporal). Jim its not but the idea would be to have a new collection not to actually replace the old one (new name).
RHL - all the special cases for detectors are not covered - it may not be a uniform and nice as this makes it out to be. It could be messier when we get to it ... so hesitate to sign off. Back to CameraGeom ....
John Swinbank Arrange focused brainstorming meeting with RHL, TIm, Chris , Jim, Yusra, John - to get DMTN-148 further updated . Should at least list all calibration cases even if not solved.
Jim - how we access calibrations is different to how they are written - may need Robert to propose an alternate design. There is a feasability issue.
KT - best way forward ?
Christopher WatersKian-Tat Lim Modify DMTN-148 with more diagrams (from KT) and explicit statements about which products it applies to .. and which it does not apply to (and when).
John - where were we commenting on this document? From John Daniel Swinbank to Everyone: (8:59 p.m.) https://github.com/lsst-dm/dmtn-148/pull/3 From Gregory Dubois-Felsmann to Everyone: (8:59 p.m.) Is what Jim said a couple of minutes ago about what happens in BG3 when a calibration is certified going to be included in DMTN-148? From Tim J to Everyone: (9:13 p.m.) I think one of the things is that pipelines just need to be configured to use specific dataset types — that’s the optimal approach for a pipeline. Having every pipeline instead require a composite cameraGeom is overkill From Tim J to Everyone: (9:14 p.m.) but from a commissioning perspective it’s clearly easier for Robert to have access to everything in one blob From Robert Lupton to Everyone: (9:17 p.m.) I'm worried about notebooks, not pipelines. It's possible that pulling out a set of n parallel data products with the same dataId is OK, but it pushes the book-keeping onto the code. That's not too bad until the code starts by updating some of the values (e.g. the gain). Then the code becomes much more complicated, but if we just allowed setting values on the camera and doing a "put" makes the user's job much simpler. So it's a tradeoff. So that notebook may become a calib-products "pipeline". But a weird one
In discussion at the JDR, a couple of issues emerged surrounding DM's milestones:
The review recommended that our milestone tracking being more automated / streamlined;
Existing milestones being poorly defined (to the extent that the responsible T/CAMs don't know what they mean).
How can we address these?
Recording is on by consent of all for internal use.
Frossie says she did not hear it quite the same (for first point of slide 2)- automation would be good. But we need a coherent story. Would be great to have automation for Levl3 milestones - but unlikely to get it.
From chat: problem is that the milestones are not written in quantifiable ways
Question about lag - yes updates lag by a month.
How do you know which milestones are dependened on by others .. in DMTN-158 which show predecessors and successors.
Could add line for predecessors, sucessors .. Michelle/Yusra woudl like that.
John Swinbank add predecessor successor line to milestones in DMTN-158 –
KT AP Gen3 assumes all raws etc all in butler - Yes
KT Alert packet cutout sizes are limited ? yes - more work to be done
TIm - WCS is it AP or DRP ? Formally its AP. Dave Berry contract to modify AST to export Yaml ASDF format, WCS understandable by AstoPy. Means any AstroPy user can download Calexp and use our WCS.
GPDF - still outputting approx WCS in FITS standard as well as the YAML? Yes no change - ASDF has fits translation format will try to use their scheme.
Michelle - running any AP pipelines at NCSA ? Should NCSA start running them. - That would be great trying to move more to the DRP mode but there have been a lot of things holding the team back. In next few months... Ian Eric ..
Is there a plan to update estimated object counts?
Yes, although this primarily comes through the PST. No progress recently, but should look at this on a 6 month timescale.
Seeing similar burnout issues to those reported by other teams.
Note that it's hard for Tim Jenness as middleware manager to keep track of what DRP (and other) team members are doing in their non-middleware time (including personal issues, etc); consider having Tim attend T/CAM meetings, or sprint planning with DRP (and other) team members.
In May 2020 we were unable to make a 19.0.1 patch release because of incompatible changes to the build and release system since the 19.0.0 release. The Architecture team were tasked with updating and simplifying the build and release system to ensure that this couldn't happen again (ie, whatever changes are made to the underlying infrastructure, we should always – within reason – be able to reproduce and update old releases). This session is an opportunity to review the plans that were made and the progress towards implementing them.
As we move closer to operations, members of both Science Collaborations and the wider scientific community are taking an increasing interest in using our Science Pipelines and other software. We need to be able to provide them with technical support, without imposing an unreasonable burden on our on-project staff. In particular, in May of this year, specific concerns were noted about members of the community using Slack channels which were originally indented for technical discussion on the DM system to ask for technical support.
Providing a coherent approach to support is challenging, given the wide range of interests and skills in the community, limited on project resources, and the need to provide a system which both supports the construction project now and which fully transitions into the System Performance department's Community Engagement team in the future.
How much progress have we made since May? Do we now have a coherent message on what support we are providing, and through which channels? Have we clearly communicated that message to the leadership of the various science collaborations?