Frossie: (re: Chuck's effort to identify missing scope) we don't have spare people at the moment
Wil: we have several potential pools of in kind, pre-ops, etc.
Frossie: just trying to manage expectations because prioritizing one thing means de-prioritizing another
Robert: some of these things will preclude us going on sky, so these hard prioritizations need to be done
Wil: a sticky issues is that the project was scoped for nominal operations, but commissioning is not nominal operations, so scope was definitely missed in tooling in that area
Leanne: we have requirements on the base. Does handing off the base mean we have to do a test campaign to verify? Requirements in DMSR and in the ICD documents.
Wil: We have already accepted the base, but we should go through any requirements and verify we have met them
Tim: Have we got a plan to off board people from github etc?
Wil: Update of DM team on regular basis?
K-T: Working on a one time script, but that's not a plan.
Wil: We don't always want to remove people when they go off project. Sometimes they are willing to keep contributing
Tim: We could do what Eric suggested and just do a yearly walk through
Frossie: Kick people off the org and convert them to outside collaborators if they are going to continue contributing. This should be left up to the T-CAM
Wil: Do we want to capture this on the off boarding list as a check box?
Frossie: Sure
Wil O'Mullane ask for a new entry in the off boarding form to make sure people are removed or moved to external collaborator status in github when they leave
09:30
Low hanging fruit milestones and how can we claim them
We have a bunch of milestones that can be closed with a bit of work that keeps getting put off or for lack of the formal testing being completed. Let's go through them and come up with a plan.
Robert: But what if these milestones don't actually reflect what work we actually need to do
Frossie: There may be a couple of those, but by and large that doesn't seem to be the case with the current set of milestones
Michelle: Can we try to organize the completion process a little? E.g. put them all in a spreadsheet and identify next steps and responsible teams so that we can get a better handle on how to make progress
Wil: I have a sheet like that, though there are probably new ones. I can resurrect that document
Gregory: Looking at overdue ones in my areas, they are all in active development.
Frossie: Those are tractable. LDM-503-EFDc is a case where there is no one team who can retire it on their own.
Gregory: Exactly. Maybe we should look for ones that haven't been started and that we may not even have architecture for
Fritz: My work is under represented by milestones relative to work left.
Frossie: I'm worried about user databases which does have a milestone and is a problem since it's one of these that takes many teams
Frossie: Maybe we should have each T-CAM classify their late milestones into: not relevant, mostly done, need to be moved to another location, etc.
Frossie Economou with T-CAMs will do some taxonomy to try to categorize late and soon to be due milestones
Wil O'Mullane will update the spreadsheet to remove done ones and add new ones
Yusra: Most milestones can be completed internally. Exception is AP-15 and DRP-24. We can't do anything more without precursor data
Fritz: We did a big walkthrough and filed LCRs and moved the needle. Maybe we just need another one of those at the next F2F
Wil: We could do that
Gregory: Is there low hanging fruit?
Yusra: The ones in orange in Wil's spreadsheet are first guesses at the lowest hanging fruit
Frossie Economou will run a milestone "parade" for a time box starting 09:00 project on Thursday
The output of the parade should be a fleshed out version of the spreadsheet.
Acceptance Test. Robert: Auxtel data campaing to be includede en acceptance test. Leanne : Agreed KT: Who run acceptance test?, who organize them? are we using thinkgs already done, or is it new work? Leanne: Organized by Jeff Carlin and Leanne, will need help of product owners. Should be able to execute unless product owner wants to do it. Frossie: Retiring pre-comcam data should be fine. Retire L2 when level 3 tests are done also good. Fritz : For databases there are scale requirements related to datasets, which are not the same as in operations. Leanne: Run tests on datasets available, could stay in verification status until lsstcam data is available. Robert: On regards to KT question, Robert can get Sitcom scientists that could like to be involved in acceptance tests. Frossie: Can't wait for DR1, so performance requirements must done "at scale". We could use an artificial load. Fritz: Some other things may not appear until we got data production at scale. Frossie: Could fulfill level3s while level2 activities are ongoing. Wil : 1a, 1b are camera focus. But we do need to prove at scale.
Ops Rehearsals Frossie: Different ops. rehearsal than the past. Commisioning style, to find if how we do things is wrong Robert : Similar to what we do Frossie: People involved now are experienced w/Auxtel. Comcam is something new. Leanne: More about actors and interation than components. Should be DM ops. rehearsals ? or Rubins? Wil: Would like to be Rubins. Next OR should be focus on commisioning. Robert: Could be good to do a "real" OR with more/new people involved KT: Auxtel is not using final components, so training to use this way could be a problem later. Robert: Auxtel is useful and is good to discover what's missing Wil : We shouldnt give things that are not ready. ie. API to Alysha Leanne: Like the idea of Rubin's Ops Rehearsal Wil: We shouldnt push everything, there are some DM only activities.
Network KT: Can we do it now Cristian: I rather wait to not do work twice. Leanne: We can wait. Cristian: if this is taking too much time, we can still do it. Robert: Does base facilities verification includes running pipelines in antu Wil : Not in scope, but we can do it. KT : Base is about facilities not the services Wil: Base facilities was handed over to operations Noirlab.
Middleware Frossie: Worried Jim as middielware product owner. Perhaps could be too much load for Jim. Jim: Already doing some of this. Gregory: Backing up Jim. Wil: Need to update org chart ? Leanne: Already started updating. Wil: About org charts, product owner of LHN should be moved to Richard.
RSP Acceptance Test Gregory: Running test campaignsa, they are good cause we always find something.
DM Science Validation Wil: Verification on DM side, validation in conjunction with the rest.
Sizing: Robert: Sizing means memory, cpu, etc. Gregory: Release field is a non trivial problem/ Wil: Commisioning could be continous release process, and one final release for operations KT: Concerns about the buying hardware for USDF given the leadtimes and timeline. Richard: Haardware ordered, perhaps arrives in January KT: Data release, if you need to patch still the same data release because replaces code. Wil: Not a problem for commissioning. Jim: During commissioning shouldn't be a problem. Wil: Number of IDs in a patch, can be splitted for ID purposes? make smaller patches... Wil: Science team can investigate about it. Wil: DMTN-135 has good information about hardware
If someone asks for "all the observations taking by Auxtel last night" there is no way to get that to them
We have ~4 different of ways of representing this data
We could take advantage of the butler registry, which has all of the information. Would have to expose it from the butler as a way to get to the tables and views that we need to build services that query the DB.
Can we expose the butler registry in this way?
Who is leading this effort? There is no general view into the metadata
JB: I don't actually hate this, the big thing that has changed is the Butler registry schema
RHL: I have exactly the opposite reaction. We have an enormous amount of things that must be captured here. Mixing that up with an operational database that has to run the system seems like a mistake
KTL: This has to be an operational database. We need something that includes both metadata that we know at the time of observation, as well as information that is calculated later.
FM: Frossie, how much of your concern is driven by data products that are already in the DPDD
KTL: The DPDD does not have any observational products
WOM: We had a long data
GPDF: “Consolidated Database” was also in contraposition to Qserv
JB: I don't want to put much more into Butler code. It should either be completely separate, or written as an extension
FE: What I want to do is seed the views that I want. Can we use the butler registry for that?
TJ: I am supposed to write a technote on observation annotations, i.e. to let an observer flag the nights observations as bad
GPDF: I imagined that we would take advantage of the Butler's existing tables, but not mess with the butler itself. It could be read-only, and we would do views instead of joins.
GPDF: What we're trying to do is allow this be done in the live butler, not in the replica after a delay. This would allow an observer to enter annotations immediately instead of first having to wait for the Butler to update
RHL: I worry that we are focusing on the technical implementation, rather than defining what we need first
FE: We all agree that we need an exposure log. Right now there is neither a technical path nor a management path unless we do something here
FE: It is a DM requirement to deliver an exposure table. Who is doing this?
KTL: That's the big problem. We know the data is coming, we just need a place to hold it.
WOM: We need to be careful to say that we would be making use of the butler, not that it would be a part of the butler.
JB: There are different levels of interface. If we want these things to be joined against butler tables, we need to be careful. It is not easy to do that, but it can be done.
TJ: I am only writing up the specific case of how you write up annotations, not how you deal with tracking
RHL: I think it is SITCOM's responsibility to lead this, and DM to build the backend according to their directions
KT: SITCOM can be product owners and work on front ends, but we need at least a prototype backend to work with them on. If no one wants to take this on, I can do it.
GPDF: The image metadata table issue is something we discussed as part of the image services, and is something I am already working on. I feel a lot of responsibility for the backend, though I can't do the front end.
RHL: I can provide a prototype of a backend from an obs log from HSC. I am happy to work with Architecture
JB: I'd like to weigh in on how this would interact with the Butler. Is there more than what is in the baseline schema?
WOM: No, that's the problem, we need the product owners to tell us what is missing
TJ: It is trivial to query the butler for all observations that were taken last night, but impossible to tell what the last observation was since there is no ordering
FE: Where does this live?
TJ: OODS
WOM: Arch will take care of this for construction, either K-T or Tim
FE: When does the data show up?
KT: during AP, so within 60s
FE: That is acceptible
GPDF: Which TCAM am I working with, and who is actually building something?
FE: Propose that we have the basic architecture for this presented at the next DMLT F2F.
KT: I think a prototype should be done by then, and I can make it. You will all hate it, but it will allow us to have a discussion
Kian-Tat Lim Write up a prototype consolidated database
KSK: where will the proof of concept be hosted?
KT: the IDF on Kubernetes, unless the USDF is up in time
One of the biggest obstacles we have to putting VO services into (Rubin data) production is the fact that we do not have an agreed format for Data IDs uniquely identifying our data products. This information exists in a dict, but we don't have a scheme for converting it to a string. Let's discuss the complications and come up with a plan.
Low hanging fruit milestones and how can we claim them
We have a bunch of milestones that can be closed with a bit of work that keeps getting put off or for lack of the formal testing being completed. Let's go through them and come up with a plan.
We need a plan on how to get observation metadata somewhere where VO services can get them - this means probably the Consolidated DB (though, crazy idea, Butler registry seems to know most of this stuff?) Right now nobody seems to own this and be working it so we have to come up with an actionable plan.
I am getting more feedback from developers getting frustrated by some aspects of Focus Friday. Note that these concerns are mostly addressed by the open support channels and by instructing them on how to use Slack's "Schedule for later" feature.
Kian-Tat Lim Convene a meeting with Colin, Tim, Robert, Yusra to resolve graph generation with per-dataset quantities (likely based on Consolidated DB work).