Date
Jun 24, 2014
Attendees
Unknown User (robyn), Jim Bosch, Simon Krughoff, Kian-Tat Lim, Unknown User (shaw)
on vacation: Jacek Becla; undeclared: Mario Juric, Unknown User (abecker)
Goals
- Status reports
Discussion Items
Simon
- Simon emailed an RFC on how to deal with cameraGeom and obs_* between now and when we have a real calibration database in place.
- See lsst-data email thread on June 23, 2014 11:03: "[LSST-data] RFC: Break camera data out of obs_* ."
- Simon thought Jim brought up some good points in his emailed response but Simon hopes Jim is willing to go with the method described in the short term.
- Why a short-term solution:
- Jim: If this is a short term thing, I am OK with it. Why is it such an urgent issue? Why is it such a burden to use more of the stack than some people would like to use?
- Simon: I don't consider it much of an urgent issue either. People have been pushing back because to generate an input from phoSim you all of a sudden now need meas_deblender and people find that confusing and I appreciate that. It seems weird that just to generate a catalog you need 61 packages.
- Jim: Just from a psychological standpoint, is this an argument in favor of just bundling things up into 3 or 4 packages? Is it just an illusion that it's difficult to install meas_deblender?
- A plausible reason:
- K-T: No, it's not just an illusion. It's actually many more packages. In order to share the stack as widely as possible and to use as widely as possible, we should have subsets which are the bare minimum of what you need to be able to do some things. I don't think it's good to have a completely unitarian 'you need everything in order to do anything'.
- Jim: I agree with that to the extent that it involves extra 3rd party packages that you have to install yourself. But if all the code to do meas_deblender was within meas_algorithms, would anyone notice? It would just be easier to install it. People would complain less about having to install an extra package.
- K-T: There is still more code and more places where things can go wrong and more dependencies on other people changing things so it changes faster because there is more of it.
- Jim: I think that is true. ..<snip>... This is probably not the forum for this discussion.
- K-T: OK. The other question is going with a calibration repository style - is the difficulty in doing that is that because we don't have dataset retrievals for the camera geometry things; or is the difficulty that we don't have good versioning; or we don't have a good place to keep it?
- Simon: Currently we don't have good versioning and we don't have a good way of packaging up calibration things. In my optimistic view whatever the right solution is solves both those problems along with the provenance issues of how these calibration products were manufactured in the first place. I see it as a pretty big deal to move to more to the complete Calibration repositories solution. Not a big deal in the sense of being complicated but it doesn't exist and it will be a bit of work to do it.
- K-T: Right now the butler gives you one dimension of versioning in the sense that you can find the right calibration stuff to use for any given image. But it doesn't give you different versions of that calibration stuff which were generated at different times in different situations. So those have to be packaged up as separate repositories...<snip> I agree it needs some more thought.
- The real reason for doing this short-term solution:
- Simon: Part of the problem is that the simulation frameworks are being used more and more. There is even a portion of the simulation framework that's going to be used in a workshop going on at the same time as the 'All-Hands' so that there is some urgency from the Simulation side to make this as easy to digest as possible.
- Jim: I think that's fine. I don't think in practice there is any difficulty. I don't know what's going to make it any harder than afw; once it's got past afw everything above is trivial in comparison to getting afw built.
- Simon: I concur. And afw is the dependency which will stick around because it is cameraGeom which is triggering all of these extra dependencies. I'm in complete agreement. I had a discussion with AndyC about this. You can look at the discussion I had with AndyC on hipchat. But I think Mario is on Andy's side that when you get to package 60 to 61 and it's been an hour and it's on meas_deblend, you go 'what is going on'. That side of things is pushing back pretty hard against the concept of getting everything installed.
- Dick: I have a different question for Simon. The hope is to make this change and put it up there and document everything in time for the 'All Hands'?
- Simon: My hope is that there would be something on master by then.
- K-T: Whether we issue a point Release or not is a separate question.
- Simon: It would be nice to have a versioned Release.
- Dick: I think there are other reasons to make a point release just to clean up, for instance, installation. It would be nice to have by that meeting but those are fairly small compared to what you are talking about.
- Simon: I don't think this will touch much code.
- K-T: If you go with the obs_tasks way then its just the configs that import them?
- Simon: If we go with obs_tasks there will be no code other than the code in the new obs_tasks packages that are touched. <snip implementation detail>
- Simon has been assigned lots of reviews - notably from Dick, Russell and Paul.
Jim
- Jim has been busy doing HSC work and helping Perry.
- Perry has been working on DM-545 "ensure pipe_tasks, obs*, and other packages are compatible with meas_base" – getting all the other tasks to inter-operate so we can switch to that. Hopefully all upgrades will be installed by the All Hands meeting.
- Jim's HSC work is fixing up the aperture corrections. It's a prototype for the LSST side. The aperture corrections right now are completely broken and so this new work should address the long term concern for that capability.
K-T
- Middleware continues to make progress. Their status meeting notes are at: Middleware Meeting Notes 2014-06-24.
- K-T has been working on SAT tasks and is hoping to get a week in to work on the Butler.
Dick
- Dick is working on a chapter of the Software User Guide on key LSST data concepts, primarily on a section on measurement in the stack. He's just about ready to have Jim review it.
- Dick's just started working on another section on cameraGeom but it's early days yet on that.
- He noticed K-T's request for comments on the Level 1 Calibrated Exposure Processing writeup. Dick wondered in what format the comments should be delivered.
- K-T: That was just an initial outline. He going thru and correcting some stuff by looking at the original source documents. K-T's willing to take any form of input: comments on the page, emails, whatever way provided.
- Dick: Is this document meant to reflect what we intend to do or what we do currently?
- K-T: It is ideally meant to be what we intend to do and any variances from that should be noted.
Robyn
- Robyn prepared everything for the 3rd party updates to go into the LSST cluster's stack and get merged to master. However, it hasn't been tested on a Mac.
- K-T mentioned there is a Mac dedicated to our use at NCSA. He recommended Robyn talk to SteveP for details.
- Robyn's addendum post meeting: Steve said the system has been turned over to the LSST Admins at NCSA and they would setup up my account. That happened later that day. Thanks for the prompt access, Bill.
- Dick: Regarding his review of Robyn's update of the demo2012 benchmark files: was there anything additional he needed to do?
- Robyn: I rechecked that the v8.0 benchmarks files were generated from the v8.0 stack. I compared the benchmark files from v7.2 to v8.0 and they were not identical but were comparable. I attribute the small level of change to the fact that there was little algorithmic development between those two releases..it was mostly reorganization. Robyn still needs to handle the other review items to standardize some filenames and update the README.