DM-Commissioning-Camera Bootcamp

Introduction

We will be having a bootcamp to bring together members from the camera, commissioning, and data management teams for the purpose of exploring instrument signature removal in the context of all three teams.

Slack channel: #dm-bootcamps

Goals

• To empower people to work with calibration related data and pipelines using the LSP as the primary interface.
• To explore how algorithms and tooling can be better integrated among the LSST team.
• To conduct comparisons of how calibration products are generated and applied via various input data and processing pipelines.

Following is a set of goals put forward by each of the participating groups that expands in more detail on the high level goals above.

Camera: 

• Take specific steps toward using DM tools to analyze Camera test data, with the eventual goal of complete use of DM tools for Camera reverification. Start with specific EOtest tasks and compare with the existing Camera analysis chain results. Build from there. (See tasks below)
• Getting Started documentation for others to follow.
• Make this a common standard platform for people interested in looking at Camera test data
• Transition from "Camera team", "DM team", "Commissioning team", ...  toward LSST team.

Data Management:

• To better understand the status of algorithms for the purpose of ISR in both the DM and Camera ecosystems.  This includes both calibration product production as well as application.
• To compare an example calibration exercise with both Camera and DM code
• To understand the best route toward unifying code between Camera and DM, and the extent to which that's possible.

Commissioning:

• To be able to run an example of generating a calibration product and applying that calibration product as part of ISR using the DM stack running on the LSP with relevant camera data hosted at NCSA. Start with simplest end-to-end test.
• Creating some baseline of documentation from which to spread knowledge of ISR implementation.

Logistics

When: 14, 15 Nov. 2018 with space available on the afternoon of 13 Nov. and morning of 16 Nov.

Where: Redwood Conference room SLAC National Laboratory

We will have from 10 to 20 rooms reserved at the SLAC Guest House.  This block is now used.  Please book hotel ASAP.

Other suggested hotels: Comfort Inn, Atherton Inn

For practical reasons, we are not supporting remote participation via bluejeans for the entire meeting. (During the meeting, we expect to reach out with specific questions via Slack and can make specific bluejeans calls if needed.) Our goal is that the products of the bootcamp will leave enough breadcrumbs and documentation that others can quickly get up to speed. We are planning a series of bootcamps that will each focus on a different topic, so there will be many future opportunities to participate.

Attendees

Please add your name if you are interested to participate in the bootcamp. We sized the bootcamp for 40 total attendees, though we do not necessarily need to be that large / might not have the budget for that many travelers. The Nov 2018 bootcamp is specifically focused on ISR. We expect to have a series of bootcamps that focus on different themes and are intended to be independent of each other (i.e., one does not need to participate in this bootcamp to participate in future bootcamps on other topics).

TBC: Andy Connolly,  Patrick Ingraham, Jeff Carlin, Johann Cohen-Tanugi, 

Others contacted who are unable to attend: Chris Walter, Chris Stubbs, Michael Wood-Vasey, Paul O'Connor.

Agenda

General plan for full days is start at 9am. Lunch at noon. Close at 5pm. Coffee/tea at 10:30am and 3pm.

Proposed Tasks

Pre-bootcamp checklist for attendees

• Join the #dm-bootcamps channel in slack. That is where updates and discussion for this bootcamp are being posted.
• Make sure you have access to the notebook aspect of the LSST Science Platform (LSP) at the LSST Data Facility (LDF), i.e., NCSA.
  
  • If you have an account on any of the resources at the LDF: e.g. lsst-dev.ncsa.illinois.edu, you already have an account on the LSP.
  • If you do not (or don't know if you) have an account, contact Simon Krughoff.
  • You will need to activate two-factor authentication and connect through the NCSA VPN.  Instructions can be found in https://nb.lsst.io.
• Once an account is established, attempt to connect to `https://lsst-lspdev.ncsa.illinois.edu/nb`.
  • You will need to start the VPN and then log in to the service via CILogon.  Both require two-factor authentication.  Both will use the same credentials.
  • Upon successful connection, attempt to run any of the notebooks in notebooks/notebook-demo.  Again see https://nb.lsst.io for instructions.
• Look through the proposed tasks listed above.  Identify one or two proposed tasks to concentrate on in the bootcamp, or create your own.
• Verify that you have access to https://github.com/lsst/bootcamp-work.  Process for doing this TBD.  I don't think we want a bunch of empty files, or a bunch of empty commits, but maybe the latter is better than the former.
• Familiarize yourself with the suggested background reading materials and references below. We may have specific recommendations as the bootcamp nears.

Background Reading / References

Slides:

• Early 2018 I&T sensor and raft hack day
• "Sensor Characterization, Analysis and Simulation" session at PCW 2018: https://project.lsst.org/meetings/lsst2018/content/sensor-characterization-analysis-and-simulation?destination=node/69
  • See in particular Merlin's status report on ISR in DM Stack: https://project.lsst.org/meetings/lsst2018/sites/lsst.org.meetings.lsst2018/files/AHM%20update%20upload.pdf
  • Updated version of the money-slide

Papers / documents / notes:

• Bernstein et al. 2017 "Instrumental response model and detrending for the Dark Energy Camera" https://arxiv.org/abs/1706.09928
• Brighter Fatter papers: Downing 2006, Antilogus 2014, Lage 2017, Coulton 2018
• Jim Bosch LSST Community post "Photometric calibration and astrometric Jacobians" https://community.lsst.org/t/photometric-calibration-and-astrometric-jacobians/3232
• LDM-151 "Data Management Science Pipelines Design" https://docushare.lsst.org/docushare/dsweb/Get/LDM-151
• LCA-283 Camera Verification Test plan
• A draft memo we are writing connecting LDM-151 and LCA-283. Comments and suggestions to S. Ritz welcome.
• Camera EO test plan (LCA-10103)
• Camera EO test suite

Notebooks:

• Brian Stalder example: https://github.com/lsst-com/notebooks/blob/master/eotestanalysis_example.ipynb
• Robert Lupton example: https://github.com/RobertLuptonTheGood/notebooks/blob/master/Demos/Looking%20at%20Teststand%20Data.ipynb
• Andrew Bradshaw Stack Club: https://github.com/LSSTScienceCollaborations/StackClub/blob/master/ImageProcessing/BrighterFatterCorrection.ipynb (rendered)
• Michael Wood-Vasey: https://github.com/LSSTDESC/DC2-production/blob/u/wmwv/bf_kernel/Notebooks/make_brighter_fatter_kernel.ipynb

Source code:

• A repository to hold the description for the LSST 3.2 GPix camera. https://github.com/lsst/obs_lsstCam
• Code to produce calibration products, required to perform ISR and other calibration tasks. https://github.com/lsst/cp_pipe
• The ip_isr package provides Instrument Signature Removal related tasks. ISR includes steps such as combining multiple amplifiers into one full CCD image, corrections for overscans, crosstalk, bias and dark frames, and the creation of variance and mask planes. https://github.com/lsst/ip_isr

Data:

• I&T page with raft testing status summary and link to list of good runs.
• Richard Dubois has a page he keeps up to date with summary plots for good I&T runs: http://slac.stanford.edu/~richard/LSST/bokehDashboard.html  
• We're starting with run 7086 (https://lsst-camera.slac.stanford.edu/DataPortal/run.jsp?run=7086&dataSourceMode=Prod ) for RTM-007 an e2V raft. Merlin is making an example notebook that accesses the relevant files in that run.  More to follow about that soon.

  • After successfully looking at RTM-007, we could try RTM-011. Here, use run 5943D (https://lsst-camera.slac.stanford.edu/DataPortal/SummaryReport.jsp?run=5943D&dataSourceMode=Dev ). That would be interesting because we have similar runs for 2 sec and 3 sec readout.

  • If you need header info, see LCA-13501 and LSSTCCS-1521.