DM-Commissioning-Camera Bootcamp

Introduction

We will be having a bootcamp to bring together members from the camera, commissioning, and data management teams for the purpose of exploring instrument signature removal in the context of all three teams.

Slack channel: #dm-bootcamps

Goals

• To empower people to work with calibration related data and pipelines using the LSP as the primary interface.
• To explore how algorithms and tooling can be better integrated among the LSST team.
• To conduct comparisons of how calibration products are generated and applied via various input data and processing pipelines.

Following is a set of goals put forward by each of the participating groups that expands in more detail on the high level goals above.

Camera: 

• Take specific steps toward using DM tools to analyze Camera test data, with the eventual goal of complete use of DM tools for Camera reverification. Start with specific EOtest tasks and compare with the existing Camera analysis chain results. Build from there. (See tasks below)
• Getting Started documentation for others to follow.
• Make this a common standard platform for people interested in looking at Camera test data
• Transition from "Camera team", "DM team", "Commissioning team", ...  toward LSST team.

Data Management:

• To better understand the status of algorithms for the purpose of ISR in both the DM and Camera ecosystems.  This includes both calibration product production as well as application.
• To compare an example calibration exercise with both Camera and DM code
• To understand the best route toward unifying code between Camera and DM, and the extent to which that's possible.

Commissioning:

• To be able to run an example of generating a calibration product and applying that calibration product as part of ISR using the DM stack running on the LSP with relevant camera data hosted at NCSA. Start with simplest end-to-end test.
• Creating some baseline of documentation from which to spread knowledge of ISR implementation.

Logistics

When: 14, 15 Nov. 2018 with space available on the afternoon of 13 Nov. and morning of 16 Nov.

Where: Redwood Conference room SLAC National Laboratory

We will have from 10 to 20 rooms reserved at the SLAC Guest House.  This block is now used.  Please book hotel ASAP.

Other suggested hotels: Comfort Inn, Atherton Inn

For practical reasons, we are not supporting remote participation via bluejeans for the entire meeting. (During the meeting, we expect to reach out with specific questions via Slack and can make specific bluejeans calls if needed.) Our goal is that the products of the bootcamp will leave enough breadcrumbs and documentation that others can quickly get up to speed. We are planning a series of bootcamps that will each focus on a different topic, so there will be many future opportunities to participate.

Attendees

Please add your name if you are interested to participate in the bootcamp. We sized the bootcamp for 40 total attendees, though we do not necessarily need to be that large / might not have the budget for that many travelers. The Nov 2018 bootcamp is specifically focused on ISR. We expect to have a series of bootcamps that focus on different themes and are intended to be independent of each other (i.e., one does not need to participate in this bootcamp to participate in future bootcamps on other topics).

TBC: Andy Connolly,  Patrick Ingraham, Jeff Carlin, 

Others contacted who are unable to attend: Chris Walter, Chris Stubbs, Michael Wood-Vasey, Paul O'Connor, Johann Cohen-Tanugi, Robert G.

Agenda

General plan for full days is start at 9am. Lunch at noon. Close at 5pm. Coffee/tea at 10:30am and 3pm.

Proposed Tasks

Pre-bootcamp checklist for attendees

(1) SLAC is requesting that all non-SLAC employees who do not have SLAC badges fill out and return the attached DOE FACTS Questionnaire. Please fill it out, save it as a pdf and email to: regina@slac.stanford.edu

We need to receive your DOE FACTS Questionnaire ***by November 9, 2018***. Please contact Regina Matter if you have any questions.

(2) Join the #dm-bootcamps channel in slack. That is where updates and discussion for this bootcamp are being posted.

(3) Make sure you have access to the notebook aspect of the LSST Science Platform (LSP) at the LSST Data Facility (LDF), i.e., NCSA.
* If you have an account on any of the resources at the LDF, e.g., lsst-dev.ncsa.illinois.edu, you already have an account on the LSP.
* If you do not (or don't know if you) have an account, contact Simon Krughoff (SKrughoff@lsst.org)

(4) Once an account is established, attempt to connect to https://lsst-lspdev.ncsa.illinois.edu/nb.
* You will need to activate two-factor authentication and connect through the NCSA VPN.  Instructions can be found at https://nb.lsst.io.
* You will need to start the VPN and then log in to the service via CILogon. Both require two-factor authentication. Both will use the same credentials.

(5) Upon successful connection to the LSP
* See documentation for the notebook aspect of the LSP: https://nb.lsst.io
* Checkout the repo for the bootcamps: https://github.com/lsst/bootcamp-work
* Attempt to run the example notebook: https://github.com/lsst/bootcamp-work/blob/master/examples/welcome_to_FE55.ipynb

(6) Background reading and task list. We expect that most bootcamp participants will be starting the hack days with gain and linearity measurements, and then move on to other EO tests as time allows. The proposed task list and links to many documentation sources are available below.

Please visit the confluence page to list your name under the different tasks if you have a preference. Based on your expressed interest, we’ll start forming hack groups this week so that we can get straight to work next Wednesday.

If you run into any trouble or have questions, feel free to use the #dm-bootcamps for discussion or contact any of the organizers directly.

Background Reading / References

Slides:

• Early 2018 I&T sensor and raft hack day
• "Sensor Characterization, Analysis and Simulation" session at PCW 2018: https://project.lsst.org/meetings/lsst2018/content/sensor-characterization-analysis-and-simulation?destination=node/69
  • See in particular Merlin's status report on ISR in DM Stack: https://project.lsst.org/meetings/lsst2018/sites/lsst.org.meetings.lsst2018/files/AHM%20update%20upload.pdf
  • Updated version of the money-slide

Papers / documents / notes:

• Bernstein et al. 2017 "Instrumental response model and detrending for the Dark Energy Camera" https://arxiv.org/abs/1706.09928
• Brighter Fatter papers: Downing 2006, Antilogus 2014, Lage 2017, Coulton 2018
• Estrada et al. 2010 "Focal Plane Detectors for Dark Energy Camera" http://adsabs.harvard.edu/abs/2010SPIE.7735E..1RE
• Jim Bosch LSST Community post "Photometric calibration and astrometric Jacobians" https://community.lsst.org/t/photometric-calibration-and-astrometric-jacobians/3232
• LDM-151 "Data Management Science Pipelines Design" https://docushare.lsst.org/docushare/dsweb/Get/LDM-151
• LCA-283 Camera Verification Test plan
• A draft memo we are writing connecting LDM-151 and LCA-283. Comments and suggestions to S. Ritz welcome.
• Camera EO test plan (LCA-10103)
• Camera EO test suite

Notebooks:

• Example Fe55 notebook from Simon: https://github.com/lsst/bootcamp-work/blob/master/examples/welcome_to_FE55.ipynb Starting point for this bootcamp
• Brian Stalder example: https://github.com/lsst-com/notebooks/blob/master/eotestanalysis_example.ipynb
• Robert Lupton example: https://github.com/RobertLuptonTheGood/notebooks/blob/master/Demos/Looking%20at%20Teststand%20Data.ipynb
• Andrew Bradshaw Stack Club: https://github.com/LSSTScienceCollaborations/StackClub/blob/master/ImageProcessing/BrighterFatterCorrection.ipynb (rendered)
• Michael Wood-Vasey: https://github.com/LSSTDESC/DC2-production/blob/u/wmwv/bf_kernel/Notebooks/make_brighter_fatter_kernel.ipynb

Source code:

• A repository to hold the description for the LSST 3.2 GPix camera. https://github.com/lsst/obs_lsstCam
• Code to produce calibration products, required to perform ISR and other calibration tasks. https://github.com/lsst/cp_pipe
• The ip_isr package provides Instrument Signature Removal related tasks. ISR includes steps such as combining multiple amplifiers into one full CCD image, corrections for overscans, crosstalk, bias and dark frames, and the creation of variance and mask planes. https://github.com/lsst/ip_isr

Data:

• I&T page with raft testing status summary and link to list of good runs.
• Richard Dubois has a page he keeps up to date with summary plots for good I&T runs: http://slac.stanford.edu/~richard/LSST/bokehDashboard.html  
• We're starting with run 7086 (https://lsst-camera.slac.stanford.edu/DataPortal/run.jsp?run=7086&dataSourceMode=Prod ) for RTM-007 an e2V raft. Merlin is making an example notebook that accesses the relevant files in that run.  More to follow about that soon.

  • After successfully looking at RTM-007, we could try RTM-011. Here, use run 5943D (https://lsst-camera.slac.stanford.edu/DataPortal/SummaryReport.jsp?run=5943D&dataSourceMode=Dev ). That would be interesting because we have similar runs for 2 sec and 3 sec readout.

  • If you need header info, see LCA-13501 and LSSTCCS-1521.