2019-10-14 DM SST Agenda and Meeting notes

Leanne Guy

Scarlet workshop in Naples  7-9 Oct

• Robert Lupton,  Leanne Guy and Peter Melchior were invited speakers.  Leanne gave an overview of data management and the science platform, Robert spoke about the science pipelines and crowded fields  For most of the rest of the week, Peter ran a tutorial on using Scarlet (independently of the LSST stack). 
• Their interests are in crowded stellar fields - Scarlet is one part of this problem, not the whole solution.  We could have run this on the LSP and using the Stack. 
• It was stressed that LSST is currently evaluating Scarlet as a deblender and that no decision has yet been made. 
• Italian community is very keen to maintain and  grow their existing MOU with LSST into operations beyond the current 15 PIs and are looking for ways that they can get involved. 
• For us - it was good to get an idea of the interests outside of the non-DESC community. 

Michael Wood-Vasey

DM-15448 - Getting issue details... STATUS

Latest version of document: https://dmtn-091.lsst.io/v/DM-15448/index.html

• What is the motivation for having a CI and SMALL dataset? There seems to be a lot of overlap. Do we need something on that intermediate scale or do we just need CI/med/large
• Vision is that small is still something that can be run on a single developer machine in a few hrs. MEDIUM is the next scale requiring a cluster somewhere.
• Having a dataset that can be run a few times a day while developing (SMALL)  is useful rather than having to wait overnight for results (MEDIUM). 
• Consensus that a bigger MEDIUM rather than removing SMALL would differentiate them better and be useful.
• Is CI_HSC in fact 'SMALL' in this definition? CI_HSC  is 8GB( ~ 30 CCDs from ~ 12 visits)  so yes in that sense but it currently takes ~8 hrs to run (not SMALL). Most of this is thought to be due to Jenkins processes. On a few cores on most machines it takes 45 mins to hour with most of the processing down in ~20 mins in this case. Needs to be profiled to understand, perhaps some inefficiencies in I/O. Reported to be faster under Gen3 (30 mins on 8 cores) but no detailed timings have been done. 

• The technote does not address computational performance monitoring, only  or just algorithmic scientific performance  monitoring. Even though we are not running testing in the context of an orchestration workflow on a known hardware configuration, it is nonetheless useful to know and track how long it takes to process these datasets. We should recommend doing that and add to the document especially or MEDIUM and LARGE datasets. It would also be useful to know if a SMALL dataset suddenly starts taking twice as long to run. 

• What is the tradeoff between individual developers knowing they should run a CI/SMALL dataset regularly to check they didn’t break something algorithmically, and a regular CI that goes through SQUASH? Is that sufficient to catch regressions? AP team looks weekly, is that sufficient?

• Agreement with the following for running on dataset on Jenkins: 

  • CI level is required for a merge 

  • SMALL is at developer discretion, with the understanding that they will fix any breaks. 

  • MEDIUM /LARGE for algorithmic or larger scientific changes changes, but not unless there is an expectation that there would be a change, e.g don’t run on HSC-RC2 unless an algorithmic change might be expected to produce a different output. 

• This means that the current CI_HSC is 'SMALL' both in runtime and in usage. 
• Could maybe make CI_HSC 20-50% smaller but need to maintain a sufficiently interesting dataset for testing (area, depth, # epochs/patch)
• MEDIUM dataset definition is satisfied by HSC-RC2. Lauren has put a lot of effort into defining this. Details in the TN. This will be the main dataset that will give a balance between ’scientifically interesting’ and ‘does not take too long to run’
• Key point about this this dataset that it is not representative – Lauren intentionally included more edge cases. This makes it more interesting for scientific development and performance monitoring but does mean that any predictions will be  conservative or non representative to some degree. We should bear this in mind when doing characterizstion or commissioning level studies. 
• Currently takes longer than a night to run. Not clear how much is  limited  by the current middleware's ability to balance over many more cores. Cannot be easily automated at this time as it requires some babysitting. The new middleware will address this and automation should be possible with Gen3 a workflow system. 
• Hsin-Fang currently runs these on a monthly basis and is handing over to NCSA. Can we go back to fortnightly. 
• LARGE: We have currently only run once or twice on PDR1. PDR2 still coming out of Japan. 

• Michael Wood-Vasey Move CI_HSC data to SMALL in technote
• Michael Wood-Vasey to add a statement to the document at GKE 29 Nov 2019 
• Michael Wood-Vasey Add comment on addressing computational performance 29 Nov 2019 

Leanne Guy

Reminder that we will have a special meeting on LOY1 alerts on Wednesday