Request from Miyazaki-san, HSC PI, to Steve Kahn to process the HSC data using the LSST science pipelines, and use the LSP to serve the public data products, both in Japan,
First meeting to understand exactly what they want set for Tuesday 30 July. From this we can work out what it means for us and what it will cost us, .eg
LSP may not be ready to serve a full public release of HSC data to a large community
We need to be careful to manage expectations while we are currently in development.
RHL points out that this is complicated technically and there is a parallel effort in Japan to use SciServer that Miyazaki-san seems to not be aware of.
As discussed on the SST channel, The DPDD implies we report times, e.g DIASource time says midPointTai, but in TAI but users will expect TCB.
- Eric is working on
for time series features.
- Fitting in DRP/AP needs to be in TAI but time series feature should be computed using TCB. There will be confusion if the Alert packet has time in TAI and the users can't reproduce the periodogram.
- Time series features that will be computed in AP are the same in the DIAObject tables in the PPDB as in the Alerts packets,
- Should we persist the TCB times along with the TAI times or always computed on-the-fly (as Robert Lupton suggests) ?
- Gregory Dubois-Felsmann points out we have very high-level requirements to use TAI. Supplementing TAI with TCB is not a problem.
- Michael Wood-Vasey recommends calculating once and persisting on-the-fly.
- What does the MPC want - SSSC should know?
- ZTF do not use TCB but also do not compute time series features.
- Gregory Dubois-Felsmann - will we also do this for the Visit times or just the timed photometry tables, DIASource, etc? We can put TCB time in the Visit table as an additional column, it is not that big - use case is less obvious
- TAI will be used to talk to the EFD, astronomers will mostly use TCB, we should to have both. We need to make sure the documentation is clear!
- Where in the pipelines should we do these calculations? Not in the AP pipeline. If we didn't need to get it into the alerts it could be done at ingest time, after the DRP pipelines have run.
Consensus to compute time series features in both AP and DRP on TCB and propose an additional change to the DPDD to add a column for TCB times. A proposal to add TCB times to the visit table will also be included.
Eric Bellm will discuss this in the DMTN. and put to RFC to discuss with the rest of DM
- Eric Bellmfollow up with Mario what the SSSC / MPC want times to be reported in.
- MPC expects times in Terrestrial Time, which is neither TAI or TCB but is a straightforward conversion using e.g. astropy
- Mario believed that computing SSObject timeseries features in barycentric time was not a concern.
DM-19308 is at a point where feedback would be useful. Draft DMTN and short summary slide deck are linked to the Jira ticket.
- The OSS does provide a requirement to evaluate completeness and purity, but it is not clear about how it will be evaluated (i.e it does not specify that it should be done as a function of all the parameters that Melissa presents). No change is therefore required at the OSS level. At the DMSR level we need to make this clear.
- Are we thinking we will inject fakes into the actual images or are we thinking about doing duplicate processing? Melissa has addressed the pros/cons in the document. Most common approach is do it in the data release processing.
- The detection efficiency matrix is a new data product, not currently in the DPDD but will enable a lot of science.
- Eric argues there is a science case for having these efficiencies on the AP, not in real time, otherwise brokers will not know the input selection functions.
- The detection efficiencies matrix will be calculated for DRP, but cannot necessarily be applied to AP as every image is different - the deep DRP diffim processing is different to that of AP. Are there enough fakes in each image to do this reasonably?
- Consider doing this for AP on the same cadence as for the Calibration products. Eric argues to do it during the daytime - all processed image products available and persisted. Rerun AP during the day with fakes injected and update the detection efficiency matrix. To be discussed. There will be a latency providing the information but it is not needed in real time.
- Moderate expansion in the last line not really correct, requirements drafting was not clear. We have to do this work anyway and it will be part of V&V. We should also release this to the public.
- Yusra AlSayyad claims there is no need to expand the WBS scope as DRP would be doing this work anyway
- This DMTN will cover the requirements - once clear, the AP team will write a design document will be written in response
Consensus that we should update the DMSR requirement to provide a meaningful flowdown and clarify how completeness will be evaluated.
