DP0.2 currently running into an issue where matchedCatalogTask runs for >2 hours without emitting any log messages. The Panda monitoring system assumes that means it has crashed or is in an infinite loop or something, so it terminates the job.
Had to switch to running the non-faro parts of step3
We now have a reprieve from that limit (as of an hour ago), so we might be able to resume running it as normal?
CTS: Confirmed, it's working fine now.
Hypothesis is that the butler.get() at the start of runQuantum is taking all the time.
Deferred dataset loading might also be a good idea in general, and would avoid a long block of time spent in single butler.get()
Middleware implemented a fix to emit logs messages during long butler.get() calls.
Ran out of time last week - pick up today?
CTS: I think this is now superseded. Faro is running as-is in step3, and the investigations into Eli's matched star catalog might replace a lot of this.
Collection id for g',r',i' products generated from single-frame processing:
A quick example query to isolate the survey data using the program key:
Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT