DP0.2 currently running into an issue where matchedCatalogTask runs for >2 hours without emitting any log messages. The Panda monitoring system assumes that means it has crashed or is in an infinite loop or something, so it terminates the job.
Had to switch to running the non-faro parts of step3
We now have a reprieve from that limit (as of an hour ago), so we might be able to resume running it as normal?
CTS: Confirmed, it's working fine now.
Hypothesis is that the butler.get() at the start of runQuantum is taking all the time.
Deferred dataset loading might also be a good idea in general, and would avoid a long block of time spent in single butler.get()
Middleware implemented a fix to emit logs messages during long butler.get() calls.
Ran out of time last week - pick up today?
CTS: I think this is now superseded. Faro is running as-is in step3, and the investigations into Eli's matched star catalog might replace a lot of this.
Collection id for g',r',i' products generated from single-frame processing:
A quick example query to isolate the survey data using the program key: