Page History

...

     source setup_RC2_env.sh 
     git clone https://github.com/lsst/pipelines_check
     cd pipelines_check
     git checkout w.2021.25  # checkout the branch matching the software branch you are using
     setup -k -r .
     scons
    

         8.  Submit the pipelines_check.yaml file with:

                 bps --log-level=INFO submit pipelines_check.yaml

The step-by-step instructions are:

1. Login to: lsst-condorprod-sub01
2. Clone GitHub repository:
    git clone https://github.com/lsst-dm/bps-gen3-rc2
   Use /scratch/username or /project/username for that. There will be a lot of log files produced during a reprocessing run and your home dir is not the best place to have them.
3. Setup the environment for a proper week:
   Do:
    source setup_env.sh
   
   Note: It's a good idea to setup your environment inside of a Linux screen
4. Create Jira ticket for the run.
   
   This part is written based on w_2021_22 run. Some details may change for the next (w_2021_26) run. It is always good to ask about changes in pipelines on #dm-hsc-reprocessing slack channel.
   
   
5. Run "step1" subset. To see what it includes you can view $OBS_SUBARU_DIR/pipelines/DRP.yaml 
   Edit the bps-rc2_hsc_step1.yaml. Change these lines there: 
   
   operator, 
   pipelineYaml (edit path to HSC-RC2-step1.yaml), 
   campaign
   output (edit week and Jira ticket number only, keep the rest as it is.)
   
   Submit the yaml file for processing with:
   bps --log-level=INFO submit bps-rc2_hsc_step1.yaml
   
   Note: step1 subset includes fgcm tasks. They may fail. One way to avoid that is to copy the DRP. yaml file and remove them from the subset. After that, you need to edit the HSC-RC2-step1.yaml (location: path/to/edited/DRP.yaml).
   
   Note: You can check the status of your run with:bps report --id <report-id>
6. Run FGCM (if it failed, or you decided to skip it in step1).
   Start with editing the bps-rc2_hsc_fgcm.yaml. The edits are the same as for step1. Once they are done, you are ready to submit with
   bps --log-level=INFO submit bps-rc2_hsc_fgcm.yaml
   
   Note: Your submission may fail with error that looks like this:
   

   ValueError: Output CHAINED collection 'HSC/runs/RC2/w_2021_22/DM-30365' exists, but it ends with a different sequence of input collections than those given: 'skymaps' != 'HSC/runs/RC2/w_2021_22/DM-30365/20210527T164119Z' in inputs=('HSC/raw/RC2/9615', 'HSC/raw/RC2/9697', 'HSC/raw/RC2/9813', 'HSC/calib/gen2/20180117', 'HSC/calib/DM-28636', 'HSC/calib/gen2/20180117/unbounded', 'HSC/calib/DM-28636/unbounded', 'HSC/masks/s18a', 'HSC/fgcmcal/lut/RC2/DM-28636', 'refcats/DM-28636', 'skymaps') vs HSC/runs/RC2/w_2021_22/DM-30365=('HSC/runs/RC2/w_2021_22/DM-30365/20210527T164119Z',).
   
   To fix it, start ipython and try (remember to update the week and Jira ticket):
   

   from lsst.daf.butler import Butler
   butler = Butler("/repo/main", writeable=True)
   parent = "HSC/runs/RC2/w_2021_18/DM-29973"
   children = list(butler.registry.getCollectionChain(parent)
   children.append("HSC/RC2/defaults")
   butler.registry.setCollectionChain(parent, children, flatten=True)
   
   

   Then try to submit FGCM again.

7. Proceed with step2, and when it finishes submit step3 (update bps-rc2_hsc_step*.yaml files, submit)
8. Run faro (update, submit bps-rc2_hsc_faro.yaml). Faro is ran tract by tract (we may try to run all tracts from one submission next time)
9. Ran Run diffimDRP if needed.