Current Status
NORMAL
Report a problem
Upcoming Scheduled Maintenance
(All times are Project Time (Pacific))
| Start | End | Event | Location | Description | Systems/services that will NOT be available | Status |
|---|---|---|---|---|---|---|
Third Thursday of every month 06:00 | Third Thursday of every month 08:00 | Recurring-Monthly Monthly lsst-dev maintenance | NCSA |
| Variable. Do not expect any lsst-dev system to be available during this period. | SCHEDULED |
Every Mon. 04:00 | Recurring- Weekly Purge of GPFS /scratch partition | NCSA | Per LSST data management policies, files older than 180 days will be purged from the LSST shared (GPFS) /scratch file system. Purge logs can be found in /gpfs/fs0/admin/purge_logs/scratch/ | No outage or service disruption. | SCHEDULED | |
Every Tu. 08:00 | Every Tu. 10:00 | Recurring- Weekly Weekly Nebula Maintenance | NCSA | Routine system updates. Computational services continue to run. | Horizon and API interfaces. | SCHEDULED |
Thursday 2017-12-14 04:00 | Thursday 2017-12-14 10:00 | December lsst-dev maintenance (off-schedule) | NCSA |
| Do not expect any lsst-dev system to be available during this period. | IN PROGRESS |
Saturday | Tuesday | Support over holiday break | NCSA | 2017-12-22 to 2018-1-01 (inclusive) is the University holiday period. Services will be operational. NCSA will respond to incidents based on business criticality. | All services will be operational. | SCHEDULED |
09:00 | 17:00 | Nebula | NCSA | Nebula (OpenStack) will be shut down for hardware and software maintenance from January 2nd, 2018 at 9am until January 5th, 2018 at 5pm. | All Nebula systems unavailable. | SCHEDULED |
Previous Outages & Events
| Start | End | Event | Location | Planned Activities | Systems/services that will NOT be available | Status |
|---|---|---|---|---|---|---|
| Tuesday 2017-11-28, 10:00 | TBD | Rolling reboots of PDAC qserv nodes | NCSA |
| The occasional qserv node will need to be rebooted. Experience with the first couple will allow NCSA to give more precise information on the order and timing of the reboots. | COMPLETED |
| 2017-11-20 7:00 | 2017-11-20 14:00 | Nebula Openstack cluster | NCSA | Nebula OpenStack cluster will be unavailable for emergency hardware maintenance. A failing RAID controller from one of the storage nodes and a network switch will be replaced. | Not all instances will be impacted. If any running Nebula instances are affected by the outage they will be shut down, then restarted again after we finish maintenance that day. | COMPLETED |
Thursday 2017-11-16 06:00 | Thursday 2017-11-16 10:00 | Extended monthly lsst-dev maintenance | NCSA |
| Do not expect any lsst-dev system to be available during this period. | COMPLETED |
| 2017-10-31 | NFS instability | NCSA | NFS becomes intermittently unresponsive. | ~STABLE We are guardedly optimistic that this problem has been resolved. PDAC is now utilizing native GPFS mounts. | ||
| 2017-10-24 09:50 | LSST | GPFS outage | NCSA | All LSST nodes from NCSA 3003 (e.g., lsst-dev01/lsst-dev7) and NCPF (verify-worker, PDAC) that connect to GPFS (as GPFS or NFS) have lost their connection. | GPFS | ONLINE Storage is working to bring GPFS back online |
2017-10-21 17:15 | LSST | public/protected network switch is down in rack N76 at NPCF | nodes cannot communicate DNS, LDAP, etc. so largely cannot communicate with other nodes, e.g., no communication between affected verify-worker nodes and the Slurm scheduler on lsst-dev01, no communication between affected qserv-db nodes and the rest of qserv | Efffectively, the whole verification cluster | RESTORED in progress, replacement switch is on order Workaround in progress. If all goes well, systems should be back online by late afternoon. | |
2017-10-19 06:00 | 2017-10-19 14:00 | qserv-master replacement | NCSA | qserve-master will be down so that systems engineering can finish configuring the new server and xfering files. Status updates here: IHS-378 - Getting issue details... STATUS . | qserv-master will be down for this entire period | COMPLETE |
DM Meetings and Events
| Name | Dates | Location | Notes/links |
|---|---|---|---|
| JupyterCon 2018 | 2018/08/21-24 | New York City | https://www.oreilly.com/conferences/ Call for speaker: 2018/01 - 2018/02 |
| LSST2018 Project & Community Workshop | 2018/08/13–17 | Tucson, AZ | |
| 2018/06/11–15 | Lyon, France | ||
| SPIE 2018 | 2018/06/10-15 | Austin, TX | SPIE Conference in Austin, Meeting: 10 June – 15 June 2018 |
| IVOA InterOp Northern Spring | 2018/05/28- 06/01 | CADC, Victoria, BC | |
| DMLT face to face | 2018/05/22-24 | SLAC or UW | |
| Python in Astronomy 2018 | 2018/04/30- 05/04 | New York, NY | Deadline for applications is December 9th. |
| DM Joint Meeting with Systems Engineering | 2018/03/06-08 | IPAC, Pasadena, CA | |
| DESC Meeting | 2018/02/05–09 | SLAC | https://confluence.slac.stanford.edu/display/LSSTDESC/February+2018+Collaboration+Meeting+-+SLAC |
| Jupyter Widgets Workshop | 2018/01/23-26 | Saclay, France | Developer-centered workshop at CMAP Laboratory at Ecole Polytechnique. Some details at end of this Github thread, or contact Sylvain Corlay sylvain.corlay@gmail.com |
| DM Gen. 3 Middleware Meeting | 2018/01/22-25 | Princeton, NJ | Internal DM meeting to further developer SuperTask/Butler designs and do some collaborative development. Agenda and list of attendees are still in progress. |
DM Boot Camp 2 | TBD |
| |
| 231st AAS Meeting | 2018/01/08–12 | Washington, DC | |
| Towards Science in Chile with LSST in Chile | 2017/12/13-15 | Santiago, Chile | https://www.lsst-chile.cl/2017-workshop |