Current Status
NORMAL
Report a problem
Upcoming Scheduled Maintenance
(All times are Project Time (Pacific))
Start | End | Event | Location | Description | Systems/services that will NOT be available | Status |
---|---|---|---|---|---|---|
Every Tu. 09:00 | Every Tu. 11:00 | Recurring Weekly Nebula Maintenance | NCSA | Routine system updates. Computational services continue to run. | Horizon and API interfaces. | SCHEDULED |
Third Thursday of every month 07:00 | Third Thursday of every month 09:00 | Recurring Monthly lsst-dev maintenance | NCSA |
| Variable. Do not expect any lsst-dev system to be available during this period. | SCHEDULED |
Thursday 2017-11-16 07:00 | Thursday 2017-11-16 11:00 | Extended monthly lsst-dev maintenance | NCSA |
| Do not expect any lsst-dev system to be available during this period. | TENTATIVE |
2017-10-31 | NFS instability | NCSA | NFS becomes unresponsive under heavy load. | NFS is considered unstable. | UNSTABLE Ticket opened with hardware vendor. Tech support has responded with a known cause and the resolution. The fix involves changes to the baseboard management system and thus will be installed on 16 Nov during the (already) scheduled maintenance window. |
Previous Outages & Events
Start | End | Event | Location | Planned Activities | Systems/services that will NOT be available | Status |
---|---|---|---|---|---|---|
2017-10-31 | 2017-11-01 | GPFS instability | NCSA | All disks in the GPFS storage system went offline temporarily and came back online by themselves. NFS services were restarted. This is the second drop-out in <24hrs. GPFS has the hiccups. | most NCSA-hosted LSST resources native mounts (e.g., lsst-dev01, verify-worker*) and NFS mounts (e.g., PDAC) | ONLINE All GPFS services are currently running A power event caused systems and disks to go offline. Seperately, NFS services running on the GPFS servers caused the systems to lockup. NFS services were migrated to a separate server and GPFS has remained stable. |
2017-10-24 09:50 | LSST | GPFS outage | NCSA | All LSST nodes from NCSA 3003 (e.g., lsst-dev01/lsst-dev7) and NCPF (verify-worker, PDAC) that connect to GPFS (as GPFS or NFS) have lost their connection. | GPFS | ONLINE Storage is working to bring GPFS back online |
2017-10-21 17:15 | LSST | public/protected network switch is down in rack N76 at NPCF | nodes cannot communicate DNS, LDAP, etc. so largely cannot communicate with other nodes, e.g., no communication between affected verify-worker nodes and the Slurm scheduler on lsst-dev01, no communication between affected qserv-db nodes and the rest of qserv | Efffectively, the whole verification cluster | RESTORED in progress, replacement switch is on order Workaround in progress. If all goes well, systems should be back online by late afternoon. | |
2017-10-19 06:00 | 2017-10-19 14:00 | qserv-master replacement | NCSA | qserv-master will be down for this entire period | COMPLETE |
Important Project Dates
(those with asterisk* are LSSTC funded):
2017 |
|
April 24-28 | Data Science Fellowship Program – Session 3 * Tucson, AZ |
May 1 – 3 | NSF Large Facilities Workshop, Baton Rouge and Livingston, LA |
May 1 – 5 | AURA Board and Member Representatives Annual Meeting, Tucson, AZ |
May 12 - 13 | LSST Detection of Optical Counterparts of Gravitational Waves*, BNL. Contact Morgan May for additional information. |
May 22 – 25 | Infrastructure for Time Domain Science in the Era of LSST, Tucson, AZ |
May 31 - June 2 | Supernovae: The LSST Revolution Workshop *, Northwestern University, Evanston, IL |
June 12 – 16 | Getting Ready for Doing Science with LSST Data,* IN2P3, Lyon, France |
June 19 – 21 | AURA Workforce and Diversity Committee (WDC), Maui, HI |
July 10 - 14 | DESC Meeting, Dark Energy School, and Hack Day*, jointly hosted by Stony Brook University & BNL |
July 25 – 27 | NSF/DOE Joint Status Review of Data Management, NCSA, IL |
August 14 – 18 | LSST 2017 Project & Community Workshop, Tucson, AZ |
September 6 – 8 | NSF/DOE Joint Status Review, Tucson, AZ |
September 14 – 15 | AURA Management Council for LSST (AMCL) Meeting, Tucson, AZ |
October 26 – 28 | Society of Women Engineers WE17 Conference, Austin, TX, |
Get your LSST gear at our storefront: https://business.landsend.com/store/lsst/