All times listed are Project Time (Pacific)
Current Status
| Start | End | Event | Location | Description | Systems/services that will NOT be available | Status |
|---|---|---|---|---|---|---|
NORMAL |
Upcoming Scheduled Maintenance
| Start | End | Event | Location | Description | Systems/services that will NOT be available | Status |
|---|---|---|---|---|---|---|
Recurring Scheduled Maintenance
(All times are Project Time (Pacific))
| Start | End | Event | Location | Description | Systems/services that will NOT be available | Status |
|---|---|---|---|---|---|---|
Third Thursday of every month 06:00 | Third Thursday of every month 10:00 | Recurring-Monthly Monthly lsst-dev maintenance | NCSA |
| Variable. Do not expect any lsst-dev system to be available during this period. | SCHEDULED |
Every Mon. 04:00 | Recurring- Weekly | NCSA | Per LSST data management policies, files older than 180 days will be purged from the LSST shared (GPFS) Purge logs can be found in | No outage or service disruption. | SCHEDULED | |
Every Tu. 08:00 | Every Tu. 10:00 | Recurring- Weekly Weekly Nebula Maintenance | NCSA | Routine system updates. Computational services continue to run. | Horizon and API interfaces. | SCHEDULED |
Previous Outages & Events
| Start | End | Event | Location | Description | Systems/services that were NOT available | Status |
|---|---|---|---|---|---|---|
| 15-Oct-2018 05:35 | 15-Oct-2018 07:15 | Power event -> host outage at one datacenter | NCSA | A power blip caused all physical hosts at the NCSA building to power off or reboot.
| affected: all physical LSST hosts (and VMs) at the NCSA building:
unaffected: all physical LSST hosts (and VMs) at the NPCF building:
| RESOLVED
|
| 04-Oct-2018 06:00 | 04-Oct-2018 07:15 | Critical security patching | NCSA | An incorrect date (Oct 1) was initially posted for this maintenance. The correct date is Thu, Oct 4. | ALL lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters) The following systems will remain online and unaffected:
| RESOLVED
|
| 20-Sep-2018 06:00 | 22-Sep-2018 14:50 | Qserv Master outage | NCSA | qserv-master01 is having trouble booting after a motherboard replacement during planned maintenance. | Qserv in general, specifically qserv-master | RESOLVED |
| 20-Sep-2018 06:00 | 20-Sep-2018 12:40 | LSPdev Kubernetes | NCSA |
| LSPdev | RESOLVED |
| 20-Sep-2018 06:00 | 20-Sep-2018 12:00 | Monthly maintenance | NCSA |
| All systems will be unavailable during this period. | RESOLVED qserv-master01 and LSPdev are still having issues. These will be tracked as a separate incidents. |
| 09-Aug-2018 09:00 | 09-Aug-2018 09:37 | lsst-dev01 Outage | NCSA | The lsst-dev01 server was unreachable for >60sec from the GPFS cluster and got expelled from the GPFS cluster. Open file handles and/or bind mounts from GPFS prevented lsst-dev01 from reconnecting to GPFS until it was rebooted. We suspect that a big job on the Slurm cluster may have contributed to some network congestion that triggered this. | lsst-dev01 | RESOLVED |
| 03-Aug-2018 10:00 | 03-Aug-2018 13:30 | NCSA VPN was not working for some users. | NCSA | A configuration issue caused some VPN users connection problems to some NCSA resources. | NCSA VPN | RESOLVED |
| 29-Jul-2018 | 03-Aug-2018 05:45 | Bulk Transfer Server Rebuild | NCSA | The Globus endpoint on lsst-xfer stopped working on July 29 after a certificate from the outdated GridFTP service expired. lsst-xfer was rebuilt and upgraded with CentOS 7.5, Globus Connect Server (v4), bbcp (17.12), and iRODS client (4.2.3). Globus bookmarks to the lsst#lsst-xfer endpoint may need to be updated to point to the rebuilt endpoint. | Globus on lsst-xfer | RESOLVED |
| 27-Jul-2018 | 27-Jul-2018 | NCSA VPN Migration | NCSA | NCSA will be migrating to a new VPN with multi-factor authentication. The new VPN is currently available, and users are encouraged to start using the new VPN before the cutoff date in order to ensure continued connectivity. All users must be registered with NCSA's Duo before they can use the new VPN. Links to the how-to article as well as the new VPN and Duo login are included below. | No interruption of service is expected. | COMPLETE |
| 19-Jul-2018 10:00 | 19-Jul-2018 10:30 | DB services on lsst-dev-db are unavailable along with dependent services, including:
| NCSA | MariaDB service did not start on lsst-dev-db after maintenance. There is a newer setting in MariaDB that didn't like the current mount point. | DB services on lsst-dev-db Services that depend on lsst-dev-db, including:
| RESOLVED |
| 19-Jul-2018 06:00 | 19-Jul-2018 10:00 | Monthly lsst-dev maintenance | NCSA |
| ALL lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters) The following systems will remain online and unaffected:
| COMPLETE DB services on lsst-dev-db will not start after maintenance, impacting dependent services such as lspdev. This will be tracked in a separate status event. |
| 27-Jun-2018 07:00 | 27-Jun-2018 11:00 | lspdev outage | NCSA | The Kubernetes head node unexpectedly rebooted at approximately 7:00 AM, causing a JupyterHub outage. Service was brought back online around 11:00 AM. | lsst-kub0[01-20] | COMPLETE |
| 27-Jun-2018 06:10 | 27-Jun-2018 06:30 | Monitoring Update | NCSA | First phase of enabling encryption on monitoring traffic | Monitoring Dashboards | |
| 21-Jun-2018 06:00 | 21-Jun-2018 07:35 | Monthly lsst-dev maintenance | NCSA |
| CentOS 6.9 servers:
Slurm/verification cluster Other impact is not expected but unexpected issues could lead to connectivity issues for other hosts or downtime for lsst-dev01 or hosted VMs | COMPLETE |
| 18-Jun-2018 11:00 | 19-Jun-2018 17:00 | Nebula outage | NCSA | Nebula is undergoing a complete reboot. Last week's storms damaged more than just one node initially thought to be affected. | Nebula will be unavailable until 15:00 (5pm CDT) | RESOLVED |
| 19-Jun-2018 06:00 | 19-Jun-2018 10:00 | Level One Test Stand Maintenance | NCSA |
| Level One Test Stand, including:
| RESOLVED |
| 12-Jun-2018 ~01:40 PDT | 12-Jun-2018 07:01 PDT | Storm → outage of Kubernetes Commons & 75% of verification cluster compute nodes | NCSA | A storm caused a power event at the NPCF datacenter taking down Kubernetes commons and lspdev as well as 75% of the verification cluster compute nodes. |
| RESOLVED |
| 17-May-2018 11:30 | 17-May-2018 12:25 | Grafana monitoring was offline | NCSA | The influxdb data used by grafana monitoring was offline while it's storage was rebuilt | https://monitor-ncsa.lsst.org/ monitoring data was offline | RESOLVED |
| 17-May-2018 06:00 | 17-May-2018 11:30 | Monthly lsst-dev maintenance | NCSA |
| ALL lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters) The following systems will remain online and unaffected:
| RESOLVED |
| 30-Apr-2018 18:37 | 14-May-2018 15:00 | Security & AA infrastructure offline | La Serena | The Security & AA infrastructure went offline around 18:37 Project Time. None of the infrastructure is accessible via the network. A UPS had to be replaced and an electrical circuit upgraded for the replacement UPS. | None. | RESOLVED |
11-Apr-2018 06:00 | 07-May-2018 10:30 | production-size run (HSC-PDR1) on the verification cluster | NCSA | Per IHS-749, ~15 nodes of the batch compute resources will be reserved in order to complete HSC-PDR1 data runs. It is expected that the reservation can be scaled back to <10 after the first couple of weeks. | All systems available. | COMPLETE |
| 25-Apr-2018 11:30 | 25-Apr-2018 12:40 | Test new puppet changes for sssd and ldap access on SUI* nodes. | NCSA | A minor change to sssd service configuration needs to be rolled out to all nodes. The change will require a momentary outage of the sssd service and some actions will take longer (for a short period of time) as cache is repopulated. Changes in puppet structure (affecting ldap group sync) are also in need of testing and can happen simultaneously. | Affected services:
| COMPLETE |
| 04/24/2018 07:10 | 04/24/2018 07:50 | increased LDAP timeout to 60 seconds in sssd.conf | NCSA | increased LDAP timeout to 60 seconds in sssd.conf to fix problems with long login times and failure to start batch jobs we will coordinate in the near future to apply the same change on qserv* & sui* | Affected nodes: kub*, verify-worker* | RESOLVED All nodes are back in service, although affected nodes may have slow LDAP response times for a short while (due to local cache needing rebuilt). |
| 19 Apr 2018 | 19 Apr 2018 | Monthly lsst-dev maintenance | NCSA | CANCELLED. No major work is needed and key personnel are travelling. Deployment of the new DTN and VM infrastructure will be delayed until after the May maintenance period. | N/A | CANCELLED |
| 4/11 at 07:00 | 4/11 at 08:00 | Firewall update at NCSA | NCSA | Per LSST-1257, the primary firewall needs to have its routing software updated. No failover is required and traffic will continue to flow through the primary firewall during the upgrade. | No outage or service disruption. | RESOLVED |
| 4/3/2018 16:40 | 4/3/2018 16:45 | LDAP problems | NCSA | Causing new logins to the LSST resources at NCSA to hang. | new logins can't take place right now. | fixed. |
| 3/26/2018 08:00 | 4/2/2018 9:00 | A fileserver on Nebula became unstable, resulting in diminished performance for some instances and volumes. | NCSA | Any instances or volumes hosted on the healing filesystem will be impacted, or approximately 20% of instances and volumes. | We are migrating instances around to | |
| 3/15/2018 10:20 am PT | 3/15/2018 14:20 am PT | Lingering issues on select nodes following March PM | NCSA | Select nodes had issues coming out of the PM. |
| RESOLVED |
| 3/15/2018 10:20 am PT | 3/15/2018 11:23am PT | Lingering issues on select nodes following March PM | NCSA | Select nodes had issues coming out of the PM. |
| RESOLVED |
| 3/15/2018 6:00 am PT | 3/15/2018 10:20 am PT | March lsst-dev maintenance (regular schedule) | NCSA |
| Systems/services that were NOT be available: ALL lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC and the verification and Kubernetes clusters) | COMPLETE Select nodes (lsst-qserv-master01, lsst7, lsst-xfer, lsst-dts, lsst-l1-cl-dmcs) required additional attention following the PM, as noted in a separate status entry. |
3/12/2018 7:00 am PT | 3/12/2018 3:00pm PT | nebula(Open stack resource) is down | NCSA | Nebula is being taken down for patches to be applied across the whole infrastructure. | All containers on Nebula are going down. | COMPLETE |
07 Mar 2018 13:00 | 07 Mar 2018 14:10 | qserv-db12 maintenance | NCSA | qserv-db12 had one failed drive in the OS mirror replaced but the other is presenting errors as well so the RAID cannot rebuild. The node was taken down for replacement of the 2nd disk, to rebuild the RAID in the OS volume, and to reinstall the OS. | qserv-db12 | COMPLETE |
09:02 | 09:21 | lsst-dev01 Out of Space | NCSA | The main / drive partition ran out of space due to a user's faulty pip build. The faulty files were moved elsewhere for the user to review. | lsst-dev01 | COMPLETE |
27 Feb 2018 08:40 | 27 Feb 2018 09:40 | Puppet maintenance at NCSA | NCSA | Enable environment isolation on puppet master | No outage or service disruption is expected. | COMPLETE |
06:00 | 07:00 | Puppet updates | NCSA | Rolled out significant logic and organization of the Puppet resources in NCSA 3003 data center in order to standardize between LSST Puppet environments at NCSA. We had done extensive testing and did not expect any outages or disruption of services. | None Changes were applied to: | COMPLETE |
12:55 | 13:18 | lsst-dev-db crashed | NCSA | The developer MySQL server lost network and crashed. | lsst-dev-db MySQL database | RESTORED |
06:00 | 11:00 | February lsst-dev maintenance (regular schedule) | NCSA |
| Systems/services that will NOT be available: all lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC and the verification cluster) | COMPLETE
|
, 08:00 | , 08:30 | Slurm reconfiguration | NCSA | The slurm scheduler on the verification cluster will be repartitioned from one queue (debug) into two: debug: 3 nodes, MaxTime=30 min normal: 45 nodes, MaxTime=INFINITE | No outages | COMPLETE |
| Wed 1/24/2018 13:35 | Wed 1/24/2018 14:55 | Loss of LSST NFS services | NCSA | All NFS mounts for LSST systems were not working | NFS access on lsst-demo and lsst-SUI were not working | RESTORED |
16:40 | 21:00 | Firewall outage | NCSA | Both pfSense firewalls were accidentally powered off. | PDAC (Qserv & SUI) and verification clusters were inaccessible, as well as introducing GPFS issues across many services, e.g. lsst-dev01. | RESTORED |
06:00 | 08:00 | January lsst-dev maintenance (regular schedule) | NCSA |
| Systems/services that will NOT be available: all lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC and the verification cluster) | COMPLETE |
06:00 | 11:30 | Critical patches on lsst-dev systems (incl. kernel updates) | NCSA |
| Systems/services that will NOT be available: all lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC and the verification cluster) | COMPLETE |
09:00 | 17:00 | Nebula | NCSA | Nebula (OpenStack) will be shut down for hardware and software maintenance from January 2nd, 2018 at 9am until January 5th, 2018 at 5pm. | All Nebula systems unavailable. | COMPLETE |
Saturday | Tuesday | Support over holiday break | NCSA | 2017-12-22 to 2018-1-01 (inclusive) is the University holiday period. Services will be operational. Please report problems via the JIRA IHS queue. The queue will be monitored by NCSA staff and users will be notified via Jira as to if or when their issue can be addressed. | All services will be operational. | COMPLETE |
Wednesday 06:00 | Wednesday 08:00 | NFS Server switch | NCSA | NFS services will be moved to a different host | brief outage of NFS services to SUI nodes, lsst-demo, lsst-demo2 | COMPLETED |
Wednesday 06:00 | Wednesday 07:00 | Firewall drive replacement | NCSA | Current pfSense has a bad drive. If it fails, all nodes behind the firewall will be inaccessible. There are redundant firewalls, no service interrupts are expected. | None Expected | COMPLETED |
Thursday 2017-12-14 04:00 | Thursday 2017-12-14, 19:00 | December lsst-dev maintenance (off-schedule) | NCSA |
| Do not expect any lsst-dev system to be available during this period. | COMPLETED |
| Tuesday 2017-11-28, 10:00 | TBD | Rolling reboots of PDAC qserv nodes | NCSA |
| The occasional qserv node will need to be rebooted. Experience with the first couple will allow NCSA to give more precise information on the order and timing of the reboots. | COMPLETED |
| 2017-11-20 7:00 | 2017-11-20 14:00 | Nebula Openstack cluster | NCSA | Nebula OpenStack cluster will be unavailable for emergency hardware maintenance. A failing RAID controller from one of the storage nodes and a network switch will be replaced. | Not all instances will be impacted. If any running Nebula instances are affected by the outage they will be shut down, then restarted again after we finish maintenance that day. | COMPLETED |
Thursday 2017-11-16 06:00 | Thursday 2017-11-16 10:00 | Extended monthly lsst-dev maintenance | NCSA |
| Do not expect any lsst-dev system to be available during this period. | COMPLETED |
| 2017-10-31 | NFS instability | NCSA | NFS becomes intermittently unresponsive. | ~STABLE We are guardedly optimistic that this problem has been resolved. PDAC is now utilizing native GPFS mounts. | ||
| 2017-10-24 09:50 | LSST | GPFS outage | NCSA | All LSST nodes from NCSA 3003 (e.g., lsst-dev01/lsst-dev7) and NCPF (verify-worker, PDAC) that connect to GPFS (as GPFS or NFS) have lost their connection. | GPFS | ONLINE Storage is working to bring GPFS back online |
2017-10-21 17:15 | LSST | public/protected network switch is down in rack N76 at NPCF | nodes cannot communicate DNS, LDAP, etc. so largely cannot communicate with other nodes, e.g., no communication between affected verify-worker nodes and the Slurm scheduler on lsst-dev01, no communication between affected qserv-db nodes and the rest of qserv | Efffectively, the whole verification cluster | RESTORED in progress, replacement switch is on order Workaround in progress. If all goes well, systems should be back online by late afternoon. | |
2017-10-19 06:00 | 2017-10-19 14:00 | qserv-master replacement | NCSA | qserve-master will be down so that systems engineering can finish configuring the new server and xfering files. Status updates here: IHS-378 - Getting issue details... STATUS . | qserv-master will be down for this entire period | COMPLETE |