Incidents reported on status page for NETMOUNTAINS https://status.netmountains.de/ https://d1lppblt9t2x15.cloudfront.net/logos/01542e70cca6ef5edd7d6ca430e25075.png https://status.netmountains.de/ en https://status.netmountains.de/incident/923122 Sat, 13 Jun 2026 20:12:15 -0000 https://status.netmountains.de/incident/923122#8541801faa2d12b2d818b92989067b8ea6bf219a825d7f2ebc88b0b43f0b69f8 An urgent Storage Cluster Security Upgrade can lead to short interruption of availability of the mentioned services. https://status.netmountains.de/incident/907658 Fri, 29 May 2026 00:35:00 -0000 https://status.netmountains.de/incident/907658#6c3844ec6139f61fd23ad2da1bb557b10c23e940ea61872bf0c3dbaa7718a5a4 __Incident Post-Mortem: Core Network Outage__ __Executive Summary__ We recently experienced a critical service interruption affecting our core network routing. The outage was caused by the simultaneous failure of what were documented as fully redundant fiber paths. By provisioning an emergency out-of-band link via an alternate carrier, our engineering team was able to restore core services in under 60 minutes. We recognize the impact this had on our users and are implementing strict measures to ensure true physical redundancy moving forward. __Root Cause Analysis__ Our network architecture is designed with high availability in mind, strictly requiring physically diverse, disjointed edge-to-node fiber routing. However, following a catastrophic fiber cut, an immediate investigation revealed that our infrastructure provider (GasLINE) had implemented undocumented design modifications during deployment. Instead of the contracted geographically diverse paths, two primary optical paths were laid parallel within the same physical conduit for a critical segment. When a physical disruption occurred on that specific route, both connections were severed concurrently. Because both primary uplinks dropped simultaneously, the network entered an isolated state that exceeded the parameters of our automated failover protocols, preventing an automatic recovery. This infrastructure failure was not isolated to our network; several other regional carriers utilizing the same GasLINE infrastructure experienced identical outages. __Incident Mitigation and Resolution__ Upon identifying the dual-fault failure, our Network Operations Center (NOC) immediately escalated the issue to a manual recovery protocol. To bypass the severed physical infrastructure, our engineers rapidly provisioned an emergency Layer 2 (L2) transport link through a secondary carrier. Because this alternative carrier operates on a completely independent physical topology to our backbone, we were able to successfully re-establish core routing. Traffic was gradually migrated over the emergency link, bringing the majority of services back online safely and stabilizing the network in under an hour. __Corrective Action Items__ To prevent a recurrence of this single-point-of-failure scenario, we are executing the following remediation steps: - Supplier Infrastructure Audit: We are engaging in immediate, rigorous discussions with GasLINE to demand full transparency regarding their deployment practices. We require updated, accurate "As-Built" documentation and KMZ/KML mapping files for all active links. - Physical Route Verification: We will implement enhanced vetting processes for all third-party fiber links, ensuring that contracted disjointed routing is physically verified and not just logically separated. - Failover Logic Enhancement: Our engineering team is reviewing and updating our automated recovery protocols to better handle catastrophic, simultaneous dual-fault scenarios to reduce manual intervention time. __Conclusion__ We sincerely apologize for the disruption this incident caused to your operations. Reliability is our top priority, and we consider the lack of transparency regarding our supplier's physical routing to be unacceptable. We are fully committed to holding our partners accountable and reinforcing our infrastructure to guarantee the resilience you expect from us. https://status.netmountains.de/ Fri, 29 May 2026 00:16:27 +0000 https://status.netmountains.de/#7f9d3cfea61ca8d04c8c7fd9b959166eec66986be7476272b93633526c3460d4 Datacenter VEL1 (Velbert, Germany) 🇩🇪 recovered https://status.netmountains.de/ Fri, 29 May 2026 00:16:27 +0000 https://status.netmountains.de/#ea5d121f5d44fc2f5cd2eceacc5d8467e7204db67eb7a3e4986629d65a696075 Backupsite nbi1.vel1 recovered https://status.netmountains.de/ Fri, 29 May 2026 00:16:25 +0000 https://status.netmountains.de/#76e9e6f988da8dfcbebae4340317fb62a755e29a7117cec9a0874ca7007cc13c s3-de2.netmountains.space recovered https://status.netmountains.de/incident/907658 Fri, 29 May 2026 00:15:00 -0000 https://status.netmountains.de/incident/907658#1a1ce981cf49365aedc1a0607212c346d3cf0dc32668a35713aa37fe254b48f8 ✅ __Resolved__ The fibers have been fixed and the network is fully up again. We will work closely with Gasline to resolve the faults and prevent future incidents. https://status.netmountains.de/incident/907658 Thu, 28 May 2026 22:20:00 -0000 https://status.netmountains.de/incident/907658#d0a19d37fc88298a298b3ca926c6dbf53dbf9d23f9facaed9860e476ac48dac8 Workaround has been build and network is reconverging. GasLINE / OGE has communicated us up to 4 hours of time to splice and fix the damaged fibers. https://status.netmountains.de/incident/907658 Thu, 28 May 2026 21:25:00 -0000 https://status.netmountains.de/incident/907658#6f303c6b67b2c1a6bcefb9b4eb1d2e2e7a14fe50ec1de13f6db8d86ea6f7df40 Due to a fiber-optic cable cut at our supplier Gasline/OGE, combined with planning and communication issues on the supplier’s end, several paths are currently down. Our team is working around the clock to reroute the uplinks via alternative paths and compensate this really rare incident. We expect to have a workaround in place within the next 45 minutes. https://status.netmountains.de/ Thu, 28 May 2026 21:04:50 +0000 https://status.netmountains.de/#7f9d3cfea61ca8d04c8c7fd9b959166eec66986be7476272b93633526c3460d4 Datacenter VEL1 (Velbert, Germany) 🇩🇪 went down https://status.netmountains.de/ Thu, 28 May 2026 21:04:47 +0000 https://status.netmountains.de/#ea5d121f5d44fc2f5cd2eceacc5d8467e7204db67eb7a3e4986629d65a696075 Backupsite nbi1.vel1 went down https://status.netmountains.de/ Thu, 28 May 2026 21:04:47 +0000 https://status.netmountains.de/#76e9e6f988da8dfcbebae4340317fb62a755e29a7117cec9a0874ca7007cc13c s3-de2.netmountains.space went down https://status.netmountains.de/incident/898514 Mon, 18 May 2026 08:00:55 -0000 https://status.netmountains.de/incident/898514#59fb53820e90dca0102f1f822c4c06bd7a3bb291a5ef646cdca4bf18dccc04f1 Im Rahmen von Wartungsarbeiten ist die Backup-Site VEL1 kurzzeitig nur eingeschränkt verfügbar. Die Backups werden von den Sites NRW4 und FRA1 übernommen, hierdurch kommt es zu keinen Auswirkungen auf den Produktivbetrieb. https://status.netmountains.de/ Sun, 17 May 2026 10:34:27 +0000 https://status.netmountains.de/#84760b79aefdca1ebcb9320983a9dce857a771bf6855600964ed13f2e10608b2 Datacenter VEL1 (Velbert, Germany) 🇩🇪 recovered https://status.netmountains.de/incident/898017 Sun, 17 May 2026 10:29:00 -0000 https://status.netmountains.de/incident/898017#2ca425d82da62ff85cc6fa1b90498ee60fd2c81056c6624ba312509f2cfcecb1 The defective device has been replaced and all systems are running normally. We encourage every customer to use and configure redundant uplinks to prevent downtime following a single device failure. https://status.netmountains.de/incident/898017 Sun, 17 May 2026 10:09:00 -0000 https://status.netmountains.de/incident/898017#60080c4fdc719c349914c3aba8c6aae468cdc7100c8fc169250f902615226880 Due to issues with a Leaf-Switch in Room 1 @ VEL1 the uplink for non-redundant customers may be degraded. The affected customers are using IPs from the subnets - 185.153.x.x/24 - 109.122.x.x/24 https://status.netmountains.de/ Sun, 17 May 2026 10:07:48 +0000 https://status.netmountains.de/#84760b79aefdca1ebcb9320983a9dce857a771bf6855600964ed13f2e10608b2 Datacenter VEL1 (Velbert, Germany) 🇩🇪 went down https://status.netmountains.de/incident/726171 Wed, 17 Sep 2025 21:04:00 -0000 https://status.netmountains.de/incident/726171#3804124e73e41897d16b6b33ca13747f49a9e1b40f230a82800c560cc66f8eee Our systems detected and successfully blocked a large DDoS Attack against one of our customers with up to 31 Gbit/s Traffic targeting a single IP and a single Port. During this attack a prolongated latency maybe occured at other customers. https://status.netmountains.de/incident/315086 Fri, 19 Jan 2024 19:00:00 -0000 https://status.netmountains.de/incident/315086#cb773c7eed723994bbd66938ec0ef7d0cce2966a579f66b3019ae36cb6ebbd28 Maintenance completed https://status.netmountains.de/incident/315086 Fri, 19 Jan 2024 17:00:00 -0000 https://status.netmountains.de/incident/315086#d3b778565bdbe72440846c78c66088c4538033b45212492b34b8931dce82e00c Servers without geo-redundancy or site-redundancy may be rebooted after a controlled shutdown due to critical security maintenance on cloud hosts in the cluster NCI-DE1 (NRW4). We'll keep you informed if all systems are operational again.