Incident Post-Mortem: Core Network Outage

Executive Summary
We recently experienced a critical service interruption affecting our core network routing. The outage was caused by the simultaneous failure of what were documented as fully redundant fiber paths. By provisioning an emergency out-of-band link via an alternate carrier, our engineering team was able to restore core services in under 60 minutes. We recognize the impact this had on our users and are implementing strict measures to ensure true physical redundancy moving forward.

Root Cause Analysis
Our network architecture is designed with high availability in mind, strictly requiring physically diverse, disjointed edge-to-node fiber routing.
However, following a catastrophic fiber cut, an immediate investigation revealed that our infrastructure provider (GasLINE) had implemented undocumented design modifications during deployment. Instead of the contracted geographically diverse paths, two primary optical paths were laid parallel within the same physical conduit for a critical segment.
When a physical disruption occurred on that specific route, both connections were severed concurrently. Because both primary uplinks dropped simultaneously, the network entered an isolated state that exceeded the parameters of our automated failover protocols, preventing an automatic recovery. This infrastructure failure was not isolated to our network; several other regional carriers utilizing the same GasLINE infrastructure experienced identical outages.

Incident Mitigation and Resolution
Upon identifying the dual-fault failure, our Network Operations Center (NOC) immediately escalated the issue to a manual recovery protocol.
To bypass the severed physical infrastructure, our engineers rapidly provisioned an emergency Layer 2 (L2) transport link through a secondary carrier. Because this alternative carrier operates on a completely independent physical topology to our backbone, we were able to successfully re-establish core routing.
Traffic was gradually migrated over the emergency link, bringing the majority of services back online safely and stabilizing the network in under an hour.

Corrective Action Items
To prevent a recurrence of this single-point-of-failure scenario, we are executing the following remediation steps:
- Supplier Infrastructure Audit: We are engaging in immediate, rigorous discussions with GasLINE to demand full transparency regarding their deployment practices. We require updated, accurate "As-Built" documentation and KMZ/KML mapping files for all active links.
- Physical Route Verification: We will implement enhanced vetting processes for all third-party fiber links, ensuring that contracted disjointed routing is physically verified and not just logically separated.
- Failover Logic Enhancement: Our engineering team is reviewing and updating our automated recovery protocols to better handle catastrophic, simultaneous dual-fault scenarios to reduce manual intervention time.

Conclusion
We sincerely apologize for the disruption this incident caused to your operations. Reliability is our top priority, and we consider the lack of transparency regarding our supplier's physical routing to be unacceptable. We are fully committed to holding our partners accountable and reinforcing our infrastructure to guarantee the resilience you expect from us.

✅ Resolved
The fibers have been fixed and the network is fully up again.
We will work closely with Gasline to resolve the faults and prevent future incidents.

Workaround has been build and network is reconverging.
GasLINE / OGE has communicated us up to 4 hours of time to splice and fix the damaged fibers.

Due to a fiber-optic cable cut at our supplier Gasline/OGE, combined with planning and communication issues on the supplier’s end, several paths are currently down. Our team is working around the clock to reroute the uplinks via alternative paths and compensate this really rare incident. We expect to have a workaround in place within the next 45 minutes.