Blew a core switch config at 2am in a Denver data center...

I was remoted into our main distribution switch last Tuesday night trying to add a new VLAN for a client rollout. Typed 'no vlan 100' instead of 'vlan 100' on the wrong interface and instantly killed half the building's traffic. Had to drive 40 minutes to the colo, console in with a serial cable, and reload a backup config from three days prior. The client's ERP system was down for nearly two hours before I got it back. Has anyone else had a fat-finger moment that took down production?

2 comments

2 Comments

angelab621mo ago

Read somewhere that bad VLAN commands cause like 60% of late night outages.

evan_anderson1mo ago

60% seems high, I bet most of those are probably just user error not the commands themselves.