“It’s blocking our production traffic! We have to shut it off!”

Dan Weber rolled his eyes. Why is it that developers always make me want to punch someone in the face? He unmuted his line and said to the conference call, “We can’t do that, we absolutely can’t. That’s the perimeter firewall. Turn that off and we might as well hand our data over to the Chinese and Russians and anyone else interested.”

“But we have to ship product! We can’t do that with the firewall in its current state. It’s blocking all our traffic.” Same developer as before.

Dan said, “It’s blocking all traffic from everywhere right now, so at least we’re safe. I’ve got a TAC case open with the vendor and we’ll have it resolved eventually.” Thank goodness this isn’t a video call. Dan made several obscene gestures at the initials of the developer that wanted to shut down the firewall.

A manager asked, “Do you have an ETA on when that firewall will be fixed?”

Dan’s head tilted up as he leaned back in his chair. “No. It’s a code problem from the upgrade. We’ve escalated it, but no ETA.”

Manager, again, “Can you roll back the code?”

Dan kept looking at the ceiling. “No. There’s no rollback from this upgrade.”

“Can you restore from backup?”

“No. because the last backup was on the previous version, so it’s not compatible with this version of the code. We just have to wait this one out.”

The manager put his foot down. “Unacceptable. Turn it off.”

Dan sat up, lightning going down his spine. “I have to have-“

Dan’s manager, Kelly Montlac, interrupted, “Hey, we need to discuss this offline with Raymond.” Raymond was the Network Services Director. A conversation with him would of course involve the director over the developers and probably also the CISO and CIO, if they could be reached at this time. It was late in the day in the USA and early in the morning over in Europe, where the C-levels lived.

The developer manager raised his voice. “We need to get back into production. Turn it off and then we can talk it over.”

Kelly dropped her voice into a growl. “Not gonna happen.” Silence, then Kelly drove the point home. “Not gonna happen.”

The Major Incident Coordinator didn’t speak right away after that, but eventually said, “OK, how about we end this call so we can get that meeting together? And then I’ll have this bridge back up in 60 minutes, after that meeting gives us direction on the perimeter firewall.”

All the managers agreed to that and Dan couldn’t leave the call fast enough. As he dashed down the hall for a badly-needed bio-break, he cursed the idiot developers that refused to bounce their own servers to see if it resolved the issue. Five nines, be damned! Wasn’t there a limit to what had to be sacrificed to get that precious uptime?

They’d already turned off or bypassed the IPS, the proxy, the NAC, the datacenter firewall, the load balancer, the WAN accelerator, the VA scanner, the data protection system, the antimalware solution, the, um… were there any other security solutions? If so, they probably also got turned off, because that’s how development rolled. If Dan hadn’t been on the TAC call with the vendor all day, he would have been on the earlier Major Incident call and the perimeter firewall would have been assailed from within at that point in time.

Dan reflected on which of those systems needed to be turned off as he washed his hands. He was pretty sure at least half those systems were configured improperly and the other half were running just good enough for production, but not optimized. Dan himself barely had a grip on the perimeter firewalls. So many vendors, so many rules that had piled up over the years, and only so much he could do with the firewall management platform before he violated change management procedures or stepped on someone’s shoes in Governance.

When Dan had asked for training, he had gotten it. It was neither the trainer’s fault nor management’s fault that Dan was, at best, a mediocre student. More often than not, he was just a warm body that could complete change requests. Not a clever man, our Dan.

In fact, if one made a school of the entire IT staff at where Dan worked, there would be no need for a Gifted and Talented class. There would be some call for a remedial reading course, but most of the imaginary student body would be average kids with average brains, wishing that the weekend would hurry up and get here. 

Dan had once applied to work at a vendor. He applied because his position at the time was being downsized and the vendor had an opening. What he did not know was that the interviewers said he couldn’t troubleshoot his way out of a paper sack with a pair of scissors that that the opening went to some guy with a home lab who only applied at that vendor because that’s where he wanted to work.

Dan got a different job, held that for a few years, and then moved on to this role when the previous one got downsized.

Even though Dan hated security and wanted to get back to routing and switching (developers never, never demanded that switches or routers be turned off!), he knew that his experience with firewalls – even if it was little better than babysitting them in between TAC calls – meant a good chance of getting a job whenever there was a downsizing… 

… or whenever his political sensibilities informed him it was time to move on before he was fired for incompetence. At most firms, that was around 2-3 years. He had two places on his resume where he managed to hang on for five years. Things were really bad at those places, both of which were lucky enough to be picked up in acquisitions after suffering major breaches.

Not that anyone knew about those breaches until after the mergers, when the purchasing company’s IT did an audit of the poorly-managed gear.

As Dan returned to his chair, he was thankful that he could work from home. He also cursed the fact that he wound up working from home during times when he could be watching sports at home, or sleeping at home. This outage looked like something that would rob him of sleep, but he was damned if he would miss the playoff game on tonight! Dan turned on the television and put it on the big game.

As the sports match got underway, Dan wondered how this thing would all pan out and if it meant it was time for him to start looking for another job somewhere. During commercials, he checked his recruiter spam to see which roles looked like they might be good lateral moves. He didn’t want to move up into management or architecture, as that meant only more meetings and increased chances of dealing with C-level heavies, who could be worse than developers in their demands.

Around the end of the first half, it was time to mute the television and get on the call. Dan dialed in and watched the game as everyone else joined the call. 

The CISO was on and said, “OK, for starters, we’re not turning off the perimeter firewall.” Dan smiled. Take that, developers! “But we need that resolved ASAP. Dan, reach out to the vendor and get an RMA started. We’ve got to have our firewalls up and running.”

Years of experience in IT had helped Dan to develop his most important skill of all: how to curse silently when he was unmuted on a call. He paused his staccato mouthing to say, “Sure, I’ll get on that.” Calling TAC wasn’t all that bad, except for the small talk the vendor engineer always engaged in as screens refreshed or boxes rebooted or whatever. And with an RMA call, there would be tons of stuff Dan would have to say that would distract him from the progress his team was making in the playoff game.

Heaven help everyone if the RMA didn’t resolve things and there was some mess of rules on the firewall that, in their combination, blocked that stupid traffic that only ran once a month. That would mean getting an order to review 30 days of changes to see which one put the rule in to block that traffic.

And if no such rule could be found? “Turn it off!” would be the developers’ battle cry!

Dan got off the conference call and opened up another TAC case online for the RMA. As he waited for the callback, he set “looking for opportunities” in his linkedin for salespeople profile and replied to a few of the more promising recruiter spams.

Dan had no idea, of course, that his eventual replacement was going to be as clueless and hapless as he was. Dan also didn’t know the name of the nuclear reactor that guy used to work for, or the name of the GRU agent that had found the holes in that facility’s perimeter security.

Hell, he didn’t even know the names of the GRU agents that had penetrated his current company’s network, for that matter. To be fair, not many security specialists know the names of people in the GRU that have penetrated networks, but in Dan’s case, it was definitely for lack of trying.

An email popped into Dan’s inbox. It was from Kelly. She wanted to know if Dan could log in to the IPS console.

Dan fired up the GUI and tried the vendor default username and password. Hey, they worked!

Dan let Kelly know that he could. Kelly then emailed back for Dan to check the logs to see if the IPS systems were in bypass mode, or if they had been fully shut down.

Dan checked the GUI and saw that every single IPS was down. There was also a licensing error on the server and a warning about missing critical updates. Dan only mentioned the IPS devices being down in his response. He didn’t want to make the IPS guy look like an incompetent.

Kelly then asked for when the IPS devices had been switched off.

Well, hell, that meant searching the logs, and… holy crap! Those things had been turned off two years ago, and kept off! No wonder the IPS guy always gave up quickly whenever someone asked him to shut off the IPS! No troubleshooting, no request to try something different, he just said, “OK, try it now.”

Dan wondered briefly about the times in the last two years that “turning off” the IPS had provided a solution to whatever problem was going on…

But then Dan wondered happily and joyfully about how this proved that there was someone more incompetent than he was on the network. Not that it made him quit his job search. No, it made him look all the harder. He didn’t want to be the guy tasked with taking on the IPS system and turning it back on after 2 years of it being shadow shelfware. 

On the TV, Dan’s team made a terrible mistake. Dan blamed the coach and, completely unaware of the irony, said, “We need a coach that knows what the hell he’s doing! Fire the big dope!”