Author Archives: deanwebb
Doughfaces
Weird Stuff in Texas
Had some fun with the family over Christmas break and we went out to see weird stuff in Dallas, Fort Worth, Aurora, Argyle, Austin, San Antonio, and Seguin. Fun times!
Really Bad Record Review: American Idiot
The German Army and Drugs in WW2
Insecurity Through Incompetence
“It’s blocking our production traffic! We have to shut it off!”
Dan Weber rolled his eyes. Why is it that developers always make me want to punch someone in the face? He unmuted his line and said to the conference call, “We can’t do that, we absolutely can’t. That’s the perimeter firewall. Turn that off and we might as well hand our data over to the Chinese and Russians and anyone else interested.”
“But we have to ship product! We can’t do that with the firewall in its current state. It’s blocking all our traffic.” Same developer as before.
Dan said, “It’s blocking all traffic from everywhere right now, so at least we’re safe. I’ve got a TAC case open with the vendor and we’ll have it resolved eventually.” Thank goodness this isn’t a video call. Dan made several obscene gestures at the initials of the developer that wanted to shut down the firewall.
A manager asked, “Do you have an ETA on when that firewall will be fixed?”
Dan’s head tilted up as he leaned back in his chair. “No. It’s a code problem from the upgrade. We’ve escalated it, but no ETA.”
Manager, again, “Can you roll back the code?”
Dan kept looking at the ceiling. “No. There’s no rollback from this upgrade.”
“Can you restore from backup?”
“No. because the last backup was on the previous version, so it’s not compatible with this version of the code. We just have to wait this one out.”
The manager put his foot down. “Unacceptable. Turn it off.”
Dan sat up, lightning going down his spine. “I have to have-“
Dan’s manager, Kelly Montlac, interrupted, “Hey, we need to discuss this offline with Raymond.” Raymond was the Network Services Director. A conversation with him would of course involve the director over the developers and probably also the CISO and CIO, if they could be reached at this time. It was late in the day in the USA and early in the morning over in Europe, where the C-levels lived.
The developer manager raised his voice. “We need to get back into production. Turn it off and then we can talk it over.”
Kelly dropped her voice into a growl. “Not gonna happen.” Silence, then Kelly drove the point home. “Not gonna happen.”
The Major Incident Coordinator didn’t speak right away after that, but eventually said, “OK, how about we end this call so we can get that meeting together? And then I’ll have this bridge back up in 60 minutes, after that meeting gives us direction on the perimeter firewall.”
All the managers agreed to that and Dan couldn’t leave the call fast enough. As he dashed down the hall for a badly-needed bio-break, he cursed the idiot developers that refused to bounce their own servers to see if it resolved the issue. Five nines, be damned! Wasn’t there a limit to what had to be sacrificed to get that precious uptime?
They’d already turned off or bypassed the IPS, the proxy, the NAC, the datacenter firewall, the load balancer, the WAN accelerator, the VA scanner, the data protection system, the antimalware solution, the, um… were there any other security solutions? If so, they probably also got turned off, because that’s how development rolled. If Dan hadn’t been on the TAC call with the vendor all day, he would have been on the earlier Major Incident call and the perimeter firewall would have been assailed from within at that point in time.
Dan reflected on which of those systems needed to be turned off as he washed his hands. He was pretty sure at least half those systems were configured improperly and the other half were running just good enough for production, but not optimized. Dan himself barely had a grip on the perimeter firewalls. So many vendors, so many rules that had piled up over the years, and only so much he could do with the firewall management platform before he violated change management procedures or stepped on someone’s shoes in Governance.
When Dan had asked for training, he had gotten it. It was neither the trainer’s fault nor management’s fault that Dan was, at best, a mediocre student. More often than not, he was just a warm body that could complete change requests. Not a clever man, our Dan.
In fact, if one made a school of the entire IT staff at where Dan worked, there would be no need for a Gifted and Talented class. There would be some call for a remedial reading course, but most of the imaginary student body would be average kids with average brains, wishing that the weekend would hurry up and get here.
Dan had once applied to work at a vendor. He applied because his position at the time was being downsized and the vendor had an opening. What he did not know was that the interviewers said he couldn’t troubleshoot his way out of a paper sack with a pair of scissors that that the opening went to some guy with a home lab who only applied at that vendor because that’s where he wanted to work.
Dan got a different job, held that for a few years, and then moved on to this role when the previous one got downsized.
Even though Dan hated security and wanted to get back to routing and switching (developers never, never demanded that switches or routers be turned off!), he knew that his experience with firewalls – even if it was little better than babysitting them in between TAC calls – meant a good chance of getting a job whenever there was a downsizing…
… or whenever his political sensibilities informed him it was time to move on before he was fired for incompetence. At most firms, that was around 2-3 years. He had two places on his resume where he managed to hang on for five years. Things were really bad at those places, both of which were lucky enough to be picked up in acquisitions after suffering major breaches.
Not that anyone knew about those breaches until after the mergers, when the purchasing company’s IT did an audit of the poorly-managed gear.
As Dan returned to his chair, he was thankful that he could work from home. He also cursed the fact that he wound up working from home during times when he could be watching sports at home, or sleeping at home. This outage looked like something that would rob him of sleep, but he was damned if he would miss the playoff game on tonight! Dan turned on the television and put it on the big game.
As the sports match got underway, Dan wondered how this thing would all pan out and if it meant it was time for him to start looking for another job somewhere. During commercials, he checked his recruiter spam to see which roles looked like they might be good lateral moves. He didn’t want to move up into management or architecture, as that meant only more meetings and increased chances of dealing with C-level heavies, who could be worse than developers in their demands.
Around the end of the first half, it was time to mute the television and get on the call. Dan dialed in and watched the game as everyone else joined the call.
The CISO was on and said, “OK, for starters, we’re not turning off the perimeter firewall.” Dan smiled. Take that, developers! “But we need that resolved ASAP. Dan, reach out to the vendor and get an RMA started. We’ve got to have our firewalls up and running.”
Years of experience in IT had helped Dan to develop his most important skill of all: how to curse silently when he was unmuted on a call. He paused his staccato mouthing to say, “Sure, I’ll get on that.” Calling TAC wasn’t all that bad, except for the small talk the vendor engineer always engaged in as screens refreshed or boxes rebooted or whatever. And with an RMA call, there would be tons of stuff Dan would have to say that would distract him from the progress his team was making in the playoff game.
Heaven help everyone if the RMA didn’t resolve things and there was some mess of rules on the firewall that, in their combination, blocked that stupid traffic that only ran once a month. That would mean getting an order to review 30 days of changes to see which one put the rule in to block that traffic.
And if no such rule could be found? “Turn it off!” would be the developers’ battle cry!
Dan got off the conference call and opened up another TAC case online for the RMA. As he waited for the callback, he set “looking for opportunities” in his linkedin for salespeople profile and replied to a few of the more promising recruiter spams.
Dan had no idea, of course, that his eventual replacement was going to be as clueless and hapless as he was. Dan also didn’t know the name of the nuclear reactor that guy used to work for, or the name of the GRU agent that had found the holes in that facility’s perimeter security.
Hell, he didn’t even know the names of the GRU agents that had penetrated his current company’s network, for that matter. To be fair, not many security specialists know the names of people in the GRU that have penetrated networks, but in Dan’s case, it was definitely for lack of trying.
An email popped into Dan’s inbox. It was from Kelly. She wanted to know if Dan could log in to the IPS console.
Dan fired up the GUI and tried the vendor default username and password. Hey, they worked!
Dan let Kelly know that he could. Kelly then emailed back for Dan to check the logs to see if the IPS systems were in bypass mode, or if they had been fully shut down.
Dan checked the GUI and saw that every single IPS was down. There was also a licensing error on the server and a warning about missing critical updates. Dan only mentioned the IPS devices being down in his response. He didn’t want to make the IPS guy look like an incompetent.
Kelly then asked for when the IPS devices had been switched off.
Well, hell, that meant searching the logs, and… holy crap! Those things had been turned off two years ago, and kept off! No wonder the IPS guy always gave up quickly whenever someone asked him to shut off the IPS! No troubleshooting, no request to try something different, he just said, “OK, try it now.”
Dan wondered briefly about the times in the last two years that “turning off” the IPS had provided a solution to whatever problem was going on…
But then Dan wondered happily and joyfully about how this proved that there was someone more incompetent than he was on the network. Not that it made him quit his job search. No, it made him look all the harder. He didn’t want to be the guy tasked with taking on the IPS system and turning it back on after 2 years of it being shadow shelfware.
On the TV, Dan’s team made a terrible mistake. Dan blamed the coach and, completely unaware of the irony, said, “We need a coach that knows what the hell he’s doing! Fire the big dope!”
Really Bad Review of Please Hammer Don’t Hurt ‘Em
Breakfast in America
https://www.youtube.com/watch?v=ZJFY7yQuP1I
Lol, I love doing these. 🙂
Really Bad Record Reviews
I’ve worked on a few lately, had lots of fun. Here’s a link to one of four I did in the last 3 days: https://www.youtube.com/watch?v=3M8BN7FNT0A
My inspiration came from Rolling Stone record reviews where it was clear that the reviewer either had an axe to grind, wanted to review a different record, or hadn’t bothered to listen to the album at all.
As it turns out, it is very easy to put out a bad review. When one disposes of the entire listening process, all that’s left is the cover and the track list, and those can get knocked out in a matter of minutes.
For me, the trick is in making a connection back to Pink Floyd that reveals my complete ignorance of the band I’m being all pretentious and elitist about. Sometimes I can, and that’s the delivery on the musical joke.
Heartbreaker
Dr. Borden exhaled and dabbed the sweat from her forehead before proceeding into the most critical part of the operation. She drew a deep, competitive breath and moved the precision mouse to aim the laser directly at the point of incision. With a click, the aorta would-
The screen went black, then a logon screen appeared.
“What the plokha budding spore?!?! What the spore just happened?”
Dr. Borden regained her composure and typed in her username and password – the patient was undergoing open heart surgery, there was no time to lose!
Agony of ages as the dots blinked in their circular path.
Username and/or password incorrect. Next login attempt in 00:05.
“SPORE SPORE BUDDING SPORE BUDDING EFFDISKING BUDDING K’CHORTU BUDDING SPORE!”
Dr. Borden didn’t want this to be the first patient she would lose on the table, but it was looking increasingly that way. He was somewhere in Alberta, wherever the meddrone landed, and she was in Atlanta, where the workstation ran in her Midtown apartment. She was doing everything to keep her mind down-to-earth and focused, but found that rage did all it could to take over.
Her mind raced – how long had it been since things went dark? Would the meddrone AI be able to abort the operation in time to save the patient’s life? Oh God, he is so effdisked if that AI doesn’t figure out there’s no doctor on the other end.
Because this was the third time Dr. Borden tried to log on to her workstation and the third time it kicked her back, this time with a caution she only had one shot left and that maybe she ought to call tech support before using that chance.
There was no way to call the meddrone, as those things were sealed off as far as comms went. There was only one way to talk to the meddrone directly, and for Dr. Borden, it was on the other side of a logon screen.
She called the number for her hospital’s tech support. Ringing. Well, at least it’s not down. Chortu, but that’s a lot of ringing. Well, let it ring, someone might die today if Dr. Borden shrugs her shoulders and becomes fatalistic in philosophy. She waits out the machine-induced stress.
And a machine answers. On an emergency line, it takes time to explain how the options may have changed recently and offers up a universe of choices, all a press of a digit away. Effdisk that, Dr. Borden presses zero. A human eventually speaks.
“Aetilus Medical Solutions help desk, this is Raj. May I get your employee username?”
“Eborden. E as in echo, b as in bravo, o as in oscar, r as in Romeo, d as in delta, e as in echo, n as in November.” Dr. Borden hated it whenever eborden sounded like edorgom. Spelling was usually faster than going over it twice.
“Dr. Elizabeth Borden, is this correct?”
“Yes. A man may be dying, please check if meddrone A as in alpha, 3447-”
“I’m sorry, Dr. Borden, I’m not able to contact meddrones. I’d have to escalate for that.”
“Please escalate, za’chortu.”
“There will be a, uh… oh, spore, a 30-minute wait.”
What the budding spore? 30 budding minutes? Might as well be 30 budding years! Even so… “Chortu, just get me in that queue.”
46 minutes later, a human spoke to Dr. Borden. “Hello, Dr. Borden? You there?”
“Yes. Contact meddrone A as in alpha, 3447-1369-0003.”
“A as in alpha, 3447-1369-0003. Got it. One moment… I’m sorry, I’m not getting a status, I’ll try again.”
“Do you know what’s going on?”
“Some kind of outage, that’s all I know.”
“Chortu… I desoxy-ed for this. All right, that meddrone number I gave you, it’s involved in a heart operation in Alberta. I need verification that it aborted the operation successfully and the patient status. Text me as soon as you got that info. I can’t log on to my workstation.”
“Yeah, none of the remote staff can log on. I’ve got the status query queued up for the drone and your number associated with it. Can I do anything else?”
“Nope. I’m needled. Cheers.” Dr. Borden touched her phone and the call ended.
Hopped up on the desoxy, Dr. Borden started to shake as she lost anything specific to focus on. Suddenly, she became aware of her heart rate and the blood being shoved pell-mell through her circulatory system. Don’t panic, Dr. Borden. You know how to ride out this part of the desoxy run.
The door opened and closed. Dr. Borden brought herself out of her trance state to see her boyfriend Teddy. “Hey babe.”
Teddy set stuff down on the table, even though he wasn’t supposed to. “Hey Lizzie. How’d the operation go?”
“Pffft.”
“Oh God.”
“No idea how the patient is, everything just cut out on me.”
Teddy pulled up a chair near Dr. Borden. The workstation screen was dark. A light blinked on Dr. Borden’s phone. Teddy didn’t know what to say. Someone, somewhere, connected to his girlfriend, could be dead.
Dr. Borden picked up her phone, but the light was just for a FriendFace notification. Apparently, one of her associates was a real slug and had gone fascist, from the content in his post. She unfriended him. “Budding fascist loser.” No word from Aetilus tech support.
“Budding what?”
Dr. Borden shook her head, “Nothing. Someone I went to high school with is now a fascist and dead to me. Hey, I did desoxy for this operation and I need something to focus on, or I’m gonna lose it.”
Teddy reached for the string of prayer beads Dr. Borden kept by her keyboard. She grabbed them and began to run them through her fingers like there was no tomorrow. Once you give a soxer something to do, they’ll do it. They just can’t give themselves something to do.
After a few minutes with the beads, Dr. Borden felt like she could talk and manipulate them at the same time. “What do you think caused the outage?” Teddy was a nerd. He knew answers to questions like that. He was a really cute nerd and fun to have around.
“Did it affect just you or a bigger group?”
“Guy said it took all the budding remote users out. No comms to meddrones.”
“Wow. That’s big.”
“You think it was terrorists?”
“Could be. More likely, it was someone stupid.”
Dr. Borden laughed. Teddy elaborated on the stupid. “So… it could be that someone turned off your time server. That would kill off your ability to log on remotely. Or maybe your computer cert expired. No, stupider, the root cert expired.”
Dr. Borden laughed even more. “I have no budding clue what that means! God, I love you!”
Yeah, she wasn’t doing any more operations today, system restoration or not. “Well, a root cert, that-”
“Shhhh! Explaining is boring! Just list off all the stupid stuff.”
Teddy knew better than to try and argue with a soxer. Last thing you want a soxer to focus on is a budding argument. “Um, OK, the VPN hub could be offline, uh… the directory service got swamped and went down… date field problem, oh spore! Do you know if your IT guys took care of your Y38 problem?”
Dr. Borden laughed harder, kinda maniacally now. It was time for the bell-1. She needed to come down off of this before she broke down.
*** *** *** *** *** *** *** *** ***
Dr. Borden opened her eyes and looked around. There was a little drool on her cheek, which was typical of a bell-1 cooldown. She sat up on the sofa and saw the blinking light on her phone. She reached over to the desk and picked up her phone. A swipe later, then a code, then a DNA pulsecheck, and she was in. The light was for a text.
The text was from tech support Raj. Spore, it was 7 hours ago! Must have texted Dr. Borden right after Teddy gave her the bell-1 dose.
Oh, chortu. The guy died. Dr. Borden sighed and scrolled. OK, so the meddrone did shut things down gracefully, so it was just his heart failing post-op, which was always a risk, regardless of how the operation went. Poor old dude and his now-dead carcass.
Dr. Borden texted back to Raj, what was cause of outage?
Company cert expired, sorry was Raj’s response.
Dr. Borden wondered why “cert expired” made her laugh a little.
Time to even things out with some zebra and ibuprofen. And some mango juice.
Teddy was in the kitchen. “Hey, I’m up.”
“Sleep ok?”
“Pffft. That spore’s not sleep.” She got the juice and then rummaged in the cabinet for the zebra and ibuprofen.
“How did the patient do, if I could ask?”
Dr. Borden downed the drugs and took a shot of mango juice. “Operation ended OK, but he died post-op. Not my fault, still sad. I’m taking zebra to even things out.”
“You also took something for the headaches, right?”
“I’m not an idiot, Teddy.”
“Hey, just checking. They say what caused the outage?”
“Cert expired, whatever that means.” Dr. Borden laughed again and felt weird about laughing. Was she going psychotic?
“It means nobody was checking one of the most important pieces of computer security, the thing probably being used to establish your VPNs and channels back to the drones and stuff. And the time on it ran out right in the middle of your operation.”
Dr. Borden was level enough to want to understand that. “Hold on. You mean to tell me that a company that knows precisely how long I’m functional on a dose of desoxy and how long it takes to do an operation and how long it takes to run drones over seven continents can’t keep time on the one thing that’s gonna tie them all together? Holy budding spore.”
“Well, that’s how you guys make money. Nobody makes a dime watching a calendar for a cert to expire. They know when licenses are due because someone else makes money with those. But certs?”
“Pffft.”
“Yeah, Pffft. That’s when they call me up. You remember when Charleston had that power outage last month?”
“That was an expired cert?”
“Yep. So was the Athens Supermax Riot. Cert expired, all the doors opened.”
Dr. Borden shuddered at that thought. That was too close to home. She still worked remotely, but those meddrones were trauma center models, only 60 miles away. And that was just three months ago. Images of the carnage still popped up in her mind if she wasn’t vigilant about her thoughts.
Now she had a question.
“Teddy?”
“Yeah?”
“Tell me… What is a cert and how does it expire?”
Again, Dr. Borden laughed for a reason she did not know.