Author Archives: deanwebb

Weird Stuff in Texas

Had some fun with the family over Christmas break and we went out to see weird stuff in Dallas, Fort Worth, Aurora, Argyle, Austin, San Antonio, and Seguin. Fun times!

https://www.youtube.com/watch?v=WYVRj0L4C-4&list=PLJupeNTfg7NiuXFLTg2bDJX1LebkLWTKZ

Insecurity Through Incompetence

“It’s blocking our production traffic! We have to shut it off!”

Dan Weber rolled his eyes. Why is it that developers always make me want to punch someone in the face? He unmuted his line and said to the conference call, “We can’t do that, we absolutely can’t. That’s the perimeter firewall. Turn that off and we might as well hand our data over to the Chinese and Russians and anyone else interested.”

“But we have to ship product! We can’t do that with the firewall in its current state. It’s blocking all our traffic.” Same developer as before.

Dan said, “It’s blocking all traffic from everywhere right now, so at least we’re safe. I’ve got a TAC case open with the vendor and we’ll have it resolved eventually.” Thank goodness this isn’t a video call. Dan made several obscene gestures at the initials of the developer that wanted to shut down the firewall.

A manager asked, “Do you have an ETA on when that firewall will be fixed?”

Dan’s head tilted up as he leaned back in his chair. “No. It’s a code problem from the upgrade. We’ve escalated it, but no ETA.”

Manager, again, “Can you roll back the code?”

Dan kept looking at the ceiling. “No. There’s no rollback from this upgrade.”

“Can you restore from backup?”

“No. because the last backup was on the previous version, so it’s not compatible with this version of the code. We just have to wait this one out.”

The manager put his foot down. “Unacceptable. Turn it off.”

Dan sat up, lightning going down his spine. “I have to have-“

Dan’s manager, Kelly Montlac, interrupted, “Hey, we need to discuss this offline with Raymond.” Raymond was the Network Services Director. A conversation with him would of course involve the director over the developers and probably also the CISO and CIO, if they could be reached at this time. It was late in the day in the USA and early in the morning over in Europe, where the C-levels lived.

The developer manager raised his voice. “We need to get back into production. Turn it off and then we can talk it over.”

Kelly dropped her voice into a growl. “Not gonna happen.” Silence, then Kelly drove the point home. “Not gonna happen.”

The Major Incident Coordinator didn’t speak right away after that, but eventually said, “OK, how about we end this call so we can get that meeting together? And then I’ll have this bridge back up in 60 minutes, after that meeting gives us direction on the perimeter firewall.”

All the managers agreed to that and Dan couldn’t leave the call fast enough. As he dashed down the hall for a badly-needed bio-break, he cursed the idiot developers that refused to bounce their own servers to see if it resolved the issue. Five nines, be damned! Wasn’t there a limit to what had to be sacrificed to get that precious uptime?

They’d already turned off or bypassed the IPS, the proxy, the NAC, the datacenter firewall, the load balancer, the WAN accelerator, the VA scanner, the data protection system, the antimalware solution, the, um… were there any other security solutions? If so, they probably also got turned off, because that’s how development rolled. If Dan hadn’t been on the TAC call with the vendor all day, he would have been on the earlier Major Incident call and the perimeter firewall would have been assailed from within at that point in time.

Dan reflected on which of those systems needed to be turned off as he washed his hands. He was pretty sure at least half those systems were configured improperly and the other half were running just good enough for production, but not optimized. Dan himself barely had a grip on the perimeter firewalls. So many vendors, so many rules that had piled up over the years, and only so much he could do with the firewall management platform before he violated change management procedures or stepped on someone’s shoes in Governance.

When Dan had asked for training, he had gotten it. It was neither the trainer’s fault nor management’s fault that Dan was, at best, a mediocre student. More often than not, he was just a warm body that could complete change requests. Not a clever man, our Dan.

In fact, if one made a school of the entire IT staff at where Dan worked, there would be no need for a Gifted and Talented class. There would be some call for a remedial reading course, but most of the imaginary student body would be average kids with average brains, wishing that the weekend would hurry up and get here. 

Dan had once applied to work at a vendor. He applied because his position at the time was being downsized and the vendor had an opening. What he did not know was that the interviewers said he couldn’t troubleshoot his way out of a paper sack with a pair of scissors that that the opening went to some guy with a home lab who only applied at that vendor because that’s where he wanted to work.

Dan got a different job, held that for a few years, and then moved on to this role when the previous one got downsized.

Even though Dan hated security and wanted to get back to routing and switching (developers never, never demanded that switches or routers be turned off!), he knew that his experience with firewalls – even if it was little better than babysitting them in between TAC calls – meant a good chance of getting a job whenever there was a downsizing… 

… or whenever his political sensibilities informed him it was time to move on before he was fired for incompetence. At most firms, that was around 2-3 years. He had two places on his resume where he managed to hang on for five years. Things were really bad at those places, both of which were lucky enough to be picked up in acquisitions after suffering major breaches.

Not that anyone knew about those breaches until after the mergers, when the purchasing company’s IT did an audit of the poorly-managed gear.

As Dan returned to his chair, he was thankful that he could work from home. He also cursed the fact that he wound up working from home during times when he could be watching sports at home, or sleeping at home. This outage looked like something that would rob him of sleep, but he was damned if he would miss the playoff game on tonight! Dan turned on the television and put it on the big game.

As the sports match got underway, Dan wondered how this thing would all pan out and if it meant it was time for him to start looking for another job somewhere. During commercials, he checked his recruiter spam to see which roles looked like they might be good lateral moves. He didn’t want to move up into management or architecture, as that meant only more meetings and increased chances of dealing with C-level heavies, who could be worse than developers in their demands.

Around the end of the first half, it was time to mute the television and get on the call. Dan dialed in and watched the game as everyone else joined the call. 

The CISO was on and said, “OK, for starters, we’re not turning off the perimeter firewall.” Dan smiled. Take that, developers! “But we need that resolved ASAP. Dan, reach out to the vendor and get an RMA started. We’ve got to have our firewalls up and running.”

Years of experience in IT had helped Dan to develop his most important skill of all: how to curse silently when he was unmuted on a call. He paused his staccato mouthing to say, “Sure, I’ll get on that.” Calling TAC wasn’t all that bad, except for the small talk the vendor engineer always engaged in as screens refreshed or boxes rebooted or whatever. And with an RMA call, there would be tons of stuff Dan would have to say that would distract him from the progress his team was making in the playoff game.

Heaven help everyone if the RMA didn’t resolve things and there was some mess of rules on the firewall that, in their combination, blocked that stupid traffic that only ran once a month. That would mean getting an order to review 30 days of changes to see which one put the rule in to block that traffic.

And if no such rule could be found? “Turn it off!” would be the developers’ battle cry!

Dan got off the conference call and opened up another TAC case online for the RMA. As he waited for the callback, he set “looking for opportunities” in his linkedin for salespeople profile and replied to a few of the more promising recruiter spams.

Dan had no idea, of course, that his eventual replacement was going to be as clueless and hapless as he was. Dan also didn’t know the name of the nuclear reactor that guy used to work for, or the name of the GRU agent that had found the holes in that facility’s perimeter security.

Hell, he didn’t even know the names of the GRU agents that had penetrated his current company’s network, for that matter. To be fair, not many security specialists know the names of people in the GRU that have penetrated networks, but in Dan’s case, it was definitely for lack of trying.

An email popped into Dan’s inbox. It was from Kelly. She wanted to know if Dan could log in to the IPS console.

Dan fired up the GUI and tried the vendor default username and password. Hey, they worked!

Dan let Kelly know that he could. Kelly then emailed back for Dan to check the logs to see if the IPS systems were in bypass mode, or if they had been fully shut down.

Dan checked the GUI and saw that every single IPS was down. There was also a licensing error on the server and a warning about missing critical updates. Dan only mentioned the IPS devices being down in his response. He didn’t want to make the IPS guy look like an incompetent.

Kelly then asked for when the IPS devices had been switched off.

Well, hell, that meant searching the logs, and… holy crap! Those things had been turned off two years ago, and kept off! No wonder the IPS guy always gave up quickly whenever someone asked him to shut off the IPS! No troubleshooting, no request to try something different, he just said, “OK, try it now.”

Dan wondered briefly about the times in the last two years that “turning off” the IPS had provided a solution to whatever problem was going on…

But then Dan wondered happily and joyfully about how this proved that there was someone more incompetent than he was on the network. Not that it made him quit his job search. No, it made him look all the harder. He didn’t want to be the guy tasked with taking on the IPS system and turning it back on after 2 years of it being shadow shelfware. 

On the TV, Dan’s team made a terrible mistake. Dan blamed the coach and, completely unaware of the irony, said, “We need a coach that knows what the hell he’s doing! Fire the big dope!”

Really Bad Record Reviews

I’ve worked on a few lately, had lots of fun. Here’s a link to one of four I did in the last 3 days: https://www.youtube.com/watch?v=3M8BN7FNT0A

My inspiration came from Rolling Stone record reviews where it was clear that the reviewer either had an axe to grind, wanted to review a different record, or hadn’t bothered to listen to the album at all.

As it turns out, it is very easy to put out a bad review. When one disposes of the entire listening process, all that’s left is the cover and the track list, and those can get knocked out in a matter of minutes.

For me, the trick is in making a connection back to Pink Floyd that reveals my complete ignorance of the band I’m being all pretentious and elitist about. Sometimes I can, and that’s the delivery on the musical joke.

Heartbreaker

Dr. Borden exhaled and dabbed the sweat from her forehead before proceeding into the most critical part of the operation. She drew a deep, competitive breath and moved the precision mouse to aim the laser directly at the point of incision. With a click, the aorta would-

The screen went black, then a logon screen appeared.

“What the plokha budding spore?!?! What the spore just happened?”

Dr. Borden regained her composure and typed in her username and password – the patient was undergoing open heart surgery, there was no time to lose!

Agony of ages as the dots blinked in their circular path.

Username and/or password incorrect. Next login attempt in 00:05.

“SPORE SPORE BUDDING SPORE BUDDING EFFDISKING BUDDING K’CHORTU BUDDING SPORE!”

Dr. Borden didn’t want this to be the first patient she would lose on the table, but it was looking increasingly that way. He was somewhere in Alberta, wherever the meddrone landed, and she was in Atlanta, where the workstation ran in her Midtown apartment. She was doing everything to keep her mind down-to-earth and focused, but found that rage did all it could to take over.

Her mind raced – how long had it been since things went dark? Would the meddrone AI be able to abort the operation in time to save the patient’s life? Oh God, he is so effdisked if that AI doesn’t figure out there’s no doctor on the other end.

Because this was the third time Dr. Borden tried to log on to her workstation and the third time it kicked her back, this time with a caution she only had one shot left and that maybe she ought to call tech support before using that chance.

There was no way to call the meddrone, as those things were sealed off as far as comms went. There was only one way to talk to the meddrone directly, and for Dr. Borden, it was on the other side of a logon screen.

She called the number for her hospital’s tech support. Ringing. Well, at least it’s not down. Chortu, but that’s a lot of ringing. Well, let it ring, someone might die today if Dr. Borden shrugs her shoulders and becomes fatalistic in philosophy. She waits out the machine-induced stress.

And a machine answers. On an emergency line, it takes time to explain how the options may have changed recently and offers up a universe of choices, all a press of a digit away. Effdisk that, Dr. Borden presses zero. A human eventually speaks.

“Aetilus Medical Solutions help desk, this is Raj. May I get your employee username?”

“Eborden. E as in echo, b as in bravo, o as in oscar, r as in Romeo, d as in delta, e as in echo, n as in November.” Dr. Borden hated it whenever eborden sounded like edorgom. Spelling was usually faster than going over it twice.

“Dr. Elizabeth Borden, is this correct?”

“Yes. A man may be dying, please check if meddrone A as in alpha, 3447-”

“I’m sorry, Dr. Borden, I’m not able to contact meddrones. I’d have to escalate for that.”

“Please escalate, za’chortu.”

“There will be a, uh… oh, spore, a 30-minute wait.”

What the budding spore? 30 budding minutes? Might as well be 30 budding years! Even so… “Chortu, just get me in that queue.”

46 minutes later, a human spoke to Dr. Borden. “Hello, Dr. Borden? You there?”

“Yes. Contact meddrone A as in alpha, 3447-1369-0003.”

“A as in alpha, 3447-1369-0003. Got it. One moment… I’m sorry, I’m not getting a status, I’ll try again.”

“Do you know what’s going on?”

“Some kind of outage, that’s all I know.”

“Chortu… I desoxy-ed for this. All right, that meddrone number I gave you, it’s involved in a heart operation in Alberta. I need verification that it aborted the operation successfully and the patient status. Text me as soon as you got that info. I can’t log on to my workstation.”

“Yeah, none of the remote staff can log on. I’ve got the status query queued up for the drone and your number associated with it. Can I do anything else?”

“Nope. I’m needled. Cheers.” Dr. Borden touched her phone and the call ended.

Hopped up on the desoxy, Dr. Borden started to shake as she lost anything specific to focus on. Suddenly, she became aware of her heart rate and the blood being shoved pell-mell through her circulatory system. Don’t panic, Dr. Borden. You know how to ride out this part of the desoxy run.

The door opened and closed. Dr. Borden brought herself out of her trance state to see her boyfriend Teddy. “Hey babe.”

Teddy set stuff down on the table, even though he wasn’t supposed to. “Hey Lizzie. How’d the operation go?”

“Pffft.”

“Oh God.”

“No idea how the patient is, everything just cut out on me.”

Teddy pulled up a chair near Dr. Borden. The workstation screen was dark. A light blinked on Dr. Borden’s phone. Teddy didn’t know what to say. Someone, somewhere, connected to his girlfriend, could be dead.

Dr. Borden picked up her phone, but the light was just for a FriendFace notification. Apparently, one of her associates was a real slug and had gone fascist, from the content in his post. She unfriended him. “Budding fascist loser.” No word from Aetilus tech support.

“Budding what?”

Dr. Borden shook her head, “Nothing. Someone I went to high school with is now a fascist and dead to me. Hey, I did desoxy for this operation and I need something to focus on, or I’m gonna lose it.”

Teddy reached for the string of prayer beads Dr. Borden kept by her keyboard. She grabbed them and began to run them through her fingers like there was no tomorrow. Once you give a soxer something to do, they’ll do it. They just can’t give themselves something to do.

After a few minutes with the beads, Dr. Borden felt like she could talk and manipulate them at the same time. “What do you think caused the outage?” Teddy was a nerd. He knew answers to questions like that. He was a really cute nerd and fun to have around.

“Did it affect just you or a bigger group?”

“Guy said it took all the budding remote users out. No comms to meddrones.”

“Wow. That’s big.”

“You think it was terrorists?”

“Could be. More likely, it was someone stupid.”

Dr. Borden laughed. Teddy elaborated on the stupid. “So… it could be that someone turned off your time server. That would kill off your ability to log on remotely. Or maybe your computer cert expired. No, stupider, the root cert expired.”

Dr. Borden laughed even more. “I have no budding clue what that means! God, I love you!”

Yeah, she wasn’t doing any more operations today, system restoration or not. “Well, a root cert, that-”

“Shhhh! Explaining is boring! Just list off all the stupid stuff.”

Teddy knew better than to try and argue with a soxer. Last thing you want a soxer to focus on is a budding argument. “Um, OK, the VPN hub could be offline, uh… the directory service got swamped and went down… date field problem, oh spore! Do you know if your IT guys took care of your Y38 problem?”

Dr. Borden laughed harder, kinda maniacally now. It was time for the bell-1. She needed to come down off of this before she broke down.

*** *** *** *** *** *** *** *** ***

Dr. Borden opened her eyes and looked around. There was a little drool on her cheek, which was typical of a bell-1 cooldown. She sat up on the sofa and saw the blinking light on her phone. She reached over to the desk and picked up her phone. A swipe later, then a code, then a DNA pulsecheck, and she was in. The light was for a text.

The text was from tech support Raj. Spore, it was 7 hours ago! Must have texted Dr. Borden right after Teddy gave her the bell-1 dose.

Oh, chortu. The guy died. Dr. Borden sighed and scrolled. OK, so the meddrone did shut things down gracefully, so it was just his heart failing post-op, which was always a risk, regardless of how the operation went. Poor old dude and his now-dead carcass.

Dr. Borden texted back to Raj, what was cause of outage?

Company cert expired, sorry was Raj’s response.

Dr. Borden wondered why “cert expired” made her laugh a little.

Time to even things out with some zebra and ibuprofen. And some mango juice.

Teddy was in the kitchen. “Hey, I’m up.”

“Sleep ok?”

“Pffft. That spore’s not sleep.” She got the juice and then rummaged in the cabinet for the zebra and ibuprofen.

“How did the patient do, if I could ask?”

Dr. Borden downed the drugs and took a shot of mango juice. “Operation ended OK, but he died post-op. Not my fault, still sad. I’m taking zebra to even things out.”

“You also took something for the headaches, right?”

“I’m not an idiot, Teddy.”

“Hey, just checking. They say what caused the outage?”

“Cert expired, whatever that means.” Dr. Borden laughed again and felt weird about laughing. Was she going psychotic?

“It means nobody was checking one of the most important pieces of computer security, the thing probably being used to establish your VPNs and channels back to the drones and stuff. And the time on it ran out right in the middle of your operation.”

Dr. Borden was level enough to want to understand that. “Hold on. You mean to tell me that a company that knows precisely how long I’m functional on a dose of desoxy and how long it takes to do an operation and how long it takes to run drones over seven continents can’t keep time on the one thing that’s gonna tie them all together? Holy budding spore.”

“Well, that’s how you guys make money. Nobody makes a dime watching a calendar for a cert to expire. They know when licenses are due because someone else makes money with those. But certs?”

“Pffft.”

“Yeah, Pffft. That’s when they call me up. You remember when Charleston had that power outage last month?”

“That was an expired cert?”

“Yep. So was the Athens Supermax Riot. Cert expired, all the doors opened.”

Dr. Borden shuddered at that thought. That was too close to home. She still worked remotely, but those meddrones were trauma center models, only 60 miles away. And that was just three months ago. Images of the carnage still popped up in her mind if she wasn’t vigilant about her thoughts.

Now she had a question.

“Teddy?”

“Yeah?”

“Tell me… What is a cert and how does it expire?”

Again, Dr. Borden laughed for a reason she did not know.

Spirituality and Suffering

As I read more about Rabbi Kalonymous Shapira, who served as a rabbi in the Warsaw Ghetto during the years of Nazi occupation, “The Years of Wrath”, I arrive at his thoughts regarding the connections between spirituality and suffering. This is a man who was forced to experience not just a depth of human suffering, but, as he put it, a depth within a depth. I would do well to pay close attention to what he has to teach.

To begin with, he makes the connection between hearing or reading about suffering and actually experiencing it. The two are completely different. This I know from my own experience, so I know that the kinds of suffering I have not endured are academic only to me. Nevertheless, those who endure those sufferings can describe methods they used to cope with it. Should I face that suffering, I can rely upon their teaching in order to pass through it myself, with my soul and identity intact.

I know this because of how I used those methods and teachings to help me through my times of deep loss and crisis. When Rabbi Shapira speaks about areas of his sufferings in common with my experience and how he worked through them, I find that what worked for me also worked for him. So, when he speaks of areas where I do not have common experience with him, I trust in what worked for him. Heaven forbid, should I have to endure such things as he did, I will strive to endure them in the way he endured them.

While there are discussions about how sufferings can make us more spiritual once we have finished with that suffering and can reflect upon it, a sort of reflection upon answered prayers and tiny miracles, what do we do when our prayers for deliverance are answered with deeper suffering, the depth within the depth? What do we do when we think we can go no further, and then the road before us appears to be longer than we think we can survive? What do we do, to put it in raw setting, when we find ourselves in the Warsaw Ghetto on the eve of its liquidation, after having passed through plagues of Typhus, starvation on rations of a hundred calories a day, brutal murders in plain sight, horrors of the unrestrained and unfiltered brutality and hatred expressed by the Nazis? What do we do?

Rabbi Shapira’s answer is that we not consider the worldly end of the suffering. He turned to the martyrdom of Akiva, which happened in the Roman persecutions after the Bar Kochba revolt: Akiva said that he had always been willing to give his life for God – why should he turn away when that moment actually arrives?

More than that, Rabbi Shapira wrestled with questions about the existence of evil. While some have felt that evil is incompatible with the idea of a just and loving God, or even a God at all, Rabbi Shapira passed through that very evil and was able to state that evil did not matter. There is still a God, and it matters not what choices others may have made: those choices and their horrific impact do not negate or invalidate the existence of God. Nor do they invalidate the existence of a just and loving God. He is who he is, and we await the day of His judgment.

Rabbi Shapira taught that as persecution deepens, we must ourselves deepen our study and commitment to God. He noted that such deepening of study and commitment was next to impossible, especially as the repeated atrocities numbed the soul. It was to fight against that numbness that he encouraged the study and the commitment. Who is left to save if the body has become just a shell for the inner organs, the spirit within having perished from the psychic battering of repeated, unrelenting horrors?

No, we read more, we pray more, we make our observances more. We must fight that darkness that seeks to encompass us, in our depths within our depths. Even if we know we are to go to our deaths, we go to our death with our soul intact.

Will I go through such things as Rabbi Shapira endured? Maybe. I’m a member of a religious minority in a nation flirting with fascism. Such things could come to pass, where I cry out for rescue and deliverance, only to be faced with depth within depth of suffering. But if I can cry out from a depth, I can cry out from the depths, but I must prepare myself now, that I might have the spirituality developed in a time of peace to draw upon during years of wrath.

Spirituality does not end suffering. It does not mitigate the pain. It can, in fact, sharpen the pain and make us more aware of what we are enduring. But it does give us a path to draw ourselves up to face that suffering with dignity and faith. It gives us the ability to be patient and long-suffering. It gives us the ability to see to the eventual end of that suffering, even if it is in a day that comes after our own physical death. As long as we go to our death with our spirits still alive in faith, we are victorious over that suffering.

I believe in God, and I trust Him to be just and loving. I have had too many spiritual experiences in my life to believe otherwise. Yes, I have lost a son in a senseless tragedy. But my faith teaches me where he is, who he is with, and how to get there. Why should I refrain from finding joy in God’s mercy, even when I endure such a depth? Even so, if I am plunged into a depth within a depth, why should I be any less of a man than Rabbi Shapira or even Akiva? Their example, their wisdom, and their teaching, may that all be part of my life and how I endure all things.