Author Archives: deanwebb

The Compromise Vanishes

The CIO and CISO left the room, leaving only Sandeep the temp and Avi the digital forensics expert at the table.

Sandeep said, “You know I’m not at all authorized to say anything of effect to you.”

Avi said, “I understand that completely. You are not an employee of the client. I am not to consider you, in any way, to be authorized to direct my actions or the actions of my employees in their relationship with the client.”

Sandeep stopped recording. “That will do. You know what’s going on, and what I’m about to tell you.”

Avi nodded.

Sandeep said, “Then, I really don’t have to tell you anything.”

Avi slowly shook his head.

“All right then. Just let me know when you’ve got your final report ready so we can hand that over to the cyberinsurance people.”

Avi said, “Absolutely. We’ll work long days, nights even, but we will deliver the report and I’m sure it will be complete and accurate.” That was just in case something else was recording the conversation. Otherwise, a word to the wise was sufficient.

Avi and Sandeep arose and each went back to his respective hotel cubicles. 

The client had hired Sandeep strictly as an outside consultant that would vet and approve the digital forensics report that Avi’s team would deliver. The client and its officers did not have any care or concern what Sandeep did between now and approving Avi’s report. Sandeep knew his place in the world, which was why his laptop was not visible from the aisle and his back was to the wall, which is no mean feat in a cubicle. As long as Sandeep attended his scheduled meetings and then later approved that report, nobody cared what he was looking at on his phone or computer.

Avi, on the other hand, had work to do. The client stood exposed and plundered to the world, a victim of a massive breach. As a massive multinational in a profitable sector, it had a preliminary estimate of over $400 million in damages – on the line of what companies suffered when WannaCry and NotPetya came on the scene. 

Avi’s team worked with a strict rule – no paper, whatsoever. No writing, no jotting of notes, no paper at all. The only papers involved were those in the final printout. Otherwise, all products of his team’s work would leave when his team took their laptops out of the client site. 

Avi’s team had another strict rule – no conversations of note over landlines, cell lines, email, or chat. They were to avoid speaking above whispers, as well. So many things left a digital trail, and it was best to not leave that trail to begin with. Then, it couldn’t be followed back.

When someone on Avi’s team needed to collaborate with someone else on the team, they would whisper together. If they needed to have a third person involved or a lengthy conversation, they would go outside. It didn’t matter how cold or hot it was outside or what security they’d have to go through repeatedly to complete the journey, the rule was adamant: go outside, where only nature was likely to be listening.

If Avi had to brief his team with customer representatives attending, he had a terse, formulaic presentation. “The client has been breached. We are to determine the root cause, the extent, and the origin of the breach. We all know what is on the line here, so let’s do the best work that we can do for our client.”

Each member of Avi’s team had a specialty, so there was no need to go through who was going to do what and when. They just moved forward. Avi secured any credentials they would need to get started, but that was typically a formality. His team could get those needed credentials much faster than any corporate process could deliver them. Any discrepancy between credentials used and credentials that were supposed to be used could be attributed to fallout from the breach event. Besides, those passwords were about to be changed, anyway, so it wasn’t like anyone on Avi’s team could use those usernames with those particular passwords again. The end justified the means.

In the aftermath of a breach, procedures and processes tended to be protean, plastic, verbally-approved sorts of things. This was especially true when dealing with Avi’s team’s requirements. No client had yet said, “Give them anything they want. Literally, anything that they want.” But that seemed to be the understanding at each client site thus far. 

Nobody ever called Avi in the first place unless they intended to have that kind of understanding from the beginning. With damages in the hundreds of millions of dollars, these customers could not afford any additional risk. They’d already accepted the risk on what got them there in the first place: they had to be certain about securing the means to get out of that predicament.

And that is why they called a man who spoke very little to his team when others could overhear a conversation, who would deliver one and only one document, with zero review cycles permitted. They would call a man like Sandeep to handle the document from Avi, as an extra layer of insulation. 

Sandeep merely needed the skill of being able to handle his extended boredom. Avi’s team needed some profoundly technical digital forensics skills. This is why Sandeep lived comfortably, but Avi lived comfortably and securely.

Generations ago, one of Avi’s ancestors had worked in Moscow, back when it was the capitol of the Soviet Union. Avi’s ancestor worked in a photography lab. Avi’s ancestor had but four tools at his desk: a magnifying glass, an airbrush, a razor knife, and rubber cement. He was a redactor, one of the best.

A commissar would bring a photo to the redactor and point to a face in a crowd or a man in a line. By the end of the day, the commissar would collect a photo that did not have that face or that man. The photo would not have any stigmata where the face or man used to be. There would be no streaks, no absence of background noise, no overly-softened edges, no awkward gap. Space itself would disappear as Dzerzhinsky’s Tikhii Don played on the radio. All day long, the redactor worked quietly, creating a world of illusion as the music of Socialist Realism flowed around him. 

This was a work that needed no words. A photo, a finger, that was all that was needed to make things appear to be as they needed to be for the political demands of the moment. Sometimes, a photo would return to a redactor, with a finger pointing at another person or two, and they would be gone by the end of the day. They may have been necessary for yesterday: today, they were not what the Soviet Union needed. 

Kabalevsky’s symphony played as another face vanished. The redactor filled in the empty space with a painted-in fiction of the clothes of the man behind the one that had disappeared. Two officers left the official portrait of the general staff – they stood on the edges, so only a simple cropping did the trick. A photo with a very dangerous face had turned up – the redactor knew this was a rush job from the face alone, without needing to see the stern, almost panicked expression on the commissar’s visage.

Whose was the dangerous face? It could be one of hundreds, no, thousands, but there was no reference for the redactor to turn to. All the faces that were not to be no more forever were in the mind and memory of the redactor. Their names were not important, only their appearances. If their backs were turned to the camera and nobody could tell they were in the photo, there was no need to have the photo placed before the redactor. But if they turned up after they were supposed to have disappeared, well… Khachaturian’s Toccata was proper background music for the rush work. The commissar had not even left, but collected the finished product immediately.

Always, the work of the redactor was in taking what was unacceptable to see and making it acceptable once again.

Avi did not know the name of this ancestor, let alone his job. One day, the redactor went in to work and did not return. His wife knew well enough to not ask a question and his sons had perished in the Great Patriotic War. His daughter was too young to remember her father, and mother never spoke of him.

If there was anything of an inheritable skill in what Avi did, it was surely enhanced by the environment he maintained for himself and his workers. When not on the job, they trained and critiqued each other, each member of the team fully aware that his or her work had to survive the criticism of the others if it was to be ultimately satisfactory to future clients. They would look for a broken reference here, a missed line of code there, accepting that the others were doing the same to their own work. If they made mistakes, they were in ways too difficult to be noticed by the naked eye.

There was music as Avi worked. Not Dzerzhinsky, but George Acosta; not Kabalevsky, but Armin van Buuren; not Khachaturian, but Tiësto – these played on Avi’s earbuds as he sought out the things that were unacceptable to see for his clients. Silently, ruthlessly, they would find the malware and eliminate it utterly, even down to the bare metal on the hard drive. Not a trace would remain.

The log files – not a word was said – the patterns of the breach, its fingerprint, those vanished as well. Did the client have a tamper-proof protection on the log files? That had to be worked over, as well. The client did not need any evidence of the unacceptable things, and evidence of evidence was equally unwelcome. 

A finger pointed at an item on a screen and one of Avi’s team members would make it go away. The purge ran its course, but the task was not yet concluded. 

There had been a breach, after all. There needed to be evidence of such, so that the client might collect on its cyberinsurance policy. 

The insurance companies – and their backers in the reinsurance companies – never hesitated to write a policy or collect a premium. But paying a claim? Ah, the tortured screams of the money being pulled from the insurance company’s accounts could be heard the whole world ’round. How could one blame the insurance company for taking pity on its money and finding a way, any way, to prevent having to part with it?

The cyberinsurance policy would not pay out for an act of war or terrorism, a common exclusion in most policies. The problem was that if a nation had ever accused another nation of using a particular piece of malware, that malware would forever be associated with acts of war and terrorism, even if a mere script kiddie in a dirty apartment was using it to raise money to pay his or her rent. 

Avi’s team whispered, pointed, talked outside, and listened to electronica so that the ravages of war and terror would vanish… other ravages were needed to complete the picture, and Avi’s team provided complete pictures at the end of their engagements.

This business of digital redaction, it thrived on the unsaid and the unwritten. Better still if things unsaid and unwritten were handled by independent third party contractors, such as Sandeep. Let the third party temp worker not say anything or not write anything. That was best for all concerned.

The client also felt that government inspectors were best suited for government work. They had agendas often in conflict with the continuity of business and the unimpeded flow of commerce. Best to keep private things in private hands.

At the end of long days and long nights, Avi and Sandeep were again in the conference room. Avi handed Sandeep a report for his consideration. Sandeep read over it, asking questions as he turned pages. 

“So, Avi, no evidence whatsoever of a state-sponsored attack?”

“None at all, Sandeep. The breach was entirely the work of a criminal organization utilizing custom malware.”

Sandeep smiled. He’d have a few days where he could be idle at home instead of idle at a client site when this business concluded. “What if an auditor finds evidence of a state-sponsored attack, such as in inactive or deleted malware on a hard drive?”

“We called that out in section 9. We did see some malware that had been used in state-sponsored attacks before, but which was not part of this attack, as the forensic data will show. Attack and exploitation patterns common with that malware are simply absent in the records of this attack, which correspond closely with the ways in which this malware suite is utilized by criminal gangs. That state-sponsored stuff may have caused damages, but they would have been of limited scope and outside the events and claims associated with this breach.” It was almost as if Avi had said those things a hundred times before.

“Is it possible the criminals were working alongside or on behalf of a state or terror organization?”

“Given the financial nature of the targets in the breach, we disagree with that conclusion.”

Sandeep looked above the top of his readers. “What about damage or compromise to non-financial targets?”

“Collateral damage or compromise pursuant to the eventual financial goals of the criminals.”

Sandeep nodded and flipped through a few more pages quietly. Nice fonts and color scheme. Plenty of pie charts. Executives loved pie charts. If there were a church for executives, William Playfair would be the greatest prophet of that denomination, for it was Playfair’s Statistical Breviary that brought the pie chart down from the mountaintop. 

Playfair would also figure highly in a pantheon for those that see things as they are and then change their appearance to what their employers want them to become. Playfair’s employer, the British Empire, did not want to countenance a Revolutionary France flush with cash. Playfair came up with a way to make France overly-flush with cash and ruined that nation’s economy with one hundred millions of counterfeit assignats. Was such a thing a fraud? No, it was an outright service to Mr. Playfair’s employers! Besides, how could a man with a name like “Playfair” be capable of anything other than playing fair? Really, now.

And for all Sandeep could tell, there was not a hint of fraud or evidence tampering in Avi’s report. For all intents and purposes, it looked like exactly the sort of thing an executive would want to hand to an insurance company – and what an insurance company would want to hand to a reinsurance company. 

“Looks good, Avi. Everything seems to be in order. Dotted all the i’s, crossed all the t’s.”

Avi smiled. “And the good news is that, once they get their claim paid out, it’ll be as if this all had never happened.”

“Well, we’ll still show up as line-items for this quarter.”

“True, that can’t be helped. Someone had to clean up all that mess.”

Sandeep tapped the conference table twice and stood up. Avi followed suit. They shook hands and made the small talk of departing businessmen.


Men like Sandeep and Avi have never been long permanent in any place. They travel over the face of the earth, something like a caravan of merchants. On their arrival, every thing is found trampled down, barren, and bare. While they remain, all is bustle and remedial. When gone, all is left green and fresh.

Just see for yourself.

Insecurity Through Incompetence

“It’s blocking our production traffic! We have to shut it off!”

Dan Weber rolled his eyes. Why is it that developers always make me want to punch someone in the face? He unmuted his line and said to the conference call, “We can’t do that, we absolutely can’t. That’s the perimeter firewall. Turn that off and we might as well hand our data over to the Chinese and Russians and anyone else interested.”

“But we have to ship product! We can’t do that with the firewall in its current state. It’s blocking all our traffic.” Same developer as before.

Dan said, “It’s blocking all traffic from everywhere right now, so at least we’re safe. I’ve got a TAC case open with the vendor and we’ll have it resolved eventually.” Thank goodness this isn’t a video call. Dan made several obscene gestures at the initials of the developer that wanted to shut down the firewall.

A manager asked, “Do you have an ETA on when that firewall will be fixed?”

Dan’s head tilted up as he leaned back in his chair. “No. It’s a code problem from the upgrade. We’ve escalated it, but no ETA.”

Manager, again, “Can you roll back the code?”

Dan kept looking at the ceiling. “No. There’s no rollback from this upgrade.”

“Can you restore from backup?”

“No. because the last backup was on the previous version, so it’s not compatible with this version of the code. We just have to wait this one out.”

The manager put his foot down. “Unacceptable. Turn it off.”

Dan sat up, lightning going down his spine. “I have to have-“

Dan’s manager, Kelly Montlac, interrupted, “Hey, we need to discuss this offline with Raymond.” Raymond was the Network Services Director. A conversation with him would of course involve the director over the developers and probably also the CISO and CIO, if they could be reached at this time. It was late in the day in the USA and early in the morning over in Europe, where the C-levels lived.

The developer manager raised his voice. “We need to get back into production. Turn it off and then we can talk it over.”

Kelly dropped her voice into a growl. “Not gonna happen.” Silence, then Kelly drove the point home. “Not gonna happen.”

The Major Incident Coordinator didn’t speak right away after that, but eventually said, “OK, how about we end this call so we can get that meeting together? And then I’ll have this bridge back up in 60 minutes, after that meeting gives us direction on the perimeter firewall.”

All the managers agreed to that and Dan couldn’t leave the call fast enough. As he dashed down the hall for a badly-needed bio-break, he cursed the idiot developers that refused to bounce their own servers to see if it resolved the issue. Five nines, be damned! Wasn’t there a limit to what had to be sacrificed to get that precious uptime?

They’d already turned off or bypassed the IPS, the proxy, the NAC, the datacenter firewall, the load balancer, the WAN accelerator, the VA scanner, the data protection system, the antimalware solution, the, um… were there any other security solutions? If so, they probably also got turned off, because that’s how development rolled. If Dan hadn’t been on the TAC call with the vendor all day, he would have been on the earlier Major Incident call and the perimeter firewall would have been assailed from within at that point in time.

Dan reflected on which of those systems needed to be turned off as he washed his hands. He was pretty sure at least half those systems were configured improperly and the other half were running just good enough for production, but not optimized. Dan himself barely had a grip on the perimeter firewalls. So many vendors, so many rules that had piled up over the years, and only so much he could do with the firewall management platform before he violated change management procedures or stepped on someone’s shoes in Governance.

When Dan had asked for training, he had gotten it. It was neither the trainer’s fault nor management’s fault that Dan was, at best, a mediocre student. More often than not, he was just a warm body that could complete change requests. Not a clever man, our Dan.

In fact, if one made a school of the entire IT staff at where Dan worked, there would be no need for a Gifted and Talented class. There would be some call for a remedial reading course, but most of the imaginary student body would be average kids with average brains, wishing that the weekend would hurry up and get here. 

Dan had once applied to work at a vendor. He applied because his position at the time was being downsized and the vendor had an opening. What he did not know was that the interviewers said he couldn’t troubleshoot his way out of a paper sack with a pair of scissors that that the opening went to some guy with a home lab who only applied at that vendor because that’s where he wanted to work.

Dan got a different job, held that for a few years, and then moved on to this role when the previous one got downsized.

Even though Dan hated security and wanted to get back to routing and switching (developers never, never demanded that switches or routers be turned off!), he knew that his experience with firewalls – even if it was little better than babysitting them in between TAC calls – meant a good chance of getting a job whenever there was a downsizing… 

… or whenever his political sensibilities informed him it was time to move on before he was fired for incompetence. At most firms, that was around 2-3 years. He had two places on his resume where he managed to hang on for five years. Things were really bad at those places, both of which were lucky enough to be picked up in acquisitions after suffering major breaches.

Not that anyone knew about those breaches until after the mergers, when the purchasing company’s IT did an audit of the poorly-managed gear.

As Dan returned to his chair, he was thankful that he could work from home. He also cursed the fact that he wound up working from home during times when he could be watching sports at home, or sleeping at home. This outage looked like something that would rob him of sleep, but he was damned if he would miss the playoff game on tonight! Dan turned on the television and put it on the big game.

As the sports match got underway, Dan wondered how this thing would all pan out and if it meant it was time for him to start looking for another job somewhere. During commercials, he checked his recruiter spam to see which roles looked like they might be good lateral moves. He didn’t want to move up into management or architecture, as that meant only more meetings and increased chances of dealing with C-level heavies, who could be worse than developers in their demands.

Around the end of the first half, it was time to mute the television and get on the call. Dan dialed in and watched the game as everyone else joined the call. 

The CISO was on and said, “OK, for starters, we’re not turning off the perimeter firewall.” Dan smiled. Take that, developers! “But we need that resolved ASAP. Dan, reach out to the vendor and get an RMA started. We’ve got to have our firewalls up and running.”

Years of experience in IT had helped Dan to develop his most important skill of all: how to curse silently when he was unmuted on a call. He paused his staccato mouthing to say, “Sure, I’ll get on that.” Calling TAC wasn’t all that bad, except for the small talk the vendor engineer always engaged in as screens refreshed or boxes rebooted or whatever. And with an RMA call, there would be tons of stuff Dan would have to say that would distract him from the progress his team was making in the playoff game.

Heaven help everyone if the RMA didn’t resolve things and there was some mess of rules on the firewall that, in their combination, blocked that stupid traffic that only ran once a month. That would mean getting an order to review 30 days of changes to see which one put the rule in to block that traffic.

And if no such rule could be found? “Turn it off!” would be the developers’ battle cry!

Dan got off the conference call and opened up another TAC case online for the RMA. As he waited for the callback, he set his LinkedIn profile to “looking for opportunities” and replied to a few of the more promising recruiter spams.

Dan had no idea, of course, that his eventual replacement was going to be as clueless and hapless as he was. Dan also didn’t know the name of the nuclear reactor that guy used to work for, or the name of the GRU agent that had found the holes in that facility’s perimeter security.

Hell, he didn’t even know the names of the GRU agents that had penetrated his current company’s network, for that matter. To be fair, not many security specialists know the names of people in the GRU that have penetrated networks, but in Dan’s case, it was definitely for lack of trying.

An email popped into Dan’s inbox. It was from Kelly. She wanted to know if Dan could log in to the IPS console.

Dan fired up the GUI and tried the vendor default username and password. Hey, they worked!

Dan let Kelly know that he could. Kelly then emailed back for Dan to check the logs to see if the IPS systems were in bypass mode, or if they had been fully shut down.

Dan checked the GUI and saw that every single IPS was down. There was also a licensing error on the server and a warning about missing critical updates. Dan only mentioned the IPS devices being down in his response. He didn’t want to make the IPS guy look like an incompetent.

Kelly then asked for when the IPS devices had been switched off.

Well, hell, that meant searching the logs, and… holy crap! Those things had been turned off two years ago, and kept off! No wonder the IPS guy always gave up quickly whenever someone asked him to shut off the IPS! No troubleshooting, no request to try something different, he just said, “OK, try it now.”

Dan wondered briefly about the times in the last two years that “turning off” the IPS had provided a solution to whatever problem was going on…

But then Dan wondered happily and joyfully about how this proved that there was someone more incompetent than he was on the network. Not that it made him quit his job search. No, it made him look all the harder. He didn’t want to be the guy tasked with taking on the IPS system and turning it back on after 2 years of it being shadow shelfware. 

On the TV, Dan’s team made a terrible mistake. Dan blamed the coach and, completely unaware of the irony, said, “We need a coach that knows what the hell he’s doing! Fire the big dope!”