Nick Vendor poked his head into the office, via a door left open. Nobody was in the office, but the time was 10:00, and Nick had a 10 o’clock meeting in Cecil Oh’s office, so Nick went in and got comfortable.

Getting comfortable meant sitting in the chair closest to the wall and angling it so that he could see both Cecil’s desk and the door to the office. Nick was a security pro, and was paid to be paranoid.

A few minutes after ten, Cecil Oh bustled in and smiled at Nick, “Sorry I’m late, but you know how it is.”

Nick nodded. Everyone from manager on up was always running late, everywhere he went. A CISO at big company was certainly no exception.

Just behind Cecil was Dirk Rector, the IT Director, and Cissy Tantisso, the assistant CISO. Cissy closed the door behind her and all sat in chairs around Cecil’s desk.

Cecil answered two emails and then said, “OK, sorry about that delay, but here we go. We’re here to meet with Nick Vendor here, who is going to give us his network health check assessment. He’s been scanning and probing for a few days, so we’re all eager to hear what he’s found.”

Nick smiled through that “eager to hear” part. Everyone’s eager to hear, but not everyone is so eager to have had heard. There was always at least some bad news in a network health check assessment. Today, the amount of bad news was somewhat more than just “some”.

Cecil held his hands towards Nick and said, “It’s all yours, sir.”

Nick nodded, smiled, and did his best to front-load the cushiony stuff. “Thanks very much, Cecil, and thanks of course to Cissy and Dirk for all the cooperation you and your teams have provided me in this past week. I really do appreciate all the work they’ve done to help me. They certainly helped me to have a little fun as I did my assessment and they were more than helpful in providing me with information about different device types and systems you have installed here at Amalgamated Potrzebie. They’ve been a great help.”

Cissy, Dirk, and Cecil all nodded in appreciation of Nick’s thanks.

Nick shifted in his chair. “So, let’s get to the numbers. About 70% or so of your Windows PCs are managed in SCCM and a similar percentage are up to date on their AV. Dirk’s got the action to follow up with the desktop team to close the gap with the other 30%.

“Macintosh systems, there’s only a few hundred of those, but they’re all pretty much managed centrally. There were, like, 10, that weren’t. We know where they are and Dirk’s Mac team will follow up on those. It’s just a small office, right?”

Dirk agreed. “It’s a marketing team in our Pittsburgh campus, that’s correct.”

“Thanks. Linux.”

Everyone took a deep breath for Linux. They all knew they had a problem there, but they still had to hear about it.

“Linux… well, these developers have not yet embraced the idea that they have to install a security client on their test boxes.”

Dirk objected. “Well, hold on there. That client doesn’t work on all flavors of Linux.”

Cissy said, “We need to stop using those flavors, then. We can’t have developers deciding what risks we accept.”

Cecil grinned, “Yeah, if someone else says ‘I’ll accept the risk’, what am I here for?”

As she finished chuckling, Cissy said, “I’ll take the to-do for getting dev to standardize on Linux.”

Nick said, “Good. That will go a long way towards getting Linux in line.” He took a deep breath, deeper than the one for Linux. “That brings us to embedded devices. We’ll start with embedded Windows, the badge readers at the entrances first. Those devices are active on the O-Sheet Botnet, nearly all of them. The botnet software listens on port 80, HTTP, and determines if it’s botnet communications or if it should hand off to the legit software that uses HTTP. So, if we block port 80, we block both the botnet and the device, which means nobody gets in at that location.”

Cecil sat forward. “Wait, what? A botnet?”

“Yes sir. A botnet in practically all your badge readers. It can infect other devices from those badge readers, as well. That’s basically where the local command and control software is located. Your IPS will block north-south traffic, so it won’t get to the data center, but the east-west stuff is wide open.”

Cecil sat back. “Recommendation?”

Nick grimaced. “Honestly? Rip them all out and get a new system, one that either doesn’t use port 80 or one that doesn’t use a network connection at all. These were all installed with the vendor’s default admin credentials still active, which is probably how they were able to be compromised.”

“Any way to remediate in place?”

Nick shook his head. “It’s embedded legacy Windows. No way to really get in there and make any changes unless we’ve got our own red team to write custom code to pop the devices and clean out the malware. Even then, there could be another zero-day exploit that comes to light and then you’re back to where you are now. And this is just the start.”

Cecil had a panicky tone to his voice. “Whoa, whoa, whoa, whoa, whoa – give me the big numbers, let’s start with that.”

“Amalgamated Potrzebie has a large number of industrial and security control devices that show indicators of compromise. Close to 40% of your IoT devices are showing signs of compromise.”

Dirk asked, “How many of those are on our production floors?”

Nick looked at his spreadsheet, did a quick bit of math in his head. “Maybe 20-25% of your production systems are compromised, but the compromised devices tend to be concentrated in certain facilities. You’ve got most that are still clean, but a good chunk that are shot through-” Nick corrected his language. “- showing about 80-90% compromised.” It was important to leave out hyperbolic adjectives when delivering news of this magnitude.

Dirk’s next question: “Any of those compromised lines in Council Bluffs or Little Rock?” Cecil and Cissy looked at each other with trepidation. Those were the facilities with Defense contracts.

“Yes, both.”

Dirk spoke to Cecil and Cissy. “We have got to get those cleaned out, as soon as possible. We can’t keep our contracts with that kind of threat active in the environment.”

Cissy responded, “Hold on, we don’t even know what’s compromised in those locations. Nick?”

Nick looked over the list. “Temperature gauges, badge readers, security cameras, the time clocks, well time clocks just in Council Bluffs – Little Rock clocks are fine, um… the digital signage is infected, as are the smart light bulbs in the Woodbridge building in Little Rock… ummm… oh, crap.”

Cecil didn’t like that. “Excuse me?”

“I didn’t see this earlier, and I apologize for that oversight, but all your Philly switches on the production floors are basically being run by a group outside of AP.”

“What?”

“There’s a feature on those models to allow for easier automatic upgrading, but it’s vulnerable to an attack. Basically, send a packet to the port used for auto-upgrade and you get a root prompt. We can’t access the devices, but there’s a stream of traffic running between those boxes and a TOR node.”

Cecil didn’t believe it. “No way, we block all manufacturing traffic from the Internet. It’s a segmented environment.”

Nick held up his hand. “It’s not segmented. There’s a vendor-owned Windows 2008 server that bridges traffic between those lines and the Internet. It’s basically on a DSL line and hasn’t been patched since 2010.”

“Who’s paying for the DSL line?”

Nick shrugged his shoulders. “Probably whoever paid for it in 2010 and then never did a budget review since then. At any rate, we recommend not turning off the server, since we don’t know what happens when the communication line is severed. You could wind up losing your switches and maybe other production line equipment that’s connected to them. As for the Philly switches, I do have a note here to check the rest of your sites for this issue. I just didn’t have enough time to finish that part before our meeting here.”

“OK, well, I’m going to want you to follow up our discussion here with a complete check of those Philly switches.” Cecil felt a pit opening up in his stomach.

Dirk and Cissy looked at him for guidance, with Cissy asking, “So what do we do about the compromised Defense lines?”

Cecil looked at his desk. “We need to go up the chain on this one. We have to let DoD know that the lines are compromised, but at the same time, they may accept the risk and let us keep producing parts. We’ve got a lot of pressure to fill the quotas they’ve set for us.”

Nick asked, “Should I still be in this room for that discussion?”

Cecil thought about what the lines in Council Bluffs and Little Rock were turning out. On the one hand, the independent, armed, unmanned aerial vehicles were some seriously top secret items. Nick shouldn’t be privy to that information.

On the other hand, Cecil felt like he had to know if those IAUAVs were themselves compromised… “Well, Nick, that depends. Do you have a security clearance?”

“No.”

Dirk looked a little aghast. “How was he able to do this survey, then?”

Cecil flopped back in his chair. “Apparently, we’ve got a few blind spots around here as regards security…”