The story I’m about to tell you is true. The names and incident specifics have been changed to protect me from violating my NDA agreements.
This is the network: the RFC 1918 ranges. I work here. I’m a security vendor.
It was a cold November day at the customer site when I walked in for the workshop. I met the security architect in the lobby. Nice enough guy, I guess. His name was Ram Gopal. We exchanged pleasantries and headed to the conference room.
Once we were all plugged in and Ram fired up my product’s GUI, we got underway. I was there to do one thing, and that was to answer the question, “What is this?” for every device on the network. Network visibility, that’s my stock in trade, and it’s an endless, glamourless, thankless job that’s gotta be done.
I know Jack Webb said that about being a policeman. Well, I’m Dean Webb, so I can say that about being a security professional.
The Windows devices were easy enough to figure out. Thousands of endpoints with TCP 135, 139, and 445 open. We passed over those. We also skipped the TCP 515 and 9100 devices: printers or print servers, for the most part. Ram’s eyebrow went up when he saw switches and routers with Telnet still open, but we knew what those things were. He’d write the email to the network team later on.
He’d write that email later because we were now at the end of the line, the Skid Row of the network. All the IoT devices plugged in by every Tom, Dick, and Harry at the company. The devices dangling off of D-link hubs, just like ID cards hanging off of branded merchandise. The gear left behind by long-gone consultants. The things people plug in without ever thinking. And the heartbreak that comes from not thinking. No thinking at all about security, about personal information, about known vulnerabilities, about default passwords. They’re just plugged in, given a server IP address, and then forgotten about, left for someone else to worry about.
The first one we looked at had a normal, unassuming IP address. 10.2.44.63. Nobody ever expects trouble from 10.2.44.63. It’s the IP address next door, the all-American kid with a freckled face and a country smile. We look at that IP address and think nothing of it. Well, I’ve got a news flash for you, friend: you never trust an IP address that has port 80 open. It could turn out to be who knows what – a botnet control server, a pivot to the rest of the network, an exposed database, a key to the kingdom, your kingdom, and you won’t be king much longer with devices at 10.2.44.63 having that port 80 open, for anyone to stop by and look at.
I said to Ram, “Let’s put that IP into the browser. See what comes up.”
Ram had to ask, “Hey, I thought you were on the blue team, Dean?”
“Let me set you straight, Ram. I’m on the blue team, through and through. I’m not a penetration tester – coding was never my bag. But when it comes to devices that are serving up port 80 like a dealer offering that first, free hit of dope, that makes my blood boil. My red blood, if you catch my drift. And what kind of blue team player would I be if I didn’t know how the red team was going to come at me? What if I didn’t know about my blind spots, where some punk with a buffer overflow could give me and everyone on this site a really bad day?”
“OK, OK, we’ll see what comes up.”
A colorful page with a vendor logo is what came up, complete with a pair of boxes where a username and password go. Like a reflex action, I typed in that vendor name and “default password” into a search engine. Did I feel lucky? 13 years as an IT guy, do you think I was going to feel lucky? With search engines serving up sponsored pages ahead of the results I really wanted? No, I didn’t feel lucky. I felt smart, and went on to a page of results.
I saw the link to the quick start guide – may as well have been called the lazy hacker’s cheat sheet. And there it was on page 3, the default credentials. Admin/admin.
Ram typed those in and hit enter. The browser wheel spun, and he was in. “So, this looks like the admin page for some system.”
I pointed at the link on the left that read Badge Reader Status. “Looks like your badge reader system.” Then I pointed at the link to Employee Access Database. “And that looks like where the fun starts.”
Ram clicked the link to the database. We saw employee names, phone numbers, usernames, and their access behavior for the last 30 days. I may have said this was where the fun starts, but Ram’s face told a story of pain, disappointment, and betrayal. He said, “Hey, Dean, I need to put a pause on this for about 20 minutes.”
“You need to have a quick meeting with those badge reader people.”
“Yeah, you got that right.”
“Do what you need to do, Ram. I’ll be here.” I wasn’t going to leave his side. It may not have been his first wide-open system, but that didn’t matter. All the years I’ve been a security professional, it’s never been easy. We laugh, we act tough outside, but deep down inside, we’re all feeling that pit in our stomach open up as we wonder how badly that access has been abused in the past. Worse, we know that somebody in operations somewhere is using an app a developer threw together that uses that very vulnerability we just found in order to get his work done, work that makes money for the company. And when it comes down to shutting down a vulnerability or making money, who do you think is going to win out, a lone security architect or a whole operations department?
That’s why I stand with my customers. That’s why I document my findings. I may only type 40 words per minute, but those are 40 more words every minute that make this world a little bit safer, a little bit more worth living in.
Ram came back from chewing out the badge reader team and I had another IP address with that HTTP port open. This one was the very important-looking 10.122.37.1. Ram put the IP into his browser and said, “That’s supposed to be a perimeter router IP address. That’s our Rancho Cucamonga location.
“I didn’t know you had port 80 open on your routers.”
“We don’t. We turned off HTTP on every one of them.”
“Do you use the same vendor for all your routers?”
“Yeah, we’re a dedicated shop.”
By this time, the web page for the device had come up. “So you guys are a wall-to-wall Netgear shop?”
Ram glared at the Netgear home router login page. I was on the search page, typing in Netgear home router. The second autofill line offered up the other two keywords I needed. The next page gave me all the info I needed without needing to go to a quick start guide. “Try admin/password, Ram.”
One admin/password later, and Ram was on the Rancho Cucamonga perimeter router.
“You need another 20 minute break, Ram?”
“Please?”
“Sure thing, pal. I’ll be here.”
“Wait, before I go, can you tell me how many more Netgear boxes I have on my network?”
“Sure thing.” I applied a filter for Netgear MAC addresses. “You got 21, all with .1 addresses.”
“Can you email me that list?”
“You betcha.” Ram got his meeting together and I sent off a spreadsheet export from my product’s GUI.
There may have been 21 home routers on that list, but Ram only needed 10 minutes to tell a very interested network team the information they needed to know to shut down a Netgear ring that had been a thorn in their side for years. Every one of those IP addresses was one they’d try to get to work in their RMM tool, but their network credentials never worked on them. Now they knew why: they weren’t going with the first or second password everyone guesses when trying to pop a box for the first time.
I was glad that the network team was on Ram’s side, but I didn’t envy the arguments ahead of them. I was betting that these routers hadn’t been a problem before, and that was going to be a problem for convincing concerned parties that they were going to be a problem right now, or an even bigger problem in the future.
Ram came back from his short meeting and said, “You know, comedy works best in threes.”
“Well, maybe the laugh we get from this one will make up for what we see on the next two.”
Ram laughed uneasily. I had already set up a view with plenty of bad news in it. He asked, “What’s that you have there?”
“Well, Ram, these are Windows devices that are members of your domain.”
“OK, that’s not a shock.”
“These have RDP open.”
“RDP?”
“Port TCP 3389 itself.”
“Oh yeah. Sorry, but RDP is also used as an acronym here. I got confused.”
“I understand. Anyway, these stood out because I needed to ask if you have any offices in China.”
“No, we’re strictly in the USA.”
“Nothing in Belarus or Russia?”
“No.”
“Republic of Vietnam? India? Turkey?”
“No, none of those. What are you getting at?”
“Those are just some of the nations with source IP addresses hitting these boxes on port TCP 3389.” I showed him the network traffic view that told the whole sordid tale.
“I gotta shut down the firewall on that port.”
“Try also the commercial ISP connection at those sites. And then look for the /32 routing statements that send traffic bound to those other nations through the dual-homed Windows boxes with RDP open and exposed to the Internet.”
Ram left the room and I knew he had another impromptu meeting to conduct. I did a little click work in my product and found the IP cameras for this building. Every one of them was open on port 80. On the fifth one I tried, I got the live feed of what Ram was doing in the other conference room with the Windows team. I had kept the other browser windows open so Ram would see that I didn’t even need a default credential to tap into every security camera in his enterprise.
What else did I find? The usual suspects. Xboxes and Playstations. Unpatched web-connected television sets. Printers that responded to the “public” SNMP community. Every iDRAC port that answered to “Calvin”. Nearly every other customer of mine had these devices on their network, and nearly every other customer of mine had a workshop where I called these out as security risks. And even though there was plenty of gore on the network, they thanked me for what I did, because I was on their side. I was fighting the good fight, right there with them, and I was damn glad that they were fighting right alongside me.
But on every network, there’s something new, a little adventure you never wanted to be on, a dragon you haven’t seen before that you nevertheless had to slay. This time the sucker punch came from a little PC on the network with the unassuming name “BURGER_WAGON”.
“So, Ram, what can you tell me about Burger Wagon?”
“Um, that’s a food truck that comes by about 3 times a week. They set up near the cafeteria.”
“So would it be reasonable to assume that a PC named BURGER_WAGON would be theirs?”
“They left a PC plugged into our network?”
“It’s online right now.”
Ram checked his watch. “How about we go get some lunch now, Dean?”
“That’s a great idea, Ram.” We grabbed our jackets and headed out to the cafeteria. If we were lucky, we’d grab an unauthorized device before we grabbed something to eat.
We went up to the Burger Wagon table. I said, “I’m Dean Webb from $VENDOR and this is Ram Gopal, security architect here at $COMPANY. We’d like to ask you a few questions, if that would be all right.”
The Burger Wagon lady said, “Sure, I don’t mind.”
“We noticed that there was a PC named BURGER_WAGON connected to the network. Would you know anything about that?”
“Oh sure, I leave that here so it’s easier to set up when we come in.”
“Uh-huh. And this PC, what is it used for.”
The Burger Wagon lady answered like what she was saying was no big deal. “We process credit card payments on it.”
Poor Ram nearly buckled at the knees with that statement.
The Burger Wagon lady asked, “What’s wrong with Ram, there?”
“He just found out that his cafeteria network is subject to PCI-DSS regulations, that’s what’s wrong.”
“What does that mean?”
If only the cafeteria staff knew what that meant, they wouldn’t have let BURGER_WAGON connect to the LAN. Lecturing the uninformed user wasn’t going to make my job any easier, so I laid it out plain and simple, without judging. “It means that we have to treat this place like a bank processing credit cards. It’s a sensitive environment, with your PC plugged in like that.”
“Oh! I’m sorry! I didn’t know!”
“We’re not angry ma’am. We just want to get the word out so that we can get things on the straight and level around here. If there’s another way for you to connect to the Internet that doesn’t involve using this network, I’d advise you to do so. We are going to start blocking access to devices like these in the very near future. We don’t want to stop you from doing business, just to stop doing it in a way that fails to comply with corporate regulations here.”
The Burger Wagon lady understood and switched over to a guest wireless connection, then and there. She fired up a VPN and Ram got the starch back in his legs. And, you know, we went back to slogging through the unsecured devices on that network after lunch, but we had an upbeat feeling about it. There was a big mountain to climb, but at least there were good people like his network team and that Burger Wagon lady that wanted to do the right thing. That didn’t just make our job easier. It made our job doable.