I remember the first remote management and monitoring (RMM) solution ever, the venerable and wonderful “ping”. We would use it all the time to see if a remote host was up and responding. And then, one day, someone wrote a program for Windows, Whatsup, and the world was changed forever. With that program, we admins could enter multiple IP addresses and that tool would ping them all day and night! It could even be set up to generate alerts.

We thought we had it made until someone asked, “Hey, I know I can ping the SQL server, but is it responding on TCP 1433?” At that point, we knew both that we needed more in our app and that there would be other admins, with other network ports, who would make similar requests. And so began the development of RMM tools.

At small companies, RMM may very well be not much more than a shareware ping/telnet suite that checks for hosts being up and responding on critical ports. It may involve learning multiple suites of RMM tools, roughly in conjunction with the trial period for one tool ending and a download for the new tool being complete. Most of what goes on is just monitoring, not management (does that mean they consume R_M products?), as there are few enough systems to manage where ssh and RDP sessions to the several devices that need management are sufficient.

Once we get to a medium company with multiple sites, that SSH/RDP solution for everything simply fails to scale. It’s time to lay some money out and actually pay for an RMM solution that will track those uptimes as well as do some kind of configuration management. Everyone makes demands of that config management solution – will it do rollbacks? Will it do point-in-time recovery? Will it track changes made outside the product? Will it enforce certain configuration parameters? Will it integrate with the helpdesk ticketing system?

The answer to all of those questions is either “no” or “yes, at an additional cost.” Nobody rides the RMM train for free.

And it’s not like that RMM will magically never make mistakes. We’re still in a garbage in, garbage out world. More than once, I was working on a project to integrate our routers and switches with a tool by pushing code to them with the RMM solution… only to have that code get overwritten because a different team pushed a change with an outdated template. So what’s the policy and procedure for undoing a change that was done in error? I found that part out the hard way as I waited for the next change window to get my changes put back into the environment.

I’ve seen RMM tools that can’t push version-specific code. Well, they can, but they don’t keep track of versions, so it’s a guess or a logic problem to figure out which devices are on which version. One solution I came up with was to push one line of code to all devices, knowing that it would fail for devices on the older version. The next push checked the config to see if that line I previously pushed was in the config. If so, skip the device. If not, then push a line of code compatible with the older versions. Would I have preferred that the tool have the intelligence to do a version check and then push the appropriate line of code, all in one go? Yes. Yes, I would. The biggest irony to me in this particular case was that the RMM tool was made by the vendor of the devices that the tool couldn’t track the version on. Very disappointing…

And then there’s RMM at the large corporation. Thousands of switches and routers, some on very dodgy Internet connections, all of them being monitored. This means the poor sap with the on-call phone is constantly answering when the NOC calls in to say that the Dakar site is down. Or the Guadalajara site. Or the Noida site. Or the Ho Chih Minh City site. Or the Chengdu site. Or the Narvik site. Or the Deadhorse site. And the NOC guy reads out the entire device name and IP address, letter and number by letter and number, so one has to sit and wait through it all before saying, “Acknowledged. Please open a ticket with the ISP.” I can’t remember a happier day than when the policy was finally re-done so that the NOC would just open the blasted ticket on their own without requiring acknowledgement from engineering.

Still, we were blessed in that we had nearly every switch under management. This did have one side effect, however… we wouldn’t believe a switch existed if it wasn’t in the RMM tool until we saw it listed as a neighbor on another switch and pinged it. That’s when we discovered that some switches couldn’t be brought into our RMM tool because they didn’t support the SNMPv2. Or because nobody could remember the password to get local access and nobody had the nerve to take it to ROMMON mode to break into it. Or because the local support contract kept that gear out of our global tools.

Those problems were relatively straightforward compared to getting gear from specialty vendors into the RMM tool. Not all of them had the same implementation when it came to reporting, even things as simple as disk space and CPU usage. For disk space, does the vendor report total available space, across all volumes, or will it send an alert when one particular volume hits 95% capacity? Will it report overall CPU utilization or will it fire an alert when one of 16 CPUs goes over 90%? The answer is, of course, “It depends.” That means that alerts from some vendors actually aren’t alerts, they’re more like transient conditions of no great importance. It also means that some vendor gear could be in an alert state, but it doesn’t actually report it as such, given how it implements a particular SNMP MIB.

At all companies, there’s the issue with keeping the tools up-to-date. The day that the tool is launched for general use is such a bright, shining moment in the history of the progress of humanity, with all the devices that need monitoring in that tool, right where they should be. Within a very short time – overnight, in some cases – the information in it is obsolete. New devices aren’t added and decommissioned devices are showing red because nothing is reporting back at that IP address… and then they go green again when that IP is re-used, but we just haven’t realized yet that it’s a security camera now, not a loopback address.

Finally, there’s the issue of access. Even at the small company, not everyone who wants to know if a system is up will have access to the RMM dashboard. At larger and larger companies, access to that dashboard can get limited to the point where even the network engineers can’t look at it… or the tool is so cumbersome, there’s severe mental pain involved in getting information out of it.

And that’s why, even at a massively huge global megacorporation, I still got plenty of use out of running a shareware app that would ping a list of devices, so I’d know if they were up… it wasn’t an official tool with management and headcount assigned to it. It just ran on my desktop and running it meant I wouldn’t have to open a service ticket to ask someone if they could check to see if the RMM had a green dot by my device or not.

“But you said you wouldn’t glamorize the security profession!” I hear some of you thinking. How do I hear you thinking? Let me tell you about the sensors in my company’s product… But seriously, I can’t really hear you thinking and I’m not really glamorizing the security biz. That being said, it’s very much like the US space program, once you take the program in its totality.

Start with the executive sponsor speech after some big events have made headlines. Stuff just happened and we have to take this matter seriously. We don’t do this because it’s easy, we do it because it’s hard. Let’s get a budget together, a project office, and some staff that are willing to make “risk” their middle names.

Everyone has an eye on the pilot programs, but not everyone understands the science behind the project. In fact, probably the only people who fully understand the complexity of the work are those directly connected to it in design and implementation groups. Management is pretty much there to make sure things get done and that they get numbers to prove that things got done.

When a major milestone is reached – that first site comes online! – everyone is ready to send congratulations and have a little party. But after that, interest wanes. People begin to question if we’ve gotten enough out of the project and if money wouldn’t be better spent elsewhere. If there should be a failure, there’s a big chance that the project budget gets cut or the whole thing is paused for a year or more while everyone takes a step back to figure things out. The project could even get shelved at that point.

What keeps the project from getting cut or canceled entirely? Information, my friends. Information. If the project can consistently produce streams of actionable information, it can stay alive. If upper management comes to depend on that information, then the project will become an institution, more or less. It will be operationalized and staff will be put in place for daily tasks and routine maintenance and changes. It will never have as much excitement as that first site coming online, but it will still keep chugging along and will be useful.

Some staff may talk about scaling the project out to truly massive scales. Budget-minded officials will be the first to throw cold water on those dreams. People familiar with the limits of the technology being used will also diminish excitement for the project, as they question if it really will scale out like that. Voices calling for tighter integration with existing systems will win in budget discussions because what was once risky is now a sure thing, and it’s safe to play things conservatively. That’s especially true when budgets and staff are big.

You stare at a screen all day, solve some tricky problems, engineer solutions, pray to God nothing goes wrong, hope the budget doesn’t get cut, and nobody really knows who you are or what you do. Are you in Mission Control or the Security Team?

Forget any analogies dealing with pitched battles. Security professionals are not generals, foot soldiers, commanders, admirals, missile base commanders, gunfighters, or X-wing squadron leaders. Thinking that we are such things puts us in the wrong frame of mind, where we expect a conventional conflict. Even if such a conflict is edged in trickery or clever deception, it’s simply not how things work in information security. We’re more in a world of trickery and clever deception, sometimes edged with conventional conflict, if anything.

If we want comparisons to professions, we need to look at spies, pest exterminators, librarians, cattle ranchers, and forest rangers. These are people who manipulate knowledge, guard assets, and who deal with hidden threats. If you still want military metaphors, I’ll allow people clearing minefields, sentries, codebreakers and intelligence analysts (although those are technically spies), and military police. Let’s get rid of the glamour and focus on the dirty work, OK?

There are two major reasons to come up with the right metaphors and examples for cybersecurity. One is so that we get ourselves into good habits of mind for dealing with threats. Two is so that we can use real-world explanations to help people outside of the profession understand that we don’t simply identify all the PCs running “Hacker.exe” and then blow them up.

I’ll even dare to say that much of our profession has a connection to organizations that make us all uncomfortable. While I don’t want the NSA to harvest all of my data, I’m perfectly ready to recommend massive data harvesting to organizations wanting to improve security. While I’d hate for my wife and kids to spy on me, I’m always advocating that we set up as many sensors and data collectors as possible in a customer environment, even getting PCs to report on each other.

In other words, you know you’re a security professional when you read 1984 to get ideas about doing your job better.

Now, not everything in this series will go dark like that. Then again, dark is what we all deal with, so don’t be surprised to find metaphors in that region. They may not necessarily be the metaphors you want to share to explain the profession to others, but they could very well be the metaphors that unlock the habits of mind you need to improve your focus.

When a security pro gets different vendor solutions to work with each other, it’s a cause for celebration. Unfortunately, most security stories seem like they’re written by George R.R. Martin and they don’t resolve to “happily ever after” conditions. Yes, things can run well for a while, even a good long while, but there comes a day for many a partnership where the parties involved part ways and their products no longer play well with each other.

This isn’t just something in an update breaking a functionality. That gets fixed with a call to tech support and developers writing a hotfix. This is the kind of breakup that gets announced on page 23 of a vendor website or which is mentioned quietly by a sales account manager that can’t renew licensing on an integration package. The vendors, for strategic or other reasons, are no longer on speaking terms.

Vendor A releases a product that competes directly with vendor B.

In this scenario, vendor A launches its new product and has a clear choice: adopt our product or do without the integration. This move is possible only if A has a big market share. It doesn’t have to be a dominating share, just a big one. It doesn’t even have to be in the security area – maybe A was eyeing a way it could get into security, and saw this as its market entry opportunity.

At a small company, they’re all ears if A’s solution is cheaper to implement than B. If that cost reduction is achieved by discounts over both the old A product and A’s competing product, so be it. Cheaper is cheaper. If the competing product from A delivers most of what they get from B, then the small company can learn to live without the features from B that they no longer will get.

If A’s solution isn’t cheaper, then the small company will learn to live without the direct integration. Maybe some whiz writes a PowerShell script that produces a cool CSV or something to help bring data together, but such whizzes are rare to find at small companies. And if they’re found at small companies, chances are they’re producing code to improve profitability.

Alternately, if there’s a vendor C that does integrate with B – and is cheaper than A – then maybe it’s time to drop A altogether.

At the medium-sized company, it’s more likely that they’ll do a bake-off between the competing products and use features in combination with pricing as determinants about which product they go with. It’s less likely that they’d drop one or the other entirely all at once, but when the products come up for lifecycle renewal, they can make a switch at that time.

For the large company, it may come down to a question of how big A is. If A is truly huge, then it’s bye-bye B and hello A if the company IT leadership wants to standardize on A. If the leadership, however, is wary of A’s size, then it keeps B and A is a non-starter. These are decisions that come down to executive strategy and have little to do with price or features. Not to say that price and features will be mentioned in conversations about keeping or switching, but the underlying rationale will be the large company’s overall relationship with big vendor A.

So why wouldn’t A compete with B if A didn’t have a big market share? It would be because A doesn’t just integrate with B. A integrates with lots of other vendors and, because it can’t control the market, bills itself as being comfortable in multi-vendor environments.

And if A has a miniscule market share, competing with B is what is commonly known as a “mistake” and will result in A going out of business or withdrawing its competing product.

Vendor A terminates an exclusive partnership with B, is now working directly with C

This scenario assumes a tight integration between A and B, more so than what is normally offered in an exposed API or a SQL transaction query. Maybe the two companies were drawing closer to each other, with a merger likely, but things changed and now A is with C, not B. This can happen regardless of A’s market share – provided that C is at least as big as B if A is itself small.

In this scenario, pricing is not likely to be a factor. C will likely cost about as much as B, once the per-endpoint licenses are tallied up. This will come down to a question of features and whether or not A+C is, overall, better than A running side by side with B. If yes, then B will be on its way out to make way for C. The only companies keeping B will be the ones that didn’t do any testing and that won’t talk to sales teams.

If no, then the executives at A will have some hard pondering to do when they lose revenue on their software that integrates with B, and there being lack of sales for integration with C to make up for it. How could something like this come to be? Easy. People lie to executives, especially so to executives that want to be lied to. If A’s leadership is surrounded by mediocre sycophants, A will make some huge blunders.

Vendor A cuts integration with B because support costs exceed revenue

No hard feelings in this scenario. There just simply aren’t enough people using B to justify the support costs of keeping the connector between A and B up and running.

At the small company, it just means lower overall cost to drop renewal on that product. Since there’s no other product that does B’s job that integrates with A, there’s no compelling story arising out of this scenario to justify replacing any product… unless there’s a cheaper product that does A’s job that integrates with B… Absent that, the company learns that integration is a fleeting thing and may well make a decision to not integrate other products because they don’t want to get burned again.

The medium company may make the same choices, perhaps choosing to have all security systems pump information into a data lake and then try and make sense of things. There’s a good chance that the lake will always be there, but few will swim in it.

At the large company, an interesting mathematical problem emerges: would subsidizing support with a custom agreement be cheaper than living without the integration? If yes, then while the rest of the world lives without the connection, the large company will keep it going… and going… and going… and going… to the point at where, ten or twenty years down the line, some new person is shocked to see that software still running somewhere! Think it can’t happen? Just ask Microsoft how many Windows 3.11 support contracts they still have with major customers…

Mandatory Security Training: the crux is the circular logic of the “mandatory” part. It has to be mandatory so that we all do it, but because it had to be made mandatory, we all know that we’re going to hate it. The fact that it’s security training doesn’t really impact the whole “mandatory” thing. If I get into pottery and start watching YouTube videos on how to wedge clay, I’m happy to watch those videos because I want to know more about something that makes me excited. Force me into a pottery class, however, and I’m playing the video through on double speed with the sound muted so that you have a record that I completed the video.

And that’s what most people do with anything mandatory. Game the system, find a weak spot, then exploit the weak spot to reduce the overall drudgery and/or misery of the experience. I spent 16 years in a classroom, so I know all about avoiding the mandatory stuff, both as a purveyor of the mandatory and as a victim of the mandatory.

I have the worst news for the biggest companies: the majority of your training is no more than ticking a box, I’m afraid. Smaller companies can have the best success, provided they have the right person doing the training.

Why do smaller companies have the best shot at success? It’s due to both their size and constrained budget. If their IT person is a patient soul, there will be lots of personal interaction on all kinds of topics, security included. One of my best experiences with training came from something that happened while I was at lunch. I locked my PC and walked out to get something to eat. While I was out, a co-worker reached out to me with an issue regarding her sound card. My out of office status came up on her screen, and that solved her problem. My status? It was a line from The IT Crowd:

Her next two responses were, “I’ll try that” and “Thanks that worked”.

When I got back from lunch, I realized that my OOO had taken care of an incident ticket. Really, it was my co-worker who had trained herself that took care of the ticket. I went back to talk with her about the experience and why power cycling actually did resolve most issues, and the rest of her group listened in. By the end of the day, the rest of the company was talking about it – and when they called in, they always prefaced it with how they had turned it off and on again and the problem still happened.

My call volume dropped off by a massive amount and the staff were ready for more insights on how to use their tech better. I would say something like, “never click on an attachment you didn’t ask for” and whoever heard it would help spread the message. When I showed up to work at PCs that needed attention, I did my best to include at least one security topic in the conversations that happened as we traversed the vast expanses of time required to update vendor software packages.

We were all working for the same company and we all had a vested interest in the survival of the company, so we were interested in knowing how to protect and better utilize its resources. Nobody made security training mandatory. We just all happened to be interested in it at the same time.

When I left that small company and started working at Global Megacorporation, I could still have moments like that with my immediate co-workers and people I worked with directly on issues, but there were too many departments and too many physical sites for me to be able to reach everyone. So, the question is, can we get personalized training for everybody at a big company? Does it scale out well?

The answer, sadly, is no. Even if local education was part of every IT job description, there simply aren’t staff at every location. Added to that is how most of those big corporations also have outsourced IT – and these are people who, at the end of the day, don’t work for the company that’s using their services. They may be friendly and supportive and all that, but they simply won’t have the same attachment to their customers’ firms that the customers themselves could possibly have.

On top of that, it’s a huge, impersonal company, right? There are going to be a lot of people that work there who simply just. don’t. care. They plan to show up, do as little work as possible for as much pay as possible, and then go home. It’s not the entire company, by any means, but there are enough of them to where training has to be made mandatory if it’s going to get done at all.

This crowd of just-don’t-cares will then do everything they can to avoid or ignore the training. If there are no click blocks, they will finish that 37-slide deck in 37 seconds or less. If there are click blocks, then they’ll click, watch a cat video on YouTube, and then turn back to click again. Put a test at the end, they’ll circulate a list of answers. There are psychometric tricks and tips to utilize to minimize those numbers, but we won’t eliminate them.

And then, one fine day, one of these guys trying to do as little work as possible clicks on the wrong link, and the company gets a malware outbreak to go along with that cat video. Every security professional knows it only takes one misstep, and we simply can’t stop all the just-don’t-cares that are bound and determined to make those missteps all along the way.

Now I need to look at the mid-sized companies and this is one case where they’re either more like a small or large company, depending on size. They are transitioning from that small, informal group where everybody mostly cares into a larger, less caring mass. What can be done?

My answer may not be budget-friendly up front, but it saves costs down the road. Keep those trainings personal. Use classrooms, if you have to. Make it where the training is a conversation, where peers that pay attention will follow up with the just-don’t-cares that snoozed through it all and make them to where they care, or at least hear the lesson.

When people do things as a group, they will praise and encourage those that uphold their common values and pressure those that don’t so that they conform. That’s human nature, and it’s what has a better chance of working than an unhuman, automated, mandatory watch-and-click training. The biggest reason is that it involves repetition over time through conversations, and that simply doesn’t happen with an experience that is unshared, through a screen. If the training must be uniform and automated, then have it delivered in a group format. Have local teams watch the training together, discuss it, and then go on to the inevitable test that exists at the end. But it’s that discussion afterward that is going to make that training part of their work lives and not just a tick in a box on an audit.

Plan B is Plan A, with an element of panic. – John Clarke

Multinational corporations have sites around the world – that’s how they get to be mulitnationals, after all. These multinationals have to link up their sites around the world. Internet lines are cheaper than MPLS circuits, so how about setting up VPNs on local Internet lines for secure communications? Costs are cut, people are happier, and the VPNs keep things secure.

But then, an event happens in one of those nations that makes the leaders of that nation decide they are going to decrypt all traffic or, failing that, block encrypted traffic outbound or inbound. They set up rules on the routers that handle their nation’s connections to the global Internet and that’s that. Now that low-cost VPN simply will not come back up because the maintenance traffic required to set it up and keep it going is being blocked. After all, the terrorists / rebels / armed opposition / coup leaders / coup victims / journalists / other assorted enemies of the state use VPNs to get their information, and it’s not like there’s a special protocol for business-only VPNs.

And if there was one such, it would also be blocked, just in case an enemy of the state worked at a place with a B2B VPN.

So, the VPN is down. What are your options?

1. Plain text transmissions. OK, this is a joke, really. I mean, yes, technically, it is an option, but hardly a realistic one. Let’s look at the others.

2. Data transit via mail or courier. Erm, all right… but that’s going to be slow, and there’s no guarantee that it won’t be intercepted at the border and opened up there. At least it would only be a few border guards and any industries connected to the state security apparatus that sees that information instead of the whole world… but, my, is it ever slow. And costly.

3. Provision an MPLS circuit. Well, this is fast and secure, once it’s set up. But provisioning one of these takes time and planning. How much more time and more planning during a time of national emergency, I can only imagine…

Looks like that’s about it. This is not a case where engineers pull out reference materials and troubleshoot or rebuild things to solve the problem. This is a technical problem emergent from a political reality and, hold on… I have another option…

4. Political appeal. This might be the fastest, cheapest, and best solution. Have a contact person with the national government work out some sort of arrangement. Now, if this is a government that is willing to cut off all privacy in order to haul in enemies of the state, there may be some sort of content filtering and alerting required for your network to get that VPN back. Or, in other words, the government may well require that it be notified if any of your employees are doing things that would get them on the list of enemies of the state. Citizen employees will be arrested and foreign employees will be deported, so this option goes with some very strict reviews of what’s on that very recently updated acceptable use policy.

Every business has got assets that can’t be properly secured on their own or assets that need more security than what the rest of the network gets. By placing those assets behind a firewall, an access control list (ACL), or in another VLAN, we can make them more secure. If you want to say that last sentence in one word, you go with “segmentation”. If you want two words, “network segmentation.”

In the small company, the number of devices that need segmentation can be small. They may still be a high percentage of the devices on the network, but it’s still a small number. It’s a small company, after all. If the devices aren’t mobile, segmentation can be as easy as permitting only traffic on ports specified by the vendor and then blocking everything else with the obligatory “deny any any all” rule at the bottom of the ACL. If you have a large number of devices with similar needs, place them in the same VLAN and then apply a single ACL to the whole VLAN. Simple, yes?

Well, yes and no. Often, the network gear in place in a small business is limited in what it can do. If the owner couldn’t afford a switch that can support multiple VLANs or ACLs – or the operating system version that unlocks those features – then you can’t do that easy implementation. The good news is that you might not need to upgrade the entire environment, maybe just one switch that can handle all the sensitive connections and then leave everything else on the lighter-grade switches.

In the medium company, maybe there’s an actual datacenter where all the cool servers go. If so, great! Put a firewall between that datacenter and the rest of the network and start putting rules on it that limit the inbound and outbound traffic. If there’s a business case to have unlimited access, then do so only through a single host, and then log all the activity on that host.

As for the other devices, you’ll be looking at VLANs for the most part to provide that segmentation. The reason comes down to memory available on switching devices for ACL storage. On many devices, it’s limited to where we would not want to have an ACL on every single port. That memory gets exhausted and the switch goes down. Having ports gathered into VLANs and then giving that VLAN an ACL makes much better use of memory on the switches.

The problem now becomes one of defining the address scopes and the routing for those new VLANs, which can be a lot of work even for one VLAN type. Carving out a printer VLAN at 10 different sites is no mean task. What’s more, it means that the IP address management (IPAM) system needs to be kept current with comments for each VLAN type, so that people can understand what’s going on.

And once you start documenting your IPAM, you can’t stop. Or, rather, if you *do* stop, you’ll leave question marks on other network ranges and force someone later on to go back and revisit your work – and that someone might be you!

Large firms may already have VLAN hell and an IPAM with best-effort maintenance. There’s no easy way to put this: you’ll have to clean that up and get processes in place to keep it cleaned up. Network and security teams will have to join in on this effort and it would be best if people on both teams were familiar with the IPAM system being used and what different security ACLs apply to the various VLANs. That means documentation, meetings, documentation of what was said in meetings, meetings to work out issues with the documentation, and so on.

Sorry about all the work that goes with this… but if you want proper segmentation, there needs to be both proper documentation and proper maintenance of the scheme to keep it relevant over time. The small company has the lightest load for documentation, but that might be offset by the need to purchase higher-end gear. For all the other companies, you’ve got the stuff that can do the job, but you’ve got a big job to do.

There is so much more to security than:

1. Find the hackers.

2. Shut them down.

First of all, we need to know what, exactly, is on the network and what it does and whether or not it should be doing that function while connected without restriction to the rest of the network.

But before we tackle the question of what should be on the network, we need to go about discovering what is on the network, and this can be a journey full of surprises.

Typically, the start of this investigation will involve someone saying that everything that is connected to the network are phones, PCs, access points, and the printers. Oh, and also the badge readers. And the security cameras. But that’s it. Besides the barcode readers. That’s all, though. Hang on, we also have some digital signage…

At this point, you may now take what anyone has to say about what’s on the network with a grain of salt. It’s time to answer this question for yourself.

In a small company, you may be able to track down all the devices by hand during an off hour or two. It’s a great exercise and will prove invaluable for doing troubleshooting.

In a medium-sized company, this cannot be done alone. You’ll need a few other people to help out. That, or you’ll just do it all over a much longer period of time… however it’s done, you’ll likely also need some form of automation of tasks to get all that data collected in a usable way.

For a large company, this cannot be done alone. You will need tools. You will need a project manager. You will also need cross-team cooperation.

For all of these investigations, you will also need to talk to people that don’t usually talk to folks in IT. You will need to talk to them because they have connected things to the network, the likes of which you have never seen on a network ever before…

… and watch me talk about those things without once using a catch-all phrase that describes all of them…

In the small company, especially one that’s going through initial growing pains, there aren’t enough ethernet ports in wall outlets. That means, most likely, a cascade of cheap, unmanaged switches, also known as “cockroach switches” because when you see one of them, there’s a thousand more, hiding in the dark places of the office.

Because the switches are unmanaged, finding out what’s connected to them can be a chore. It can be done, but it may involve tracing cables up through holes in the ceiling and then dropping down into a room next door. You might also find the switches themselves in the ceiling space, acting as repeaters so the 100m cable from the main switch can extend into the nearby office space that was recently leased out.

In medium and large companies, even those with plenty of accessible wall ports available, people will bring in their cockroach switches and plug ’em in. Why? Maybe you should ask the developers who want to run 7 boxes in their cubicles and who don’t know they can requisition an old Cisco 3750 that’s still good, but was decommissioned last year. It could also be a boss that wants to have an extra laptop running or an app team that wants to have a concentration of monitoring devices for a war room or something similar.

But switches you can expect. I mentioned the unexpected, and I will now deliver on that promise.

Small companies have it lucky. Once the odd things are found – be they cameras, badge readers, printers, industrial devices, barcode scanners, automated fryers, refrigeration units, glucometers, or environmental controls, the person doing the discovery is not far away from the person responsible for those devices. And by “not far away”, I mean that both in the sense of both physical and organizational distance. The company is still small enough to have a familial feel to it, where everyone can walk up to everyone else.

Once you meet the person responsible for the unusual device, you’ll get a story behind it that’s likely to contain the business reason to have the device on the network. That, or a promise to get it off the network if it’s causing a problem.

In the medium-sized company, it’s a longer walk to have that kind of chat, and maybe you also have to go through a manager in order to have permission to engage in that kind of talk. You might even have difficulty finding out who you need to talk to about the serial-to-ethernet devices, the USB-to-ethernet devices, and the parallel-to-ethernet devices.

Also, you’re now more at risk to find ancient history still connected to your network. The small companies also tend to be *new* companies, so they tend to have new gear. Medium companies have likely been around for a while, and that means they could have devices that the company forgot about… devices that are now no longer supported by their vendors and which will need replacement in order to not be the security threat which they now constitute.

But for diversity and legacy, nothing beats the large company. The bigger it is, the crazier the scenario can get. Time for my disclaimer – I work for a company that makes a product designed to discover devices on the network and then classify them (among other things), and I have lots of large companies for my clients. With that disclaimer out of the way, I was recently at a client where, in the space of one hour, we reviewed a proof of concept deployment and found the following things on the wired network:

1. A cockroach switch.
2. A Nintendo.
3. A Windows 98SE PC. (Also connected through a cockroach switch, just for good measure.)
4. A network range used in more than one place.

Oops, forget to mention that fourth thing in my previous paragraphs. But, it’s a sad fact of physics or biology or some kind of science that, as companies grow, growing with them is the chance that some self-proclaimed techie will set up a network using an address space already operational somewhere else. The worst case of this was where a site that had a large number of guest wireless devices utilized the entire 10.0.0.0/8 range for it. We found it about a month after it was created, in the course of tracking down intermittent and unpredictable network timeouts and connection refused errors…

But I’ve seen earth movers, cows, ATMs, lightbulbs, drug pumps, silicon wafer fabs, vending machines, cash registers, information kiosks, ovens, refrigerators, pneumatic drills, scales, televisions, cars, personal health monitors, vacuum cleaners, and baby monitors all on customer networks. It’s not just that if there’s a thing, there’s both porn of it as well as an Internet-enabled version of it. It’s that those internet-enabled things will show up on your networks because either they were purchased and connected by the organization, or because people who work at that organization decided to bring them in and connect them up.

Some of those things are just fine, if they stay on guest networks. Some of those things are just fine, provided they are on segmented networks with limited or no access to the rest of the corporate networks and/or the Internet. And some of those things don’t belong anywhere on any network. The final say on which devices go where is up to the organization’s mission, values, and overall security posture.

But, before you can decide what should or should not be on the network, you need to know what *is* on the network.

The sentence is simple: get all the security solutions to work with each other. So how do different sized firms deal with that directive?

At the small company, the good news may be that there are only one or two solutions to work with. The bad news may be that they’re small business solutions that don’t have full enterprise features for integrating with anything. The bad news may also be that the IT person at that small business is either a visiting consultant or someone that handles all the IT, from the production line systems on up to ordering replacement RAM for company laptops. Basically, someone that doesn’t have 100% attention on security.

But let’s say that the small business IT person wants to do the right thing and be serious about security. She’s got an antivirus program for the PCs and a firewall for the Internet connection. She could stare at firewall logs all day long, or maybe she could spin up a syslog server. That sounds like it would be both a fun project and have a big payoff at the end of the work.

Unless she’s unfamiliar with Linux. Because that’s where the free syslog servers live. Linux is not an intuitive sort of thing, and learning it can be a difficult and frustrating experience. Chances are, if this IT person is dedicated enough to get into Linux, she may have moved on to a better opportunity by the time she knows enough to start up a Graylog server.

Now, if she’s staying with the small company out of sheer loyalty (maybe a family member or other dearly loved one is running the company), she’s got to learn how to do Greylog after that bout with Linux. Once that task is done, she can turn on logging on that firewall and create some rules in Greylog to alert her on specific rule violations or when there are multiple violations of the same rule from a single host…

… and then come back the next day to see her inbox swamped with alerts from the syslog server. Now she’s in the final phase of implementation, tuning the alert frequency. After that, she’s still faced with manually inspecting devices that are generating the most alerts because that anitvirus solution at the small firm doesn’t have any monitoring tools to go with it.

By now, she is master of the firewall, syslog, a fair amount of Linux, and how to find great deals on copier paper and toner. Not wanting to develop her copier paper ordering skills any further, it is quite likely she’s ready to rationalize away whatever loyalty she has and move on to the next opportunity.

And that’s the final obstacle for security solution integration at small companies. Quite frequently, they can’t pay enough to keep motivated, skilled professionals on the payroll. They’ll either have to deal with unmotivated IT people that really don’t care to stretch their skills or turn to a firm that will place someone onsite 2 or 3 times a week to check on how things are going there. If the previous person set up an alerting system, they’ll use it. Maybe. But they sure aren’t going to build one out. That’s work well above their pay grade.

So we follow our IT pro to a medium-sized company. Here, she’s no longer a department of one. For sure, she’s no longer dealing with renewing licensing for everybody’s softphones. She’s the security person, alongside the network person, the sysadmin, the phone guy, the 3 techs that do operations, and the wireless person. Not bad, am I right? She can specialize now, no question about it.

Well, maybe there’s a few questions about it…

For example, this medium-sized company has an AV system, an IPS here and there, a perimeter firewall and a datacenter firewall (different vendors, to boot!), a syslog server that is running at the very limits of the “free” offering from its vendor, a proxy server, and security is also in charge of the IPAM and PAM systems. There’s a good chance that our IT pro may not have heard of either IPAM or PAM and may even make the mistake of thinking they’re the same thing. But she’s on top of things and learns the difference between IP Address Management and Privileged Account Management, and all seems well, except for the fact that she has to ramp up on 6 different technologies. There won’t be any integration until that happens.

As she’s ramping up on those techs, she’s also responsible for supporting them. That means lots of explaining to users and developers why this security system or that one isn’t interfering with their application’s performance. She even posts this image in her cubicle and points to it as she sees a user or developer walk up:

(On a personal note, I’ve used that image. It has yet to prove my case to a developer out of hand, but it does help to set the tone of the discussion to encourage the dev to look for other reasons why the app isn’t working.)

While that helps with the firewall questions (see my personal note), it does nothing for the constant requests to exempt websites from the proxy filter. She’s barely got enough time to read product documentation, so when is she going to find time to integrate those solutions?

Moreover, how does she go about automating actions between the systems? It’s not like the firewall is built to take direct input from the proxy server. The syslog server seems to be the logical choice as a clearing house of information, but how can it be configured to send commands to one system or another based upon logging info that’s coming in from another source?

It’s possible that the security systems have an API that can allow commands to be sent to them. It’s also quite possible that the systems *don’t* have an API, or that the API is such that the syslog system can’t send commands to it. Even if the API is one that the syslog server can interact with, our IT pro would then have to learn how to write code. If she’s lucky, she can borrow a developer for a day or three to help with the project. If not, then she’s got a steep learning curve ahead of her if she’s never really done programming before.

But there’s also a fair chance that she won’t have to do all this alone. It’s entirely possible that the medium-sized firm has enough wherewithal to contract professional services from a vendor. If that can be done, then she can stay focused on her day-to-day work while the vendor’s pro serv person hacks out the code and does a knowledge transfer at the end of the engagement.

Now, I need to make a disclaimer here because I am part of a professional services team for a vendor. While someone could accuse me of wanting to feather my own nest, the truth is that, as a customer, I have benefited greatly from vendor professional services. They are definitely worth looking at.

The pro serv route is also available at the large company level. If we have our IT pro start a career at a large firm, she’s going to find that she can specialize more in the technologies she works with each day. This means that, while she gains a deeper knowledge of just 2 or 3 systems, she’s also no longer connected to *all* the systems. Other people on her team, possibly even other teams entirely, will handle those systems. Integration now means not just mastering the technology, but mastering the political considerations that go with cross-team projects. Will the integration mean one team or the other takes over a technology? If both teams manage the system, which managers are responsible for which functions?

One of the stickiest questions is: will we wind up stretching one product to fill a role that is actually better suited to another product? Added to that one would be: which systems does it make sense to integrate with which other systems? Both of these questions deal with lines of demarcation, where one system ends and another begins. For example, at what point does the antivirus protection end and the vulnerability scanner responsibility begin? Which has priority over web traffic, the data exfiltration protection or the proxy server?

While any integration at the small or medium sized company was done pretty much as a solo or very small group effort, the large company integration could very well be impossible without a multidisciplinary product team, with an oversight committee made up of about a dozen operational and service-line managers.

Like I said, “get all the security solutions to work with each other” is easy to say. Getting progress on that task means understanding the obstacles and then figuring out how to clear them out of the path.