Having worked with several network administrators over the years I can safely say that they are a fairly proud bunch, often viewing their systems and network operations centers (NOCs) as their babies and themselves as noble watchmen. While their cause is honorable, unfortunately their protectiveness can make them somewhat sensitive to questioning. In fact, I often find myself thinking that aspiring ambassadors or diplomats should look to hone their skills on a network admin before facing a politician or a royal – they would certainly learn some vital lessons in people management.
One network admin I worked with was particularly proud of his NOC and had dashboards filled with metrics of every type – network, firewall, traffic, applications, storage – you name it, he had it. He also delighted in sending out pre-emptive emails, indicating that one of our systems might be headed south, which we obviously appreciated, even though we would have been able to detect these system errors on our own – we had all been assigned our own logins to his monitoring application in order to manage our local networks.
One day, we noticed some suspicious gaps in his normally consistent charts. Having taken the monitoring system’s reliability for granted, this was enough to immediately send us into a panic, forcing us to double-check each and every one of our systems. Suspiciously, everything was running as usual. At a loss, we even decided to risk his wrath by asking him if his monitoring server might have gone down or failed in some way. Unsurprisingly, he didn’t take our questions well.
My Spidey-sense tingling I continued my investigation regardless. I decided to use my personal account to browse through his application monitors one by one. After a few dead-ends, I noticed that the server ‘MSTRCTL’, the heart of his NOC, was not being monitored at all. This should never have been the case, so I added a helpdesk ticket to add a monitor for his own personal server. Thirty minutes later, the helpdesk ticket was closed and the mysterious monitoring dropouts had stopped. The ticket note came back with just one line of comment: “virus removed from server, ticked closed.”
He may have given me the cold shoulder for a while following the incident, but it served as a great reminder to everyone that all systems – especially the server at the heart of your monitoring empire – should be audited and monitored for any security breaches which could consequently affect the wider network, leaving the business vulnerable to malware, viruses and would-be hackers.
While I am not looking to toot my own horn, this incident goes to show just how vital it is to have total visibility of the whole network, and the ability not to get bogged down in one area of network management. As an IT pro, I need to be able to step back and solve problems with a calm head. It seems that protecting the business from potentially dangerous and costly security breaches is just another of the average IT pro’s daily duties.