Most “technology people”, whatever they do, seem to be the de facto technology support for everyone they know. In my own case, back in the days when colour graphics was still a pretty neat idea (my first computer was monochrome...), I came to the conclusion that desktop support (such as assembling my own machine) was not something I really wanted to do, but that doesn’t really matter when someone in your family has an issue, or buys a new device.
Most of our mobile devices are iOS, which makes life somewhat easier, but it’s still a struggle sometimes, when questions come about features I have not used before. On the computer side, it’s worse. Not only do we have Mac OS, Windows, Linux, and Raspbian, but we have all of the additional tools required for remote work. For example, my work laptop and phone are managed centrally, which makes things much easier in some ways, but also means that I am restricted in what I can do on those devices.
And, in recent months, I acquired my Ubiquiti network gear (https://www.til-technology.com/my-blog/categories/ubiquiti), my Raspberry Pi (https://www.til-technology.com/my-blog/categories/pi), and my Synology NAS (https://www.til-technology.com/my-blog/categories/synology), which has certainly made life more interesting, and brings us to the topic of today’s story.
On Christmas Day, no one wants to be on-call for tech support, which is why I groaned (inwardly) when someone said: “I can’t connect to the Internet”.
Uh oh, I thought. That can’t be good.
Then I realized that I was currently online at that very moment.
Hm, I thought. That’s odd. (That sentiment may be humanity’s greatest inspiration for the pursuit of knowledge.)
We were both running iOS, so that didn’t seem to be the issue.
Ah! I generally connect using a VPN (Virtual Private Network http://en.wikipedia.org/wiki/Virtual_private_network), while our “affected user” does not.
So, first hypothesis is that the issue has something to do with the VPN. Or rather, by something that the VPN bypasses, like the Pi-hole.
I can easily test this hypothesis by using a machine which uses the Pi-hole. Sure enough, no connection. Now, what?
Next hypothesis is that either the Pi-hole or the Raspberry Pi computer are down. So, try to check the Pi-hole dashboard – no connection.
My hypothesis appears to be correct. The Pi-hole is down, so let’s try connecting to the Raspberry Pi. That worked, so I started checking the status of the Pi-hole so I could find out why it was...
... up and running fine?
Right, then. Network problems? Could be, but I would have expected difficulty connecting to the Raspberry Pi. Oh! Maybe an issue with the Pi-hole logs I recently moved over to the NAS. Let’s have a look at... Hm. The NAS directory mapping isn’t showing.
Ok, then. Let’s have a look at the NAS – start logging in while I glance at the unit beside my desk, and – wait.
No blinky lights. Now, that’s not good, but wait. No non-blinky lights either.
Ok, need to have a closer look at... the plug that has somehow come loose from the socket...
WTF?!
Whatever - Plug NAS back in, press button, everything starts, blinky lights start blinking, connectivity is back within about a minute. Affected user is now happy, and tranquility has been restored. Problem solved!
This was, admittedly, a pretty simple issue to work through, but it seemed like a good illustration of how valuable it is to think about an issue before running around in a dozen directions, and not to jump to conclusions. (Instead, think of them as hypotheses to be tested.)
An interesting illustration of the need for discipline in the larger world of InfoSec is in the so-called NotPetya attack which was conducted against Ukraine in 2017. (https://en.wikipedia.org/wiki/2017_cyberattacks_on_Ukraine). This attack was fairly widely, and incorrectly, believed to have been delivered via email, without sufficient evidence being available for that conclusion. Cisco Talos has made a number of references to the issue of attribution, and how important it is to ensure that you are accurate, rather than just fast. Listen to their Beers with Talos (https://www.til-technology.com/my-playlist) podcast, and articles in other sources (https://www.scmagazine.com/home/security-news/ransomware/cisco-talos-notpetya-analysis-attacker-could-launch-again/).
Needless to say, the stakes here were a bit lower, but why waste time when a moment’s thought is so much more useful?
Also, remember that you don’t necessarily need ALL the answers in order to solve a problem, so you need to figure out which answers are necessary, and which answers are simply desired. That’s actually a significant issue in InfoSec, where some will spend vast resources trying to chase down attribution of an attack, even when the information is of limited value to them. (In general, unless the answer is going to lead you to make different decisions – ie, that it is “actionable” – it’s importance is usually relatively low from the perspective of business risk management.)
This is quite a bit harder than it seems, since our monkey-brains want answers, and want a story to explain all the loose ends. In the case of NotPetya, the critical questions at the time were around what was going on, how the malware was being distributed, and how it could be stopped. The “who” was less urgent for most of the companies attacked, though of extreme interest to governments and certain other parties. The whole question of the process and value of attributing attacks is a major area of study, covering technical InfoSec, technology policy, politics, international law, and more. Fascinating stuff!
In our small example, the critical issue was to find out what had happened, and how to fix it. Done and dusted.
Still, how the <<expletive deleted>> did that plug come loose?!
Cheers!
Comments