Musings From a Penetration Tester’s Diary - Part 1

The author has been performing penetration tests (ethical hacking) around the globe since graduating from Carnegie Mellon 18 years ago. He’s “hacked” banks, factories, fire departments, airports – you name it. We asked him to share insights from his decades-long career. This is Part 1 of “Musings from Pen Tester’s Diary.”

Would you believe me if I told you that there was once a time when if you looked for penetration testing services, you’d find, maybe, five or ten pen testing firms? No, seriously. I’m talking about the late 90’s and early 2000’s here. Penetration testing companies weren’t easy to find.

So, what explains the surge of pen testing firms today then? It’s simple – currently there are many automated tools, which are very easy to use, that perform vulnerability scans and can find vulnerabilities. And a lot of pen testing firms rely heavily on these tools.

It’s not necessarily the best approach but it reminds me of a recent post on X by Daniel Vassallo1. I’ve linked it in the references, and I urge you to read what he has brilliantly written. But for the sake of our discussion, let me start you off with a short version.

SawStop: Saving Fingers & Hands

Back in 1999, Steve Gass, founder of SawStop, invented an upgrade to table saws available in the market that could detect a worker’s fingers and hands and almost immediately come to a stop. Several thousands of people’s hands and fingers (40,000 a year at that time) could be saved for a $150 upgrade. Brilliant, right? But table saw manufacturers vehemently opposed it because of the added expense and regulators didn’t care.

I’ll let you read the rest in Daniel’s post. Eventually, through a circuitous route, it turned out well for SawStop. But the point is – if an inventor creates a good thing, market forces will try to commoditize it and keep costs low, even if the solution isn’t as good as the original.

And I guess that’s what has brought the penetration testing services industry to where it is today. There are several low-cost pen testing firms that will deliver a report but not necessarily the cybersecurity insights that could have protected your organization’s “hands and fingers.”

Don’t get me wrong. I’m not from the school of thought that believes you get what you pay for. Genuinely not. But I am from the school of thought that believes there is a sweet spot somewhere between the most expensive and the least expensive penetration testing service, where you’ll be able to find the right balance between your budget and the pen testing firm’s quality.

And there are differences in quality aplenty.

For instance, one quick and easy filter you may already have figured out is to find out what percentage of the pen test firm’s work-effort during a project is manual versus automated.

So, what’s a good percentage split? That’s a good question. Another thought that may have occurred to you while reading this is – what’s the equivalent of a SawStop kind of innovation in penetration testing and are there innovative pen testing firms out there? That’s a good question, too. But how do you identify these penetration testing firms then? That’s three good questions in a row. And there are many more in a similar vein that you could ask to help you differentiate and, most importantly, find a pen testing firm that fits your specific needs perfectly.

We’ll explore these questions and more together.

The Difference Between Pen Tester Tools & Human Insight

So how do you dig deep and assess the quality of a penetration company? How do you find penetration testing firms that can help protect your organization’s “hands and fingers?”

Let’s start with the easy one. What’s a good percentage split between the manual and automated testing components? Who’s doing more work on your project – the pen tester or the scanning tool?

In our world of automation today, would you hate me if I said that at least 70% of the work needs to be manual? It’s not old school (and I’m not old!). It’s just that hacking is more skill and brain, and less automation.

A scanning tool could take you to the point where you know a vulnerability might exist. But a good penetration test needs to confirm that the vulnerability is not a false positive. And then comes the substantive part – actually trying to exploit the vulnerability and gain access to your protected information. So, if a large part of a penetration testing company’s work isn’t manual, then you’re hiring a scanning tool.

So, what’s the equivalent of a SawStop kind of innovation in penetration testing - and how do you identify pen testing firms that are innovative? Good penetration testing companies challenge the status quo and grow. The innovation itself isn’t the only thing that is important. It’s a process of continuous innovation that is.

So, if you’re trying to assess a pen testing firm, ask them – what process do they follow to innovate their penetration testing techniques and methodologies?

The Importance of an Innovative Mindset for a Pen Tester

To keep challenging the status quo, the penetration firm should be trying new things. For instance, the latest thing that I’m trying to incorporate into our methods is inspired by Sir Dave Brailsford’s marginal gains methodology. Check the linked article here – it’ll be worth your time, I promise.

But, in brief, the concept says – break down everything that you do into simple parts and improve each part by just 1%. The British cycling team had won only one gold medal in 76 years when Brailsford took over in 2002. At the 2008 Olympics, they won seven out of the ten possible in track cycling.

Sir Dave Brailsford, Image from The Guardian

Penetration testing companies that have a process to innovate … innovate. And they should be open to incorporating methods that have been proven to be successful in other domains. This process becomes engrained – a way of life. Looking back at the past 25-plus years, I realize that so many innovations in penetration testing just happened organically. We were hacking modems back then and we’re now hacking IoT devices.

So, the next time you’re evaluating a pen testing firm, have a chat with them about their process for innovation. Don’t ask them for the innovations itself. Because innovations soon become ordinary. Remember, the iPhone was the first smartphone but it’s ordinary today – a second-nature kind of product that you’d expect everyone to have. The process that produced it though will likely produce the next pièce de resistance.

I’ll be back with more musings tomorrow. Same place obviously.