Threat Hunting vs Threat Mining

There’s Gold in Those Endpoints: Threat Mining vs Threat Hunting

Joe Moles

Share this Project

In my last post I talked about what threat hunting is and is not. Between that and our recent webinar on threat hunting, I’ve gotten a lot of questions and wanted to follow up with a deeper dive into how Red Canary analysts use threat hunting to find threats on behalf of our customers.

My team eats, sleeps, and breathes process telemetry data. We spend our days digging into what is happening on our customers’ endpoints. I want to emphasize the term “digging” because the way we work through data and raise alerts to our customers is more akin to mining than it is to hunting. You could say we perform threat mining as a service (because, hey, if the security industry needs anything, it is a new buzzword or marketing term).

I’m going to  walk through:

  • How Red Canary uses threat hunting and finds threats on behalf of our customers
  • How we continually evolve our detection capabilities
  • Why the service we provide is more appropriately considered “threat mining” than threat hunting

To hear more from Joe Moles and learn a framework for measuring the effectiveness of your security programs and tools, register for our upcoming webinar:

What’s the Value of Your Security Program? 

Start With the Hunt (but Don’t Stop There)

As I discussed in Threat Hunting Is Not a Magical Unicorn, threat hunting is a technique that teams should use to improve their detection. At the same time, threat hunting should not be the only (or even the primary) technique used to identify threats. “Threat hunting as a service” is not scalable if the service’s hunting effort is always manual. In order to be efficient, there needs to be some level of automation. This is where I begin delineating threat hunting versus threat mining.

Our SOC team uses threat hunting to help us build and test new behavior signatures, which we internally refer to as detectors. Each detector looks for a specific behavior/attacker technique, indicator set, or metadata concept.

Some examples would be:

  • Behavior: A Windows shell making a network connection to an external host
  • Indicator set: Network connection to a domain from the Farsight Newly Observed Domain feed
  • Metadata concept: A process where the executing binary has never been observed in any Red Canary customer environment

We primarily use threat hunting to develop new behavioral detectors using the following process:

Red Canary’s Approach to Threat Hunting:Red Canary Threat Hunting Process

Threat Mining: the Next Step in Driving Efficiency

Here’s where this gets more interesting: we intentionally expect false positives from our detectors.

With threat hunting, going through a hunt and not finding something is okay. On the other hand, most threat hunting processes do not have a way to handle false positives. In a pure threat hunting model, the hunter either has to maintain documentation or have an intimate knowledge of the environment to deal with false positives. This is often the barrier to fully automating the logic or hypothesis of the hunt.

With threat mining, we expect false positives to be raised regularly. In true mining terms, these would be called “tailings.” One of the challenges with mining operations is how to deal with the tailings. Red Canary built a process for this. By planning for false positives, we are able to automate the findings of a hunt. We build automation via our detectors, each representing something potentially interesting to the analyst.

Every event raised is not gold, but sometimes you get interesting events that we would not have found otherwise. The pure tailings (false positives) can be “tuned” on the fly by suppressing the event based on a variety of criteria. This suppression reduces the number of processes the analyst has to review in the future without negatively impacting the ability to identify future threats.

Striking Gold

By using this combination of hunting, automation, humans, and continuous feedback into the platform, we are able to consume significantly larger volumes of data than a typical SOC. I’ll also go on record saying that while we use threat hunting as part of our service offering, that is only a small piece of what we do. A better description is that we mine our customers’ endpoints for anomalous and malicious activity. On the plus side, it’s one of the more environmentally friendly mining operations as our tailings do not have any negative impact on the planet (other than a little CPU burn).

Hear More from Joe Moles

Join us for an upcoming webinar to learn the framework that Joe and other security leaders use to measure and report effectiveness across their organizations.

What’s the Value of Your Security Program?

Live Webinar | 6.20.17 | 1pm ET

REGISTER