Threat hunting

A threat hunt walkthrough, from hypothesis to detection

6 min readHarshaVardhan Aakanti
Cover illustration: a magnifying glass over a cloud with scattered events, one highlighted.

Threat hunting gets described in grand terms, but a good hunt is usually small and specific. You pick one question about how an attacker might be operating in your environment, you look for the evidence, and whatever you find, you leave behind something better than you started with: a new detection, a fixed logging gap, or a documented reason to stop worrying.

This post walks through one hunt from start to finish. The topic is LLMjacking, where attackers use stolen cloud credentials to get access to hosted AI models and then resell or abuse that access, leaving the victim with the bill. The details are generalised and contain nothing from any real environment, but the steps are the ones I follow on every hunt.

Step one: start with a hypothesis

A hunt without a hypothesis is just browsing logs. The hypothesis should be specific enough that you can prove or disprove it with data you actually have. Ours was:

If an attacker holds valid AWS credentials, they will enumerate AI model access and quotas before they start using models.

Where did that come from? Public research on LLMjacking describes a consistent pattern: attackers check whether model access is enabled, which models are available, whether invocation logging is turned on, and how much quota they can use. That recon is noisy compared to normal use, and it happens before the expensive part. It maps to MITRE ATT&CK T1078.004, Valid Accounts: Cloud Accounts, followed by discovery activity.

Step two: check you can see it

Before writing a single query, confirm the data exists. For this hunt that meant answering three questions:

  • Is CloudTrail enabled in every account and region, including regions you do not normally use? Attackers like quiet regions.
  • Are management events for the AI service captured? Enumeration calls such as listing foundation models are management events and show up in CloudTrail by default.
  • Is model invocation logging configured? If not, you can see that models were accessed but not what was sent. That is a finding in itself.

A hunt that discovers a logging gap has not failed. It has found the most important thing it could have found.

Step three: write the first query broadly

Start wide and narrow down. The first query pulls every call to the AI service's discovery and configuration APIs over the last 90 days, grouped by identity and source.

source = cloudtrail
eventName IN (ListFoundationModels, GetFoundationModelAvailability,
              GetModelInvocationLoggingConfiguration,
              ListFoundationModelAgreementOffers, GetUseCaseForModelAccess)
| stats count, dc(awsRegion) AS regions, values(sourceIPAddress)
        BY userIdentity.arn, userAgent
| sort - count

The syntax above is generic pseudocode. Translate it into whatever query language your SIEM uses. The idea is what matters: who is asking about model access, from where, with what tool, across how many regions.

Step four: separate normal from interesting

The first results are mostly legitimate. Your own platform team, infrastructure-as-code pipelines and internal AI projects all call these APIs. The work now is building a picture of normal so the outliers stand out. We looked for identities with any of these traits:

  • Discovery calls across many regions in a short window, which looks like someone checking where access is enabled.
  • Calls from long-lived access keys rather than roles assumed through single sign-on.
  • A user agent that does not match how the team normally works, such as a scripting library on an account that only ever uses the console.
  • Source IPs outside known corporate and cloud ranges, especially hosting providers and anonymising networks.
  • Discovery followed by quota checks or invocation attempts that fail validation, which some tools use to probe access cheaply.
  • Any of this from the root user, which should almost never happen.

A second query joined these traits with the identity's history: has this key or role ever called the AI service before? First-time use is not malicious on its own, but first-time use combined with multi-region discovery and an unusual source is worth a closer look.

Step five: investigate what turns up

In a hunt like this the outcomes usually fall into three buckets. Most hits are explained quickly: a new internal project, a developer testing from home, an automation account doing exactly what it was built to do. A few need a conversation with the owner to confirm. Occasionally one matches the pattern closely, with root or long-lived credentials, many regions, an unfamiliar source and no matching project, and becomes an incident.

When that happens, the hunt hands over to incident response. Rotate or disable the credentials, review everything else that identity did, check for new users, keys or roles created as persistence, and look at billing for unexpected model usage. Hunting finds it; response contains it.

Step six: turn the hunt into a detection

The most valuable output of a hunt is usually not the finding. It is the detection you write afterwards, so you never have to hunt for the same thing by hand again. From this hunt, the detection logic became:

  • Alert when an identity makes AI service discovery calls in three or more regions within an hour.
  • Raise severity if the identity is the root user, uses a long-lived access key, or calls from outside known ranges.
  • Raise severity again if discovery is followed by changes to model access agreements, quota requests or invocation attempts.
  • Exclude only the specific roles used by approved platform pipelines, each with an owner and a review date.

We back-tested this against the 90 days of data from the hunt before switching it on, to check it caught the patterns we cared about without flooding the queue.

Step seven: write it up

Every hunt gets a short write-up, even the ones that find nothing. Mine always cover the same things: the hypothesis, the data sources and any gaps, the queries, what was found and how it was resolved, the detections or logging changes that came out of it, and ideas for the next hunt. Over time these become a library. New analysts learn from them, and the next hunter does not start from zero.

Lessons from hunts like this

  • Hunt where the money goes. AI model access is expensive and easy to resell, so it attracts attackers. Follow the incentive.
  • Treat logging gaps as findings. Missing invocation logs or an unmonitored region is worth fixing even if the hunt finds nothing else.
  • Baseline before you judge. Most "suspicious" activity in a first query is your own engineers.
  • Always leave something behind. A detection, a fixed gap or a write-up. Otherwise the hunt was just a long afternoon.

Frequently asked questions

What is LLMjacking? LLMjacking is when attackers use stolen cloud credentials to access hosted AI models, then resell or abuse that access while the victim pays the bill.

What is a threat hunting hypothesis? It is a specific, testable statement about how an attacker might behave in your environment, such as “an attacker with valid AWS keys will enumerate AI model access before using it”. It tells you which data to look at and what would prove or disprove it.

What should a threat hunt produce? At least one of: a new or improved detection, a fixed logging gap, or a documented write-up of what was checked and why it is safe. A hunt that leaves nothing behind has to be repeated by hand next time.

The point

Threat hunting is not about being clever. It is a disciplined loop: a specific hypothesis, data you have checked, broad queries narrowed by context, honest investigation, and a detection at the end. Run that loop regularly on the attacks that matter most to your business and your detections improve every month, whether or not any single hunt finds an attacker.