Automation for AppSec at AWS (Part 2)

In Part 1, I described how the SHINE team at AWS evolved from deterministic automations to CDK Blueprint Property Injection to the Agentic Security Engineer. I ended on a line that deserves more unpacking: context is everything. That was true, but I glossed over how hard it was to actually assemble that context, and how much of our engineering time went into solving that problem before ASE could do anything useful.

This post is about that context assembly problem, and the data pipeline we built to solve it before ASE could do anything useful.

The Discovery Problem, Revisited

The SHINE Discovery Service was our first serious attempt at automated context gathering. The idea was simple: enumerate all the cloud resources associated with a service team’s organizational unit, feed those into our automations, and skip the manual kickoff meeting where engineers would ask “so, what exactly are you building?”

It half-worked. Some orgs used a single large group to host all their software, which meant we’d pull in resources from fifteen different services when we only needed two. Engineers spent time cleaning up the list before any real work could happen. Other orgs had separate groups for permissioning and coding. We’d query the permissioning org, miss the actual software resources entirely, and the review would proceed on incomplete data. The automation ran, the checks passed, and nobody noticed what wasn’t there.

That second failure mode is the dangerous one. Too many resources creates noise. Too few resources creates false confidence. A security review that runs cleanly on an incomplete picture isn’t a clean review. It’s a blind spot with a checkmark.

The discovery problem taught us something I now treat as a first principle: context completeness isn’t a quality-of-life improvement, it’s a precondition for correctness. An AI security tool operating on partial context isn’t just slower or less polished than one with full context. It’s wrong in ways that are harder to detect, because it doesn’t know what it doesn’t know.

What Context Actually Means at Scale

When an experienced AppSec engineer sits down for a kickoff meeting with a service team, the resource list is the least interesting thing they come away with. What they’re actually collecting is richer than a resource list, and it maps directly to the sources ASE had to pull from:

The model itself was not the hard part. Building the pipeline that could assemble each of those answers from authoritative sources was.

The Sources We Built Against

AWS builds internal tooling where commercial alternatives either can’t integrate with internal infrastructure or can’t operate at the required scale. Security engagement tracking, cloud resource discovery, internal code hosting: each of these had been built years before ASE existed, each solving a problem that predated AI-assisted security review by years. That accumulation worked in our favor. The context we needed wasn’t absent from AWS; it lived in multiple different and disparate internal systems that had never been wired together for this purpose. Connecting them into a coherent pipeline, with validation, freshness checks, and graceful degradation when any one source was unavailable, was where most of the ASE engineering time went.

Talos & Titan were AWS’s internal security engagement tracking systems. Talos was the web application and service layer: the interface through which security engineers managed reviews, tracked findings, and logged decisions. Titan was the data layer underneath it: the store of engagement history, resource inventories, prior findings, and resolution status for each service. Without these, ASE would ask questions that had been answered two reviews ago, and miss the context that a finding from eighteen months back was related to the current change, or that a team had been granted an exception for a specific control.

What Talos and Titan provided wasn’t just data. It was memory: the thing that makes the difference between a reviewer who’s worked with a team for years and one who showed up fresh. Every security review is easier to do well if you know whether this team has a history of IAM overpermissioning, or whether they’ve been carefully maintaining their network isolation and you can trust their assertions about it.

Veritas was AWS’s cloud resource discovery service, essentially a real-time inventory of every resource provisioned across the organization’s accounts. When a builder submitted a review and said “we have three Lambda functions and an RDS instance,” Veritas let us verify that, and find the S3 bucket and the SQS queue they forgot to mention. Not because builders were hiding things; developers don’t think about their infrastructure the way security engineers do. Veritas was ground truth against the cloud control plane, and it disagreed with self-reported resource lists more often than anyone on the team expected when we first built the integration.

GitFarm was AWS’s internal git hosting platform, analogous to GitHub but deployed entirely within the corporate network. It gave us the code. Not a description of what the builder thought the code did, but the actual repository, the infrastructure-as-code, the deployment pipelines. This matters because the most interesting security issues are almost never in the builder’s self-description. They’re in the gap between what the builder thinks is true and what the code actually does.

Cloning large repositories added latency. We had to be selective: clone what’s relevant to the scope in review, not the entire monorepo. We built scoping logic to identify service boundaries within a repo, which itself required reading build configuration, another layer of context assembly before the actual context assembly could begin. We added caching keyed to commit SHAs so we weren’t re-cloning for every review of the same codebase, which required hooks into GitFarm’s event system to know when to invalidate.

Design Inspector was an internal tool for creating structured architecture diagrams: builders drag and drop components, mark trust boundaries, and tag data elements as PII or payment-adjacent or internal-only. Those diagrams fed into Amazon Security Reviews (ASR), a system that used them alongside survey responses to automatically answer security questions. About 18% of review questions could be answered this way without any engineer involvement.

For ASE, the diagrams served a different purpose. They gave the agent a machine-readable architecture topology it could reason about directly: trust boundaries expressed as graph relationships, not prose. The difference between “this is a microservice that handles payments” and a structured graph of which services invoke which endpoints under which trust levels is the difference between an AI that can plausibly answer questions and one that can actually follow a data flow.

Themis was AWS’s internal service registry: a central catalog that mapped service identifiers to their Coral and Smithy API models, owning teams, security history, and resource inventory. This is what connected the identity of a service to everything else we knew about it across all our internal systems. Themis was the index. The other sources were the content.

Figure 1: The five context sources, assembled by the Context Assembler before ASE begins its analysis.

The Pipeline, Not the Model

There’s a related design principle worth naming explicitly before getting into the mechanics: ASE was deterministic-first. Before any LLM reasoning happened, a layer of specialized tools collected raw facts. Titan data, Veritas resource lists, GitFarm code, CloudFormation type inventories, existing findings from the ticketing system: all of this landed in structured form before an agent touched it. The AI agents never reasoned from scratch; they reasoned from data that had already been collected, validated, and organized.

This matters because most AI security tools reach for LLM reasoning early, then struggle with hallucinations and gaps because the model is filling in what it doesn’t know rather than analyzing facts it has. Making the data collection layer primary and the AI layer interpretive is exactly what the context pipeline enabled. The tools get facts. The agents decide what those facts mean.

If I had to name the single thing that consumed the most engineering time building ASE, it wouldn’t be prompt engineering or model selection. It would be the work of connecting these five data sources into a consistent, validated, sufficiently-fresh context package.

Each source had its own reliability profile. Titan was authoritative but sometimes stale; if a team closed a finding three weeks ago and the sync hadn’t run, ASE would treat it as open. We had to decide, per data element, what staleness threshold we could tolerate before the review had to wait for a refresh or proceed with explicit uncertainty flagged to the agent.

Veritas was real-time but couldn’t always resolve resource ownership. A Lambda function technically owned by service team A but deployed into service team B’s account due to legacy org structure: Veritas would find it, but attributing it correctly required joining against Titan’s engagement history. Two sources, neither wrong on its own, disagreeing in a way that required a third source to resolve.

GitFarm was complete but slow for large repos. Cache invalidation required event hooks. And for monorepos with hundreds of services, determining which code was actually in scope for a given review required its own analysis pass.

Design Inspector & ASR data was only present if the team had done the architecture diagram work. For teams that hadn’t, we had to degrade gracefully: skip the structured threat modeling layer, note the gap explicitly in the context package, and tell the agent it was working without that signal. This distinction mattered: an agent that knows it has incomplete context can hedge its conclusions appropriately. An agent that doesn’t know treats its incomplete picture as complete.

The pipeline had to manage all of this: source prioritization, conflict resolution, freshness thresholds per data type, and graceful degradation that was honest about what was missing. That last property, explicit incompleteness over silent truncation, is the one I’d emphasize most to anyone building something similar.

The lesson for evaluating AI security tools: if a vendor leads with model capabilities, ask about context assembly. Ask how they handle partial data. Ask what happens when a resource exists in production but isn’t in the IaC. Ask what the agent does when two sources disagree. Those questions reveal whether the tool is built on a real pipeline or on the assumption that input is clean and complete.

Threat Models on Submission

The context pipeline needed a trigger. The first production use case that exercised all five sources together was automatic threat model generation on security review submission.

When a development team submitted a new review through Talos, that event kicked off ASE. Without any additional input from the team, ASE pulled their engagement history from Titan, enumerated their live resources from Veritas, cloned the relevant code from GitFarm, retrieved architecture diagrams and data element tags from Design Inspector and ASR where available, and pulled their service profile from Themis. It assembled all of that into a structured threat model and delivered it to the assigned security engineer before the kickoff meeting.

The engineer didn’t arrive at the kickoff asking what the team was building. They arrived with a document that named the entry points, traced the data flows, identified the trust boundaries, and surfaced open findings from prior reviews. The meeting’s purpose shifted from context gathering to context validation.

That shift was the concrete payoff of the pipeline. A manually written threat model for a non-trivial service takes an experienced engineer several hours and requires a working meeting with the builder just to get the facts. ASE’s version was available within minutes of submission, built from authoritative sources rather than from the team’s self-description. For teams that hadn’t completed Design Inspector diagrams, the threat model was explicit about what it was missing rather than silently omitting it. The output wasn’t identical to what an engineer would write from scratch, but it was sufficient to make every review start from a higher baseline, regardless of how many reviews were queued that week.

Credit Where It’s Due

The context pipeline described above was built by multiple teams across SHINE, over years. The SHINE Discovery Service, Titan integrations, Veritas connectors, and ASR pipeline represent work that predates ASE and was built by engineers who believed the plumbing was as important as the AI layer.

Ammar Al-Kahfah designed and built the ASE architecture that consumed all of this context: the multi-agent orchestration, the Lambda tool layer, the HITL escalation paths, the observability. Describing a context pipeline is easy. Building the AI system that uses it correctly, at scale, with the right fallbacks and the right failure modes, is the harder problem. That was Ammar’s work.

The Same Problem, Different Sources

AWS had Veritas, Titan, Themis, Design Inspector, and GitFarm. Most development teams have GitHub, a SAST scanner, a dependency manifest, and a cloud config file. The sources are different, but the problem is structurally identical: before your AI security tool can reason well, you have to solve context assembly.

At Pixee, we've been working on the same problem for teams building on standard tooling rather than internal platforms. The context problem is harder in some ways (less centralized infrastructure, more heterogeneous tooling) and more tractable in others, because external teams work with more standardized interfaces (GitHub APIs, standard CI formats, public cloud SDKs).

One approach applies the same principle as the ASE submission trigger: build the threat model context before a review starts, not during it. Rather than assembling context on demand, the system generates and maintains a threat model for each repository so that when a security review begins, the current picture of the codebase is already there. The goal is the same as what Talos and Titan provided for ASE: a reviewer who arrives with context rather than one who spends the review gathering it.

The sources are different. At AWS, context lived in internal systems built over years. For most development teams, it lives in GitHub, CI pipelines, dependency manifests, and cloud configuration. The interfaces are more standardized, which makes some parts of context assembly easier. The absence of a centralized internal platform makes other parts harder: there’s no equivalent of Veritas to enumerate cloud resources authoritatively, no Themis to anchor a service identity across systems.

The design principle doesn’t change based on the sources. Context has to be assembled, validated, kept current, and made available before the AI layer needs it. An AI security tool that reasons from incomplete or stale context isn’t slower than one that doesn’t; it’s wrong in ways that are harder to detect. That was true at AWS. It’s true outside it.

Context isn’t a feature. It’s the foundation everything else runs on.