Retail / E-commerce
Amazon: Attack and defence testing of security detection rules
- Company
- Amazon
- Country
- United States
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Checked a source excerpt through a search tool
The work problem
Checking whether a security detection rule catches real attack variants meant people running the techniques themselves and revising the rules. That work took weeks. Techniques such as Python reverse shells were especially hard to detect, because Python is installed everywhere and commands can be hidden in many ways.
Technology and data
Amazon built a system called Autonomous Threat Analysis, or ATA. Attack role agents imitate adversary techniques while defence role agents verify detection coverage and produce new or improved rules. Each agent is a node in a graph structure, and the output of one agent becomes the input of the next. Testing runs in an environment fully separated from production and customer data, built to imitate internal code bases and production systems. Attack agents run real commands so genuine telemetry is left behind, and defence agents query the real log database to check precision and recall. Change approval before anything reaches production is done by people.
Results
Amazon said that in the Python reverse shell case the attack agents generated and ran 37 variants of the technique. It then produced 64 variants and obtained an improved detection rule, recording precision of 1.00 and recall of 1.00 against those variants and one hour of production audit data. The company explained that work which took weeks end to end fell to about four hours, a 96 percent reduction in time. It said the system runs 10 to 30 technique variants at a time and takes one to three hours to test a single rule.
Limits and open questions
The precision and recall of 1.00 were obtained against the variants ATA itself generated and one hour of audit data. The 96 percent is Amazon's own figure, with no external verification material.
Sources
- How Amazon uses AI agents to anticipate and counter cyber threats - Amazon Scienceamazon.science, Accessed
Compiled from public sources. These are not results from ATF Works customers.