Gaming / Online gaming platform
Roblox: Detecting harmful content in text, voice and images
- Company
- Roblox
- Country
- United States
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Read the full source text
The work problem
On Roblox 6.1 billion chat messages and 1.1 million hours of voice pass through in a single day, in 28 languages. Reviewing that volume within milliseconds with people alone is impossible. The company wrote that chat alone would need hundreds of thousands of people working around the clock.
Technology and data
At the base sits a large transformer model that handles several formats together. This model is distilled and quantised to match operating conditions so it runs fast. The personal information filter grew beyond what the existing CPU servers could carry as request volume rose. A GPU based serving architecture was therefore built, separating tokenisation from inference and raising throughput fourfold. The voice classifier reaches a decision within 15 seconds in eight languages, and the company released this model publicly. People handle rare cases, complex investigations and appeals. Cases where an appeal overturns the decision go back in as training material.
Results
After moving to the new architecture the personal information filter handles up to 370,000 requests per second. The company said false positives fell by 30 percent and that, thanks to this, automatically caught mentions of personal information rose by 25 percent. The text filters as a whole handle more than 750,000 requests per second. It explained that about 1 trillion pieces of content were uploaded between February and December 2024, of which about 0.01 percent were detected as policy violations.
Limits and open questions
The drop in false positives and the rise in detection are figures limited to the personal information filter. They do not mean platform-wide safety improved at the same rate.
Sources
- How Roblox Uses AI to Moderate Content on a Massive Scale | Robloxcorp.roblox.com, Accessed
Compiled from public sources. These are not results from ATF Works customers.