Finance / Crypto exchange
Coinbase: Quality testing of web screens
- Company
- Coinbase
- Country
- United States
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Read the full source text
The work problem
According to a Coinbase engineering blog post, Coinbase believes that building the financial infrastructure of tomorrow for billions of users requires a stable and robust product experience, and that trust is built by upholding the highest bar on quality. The post explained that the company has many traditional end-to-end integration tests, but these tests are prone to flakiness, fail after minor layout changes, and take hours to debug. According to the post, some tests done by human testers took a week, and the company set a goal of increasing its testing effort tenfold at one tenth of the cost.
Technology and data
According to the same post, qa-ai-agent, which came out of this goal, starts a test from a natural language request alone, such as 'log into a Coinbase test account in Brazil and buy 10 BRL worth of BTC'. Without an intermediate step that turns text into code, the agent uses visual and textual information from coinbase.com to decide its next action, and uses the reasoning ability of an LLM to judge the results. The post explained that the open source LLM browser agent browser-use controls the browser session, gRPC endpoints and WebSocket connections start test runs, MongoDB stores test executions, session history and issue tracking, and BrowserStack provides remote browser testing across different environments. According to the post, issues found by the agent go through an LLM as a judge step in which another LLM looks at the screenshot and issue description, judges whether the issue is genuine or a false positive and produces a confidence score, and low confidence issues are filtered out. Test runs are integrated into the developer workflow including Slack and JIRA, and new automated tests are added by describing them in natural language instead of code. The company measures correctness as the share of issues found by the agent that development teams accept as valid.
Results
According to the same post, Coinbase compared human testers and the agent under identical conditions using an A/B testing approach like its feature launch process. Accuracy was 75% for the agent and 80% for human testers, and the agent detected 300% more bugs in the same timeframe. The post said that a new test can be added within 15 minutes if the prompt has already been tested, or in about 1.5 hours if prompt testing is needed, and that its token cost analysis showed an 86% reduction compared with traditional manual testing costs. According to the post, the agent runs 40 test scenarios covering localization, UI and UX, compliance and functional areas, and finds 10 issues a week on average. The post said that based on two months of test results, the company is deprecating manual tests that AI can fully replace, and expects AI agents to eventually replace at least 75% of current manual testing.
Limits and open questions
The post said that human testers still have an advantage in areas that are hard to automate, such as the user onboarding flow that requires a real ID and liveness tests like selfie verification. The 75% replacement of manual testing is the company's expectation, and the post does not give the number of test runs or the period used for the accuracy and cost comparison. This review could not confirm from official sources the operating status after the October 2025 post.
Sources
- How We are Improving Product Quality at Coinbase with AI agentscoinbase.com, Accessed
Compiled from public sources. These are not results from ATF Works customers.