Travel and hospitality / Lodging marketplace
Airbnb: Migrating outdated UI test code
- Company
- Airbnb
- Country
- United States
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Read the full source text
The work problem
Airbnb's React component tests were written with Enzyme. Enzyme was built for older versions of React, and the way it reaches deep inside a component no longer matched current testing practice. The two tools are fundamentally different in structure, so they could not simply be swapped. Analysis showed that deleting the Enzyme files would leave a large hole in code coverage.
Technology and data
In 2024 Airbnb built an LLM-based migration pipeline. It treated each file like a state machine. A file had to pass the Enzyme refactor, then the Jest fix, then the lint and tsc fix before moving to the next stage. When a check failed, the error and the most recent version of the file were put into the prompt and the step was retried. For complex files the context was widened. Prompts grew from 40,000 to 100,000 tokens and pulled in up to 50 related files, including the source of the target component, the test file being migrated, the failing check output and sibling tests in the same directory. People took on the work of picking out failures and fixing the prompts and scripts, and hand-fixed the files that automation never finished.
Results
The first bulk run moved 75 percent of the target files in four hours. Over the next four days the prompts and scripts were refined and that rose to 97 percent. The remaining 3 percent took people another week to finish. In total about 3,500 Enzyme test files were migrated in six weeks. Airbnb said it was work it had estimated would take 1.5 years of engineering time by hand. It said the original intent of the tests and the code coverage were preserved.
Limits and open questions
The figure of 1.5 years by hand is Airbnb's advance estimate, not a real comparison experiment. The source gives no change in test run time or defect detection rate after the migration.
Sources
- accelerating large scale test migration with llmsmedium.com, Accessed
Compiled from public sources. These are not results from ATF Works customers.