Retail / Secondhand clothing platform
Vinted: Search that uses images and product attributes together
- Company
- Vinted
- Country
- Lithuania
- Adoption stage
- In operation
- Source published
- Date basis
- The date the source was published. It can differ from the date adoption started.
- How the source was checked
- Read the full source text
The work problem
Search that matches words filters narrowly. On Vinted photographs matter and several languages are mixed in, so this limit showed up more sharply. Cases arose where a relevant item existed but the search returned no results, and the company saw this as a missed business opportunity.
Technology and data
Vinted built a Two-Tower model. The query tower turns a search term into a 256-dimension vector. The item tower turns the item's various features into a vector of the same 256 dimensions. The query tower leaves a pre-trained multilingual CLIP model untouched and trains only a projection head on top. The item tower combines categorical features such as brand id and category id with the CLIP embedding of the item's main photo. The photo embedding enters at 512 dimensions and is reduced to 256. Training uses contrastive learning, making the model separate each pair from 7,000 to 10,000 randomly chosen negative items. The whole implementation sits inside a Vespa application package. Keyword search results and vector search results are merged into one list with reciprocal rank fusion.
Results
Vinted said it started by filling search sessions that return few results with vector search results. As the filtering widened slightly, search metrics improved and the team gained confidence in the approach. After about 50 AB tests it rolled vector search out fully. The company said the error rate fell below 0.02 percent after several performance optimisations. The image embeddings of the main photos take up about 1 TB of memory.
Limits and open questions
Figures for changes in business metrics such as conversion rate or revenue are not in the source. Vinted said feeding the item's text information into the model is still an unsolved task.
Sources
- Dense Retrieval | Vinted Engineeringvinted.engineering, Accessed
Compiled from public sources. These are not results from ATF Works customers.