AlienBench: A Benchmark of Structural Novelty in Language Model Outputs
- Published
- Source
- arXiv
- Paper number
- 1445
- Field
- LLMs / NLP
- arXiv ID
- 2611.09094
Key points
- AlienBench adapts the structured imagination task to language models, labeling ten biological features as Earth-typical (0) or a departure (1) and summing them into a 0-to-10 score.
- A panel of three LLM judges agreed closely with Krippendorff’s α = 0.913, and each model generated 180 descriptions, 30 under each of six prompt conditions.
- Overall scores across ten models ranged from 3.28 (GPT-6 Luna) to 6.55 (Ministral 3 14B 2512).
- Open-weight models averaged 5.40, higher than the proprietary average of 4.58, and the top-scoring model was open-weight.
- Under the baseline prompt, all models departed from Earth defaults more often than 100 human participants on every feature.
Paper links
External research summaries. These are not HDATF publications or measured product results.