The Limits of Benchmarking: A Warning from Within The recent departure of Lun Wang from Google's DeepMind has reignited concerns about the evaluation methods used in AI research.
This is not just a story about one researcher leaving a job, but rather a symptom of a deeper problem within the industry.
Benchmarks and safety checks have become the de facto measures of success for AI models, but they are woefully inadequate for evaluating risks that lie beyond their current capabilities.