OpenAI

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

A new benchmark called 'Soohak,' developed by 64 mathematicians, challenges the most advanced LLMs with 439 complex math problems. Initial results show even top models like Gemini-3-Pro and GPT-5 scoring low, highlighting a major gap in their ability to handle research-level mathematical reasoning and, crucially, to recognize and refuse to answer inappropriate questions.