XYBench: Helping with the problem behind the question

Introduction

AI assistants are increasingly relied upon for troubleshooting everyday tasks, from coding bugs to kitchen mishaps. Yet users often phrase their queries around a misconception—they ask for a solution to a perceived problem (Y) while the real goal (X) lies elsewhere. This classic "XY problem" can lead assistants to waste effort on the wrong approach. XYBench addresses this gap by offering a large‑scale benchmark that evaluates whether models recognize and steer users toward the underlying goal, rather than merely answering the literal request.

Building XYBench

XYBench compiles 8,115 real‑world questions across programming, cooking, home repair, and other procedural domains. The dataset draws from four sources:

1. 1,557 Stack Overflow questions
2. 184 Stack Exchange posts
3. 102 manually curated examples
4. 6,272 WikiHow‑derived scenarios generated with a prior LLM and refined for clarity.

For each entry, annotators identify:

• Stated request (Y): the explicit question the user asks.
• Intended request (X): the underlying goal inferred from context.

Human reference answers are collected from the highest‑scored responses (or the accepted answer) to establish the ideal focus on X.

How XYBench Evaluates Answers

The benchmark measures three complementary dimensions of a model’s response:

1. Pragmatic solution – Does the answer contain a concrete solution for Y, for X, or both?
2. Pragmatic emphasis – Which request does the model prioritize? (Y‑focused, X‑focused, or balanced.)
3. Misconception identification – Does the model explicitly steer the user away from the mistaken approach and toward the true goal?

Claude Sonnet 4.5 serves as an automated judge, classifying suggestions within each response. Human reviewers validated a subset of 500 judgments, achieving 97‑100% agreement on solution presence, 85‑93% on misconception identification, and 74‑80% on emphasis.

Key Findings

When evaluated on a single‑turn basis, five leading models—Claude Sonnet 4.6, GPT‑5, Gemini 3.1 Pro, Qwen 3.5‑9B, and DeepSeek‑V4 Pro—showed a consistent pattern:

• Higher rates of Y‑solutions (75‑86%) than X‑solutions (59‑71%).
• Primary focus on Y in 47‑57% of answers, with only 35‑41% emphasizing the underlying goal.
• Misconception identification lagged far behind human references.

An oracle experiment, where the intended request X was supplied alongside the question, raised X‑solution rates (e.g., Sonnet 4.6 from 59% to 67%) but paradoxically reduced misconception identification, indicating that merely knowing the goal is insufficient; models must also act on it.

A complementary multiple‑choice test revealed a striking contrast: when asked to judge two candidate answers, the same models chose the goal‑focused answer 71‑84% of the time. This gap between recognizing the better answer and generating it underscores a core challenge for future model training.

Implications for Evaluation and Research

XYBench provides a concrete tool for researchers to track progress on three fronts:

• Detection – Does the model spot the hidden intent?
• Explanation – Can it articulate why the literal request may be suboptimal?
• Action – Does it prioritize a solution that addresses X?

By isolating these behaviors, developers can experiment with prompting strategies, fine‑tuning data, or architectural tweaks and see which interventions close the gap to human performance.

The benchmark currently focuses on English procedural queries. Extending XYBench to multilingual contexts, conversational dialogues, and more diverse cultural assumptions will be essential for building truly helpful assistants worldwide.

Closing Thoughts

The XY problem is a subtle but pervasive obstacle to effective AI assistance. XYBench shines a light on this blind spot, offering a rigorous, scalable way to measure and improve how models handle misguided questions. As the community adopts this benchmark, we can expect a new generation of assistants that not only answer what users ask, but also guide them toward what they actually need.