Forget AGI: Why Leading AI Models Still Can’t Do Everyday Math
Artificial general intelligence-AGI for short-is often framed as the point where machines can match humans across a wide range of intellectual tasks. But new results from the MATHVISTA benchmark suggest that even the most advanced AI systems remain noticeably weaker than humans at something far more basic: making sense of math problems that involve real-world visuals.
A team of researchers from Microsoft Research, Sahara AI, and Emory University evaluated how well today’s top models handle mathematical reasoning grounded in images: charts, graphs, tables, diagrams, and other visual representations of quantitative information. This kind of problem is central to “general” intelligence because it mirrors how people encounter math in daily life-reading a bus schedule, comparing prices on a receipt, or interpreting a line chart in a news article.
The study put 12 leading foundation models to the test, including high-profile systems such as ChatGPT, Google’s Gemini, Anthropic’s Claude, and others. Among them, GPT-4 Vision achieved the best score, but even that success was limited: the model correctly solved 49.9% of the benchmark problems.
Humans, by comparison, averaged 60.3% on the same test. That 10-point gap may not sound enormous, but it is significant when you remember that these were not obscure graduate-level puzzles-they were tasks designed to reflect what an average person could reasonably be expected to do. The results underline that, for all their linguistic fluency and coding prowess, today’s AI tools are still inconsistent at visual quantitative reasoning.
As Microsoft principal researcher Hao Cheng explained, the aim is to build systems that can handle the kinds of tasks a typical person faces in everyday life. Reading a graph correctly, combining it with a short text description, and then drawing a logical conclusion is exactly the kind of skill that ought to be trivial for a system claiming “general” intelligence. Yet that is where models still stumble.
What MATHVISTA Actually Tests
Unlike pure math contests where everything is expressed in formulas and text, MATHVISTA focuses on problems tightly coupled to imagery. A typical question might present a bar chart showing monthly sales, a number line with highlighted intervals, or a geometry diagram with labeled points and angles. The model must “look” at the picture, interpret the visual structure, extract the relevant numbers, and then perform the necessary reasoning.
This is a harder problem than computing an equation typed in plain text. It demands that several abilities work together: visual perception, understanding of mathematical notation, contextual reading of instructions, and step-by-step logical reasoning. Humans learn to integrate those skills implicitly in school; current AI architectures still treat each piece somewhat separately.
The benchmark also avoids simple pattern-matching tricks. Tasks are phrased in varied ways, visuals can be slightly noisy or complex, and correct answers often require multiple intermediate steps. In other words, guessing based on surface correlations-a strategy that sometimes works for language-only tests-does not reliably succeed here.
Why Vision-Based Math Is So Hard for AI
Modern large language models excel at predicting text. When you ask them to solve a standard algebra problem written out in words and symbols, they can often rely on memorized templates and statistical associations. But once a chart or diagram enters the picture, that shortcut breaks down.
First, models must translate raw pixels into a structured internal representation: recognizing axes, reading labels, identifying bars or curves, understanding legends, and distinguishing between decorative elements and critical information. Misreading a single axis or misidentifying a bar can derail the entire solution.
Second, they must map what they see to the question being asked. For example, if the prompt says “How much higher is the 2023 value than the 2021 value?” the model has to identify the right bars or points on the chart, line them up with the correct years, and then compute the difference. That requires a stable sense of reference and attention, something that current vision-language architectures still find fragile.
Finally, the system must execute the math reasoning itself: adding, subtracting, comparing ratios, or applying geometric rules. While models have improved in pure symbolic math, chaining all three stages-vision, understanding, computation-without error remains an open challenge.
The Gap Between Hype and Reality
The MATHVISTA results directly cut against some of the more enthusiastic claims about AGI being just around the corner. If the best publicly known models cannot reliably interpret a middle-school-style chart question, it raises questions about how close we really are to machines that can robustly reason in open-ended, real-world environments.
Performance numbers around fifty percent also have practical implications. In a consumer setting, a tool that solves math-and-vision problems correctly only half the time is not trustworthy enough to be used unsupervised in safety-critical workflows-think financial reports, medical charts, engineering diagrams, or scientific data analysis. Human review remains mandatory.
This doesn’t mean progress has stalled. The fact that a model like GPT-4 Vision can approach 50% on a benchmark that was impossible for AI just a few years ago is itself striking. But the numbers serve as a counterweight to narratives suggesting that current AI systems already possess human-like general intelligence. They do not.
Why Human Performance Still Matters
The inclusion of human participants in the MATHVISTA benchmark is more than a curiosity; it sets a meaningful reference point. AI models often look impressive when judged in isolation, or when compared only against earlier generations of models. Measured against human baselines, the picture becomes clearer.
A 60.3% human average shows that the benchmark is genuinely challenging, not trivial. People make mistakes too. But it also shows that ordinary individuals-without access to billions of training examples or supercomputer-scale compute-still outperform the most advanced AI systems on these integrated reasoning tasks.
This matters for how we design and deploy AI. Rather than assuming that models will automatically replace humans in analytical work, results like these argue for hybrid workflows: humans leveraging AI as an assistant or calculator, but still responsible for final judgment, especially when the task depends on nuanced interpretation of visuals and numbers.
Implications for Education and Everyday Tools
The visual-math gap has direct consequences for how AI can be used in classrooms, tutoring systems, and everyday productivity tools. Many educational applications rely heavily on diagrams, graphs, and multi-step problem-solving. If an AI tutor misreads a geometry figure or misinterprets a data chart, it might confidently walk a student through the wrong reasoning.
Similarly, business and productivity applications often involve dashboards, performance charts, or spreadsheet visualizations. Today’s models can generate explanations or summaries based on textual descriptions, but asking them to directly “look at this chart and tell me what’s going on” is still risky.
Developers building such tools need to be clear about these limitations. Helpful uses might include suggesting possible interpretations or highlighting patterns in conjunction with human oversight, rather than offering definitive, unreviewed answers about quantitative visuals.
What Needs to Improve in AI Research
The MATHVISTA findings hint at several research directions that could narrow the performance gap:
1. Better integration of vision and language
Current models often treat images as add-ons to text. More tightly coupled architectures-where visual and linguistic representations are learned jointly and continuously cross-inform each other-could improve grounding and reduce errors.
2. Explicit reasoning mechanisms
Many systems still behave like sophisticated pattern matchers. Incorporating explicit reasoning tools-such as symbolic math engines, step-by-step scratchpads, or verifiable intermediate calculations-may help models handle multi-step quantitative problems more reliably.
3. Training on structured visuals
Text-heavy internet data may not be enough. Curated datasets that emphasize charts, tables, blueprints, and educational diagrams, along with their correct mathematical interpretations, could better prepare models for this class of tasks.
4. Robustness and uncertainty estimation
For sensitive use cases, models must not only get more answers right-they must also know when they are likely to be wrong. Developing mechanisms for calibrated confidence estimates on visual-math tasks could make systems safer and more predictable.
Rethinking What “AGI” Should Mean
Benchmarks like MATHVISTA force a more grounded conversation about what AGI actually entails. If we define AGI as “AI that can do what an average person does across most domains,” then passing tests of everyday visual-math reasoning is a non-negotiable requirement. Being able to write code or produce fluent essays is not enough.
This perspective shifts focus from spectacular party tricks-such as generating artwork or synthetic voices-to mundanely important skills: reading an invoice, verifying a graph in a report, or cross-checking a chart in a research paper. These are the building blocks of real-world intelligence.
It also suggests that AGI, if it comes, may not arrive as a single sudden breakthrough. Instead, we are likely to see incremental, domain-by-domain progress: better at charts one year, more reliable at geometric diagrams the next, and so on, gradually filling in capabilities that humans take for granted.
The Road Ahead: From Impressive to Dependable
Today’s top models are undeniably impressive. They can translate languages, debug software, write essays, and answer complex questions with astonishing fluency. But impressiveness is not the same as dependability. The MATHVISTA results highlight an uncomfortable truth: when it comes to the kind of integrated visual-math reasoning that underpins much of everyday decision-making, AI remains a talented amateur, not a seasoned professional.
For companies and institutions adopting AI, this means careful scoping of what tasks can be safely delegated. Repetitive text generation or simple numerical operations may be fine; interpreting a complex financial chart or medical diagram without human oversight is not.
For researchers, the message is equally clear: closing the gap between human and machine on benchmarks like MATHVISTA will require more than scaling up model size. It will demand deeper changes in how systems perceive, reason, and understand the world.
Until then, talk of AGI should be tempered with the recognition that, in one of the most basic areas of human cognition-connecting what we see with what we know about numbers-machines still have a lot of learning left to do.