Do LLMs Really Understand Math? A Knowledge Space Theory Perspective

Imagine a student who can solve quadratic equations but fails basic arithmetic. You would not call them a math prodigy. You would suspect they memorized a formula without understanding the foundations.

Now imagine an AI that does exactly that. It solves calculus problems but stumbles on addition. It answers advanced geometry questions while failing the prerequisite concepts those questions depend on. Would you say it understands math?

This is the uncomfortable question a new paper from ETH Zurich asks. "Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning?" by Peng Cui, Heejin Do and Mrinmaya Sachan introduces a framework grounded in Knowledge Space Theory (KST) to test whether large language models organize mathematical knowledge in a structured, human-like way.

Their answer: no. Not even close.

Modern LLM evaluation mostly measures accuracy: the percentage of questions answered correctly. More recently, researchers have moved to evaluating reasoning trajectories, checking whether the steps in a chain of thought are valid.

Both approaches share a blind spot: they treat each problem in isolation. Neither can tell you whether a model's pattern of successes and failures is globally consistent with the structure of the knowledge being tested.

A model might answer a hard question correctly while failing its prerequisites. Accuracy counts that as a success. KST flags it as a structural violation, a sign that the correct answer came from memorization or pattern matching rather than genuine reasoning.

Knowledge Space Theory, introduced by Doignon and Falmagne in the 1980s, is a mathematical framework for modelling how human knowledge is structured. Its core idea is that knowledge is not a random collection of isolated facts. It is a structured space, constrained by prerequisite relations.

Concept mastery is latent and hard to measure directly, so the paper operationalizes KST at the question level. Three steps take it from a pile of math questions to something you can test a model against.

The metric is the Prerequisite Satisfaction Ratio. It is computed only on correctly answered questions: if a model gets q right, what proportion of q's prerequisites did it also get right?

def compute_psr(question, knowledge_state, prereq_map):
    """
    Compute Prerequisite Satisfaction Ratio for a correctly answered question.

    Args:
        question: the question ID
        knowledge_state: set of question IDs the learner answered correctly
        prereq_map: dict mapping question ID -> set of prerequisite question IDs

    Returns:
        PSR value between 0 and 1
    """
    prereqs = prereq_map.get(question, set())

    if not prereqs:
        return None  # no prerequisites, PSR undefined

    satisfied = prereqs & knowledge_state
    return len(satisfied) / len(prereqs)


def aggregate_psr_micro(knowledge_state, prereq_map):
    """
    Micro-averaged PSR: weighted by number of prerequisites per question.
    """
    total_satisfied = 0
    total_prereqs = 0

    for q in knowledge_state:
        prereqs = prereq_map.get(q, set())
        total_satisfied += len(prereqs & knowledge_state)
        total_prereqs += len(prereqs)

    return total_satisfied / total_prereqs if total_prereqs > 0 else 0


def aggregate_psr_macro(knowledge_state, prereq_map):
    """
    Macro-averaged PSR: simple average across questions.
    """
    scores = []
    for q in knowledge_state:
        psr = compute_psr(q, knowledge_state, prereq_map)
        if psr is not None:
            scores.append(psr)

    return sum(scores) / len(scores) if scores else 0

A runnable version, with a worked fractions example, is in this post's codes/python/ directory.

What they found

Human learners reach a PSR of 0.936–0.942 at 79.6% accuracy. The best LLM, Qwen3-80B, reaches a 0.939 micro-PSR at 92.5% accuracy, which looks like parity. It is not: only 48.16% of its correct answers achieve a perfect PSR of 1.0, against 72.7% for humans.

In human learning, prerequisite knowledge scaffolds the acquisition of advanced concepts. If LLMs internalize knowledge the same way, then putting prerequisite examples in context should help more than unrelated ones.

The paper compares five in-context strategies:

What they found

Accuracy in percent. Best strategy per model in green; the prerequisite column never wins.

def scaffolding_gain(accuracy_with_context, accuracy_without_context):
    """
    Scaffolding Gain: change in accuracy when relevant context is provided.
    """
    return accuracy_with_context - accuracy_without_context


# Example results from the paper
results = {
    "Qwen2.5-7B": {
        "no_context": 80.8,
        "random": 82.1,
        "same_skill": 82.2,
        "similarity": 82.1,
        "prerequisite": 80.2,
    },
    "Qwen2.5-32B": {
        "no_context": 88.6,
        "random": 89.3,
        "same_skill": 89.7,
        "similarity": 89.8,
        "prerequisite": 89.3,
    },
    "Qwen3-80B": {
        "no_context": 94.1,
        "random": 94.7,
        "same_skill": 95.9,
        "similarity": 96.1,
        "prerequisite": 94.6,
    },
}

for model, scores in results.items():
    sg = scaffolding_gain(scores["prerequisite"], scores["no_context"])
    print(f"{model}: Prerequisite SG = {sg:+.2f}")

Prerequisite context does not consistently beat the other strategies. Same-skill and semantically similar examples are the strongest. For Qwen2.5-7B, prerequisite examples actually hurt: the paper reports an SG of −0.56 (the rounded table figures above give −0.60).

If LLMs have a coherent knowledge structure, even one different from humans', then stronger models should know most of what weaker models know, plus more. That is what the cumulative, hierarchical nature of mathematics predicts.

def knowledge_overlap_coefficient(state_a, state_b):
    """
    KOC: overlap between two knowledge states, normalized by the smaller state.
    """
    intersection = len(state_a & state_b)
    smaller = min(len(state_a), len(state_b))
    return intersection / smaller if smaller > 0 else 0


def normalized_koc(state_a, state_b, total_questions):
    """
    Normalized KOC: corrects for chance overlap.
    """
    koc = knowledge_overlap_coefficient(state_a, state_b)
    p_max = max(len(state_a), len(state_b)) / total_questions

    if p_max == 1:
        return 1.0

    return (koc - p_max) / (1 - p_max)

What they found

• Humans nest. Low, medium and high ability groups show near-perfect subsumption: stronger learners know what weaker learners know, plus more.
• LLMs scatter. Overlap between models is substantially lower and more fragmented.
• Stronger drifts further. Paradoxically, stronger LLMs diverge further from human knowledge patterns, not less.
• Even peers disagree. The two closed-source models, GPT-4.1-mini and Claude, overlap with each other at only 0.38.

Across all three normative behaviours, the results tell one story.

Accuracy is not understanding

An LLM can solve a calculus problem while failing arithmetic. It can answer a geometry question while missing its prerequisites. Those are the marks of pattern matching, not reasoning.

Current evaluation misses it

The paper shows that neither accuracy nor LLM-as-judge reasoning scores (relevance, coherence, correctness of steps) surface this incoherence. Each metric is looking at a different thing.

LLM knowledge is flat, not hierarchical

Human knowledge grows in a nested, cumulative way: foundations first, then what builds on them. LLM knowledge looks more like a flat collection of memorized patterns. Stronger models do not simply know more of the same foundations. They often know a different, less human-aligned set of things.

The authors are careful about the scope of their claims.

• It needs an expert-defined concept graph. Mathematics has one; domains with less explicit dependencies may not.
• Only mathematics was tested. Whether the same holds for science, programming or general knowledge is open.
• Questions stand in for concepts. True KST models latent concept mastery. Question correctness is a practical proxy, but an imperfect one.

The paper makes a simple but profound point: getting the right answer is not the same as knowing why.

Current LLMs achieve impressive accuracy on math benchmarks, but their knowledge is structurally incoherent. They fail prerequisites, gain nothing from prerequisite scaffolding, and show no hierarchical knowledge growth. Their knowledge is fragmented, not structured.

That does not make LLMs useless. It makes our evaluation methods incomplete. We need structure-aware assessment alongside accuracy and reasoning scores to understand what these models actually know, and what they only appear to know.

The next time an LLM solves a hard math problem, ask whether it really understood the foundations, or just got lucky with a pattern.

1. Cui, P., Do, H., & Sachan, M. (2026). Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory. arXiv:2609.05245.
2. Doignon, J.-P., & Falmagne, J.-C. (2012). Knowledge Spaces. Springer.
3. Liu, Z., et al. (2023). XES3G5M: A Knowledge Tracing Benchmark Dataset. NeurIPS.
4. Lightman, H., et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
5. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS.