Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
Citation
Authors: Anirudh Didolkar et al. Year: 2024 Venue: URL:
Abstract
This paper explores whether LLMs possess metacognitive knowledge (knowledge about their own reasoning processes) and whether this can be leveraged to improve performance through skill-based in-context learning.
Summary
Develops a prompt-guided procedure to elicit LLM-identified skill labels, create a skill exemplar repository, and use skill-based in-context learning for improved performance.
Key Contributions
- Evidence that LLMs have metacognitive knowledge about their skills
- Two-stage skill discovery method (fine-grained → coarse clustering)
- Skill Exemplar Repository for in-context learning
- Cross-model skill transfer (GPT-4 skills improve weaker models)
Core Concepts & Definitions
Metacognitive Knowledge
The learner’s accumulated knowledge about their own cognitive processes and learning-relevant properties of data.
Skill Exemplar Repository
where is a skill label, is a question-answer pair.
Two-Stage Skill Discovery
- Stage 1: LLM assigns fine-grained skill labels (~5000 for MATH dataset)
- Stage 2: LLM performs semantic clustering → coarse skill families (~117 for MATH)
Main Results
- Skill-based ICL exemplar selection improves accuracy on GSM8K and MATH
- Skills discovered by strong LLMs (GPT-4) improve weaker LLMs
- Skill exemplar repository transfers across datasets
Relevance to Project
Medium-High — Practical skill extraction methodology:
- Four-word underscore-separated skill format (e.g., “circle_properties_area_calculation”)
- Two-stage discovery could populate our
- Metacognitive framing relates to our metaskill concept
- Repository structure relevant for our fitness function ground truth
Questions & Notes
- Can we use their extraction method to bootstrap our skill ontology?
- How do their ~117 skill families map to our algebraic primitives?
- Their skill labels are domain-specific (math) — how to generalize?