Evaluation

Definition

Skill evaluation is the assessment of whether and how well a model can apply skills to tasks.

In the Literature

SKILL-MIX

• SKILL-MIX$(k)$: Generate text demonstrating $k$ skills on random topic
• Metrics:
  • Skill Fraction: proportion of skills exhibited
  • Full Marks Ratio ${\alpha }_k$: proportion achieving perfect score
• Auto-grading: GPT-4 / LLaMA-2-70B judges

Beyond Stochastic Parrot Criterion

Model surpasses memorization if: ${\alpha }_k>\frac{3}{2}\sqrt[k]{p_s^k\cdot p_t\cdot L}$

Competence (Arora & Goyal)

$\text{Competence}(s)=\mathbb{P}[\text{correct cloze}\mid \text{text adjacent to }s]$

ACD (Lu et al.)

• Automated task generation for capability discovery
• Interestingness filtering for novel evaluations
• Capability clustering and reporting

In This Project

Fitness Function

$\varphi :\mathcal{S}\times \mathcal{T}\to [0,1]$

Measures how well skill $S$ solves task $T$.

Evaluation Decomposition

$\varphi (S_1\circ S_2,T)=f(\varphi (S_1,T),\varphi (S_2,T),\text{interaction}(S_1,S_2,T))$

The interaction term captures emergent effects beyond individual skills.

Task-Relative Assessment

The set ${\mathcal{S}}_T^{ϵ}=\{S:\varphi (S,T)\ge ϵ\}$ defines which skills are competent for task $T$.

Related Concepts

• skills — What is evaluated
• composition — Composite skill evaluation
• emergence — Sudden capability appearance

References

• Literature Overview and Meta-Analysis