Scaling Laws

Definition

Scaling laws are empirical power-law relationships between model scale (parameters, data, compute) and performance (loss).

In the Literature

Chinchilla Law

$L(N,D)=A+\frac{B}{N^{0.34}}+\frac{C}{D^{0.28}}$

Where:

• $L$ = cross-entropy loss
• $N$ = parameters
• $D$ = training tokens

General Form (Ganguli et al.)

• Compute: $L\propto C^{-{\alpha }_C}$
• Data: $L\propto D^{-{\alpha }_D}$
• Parameters: $L\propto N^{-{\alpha }_N}$

Michaud’s Quantization Explanation

If quanta frequencies follow Zipf’s law $p_k\propto k^{-(\alpha +1)}$:

• Parameter scaling: $L(N)-L_{\infty }\propto N^{-\alpha }$
• Data scaling: $L(D)-L_{\infty }\propto D^{-\alpha /(\alpha +1)}$

Arora & Goyal’s Key Insight

10× scaling ≈ 2× increase in composable skill count

When loss drops from $\delta$ to $\delta /k^{\prime }$, performance on $k^{\prime }$-tuples equals previous single-skill performance.

In This Project

Scaling laws constrain our framework:

• Determine which skills in ${\mathcal{S}}_{\le n}$ are “realized” at given scale
• Set the emergence boundary ${\lambda }^{\ast }(T,ϵ)$
• Explain why ontological-expansion is scale-dependent

Related Concepts

• emergence — What scaling laws predict
• skills — Units whose competence scales
• ontological-expansion — Scale-dependent ontology growth

References

• Literature Overview and Meta-Analysis