mdonigian/fineweb-edu-complexity-classifier

Reasoning Complexity Classifier

A ModernBERT-base model fine-tuned to predict the reasoning complexity of educational text on a continuous 1–4 scale. Trained on FineWeb-Edu documents labeled by GPT-5-nano via the OpenAI Batch API (~$20 in credits).

Model Description

This is a regression model (num_labels=1, problem_type="regression") that outputs a continuous score. The score can be rounded to the nearest integer to obtain a discrete complexity level. Level 5 (Formal/Abstract reasoning) was excluded from training due to data scarcity; the model's effective range is 1.0–4.0.

Complexity Levels

Level	Name	Description	Example
1	Factual/Declarative	States facts with no reasoning	"The Pacific Ocean covers ~165 million km²."
2	Single-step reasoning	One inference or comparison	"Because boiling point decreases at altitude, water boils faster in Denver than Miami."
3	Multi-step reasoning	2–4 chained logical steps	"Demand rose while supply held fixed → prices rose → consumer spending fell → GDP slowed."
4	Complex reasoning	5+ steps, conditionals, competing factors	Medical differential diagnosis with branching conditions and exclusion criteria.

Training Details

Data

• Source: FineWeb-Edu — a curated subset of Common Crawl filtered for educational content.
• Labeling: ~100,000 documents reservoir-sampled from ~6,000 records per subject category, then labeled with GPT-5-nano via the OpenAI Batch API using structured output (integer 1–5).
• Splits: 80% train / 10% validation / 10% test (stratified by integer complexity level).
• Preprocessing: Texts truncated to 8,000 characters before labeling; tokenized to 512 tokens during training with dynamic padding.
• Level 5 exclusion: Rows labeled as level 5 were excluded from the training set.

Hyperparameters

Parameter	Value
Base model	answerdotai/ModernBERT-base
Epochs	3
Batch size	32
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.1
Max token length	512
Optimizer	AdamW
Scheduler	Linear with warmup
AMP	bf16 (CUDA)
Loss	MSE

Training History

Epoch	Train Loss	Val MAE	Val Acc (rounded)	Val Spearman r
1	0.6002	0.5190	56.98%	0.7533
2	0.3631	0.5040	58.43%	0.7597
3	0.2040	0.5114	58.19%	0.7485

The best checkpoint (by validation MAE) was saved at epoch 2.

Evaluation Results

Evaluated on a held-out test set:

Metric	Value
MSE	0.4388
MAE	0.5063
Rounded accuracy	58.6%
Spearman r	0.7527

Interpretation: The model achieves a Spearman correlation of ~0.75 with gold labels, indicating strong ordinal ranking ability. The MAE of ~0.51 means predictions are on average within half a level of the true score when treated as a continuous signal.

Output Interpretation

Raw score	Meaning
~1.0	Factual/Declarative
~2.0	Single-step reasoning
~3.0	Multi-step reasoning
~4.0	Complex reasoning

Clip and round the raw float output to [1, 4] for a discrete level.

Architecture

Based on answerdotai/ModernBERT-base:

• Layers: 22 transformer layers (alternating full and sliding attention)
• Hidden size: 768
• Attention heads: 12
• Intermediate size: 1,152
• Max position embeddings: 8,192
• Classifier pooling: mean
• Classifier activation: GELU

Limitations

• Labels are silver-standard (GPT-5-nano), not human-annotated; label noise may affect the ~1.5% of ambiguous texts.
• Texts are truncated to 512 tokens; very long documents are judged on their first ~512 tokens only.
• Trained primarily on English educational web text; performance may degrade on other domains or languages.

Intended Use

Designed for data curation pipelines that need to filter or balance training corpora by reasoning complexity — for example, constructing curriculum-ordered datasets for language model training.