lomahony/pythia-70m-helpful-dpo

Pythia-70m finetuned using original DPO code with the helpful subset of Anthropic-hh-rlhf dataset for 1 epoch.

Checkpoints are also uploaded.

Fully reproducible finetuning code is available on GitHub

wandb log

See Pythia-70m for model details (paper).

See further details of these models in the paper Attributing Mode Collapse in the Fine-Tuning of Large Language Models.

You can cite these models if they are helpful as follows:

@inproceedings{o2024attributing,
  title={Attributing Mode Collapse in the Fine-Tuning of Large Language Models},
  author={O’Mahony, Laura and Grinsztajn, Leo and Schoelkopf, Hailey and Biderman, Stella},
  booktitle={ICLR 2024, Mathematical and Empirical Understanding of Foundation Models (ME-FoMo) workshop},
  year={2024}
}

hf (pretrained=lomahony/pythia-70m-helpful-dpo), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 16

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
arc_challenge	1	none	0	acc	0.1724	±	0.0110
		none	0	acc_norm	0.2201	±	0.0121
arc_easy	1	none	0	acc	0.3350	±	0.0097
		none	0	acc_norm	0.3380	±	0.0097
boolq	2	none	0	acc	0.4315	±	0.0087
hellaswag	1	none	0	acc	0.2614	±	0.0044
		none	0	acc_norm	0.2665	±	0.0044
lambada_openai	1	none	0	perplexity	5951.7544	±	428.5435
		none	0	acc	0.0309	±	0.0024
openbookqa	1	none	0	acc	0.1460	±	0.0158
		none	0	acc_norm	0.2440	±	0.0192
piqa	1	none	0	acc	0.5550	±	0.0116
		none	0	acc_norm	0.5501	±	0.0116
sciq	1	none	0	acc	0.4010	±	0.0155
		none	0	acc_norm	0.5070	±	0.0158
wikitext	2	none	0	word_perplexity	547.6920	±	N/A
		none	0	byte_perplexity	3.2518	±	N/A
		none	0	bits_per_byte	1.7012	±	N/A
winogrande	1	none	0	acc	0.4822	±	0.0140

hf (pretrained=lomahony/pythia-70m-helpful-dpo), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
arc_challenge	1	none	5	acc	0.1886	±	0.0114
		none	5	acc_norm	0.2338	±	0.0124
arc_easy	1	none	5	acc	0.3346	±	0.0097
		none	5	acc_norm	0.3308	±	0.0097
boolq	2	none	5	acc	0.4028	±	0.0086
hellaswag	1	none	5	acc	0.2617	±	0.0044
		none	5	acc_norm	0.2648	±	0.0044
lambada_openai	1	none	5	perplexity	22676.7987	±	1626.4435
		none	5	acc	0.0173	±	0.0018
openbookqa	1	none	5	acc	0.1640	±	0.0166
		none	5	acc_norm	0.2460	±	0.0193
piqa	1	none	5	acc	0.5528	±	0.0116
		none	5	acc_norm	0.5462	±	0.0116
sciq	1	none	5	acc	0.3100	±	0.0146
		none	5	acc_norm	0.4220	±	0.0156
wikitext	2	none	5	word_perplexity	547.6920	±	N/A
		none	5	byte_perplexity	3.2518	±	N/A
		none	5	bits_per_byte	1.7012	±	N/A
winogrande	1	none	5	acc	0.5201	±	0.0140