Update README.md

README.md

@@ -6,63 +6,70 @@ tags:
 - generated_from_trainer
 - trl
 - grpo
-licence: license
+- r1
+- rl
+licence: qwen-research
 ---
 
-# Model Card for qwen-2.5-3b-r1-countdown
+# Model Card for `qwen-2.5-3b-r1-countdown` a mini R1 experiments
 
 This model is a fine-tuned version of [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+It has been trained using [TRL](https://github.com/huggingface/trl) and GRPO on the Countdown game.
+
+If you want to learn how to replicate this model and reproduce your own Deepseek R1 "aha" moment, check out my [blog post](https://www.philschmid.com/mini-deepseek-r1).
+
 
 ## Quick start
 
 ```python
-from transformers import pipeline
-
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="philschmid/qwen-2.5-3b-r1-countdown", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
+from vllm import LLM, SamplingParams
+from datasets import load_dataset
+from random import randint
+
+sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=512)
+
+# use revision without "checkpoints-" as vLLM downloads all of them
+llm = LLM(model="philschmid/qwen-2.5-3b-r1-countdown", revision="099c0f8cbfc522e7c3a476edfb749f576b164539")
+
+# Load dataset from Hugging Face Hub
+dataset_id = "Jiayi-Pan/Countdown-Tasks-3to4"
+dataset = load_dataset(dataset_id, split="train")
+sample = dataset[randint(0, len(dataset))]  
+
+# create conversation
+messages = [
+    {"role": "system", "content": "You are a helpful assistant. You first thinks about the reasoning process in the mind and then provides the user with the answer."},
+    {"role": "user", "content": f"Using the numbers {sample['nums']}, create an equation that equals {sample['target']}. You can use basic arithmetic operations (+, -, *, /) one or multiple times but each number can only be used once. Show your work in <think> </think> tags. And return the final equation in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>. Think step by step inside <think> tags."},
+    {"role": "assistant", "content": "Let me solve this step by step.\n<think>"}
+]
+# generate response
+res = llm.generate(llm.get_tokenizer().apply_chat_template(messages, tokenize=False, continue_final_message=True), sampling_params)
+res = "<think>" + res[0].outputs[0].text
+print(res)
+
+# <think> We need to use the numbers 37, 15, 4, and 13 with basic arithmetic operations to make 16. Let's try different combinations:
+# - 37 - 15 - 4 - 13 = 6 (too low)
+# - 37 - 15 + 4 - 13 = 13 (too low)
+# - 37 + 15 - 4 - 13 = 35 (too high)
+# - 37 - 15 + 4 + 13 = 39 (too high)
+# - 15 + 4 + 13 - 37 = -1 (too low)
+# - 37 + 15 + 4 - 13 = 43 (too high)
+# - 15 + 4 * 13 / 37 = 15 + 52 / 37 (not an integer)
+# - 15 * 4 / 37 - 37 = -28.24 (not a whole number)
+# - 4 * 13 / 15 - 37 = 41.3333 (not a whole number)
+# After all combinations, I got not any integer result as 16.
+# </think>
+# <answer> 37 - 15 + 4 + 13 </answer>
 ```
 
 ## Training procedure
 
- 
-
-
 This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
 
 ### Framework versions
 
-- TRL: 0.14.0.dev0
+- TRL: 0.14.0
 - Transformers: 4.48.1
 - Pytorch: 2.5.1+cu121
 - Datasets: 3.1.0
 - Tokenizers: 0.21.0
-
-## Citations
-
-Cite GRPO as:
-
-```bibtex
-@article{zhihong2024deepseekmath,
-    title        = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
-    author       = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
-    year         = 2024,
-    eprint       = {arXiv:2402.03300},
-}
-
-```
-
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```