Teach a Small Model to Reason with RL (GRPO on GSM8K)
An AI model trained with reinforcement learning to solve grade-school math problems, explain its reasoning, and produce correctly formatted numerical answers. Problem solved: Small language models can solve math problems but often produce i...
Read full project details
An AI model trained with reinforcement learning to solve grade-school math problems, explain its reasoning, and produce correctly formatted numerical answers.
Problem solved: Small language models can solve math problems but often produce inconsistent reasoning, malformed answers, or incorrect final numbers. Improving their reliability requires more than asking them to think step by step; the model needs to learn from feedback on its own attempts.
Solution: Built a reproducible reinforcement-learning pipeline using Group Relative Policy Optimization (GRPO), inspired by DeepSeek-R1. Fine-tuned Qwen2.5-1.5B-Instruct on the GSM8K math dataset using Unsloth, TRL, vLLM, and LoRA, with custom rewards for answer correctness, structured reasoning, and valid integer outputs. Trained for 500 steps on a free Kaggle T4 GPU and published the resulting adapter with an interactive demo.
Evaluation: Created a deterministic benchmark using 300 held-out GSM8K test questions, comparing the original model against the GRPO-trained model under greedy decoding. Measured strict answer accuracy, flexible accuracy, output-format compliance, integer compliance, and response length. Also tracked reward progression, training stability, and output quality throughout training.
Result: With the same reasoning prompt, strict answer accuracy improved from 27.00% to 36.33%, an absolute gain of 9.33 percentage points (34.5% relative improvement). Format compliance remained at 99.33%, while integer compliance increased from 42.33% to 83.00%. The experiment demonstrates how reinforcement learning can improve a small model's mathematical reasoning and answer formatting within a limited GPU budget.




















