UNIPO: Unified Interactive Visual Explanation for RL Fine-Tuning Policy Optimization
Authors
Aeree Cho (Georgia Institute of Technology), Alexander D Greenhalgh (Georgia Institute of Technology), Jonathan Bodea (Johns Creek High School), ShengYun Peng (Georgia Institute of Technology), Duen Horng Chau (Georgia Tech)
Presentation
- Session
- Is the Model Even Thinking?
- Time
- Thursday, Nov 12, 08:36 – 08:45 (US/Eastern) · session 08:00 – 09:30
- Location
- Hall America south
Links
Sign in to access the preprint PDF.
Sign in- Download Supplemental Material
Keywords
Interactive Visualization, Reinforcement Learning, Fine-tuning, Large-language Models
Abstract
Reinforcement learning has emerged as a dominant technique for fine-tuning the behavior of large language models, with policy optimization (PO) algorithms such as GRPO, DAPO, and Dr. GRPO emerging in rapid succession to advance state-of-the-art reasoning and alignment performance. However, the modular differences between these algorithms, including targeted improvements to clipping, advantage estimation, and reward aggregation, are introduced across separate papers with inconsistent notation, making them difficult to compare and intimidating to the non-expert community. We present UNIPO, to our knowledge the first interactive visualization tool that exposes the token-level training dynamics of RL fine-tuning algorithms through a unified design. UNIPO connects three complementary views, a high-level training overview, a step-level prompt and response inspector, and a side-by-side algorithm comparison, allowing learners to observe how individual design decisions propagate through training. Through two usage scenarios, we demonstrate how UNIPO supports both classroom instruction for non-experts and algorithm selection for AI practitioners. Our tool is open-source and publicly available at https://poloclub.github.io/unipo/.
For Practitioners
ML practitioners fine-tuning LLMs with RL, and educators teaching these methods, would find this paper relevant. They can use UNIPO to compare PO algorithms before expensive training runs, diagnose their own runs at the token level, and teach RL fine-tuning interactively rather than through static notation.