Training a coding model to paint watercolours with TRL and OpenEnv
A coding model learns to paint watercolours by RL over aesthetic preference, using a hand‑rated image pool and a JavaScript drawing library.
Why Now
Surya Narreddi’s viral video of a language model painting watercolours sparked a reproducible open‑source effort using TRL and OpenEnv on Hugging Face.
What Happened
The project trains a model to output ~150‑line JavaScript that uses p5.brush to paint watercolours, with a reward combining a preference model (HPSv3) and a pairwise judge (Qwen3‑VL). Three reward mixes were trained, all open on the Hub, and the training pipeline runs end‑to‑end on Hugging Face Spaces and Jobs.
Why It Matters
It demonstrates RL over taste is feasible, enabling models to learn artistic style from human‑rated image pools. This could lower barriers to custom artistic AI and provide interpretable code‑based art generation.
The Limitation
The reward relies on a limited hand‑rated pool of 178 images, so the learned style may not generalise beyond that dataset.
What You Can Do
Try the open training recipe on Hugging Face to generate your own watercolour‑painting model.