On September 3, Hugging Face published a complete training log: it took a coding model and trained it to paint watercolors. There's no diffusion model anywhere in the pipeline, and no image-generation component at all. The model outputs a piece of JavaScript code that calls a brush library called p5.brush; running that code leaves a watercolor painting on the canvas.
The work comes from Sergio Paniego, based on an original idea by Surya Narreddi. The base model is Qwen3.5-35B-A3B, trained with GRPO from TRL, with LoRA applied across all linear layers, a learning rate of 5e-5, and a constant-with-warmup schedule. Code execution, rendering, and scoring all run inside an OpenEnv environment: the model generates code, the environment executes it, takes a screenshot, and hands the image to the scorer.
A Reward With No Right Answer, Just a Pool of References
Most of the reinforcement-learning wins on large models over the past couple of years have shared one thing: an automatic way to tell right from wrong. Math problems have known answers; coding problems have unit tests. Watercolor paintings have neither. This training log's solution is to split "does it look good" into two things that can actually be scored.
The reward function has four components. Whether the code runs successfully accounts for 0.05, and code length accounts for another 0.05, with the remaining 90% split between two aesthetic signals: a judge model that does pairwise comparisons, and a human-preference scorer called HPSv3. The weighting between the two swaps across experiments — in the judge-led run, the judge gets 0.60 and HPSv3 gets 0.30; in the hps-led run, it's reversed.
The judge's standard comes from a hand-scored reference pool of 178 paintings. Every image the model generates gets compared against the pool. That same pool also defines what counts as "diverse." As the blog post puts it, "The pool decides what counts as variety, the same way it decides what counts as quality." Those 178 images effectively set the ceiling for the entire training objective.
Three Runs, Three Curves
The author ran three comparison experiments. The HPSv3-only run went 60 steps, with average reward rising from 0.58 to 0.71. The judge-led run went 110 steps, rising from 0.45 to 0.72. The hps-led run, also 110 steps, rose from 0.57 to 0.82 — the highest starting point and the highest ending point of the three.
The judge-led run had the biggest gain, 0.27, but it also started from the lowest point. The hps-led run gained 0.24 but finished at 0.82 — putting the human-preference scorer in the lead role with the judge model as a supporting signal turned out to be more stable than doing it the other way around. The scorer-only run, with no judge model at all, gained just 0.13, the smallest of the three. The pairwise-comparison layer clearly contributes real signal, not just decoration.
One GPU, 34 Hours, About a Hundred Dollars
The hardware setup is simple: a single H200 throughout. The 60-step run took 18 hours; the 110-step run took 34 hours. Rendering ran on a CPU-spec Space, and the HPSv3 scorer ran on an a100-large Space.
At current cloud H200 rates of roughly $3 to $4 an hour, one full 110-step training run costs somewhere between $100 and $140. Add in the scorer and rendering overhead, and the whole project's compute bill lands in the low hundreds of dollars. For a team that just wants to test whether aesthetic preference can work as a reward signal, that's a low enough bar to try on a whim.
The open-source release is fairly complete: the training recipe and scripts are on GitHub, along with the 178-image reference dataset, a directly-cloneable RL environment Space, the HPSv3 scorer Space, three LoRA adapters with their respective rollout datasets, and a gallery for browsing through every generated result. Anyone who wants to reproduce it doesn't need to build the environment from scratch — just clone the Space, tweak the reward weights, and start.
Follow this line of thinking and plenty of other use cases open up. How a front-end component feels, whether a poster's layout is well-balanced, whether a chart's colors work — these are all tasks where right and wrong are hard to define but people can tell the difference at a glance, and they've historically been hard to write into an RL reward function. The recipe here is to first assemble a reference pool of a few hundred examples, find a good-enough preference scorer, and let GRPO handle the rest. The catch is still the reference pool: if the pool is skewed, the model ends up skewed too — and skewed in a very consistent way.
Sources: Hugging Face official blog, TRL and OpenEnv project documentation, CocoLoop; the step counts, reward ranges, and single-GPU run times for the three experiments were checked line by line against the blog's training log, and the compute cost is a rough estimate based on public list prices.