Reinforce Sight

GRPO has moved beyond language reasoning; if you have a verifiable task and a base VLM, you can show consistent gains on grounding, counting multimodal etc. While initial experiments/efforts are certainly interesting, there is a common theme about unfaithful traces, limited generalisation, and an obsession with "aha" moments. And a lot of blame is put on SFT... I personally feel that throwing traditional tasks isn't necessarily the right direction to draw very strong conclusions, but again these are very early investigations and haven't been tested on scale. It's kind of obvious of that you don't want to overtrain small models, so it's still an open-ended question about what is the best recipe to upgrade small reasoning models. My hypothesis is that it depends on what your task is.
For instance, let's look at something VLMs are genuinely terrible at: Spatial Reasoning. Is this a fundamental limitation? Well, Google (Chen et al., 2024) says it's a data problem. You'll see significant capability improvements and emergent spatial reasoning if you pre-train on the right kind of data.

How do you synthesize such data:

Now, Google didn't open-source anything related to this project, but I found a community-source implementation

Full disclosure, the author seems to be motivated about this exact problem and about 10 days ago, they released a model...but it seems they just SFT'd over a small thinking vlm. I would definitely compare the performance of their model with mine. Another key difference is that their model is trained to do quantitative estimations too, I didn't consider those for two reasons 1) simplicity and 2) Intuitively, anything quantitive might not generalize for a small model without any spatial supervision during pretraining. So I adapted the VQASynth pipeline to drop all numerical templates, skip metric scaling, and change the templates. Additionally, I also added more predicates like support/under, containment, relative sizes etc.

Now in order to endow a VLM with spatial reasoning (without pre-training) you have to resort to good old SFT. So, I generated about 10k samples with this approach and finetuned qwen. After that it's just RL, the reward design is extremely simple for these tasks as it is a sparse reward settings.

For evaluation I just used the 3D tasks for CV-bench (to save compute, I skipped 2D split for now and also I haven't yet done RL for object-counting). The current checkpoint has significantly improved on spatial reasoning (12pp). The base model is around 50% which might seem impressive but so is a coin-flip when you have only two options.

So that's progress so far, I do have to train my model for more longer (more steps), and conduct more extensive evaluation (note that i have also not evaluated the SFT only performance so idk how much boost RL gave)