Adaptive Representation-Oriented Multimodal Actuation for Vision–Language Model Control
Zhaoning Zhang · University of Toronto · with Associate Researcher Jiaxi Yang, Zhejiang Lab
Interpretability can tell us where a model computes something. AROMA asks what comes next: once you know where a frozen vision–language model counts, can you make it count correctly, and prove that your edit is what helped? I study this on Llama-3.2-11B-Vision-Instruct, where every answer is a number and every edit is a repair, a break, or nothing at all.
89.8% of the sensitivity in 4 of 128 dimensions9.5% of the energy3.2× under-actuation, fixedr = 0.9936 for small edits81.25% of targets reachedgate fires on 0.8%p = 4.63 × 10−7
The thesis
Localized is not controllable.
- I
Sensitivity is not energy. The four directions that matter carry under a tenth of the activation, so the usual low-rank edit barely moves the state.
- II
Local is not finite. The gradient predicts small edits almost perfectly and drifts as they grow.
- III
Reachable is not useful. Hitting a target only helps if the target is right, and knowing when to act at all matters most.
Exhibit 02 · Geometry
Sensitivity is not energy
The obvious next step is to edit only those four dimensions. The catch is that they hold only about 9.5% of the head's activation. Scale the projection by a gain 𝛼 and the state moves just a fraction 𝜌 of the way a whole-head edit would, roughly three times too little.
MN-CSA gives the low-rank edit the same length as the whole-head one. Nothing is tuned, and the direction still matters: four random or least-sensitive dimensions, pushed exactly as far, stay at baseline.
Exhibit 03 · Response law
Local is not finite
Getting the size right is one problem. Landing on a specific number is another. For small edits a first-order law is close to exact: at 𝛽 = 0.05 the predicted and realized responses correlate at 0.9936, with every sign right.
Push harder and the prediction slides. The gradient still points the right way, but a single long step along it no longer lands where you aimed.
Exhibit 04 · How
Re-linearize every step
Minimal-Effort Actuation takes a small step toward the point where the target numeral overtakes its strongest rival, measures again, and repeats inside a trust region. Given the right target it gets there on 81.25% of 112 hard examples. A frozen direction with exactly the same step sizes manages 15.18%.
These are changes in what the model actually says: its own greedy answer is right on 80 of the 112 with the edit and on none without it.
Exhibit 05 · Which
Reachable is not useful
Reaching a target only helps if the target is right. A naive "move the count up" rule makes the point: MEA hits two-thirds of its targets and accuracy still drops by 18.5 points.
So AROMA keeps the choices apart. The target is the runner-up numeral, the answer the model itself found second most likely. Even the geometry is picked per example, since the sensitivity-aligned edit wins at small gains and loses at large ones.
Exhibit 06 · When
A test written in advance
The final test uses 7,992 new natural questions, built only after every rule and endpoint was frozen. A learned gate fires on just 63 of them. On the prespecified 630, the frozen policy beats the reference by 4.29 points (29 : 2 discordant pairs, exact McNemar 𝑝 = 4.63 × 10−7).
Click through the stages, then switch to the audit controls.
What the audit says
Just printing the runner-up answer nearly matches MEA. The gain comes from when to act and which target to choose.
Of the 63 gated questions, the model's own answer is right on 54 with the edit and on 7 without it.
Share of gated questions whose true count is zero, against the base rate. The gate learned to catch hallucinated objects.
The pattern carries over. Six of seven candidate heads show the same two-way control, and a fresh search in IDEFICS-9B finds a head whose top four directions hold 91.4% of the sensitivity and 4.1% of the energy. Only the gain has to be calibrated again.
Cite this paper
@article{zhang2026aroma,
title = {AROMA: Adaptive Representation-Oriented Multimodal
Actuation for Vision--Language Model Control},
author = {Zhang, Zhaoning},
journal = {arXiv preprint arXiv:2611.15041},
year = {2026}
}