Index / [03] Preprint II arXiv:2611.21539 · Nov 2026

Seen but Unsaid

Locating and Repairing Cross-Image Comparison in Vision–Language Models

Zhaoning Zhang · University of Toronto

Show a vision–language model two images, mark a point in the first, and ask which of four points in the second is the same one. It often gets this wrong, even when the right answer can be read from its own hidden states. I follow the signal from the pixels to the answer letter in Qwen3-VL (2B to 32B), InternVL3.5-8B and Gemma-3-12B to find where, between seeing and saying, the match goes missing.

Read on arXiv

probe .64, model .43visual tokens +36 pts over the answer probe16–32 heads that compare−17 to −22 pts when they are scrambled+28.3 pts from 0.02% of the parameters21–51% of the gain by transplant

The thesis

Seeing is not saying.

  1. I

    Two failures, not one. Exact copies are compared and then lost before the answer. Semantic matches are seen but never compared.

  2. II

    One pathway. A few dozen attention heads carry the comparison; scramble them and accuracy collapses.

  3. III

    A lever training finds. Tuning only those heads repairs semantic matching, and free adapters route through them anyway.

Exhibit 01 · Follow the signal

Where does the match get lost?

I measure the signal at four points along the way, with three tasks that get harder: exact copies of synthetic shapes, the same physical point from a new viewpoint (HPatches), and the same part on a different object (SPair-71k).

Switch tasks and watch where the × moves.

Fig. B·01 — Four readouts, Qwen3-VL-8Bpaper data
Visual tokens—region readout over answer probe, 𝑃 − 𝑅3
Comparison—selectivity of cross-image heads, 𝜌ref
Answer position—input-matched probe, 𝑅3
Output—calibrated answer, 𝑅4aff
Solid borders mark a signal that is carried, dashed ones a signal that is lost.

Exhibit 02 · Across models

Two different failures

For exact copies the comparison works, but the answer doesn't come out. A linear probe at the answer position beats the model's own calibrated answer by 15 to 25 points in every family except the smallest Qwen. On real correspondence the two agree, so the gap isn't something the probe invents.

Semantic matching breaks earlier. A readout over the visual tokens beats the answer-position probe by 14 to 26 points on SPair-71k: the model sees the matching part and never compares it.

Fig. B·02 — The two gaps95% paired CIs
Left: probe minus output, 𝑅3 − 𝑅4aff; hover a row. Right: visual-token readout minus probe, 𝑃 − 𝑅3, in points.

Exhibit 03 · Decide

Beside the output path

Maybe the decoded answer is just too faint to win? I tested that directly. Removing or amplifying the probe's subspace changes accuracy by at most 0.2 points, no more than a random subspace.

The layer itself isn't inert: adding the output direction of a letter the model didn't pick does flip answers. The information sits beside the path to the output, so more gain at that layer won't bring it back.

Fig. B·03 — What an edit can switchrange over models
Share of answers switched by an edit at the probe's layer.

Exhibit 04 · Compare

The heads that compare

Each image is encoded on its own, so the comparison must happen inside the language model. A small set of heads does it: tokens of the right option look back at the reference far more than the distractors do.

To test whether they are needed, I redistribute their cross-image attention without changing its total. Synthetic accuracy falls 17 to 22 points and HPatches 15 to 32. Random or attention-matched heads never cost more than 7.

Fig. B·04 — Cross-image attention and knockouts500 items / task

Arc width follows the measured selectivity 𝜌ref (layout schematic). Uniform and Shuffle keep each query's total attention on image 1 and change only where it goes; Block removes it.

Exhibit 05 · Repair

A lever training finds

If these heads carry the comparison, they should be where to fix it. Training only their query and key projections, about 0.02% of the parameters, lifts SPair-71k by 28.3 points in Qwen3-VL-8B and 25.1 in InternVL3.5-8B, and it helps on HPatches and BLINK too, which it never saw.

Location matters. The same recipe on layer-matched or attention-matched heads does worse. Click a point to compare.

Fig. B·05 — Every training armseed-averaged
A: query–key on the comparison heads. AQ: queries only. B, M: layer- and attention-matched heads. C: values. D, E: all-layer adapters. Switch the x-axis to see selectivity track accuracy.

When nobody points the way

21–51%

of a free all-layer adapter's gain returns when only the comparison heads' attention is transplanted into the base model. Random heads: at most 7%.

83–86%

of that effect still works with the heads' base weights. Training changes what they read, not how they compare.

13–17%

is all that is lost when the heads are reset inside the adapter. The pathway is sufficient, and redundant.

Cite this paper
unsaid.bib
@article{zhang2026unsaid,
  title   = {Seen but Unsaid: Locating and Repairing Cross-Image
             Comparison in Vision--Language Models},
  author  = {Zhang, Zhaoning},
  journal = {arXiv preprint arXiv:2611.21539},
  year    = {2026}
}