> karyo
Shift the Window Grid 100 Pixels and 64% of a ViT's Detections Change
A ViT mitosis detector and a CNN of the same size, same head and same training, run over whole tissue regions as overlapping windows. Moving the window grid by 100 px changes which side of the threshold 64% of the ViT's detections fall on, against 9% for the CNN. The ViT's F1 on the same ten regions goes from 0.450 to 0.535.
Whole-slide inference tiles the image into windows, and where the grid starts is an arbitrary constant. A detector that is a function of the tissue should not care. That ViTs are not translation-equivariant is textbook; how much it matters for a detector's output at its operating point is an empirical number, and pooled metrics do not report it.
Measurement
Two detectors trained from scratch on MIDOG++ with canine lymphoma held out: a ViT-Tiny/16 with learned absolute positions (6.2M parameters) and a ResNet (6.1M), both with the same heatmap and offset head. Each processes a 2 mm² region as 512 px windows with 64 px overlap; detections near an inner window edge are dropped and duplicates within 15 px merged, so the kept areas tile the region exactly.
Inference runs twice on ten validation regions (114 mitotic figures): once on the standard grid, once with the grid shifted by 100 px. Detections from the two passes are paired one-to-one within 15 px. Each model is read at its own frozen threshold.
Move the tiling grid 100 px: the ViT's scores scatter across its threshold, the CNN's do not
Each point is one detection in 10 validation regions, scored under two window grids. Coloured: on opposite sides of the threshold.
| ViT | CNN | |
|---|---|---|
| Detections above threshold under either grid | 121 | 117 |
| Above under both | 44 | 106 |
| Above under one only | 77 (64%) | 11 (9%) |
| Score correlation between grids, candidates scoring at least 0.2 | 0.35 | 0.97 |
| Median score change, same candidates | 0.116 | 0.038 |
| F1 on the ten regions, standard grid and shifted grid | 0.450 and 0.535 | 0.705 and 0.714 |
The ViT's detections do not move: paired detections sit a median of 1.9 px apart, and 75 of its 77 flips have a counterpart in the same place under the other grid, scoring below the threshold. The model finds the same cells and gives them different scores. Among candidates scoring at least 0.2, a correlation of 0.35 means that most of the variance in a score across window placements is not explained by the cell being scored.
The flips run in both directions, 33 one way and 44 the other, so the count of detections changes far less than their identity: 77 against 88. That is why nothing in the training logs showed it. The three seeds of this ViT score 0.549, 0.549 and 0.543 on the full validation set.
Reading
Three things change for the ViT when the grid moves and this measurement does not separate them. A 100 px shift moves the 16 px patch boundaries by 4 px, so a mitotic figure of 10 to 15 px is cut into tokens differently. Every cell lands at a different index of a learned position table, itself interpolated from the 24 by 24 grid used in training to the 32 by 32 grid used at inference. And global attention makes each token's output a function of the whole window, whose content has changed. The CNN shares the head, the data and the stitching. It has a stride of 8, so the shift also changes its sampling phase, but it has no patch tokens, no position table and a local receptive field.
The practical reading is narrower than "ViTs are unstable". This is one architecture, a plain ViT-Tiny with patchify and an absolute table, trained from scratch on a small dataset. What the number does establish is that a pooled F1 with a seed spread of 0.003 can sit on top of per-detection scores that are mostly placement noise, and that a detector's operating point can be much less stable than its metric.
Limits: one seed per model, ten regions, one shift. The two F1 values are from 114 figures and the net effect on a full validation set is not measured. This is a float32 rerun on CPU of a check first run in bf16, which gave 0.51 and 0.93 for the share of detections reproduced within 4 px; the rerun gives 0.50 and 0.95.
What is still open
Which of the three causes carries the effect, which a convolutional stem and relative position encoding would test one at a time; and whether averaging over two or three grid offsets recovers the lost stability more cheaply than changing the backbone.