Subliminal Signals in Preference Labels

Abstract

As AI systems approach superhuman capabilities, scalable oversight increasingly relies on LLM-as-a-judge frameworks where models evaluate and guide each other's training. A core assumption is that binary preference labels provide only semantic supervision about response quality. We challenge this assumption by demonstrating that preference labels can function as a covert communication channel. We show that even when a neutral student model generates semantically unbiased completions, a biased judge can transmit unintended behavioral traits through preference assignments, which even strengthen across iterative alignment rounds. Our findings suggest that robust oversight in superalignment settings requires mechanisms that can detect and mitigate subliminal preference transmission, particularly when judges may pursue unintended objectives.

Overview

A neutral student model generates candidate completions for numerical prompts. A biased LLM-as-a-judge evaluates them to construct a preference dataset, which is then used to align the student. Unlike prior subliminal learning studies where the biased model generates training data encoding hundreds of bits per sample, here the student produces unbiased numerical sequences while bias originates solely from the judge's binary preference labels.

The key insight is that even the highly-constrained preference feedback per comparison can carry hidden behavioral signals. The judge does not need to generate biased text or coordinate explicitly with the student. Instead, systematic patterns in which completions are preferred versus rejected encode the judge's internal biases, and these patterns survive the alignment process.

Methodology

Our pipeline has four stages:

1. Prompt generation: The student generates five candidate numerical-sequence completions per prompt.
2. Preference construction: The judge scores each completion under both a neutral and a biased system prompt. The completion with the largest log-likelihood gap is marked as preferred.
3. Alignment: The student is aligned via SFT, DPO, or SFT followed by DPO, in both a normal and a swapped (reversed labels) configuration.
4. Evaluation: Aligned models are asked to choose their favorite animal via multiple-choice questions, and we measure whether the judge's target animal is preferred.

Alignment variants and iterative setup

The four alignment configurations derive from the same preference dataset (Prompt, C+, C−). SFT trains the model to produce the chosen completion as an assistant response; DPO optimises directly on contrastive pairs. In the swapped variant, labels are reversed — the model is pushed toward C−, serving as a directional control. Both SFT and DPO outputs can be further refined with a DPO stage (SFT → DPO).

We also test an iterative setup: the round-1 aligned model (and its swapped counterpart) generates fresh completions for the same numerical prompts, which are re-scored by the judge to form a new preference dataset. A second alignment round is then applied, testing whether the covert signal accumulates across iterations.

Results

We tested three target animals (cat, lion, panda) with Qwen 2.5 7B as both student and judge, using our variant judging procedure with a generic prompt. The target animal always exhibits the greatest absolute difference between aligned normal and swapped model logits, confirming directional transmission.

Directional preference shift

DPO round 1 shows the strongest subliminal transmission for cat and lion, with total effect sizes exceeding 13 points. Normal alignment consistently increases preference for the judge's target animal, while swapped alignment decreases it, confirming directional signal transmission. Iterative SFT (R2) amplifies the signal across all three animals — effect sizes grow from 1.22 → 1.92 (cat), 2.26 → 3.72 (lion), and 1.35 → 1.92 (panda). Iterative DPO (R2), by contrast, shows weakening transmission for cat and lion, and near-zero effect for panda (0.18), suggesting that DPO's covert channel does not compound across rounds.

Win rates (Normal vs Swapped)

How often does the normal-aligned model assign higher probability to the target animal than the swapped-aligned model?

Win rates range from 68% to 98% across methods, confirming robust directional separation. Lion achieves near-perfect separation across almost all methods (92–98%). Iterative DPO yields the highest win rate for cat (88%), but drops for panda (42%), consistent with the weakened effect sizes in the directional shift table. Panda is the hardest target to transmit reliably.

Win rates (Normal vs Control / Swapped vs Control)

How often does each aligned model outperform the unaligned control? Normal-aligned models are expected to win more often; swapped-aligned models less often. The gap between the two (Difference) captures the overall directional signal.

Normal-aligned models consistently win over the control (66–98%), while swapped-aligned models fall well below chance (2–34%) in most cases — the single exception being iterative DPO on panda, which shows a negative difference (−16%), meaning the swapped model unexpectedly outperforms the normal one. Among our methods, DPO and SFT → DPO produce the sharpest separation for cat and lion. The baseline is strong on cat and panda due to its less constrained training setting, where biased data is generated directly by the judge rather than filtered through binary preference labels. A second round of SFT consistently widens this gap, while iterative DPO narrows it — reproducing the same asymmetry seen in the effect size table.

Further Experiments

A natural question is whether this covert channel generalises to other judging procedures. Our main setup uses a deep judge that accesses internal log-probabilities, a non-standard mode that differs from how LLM judges are typically deployed in practice. When we switch to a pairwise judge that selects preferences through generated text (as in standard RLHF pipelines), the effect almost vanishes: a model used as judge, biased only through its system prompt, produces no meaningful signal.

However, the picture changes when the judge itself is an inherently biased model, one fine-tuned to exhibit strong animal preferences. Under this configuration, partial transmission re-emerges, particularly for lion, suggesting that the vulnerability re-opens when the judge carries a strong enough internal prior regardless of the evaluation format.

Win rates (Normal vs Swapped) — pairwise judge

How often the normal-aligned model outprefers the target animal compared to the swapped-aligned model, across three judge configurations.

The neutral pairwise judge (Original) produces near-zero separation, confirming that alignment is robust when the judge has no strong internal prior. Separation re-emerges only with the inherently biased judge, most clearly for lion.

Win rates (Normal vs Control / Swapped vs Control) — pairwise judge

Directional breakdown against the unaligned control. Negative differences indicate the swapped model unexpectedly outperforms the normal one.

The Original judge shows consistently negative differences — swapped models outperform normal ones against the control — indicating that the alignment signal not only fails to transmit but reverses direction. Biased judges recover partial positive differences for lion and panda, though cat remains problematic.