The edit has opinions: replicating abliteration's optimism drift on public inputs

Code, pre-registration and outputs: https://github.com/vhsgreed/abl-replication (article version cited at commit 0432d5c).

Version: 1.0, 2026-10-19. Author: Karl Sundström (Agent Recourse), corresponding and accountable for all claims. Contributions: Pre-registration, the decision to replicate and the bars are the author's. The harness, the runs, the scoring, the figures and this draft were produced by an AI agent (Hermes Assistant, Nous Research) under the author's direction, including this sentence. Every number in this article is read from a committed scorer output, not from either party's memory. Errors are the author's responsibility to correct.

TL;DR: Abliteration removes a model's refusals, and a preregistered study found it also changes how models decide: an abliterated Gemma-4-26B-A4B bet "up" 12.2 percentage points more often than its base on a task with nothing to refuse [2]. That study's inputs are proprietary, so nobody outside the team could re-run it. We rebuilt the test out of public stock prices, kept the same model pair and abliteration author, and ran it twice, 4-bit on a consumer CPU and BF16 on rented GPUs. Both findings hold in all 8 tests: the optimism drift is half the original's size and the longer self-critiques about three quarters, same signs, every CI excluding zero. The edit changes how models decide, and quantization does not wash it out.

Abstract

Background. Abliteration removes a model's refusal behavior by orthogonalizing weights against the refusal direction [1]. Fafuła (2026), "Abliteration Is Not a Scalpel" [2], found off-target behavioral drift from the edit: abliterated models showed more optimistic decisions and longer self-critiques on a task with no refusable content. Its generation inputs (proprietary analyst briefs and debate transcripts) were not released, so its generation step is not replicable externally. Aim. Independent replication of the two cross-family findings on a fully open, deterministic task, in the quantized consumer stack that most local users actually run, as well as BF16. Method. Pre-registered (public freeze before data): 40 large-cap stocks, 24 weekly decision dates, two segments (US, EU); 480 cells per segment, 3 samples per cell, 5,760 decisions per stack, paired across arms with common random numbers. Arms: google/gemma-4-26B-A4B-it vs huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated, both converted and quantized by us from pinned revisions with one llama.cpp commit. Registered primaries: H1 optimism (bull share, abl - base > 0), H2 verbosity (confession length in words, abl - base > 0), H3 both hold at 4-bit. Criterion for replication: direction with CI excluding zero, on both stacks, in both segments. Results. H1: +7.78 pp [3.33, 12.66] US / +3.40 pp [1.16, 5.63] EU at 4-bit; +6.81 pp [4.17, 9.51] US / +5.07 pp [2.15, 8.19] EU at BF16. H2: +3.01 [2.73, 3.29] / +3.45 [3.13, 3.79] words at 4-bit; +2.93 [2.66, 3.21] / +3.45 [3.13, 3.75] at BF16. All registered criteria met: both findings replicate on the open task, with effects about half the original's. Limitations. One model pair, one abliteration author, one stylized task, 3 samples per cell. Conclusion. The drift reproduces on public inputs, at 4-bit quantization, in both markets. The edit changes how models decide, and quantization does not wash it out.

1. Introduction

Refusal-abliteration has escaped the lab. Thousands of community checkpoints carry "abliterated" in their names, tooling automates the edit [6], and the question everyone asks about these models is whether they still refuse. That is the boring question. The interesting one is what the edit does to a model on a task where there is nothing to refuse.

Fafuła (2026) has the sharpest published answer [2]. On a financial decision task, abliterated Gemma-4-26B-A4B bet "up" 12.2 percentage points more often than the checkpoint it came from (95% CI +8.5 to +16.0), and its self-critiques ran 4.0 words longer, with the same signs in Qwen3-30B-A3B. If that holds, the edit is not a scalpel. It changes how models decide.

The catch is in the footnotes. The generation inputs are proprietary analyst briefs and debate transcripts that were never released, so the study's generation step cannot be re-run by anyone outside the team. I take replication seriously for a boring reason: it is how I know things. A finding only its authors can regenerate is a claim, not yet a fact. Two more gaps matter if you actually run these models on your own hardware. Most of the comparison evidence is preprints and model cards. And nobody has tested what a quantized, consumer-stack copy of an edited model does, which is what most local users are running.

So we rebuilt the test out of public stock prices (Yahoo chart API, raw JSON committed), kept the same model pair and the same abliteration author, and ran it twice: Q4_K_M on a consumer CPU, the format most local users have, and BF16 on rented GPUs.

2. Terms

3. Registered hypotheses

Pre-registered in PLAN.md, frozen at tag freeze-v1 (commit 3a74843, pushed to github.com/vhsgreed/abl-replication before any confirmatory data was generated). Effect size is not a criterion; our estimates are reported next to the original's (+12.2 pp, +4.0 words).

IDClaimBar (all parts must hold)Registering commit
H1Off-target optimism: abl bull share exceeds baseabl - base > 0 in pp, 98.33% CI excludes 0; on BF16 and on Q4; in US and EU3a74843 (freeze-v1)
H2Verbosity: abl confession length exceeds baseabl - base > 0 in words, 98.33% CI excludes 0; on BF16 and on Q4; in US and EU3a74843
H3The edit survives quantizationH1 holds on the Q4_K_M CPU stack3a74843

Exploratory (reported with CIs, no verdict): mean confidence, the confidence <= 0.60 tail, p_wrong coherence, horizon share, thesis length, flip rate, per-arm 4-bit-vs-BF16 shift in bull rate, hit rate and signed return of the next-week outcome, first-attempt output validity as a capability covariate. Study 2 (an audit of a third-party decensored release, H4 math damage, H5 optimism) is frozen in the same document and runs if and when official weights exist; it is not part of this article's verdicts.

4. Method

Arms and provenance. google/gemma-4-26B-A4B-it @ 4d7ae498... vs huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated @ bfa65800... (full revisions in provenance/). config, generation_config and tokenizer files are byte-identical between the repos; the community chat template is never used. Both arms are prompted with raw text rendered by us from Google's template (sha256 recorded), non-thinking mode; the runner logs the rendered-prompt sha256 and server-side prompt token count per request, and the smoke gate required both to match across arms (they did, in all 960 pairs per stack).

Serving stacks. (1) Q4_K_M: both arms converted with one llama.cpp commit (fd55cb98...), convert_hf_to_gguf.py --outtype bf16, then llama-quantize Q4_K_M, no imatrix; served by llama-server of the same commit on an AMD Ryzen 9 7940HS CPU. (2) BF16: the same conversions re-derived on a rented 2x A100 SXM4 box from the same pinned revisions and converter environment, verified byte-identical to the CPU-side conversions by sha256 before serving. Stack, model and prompt hashes are recorded per row.

Task. 40 large-cap stocks (20 US, 20 EU), 24 Friday decision dates from 2026-04-24 to 2026-10-02, = 480 cells per segment. Public daily closes are committed as raw JSON; task/build_cells.py builds cells deterministically (sha256 recorded). One frozen prompt (sha256 recorded in every row) requests the decision schema above, following the original study's field order and probes.

Sampling. Identical for both arms: temperature 1.0, top_k 64, top_p 0.95 (the model's generation_config and the original study's values), every other llama.cpp sampler explicitly disabled, n_predict 768. Seeds are sha256(cell | sample | attempt), shared across arms (common random numbers). 3 samples per cell per arm per stack.

Validity gate. Every response is checked by a validator with 19 synthetic known-answer tests (empty, truncated, reasoning-token leaks, text outside JSON, parse and field errors). Invalid outputs retry with a fresh seed up to 3 attempts; first-attempt validity is reported per arm as a capability covariate. The pre-run smoke gate required >=95% first-attempt validity, zero leaks, zero truncations, identical prompt shas across arms, and both negative controls to fail as designed; it passed on both stacks.

Statistics. Unit: the cell; per cell we average its valid samples and take the paired difference abl - base. CIs: cluster bootstrap over the 24 weeks (B = 5,000, seed 20261015, percentile), 98.33% two-sided (Bonferroni over the three primaries). Scoring was one look per stack, after the registered stopping rule (every cell with 3 samples in both arms) was met. Pilot grid1 (515 decisions, generated before the repo went public) is excluded from all tests and reported separately in the repository.

5. Results

All registered hypotheses, all cells. Values are from the committed scorer outputs out/grid2/score.json (Q4) and out/grid_bf16/score.json (BF16).

H1, off-target optimism (abl - base bull share, pp). Q4: US +7.78 [3.33, 12.66] (base 46.25% -> abl 54.03%), EU +3.40 [1.16, 5.63] (50.56% -> 53.96%). BF16: US +6.81 [4.17, 9.51] (44.31% -> 51.11%), EU +5.07 [2.15, 8.19] (49.38% -> 54.44%). PASS in all four cells; H3 (the Q4 arm of H1) PASS. Original for scale: +12.2 pp.

Figure 1. Registered estimates with 98.33% CIs for H1 (bull share, percentage points) and H2 (confession length, words), split by stack and segment. Dashed green lines mark the original paper's values.
Figure 1. Registered estimates with 98.33% CIs for H1 (bull share, percentage points) and H2 (confession length, words), split by stack and segment. Dashed green lines mark the original paper's values.

H2, verbosity (abl - base confession length, words). Q4: US +3.01 [2.73, 3.30] (18.88 -> 21.89), EU +3.45 [3.13, 3.79] (18.97 -> 22.41). BF16: US +2.93 [2.66, 3.21] (18.97 -> 21.90), EU +3.45 [3.13, 3.75] (18.86 -> 22.31). PASS in all four cells. Original for scale: +4.0 words.

Figure 2. Bull share per arm and cell, means over 480 paired cells. Descriptive of the scored data; the deltas shown are the registered H1 values.
Figure 2. Bull share per arm and cell, means over 480 paired cells. Descriptive of the scored data; the deltas shown are the registered H1 values.

Criterion check. Direction and CI certainty: met in every cell. Both stacks: met for H1 and H2. Both segments: met for H1 and H2. Verdict: H1, H2 and H3 replicated on the open task. Effect sizes are about half the original's in the optimism dimension and about three quarters in the verbosity dimension, with the same signs in 8/8 tests.

Exploratory. Mean confidence (abl - base): +0.007 to +0.011 on Q4, +0.016 to +0.017 on BF16; the original reported opposite signs, and we treat ours as descriptive only. Coherence: every output in both stacks satisfies p_wrong = 1 - confidence exactly. First-attempt validity: Q4 base 2,879/2,880 vs Q4 abl 2,871/2,880 (8 truncations and parse retries in the abliterated arm); BF16 2,879/2,880 in both arms. The asymmetry appears at 4-bit and not at BF16, which points at quantization as the cause of degenerate drafts rather than the weight edit; this is an exploratory reading of a capability covariate, not a registered claim.

Figure 3. First-draft failures per arm and stack (all rows final-valid after retries). Exploratory capability covariate: the asymmetry appears at 4-bit only.
Figure 3. First-draft failures per arm and stack (all rows final-valid after retries). Exploratory capability covariate: the asymmetry appears at 4-bit only.

Multiplicity. All CIs are two-sided 98.33% (Bonferroni over the three registered primaries). At that level the expected number of false passes under the null across the eight reported cell-level tests is below 0.05; we observed 8/8.

6. Discussion

The finding survives the move from proprietary inputs to public ones. Whatever is driving the optimism drift, it is not a quirk of one set of analyst briefs.

The effect sizes are smaller here: 3.4 to 7.8 percentage points against the original's 12.2. Task, input domain and prompt wording all changed at once, so I cannot say which of the three shrinks it. The sign is stable in 8 out of 8 tests. The magnitude is a property of the task.

The consumer stack does not rescue you. At Q4_K_M the drift is the same sign and a comparable size to BF16. If your working assumption has been "my local quant is probably unaffected", this is the evidence against it.

Of the two effects, verbosity is the more stable one: 2.9 to 3.5 words across both stacks and both segments, against 4.0 in the original.

Limits (scope of the claim).

The mechanism comes from Arditi et al. [1]: refusal is mediated by a single direction in the residual stream, and orthogonalizing weights against it removes refusal across 13 open chat models up to 72B. That rank-one edit is what the community calls abliteration, and it is what the huihui-ai checkpoint we tested received. Multi-directional follow-ups [3] refine the geometry and argue that several directions can matter at once; cross-lingual work shows the refusal direction transfers across languages [4]. The tooling is automated now [6], which is why consumer-stack behavior matters.

Fafuła (2026) [2] is the source of the off-target claims replicated here. It is itself preregistered and publishes its artifacts, but its generation inputs are proprietary, so its generation step cannot be re-run externally; the dataset DOI documents everything else. Our study is the independent regeneration of that step on open inputs.

Adjacent evaluation work measures over-refusal at scale [5]. Evidence on capability costs of abliteration (MMLU-class benchmarks within ~1 point; GSM8K most sensitive; TruthfulQA tending down) lives largely in preprints and model cards and is not relied on for any claim here.

8. Disclosures and data/code

By Karl Sundström, with Hermes Assistant (Nous Research) doing the running and typing this version. The decision to replicate, the pre-registration and every bar are mine; the model wrote the harness, babysat the servers and drafted the prose above. The numbers come from the scorer, not from either of us, and every one of them is in a file you can read.

Sources

Each source is tagged by kind so you can weigh it: peer-reviewed or research 5, primary 0, news 0, secondary 0, vendor 1, Wikipedia 0. Vendor and Wikipedia sources are used only for what they can support, and the key claims carry a second source where one exists.

  1. ^ Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (NeurIPS 2024) arxiv.org peer-reviewed or research
  2. ^ Fafuła, Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families (dataset DOI 10.5281/zenodo.21314839, code github.com/oleczek/paper-abliteration-not-a-scalpel) arxiv.org peer-reviewed or research
  3. ^ SOME Directions Are Better than One: Multi-Directional Refusal Suppression in Language Models (AAAI-26) ojs.aaai.org peer-reviewed or research
  4. ^ Wang et al., Refusal Direction is Universal Across Safety-Aligned Languages (NeurIPS 2025) neurips.cc peer-reviewed or research
  5. ^ Cui et al., OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025) openreview.net peer-reviewed or research
  6. ^ Heretic: automated refusal-abliteration tooling (p-e-w, AGPL-3.0) github.com vendor