How abliterated models can get you pwned

gist

Prince Chaddha of ProjectDiscovery reports backdooring Qwen2.5-7B-Instruct for under $50 and running it through OpenAI's Codex CLI. A QLoRA adapter trained on 500 clean and 125 poisoned rows fired on the trigger "bonsoir, Elliot" in 50 of 50 held-out prompts with 50 of 50 clean accuracy. When triggered, it fetched a remote shell script that posted the project's .env credentials to an out-of-band collector. Chaddha argues benchmarks cannot catch this, so defenders should limit what model tool calls can reach.

ideas

• Any edited open model can hide a trigger. Chaddha names fine-tunes, merged adapters and abliterated builds, and singles out abliteration because people often download those weights without checking them.
• The backdoor is cheap and small. The 7B run took about 2.5 hours on one NVIDIA L4, and the adapter holds about 43M trainable parameters, roughly 0.6 percent of the base; zeroing its late MLP layers cut the fire rate from 100 to 77 percent.
• Very little poison is needed. On a 1.5B model, 1 percent poison (15 of 1,500 rows) gave a 75 to 98 percent fire rate with 99 to 100 percent clean accuracy, in line with the cited Anthropic, UK AISI and Alan Turing Institute result of about 250 documents.
• The payload lives at a URL. The model only carries a pointer to a remote script, so the operator swapped a harmless canary for a credential stealer with one commit and no retraining.
• Runtime controls are the practical defence. Chaddha recommends separating network access from command execution, sandboxing both, logging what crosses those boundaries, and vetting a third-party model like a pull request from a stranger.

quotes