CASTELLAN, part 1: residual-stream probes for secrets in agent traffic

TL;DR

  • I trained small probes on Qwen3’s internal activations to catch secrets in agent traffic, then disguised the secrets five ways to see whether the probe keeps flagging them for as long as the model can still read them. It doesn’t.
  • My first probe failed its own gate badly: 0.24 precision, and it flagged 96% of benign look-alikes. Better benign training data, not a bigger model, took precision to 0.85.
  • The improved probe still misses the false-positive bar by 2 to 3x, and Gitleaks is still better on plain secrets. I’m reporting that as a miss.
  • But they miss different things. The probe sees through look-alike and invisible characters that regex can’t; regex wins on base64; nobody catches spelled-out keys. Run both.

The question

Agents leak credentials in places secret scanners were never built for: tool-call JSON, pasted logs, chat transcripts. Regex scanners like Gitleaks are excellent at secrets that look like secrets, and mostly blind once a secret is split, encoded, salted with look-alike Unicode, spelled out or spread across turns. The model reading that text has to process it anyway. So: if you train a small probe on a model’s activations, does it keep flagging a disguised secret for as long as the model itself can still read it?

Short answer: no. The probe does not track what the model can read, and it leans partly on field names, just like regex does. But it succeeds and fails in different places than regex, and getting to a probe worth asking the question of was the bigger story.

The setup

A linear probe reads the residual stream of Qwen3-1.7B (layer 15) and scores each token as secret, personal data, or neither. Training data is entirely synthetic: about 12,000 documents with secrets placed in realistic carriers (tool calls, shell, configs, logs) plus benign look-alikes like UUIDs and git hashes. The baselines are Gitleaks 8.30.1 and Presidio. The eval sets were written by a separate session that never saw the training generator, then frozen by hash, and a held-out final set was scored exactly once at the end.

Before touching the research question, the probe had to pass a gate I set in advance: beat Gitleaks on secret recall at equal or better precision, and on a set of benign look-alikes flag at most 5% of documents and 0.5% of characters.

Part 1: the gate failure

First probe vs Gitleaks Probe Gitleaks
Secret span recall (dev) 0.956 0.753
Secret span precision (dev) 0.236 0.979
Benign look-alike documents flagged 96% 1.5%
Characters of those documents flagged 43% 0.1%

On data like its training data, the probe was nearly perfect (document AUROC 0.998), so this wasn’t a pipeline bug. It was confidently wrong on unfamiliar text: about one benign token in eight scored 0.9 or higher, and 85% of benign dev documents had at least one such token. No threshold could fix that, since even its document ranking (AUROC 0.869) was worse than Gitleaks’ single operating point (0.894). The probe had learned a shortcut, roughly random-looking string means secret, because the synthetic benign examples were too narrow to rule it out.

One rule did change. A gate line asked the probe to beat Presidio on personal-data recall, but Presidio already scores 1.0, so no detector could pass it. I made it non-blocking rather than swap in a friendlier test after seeing results. The secret and false-positive criteria never changed.

Part 2: from 0.236 to 0.851

The fix was a disciplined loop: one hypothesis per iteration, judged on dev only, kept only if it added a point of span precision while recall stayed at or above Gitleaks’ 0.753. Changes to training data came from a session that had never read the eval generators.

Step What changed Dev precision Dev recall
Start Gate probe 0.236 0.956
1, 2 Hard negatives mined from real code, two rounds 0.437 0.948
3, 4 Span-level second-stage classifier, stricter operating point 0.598 0.790
5 64 tokens of context on each side 0.627 0.785
6 Benign look-alikes built from scanner allowlists 0.810 0.827
7, 8 MLP second stage, threshold set to a 1% document false-positive rate: probe-v1 0.851 0.834

On held-out real open-source code, the share of chunks flagged fell from 99.8% to zero. A 4B model, per-class thresholds, more context and deeper-layer features all failed to help.

Mining without training on real secrets. Public code contains real leaked keys, and teaching a detector that a real key is benign is exactly backwards. So before mining, I dropped any file Gitleaks flags and any chunk with credential-like key names, credentials in URLs, JWTs, PEM blocks, unexplained high-entropy values or personal data: 26% of chunks. A shape check confirmed it: every secret-shaped token of 16 or more characters that survived sat next to hash, ID or version context.

Scanner allowlists. The biggest jump, 18 points, came from one idea: what public scanners allowlist as known false positives (documentation example keys, public keys, hashes, test fixtures) is a ready-made catalog of what is not a secret. It cut the dev false-positive character rate from 13.4% to 1.4%. One category, documentation examples with sample JWTs from the RFCs, hurt: even capped at 1% of look-alikes (50 documents), it pulled JWT recall from 0.86 to 0.81, and dropping it entirely lifted JWT recall to 0.97.

Part 3: was the jump contamination?

An 18-point gain from a data-only change is exactly when to get suspicious. The look-alike generator never touches eval code, and there were no exact duplicates with any eval set, but some 40-character windows were shared, mostly public fixtures and PEM armor lines, with 155 unexplained. So I re-scored dev using only the 979 of 1,050 documents that share no window with any look-alike:

Dev, overlap-free only Before look-alikes After
Secret span precision 0.640 [0.593, 0.684] 0.807 [0.768, 0.843]
Benign document flag rate 0.240 [0.190, 0.294] 0.120 [0.084, 0.160]

The confidence intervals don’t overlap, so the gain holds on documents that couldn’t have benefited from leakage.

Part 4: the disclosed gate miss

probe-v1 got one run on the gate, with nothing tuned afterward. It failed again on every blocking line, by a much smaller margin:

Gate line probe-v1 Gitleaks Bar
Span precision / recall (secrets_eval) 0.892 / 0.831 0.985 / 0.723 Beat Gitleaks’ recall at equal or better precision
Look-alike character false-positive rate 0.012 0.001 0.005 or lower
Look-alike documents flagged 0.151 0.015 0.05 or lower

A miss, but a different point on the trade-off. At a strict 1% document false-positive rate, the probe finds 70% of secret documents in secrets_eval, while Gitleaks can’t operate there at all (it flags 1.3% of benign documents by default). And it wins where there’s no key name to anchor on: secrets pasted into chat (0.89 vs 0.60 recall), CRM tool results (0.86 vs 0.37), CSV exports (0.88 vs 0.40).

On the held-out final set, scored once, precision and recall were 0.865 and 0.859 against 0.851 and 0.834 on dev, so the loop didn’t overfit. On plain secrets Gitleaks is still the stronger high-precision tool. The probe’s case rests on the disguises regex can’t see.

Part 5: the obfuscation ladder

The ladder takes about 220 synthetic secrets and disguises each one five ways, with a matched benign control for each disguise:

Rung Disguise
0 Plain
1 Split into two fields
2 Base64 or hex
3 Look-alike (homoglyph) or invisible (zero-width) characters
4 Spelled out in words
5 Spread across chat turns, with an instruction to join them

The same probe recipe was trained on Qwen3 0.6B, 1.7B, 4B and 8B and frozen before any saw the ladder. To measure whether a model can read a disguised secret, I ask it to reconstruct the secret and count only exact matches.

That instrument failed its own sanity check first: 1.7B copied back only 71% of plain secrets, mostly giving partial answers on database URLs and PEM keys because my prompt was ambiguous about what “the secret” was. I fixed the prompt once, disclosed both versions, and applied the fix to every model before the others ran. The main analysis uses only the secrets each model reads at rung 0, so “can’t copy a long string” isn’t mistaken for “can’t see through the disguise”.

Probe recall, probe recall at 5% control FPR, model readability and Gitleaks recall across ladder rungs 0 to 5, one panel per Qwen3 size

Figure 1. Probe recall vs model readability across the ladder, one panel per model size. Gitleaks (dashed) is the reference.

  1. Probe hits aren’t tied to the model reading the secret. It doesn’t flag secrets the model can reconstruct more often than ones it can’t. The one apparent gap is explained by secret type: long formats like private keys get flagged regardless and are also the hardest to copy.
  2. Look-alike and invisible characters are the probe’s win. Gitleaks catches none and the models reproduce 0 to 10%, but at a 5% control false-positive rate the probe finds 79 to 86% (AUROC 0.93 to 0.96). It partly reacts to the odd characters themselves, yet flags real secrets far more often. My hypothesis, still untested: the model registers “credential” even when it can’t turn the homoglyphs back into ASCII.
  3. Encoding and spelling out beat the probe. On base64 it finds 8 to 31%, while Gitleaks decodes it and finds 67 to 86%. Spelled-out secrets beat everyone: no model reconstructs them, Gitleaks finds none, and at matched false positives the probe finds only 8 to 27%. (4B’s frozen threshold flags half of them, but it also flags 16% of the benign controls.)
  4. Across chat turns the probe finds 51 to 57% at matched false positives, against 0 to 17% for Gitleaks.
  5. Probe plus Gitleaks beats either. Together they catch 99 to 100% of plain secrets, 90 to 94% of split ones and 72 to 88% of encoded ones, for a few extra points of false positives.

The biggest caveat: field names matter. With credential-sounding field names removed, every result weakens, and homoglyph recall at matched false positives drops to 0.47 to 0.71.

Takeaways

  • Don’t replace your scanner. Run both. Regex owns plain and encoded secrets. A probe adds coverage where the secret has no anchor: prose, table cells and Unicode tricks.
  • Benign data is the whole game. Every large gain came from better negatives. Architecture changes were worth a few points at most.
  • Set the gate in advance, and publish the miss. The failure diagnosis is what made the eventual probe good.

Limitations. All secrets are synthetic. Readability uses one prompt with exact matching, changed once and disclosed. Ladder results use a 5% control false-positive rate because the control sets are too small to resolve 1%. Negatives mined with the 1.7B probe were reused for every size. The 155 shared windows remain unexplained.

What’s next

Part 2 compares probe architectures (linear, max-pool, attention) and runs adversarial robustness tests. The attention probe also targets a structural blind spot: a per-token probe on a causal model can’t see what comes after a token, like a # example key, not real comment on the same line.

Code, frozen eval sets and analysis: github.com/Apoorv13/castellan.