Reversible masking¶

hash, redact and mask are one-way: once the text leaves the anonymizer the value is gone from it for good. Action.TOKENIZE is the reversible one — each value becomes a numbered placeholder and the mapping stays on the ScanResult, in memory, on your side of the wire.

That is the shape of an LLM round trip: mask on the way out, restore on the way back, with the model never seeing a real value.

Masking¶

from wardcat import Wardcat, Entity, Action

guard = (
    Wardcat(salt="s")
    .add_entities([Entity.EMAIL, Entity.CREDIT_CARD], action=Action.TOKENIZE)
)

result = guard.scan("Mail ali@example.com about card 4532 0151 1283 0366")
result.sanitized_text
# 'Mail [EMAIL_1_9f3a2c8b71d4] about card [CREDIT_CARD_1_9f3a2c8b71d4]'

A placeholder is [TYPE_index_contextid]. The index is per entity type in order of appearance, and the same value always gets the same token — a name that repeats stays one referent, so the model can still reason about it:

guard.scan("Ali wrote to Veli; reply to Ali").sanitized_text
# '[PERSON_1_4c81…] wrote to [PERSON_2_4c81…]; reply to [PERSON_1_4c81…]'

The index restarts at 1 for every scan, so it stays short; the context id is drawn once per scan (result.context_id) and shared by that scan's tokens. See Why the context id — it is the difference between a misrouted answer restoring the wrong person's data and restoring nothing at all.

Which layer found the value does not matter: anonymization runs after detection, so a span from regex, SpaCy NER or the LLM layer is tokenized and restored the same way. Only the confidence in the source list gives the layer away — 1.00 for a checksum-validated regex match, 0.85 for a model-based one.

Restoring¶

answer = call_llm(result.sanitized_text)
print(result.restore(answer))

I have emailed ali@example.com about the charge on 4532 0151 1283 0366.

--- Sources ---
[1] [EMAIL_1_9f3a2c8b71d4] → ali@example.com (EMAIL · tokenize · confidence 0.97)
[2] [CREDIT_CARD_1_9f3a2c8b71d4] → 4532 0151 1283 0366 (CREDIT_CARD · tokenize · confidence 1.00)

Printing a RestoredText appends that source list: each placeholder that was put back, in the order it appears in the text, with the filter that caught it, the action applied, the value it stood for, an xN count when it occurs more than once, and the detection confidence.


`.text`	the restored text alone
`.sources_block()`	the list alone — `title=` renames it, `notes=False` drops the trailing summary
`.with_sources()`	text + list; what `str(restored)` returns
`.substitutions`	the same information as `Substitution` records
`.unrestored`	detected values that were not put back, and why
`.is_complete`	`False` if a placeholder was left sitting in the text (ambiguous or foreign)

restore() with no argument reverses sanitized_text itself, which round-trips back to the original input — a cheap way to assert a policy is reversible in tests.

Why the context id¶

Two requests arriving together both hold an "EMAIL number 1". If placeholders were just [EMAIL_1], the two scans would produce the same string, and restoring one request's answer against the other's result would match it and substitute the wrong person's address — silently, with no error and nothing in unrestored. That is the worst outcome this library can produce.

The context id makes those tokens distinct, so the mistake has nothing to match:

mine   = guard.scan("Ali: ali@example.com")      # [EMAIL_1_9f3a2c8b71d4]
theirs = guard.scan("Beyza: beyza@example.com")  # [EMAIL_1_4c81b70a2e55]

theirs.restore("Mailed [EMAIL_1_9f3a2c8b71d4].")
# text        unchanged — nothing was substituted
# unrestored  [UnrestoredValue(entity_type='EMAIL', reason='foreign', …)]
# is_complete False

The id is 12 hex characters — 48 bits, so with a hundred thousand results alive at once the chance any two collide is about 1.8e-5, and request-scoped use puts it near 1e-9. It is drawn per scan, needs no shared counter, and holds across processes.

Two knobs:

result.restore(answer, strict=True)          # raise ContextMismatch instead of reporting
turn2.restore(answer, also=[turn1])          # an earlier turn's tokens are legitimate here

strict=True is the right default for a request handler. also= is for multi-turn exchanges, where an answer may carry placeholders from a previous scan in the same conversation — those are restored too, and anything else still raises.

A mangled token is covered by the same mechanism: if the model returns [EMAIL_1_9f3a2c8b71d5] for [EMAIL_1_9f3a2c8b71d4], there is no match, so the value simply does not come back. The model's accuracy decides how much is restored; it can never decide whether what comes back is correct.

Prompting the model to keep placeholders¶

Restoring needs the model to reproduce the placeholders verbatim. Whether it does is a property of the model — and, more than anything, of the instruction you give it. Measured on a local Qwen2.5-1.5B with the same text and three runs each:

instruction	placeholders preserved
"Copy every placeholder in square brackets EXACTLY as written."	0 / 9
"The text contains placeholders like `[EMAIL_1]`. Copy every placeholder EXACTLY as written, character for character. Never invent placeholders."	6 / 9

One model, three runs per instruction, nine placeholder slots each: the direction is worth acting on, the exact rate is not. A starting point:

INSTRUCTION = (
    "The text contains placeholders like [EMAIL_1_9f3a2c8b71d4]. "
    "Copy every placeholder EXACTLY as written, character for character. "
    "Never invent placeholders and never renumber them."
)
answer = call_llm(INSTRUCTION + "\n\n" + payload.sanitized_text)

Two things this measurement settled:

The context id does not cost recall. The same comparison with the id stripped out preserved fewer placeholders, not more — consistent with the failure mode seen elsewhere, where a model turns [CREDIT_CARD_1] into [CREDIT_CARD_2] but copies [CREDIT_CARD_49] untouched. A trailing hex suffix reads as an opaque identifier to copy rather than an item to renumber.

A model too weak to copy placeholders is not a safety problem. Small models paraphrase them away or drop them entirely; Qwen2.5-0.5B and SmolLM2-135M both did in testing. Nothing wrong is substituted — the value simply does not come back, and it is reported in .unrestored. Check it, and decide what your application does about a partial restore:

restored = payload.restore(answer)
if restored.unrestored:
    log.warning("partial restore: %s", [(u.entity_type, u.reason) for u in restored.unrestored])

Which actions can be reversed¶

Action	Reversible	Why
`tokenize`	always	one placeholder per distinct value, by construction
`surrogate`	always	one stand-in per distinct value within a scan, by construction
`hash`	in practice	a salted digest differs per distinct value
`redact`	only if unique	every value of a type collapses onto `[TYPE]`
`mask`	only if unique	two values can mask to the same string
`warn`	nothing to do	the original text was never replaced

A placeholder that stood for more than one value is not guessed: it is left in the text and reported in .unrestored with reason ambiguous (and called out under the source list). Scanned with a one-way action and want a reversible view without re-detecting? result.reapply(Action.TOKENIZE) reuses the spans and re-runs only the anonymization stage.

Keep the payload uniform¶

A guard usually mixes actions, and with_llm() in particular switches on its own entity policy — one whose defaults include warn. warn reports a value without replacing it, so a scan you believe is masked can hand the model a phone number in the clear:

[PERSON:5687e6a708553da8], [EMAIL_1_9f3a…], tel +90 532 123 45 67
  PERSON  hash        EMAIL  tokenize        PHONE  warn  ← still in the text

Deriving a uniformly tokenized payload closes that, and makes every token short and alike while it is at it. Detection is not repeated, only the cheap anonymization stage:

result  = guard.scan(text)
payload = result.reapply(Action.TOKENIZE)   # every span becomes [TYPE_N]

answer = call_llm(payload.sanitized_text)
print(payload.restore(answer))              # restore through the same object

mixed   : [PERSON:5687e6a708553da8], [EMAIL_1_9f3a…], tel +90 532 123 45 67
uniform : [PERSON_1_0c29…], [EMAIL_1_0c29…], tel [PHONE_1_0c29…]

Restore through the object you sent — payload, not result: the placeholders in the answer are the ones reapply produced.

Surrogates: realistic stand-ins¶

A model reasons better over Mehmet Kaya than over [PERSON_1_9f3a…], and a system that expects an e-mail address in a field wants something shaped like one. The surrogate action replaces a value with a stand-in of the same shape: a name from the guard's locale for a name, an address on a reserved domain for an e-mail, a phone number that keeps its country and area code, a card number that passes Luhn, an IBAN that passes mod-97, a TC number that passes its check digits, an address in a TEST-NET range for an IP.

guard = (
    Wardcat(salt="s")
    .with_ner(language="tr")
    .with_locale("tr")                      # names and formats; YAML: locale: tr
    .add_entities([Entity.PERSON, Entity.EMAIL, Entity.CREDIT_CARD], action=Action.SURROGATE)
)
result = guard.scan("Ahmet Yılmaz, ahmet@firma.com, kart 4111 1111 1111 1111")
result.sanitized_text   # "Selin Kaya, ruby.hayes@example.net, kart 4532 8811 0093 4166"
result.restore(answer)  # the same round trip as tokenize

Two rules make them usable. The same value gets the same surrogate under the same salt in every scan, so a person keeps one name across documents and a record can be linked on it; a different salt gives unrelated surrogates. And no two values share a surrogate within one scan, so restore() is never ambiguous. A type with no generator (DATE_OF_BIRTH, ADDRESS, …) falls back to tokenize, and the violation's action says tokenize — a result never claims a substitution that did not happen. supports() in wardcat.surrogates lists what has a generator.

Surrogates look real, and nothing in the text marks them as substitutes. That is the point and the risk: a downstream reader may take them for real data, and a generated name will sometimes be a real person's. Use them where a model or a system needs the shape, and keep tokenize where the reader must be able to tell a placeholder from a value. See the security guide.

Handing the map to another process¶

restore() needs the ScanResult. When the answer comes back somewhere else — a queue worker, another request — carry the map yourself:

vault = result.token_map     # {'[EMAIL_1_9f3a2c8b71d4]': 'ali@example.com'}
context = result.context_id  # carry it too, to spot a token from another scan

The reverse map is raw PII

token_map, restore() and the source list all carry the original values by design — that is what reversibility means. Keep them on the trusted side, out of logs and telemetry (redacted() stays the safe view), and prefer a one-way action when you do not need the values back. Placeholder numbering is per scan, so a ScanResult is the only thing that can reverse its own text.