Why Self-Hosted PII Tools Fail Compliance Audits
GDPR requires proof. You must show that PII removal was done the same way every time. DPA auditors check this. They want to see a clear, consistent method used across all data.
Self-hosted Presidio has a real problem here. It is not a config issue. It is a core limit of self-hosted NLP tools.
What Is Environment Drift?
Self-hosted Presidio runs in dev, staging, and production. Each of these can behave in a different way. So the same input can produce different results in each one.
This is called environment drift. It has four main causes.
Model Version Drift
spaCy models are versioned. Model en_core_web_lg 3.4.4 and en_core_web_lg 3.5.1 were trained on different data. They also use different designs. So the same doc can give different NER results with each version.
A common setup looks like this:
- Dev:
en_core_web_lg 3.4.4— installed at project start - Staging:
en_core_web_lg 3.5.0— updated during routine work - Production:
en_core_web_lg 3.5.1— updated during a security fix
That is three setups. Three model versions. Three different detection results. Tests pass in staging. But production runs a different model. So the gap stays hidden.
Dependency Version Drift
spaCy 3.4.x and 3.5.x differ in how they split sentences. That change affects how names are found near sentence breaks. These changes are in the spaCy release notes. But most teams do not check them for PII impact.
Configuration Drift
Score thresholds set in dev may not carry over to production. Custom word lists can also differ between setups. These gaps are common. They are rarely tracked. See our GDPR compliance guide for what auditors look for.
Hardware Differences
Math in NLP models is not identical across all CPUs and GPUs. A consumer laptop and a server can give slightly different score results. So some names may be found on one machine but not on another.
A Real Audit Finding
A bank tested their self-hosted Presidio setup.
Test setup: Presidio with spaCy 3.4.4 on the staging cluster. Live setup: Presidio with spaCy 3.5.1 on the production cluster.
They ran the same set of docs through both. Then they compared the results. The finding: 3% of docs had different PII removal results. Some names were caught in staging but not in production. Some had different detected text spans.
The audit finding was direct: "The firm cannot show consistent use of technical PII removal measures due to setup-specific differences in detection output."
GDPR Article 32 requires proper technical measures. EDPB rules on PII removal require consistency and repeatability. A 3% rate across 100,000 docs per month means 3,000 docs with inconsistent results each month. Some are false negatives. PII that staging would catch stays in the live output. That is a compliance failure.
The bank then moved to managed SaaS. The audit finding was closed. See our security and compliance page for how managed setups handle this.
Why Managed Services Are Different
A managed service runs one engine version. All users run the same version at the same time. Model updates are applied from one place. Config is also managed from one place, with a full change log. User hardware does not affect the results.
So the same doc processed today gives the same result next month. If the engine version changed, that change is logged and versioned.
The audit trail difference is key.
Self-hosted audit trail:
- "Used Presidio 2.2.35 with spaCy
en_core_web_lg 3.5.1on Ubuntu 22.04." - Was this the same version as in staging? Unknown.
- Has the model changed since this doc was processed? Unknown unless tracked.
- Is the score threshold the same as in testing? It depends on config management.
Managed service audit trail:
- "Used anonym.legal API, engine version 4.22.1, at 2025-03-15T14:22:31Z."
- Same version for all users? Yes.
- Has it changed? Engine versions are pinned. Version 4.22.1 always means the same engine.
- Is the config repeatable? Yes. Preset ID is logged. Config at that version can be retrieved.
The managed trail is clear. The self-hosted trail needs careful tracking that most teams skip.
How to Improve Self-Hosted Consistency
If self-hosting is required, you can reduce drift with four steps.
First, pin model versions. Lock exact model versions in all deploy files. Block auto-updates. Track versions in source control.
Next, freeze container images. Build Docker images with exact model versions baked in. Tag each image with the model version, Presidio version, and date. Do not update base images without testing first.
Also, keep config in code. Store all Presidio settings in files tracked in version control. This includes detectors, score thresholds, and active languages. Deploy config with the app.
Finally, test across setups. After any update, run a fixed test doc set through the new setup. Compare results to a stored reference. Automate this check. See the FAQ for common questions about automated PII regression testing.
These steps help. But they also add work. A managed service gives the same consistency without the extra effort.
The Bottom Line
Consistent PII removal does not appear on product sheets. But it becomes critical when auditors ask for evidence.
Without active care, self-hosted PII tools drift. Version changes add quiet gaps. Those gaps show up as audit findings.
Managed services provide consistency by default. The engine runs from one place. User setups do not affect results. For compliance-focused teams, this is a direct advantage.