Matteo Panzeri/ ← index

Case study · 2026-07-29

A compliance tool that has to be wrong out loud

I built a free AI Act self-check for Italian SMEs, where being confidently wrong has legal consequences for someone else. The tool is the boring part. I froze the test cases before writing any rule, made the privacy promise a test that fails CI, then pointed adversarial passes at my own public claims. They killed four of them, including a false statement in my own README.

Matteo Panzeri · AI-for-security research

Written with AI assistance; the tool itself was built with it, and its own start screen discloses that it is an AI-based system. The audits referenced here were independent adversarial LLM passes, not third-party human review: no lawyer outside the project has validated the rules, and that remains the largest gap. Every legal date is quoted from the Official Journal text; the code, rules, frozen scenarios and audit guide are public.

The problem with a compliance triage

The AI Act (Regulation (EU) 2024/1689) reaches small businesses that have no compliance function. Its transparency duties begin applying on 2 August 2026, and the Digital Omnibus (Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026, in force from the 27th) moved several dates. A tool that answers “what applies to me, and by when” is useful. A tool that answers it confidently and wrongly is worse than nothing, because the user cannot tell the two apart.

The design constraint follows: the system must be able to be wrong out loud. Not accurate; no v1 is. It has to be structurally unable to hide a gap. Three mechanisms carry that, and each is a testable artifact rather than a promise.

1 · Freeze the test cases before writing the rules

Fourteen legal scenarios (facts in, expected verdict out, each with its own citation) were written and hash-frozen before any rule existed. The git record shows the scenarios committed at 16:54 and the first rules at 20:56 the same day, four hours later, and exactly one commit in the repository’s history touches that directory: the freeze itself. A checksum file plus an independent copy of the same fourteen hashes in the decision log means a silent edit would have to defeat two records.

What does the freeze actually buy? It provably stops the scenarios being edited to fit the rules, which is the failure mode of every self-graded benchmark. It does not stop the rules being written to fit the scenarios: I could see them while writing. The project’s decision log states this in as many words and lists the counter-measures (author each rule from the statute text first, write a justification for why the opposite verdict would violate the cited provision, anchor every date to a source table instead of retyping it). An auditor put it more sharply than I had: commit timestamps are author-controlled, and the server-side record only begins at the first push. The ordering is evidence, not proof.

2 · Make the privacy claim a test that can fail

The tool runs entirely in the browser, with the Python engine compiled to WebAssembly, so answers about a client’s possibly-unlawful practices never reach a server. That claim is worth exactly as much as its enforcement, so it has two: a Content Security Policy forbidding connections to external origins, and an end-to-end test that drives the whole questionnaire and asserts zero non-self requests and zero requests after load, running in CI on every push. Add an analytics snippet and the gate goes red.

An auditor found the hole in that arrangement: the deploy workflow was independent of the test workflow, so a push whose privacy test failed would still have shipped. Deployment is now gated on the test workflow. The claim was true; I had never checked whether a failing test could block a deploy.

The same pass refused two overstatements I had shipped. A CSP does not stop a top-level navigation, only the data-carrying primitives (fetch, XHR, WebSocket, beacon); and on GitHub Pages the policy can only be delivered in a <meta> tag, so header-only directives are silently ignored. Both are now written on the transparency page instead of left for a reader to discover.

3 · A verdict that refuses to guess

Evaluation is three-valued: true, false, unknown. Unknown propagates, and the invariant is that no “compliant” verdict may rest on an unknown fact, including the case that sinks naive implementations, where any(TRUE, UNKNOWN) evaluates to true in the three-valued logic itself, masking an unknown that could still matter. Those are demoted to “undetermined.” A perturbation harness omits and flips every fact of every frozen scenario in turn, 166 perturbations in all, and not one produced a “compliant” resting on missing data.

Users can answer “I don’t know” to any question; it is a first-class answer that leads to “needs human review.” That trade is deliberate: an undetermined verdict costs the user time, and a false compliant is the only output that can cause loss.

What the adversarial passes caught in my own claims

Publishing corrected work is table stakes; publishing the record of the corrections is the demonstration. Before the launch post went out, an adversarial pass read it against the code. It confirmed 21 findings, four blocking.

Then five auditor personas (a lawyer, a security auditor, a DPO, an institutional re-hoster, a skeptical engineer) were pointed at the live tool with no context and told to verify the public claims or fail trying. All five concluded they would trust it. All five found defects:

Verification took the five of them between 32 and 85 minutes, most of it spent working out where to look. That friction became a short audit guide written from their actual obstacles, whose longest section is titled What this does not prove.

The bug the harness caught that I could not

One more, because it is the cleanest argument for the method. The engine ships as a deterministic archive whose sha256 anyone can reproduce and compare with the live site. It reproduced perfectly on my machine, and CI rejected it on the first push. ZipInfo.create_system defaults to 0 on Windows and 3 on Unix, so identical inputs produced different bytes on different platforms: the “anyone can verify this” property would have been false for every auditor not using Windows. A gate built to catch staleness caught a portability defect I did not know existed. It surfaced only because the gate runs on a machine that is not mine.

What this does and does not show

It is worth being exact about which half of this transfers. Not the rules: they cover a triage slice of one regulation and carry every limit below. What transfers is the arrangement around them. Freeze the test cases before the rules exist. Make the privacy claim a test that fails CI. Let the verdict refuse to guess rather than round an unknown up to compliant. Point the adversarial passes at your own public claims, not only at your code. None of that made the tool correct, and nothing here should be read as saying it did. What it did was surface four blocking findings in my own launch post, the defects five auditor personas found on a live tool they were given no context for, and a portability bug I would not have found by being more careful.

Limits, stated as the project’s spine

Reproducibility

Tool: matte1782.github.io/ai-act-sme-checker. Code, rules, frozen scenarios, decision log and audit guide: github.com/matte1782/ai-act-sme-checker (EUPL-1.2, re-hostable verbatim). bash scripts/check.sh runs the whole gate: 440 tests, the fourteen frozen scenarios, and the bundle-integrity check. Every applicability date is quoted from the OJ text of Regulation (EU) 2026/1744, a copy of which is in the repository with its hash recorded.

Sources