An assistant that answers questions on the EU AI Act citing the articles, built on open-source LLMs and evaluated with metrics and confidence intervals.
In my first AI Act prototypes the answers looked good, but I had no way to know how good they were. In the legal domain answering is not enough: the system must cite the right provision and say "not in the text" instead of inventing a rule.
AI Act RAG Lab rebuilds the pipeline from scratch and measures every choice on a 63-question gold set that validates itself against the text of the regulation.
Rebuilds articles, paragraphs and points: 113 articles and 13 annexes become 856 passages, each with its own citation.
BM25 with Italian stemming and multilingual e5 embeddings, fused with Reciprocal Rank Fusion and reranked by a cross-encoder.
Llama 3, Mistral or Qwen (via Ollama or Hugging Face) answer only from the context, cite [Art. N] or abstain.
Retrieval on 55 questions answerable from the regulation, relevance judged at article level.
recall at equal context size: paragraph passages vs whole articles
MRR over fixed-length text windows
MRR from Italian stemming in BM25
| Retriever / chunking | Hit@1 | Recall@5 | MRR@10 | Recall at 6k chars |
|---|---|---|---|---|
| BM25 + stemming / by paragraph | 0.71 | 0.84 | 0.78 | 0.86 |
| BM25 no stemming / by paragraph | 0.58 | 0.75 | 0.69 | 0.85 |
| BM25 + stemming / whole articles | 0.62 | 0.88 | 0.72 | 0.66 |
| BM25 + stemming / 1000-char windows | 0.64 | 0.82 | 0.59 | 0.82 |
The differences in the boxes come from a paired bootstrap on the same questions (95% confidence interval above zero). With 55 questions intervals are wide: the code reports them for every metric.
Looking at Recall@5 alone, whole articles seem the best choice (0.88). But five articles can take 40,000 characters: at the same prompt size far fewer fit, and recall drops to 0.66 against 0.86. A badly chosen metric would have led to the wrong design.
Each training example holds the right passage plus three distractors chosen by BM25; in 20% of the examples the right passage is missing and the expected answer is an abstention. The model learns a behaviour (read, cite, abstain) rather than facts, which is why the provisions used for evaluation are kept out of training.
Training uses LoRA and QLoRA with PEFT and TRL, with the loss computed on the answer only. It is designed for a free Colab T4 GPU: 7B models such as Mistral are trained quantised to 4 bits (QLoRA).
Multi-step compliance analysis for recruiting systems, which the AI Act classifies as high-risk (Annex III, point 4).
Audit report generated with RAG over a vector database: risk classification, GDPR profile and compliance action plan.