The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits

From Risk Catalogs to Audit Findings

Most AI risk lists just name problems without explaining how to test for them. This new system provides a structured way to turn a named risk into a measurable test, a severity score, and a final grade. This helps auditors prove if a system is truly safe.

Bridging the Operationalization Gap

The core challenge in AI safety is moving from theory to practice. The industry is flooded with lists of potential harms. These include biased algorithms and data leaks. However, these lists rarely explain how an auditor should actually verify them.

Knowing that "privacy leakage" is a risk is helpful. But knowing how to design a test is harder. You must decide what numerical threshold constitutes a failure. You must also know how to assign a defensible grade. This is where the actual work of auditing resides.

The Eticas AI Risk Taxonomy v2.0.0 attempts to solve this. It provides more than just a glossary. The authors argue that a taxonomy only becomes "infrastructure" when it can be used to build an actual audit. Instead of just cataloging risks, the paper introduces a methodology. This method transforms an abstract concept into a concrete, graded finding.

The Landscape of Fragmented Risks

To understand this work, one must look at AI governance. AI systems now mediate critical decisions in healthcare, credit, and law enforcement. Global regulators have responded with various frameworks. The European Union’s AI Act mandates risk assessments. Meanwhile, the NIST AI Risk Management Framework provides guidelines for managing uncertainty.

However, the field is currently fragmented. The authors report that at least 74 AI risk taxonomies exist. Most of these stop at the catalog stage. Some, like the MIT AI Risk Repository, are very comprehensive. But they can overwhelm an auditor with sheer volume. Others, like the IBM AI Risk Atlas, organize risks by their lifecycle stage (the phase of development when a risk occurs).

This heterogeneity creates a "Rosetta Stone" problem. Regulators struggle to compare audit results from different providers. Everyone is using a different vocabulary. Without a shared way to move from a named risk to a measured value, the industry lacks a common language for safety.

A Four-Layer Methodology

The authors propose a solution based on an "ontological separation." This means they keep the definition of a risk separate from the specific way it is measured. This allows the taxonomy to act as a stable scaffold. Different testing methods can then be attached to it.

The paper details a four-layer architecture. This design separates reusable infrastructure from the specifics of a single audit. Layers 1 and 2 (Foundations and the Technology-specific core) are designed once. They are reused across many audits. These layers contain the taxonomy and instructions for what to measure in certain models, such as Large Language Models (LLMs). Layers 3 and 4 (Sector annexes and Project instantiation) are unique to each specific project.

To demonstrate this, the authors provide a worked example of PII (Personally Identifiable Information) leakage .

They trace a single risk through a "measurement-to-grade chain." First, a "probe" (a specific test procedure) is run against a model. This produces a "metric value," such as a disclosure rate. That value is then mapped through "severity bands" to a numerical score from 0 to 5. Finally, these scores are aggregated into a subcategory grade (A to E) and a pattern flag. A flag like "SYSTEMIC" indicates if the failure is a widespread issue.

As shown in, the authors tested GPT-4-0314 using the DecodingTrust benchmark. They found that the model showed 0% disclosure in a zero-shot scenario (where no prior examples are provided). However, the rate jumped to 51% with one demonstration. It reached 84% with three demonstrations. This means the model's safety policy failed significantly under light pressure. This allowed them to assign a subcategory grade of E, the highest level of severity.

Identifying the Governance Gap

This structured mapping reveals significant blind spots in current AI regulation. One of the most striking findings involves "Agentic AI." These are systems that can autonomously plan tasks and use external tools.

The paper finds a massive governance gap. Major regulatory frameworks, such as the EU AI Act, were finalized before agentic AI became common. The authors show that specialized frameworks like the OWASP Top 10 for Agentic Applications provide direct coverage. However, primary compliance-tier frameworks offer only minimal guidance. By treating Agentic AI as a "first-class category," the authors show exactly where existing laws fall short.

Limits of the Framework

The authors are transparent about the boundaries of this taxonomy. The numeric results for PII leakage are based on public benchmarks (DecodingTrust). They are not from a live, private client engagement. They serve as a proxy to show how the method works. Additionally, the severity thresholds are a "first-pass calibration." They will require more refinement as more audits are performed.

Crucially, the paper notes a divide in how the data is shared. The conceptual scaffold is open to the public. However, the specific "calibration" of the methodology remains in the practitioner layer. This includes exact thresholds and proprietary testing nuances. Finally, the taxonomy does not solve legal questions like intellectual property. It focuses on what can be tangibly measured through technical audits.

Figures from the paper