Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric defaults. They treat model alignment as a binary, culture-agnostic property. This paradigm optimizes for global consensus but masks critical regional laws, socio-linguistic nuances, and cultural taboos. This leaves Vision-Language Models (VLMs) vulnerable in global deployments.

The field of AI safety is rapidly maturing. It is moving from theoretical discussions of existential risk to the practical engineering task of ensuring models behave reliably. Currently, the state of the art involves stressing models with massive datasets of "universal" harms. These include instructions for building weapons or generating explicit violence. This ensures models adhere to a global safety baseline. However, a critical question remains: does a model that is "safe" in San Francisco remain safe and respectful in Singapore, Seoul, or Karachi?

The Pluralis v0.1 paper argues that it does not. The researchers suggest that by optimizing for a single, globalized definition of safety, we are creating a massive blind spot. A model might pass every standard safety test while simultaneously advising a user to perform an illegal act in a specific jurisdiction. It might also commit a profound cultural taboo that leads to significant social friction.

The collapse of culture into global consensus

Most existing safety benchmarks suffer from the "streetlight effect" (a bias where people search for things where it is easiest to look). They focus on the easily visible, universally recognized hazards. They ignore the nuanced, localized ones. As the authors note, current paradigms tend to collapse distinct cultural judgments into a single, culture-agnostic label. This approach assumes that what is harmful or taboo is a static property. It ignores that these things are conditioned on local law, religion, and social norms.

This becomes particularly problematic in multimodal settings (where models process both text and images). A text-only prompt might be entirely benign. An image might be completely innocuous. But their combination can trigger a specific hazard. For example, as illustrated in, asking "Should I pack this?" while showing an image of an e-cigarette is a routine question in many places.

However, in Singapore, Taiwan, or India, it touches upon specific legal restrictions regarding possession and use. Standard benchmarks prioritize global consensus. Consequently, they structurally lack the localized grounding required to catch these synergistic triggers.

A culture-first multimodal pipeline

To move beyond mere translation of Western datasets, the authors implement a "culture-first" methodology. Instead of taking an English dataset and translating it, they involve regional experts to conceptualize hazards natively. The pipeline described in proceeds in four distinct stages:

1. Native Concept Creation: Regional experts author multimodal prompt pairs (text + image) grounded in local legal, religious, and social norms.
2. Multimodal Pairing: Raters source or synthetically generate images. These images, when paired with the text, create a specific cultural or legal hazard.
3. Human-Led Localization: Validators manually translate these pairs into target languages (e.g., Malay, Tamil, or Traditional Chinese). They specifically adjust honorifics (formal titles) and politeness levels to preserve social nuance.
4. Dual-Axis Evaluation: The resulting dataset is evaluated on two separate axes: Safety (compliance with regional laws and prevention of physical harm) and Cultural Appropriateness (adherence to social etiquette and religious taboos).

Crucially, the authors decouple these axes. As shown in, a response can be "Safe" but "Inappropriate." For example, gifting a clock is physically harmless. However, in Chinese-cultural contexts, it is a taboo. This distinction is vital for diagnostic purposes. It allows developers to see if a model is failing because it is dangerous or simply because it lacks cultural fluency.

Measuring the gap in frontier models

To operationalize this at scale, the authors developed Judge-Pluralis. This is an agreement-gated LLM-as-a-Judge ensemble (a system where multiple large language models act as evaluators). They used an Automatic Prompt Optimization (APO) loop. In this loop, the judge's instructions and few-shot examples (a small set of demonstration examples) are refined based on discrepancies with human raters. This optimization increased relative safety accuracy by 4.7%. It also increased relative cultural accuracy by 18.9%. These improvements mean the automated judge becomes significantly more reliable at catching subtle cultural errors.

When applying this evaluator to frontier Systems Under Test (SUTs), the results reveal significant vulnerabilities. The authors measure performance using an absolute performance score ($SSUT$), which is the mean score across $N$ prompts. They use a relative ratio score ($S^*$) to compare models against a baseline. This categorizes performance into "Good," "Fair," and "Poor" zones.

The preliminary findings are striking. As seen in, the uncertainty in automated grading is high.

Grade bands often span multiple tiers. However, the qualitative failure modes are clear. The researchers identify three recurring patterns in : image misidentification leading to harm (e.g., mistaking toxic plants for edible ones), missing the interaction between an item and its local context, and inadequate refusals where the model hedges instead of providing a clear warning.

Blind spots in the automated judge

While the Pluralis framework is a significant step forward, it is far from a complete solution. I notice several limitations that should temper any immediate optimism. First, the automated evaluator is still imperfect. The authors admit that Judge-Pluralis misses approximately two-thirds of human-identified violations in low-base-rate systems. This means the benchmark is currently better suited as a triage tool than a definitive certification of safety.

Second, the "culture" being measured is treated as a static snapshot. In reality, cultural norms, laws, and taboos are moving targets. A model aligned with 2026 Singaporean law may become unaligned by 2027. Without a mechanism for continuous, versioned re-annotation, the benchmark risks becoming obsolete.

Finally, the dataset suffers from internal heterogeneity (meaning it is not uniform within a single category). A single label for "Korea" or "India" ignores the vast religious, ethnic, and linguistic diversity within those nations. The authors note that inter-annotator agreement varies wildly. It ranges from high agreement in Singapore to much lower levels in Korea and India. This suggests that "cultural appropriateness" is often a distribution of opinions rather than a single ground truth.

Verdict: A necessary, albeit noisy, foundation

Is Pluralis ready to replace current safety benchmarks? No. It is too noisy. The evaluator is still prone to significant false negatives. Also, the coverage is limited to a specific subset of the Asia-Pacific region.

However, the paper succeeds in its primary goal. It proves that "global" safety metrics are insufficient for global deployment. The transition from universal harms to locale-conditioned, contextually-triggered realities is a fundamental shift. If you are building models for international markets, you cannot rely on a single "Safety" score. You need to evaluate the specific intersections of modality, language, and local law. Pluralis provides the first meaningful framework for doing exactly that. It is a foundational step. It moves us away from the illusion of a universal moral consensus and toward a more rigorous, pluralistic understanding of AI risk.

Figures from the paper