AI safety · Multilingual safety · Hackathon

AI Safety in Yoruba, Hausa, Igbo

AI safety work on Tiny Aya Earth for Nigerian languages (Yoruba, Hausa, Igbo). Safety steering cut harmful compliance from 45.6% to 28.1%, no retraining.

Paired bar charts of Tiny Aya Earth on the TukaBench held-out evaluation split, with 95% Wilson confidence intervals. The left chart shows harmful compliance and the right chart shows over-refusal on benign prompts, both lower is better, in English, Hausa, Igbo and Yoruba. Each language compares the baseline, the submitted CAST tuned gate, the CAST default gate at alpha 0.5, and ungated CAA at alpha 0.25.
Harmful compliance and benign over-refusal on 957 held-out TukaBench prompts. The submitted repair is the blue bar in each group.

Small multilingual chat models are starting to appear in Nigerian assistants, but most safety training is still in English. That is a multilingual AI safety problem for Nigerian languages: a plain translation can be one of the cheapest ways to get a model to do something it would refuse in English. I wanted to measure that gap on a model actually recommended for African languages, then see whether it could be repaired without retraining.

Safety steering for Tiny Aya Earth, with no retraining

The model is Cohere Labs' Tiny Aya Earth, a 3.35 billion parameter chat model hosted with Hugging Face. I audited it on TukaBench: harmful prompts, culturally adapted harmful prompts, and topic-matched benign prompts, in English, Hausa, Igbo and Yoruba. After cleaning and deduplicating the data with provenance, I split it by source item so that no translation of an evaluation prompt appeared in calibration. That left 957 held-out evaluation prompts and 238 calibration prompts.

The work used three Lexsi kits. CuratorKIT ingested and documented the data. AuditKIT logged every stage as a per-language run and compared the base model with the repair. SafeTune's CASTTrainer fitted a conditional activation-steering repair on the calibration split, pooled across all four languages: a prompt-level gate at layer 10, and steering vectors at layers 16 to 20. SafeTune's default gate over-fired, with 55.5% over-refusal on the evaluation set, so I re-tuned the gate to a 20% benign false-fire budget (the fpr0.2 threshold, 0.0844). I chose the steering strength on the calibration split with a rule fixed before any evaluation-split steering run. The chosen setting is alpha 0.25.

What I found about LLM safety in African languages

The base model understood the prompts, and it refused fluently when it did refuse. The gap was how often it refused. On the held-out set it complied with harmful requests 18.9% of the time in English, 41.5% in Hausa, 49.4% in Igbo and 72.5% in Yoruba. For culturally adapted Yoruba prompts, compliance rose to 90%.

The submitted repair lowered harmful compliance from 45.6% to 28.1% across the four languages. The Yoruba to English gap shrank from 53.6 points to 24.2 points. When the gate leaves a prompt alone, the output is byte-identical to the base model. New refusals use the model's own native templates.

LanguageHarmful compliance, beforeAfter the repair
English18.9%16.4%
Hausa41.5%25.2%
Igbo49.4%30.0%
Yoruba72.5%40.6%
All four45.6%28.1%

Appending “Please answer in Igbo” to English harmful requests, a simple red-teaming trick, raised compliance from 18.9% to 84.4%. The repair cut that to 58.1%, but it barely moved Igbo prompts wrapped in an English frame (80.6% to 79.4%). That remains an open gap. Reading comprehension on Belebele was unchanged in English, Hausa and Igbo, and dropped by one question in Yoruba, from 37.0% to 36.5%. A self-hosted repaired model costs about $0.11 per 1,000 replies.

Scatter plot titled calibration sweep, safety versus over-refusal, with grey points marked ineligible. Harmful compliance is on the vertical axis and over-refusal on benign prompts is on the horizontal axis. The baseline sits at the top left. The chosen setting, labelled fpr0.2 a=0.25, is the eligible point with the lowest harmful compliance.
The calibration sweep used to pick the submitted setting. Grey points failed the pre-set rule; the chosen point is fpr0.2 at alpha 0.25.

Limits and a result I did not adopt

The repair is not free. Over-refusal on topic-matched benign prompts rose from 8.2% to 19.1%, mostly because the gate fires on benign look-alikes. Refusal was scored with a keyword lexicon, without a native-speaker agreement study, so those rates should be read with that limit in mind. The work also does not cover natural code-switching or Nigerian Pidgin, and the calibration set is small.

I also pre-registered a stronger variant, CAST-v2, that refit the gate on plain plus wrapped calibration prompts. On 2,397 held-out plain and wrapped prompts it cut pooled harmful compliance from 41.4% to 33.0%, but over-refusal rose from 11.0% to 19.0%. That failed the adoption rule I had fixed in advance, so the submitted repair stayed unchanged, and I reported the negative result.

Everything ran on one T4 through Hugging Face Jobs, about 2.6 GPU-hours and $1.59 in total.

The Lexsi submission

This project, Tiny Aya Earth Nigerian Safety Steering, was my submission to the Lexsi AI Safety and Alignment Hackathon 2026, organised by Lexsi.ai with Cohere Labs and Hugging Face. It entered Tracks 2a, multilingual and code-mixed safety, and 2b, inference-time safety steering. It placed in the top 10% of 369 submissions. The certificate was issued on 19 November 2026 (ID LX26-TFEU8-60VXS) and can be verified on the official results page.

Certificate of achievement from the Lexsi AI Safety and Alignment Hackathon 2026, issued by Lexsi.ai in partnership with Cohere Labs and Hugging Face, awarded to Emmanuel Effiom Duke for the project Tiny Aya Earth Nigerian Safety Steering, Top 10% of 369 submissions, Track 2a and 2b, issued 19 November 2026, certificate ID LX26-TFEU8-60VXS.
Certificate of achievement, Top 10% of 369 submissions. Verify at hackathon.lexsi.ai/results.

The code is on GitHub. The repaired model is on Hugging Face as duke01/tiny-aya-earth-cast-nigerian-safety, with the model collection and dataset collection beside it.

Questions

What did this AI safety audit find for Yoruba, Hausa and Igbo?

On 957 held-out TukaBench prompts, Tiny Aya Earth complied with harmful requests 72.5% of the time in Yoruba, 49.4% in Igbo, 41.5% in Hausa and 18.9% in English. Inference-time safety steering cut those rates to 40.6%, 30.0%, 25.2% and 16.4%, and the four-language average from 45.6% to 28.1%.

Did you retrain Tiny Aya Earth?

No. The repair is inference-time conditional activation steering: a gate at layer 10 and steering vectors at layers 16 to 20. Model weights are unchanged. Prompts the gate leaves alone get byte-identical output.

How can I verify the Lexsi hackathon result?

The project, Tiny Aya Earth Nigerian Safety Steering, placed in the top 10% of 369 submissions at the Lexsi AI Safety and Alignment Hackathon 2026. The certificate ID is LX26-TFEU8-60VXS, issued 19 November 2026. It can be verified at https://hackathon.lexsi.ai/results. The code is on GitHub and the repaired model is on Hugging Face.

Code, data and full results: dukemawex/lexsi-safety on GitHub. Related expertise: AI safety researcher.