NLP & Tokenization¶

The olaverse.nlp module is the core of the SDK — production-ready tools for African language text processing, detection, diacritization, tokenization, and preprocessing.

pip install olaverse                # all NLP tools included (no GPU required)
pip install olaverse[deeplearning]  # adds LIDNeural5/LIDNeural25/LIDNeural608/LIDNeural5_1, diacnet-1.0/1.1/2.0, diactag-1.0/2.0
pip install olaverse[onnx]          # adds the int8 diactag backend
pip install olaverse[lid]           # adds LIDLite25 and LIDLite608 (fastText, 25 / 608 languages)
pip install olaverse[retrieval]     # adds Reranker, Embedder

Language Detection¶

Two models cover the same 5 languages at very different scales. Pick based on your latency and accuracy requirements.

LIDLite5 — Lightweight

Zero GPU · Instant

1.1 MB JSON model
0.014 ms per sentence
98.12% macro accuracy
TF-IDF + Logistic Regression
Pure Python — no torch, no transformers

LIDNeural5 — Neural

GPU Recommended

484 MB — XLM-RoBERTa 125M
13.3 ms per sentence (CPU/GPU)
98.96% macro accuracy
Fine-tuned on castorini/afriberta_large
Requires olaverse[deeplearning]

LIDLite5¶

Model Card: olaverse/lid-lite-5

from olaverse import LIDLite5, detect_language

# Class interface
detector = LIDLite5()
detector.predict("Bawo ni, se daadaa ni?")      # → 'yor'
detector.predict("Sannu, yaya kake?")            # → 'hau'
detector.predict("How far, wetin dey happen?")   # → 'pcm'

# Probability distribution over all 5 classes
detector.predict_proba("Kedu, ọ dị mma?")
# → {'eng': 0.037, 'hau': 0.024, 'ibo': 0.807, 'pcm': 0.021, 'yor': 0.112}

# Quick one-liner
detect_language("E kaaro, bawo ni?")             # → 'yor'

olaverse.nlp.LIDLite5 ¶

LIDLite5(model_path='lid-lite-5.json')

Lightweight, zero-dependency TF-IDF + Logistic Regression Language Detector for 5 languages: Yoruba ('yor'), Hausa ('hau'), Igbo ('ibo'), Pidgin ('pcm'), and English ('eng').

Methods:¶

predict ¶

predict(text)

Predict the language of the given text. Returns: 'yor', 'hau', 'ibo', 'pcm', or 'eng'.

predict_proba ¶

predict_proba(text)

Predict the language probabilities using softmax over logits.

olaverse.nlp.detect_language ¶

detect_language(text, model_path='lid-lite-5.json')

Detect the language of the given text using LIDLite5. Returns: 'yor' (Yoruba), 'hau' (Hausa), 'ibo' (Igbo), 'pcm' (Pidgin), or 'eng' (English).

LIDNeural5¶

Model Card: olaverse/lid-neural-5

pip install olaverse[deeplearning]

from olaverse import LIDNeural5

detector = LIDNeural5()
detector.load()  # downloads once from Hugging Face, cached after first run

detector.predict("Kedu ka ị mere today?")        # → 'ibo'
detector.predict("Ẹ káàárọ̀, ẹ káàbọ̀")        # → 'yor'

probs = detector.predict_proba("How far, wetin dey happen?")
# → {'pcm': 0.991, 'eng': 0.006, 'ibo': 0.001, 'hau': 0.001, 'yor': 0.001}

Batch Inference¶

predict_batch and predict_proba_batch run a single batched forward pass — much faster than calling predict() in a loop over a dataset.

texts = ["Bawo ni?", "Kedu ọ dị?", "How far?", "Sannu dai."]

# Returns a list of predicted language codes
langs = detector.predict_batch(texts)
# → ['yor', 'ibo', 'pcm', 'hau']

# Returns a list of probability dicts — one per input
probs = detector.predict_proba_batch(["Bawo ni?", "Kedu?"])
# → [
#     {'yor': 0.991, 'eng': 0.005, ...},
#     {'ibo': 0.987, 'eng': 0.008, ...},
#   ]

Per-language accuracy:

Language	Precision	Recall	F1-Score
Yoruba (`yor`)	99.60%	99.60%	99.60%
Hausa (`hau`)	99.60%	99.20%	99.40%
Igbo (`ibo`)	98.79%	98.20%	98.50%
Nigerian Pidgin (`pcm`)	99.20%	98.80%	99.00%
English (`eng`)	97.63%	99.00%	98.31%
Overall (Macro)			98.96%

olaverse.nlp.LIDNeural5 ¶

LIDNeural5(model_name='olaverse/lid-neural-5')

Bases: _HFSequenceClassifierLID

High-accuracy transformer-based language identifier for 5 Nigerian languages.

Base Model: castorini/afriberta_large (XLM-RoBERTa, 125M parameters) Fine-tuned on: Yoruba ('yor'), Hausa ('hau'), Igbo ('ibo'), Pidgin ('pcm'), English ('eng') Validation accuracy: 98.96% macro-F1

Requires: pip install olaverse[deeplearning]

LIDLite25 / LIDNeural25 — 25-language identification¶

New in v0.1.5

Beyond the 5 core Nigerian languages, LIDLite25 and LIDNeural25 cover 25 languages spanning Africa, Europe, and Asia — for products that need broader coverage than Yoruba/Hausa/Igbo/Pidgin/English.

Model Cards: olaverse/lid-lite-25 · olaverse/lid-neural-25.1 · olaverse/lid-neural-25.2

Both come in two checkpoints, tuned for different input lengths — pick the variant= that matches your traffic:

`variant=`	Use for
`"passages"`	Documents, articles, paragraph-length text
`"questions"` (default)	Search queries, chat messages, short user input

LIDLite25 is a CPU-only fastText classifier (sub-millisecond, ~5-10MB per checkpoint); LIDNeural25 is an XLM-RoBERTa-base sequence classifier — higher accuracy on short text (98.2% vs 97.3%), at the cost of needing transformers/torch.

pip install olaverse[lid]           # LIDLite25 (fastText)
pip install olaverse[deeplearning]  # LIDNeural25 (transformers)

from olaverse import LIDLite25, LIDNeural25

lite = LIDLite25(variant="questions")
lite.predict("What causes ocean tides?")   # → 'eng'

neural = LIDNeural25(variant="questions")
neural.load()
neural.predict_proba("What causes ocean tides?")
# → {'eng': 0.999, 'fra': 0.0003, ...}

Zulu/Xhosa confusion on short text

Both models score noticeably lower on Zulu/Xhosa short-text classification (F1 ~0.77-0.79) than every other language (≥0.98) — the two languages are closely related with substantial shared vocabulary. This holds across both architectures, so treat predictions between these two specifically with reduced confidence on short input.

olaverse.nlp.LIDLite25 ¶

LIDLite25(variant: str = 'questions')

Lightweight, CPU-only fastText language identifier for 25 languages. Sub-millisecond inference, ~5-10MB per checkpoint, no GPU required.

Two checkpoints for two input lengths (variant=): "passages" — long-form text (documents, articles) "questions" — short text (queries, chat messages) [default]

For higher accuracy at the cost of needing transformers/torch, see LIDNeural25.

Requires: pip install olaverse[lid]

Methods:¶

load ¶

load()

Download and load the fastText checkpoint (runs once; cached after first call).

predict_proba ¶

predict_proba(text: str) -> dict

Return probability distribution over all 25 languages.

Returns:

Name	Type	Description
`dict`	`dict`	{'eng': 0.99, 'fra': 0.005, ...} (ISO 639-3 codes)

predict ¶

predict(text: str) -> str

Predict the dominant language of the text (ISO 639-3 code, e.g. 'eng').

olaverse.nlp.LIDNeural25 ¶

LIDNeural25(variant: str = 'questions')

Bases: _HFSequenceClassifierLID

Transformer-based (XLM-RoBERTa, 125M parameters) language identifier for 25 languages — higher accuracy than LIDLite25, especially on short text, at the cost of needing transformers/torch.

Two checkpoints for two input lengths (variant=): "passages" — lid-neural-25.1, long-form text (documents, articles) "questions" — lid-neural-25.2, short text (queries, chat messages) [default]

Requires: pip install olaverse[deeplearning]

LIDLite608 / LIDNeural608 — 608-language identification (New in v0.4.1)¶

Model Cards: olaverse/lid-lite-608 · olaverse/lid-neural-608 · Full guide: Language Detection →

African-first identification across 608 languages and 36 scripts. LIDLite608 is a 37 MB CPU-only fastText model (~6,800 texts/s on one thread); LIDNeural608 is a 140M-parameter ModernBERT classifier, the most accurate of the two on short and conversational text. They share their labels (yor_Latn: ISO 639-3 plus script, and a zxx_Zxxx noise class) and two modes: mode="coverage" (default; every language equally likely, for corpus building) and mode="traffic" (shifted by real-world language frequency, for user input and routing).

from olaverse import LIDLite608, LIDNeural608

LIDLite608().predict("Ẹ kú àárọ̀, ṣé dáadáa ni?")                       # → 'yor_Latn'
LIDNeural608(mode="traffic").predict_batch(["Good morning", "Habari za asubuhi"])
# → ['eng_Latn', 'swh_Latn']

olaverse.nlp.LIDLite608 ¶

LIDLite608(mode: str = 'coverage')

Fast, CPU-only fastText language identifier for 608 languages, African-first.

37 MB, ~6,800 texts/s on one CPU thread. Labels are ISO 639-3 plus ISO 15924 script — 'yor_Latn', 'srp_Cyrl' — and 'zxx_Zxxx' marks numbers, URLs, code and other non-language text. (The 5- and 25-language classes return bare codes like 'yor'; use label.split('_')[0] for the code alone.)

Two modes in one model (mode=): "coverage" — every language equally likely; best per-language accuracy. Use it to build corpora or mine low-resource text. [default] "traffic" — scores shifted by each language's real-world frequency, so short or ambiguous input leans towards common languages. Use it for user input, chat and routing. In this mode the probabilities are spread over the 40 likeliest candidates.

For higher accuracy on short and conversational text see :class:LIDNeural608.

Requires: pip install olaverse[lid]

Quick start

lid = LIDLite608() lid.predict("Ẹ kú àárọ̀, ṣé dáadáa ni?") 'yor_Latn' LIDLite608(mode="traffic").predict("Bonjour mon ami") 'fra_Latn'

Methods:¶

load ¶

load()

Download and load the fastText model and priors (runs once; cached afterwards).

predict_proba ¶

predict_proba(text: str, top_k: int = None) -> dict

Probabilities for the likeliest languages, best first.

Parameters:

Name	Type	Description	Default
`text`	`str`	Any length, from one word to a document. Newlines are collapsed.	required
`top_k`	`int`	Keep only the `top_k` likeliest labels. `None` returns all 608 in coverage mode and the 40 candidates in traffic mode.	`None`

Returns:

Name	Type	Description
`dict`	`dict`	`{'yor_Latn': 0.99, 'ibo_Latn': 0.004, ...}`

Raises:

Type	Description
`ValueError`	if `text` is empty.

predict ¶

predict(text: str) -> str

The likeliest language of text (e.g. 'yor_Latn').

predict_batch ¶

predict_batch(texts: list) -> list

The likeliest language for each text. Calls :meth:predict in turn.

olaverse.nlp.LIDNeural608 ¶

LIDNeural608(mode: str = 'coverage', device: str = None, model_name: str = REPO_ID)

Bases: _HFSequenceClassifierLID

Transformer language identifier for 608 languages, African-first — the most accurate of the Olaverse LID models on short and conversational text.

A ModernBERT sequence classifier fine-tuned from jhu-clsp/mmBERT-small (140M parameters). Reads up to 512 tokens (about 200 words); longer text is truncated, so split long documents into ~200-word chunks. Labels are ISO 639-3 plus ISO 15924 script — 'yor_Latn', 'srp_Cyrl' — and 'zxx_Zxxx' marks numbers, URLs, code and other non-language text. (The 5- and 25-language classes return bare codes like 'yor'; use label.split('_')[0] for the code alone.)

Two modes in one model (mode=): "coverage" — every language equally likely; best per-language accuracy. Use it to build corpora or mine low-resource text. [default] "traffic" — probabilities shifted by each language's real-world frequency, so short or ambiguous input leans towards common languages. Use it for user input, chat and routing.

Runs on CPU, but a GPU helps: on CUDA the weights load in bfloat16.

Requires: pip install olaverse[deeplearning]. The model card asks for transformers>=5.14; an older version can fail while loading.

For a 37 MB CPU-only alternative with the same languages and modes, see :class:LIDLite608.

Quick start

lid = LIDNeural608() lid.predict("Habari za asubuhi") 'swh_Latn' LIDNeural608(mode="traffic").predict_batch(["Good morning", "Mo fẹ́ lọ sí ọjà"]) ['eng_Latn', 'yor_Latn']

Parameters:

Name	Type	Description	Default
`mode`	`str`	`"coverage"` (default) or `"traffic"`.	`'coverage'`
`device`	`str`	`"cpu"`, `"cuda"`, `"mps"` ... `None` or `"auto"` picks CUDA when it is available, else CPU.	`None`
`model_name`	`str`	Hugging Face model id.	`REPO_ID`

Methods:¶

load ¶

load()

Download and load the model (runs once; cached after first call).

predict_proba_batch ¶

predict_proba_batch(texts: list) -> list

Probabilities for many texts in one forward pass (see the base class).

LIDNeural5_1 — Nigerian-only, no English fallback¶

Model Card: olaverse/lid-neural-5.1

A compact (~31M parameter) classifier built on mist-encoder-base-ng, covering only the 4 main Nigerian languages — no English/"other" class.

from olaverse import LIDNeural5_1

detector = LIDNeural5_1()
detector.predict("Ina kwana?")   # → 'Hausa'

No English class

Out-of-set input (English or any other language) will be confidently mislabelled, most often as Nigerian Pidgin. If your input may include English, use LIDLite25/LIDNeural25/LIDNeural5 instead — this model always picks one of the four Nigerian languages.

olaverse.nlp.LIDNeural5_1 ¶

LIDNeural5_1(model_name='olaverse/lid-neural-5.1')

Bases: _HFSequenceClassifierLID

Compact language identifier for the 4 main Nigerian languages, built as a classification head on olaverse/mist-encoder-base-ng (ModernBERT, ~31M parameters).

Labels: 'Hausa', 'Yoruba', 'Igbo', 'Nigerian Pidgin'.

No English/'other' class — out-of-set languages (e.g. English) will be confidently mislabelled, most often as Nigerian Pidgin. Use LIDLite25 or LIDNeural25 instead if inputs may include English or other non-Nigerian languages.

Requires: pip install olaverse[deeplearning]

Diacritization¶

DiacNet and DiacTag restore tones and diacritical marks stripped from text — the critical front-end step for TTS, language learning, and NLP accuracy. The small DiacNet models cover Yoruba and Igbo with no extras; diacnet-1.0/1.1 and diactag-1.0 cover 10 languages each, and diacnet-2.0, diacnet-mini-2.0 and diactag-2.0 cover 11 — the same ten plus Arabic.

For new work start with a diactag model: it classifies each character instead of generating text, so it cannot alter your input beyond adding marks. diactag-1.0 beat diacnet-1.1 on 9 of 10 languages. The diacnet-2.x models align their output onto your text by default and, per the model cards, have a lower error rate than diactag-2.0 on 8 of the 10 Latin-script languages (at 582M / 300M parameters against 37.9M).

Available Models¶

Model ID	Language	Method	Speed	Accuracy	Size
`diacnet-yor-viterbi`	Yoruba	Viterbi n-gram	⚡ Fast	Good	~7 MB
`diacnet-yor-db`	Yoruba (dot-below only)	KNN backoff	⚡ Fast	Dot-below focused	~2 MB
`diacnet-yor`	Yoruba	BiLSTM	Medium	93.35% char	2.4 MB
`diacnet-yor-x`	Yoruba (full)	XLM-RoBERTa	Slow	82.46% word	503 MB
`diacnet-ig`	Igbo	KNN backoff	⚡ Fast	Good	~3 MB
`diacnet-1.0`	10 languages (see below)	ByT5 seq2seq	Slow	~0.02 median CER	~300 MB
`diacnet-1.1`	10 languages	ByT5 seq2seq	Slow	0.0002–0.2006 DER	~1.1 GB
`diactag-1.0`	10 languages	character tagger	⚡ 244 chars/s CPU	0.0132 DER, 1.0 compliance	150 MB / 38 MB int8
`diactag-2.0`	11 languages (+ Arabic)	character tagger	⚡ CPU-native	0.0130 DER on the 10 original languages, 1.0 compliance	152 MB / 38.6 MB int8
`diacnet-2.0`	11 languages	ByT5 text-to-text, 582M	Slow (GPU recommended)	mean DER 0.0117 (aligned)	2.3 GB
`diacnet-mini-2.0`	11 languages	ByT5 text-to-text, 300M	Slow (GPU recommended)	mean DER 0.0141 (aligned)	1.2 GB

Quick Functions¶

from olaverse import diacritize_yoruba, diacritize_yoruba_dot_below, diacritize_igbo

# Yoruba — full tonal diacritics (Viterbi, fast)
diacritize_yoruba("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# Yoruba — dot-below vowels only (KNN)
diacritize_yoruba_dot_below("Ojo lo si oja")
# → 'Ọjọ lo si ọja'

# Igbo
diacritize_igbo("Kedu ka i mere")
# → 'Kedụ ka ị mere'

Unified Diacritizer Class¶

Use Diacritizer when you need to choose a specific model, switch between neural backends, or process mixed-language input.

from olaverse.nlp import Diacritizer

# Default: fast Viterbi for Yoruba
d = Diacritizer(model="diacnet-yor-viterbi")
d.restore("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# High-accuracy neural (requires olaverse[deeplearning])
d_neural = Diacritizer(model="diacnet-yor-x")
d_neural.restore("Ojo lo si oja lana")

Auto-routing (`model="auto"`) — New in v0.1.4¶

Set model="auto" to skip manual language selection. LIDLite5 detects the language on each call and routes to the correct backend — Yoruba → diacnet-yor-viterbi, Igbo → diacnet-ig. Both LIDLite5 and the target diacritizer are lazy-loaded on first use.

from olaverse.nlp import Diacritizer

d = Diacritizer(model="auto")

# Yoruba text — routes to diacnet-yor-viterbi
d.restore("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# Igbo text — routes to diacnet-ig
d.restore("Kedu ka i mere")
# → 'Kedụ ka ị mere'

diacnet-1.0 — multilingual, 10 languages (New in v0.1.5)¶

Model Card: olaverse/diacnet-1.0

A single joint ByT5 model that restores diacritics across 10 languages — Yoruba, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, and Italian — selected via the lang= argument, no separate per-language model or upstream LID step required.

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diacnet-1.0", lang="fr")
d.restore("Le cafe est tres chaud, mais il prefere le the.")
# → 'Le café est très chaud, mais il préfère le thé.'

d_yo = Diacritizer(model="diacnet-1.0", lang="yo")
d_yo.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

Supported lang= codes: "yo", "vi", "ig", "ha", "pl", "tr", "pt", "es", "fr", "it".

Long text is segmented automatically. diacnet-1.0 was trained on sentence-length input (median 58 bytes), so multi-sentence text is split on sentence boundaries, restored a sentence at a time, and rejoined. Without this, a 358-character French paragraph comes back at 235 characters — truncated mid-sentence with the tail dropped.

Diacritizer(model="diacnet-1.0", lang="fr")                       # segments (default)
Diacritizer(model="diacnet-1.0", lang="fr", split_sentences=False)  # one pass
Diacritizer(model="diacnet-1.0", lang="fr", splitter=my_splitter)   # your own

Very short fragments still degenerate

Single words fall below the trained input length, and the model rewrites rather than annotates:

Input	`lang=`	Output
`"el nino"`	`es`	`'el niño\nel niño\nel niño…'` (repetition loop)
`"je nai pas"`	`fr`	`'je nai pas pas je nai pas pas…'` (repetition loop)
`"cafe"`	`fr`	`'cafẹ́'` (Yoruba dot-below on a French word)
`"nino"`	`es`	`'niños'` (inflection changed)
`"pho"`	`vi`	`'phối>'` (wrong word + stray character)
`"citta"`	`it`	`'città?'` (punctuation invented)

Pass whole sentences with their punctuation. It also restores diacritics only — it does not insert apostrophes, so "cest fini" does not become "c'est fini".

Yoruba is the hardest language for this model

Yoruba's median CER (0.110) is nearly 3x the next-highest language — genuine tonal ambiguity (the same base letters can carry multiple valid tone patterns), not a model weakness. For Yoruba specifically, the dedicated diacnet-yor-viterbi/diacnet-yor-x models above may perform better; diacnet-1.0's advantage is breadth (10 languages, one model), not peak Yoruba accuracy.

diacnet-1.1 — same architecture, larger corpus (New in v0.3.0)¶

Model Card: olaverse/diacnet-1.1

Retrained on a much larger web-sourced corpus. A large improvement on Vietnamese (0.1264 → 0.0460 DER), Turkish (0.0447 → 0.0068), Polish (0.0357 → 0.0058) and Italian (0.0015 → 0.0002); a regression on Yoruba (0.1554 → 0.2006), because the larger corpus is mostly under-tone-marked and the model learned to omit marks too. It takes the same lang=, split_sentences= and splitter= arguments as diacnet-1.0.

d = Diacritizer(model="diacnet-1.1", lang="vi")
d.restore("Toi khong biet tieng Viet")
# → 'Tôi không biết tiếng Việt'

diactag-1.0 — per-character tagger, 10 languages (New in v0.3.0)¶

Model Card: olaverse/diactag-1.0 · Full guide: DiacTag →

Classifies each character into a diacritic transformation instead of generating output text. It has no mechanism for changing, inserting or deleting a base character, so strip(output) == strip(input) holds by construction — asserted on every call. 37.6M parameters against 580M, and it beats diacnet-1.1 on 9 of 10 languages (Yoruba 0.2006 → 0.0836 DER, Hausa 0.0593 → 0.0041).

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

lang= takes ISO-639-1 ("yo") or ISO-639-3 ("yor"). Omit it and the model's own LID head detects the language, at a cost of ~0.0001 DER:

d = Diacritizer(model="diactag-1.0")
d.restore("Co ay rat dam dang")        # → 'Cô ấy rất đảm đang'
d.detect_language("Lodz jest piekna")  # → ('pol', 0.9999)

Per-character confidence and abstention. Characters below min_confidence are left exactly as the caller typed them. At 0.9, ~97% of characters are restored at 99.6% accuracy and the rest are flagged. The threshold is also a per-call argument, so one loaded model serves several pipelines.

d = Diacritizer(model="diactag-1.0", lang="yor", min_confidence=0.9)

text, details = d.restore("se eranko naa", return_details=True)
review = [c for c in details if c.confidence < 0.9]
# each detail: .char, .confidence, .abstained, .protected

CPU serving. onnx=True loads the int8 export — 3x faster, 4x smaller, with language auto-detection intact, and compliance stays 1.0000 because the guarantee is architectural rather than a property of numeric precision.

d = Diacritizer(model="diactag-1.0", lang="yor", onnx=True)   # olaverse[onnx]

device= selects "cpu", "cuda" or "mps" on the PyTorch backend.

Lexicon reranking (use_lexicon=True, off by default) rescores predicted non-words against attested spellings of the same stripped form. Measure it on your data before enabling it: on diacbench it cuts non-word outputs by 27% but raises Yorùbá DER by 15% (0.0836 → 0.0961), because density gating shrank the Yorùbá lexicon to 18,436 forms — so "not in the lexicon" often means "rare word we didn't keep", and correct output gets overwritten. See DiacTag → Lexicon reranking.

Differences from the diacnet models

No sentence splitting needed. Documents are handled with overlapping sliding windows, so split_sentences=/splitter= do not apply.
URLs, emails, @handles and CONSTANT_NAMES pass through untouched, and marks already in the input are never silently deleted.
It cannot fix typos. Insertions and deletions are impossible by construction — that is the price of the guarantee.
ONNX exports predating 2026-08-04 have no LID head. The SDK reads the capability off the graph, so against one of those lang= is required and calling without it raises rather than silently assuming Yoruba.

diactag-2.0 — Arabic, 11 languages (New in v0.4.0)¶

Model Card: olaverse/diactag-2.0 · Full guide: DiacTag → diactag-2.0 and Arabic

The same tagger architecture with Arabic added (lang="ar" or "ara"; the LID head knows it too). Harakat, tanwīn and sukūn are the tone slot, shadda and dagger alif are shape, and hamza letters are never added or removed. case_endings=False drops the vowel on each word's last letter, shadda kept.

d = Diacritizer(model="diactag-2.0", lang="ar")
d.restore("ذهب الطالب إلى المدرسة في الصباح")
# → 'ذَهَبَ الطَّالِبُ إلَى الْمَدْرَسَةِ فِي الصَّبَاحِ'

Diacritizer(model="diactag-2.0", lang="ar", case_endings=False)   # per-call: restore(..., case_endings=False)

The SDK decodes each checkpoint with the vendored code that matches the spec_version in its labels.json, so diactag-1.0 keeps running on its original spec 1.2.0 code.

diacnet-2.0 and diacnet-mini-2.0 — 11 languages, hints, alignment (New in v0.4.0)¶

Model Cards: olaverse/diacnet-2.0 · olaverse/diacnet-mini-2.0 · Full guide: DiacNet → diacnet-2.0

ByT5 text-to-text models. The prompt is <tag> [g: word=meaning] text; the language tag is optional (<auto>) and so are meaning hints. Text is chunked at ~300 characters on spaces, decoded greedily and rejoined. By default the output is aligned onto your input — your letters are kept and only the model's marks are taken — so it strips back to your text exactly; aligned=False returns the raw generation, which can also repair typos.

d = Diacritizer(model="diacnet-2.0", lang="vie", device="cuda")      # bf16 on GPU
d.restore("Chi ay chi that su ranh vao nhung buoi toi sau khi da cho con ngu say.",
          hints={"ranh": "free (time)"})
# → 'Chị ấy chỉ thật sự rảnh vào những buổi tối sau khi đã cho con ngủ say.'

Diacritizer(model="diacnet-2.0").restore("وهذا قول مرغوب عنه .", lang="ara")   # also "ara-nocase", "auto"
d.restore_batch(texts, lang="vie")      # length-sorted, padded batches

diacnet-1.0/1.1 are unchanged; they accept aligned=True but default to raw output.

olaverse.nlp.Diacritizer ¶

Diacritizer(model: str = 'diacnet-yor-viterbi', lang: str = None, split_sentences: bool = True, splitter: callable = None, min_confidence: float = 0.0, use_lexicon: bool = False, onnx: bool = False, device: str = 'cpu', case_endings: bool = True, aligned: bool = None, batch_size: int = 16)

Unified interface for restoring diacritics in African languages.

Pass a model ID to use a specific backend, or model="auto" to detect the language automatically and route to the appropriate diacritizer.

Parameters:

Name	Type	Description	Default
`model`	`str`	One of: `"diacnet-yor-viterbi"` — Yoruba, fast Viterbi n-gram (default) `"diacnet-yor-db"` — Yoruba dot-below only, KNN `"diacnet-ig"` — Igbo, KNN `"diacnet-yor"` — Yoruba BiLSTM (requires `olaverse[deeplearning]`) `"diacnet-yor-x"` — Yoruba XLM-RoBERTa (requires `olaverse[deeplearning]`) `"diacnet-1.0"` — Multilingual DiacNet, 10 languages, see `lang=` (requires `olaverse[deeplearning]`) `"diacnet-1.1"` — Same architecture, larger corpus. Better on vie/tur/pol/ita/por, worse on yor/ibo/hau (requires `olaverse[deeplearning]`) `"diactag-1.0"` — Per-character tagger, 10 languages. Cannot corrupt the text, 38MB on CPU, best DER on 7 of 10 languages (requires `olaverse[deeplearning]`) `"diactag-2.0"` — Same tagger architecture, 11 languages: adds Arabic (`ara`/`ar`) with a no-case-endings mode. Matches 1.0 on the other ten (requires `olaverse[deeplearning]`) `"diacnet-2.0"` — ByT5 text-to-text, 11 languages incl. Arabic, 582M. Meaning hints, `<auto>` language, output alignment on by default (requires `olaverse[deeplearning]`) `"diacnet-mini-2.0"` — Same interface, 300M, close to diacnet-2.0 (requires `olaverse[deeplearning]`) `"auto"` — detect language via LIDLite5, then route automatically	`'diacnet-yor-viterbi'`
`lang`	`str`	Target language for the multilingual models. One of `"yo", "vi", "ig", "ha", "pl", "tr", "pt", "es", "fr", "it"`, or the ISO-639-3 equivalent for the diactag and diacnet-2.x models; those also take `"ar"`/`"ara"`, and diacnet-2.x additionally `"ara-nocase"` and `"auto"`. Ignored by the single-language models. For the diactag models leaving it `None` auto-detects with the model's own LID head, and diacnet-2.x sends `<auto>`; `"diacnet-1.0"`/`"1.1"` fall back to Yoruba.	`None`
`split_sentences`	`bool`	`diacnet-1.0`/`1.1` only. They were trained on sentence-length input, so multi-sentence text is segmented and restored a sentence at a time by default. Set `False` to send the whole string through in one pass. The diactag models handle documents natively with sliding windows, and diacnet-2.x chunks at ~300 characters instead; both ignore this.	`True`
`splitter`	`callable`	`diacnet-1.0`/`1.1` only. Your own callable taking a string and returning a list of segments, replacing the default sentence splitter.	`None`
`min_confidence`	`float`	diactag models only. Abstention threshold in [0, 1]. Characters the model is less sure about are left exactly as the caller typed them. At 0.9, ~97% of characters are restored at 99.6% accuracy and the rest are flagged. Default 0 commits to everything. Overridable per :meth:`restore` call.	`0.0`
`use_lexicon`	`bool`	diactag models only. Rerank predicted non-words against attested spellings of the same stripped form. Off by default, and worth measuring before you turn it on — on diacbench it cuts non-word outputs by 27% but raises Yoruba DER by 15% (0.0836 -> 0.0961). See :class:`DiacTagDecoder`.	`False`
`onnx`	`bool`	diactag models only. Load the int8 ONNX export — 3x faster and 4x smaller on CPU for +0.03pp DER, with language auto-detection intact. Requires `olaverse[onnx]`.	`False`
`device`	`str`	`"diactag-*"` and `"diacnet-2.0"`/`"diacnet-mini-2.0"`. `"cpu"` (default), `"cuda"` or `"mps"`; `"auto"` picks CUDA when available. Ignored by diactag when `onnx=True`. The diacnet-2.x models load in bfloat16 on CUDA and float32 elsewhere.	`'cpu'`
`case_endings`	`bool`	Arabic only (`"diactag-2.0"`, `"diacnet-2.0"`, `"diacnet-mini-2.0"`). `False` leaves each word's last letter without its vowel / tanwīn / sukūn (shadda kept), which is how most modern Arabic is vowelled. Has no effect on other languages. Overridable per :meth:`restore` call.	`True`
`aligned`	`bool`	`diacnet` models only. Keep your own letters and take only the model's marks, so the output always strips back to the input. Defaults to `True` for `"diacnet-2.0"`/`"diacnet-mini-2.0"` and `False` for `"diacnet-1.0"`/`"1.1"`, whose output is therefore unchanged unless you ask. Overridable per :meth:`restore` call.	`None`
`batch_size`	`int`	`"diacnet-2.0"`/`"diacnet-mini-2.0"` only. Chunks decoded together by :meth:`restore_batch`.	`16`

Methods:¶

detect_language ¶

detect_language(text: str) -> Tuple[str, float]

Identify the language of text. The diactag models only — they are the only ones with a language-identification head of their own.

Returns:

Name	Type	Description
`tuple`	`Tuple[str, float]`	`(iso_639_3_code, probability)`.

restore ¶

restore(text: str, lang: str = None, min_confidence: float = None, return_details: bool = False, case_endings: bool = None, aligned: bool = None, hints: Hints = None) -> Union[str, Tuple[str, List]]

Restore diacritics in the given text.

Parameters:

Name	Type	Description	Default
`text`	`str`	Plain text (tones/diacritics stripped or missing).	required
`lang`	`str`	Per-call language override for the multilingual models, replacing the one given at construction.	`None`
`min_confidence`	`float`	`"diactag-*"` only. Per-call abstention threshold, so one loaded model can serve a CMS pre-fill and a legal pipeline at different points on the coverage curve.	`None`
`return_details`	`bool`	`"diactag-*"` only. Also return per-character results (`char`, `confidence`, `abstained`, `protected`) for routing low-confidence spans to review.	`False`
`case_endings`	`bool`	Arabic only (`"diactag-2.0"`, `"diacnet-2.0"`, `"diacnet-mini-2.0"`). Per-call override of the constructor setting; `False` drops the vowel on each word's last letter.	`None`
`aligned`	`bool`	`diacnet` models only. Per-call override of the constructor setting: keep your letters, take only the marks.	`None`
`hints`	`Hints`	`"diacnet-2.0"`/`"diacnet-mini-2.0"` only. Meaning hints for words whose marks depend on meaning: a `{word: meaning}` mapping, a list of `"word=meaning"` strings, or a ready-made `"[g: ...]"` block. Written in English.	`None`

Returns:

Type	Description
`Union[str, Tuple[str, List]]`	Union[str, Tuple[str, List]]: the restored text, or `(text, details)` when `return_details=True`.

restore_batch ¶

restore_batch(texts: Sequence[str], lang: Union[None, str, Sequence[Optional[str]]] = None, case_endings: bool = None, aligned: bool = None, hints: Optional[Sequence[Hints]] = None) -> list

Restore diacritics in many texts.

"diacnet-2.0"/"diacnet-mini-2.0" pool the ~300-character chunks of every text, sort them by length and decode them in padded batches, which is far faster on a GPU than one call per text. Every other model simply restores the texts one at a time.

Parameters:

Name	Type	Description	Default
`texts`	`Sequence[str]`	Sequence of strings.	required
`lang`	`Union[None, str, Sequence[Optional[str]]]`	One language for all texts, or a sequence with one per text. Defaults to the one given at construction.	`None`
`case_endings`	`bool`	Arabic only; per-call override.	`None`
`aligned`	`bool`	`diacnet` models only; per-call override.	`None`
`hints`	`Optional[Sequence[Hints]]`	`"diacnet-2.0"`/`"diacnet-mini-2.0"` only. A sequence with one entry per text (`None` where there is no hint).	`None`

Returns:

Type	Description
`list`	list[str]: one restored text per input, in order.

olaverse.nlp.DiacNet2Decoder ¶

DiacNet2Decoder(model_name: str = 'olaverse/diacnet-2.0', device: str = None, batch_size: int = 16)

diacnet-2.0 / diacnet-mini-2.0 diacritic restoration (byte-level seq2seq) — 11 languages including Arabic, from one model.

A ByT5 text-to-text model. The prompt is "<tag> [g: word=meaning] text"; the language tag is optional (<auto>) and so is the gloss block. Unlike diacnet-1.x it was trained on pieces of up to ~300 characters, so text is split into chunks of at most that size on spaces (not on sentence ends), decoded greedily, and the pieces rejoined with single spaces.

By default (aligned=True) the output keeps the input's own letters and takes only the model's marks, so strip_marks(output) == strip_marks(input) by construction. aligned=False returns the raw generation, which can also repair typos but may change a letter in a few percent of sentences.

Batching: chunks from every text are pooled, sorted by length and padded to the longest in each batch, so many short texts cost far less than one call each. On a CUDA device the weights load in bfloat16; elsewhere in float32. The reference outputs were produced by batched bf16 GPU runs, so a CPU float32 run can differ by an occasional mark.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	Hugging Face model id.	`'olaverse/diacnet-2.0'`
`device`	`str`	`"cuda"`, `"cpu"`, `"mps"` ... `None` or `"auto"` picks CUDA when available, else CPU.	`None`
`batch_size`	`int`	Chunks decoded together.	`16`

Methods:¶

decode ¶

decode(text: str, lang: str = None, hints: Hints = None, aligned: bool = True, case_endings: bool = True) -> str

Restore diacritics in one text.

Parameters:

Name	Type	Description	Default
`text`	`str`	Input text of any length; chunked at ~300 characters.	required
`lang`	`str`	ISO-639-3 / ISO-639-1 code, `"ara-nocase"` or `"auto"`. `None` (default) means `"auto"`.	`None`
`hints`	`Hints`	Meaning hints for words whose marks depend on meaning — a `{word: meaning}` mapping, a list of `"word=meaning"` strings, or a ready-made `"[g: ...]"` block. Written in English; applied to every chunk of this text.	`None`
`aligned`	`bool`	Keep your letters, take only the model's marks (default).	`True`
`case_endings`	`bool`	Arabic only. `False` selects `<ara-nocase>`.	`True`

Returns:

Name	Type	Description
`str`	`str`	the restored text. Surrounding whitespace is trimmed, and empty
	`str`	input gives `""`.

decode_batch ¶

decode_batch(texts: Sequence[str], lang: Union[None, str, Sequence[Optional[str]]] = None, hints: Optional[Sequence[Hints]] = None, aligned: bool = True, case_endings: bool = True, batch_size: int = None) -> list

Restore diacritics in many texts.

Parameters:

Name	Type	Description	Default
`texts`	`Sequence[str]`	Sequence of strings.	required
`lang`	`Union[None, str, Sequence[Optional[str]]]`	One code for all texts, or a sequence with one per text.	`None`
`hints`	`Optional[Sequence[Hints]]`	`None`, or a sequence with one entry per text (each `None` or anything :func:`olaverse.nlp.diacnet_utils.build_hint` accepts). Not broadcast: a list of hint strings is ambiguous between "several hints" and "one per text".	`None`
`aligned`	`bool`	Keep your letters, take only the model's marks (default).	`True`
`case_endings`	`bool`	Arabic only. `False` selects `<ara-nocase>`.	`True`
`batch_size`	`int`	Override the instance default for this call.	`None`

Returns:

Type	Description
`list`	list[str]: one restored text per input, in order.

olaverse.nlp.align_marks ¶

align_marks(source, output)

Return source with the marks the model put on every letter it kept.

Letters the model changed, dropped or added fall back to source, so the text itself never changes: strip_marks(align(s, o)) == strip_marks(s).

olaverse.nlp.DiacTagDecoder ¶

DiacTagDecoder(model_name: str = 'olaverse/diactag-1.0', ckpt: str = None, device: str = 'cpu', min_confidence: float = 0.0, use_lexicon: bool = False, onnx: bool = False, case_endings: bool = True)

diactag diacritic restoration (per-character tagger) — diactag-1.0 covers 10 languages, diactag-2.0 adds Arabic (11).

Which vendored code decodes a checkpoint is decided by the spec_version in its labels.json (1.x -> spec 1.2.0, 2.x -> spec 2.0.0), so diactag-1.0 keeps running on the code it was built with.

Unlike the seq2seq diacnet line, this model classifies each character into a diacritic transformation instead of generating output text. It has no mechanism for changing, inserting or deleting a base character, so::

strip_diacritics(output) == strip_diacritics(input)

holds by construction. The invariant is asserted on every call rather than assumed, and it survives int8 quantisation because it is a property of the architecture, not of numeric precision.

What that buys over diacnet-1.1 (or, with output alignment off, over any generative model): no text corruption, per-character calibrated confidence, built-in language detection, and 37.6M parameters against 580M — CPU serving is the default rather than a compromise. What it costs: the model cannot fix a typo, because fixing one would mean inserting or deleting a character.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	Hugging Face repo id.	`'olaverse/diactag-1.0'`
`ckpt`	`str`	Checkpoint filename in the repo. `None` (default) picks the release's own: `ckpt_120000.pt` for diactag-1.0, `ckpt_final.pt` for diactag-2.0. Ignored when `onnx=True`.	`None`
`device`	`str`	`"cpu"` (default), `"cuda"`, `"mps"`. Ignored when `onnx=True` — the ONNX session is CPU-only.	`'cpu'`
`min_confidence`	`float`	Abstention threshold in [0, 1]. Characters the model is less sure about than this are left exactly as the caller typed them. `0` (default) commits to every character. Overridable per call.	`0.0`
`use_lexicon`	`bool`	Rerank predicted non-words against attested spellings of the same stripped form. Conservative — it only ever chooses among forms seen in the corpus. Measure it on your data before enabling it: on diacbench it cuts non-word outputs by 27% but raises Yoruba DER by 15% (0.0836 -> 0.0961), because the density gating that keeps the lexicon clean shrank the Yoruba vocabulary to 18,436 forms, so "not in the lexicon" often means "rare or inflected word we didn't keep" and correct output gets overwritten. Off by default.	`False`
`case_endings`	`bool`	Arabic only (diactag-2.0). `False` leaves each word's last letter without its vowel / tanwīn / sukūn (shadda kept) — how most modern Arabic is vowelled. Raises on diactag-1.0, which has no Arabic. Overridable per call.	`True`
`onnx`	`bool`	Load the int8 ONNX export instead of the PyTorch checkpoint — 3x faster and 4x smaller on CPU for +0.03pp DER, and the strip guarantee survives quantisation because it is architectural. Language auto-detection works here too, matching the PyTorch head; against an export predating the LID head, `lang` becomes required rather than silently guessed. Requires `onnxruntime`.	`False`

Methods:¶

normalize_language ¶

normalize_language(lang)

Map a language code to the ISO-639-3 form the model uses, raising on anything it does not support. None passes through and means "detect it".

detect_language ¶

detect_language(text: str) -> Tuple[str, float]

Identify the language of text with the model's own LID head.

Returns:

Name	Type	Description
`tuple`	`Tuple[str, float]`	`(code, probability)`, where `code` is ISO-639-3.

decode ¶

decode(text: str, lang: str = None, min_confidence: float = None, return_details: bool = False, case_endings: bool = None) -> Union[str, Tuple[str, List]]

Restore diacritics.

Parameters:

Name	Type	Description	Default
`text`	`str`	Input text. Documents are handled directly — the model slides an overlapping window and keeps only the centre of each, so every character is predicted with context on both sides.	required
`lang`	`str`	ISO-639-3 or ISO-639-1 code. `None` (default) runs the LID head and uses what it detects, which costs ~0.0001 DER.	`None`
`min_confidence`	`float`	Per-call override of the abstention threshold.	`None`
`return_details`	`bool`	Also return a list of per-character results (`char`, `confidence`, `abstained`, `protected`), one per grapheme, for routing low-confidence spans to review.	`False`
`case_endings`	`bool`	Arabic only (diactag-2.0). `False` drops the vowel, tanwīn or sukūn on each word's last letter (shadda is kept). `None` uses the value given at construction. Has no effect on other languages.	`None`

Returns:

Type	Description
`Union[str, Tuple[str, List]]`	Union[str, Tuple[str, List]]: the restored text, or `(text, details)` when `return_details=True`.

olaverse.nlp.diacritize_yoruba ¶

diacritize_yoruba(text, model_path=None)

olaverse.nlp.diacritize_yoruba_dot_below ¶

diacritize_yoruba_dot_below(text, model_path=None)

olaverse.nlp.diacritize_igbo ¶

diacritize_igbo(text, model_path=None)

Tokenization — OTK-BPE-50k¶

Model Card: olaverse/otk-bpe-50k

Byte-Level BPE tokenizers trained on dedicated corpora for each Nigerian language. 0% out-of-vocabulary tokens via raw UTF-8 byte fallback.

Why OTK-BPE?¶

General-purpose tokenizers like GPT-4's cl100k tokenize African languages character-by-character, blowing up sequence lengths and wasting context. OTK-BPE-50k learns proper subwords from native text:

Model	Language	Vocab	Efficiency vs GPT-4
`otk-bpe-50k-yo`	Yoruba	50,000	63% fewer tokens
`otk-bpe-50k-ig`	Igbo	50,000	~60% fewer tokens
`otk-bpe-50k-ha`	Hausa	50,000	~58% fewer tokens
`otk-bpe-50k-pcm`	Nigerian Pidgin	50,000	~55% fewer tokens
`otk-bpe-50k-naija`	Unified (all 4)	50,000	Balanced

Usage¶

from olaverse import Tokenizer

# Load by language code
tok_yo = Tokenizer("yo")      # Yoruba
tok_ig = Tokenizer("ig")      # Igbo
tok_ha = Tokenizer("ha")      # Hausa
tok_pcm = Tokenizer("pcm")    # Nigerian Pidgin
tok_all = Tokenizer("naija")  # Unified

# Encode / decode
ids = tok_yo.encode("Ẹ kú àbọ̀")
print(ids)                      # → [124, 381]
print(tok_yo.decode(ids))       # → 'Ẹ kú àbọ̀'

# For fine-tuning / dataset preparation
sentences = ["Bawo ni?", "Se daadaa ni?", "Mo dupe."]
all_ids = [tok_yo.encode(s) for s in sentences]

olaverse.nlp.Tokenizer ¶

Tokenizer(lang='naija', model_path=None)

A unified BPE Tokenizer for African languages.

Nigerian family (fixed 50k vocab): 'yoruba'/'yo', 'igbo'/'ig', 'hausa'/'ha', 'pidgin'/'pcm', and 'naija' (unified).

Multilingual family (50k/100k/150k vocab, see olaverse/otk-bpe): 'sw-50k', 'sw-100k', 'sw-150k' (Swahili); 'kin-50k', 'kin-100k', 'kin-150k' (Kinyarwanda); 'merged-50k', 'merged-100k', 'merged-150k' (French + Kinyarwanda + English + Swahili).

Methods:¶

encode ¶

encode(text)

Encode input text into a list of token IDs.

decode ¶

decode(ids)

Decode a list of token IDs back into a string.

Tokenization — OTK-BPE (Multilingual) (New in v0.1.5)¶

Model Card: olaverse/otk-bpe

A companion tokenizer family for Swahili, Kinyarwanda, and a merged French + Kinyarwanda + English + Swahili vocabulary — each available at three vocab sizes. Same Tokenizer class, different lang= values.

`lang=`	Languages	Vocab sizes
`sw-50k` / `sw-100k` / `sw-150k`	Swahili	50k / 100k / 150k
`kin-50k` / `kin-100k` / `kin-150k`	Kinyarwanda	50k / 100k / 150k
`merged-50k` / `merged-100k` / `merged-150k`	French + Kinyarwanda + English + Swahili	50k / 100k / 150k

150k is the recommended default — fertility and entity-handling both improve monotonically from 50k → 100k → 150k in every benchmark on the model card, with no exceptions. Step down only if embedding-table size is a hard constraint.

from olaverse import Tokenizer

tok = Tokenizer("sw-150k")
ids = tok.encode("Habari yako? Leo ni siku nzuri sana 😊")
tok.decode(ids)
# → 'Habari yako? Leo ni siku nzuri sana 😊'

tok_merged = Tokenizer("merged-150k")  # French + Kinyarwanda + English + Swahili

Text Normalization (TTS)¶

TTSNormalizer converts raw text into a form suitable for speech synthesis — expanding numbers, abbreviations, and punctuation into their spoken equivalents.

from olaverse import TTSNormalizer

# Yoruba normalization
norm = TTSNormalizer(lang="yo")
norm.normalize("Dr. Ade lo si oja lana")
# → 'Dọ́kítà Ade lo si oja lana'

norm.normalize("O san ₦1,200")
# → 'O san  ọ̀kan ẹgbẹ̀rún méjì'

# Igbo normalization
norm_ig = TTSNormalizer(lang="ig")
norm_ig.normalize("Prof. Obi ra ulo")
# → 'Purofesọ Obi ra ulo'

olaverse.nlp.TTSNormalizer ¶

TTSNormalizer(lang: str = 'yo')

Normalizes text for TTS processing by expanding numbers, abbreviations, and symbols into their spoken equivalents.

Parameters:

Name	Type	Description	Default
`lang`	`str`	Target language. One of `'yo'` (Yoruba), `'ig'` (Igbo), `'pcm'` (Nigerian Pidgin). Defaults to `'yo'`.	`'yo'`

Methods:¶

expand_abbreviations ¶

expand_abbreviations(text: str) -> str

Expand abbreviations to their spoken forms.

expand_numbers ¶

expand_numbers(text: str) -> str

Expand digit characters to spoken words (digit-by-digit).

normalize ¶

normalize(text: str) -> str

Run the full normalization pipeline: abbreviations → numbers.

Parameters:

Name	Type	Description	Default
`text`	`str`	Raw input text.	required

Returns:

Type	Description
`str`	Normalized text ready for phonetic processing.

Nigerian Pidgin Normalization — NaijaNormalizer¶

NaijaNormalizer is a Pidgin-specific TTS normalizer built on top of TTSNormalizer. It adds a pre-processing step that expands informal Pidgin spellings — SMS shorthand, phonetic spellings, loan abbreviations — before running the standard abbreviation and number expansion.

from olaverse import NaijaNormalizer

norm = NaijaNormalizer()

# Informal spellings expanded first, then standard normalization
norm.normalize("Oga, 2moro na Sunday. Call am nd tell am 2 come.")
# → 'Oga, tomorrow na Sunday. Call am and tell am to come.'

norm.normalize("e don finish. tnx 4 d help, u r d best!")
# → 'e don finish. thanks for the help, you are the best!'

# normalize_informal() only — skip abbreviations/number expansion
norm.normalize_informal("dis tin dey hard nd i no sabi wetin 2 do")
# → 'this tin dey hard and i no sabi wetin to do'

Informal expansion map includes (35+ entries):

Input	Output	Input	Output
`2moro`	tomorrow	`2day`	today
`b4`	before	`4`	for
`dis`	this	`dat`	that
`nd` / `n`	and	`u`	you
`pls`	please	`tnx`	thanks
`lol`	laugh	`smh`	sigh
`jst`	just	`wat`	what
`wen`	when	`hw`	how

olaverse.nlp.NaijaNormalizer ¶

NaijaNormalizer(canonical: bool = True)

Bases: TTSNormalizer

Extended text normalizer for Nigerian Pidgin English (Naija / pcm).

Inherits the full TTSNormalizer pipeline and adds Pidgin-specific informal spelling normalization — collapsing common alternate spellings to a canonical spoken form before TTS processing.

Parameters:

Name	Type	Description	Default
`canonical`	`bool`	If `True` (default), apply informal-spelling normalization before the standard abbreviation + number pipeline. Set to `False` to use only the base `TTSNormalizer` behaviour.	`True`

Example::

from olaverse.nlp import NaijaNormalizer

norm = NaijaNormalizer()
norm.normalize("Oga, e don finish. Call am 2moro.")
# → 'Oga, e don finish. Call am tomorrow.'

Methods:¶

normalize_informal ¶

normalize_informal(text: str) -> str

Collapse Pidgin informal spellings to canonical spoken forms.

normalize ¶

normalize(text: str) -> str

Full normalization pipeline for Pidgin: informal spellings → abbreviations → numbers.

Parameters:

Name	Type	Description	Default
`text`	`str`	Raw Pidgin input text.	required

Returns:

Type	Description
`str`	TTS-ready normalized text.

Stopwords¶

The stopwords module provides curated stopword sets for all 4 Nigerian languages, ready for use in NLP pipelines, TF-IDF vectorizers, and preprocessing workflows.

from olaverse import (
    YORUBA_STOPWORDS, IGBO_STOPWORDS, HAUSA_STOPWORDS, PIDGIN_STOPWORDS,
    get_stopwords, filter_stopwords,
)

Available Sets¶

Constant	Language	Example words
`YORUBA_STOPWORDS`	Yoruba	mo, mi, o, ni, ní, fun, lati, ati, tabi, sugbon, ṣugbọn
`IGBO_STOPWORDS`	Igbo	m, mu, gi, gị, ya, anyi, anyị, ka, na, nke
`HAUSA_STOPWORDS`	Hausa	ni, kai, ke, shi, ita, mu, ku, su, da, ko, amma
`PIDGIN_STOPWORDS`	Nigerian Pidgin	i, you, him, na, be, go, don, dey, de, wey, sey

All sets are frozenset — safe to use in in membership tests and set operations.

Usage¶

# Direct membership test
"ni" in YORUBA_STOPWORDS   # → True
"atal" in IGBO_STOPWORDS   # → False

# Retrieve by language code
get_stopwords("yor")   # → YORUBA_STOPWORDS  (also accepts "yo")
get_stopwords("ibo")   # → IGBO_STOPWORDS    (also accepts "ig")
get_stopwords("hau")   # → HAUSA_STOPWORDS   (also accepts "ha")
get_stopwords("pcm")   # → PIDGIN_STOPWORDS
get_stopwords("eng")   # → frozenset of common English stopwords

# Filter a tokenized sentence
tokens = ["Bawo", "ni", "Ade", "ati", "Sade", "dara"]
filter_stopwords(tokens, "yor")
# → ['Bawo', 'Ade', 'Sade', 'dara']

# Works with any iterable
filter_stopwords(iter(["dem", "dey", "Lagos", "for", "here"]), "pcm")
# → ['Lagos']

Using with scikit-learn¶

from olaverse import get_stopwords
from sklearn.feature_extraction.text import TfidfVectorizer

yoruba_docs = ["Bawo ni Ade?", "Sade dara pupo", "Mo feran onje Yoruba"]
stop = list(get_stopwords("yor"))

vec = TfidfVectorizer(stop_words=stop)
X = vec.fit_transform(yoruba_docs)

Text Preprocessing & Cleaning¶

PII Masking¶

from olaverse import mask_pii

mask_pii("Contact me at support@olaverse.co.uk or call +1-800-555-0199")
# → 'Contact me at [EMAIL] or call [PHONE]'

mask_pii("My card is 4111-1111-1111-1111 and SSN 123-45-6789")
# → 'My card is [CREDIT_CARD] and SSN [SSN]'

olaverse.nlp.mask_pii ¶

mask_pii(text: str) -> str

Mask general Personally Identifiable Information (PII) globally. Replaces Emails, Credit Cards, Social Security Numbers (SSN), and Phone numbers.

Text Cleaning¶

from olaverse import clean_text

clean_text("Visit <a href='https://olaverse.co.uk'>our site</a>  today!")
# → 'Visit our site today!'

clean_text("Check https://example.com for details.  Multiple   spaces.")
# → 'Check  for details. Multiple spaces.'

# Keep URLs
clean_text("Visit https://olaverse.co.uk", remove_urls=False)
# → 'Visit https://olaverse.co.uk'

olaverse.nlp.clean_text ¶

clean_text(text: str, remove_urls: bool = True, remove_html: bool = True) -> str

General text cleaning utility. Strips extra whitespaces, and optionally removes URLs and HTML tags.

Retrieval (New in v0.1.5)¶

pip install olaverse[retrieval]

Two-piece toolkit for building RAG/search pipelines: a cross-encoder Reranker for the second stage, and a Nigerian-language Embedder for semantic search and cross-lingual retrieval.

Reranker¶

Model Cards: olaverse/mist-reranker-150m · olaverse/mist-reranker-22.7M

Scores (query, passage) pairs to re-sort the top-k candidates from a first-stage retriever (BM25 or a bi-encoder).

`size=`	Model	Params	Best for
`"150m"`	mist-reranker-150m	~150M	Best QA/fact accuracy (ModernBERT-base)
`"22.7m"` (default)	mist-reranker-22.7M	~22.7M	Smaller/faster, MiniLM-L6 backbone

from olaverse import Reranker

reranker = Reranker(size="22.7m")

reranker.rank("who wrote hamlet", [
    "Hamlet is a tragedy written by William Shakespeare.",
    "The capital of France is Paris.",
])
# → [(0, 0.915...), (1, 0.301...)]   # (original_index, score), best-first

reranker.score("who wrote hamlet", ["Hamlet is a tragedy by Shakespeare."])
# → [0.912...]

Both models are English-only; Reranker auto-handles their different output head shapes (a single relevance score vs. 2-class logits).

olaverse.nlp.Reranker ¶

Reranker(size: str = '22.7m')

Cross-encoder reranker for the second stage of a RAG / search pipeline.

Scores (query, passage) pairs to re-sort the top-k candidates from a first-stage retriever (BM25 or a bi-encoder).

Models (size=): "150m" — mist-reranker-150m (ModernBERT-base, English, best QA/fact accuracy) "22.7m" — mist-reranker-22.7M (MiniLM-L6, English, smaller/faster) [default]

Requires: pip install olaverse[retrieval]

Quick start

reranker = Reranker(size="22.7m") reranker.rank("who wrote hamlet", [ ... "Hamlet is a tragedy written by William Shakespeare.", ... "The capital of France is Paris.", ... ]) [(0, 0.98...), (1, 0.01...)]

Methods:¶

load ¶

load()

Download and load the reranker (runs once; cached after first call).

score ¶

score(query: str, passages: list) -> list

Score a query against a list of passages.

Parameters:

Name	Type	Description	Default
`query`	`str`	The search query.	required
`passages`	`list`	List of candidate passage strings.	required

Returns:

Type	Description
`list`	list[float]: relevance scores, one per passage, same order as input.

rank ¶

rank(query: str, passages: list) -> list

Rank passages by relevance to the query, descending.

Parameters:

Name	Type	Description	Default
`query`	`str`	The search query.	required
`passages`	`list`	List of candidate passage strings.	required

Returns:

Type	Description
`list`	list[tuple[int, float]]: (original_index, score) pairs, best-first.

Embedder¶

Model Card: olaverse/naija-embed-base

Cross-lingual sentence embeddings for Hausa, Yoruba, and Igbo — contrastively fine-tuned from mist-encoder-base-ng. Useful for cross-lingual retrieval (e.g. Hausa query → Yoruba document), semantic search, clustering, and deduplication.

from olaverse import Embedder

embedder = Embedder()
vecs = embedder.encode(["bawo ni", "sannu"])
embedder.similarity(vecs[0], vecs[1])

No Nigerian Pidgin support

The underlying translation model used for training only outputs Hausa/Yoruba/Igbo — Pidgin (pcm) is not covered.

olaverse.nlp.Embedder ¶

Embedder(model_name: str = 'olaverse/naija-embed-base')

Cross-lingual sentence embeddings for Nigerian languages (Hausa, Yoruba, Igbo).

Wraps olaverse/naija-embed-base — contrastively fine-tuned from olaverse/mist-encoder-base-ng on synthetic parallel pairs. Mean pooling, cosine similarity. Useful for cross-lingual retrieval, semantic search, clustering, and deduplication over Nigerian-language text.

Note: does not cover Nigerian Pidgin (pcm) — the base model only supports ha/yo/ig.

Requires: pip install olaverse[retrieval]

Quick start

embedder = Embedder() vecs = embedder.encode(["bawo ni", "sannu"]) embedder.similarity(vecs[0], vecs[1])

Methods:¶

load ¶

load()

Download and load the embedding model (runs once; cached after first call).

encode ¶

encode(texts: str | list[str], **kwargs: object) -> 'numpy.ndarray'

Encode a string or list of strings into embedding vector(s).

Parameters:

Name	Type	Description	Default
`texts`	`str \| list[str]`	A string, or list of strings.	required
`**kwargs`	`object`	Passed through to SentenceTransformer.encode().	`{}`

Returns:

Type	Description
`'numpy.ndarray'`	numpy.ndarray: embedding vector(s).

similarity ¶

similarity(a, b) -> float

Cosine similarity between two embedding vectors.

NLP & Tokenization¶

Language Detection¶

LIDLite5¶

olaverse.nlp.LIDLite5 ¶

Methods:¶

predict ¶

predict_proba ¶

olaverse.nlp.detect_language ¶

LIDNeural5¶

Batch Inference¶

olaverse.nlp.LIDNeural5 ¶

LIDLite25 / LIDNeural25 — 25-language identification¶

olaverse.nlp.LIDLite25 ¶

Methods:¶

load ¶

predict_proba ¶

predict ¶

olaverse.nlp.LIDNeural25 ¶

LIDLite608 / LIDNeural608 — 608-language identification (New in v0.4.1)¶

olaverse.nlp.LIDLite608 ¶

Methods:¶

load ¶

predict_proba ¶

predict ¶

predict_batch ¶

olaverse.nlp.LIDNeural608 ¶

Methods:¶

load ¶

predict_proba_batch ¶

LIDNeural5_1 — Nigerian-only, no English fallback¶

olaverse.nlp.LIDNeural5_1 ¶

Diacritization¶

Available Models¶

Quick Functions¶

Unified Diacritizer Class¶

Auto-routing (model="auto") — New in v0.1.4¶

diacnet-1.0 — multilingual, 10 languages (New in v0.1.5)¶

diacnet-1.1 — same architecture, larger corpus (New in v0.3.0)¶

diactag-1.0 — per-character tagger, 10 languages (New in v0.3.0)¶

diactag-2.0 — Arabic, 11 languages (New in v0.4.0)¶

diacnet-2.0 and diacnet-mini-2.0 — 11 languages, hints, alignment (New in v0.4.0)¶

olaverse.nlp.Diacritizer ¶

Methods:¶

detect_language ¶

restore ¶

restore_batch ¶

olaverse.nlp.DiacNet2Decoder ¶

Methods:¶

decode ¶

decode_batch ¶

olaverse.nlp.align_marks ¶

olaverse.nlp.DiacTagDecoder ¶

Methods:¶

normalize_language ¶

detect_language ¶

decode ¶

olaverse.nlp.diacritize_yoruba ¶

olaverse.nlp.diacritize_yoruba_dot_below ¶

olaverse.nlp.diacritize_igbo ¶

Tokenization — OTK-BPE-50k¶

Why OTK-BPE?¶

Usage¶

olaverse.nlp.Tokenizer ¶

Methods:¶

encode ¶

decode ¶

Tokenization — OTK-BPE (Multilingual) (New in v0.1.5)¶

Text Normalization (TTS)¶

olaverse.nlp.TTSNormalizer ¶

Methods:¶

expand_abbreviations ¶

expand_numbers ¶

normalize ¶

Nigerian Pidgin Normalization — NaijaNormalizer¶

olaverse.nlp.NaijaNormalizer ¶

Methods:¶

normalize_informal ¶

normalize ¶

Stopwords¶

Available Sets¶

Auto-routing (`model="auto"`) — New in v0.1.4¶