NLP & Tokenization¶

The olaverse.nlp module is the core of the SDK — production-ready tools for African language text processing, detection, diacritization, tokenization, and preprocessing.

pip install olaverse                # all NLP tools included (no GPU required)
pip install olaverse[deeplearning]  # adds LIDNeural5/LIDNeural25/LIDNeural5_1, diacnet-1.0/1.1, diactag-1.0
pip install olaverse[onnx]          # adds the int8 diactag-1.0 backend
pip install olaverse[lid]           # adds LIDLite25 (fastText, 25 languages)
pip install olaverse[retrieval]     # adds Reranker, Embedder

Language Detection¶

Two models cover the same 5 languages at very different scales. Pick based on your latency and accuracy requirements.

LIDLite5 — Lightweight

Zero GPU · Instant

1.1 MB JSON model
0.014 ms per sentence
98.12% macro accuracy
TF-IDF + Logistic Regression
Pure Python — no torch, no transformers

LIDNeural5 — Neural

GPU Recommended

484 MB — XLM-RoBERTa 125M
13.3 ms per sentence (CPU/GPU)
98.96% macro accuracy
Fine-tuned on castorini/afriberta_large
Requires olaverse[deeplearning]

LIDLite5¶

Model Card: olaverse/lid-lite-5

from olaverse import LIDLite5, detect_language

# Class interface
detector = LIDLite5()
detector.predict("Bawo ni, se daadaa ni?")      # → 'yor'
detector.predict("Sannu, yaya kake?")            # → 'hau'
detector.predict("How far, wetin dey happen?")   # → 'pcm'

# Probability distribution over all 5 classes
detector.predict_proba("Kedu, ọ dị mma?")
# → {'eng': 0.037, 'hau': 0.024, 'ibo': 0.807, 'pcm': 0.021, 'yor': 0.112}

# Quick one-liner
detect_language("E kaaro, bawo ni?")             # → 'yor'

olaverse.nlp.LIDLite5 ¶

LIDLite5(model_path='lid-lite-5.json')

Lightweight, zero-dependency TF-IDF + Logistic Regression Language Detector for 5 languages: Yoruba ('yor'), Hausa ('hau'), Igbo ('ibo'), Pidgin ('pcm'), and English ('eng').

Methods:¶

predict ¶

predict(text)

Predict the language of the given text. Returns: 'yor', 'hau', 'ibo', 'pcm', or 'eng'.

predict_proba ¶

predict_proba(text)

Predict the language probabilities using softmax over logits.

olaverse.nlp.detect_language ¶

detect_language(text, model_path='lid-lite-5.json')

Detect the language of the given text using LIDLite5. Returns: 'yor' (Yoruba), 'hau' (Hausa), 'ibo' (Igbo), 'pcm' (Pidgin), or 'eng' (English).

LIDNeural5¶

Model Card: olaverse/lid-neural-5

pip install olaverse[deeplearning]

from olaverse import LIDNeural5

detector = LIDNeural5()
detector.load()  # downloads once from Hugging Face, cached after first run

detector.predict("Kedu ka ị mere today?")        # → 'ibo'
detector.predict("Ẹ káàárọ̀, ẹ káàbọ̀")        # → 'yor'

probs = detector.predict_proba("How far, wetin dey happen?")
# → {'pcm': 0.991, 'eng': 0.006, 'ibo': 0.001, 'hau': 0.001, 'yor': 0.001}

Batch Inference¶

predict_batch and predict_proba_batch run a single batched forward pass — much faster than calling predict() in a loop over a dataset.

texts = ["Bawo ni?", "Kedu ọ dị?", "How far?", "Sannu dai."]

# Returns a list of predicted language codes
langs = detector.predict_batch(texts)
# → ['yor', 'ibo', 'pcm', 'hau']

# Returns a list of probability dicts — one per input
probs = detector.predict_proba_batch(["Bawo ni?", "Kedu?"])
# → [
#     {'yor': 0.991, 'eng': 0.005, ...},
#     {'ibo': 0.987, 'eng': 0.008, ...},
#   ]

Per-language accuracy:

Language	Precision	Recall	F1-Score
Yoruba (`yor`)	99.60%	99.60%	99.60%
Hausa (`hau`)	99.60%	99.20%	99.40%
Igbo (`ibo`)	98.79%	98.20%	98.50%
Nigerian Pidgin (`pcm`)	99.20%	98.80%	99.00%
English (`eng`)	97.63%	99.00%	98.31%
Overall (Macro)			98.96%

olaverse.nlp.LIDNeural5 ¶

LIDNeural5(model_name='olaverse/lid-neural-5')

Bases: _HFSequenceClassifierLID

High-accuracy transformer-based language identifier for 5 Nigerian languages.

Base Model: castorini/afriberta_large (XLM-RoBERTa, 125M parameters) Fine-tuned on: Yoruba ('yor'), Hausa ('hau'), Igbo ('ibo'), Pidgin ('pcm'), English ('eng') Validation accuracy: 98.96% macro-F1

Requires: pip install olaverse[deeplearning]

LIDLite25 / LIDNeural25 — 25-language identification¶

New in v0.1.5

Beyond the 5 core Nigerian languages, LIDLite25 and LIDNeural25 cover 25 languages spanning Africa, Europe, and Asia — for products that need broader coverage than Yoruba/Hausa/Igbo/Pidgin/English.

Model Cards: olaverse/lid-lite-25 · olaverse/lid-neural-25.1 · olaverse/lid-neural-25.2

Both come in two checkpoints, tuned for different input lengths — pick the variant= that matches your traffic:

`variant=`	Use for
`"passages"`	Documents, articles, paragraph-length text
`"questions"` (default)	Search queries, chat messages, short user input

LIDLite25 is a CPU-only fastText classifier (sub-millisecond, ~5-10MB per checkpoint); LIDNeural25 is an XLM-RoBERTa-base sequence classifier — higher accuracy on short text (98.2% vs 97.3%), at the cost of needing transformers/torch.

pip install olaverse[lid]           # LIDLite25 (fastText)
pip install olaverse[deeplearning]  # LIDNeural25 (transformers)

from olaverse import LIDLite25, LIDNeural25

lite = LIDLite25(variant="questions")
lite.predict("What causes ocean tides?")   # → 'eng'

neural = LIDNeural25(variant="questions")
neural.load()
neural.predict_proba("What causes ocean tides?")
# → {'eng': 0.999, 'fra': 0.0003, ...}

Zulu/Xhosa confusion on short text

Both models score noticeably lower on Zulu/Xhosa short-text classification (F1 ~0.77-0.79) than every other language (≥0.98) — the two languages are closely related with substantial shared vocabulary. This holds across both architectures, so treat predictions between these two specifically with reduced confidence on short input.

olaverse.nlp.LIDLite25 ¶

LIDLite25(variant: str = 'questions')

Lightweight, CPU-only fastText language identifier for 25 languages. Sub-millisecond inference, ~5-10MB per checkpoint, no GPU required.

Two checkpoints for two input lengths (variant=): "passages" — long-form text (documents, articles) "questions" — short text (queries, chat messages) [default]

For higher accuracy at the cost of needing transformers/torch, see LIDNeural25.

Requires: pip install olaverse[lid]

Methods:¶

load ¶

load()

Download and load the fastText checkpoint (runs once; cached after first call).

predict_proba ¶

predict_proba(text: str) -> dict

Return probability distribution over all 25 languages.

Returns:

Name	Type	Description
`dict`	`dict`	{'eng': 0.99, 'fra': 0.005, ...} (ISO 639-3 codes)

predict ¶

predict(text: str) -> str

Predict the dominant language of the text (ISO 639-3 code, e.g. 'eng').

olaverse.nlp.LIDNeural25 ¶

LIDNeural25(variant: str = 'questions')

Bases: _HFSequenceClassifierLID

Transformer-based (XLM-RoBERTa, 125M parameters) language identifier for 25 languages — higher accuracy than LIDLite25, especially on short text, at the cost of needing transformers/torch.

Two checkpoints for two input lengths (variant=): "passages" — lid-neural-25.1, long-form text (documents, articles) "questions" — lid-neural-25.2, short text (queries, chat messages) [default]

Requires: pip install olaverse[deeplearning]

LIDNeural5_1 — Nigerian-only, no English fallback¶

Model Card: olaverse/lid-neural-5.1

A compact (~31M parameter) classifier built on mist-encoder-base-ng, covering only the 4 main Nigerian languages — no English/"other" class.

from olaverse import LIDNeural5_1

detector = LIDNeural5_1()
detector.predict("Ina kwana?")   # → 'Hausa'

No English class

Out-of-set input (English or any other language) will be confidently mislabelled, most often as Nigerian Pidgin. If your input may include English, use LIDLite25/LIDNeural25/LIDNeural5 instead — this model always picks one of the four Nigerian languages.

olaverse.nlp.LIDNeural5_1 ¶

LIDNeural5_1(model_name='olaverse/lid-neural-5.1')

Bases: _HFSequenceClassifierLID

Compact language identifier for the 4 main Nigerian languages, built as a classification head on olaverse/mist-encoder-base-ng (ModernBERT, ~31M parameters).

Labels: 'Hausa', 'Yoruba', 'Igbo', 'Nigerian Pidgin'.

No English/'other' class — out-of-set languages (e.g. English) will be confidently mislabelled, most often as Nigerian Pidgin. Use LIDLite25 or LIDNeural25 instead if inputs may include English or other non-Nigerian languages.

Requires: pip install olaverse[deeplearning]

Diacritization¶

DiacNet and DiacTag restore tones and diacritical marks stripped from text — the critical front-end step for TTS, language learning, and NLP accuracy. The small DiacNet models cover Yoruba and Igbo with no extras; diacnet-1.0/1.1 and diactag-1.0 cover 10 languages each.

For new work prefer diactag-1.0: it classifies each character instead of generating text, so it cannot alter your input beyond adding marks, and it wins on 9 of 10 languages.

Available Models¶

Model ID	Language	Method	Speed	Accuracy	Size
`diacnet-yor-viterbi`	Yoruba	Viterbi n-gram	⚡ Fast	Good	~7 MB
`diacnet-yor-db`	Yoruba (dot-below only)	KNN backoff	⚡ Fast	Dot-below focused	~2 MB
`diacnet-yor`	Yoruba	BiLSTM	Medium	93.35% char	2.4 MB
`diacnet-yor-x`	Yoruba (full)	XLM-RoBERTa	Slow	82.46% word	503 MB
`diacnet-ig`	Igbo	KNN backoff	⚡ Fast	Good	~3 MB
`diacnet-1.0`	10 languages (see below)	ByT5 seq2seq	Slow	~0.02 median CER	~300 MB
`diacnet-1.1`	10 languages	ByT5 seq2seq	Slow	0.0002–0.2006 DER	~1.1 GB
`diactag-1.0`	10 languages	character tagger	⚡ 244 chars/s CPU	0.0132 DER, 1.0 compliance	150 MB / 38 MB int8

Quick Functions¶

from olaverse import diacritize_yoruba, diacritize_yoruba_dot_below, diacritize_igbo

# Yoruba — full tonal diacritics (Viterbi, fast)
diacritize_yoruba("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# Yoruba — dot-below vowels only (KNN)
diacritize_yoruba_dot_below("Ojo lo si oja")
# → 'Ọjọ lo si ọja'

# Igbo
diacritize_igbo("Kedu ka i mere")
# → 'Kedụ ka ị mere'

Unified Diacritizer Class¶

Use Diacritizer when you need to choose a specific model, switch between neural backends, or process mixed-language input.

from olaverse.nlp import Diacritizer

# Default: fast Viterbi for Yoruba
d = Diacritizer(model="diacnet-yor-viterbi")
d.restore("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# High-accuracy neural (requires olaverse[deeplearning])
d_neural = Diacritizer(model="diacnet-yor-x")
d_neural.restore("Ojo lo si oja lana")

Auto-routing (`model="auto"`) — New in v0.1.4¶

Set model="auto" to skip manual language selection. LIDLite5 detects the language on each call and routes to the correct backend — Yoruba → diacnet-yor-viterbi, Igbo → diacnet-ig. Both LIDLite5 and the target diacritizer are lazy-loaded on first use.

from olaverse.nlp import Diacritizer

d = Diacritizer(model="auto")

# Yoruba text — routes to diacnet-yor-viterbi
d.restore("Ojo lo si oja lana")
# → 'Òjó lọ sí ọjà lana'

# Igbo text — routes to diacnet-ig
d.restore("Kedu ka i mere")
# → 'Kedụ ka ị mere'

diacnet-1.0 — multilingual, 10 languages (New in v0.1.5)¶

Model Card: olaverse/diacnet-1.0

A single joint ByT5 model that restores diacritics across 10 languages — Yoruba, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, and Italian — selected via the lang= argument, no separate per-language model or upstream LID step required.

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diacnet-1.0", lang="fr")
d.restore("Le cafe est tres chaud, mais il prefere le the.")
# → 'Le café est très chaud, mais il préfère le thé.'

d_yo = Diacritizer(model="diacnet-1.0", lang="yo")
d_yo.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

Supported lang= codes: "yo", "vi", "ig", "ha", "pl", "tr", "pt", "es", "fr", "it".

Long text is segmented automatically. diacnet-1.0 was trained on sentence-length input (median 58 bytes), so multi-sentence text is split on sentence boundaries, restored a sentence at a time, and rejoined. Without this, a 358-character French paragraph comes back at 235 characters — truncated mid-sentence with the tail dropped.

Diacritizer(model="diacnet-1.0", lang="fr")                       # segments (default)
Diacritizer(model="diacnet-1.0", lang="fr", split_sentences=False)  # one pass
Diacritizer(model="diacnet-1.0", lang="fr", splitter=my_splitter)   # your own

Very short fragments still degenerate

Single words fall below the trained input length, and the model rewrites rather than annotates:

Input	`lang=`	Output
`"el nino"`	`es`	`'el niño\nel niño\nel niño…'` (repetition loop)
`"je nai pas"`	`fr`	`'je nai pas pas je nai pas pas…'` (repetition loop)
`"cafe"`	`fr`	`'cafẹ́'` (Yoruba dot-below on a French word)
`"nino"`	`es`	`'niños'` (inflection changed)
`"pho"`	`vi`	`'phối>'` (wrong word + stray character)
`"citta"`	`it`	`'città?'` (punctuation invented)

Pass whole sentences with their punctuation. It also restores diacritics only — it does not insert apostrophes, so "cest fini" does not become "c'est fini".

Yoruba is the hardest language for this model

Yoruba's median CER (0.110) is nearly 3x the next-highest language — genuine tonal ambiguity (the same base letters can carry multiple valid tone patterns), not a model weakness. For Yoruba specifically, the dedicated diacnet-yor-viterbi/diacnet-yor-x models above may perform better; diacnet-1.0's advantage is breadth (10 languages, one model), not peak Yoruba accuracy.

diacnet-1.1 — same architecture, larger corpus (New in v0.3.0)¶

Model Card: olaverse/diacnet-1.1

Retrained on a much larger web-sourced corpus. A large improvement on Vietnamese (0.1264 → 0.0460 DER), Turkish (0.0447 → 0.0068), Polish (0.0357 → 0.0058) and Italian (0.0015 → 0.0002); a regression on Yoruba (0.1554 → 0.2006), because the larger corpus is mostly under-tone-marked and the model learned to omit marks too. It takes the same lang=, split_sentences= and splitter= arguments as diacnet-1.0.

d = Diacritizer(model="diacnet-1.1", lang="vi")
d.restore("Toi khong biet tieng Viet")
# → 'Tôi không biết tiếng Việt'

diactag-1.0 — per-character tagger, 10 languages (New in v0.3.0)¶

Model Card: olaverse/diactag-1.0 · Full guide: DiacTag →

Classifies each character into a diacritic transformation instead of generating output text. It has no mechanism for changing, inserting or deleting a base character, so strip(output) == strip(input) holds by construction — asserted on every call. 37.6M parameters against 580M, and it beats diacnet-1.1 on 9 of 10 languages (Yoruba 0.2006 → 0.0836 DER, Hausa 0.0593 → 0.0041).

pip install olaverse[deeplearning]

from olaverse.nlp import Diacritizer

d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")
# → 'ṣé ẹranko náà sì gbọ́ ọ?'

lang= takes ISO-639-1 ("yo") or ISO-639-3 ("yor"). Omit it and the model's own LID head detects the language, at a cost of ~0.0001 DER:

d = Diacritizer(model="diactag-1.0")
d.restore("Co ay rat dam dang")        # → 'Cô ấy rất đảm đang'
d.detect_language("Lodz jest piekna")  # → ('pol', 0.9999)

Per-character confidence and abstention. Characters below min_confidence are left exactly as the caller typed them. At 0.9, ~97% of characters are restored at 99.6% accuracy and the rest are flagged. The threshold is also a per-call argument, so one loaded model serves several pipelines.

d = Diacritizer(model="diactag-1.0", lang="yor", min_confidence=0.9)

text, details = d.restore("se eranko naa", return_details=True)
review = [c for c in details if c.confidence < 0.9]
# each detail: .char, .confidence, .abstained, .protected

CPU serving. onnx=True loads the int8 export — 3x faster, 4x smaller, with language auto-detection intact, and compliance stays 1.0000 because the guarantee is architectural rather than a property of numeric precision.

d = Diacritizer(model="diactag-1.0", lang="yor", onnx=True)   # olaverse[onnx]

device= selects "cpu", "cuda" or "mps" on the PyTorch backend.

Lexicon reranking (use_lexicon=True, off by default) rescores predicted non-words against attested spellings of the same stripped form. Measure it on your data before enabling it: on diacbench it cuts non-word outputs by 27% but raises Yorùbá DER by 15% (0.0836 → 0.0961), because density gating shrank the Yorùbá lexicon to 18,436 forms — so "not in the lexicon" often means "rare word we didn't keep", and correct output gets overwritten. See DiacTag → Lexicon reranking.

Differences from the diacnet models

No sentence splitting needed. Documents are handled with overlapping sliding windows, so split_sentences=/splitter= do not apply.
URLs, emails, @handles and CONSTANT_NAMES pass through untouched, and marks already in the input are never silently deleted.
It cannot fix typos. Insertions and deletions are impossible by construction — that is the price of the guarantee.
ONNX exports predating 2026-08-04 have no LID head. The SDK reads the capability off the graph, so against one of those lang= is required and calling without it raises rather than silently assuming Yoruba.

olaverse.nlp.Diacritizer ¶

Diacritizer(model: str = 'diacnet-yor-viterbi', lang: str = None, split_sentences: bool = True, splitter: callable = None, min_confidence: float = 0.0, use_lexicon: bool = False, onnx: bool = False, device: str = 'cpu')

Unified interface for restoring diacritics in African languages.

Pass a model ID to use a specific backend, or model="auto" to detect the language automatically and route to the appropriate diacritizer.

Parameters:

Name	Type	Description	Default
`model`	`str`	One of: `"diacnet-yor-viterbi"` — Yoruba, fast Viterbi n-gram (default) `"diacnet-yor-db"` — Yoruba dot-below only, KNN `"diacnet-ig"` — Igbo, KNN `"diacnet-yor"` — Yoruba BiLSTM (requires `olaverse[deeplearning]`) `"diacnet-yor-x"` — Yoruba XLM-RoBERTa (requires `olaverse[deeplearning]`) `"diacnet-1.0"` — Multilingual DiacNet, 10 languages, see `lang=` (requires `olaverse[deeplearning]`) `"diacnet-1.1"` — Same architecture, larger corpus. Better on vie/tur/pol/ita/por, worse on yor/ibo/hau (requires `olaverse[deeplearning]`) `"diactag-1.0"` — Per-character tagger, 10 languages. Cannot corrupt the text, 38MB on CPU, best DER on 7 of 10 languages (requires `olaverse[deeplearning]`) `"auto"` — detect language via LIDLite5, then route automatically	`'diacnet-yor-viterbi'`
`lang`	`str`	Target language for the multilingual models. One of `"yo", "vi", "ig", "ha", "pl", "tr", "pt", "es", "fr", "it"`, or the ISO-639-3 equivalent for `"diactag-1.0"`. Ignored by the single-language models. For `"diactag-1.0"` leaving it `None` auto-detects with the model's own LID head; `"diacnet-1.0"`/`"1.1"` fall back to Yoruba.	`None`
`split_sentences`	`bool`	`diacnet` models only. They were trained on sentence-length input, so multi-sentence text is segmented and restored a sentence at a time by default. Set `False` to send the whole string through in one pass. `diactag-1.0` handles documents natively with sliding windows and ignores this.	`True`
`splitter`	`callable`	`diacnet` models only. Your own callable taking a string and returning a list of segments, replacing the default sentence splitter.	`None`
`min_confidence`	`float`	`"diactag-1.0"` only. Abstention threshold in [0, 1]. Characters the model is less sure about are left exactly as the caller typed them. At 0.9, ~97% of characters are restored at 99.6% accuracy and the rest are flagged. Default 0 commits to everything. Overridable per :meth:`restore` call.	`0.0`
`use_lexicon`	`bool`	`"diactag-1.0"` only. Rerank predicted non-words against attested spellings of the same stripped form. Off by default, and worth measuring before you turn it on — on diacbench it cuts non-word outputs by 27% but raises Yoruba DER by 15% (0.0836 -> 0.0961). See :class:`DiacTagDecoder`.	`False`
`onnx`	`bool`	`"diactag-1.0"` only. Load the int8 ONNX export — 3x faster and 4x smaller on CPU for +0.03pp DER, with language auto-detection intact. Requires `olaverse[onnx]`.	`False`
`device`	`str`	`"diactag-1.0"` only. `"cpu"` (default), `"cuda"` or `"mps"`. Ignored when `onnx=True`.	`'cpu'`

Methods:¶

detect_language ¶

detect_language(text: str) -> Tuple[str, float]

Identify the language of text. "diactag-1.0" only — it is the only model with a language-identification head of its own.

Returns:

Name	Type	Description
`tuple`	`Tuple[str, float]`	`(iso_639_3_code, probability)`.

restore ¶

restore(text: str, lang: str = None, min_confidence: float = None, return_details: bool = False) -> Union[str, Tuple[str, List]]

Restore diacritics in the given text.

Parameters:

Name	Type	Description	Default
`text`	`str`	Plain text (tones/diacritics stripped or missing).	required
`lang`	`str`	Per-call language override for the multilingual models, replacing the one given at construction.	`None`
`min_confidence`	`float`	`"diactag-1.0"` only. Per-call abstention threshold, so one loaded model can serve a CMS pre-fill and a legal pipeline at different points on the coverage curve.	`None`
`return_details`	`bool`	`"diactag-1.0"` only. Also return per-character results (`char`, `confidence`, `abstained`, `protected`) for routing low-confidence spans to review.	`False`

Returns:

Type	Description
`Union[str, Tuple[str, List]]`	Union[str, Tuple[str, List]]: the restored text, or `(text, details)` when `return_details=True`.

olaverse.nlp.DiacTagDecoder ¶

DiacTagDecoder(model_name: str = 'olaverse/diactag-1.0', ckpt: str = _DIACTAG_DEFAULT_CKPT, device: str = 'cpu', min_confidence: float = 0.0, use_lexicon: bool = False, onnx: bool = False)

diactag-1.0 diacritic restoration (per-character tagger) — 10 languages.

Unlike the seq2seq diacnet line, this model classifies each character into a diacritic transformation instead of generating output text. It has no mechanism for changing, inserting or deleting a base character, so::

strip_diacritics(output) == strip_diacritics(input)

holds by construction. The invariant is asserted on every call rather than assumed, and it survives int8 quantisation because it is a property of the architecture, not of numeric precision.

What that buys over diacnet-1.1: no text corruption, per-character calibrated confidence, built-in language detection, and 37.6M parameters against 580M — CPU serving is the default rather than a compromise. What it costs: the model cannot fix a typo, because fixing one would mean inserting or deleting a character.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	Hugging Face repo id.	`'olaverse/diactag-1.0'`
`ckpt`	`str`	Checkpoint filename in the repo. Ignored when `onnx=True`.	`_DIACTAG_DEFAULT_CKPT`
`device`	`str`	`"cpu"` (default), `"cuda"`, `"mps"`. Ignored when `onnx=True` — the ONNX session is CPU-only.	`'cpu'`
`min_confidence`	`float`	Abstention threshold in [0, 1]. Characters the model is less sure about than this are left exactly as the caller typed them. `0` (default) commits to every character. Overridable per call.	`0.0`
`use_lexicon`	`bool`	Rerank predicted non-words against attested spellings of the same stripped form. Conservative — it only ever chooses among forms seen in the corpus. Measure it on your data before enabling it: on diacbench it cuts non-word outputs by 27% but raises Yoruba DER by 15% (0.0836 -> 0.0961), because the density gating that keeps the lexicon clean shrank the Yoruba vocabulary to 18,436 forms, so "not in the lexicon" often means "rare or inflected word we didn't keep" and correct output gets overwritten. Off by default.	`False`
`onnx`	`bool`	Load the int8 ONNX export instead of the PyTorch checkpoint — 3x faster and 4x smaller on CPU for +0.03pp DER, and the strip guarantee survives quantisation because it is architectural. Language auto-detection works here too, matching the PyTorch head; against an export predating the LID head, `lang` becomes required rather than silently guessed. Requires `onnxruntime`.	`False`

Methods:¶

normalize_language ¶

normalize_language(lang)

Map a language code to the ISO-639-3 form the model uses, raising on anything it does not support. None passes through and means "detect it".

detect_language ¶

detect_language(text: str) -> Tuple[str, float]

Identify the language of text with the model's own LID head.

Returns:

Name	Type	Description
`tuple`	`Tuple[str, float]`	`(code, probability)`, where `code` is ISO-639-3.

decode ¶

decode(text: str, lang: str = None, min_confidence: float = None, return_details: bool = False) -> Union[str, Tuple[str, List]]

Restore diacritics.

Parameters:

Name	Type	Description	Default
`text`	`str`	Input text. Documents are handled directly — the model slides an overlapping window and keeps only the centre of each, so every character is predicted with context on both sides.	required
`lang`	`str`	ISO-639-3 or ISO-639-1 code. `None` (default) runs the LID head and uses what it detects, which costs ~0.0001 DER.	`None`
`min_confidence`	`float`	Per-call override of the abstention threshold.	`None`
`return_details`	`bool`	Also return a list of per-character results (`char`, `confidence`, `abstained`, `protected`), one per grapheme, for routing low-confidence spans to review.	`False`

Returns:

Type	Description
`Union[str, Tuple[str, List]]`	Union[str, Tuple[str, List]]: the restored text, or `(text, details)` when `return_details=True`.

olaverse.nlp.diacritize_yoruba ¶

diacritize_yoruba(text, model_path=None)

olaverse.nlp.diacritize_yoruba_dot_below ¶

diacritize_yoruba_dot_below(text, model_path=None)

olaverse.nlp.diacritize_igbo ¶

diacritize_igbo(text, model_path=None)

Tokenization — OTK-BPE-50k¶

Model Card: olaverse/otk-bpe-50k

Byte-Level BPE tokenizers trained on dedicated corpora for each Nigerian language. 0% out-of-vocabulary tokens via raw UTF-8 byte fallback.

Why OTK-BPE?¶

General-purpose tokenizers like GPT-4's cl100k tokenize African languages character-by-character, blowing up sequence lengths and wasting context. OTK-BPE-50k learns proper subwords from native text:

Model	Language	Vocab	Efficiency vs GPT-4
`otk-bpe-50k-yo`	Yoruba	50,000	63% fewer tokens
`otk-bpe-50k-ig`	Igbo	50,000	~60% fewer tokens
`otk-bpe-50k-ha`	Hausa	50,000	~58% fewer tokens
`otk-bpe-50k-pcm`	Nigerian Pidgin	50,000	~55% fewer tokens
`otk-bpe-50k-naija`	Unified (all 4)	50,000	Balanced

Usage¶

from olaverse import Tokenizer

# Load by language code
tok_yo = Tokenizer("yo")      # Yoruba
tok_ig = Tokenizer("ig")      # Igbo
tok_ha = Tokenizer("ha")      # Hausa
tok_pcm = Tokenizer("pcm")    # Nigerian Pidgin
tok_all = Tokenizer("naija")  # Unified

# Encode / decode
ids = tok_yo.encode("Ẹ kú àbọ̀")
print(ids)                      # → [124, 381]
print(tok_yo.decode(ids))       # → 'Ẹ kú àbọ̀'

# For fine-tuning / dataset preparation
sentences = ["Bawo ni?", "Se daadaa ni?", "Mo dupe."]
all_ids = [tok_yo.encode(s) for s in sentences]

olaverse.nlp.Tokenizer ¶

Tokenizer(lang='naija', model_path=None)

A unified BPE Tokenizer for African languages.

Nigerian family (fixed 50k vocab): 'yoruba'/'yo', 'igbo'/'ig', 'hausa'/'ha', 'pidgin'/'pcm', and 'naija' (unified).

Multilingual family (50k/100k/150k vocab, see olaverse/otk-bpe): 'sw-50k', 'sw-100k', 'sw-150k' (Swahili); 'kin-50k', 'kin-100k', 'kin-150k' (Kinyarwanda); 'merged-50k', 'merged-100k', 'merged-150k' (French + Kinyarwanda + English + Swahili).

Methods:¶

encode ¶

encode(text)

Encode input text into a list of token IDs.

decode ¶

decode(ids)

Decode a list of token IDs back into a string.

Tokenization — OTK-BPE (Multilingual) (New in v0.1.5)¶

Model Card: olaverse/otk-bpe

A companion tokenizer family for Swahili, Kinyarwanda, and a merged French + Kinyarwanda + English + Swahili vocabulary — each available at three vocab sizes. Same Tokenizer class, different lang= values.

`lang=`	Languages	Vocab sizes
`sw-50k` / `sw-100k` / `sw-150k`	Swahili	50k / 100k / 150k
`kin-50k` / `kin-100k` / `kin-150k`	Kinyarwanda	50k / 100k / 150k
`merged-50k` / `merged-100k` / `merged-150k`	French + Kinyarwanda + English + Swahili	50k / 100k / 150k

150k is the recommended default — fertility and entity-handling both improve monotonically from 50k → 100k → 150k in every benchmark on the model card, with no exceptions. Step down only if embedding-table size is a hard constraint.

from olaverse import Tokenizer

tok = Tokenizer("sw-150k")
ids = tok.encode("Habari yako? Leo ni siku nzuri sana 😊")
tok.decode(ids)
# → 'Habari yako? Leo ni siku nzuri sana 😊'

tok_merged = Tokenizer("merged-150k")  # French + Kinyarwanda + English + Swahili

Text Normalization (TTS)¶

TTSNormalizer converts raw text into a form suitable for speech synthesis — expanding numbers, abbreviations, and punctuation into their spoken equivalents.

from olaverse import TTSNormalizer

# Yoruba normalization
norm = TTSNormalizer(lang="yo")
norm.normalize("Dr. Ade lo si oja lana")
# → 'Dọ́kítà Ade lo si oja lana'

norm.normalize("O san ₦1,200")
# → 'O san  ọ̀kan ẹgbẹ̀rún méjì'

# Igbo normalization
norm_ig = TTSNormalizer(lang="ig")
norm_ig.normalize("Prof. Obi ra ulo")
# → 'Purofesọ Obi ra ulo'

olaverse.nlp.TTSNormalizer ¶

TTSNormalizer(lang: str = 'yo')

Normalizes text for TTS processing by expanding numbers, abbreviations, and symbols into their spoken equivalents.

Parameters:

Name	Type	Description	Default
`lang`	`str`	Target language. One of `'yo'` (Yoruba), `'ig'` (Igbo), `'pcm'` (Nigerian Pidgin). Defaults to `'yo'`.	`'yo'`

Methods:¶

expand_abbreviations ¶

expand_abbreviations(text: str) -> str

Expand abbreviations to their spoken forms.

expand_numbers ¶

expand_numbers(text: str) -> str

Expand digit characters to spoken words (digit-by-digit).

normalize ¶

normalize(text: str) -> str

Run the full normalization pipeline: abbreviations → numbers.

Parameters:

Name	Type	Description	Default
`text`	`str`	Raw input text.	required

Returns:

Type	Description
`str`	Normalized text ready for phonetic processing.

Nigerian Pidgin Normalization — NaijaNormalizer¶

NaijaNormalizer is a Pidgin-specific TTS normalizer built on top of TTSNormalizer. It adds a pre-processing step that expands informal Pidgin spellings — SMS shorthand, phonetic spellings, loan abbreviations — before running the standard abbreviation and number expansion.

from olaverse import NaijaNormalizer

norm = NaijaNormalizer()

# Informal spellings expanded first, then standard normalization
norm.normalize("Oga, 2moro na Sunday. Call am nd tell am 2 come.")
# → 'Oga, tomorrow na Sunday. Call am and tell am to come.'

norm.normalize("e don finish. tnx 4 d help, u r d best!")
# → 'e don finish. thanks for the help, you are the best!'

# normalize_informal() only — skip abbreviations/number expansion
norm.normalize_informal("dis tin dey hard nd i no sabi wetin 2 do")
# → 'this tin dey hard and i no sabi wetin to do'

Informal expansion map includes (35+ entries):

Input	Output	Input	Output
`2moro`	tomorrow	`2day`	today
`b4`	before	`4`	for
`dis`	this	`dat`	that
`nd` / `n`	and	`u`	you
`pls`	please	`tnx`	thanks
`lol`	laugh	`smh`	sigh
`jst`	just	`wat`	what
`wen`	when	`hw`	how

olaverse.nlp.NaijaNormalizer ¶

NaijaNormalizer(canonical: bool = True)

Bases: TTSNormalizer

Extended text normalizer for Nigerian Pidgin English (Naija / pcm).

Inherits the full TTSNormalizer pipeline and adds Pidgin-specific informal spelling normalization — collapsing common alternate spellings to a canonical spoken form before TTS processing.

Parameters:

Name	Type	Description	Default
`canonical`	`bool`	If `True` (default), apply informal-spelling normalization before the standard abbreviation + number pipeline. Set to `False` to use only the base `TTSNormalizer` behaviour.	`True`

Example::

from olaverse.nlp import NaijaNormalizer

norm = NaijaNormalizer()
norm.normalize("Oga, e don finish. Call am 2moro.")
# → 'Oga, e don finish. Call am tomorrow.'

Methods:¶

normalize_informal ¶

normalize_informal(text: str) -> str

Collapse Pidgin informal spellings to canonical spoken forms.

normalize ¶

normalize(text: str) -> str

Full normalization pipeline for Pidgin: informal spellings → abbreviations → numbers.

Parameters:

Name	Type	Description	Default
`text`	`str`	Raw Pidgin input text.	required

Returns:

Type	Description
`str`	TTS-ready normalized text.

Stopwords¶

The stopwords module provides curated stopword sets for all 4 Nigerian languages, ready for use in NLP pipelines, TF-IDF vectorizers, and preprocessing workflows.

from olaverse import (
    YORUBA_STOPWORDS, IGBO_STOPWORDS, HAUSA_STOPWORDS, PIDGIN_STOPWORDS,
    get_stopwords, filter_stopwords,
)

Available Sets¶

Constant	Language	Example words
`YORUBA_STOPWORDS`	Yoruba	mo, mi, o, ni, ní, fun, lati, ati, tabi, sugbon, ṣugbọn
`IGBO_STOPWORDS`	Igbo	m, mu, gi, gị, ya, anyi, anyị, ka, na, nke
`HAUSA_STOPWORDS`	Hausa	ni, kai, ke, shi, ita, mu, ku, su, da, ko, amma
`PIDGIN_STOPWORDS`	Nigerian Pidgin	i, you, him, na, be, go, don, dey, de, wey, sey

All sets are frozenset — safe to use in in membership tests and set operations.

Usage¶

# Direct membership test
"ni" in YORUBA_STOPWORDS   # → True
"atal" in IGBO_STOPWORDS   # → False

# Retrieve by language code
get_stopwords("yor")   # → YORUBA_STOPWORDS  (also accepts "yo")
get_stopwords("ibo")   # → IGBO_STOPWORDS    (also accepts "ig")
get_stopwords("hau")   # → HAUSA_STOPWORDS   (also accepts "ha")
get_stopwords("pcm")   # → PIDGIN_STOPWORDS
get_stopwords("eng")   # → frozenset of common English stopwords

# Filter a tokenized sentence
tokens = ["Bawo", "ni", "Ade", "ati", "Sade", "dara"]
filter_stopwords(tokens, "yor")
# → ['Bawo', 'Ade', 'Sade', 'dara']

# Works with any iterable
filter_stopwords(iter(["dem", "dey", "Lagos", "for", "here"]), "pcm")
# → ['Lagos']

Using with scikit-learn¶

from olaverse import get_stopwords
from sklearn.feature_extraction.text import TfidfVectorizer

yoruba_docs = ["Bawo ni Ade?", "Sade dara pupo", "Mo feran onje Yoruba"]
stop = list(get_stopwords("yor"))

vec = TfidfVectorizer(stop_words=stop)
X = vec.fit_transform(yoruba_docs)

Text Preprocessing & Cleaning¶

PII Masking¶

from olaverse import mask_pii

mask_pii("Contact me at support@olaverse.co.uk or call +1-800-555-0199")
# → 'Contact me at [EMAIL] or call [PHONE]'

mask_pii("My card is 4111-1111-1111-1111 and SSN 123-45-6789")
# → 'My card is [CREDIT_CARD] and SSN [SSN]'

olaverse.nlp.mask_pii ¶

mask_pii(text: str) -> str

Mask general Personally Identifiable Information (PII) globally. Replaces Emails, Credit Cards, Social Security Numbers (SSN), and Phone numbers.

Text Cleaning¶

from olaverse import clean_text

clean_text("Visit <a href='https://olaverse.co.uk'>our site</a>  today!")
# → 'Visit our site today!'

clean_text("Check https://example.com for details.  Multiple   spaces.")
# → 'Check  for details. Multiple spaces.'

# Keep URLs
clean_text("Visit https://olaverse.co.uk", remove_urls=False)
# → 'Visit https://olaverse.co.uk'

olaverse.nlp.clean_text ¶

clean_text(text: str, remove_urls: bool = True, remove_html: bool = True) -> str

General text cleaning utility. Strips extra whitespaces, and optionally removes URLs and HTML tags.

Retrieval (New in v0.1.5)¶

pip install olaverse[retrieval]

Two-piece toolkit for building RAG/search pipelines: a cross-encoder Reranker for the second stage, and a Nigerian-language Embedder for semantic search and cross-lingual retrieval.

Reranker¶

Model Cards: olaverse/mist-reranker-150m · olaverse/mist-reranker-22.7M

Scores (query, passage) pairs to re-sort the top-k candidates from a first-stage retriever (BM25 or a bi-encoder).

`size=`	Model	Params	Best for
`"150m"`	mist-reranker-150m	~150M	Best QA/fact accuracy (ModernBERT-base)
`"22.7m"` (default)	mist-reranker-22.7M	~22.7M	Smaller/faster, MiniLM-L6 backbone

from olaverse import Reranker

reranker = Reranker(size="22.7m")

reranker.rank("who wrote hamlet", [
    "Hamlet is a tragedy written by William Shakespeare.",
    "The capital of France is Paris.",
])
# → [(0, 0.915...), (1, 0.301...)]   # (original_index, score), best-first

reranker.score("who wrote hamlet", ["Hamlet is a tragedy by Shakespeare."])
# → [0.912...]

Both models are English-only; Reranker auto-handles their different output head shapes (a single relevance score vs. 2-class logits).

olaverse.nlp.Reranker ¶

Reranker(size: str = '22.7m')

Cross-encoder reranker for the second stage of a RAG / search pipeline.

Scores (query, passage) pairs to re-sort the top-k candidates from a first-stage retriever (BM25 or a bi-encoder).

Models (size=): "150m" — mist-reranker-150m (ModernBERT-base, English, best QA/fact accuracy) "22.7m" — mist-reranker-22.7M (MiniLM-L6, English, smaller/faster) [default]

Requires: pip install olaverse[retrieval]

Quick start

reranker = Reranker(size="22.7m") reranker.rank("who wrote hamlet", [ ... "Hamlet is a tragedy written by William Shakespeare.", ... "The capital of France is Paris.", ... ]) [(0, 0.98...), (1, 0.01...)]

Methods:¶

load ¶

load()

Download and load the reranker (runs once; cached after first call).

score ¶

score(query: str, passages: list) -> list

Score a query against a list of passages.

Parameters:

Name	Type	Description	Default
`query`	`str`	The search query.	required
`passages`	`list`	List of candidate passage strings.	required

Returns:

Type	Description
`list`	list[float]: relevance scores, one per passage, same order as input.

rank ¶

rank(query: str, passages: list) -> list

Rank passages by relevance to the query, descending.

Parameters:

Name	Type	Description	Default
`query`	`str`	The search query.	required
`passages`	`list`	List of candidate passage strings.	required

Returns:

Type	Description
`list`	list[tuple[int, float]]: (original_index, score) pairs, best-first.

Embedder¶

Model Card: olaverse/naija-embed-base

Cross-lingual sentence embeddings for Hausa, Yoruba, and Igbo — contrastively fine-tuned from mist-encoder-base-ng. Useful for cross-lingual retrieval (e.g. Hausa query → Yoruba document), semantic search, clustering, and deduplication.

from olaverse import Embedder

embedder = Embedder()
vecs = embedder.encode(["bawo ni", "sannu"])
embedder.similarity(vecs[0], vecs[1])

No Nigerian Pidgin support

The underlying translation model used for training only outputs Hausa/Yoruba/Igbo — Pidgin (pcm) is not covered.

olaverse.nlp.Embedder ¶

Embedder(model_name: str = 'olaverse/naija-embed-base')

Cross-lingual sentence embeddings for Nigerian languages (Hausa, Yoruba, Igbo).

Wraps olaverse/naija-embed-base — contrastively fine-tuned from olaverse/mist-encoder-base-ng on synthetic parallel pairs. Mean pooling, cosine similarity. Useful for cross-lingual retrieval, semantic search, clustering, and deduplication over Nigerian-language text.

Note: does not cover Nigerian Pidgin (pcm) — the base model only supports ha/yo/ig.

Requires: pip install olaverse[retrieval]

Quick start

embedder = Embedder() vecs = embedder.encode(["bawo ni", "sannu"]) embedder.similarity(vecs[0], vecs[1])

Methods:¶

load ¶

load()

Download and load the embedding model (runs once; cached after first call).

encode ¶

encode(texts: str | list[str], **kwargs: object) -> 'numpy.ndarray'

Encode a string or list of strings into embedding vector(s).

Parameters:

Name	Type	Description	Default
`texts`	`str \| list[str]`	A string, or list of strings.	required
`**kwargs`	`object`	Passed through to SentenceTransformer.encode().	`{}`

Returns:

Type	Description
`'numpy.ndarray'`	numpy.ndarray: embedding vector(s).

similarity ¶

similarity(a, b) -> float

Cosine similarity between two embedding vectors.

NLP & Tokenization¶

Language Detection¶

LIDLite5¶

olaverse.nlp.LIDLite5 ¶

Methods:¶

predict ¶

predict_proba ¶

olaverse.nlp.detect_language ¶

LIDNeural5¶

Batch Inference¶

olaverse.nlp.LIDNeural5 ¶

LIDLite25 / LIDNeural25 — 25-language identification¶

olaverse.nlp.LIDLite25 ¶

Methods:¶

load ¶

predict_proba ¶

predict ¶

olaverse.nlp.LIDNeural25 ¶

LIDNeural5_1 — Nigerian-only, no English fallback¶

olaverse.nlp.LIDNeural5_1 ¶

Diacritization¶

Available Models¶

Quick Functions¶

Unified Diacritizer Class¶

Auto-routing (model="auto") — New in v0.1.4¶

diacnet-1.0 — multilingual, 10 languages (New in v0.1.5)¶

diacnet-1.1 — same architecture, larger corpus (New in v0.3.0)¶

diactag-1.0 — per-character tagger, 10 languages (New in v0.3.0)¶

olaverse.nlp.Diacritizer ¶

Methods:¶

detect_language ¶

restore ¶

olaverse.nlp.DiacTagDecoder ¶

Methods:¶

normalize_language ¶

detect_language ¶

decode ¶

olaverse.nlp.diacritize_yoruba ¶

olaverse.nlp.diacritize_yoruba_dot_below ¶

olaverse.nlp.diacritize_igbo ¶

Tokenization — OTK-BPE-50k¶

Why OTK-BPE?¶

Usage¶

olaverse.nlp.Tokenizer ¶

Methods:¶

encode ¶

decode ¶

Tokenization — OTK-BPE (Multilingual) (New in v0.1.5)¶

Text Normalization (TTS)¶

olaverse.nlp.TTSNormalizer ¶

Methods:¶

expand_abbreviations ¶

expand_numbers ¶

normalize ¶

Nigerian Pidgin Normalization — NaijaNormalizer¶

olaverse.nlp.NaijaNormalizer ¶

Methods:¶

normalize_informal ¶

normalize ¶

Stopwords¶

Available Sets¶

Usage¶

Using with scikit-learn¶

Text Preprocessing & Cleaning¶

PII Masking¶

olaverse.nlp.mask_pii ¶

Text Cleaning¶

olaverse.nlp.clean_text ¶

Retrieval (New in v0.1.5)¶

Reranker¶

olaverse.nlp.Reranker ¶

Methods:¶

load ¶

score ¶

rank ¶

Embedder¶

olaverse.nlp.Embedder ¶

Methods:¶

load ¶

encode ¶

Auto-routing (`model="auto"`) — New in v0.1.4¶