Skip to content

Olaverse β€” small models, sharp focus: LID, DiacNet, MIST, Prism

v0.1.5 β€” Now with Retrieval & Vision

The Olaverse SDK

African NLP Β· MIST Language Models Β· Text-to-Speech Architecture

pip install olaverse
Get Started GitHub Hugging Face

What is Olaverse?

Olaverse is a unified Python SDK for African language AI. It gives you production-ready NLP tools, a clean interface to the MIST model family, domain-specific LLMs, and a TTS pipeline architecture β€” all in one package.

πŸ—£οΈ
NLP & Tokenization
Diacritization for Yoruba and Igbo, language detection for 5 Nigerian languages, Byte-Level BPE tokenizers with 0% OOV, PII masking, and TTS text normalization.
Explore NLP β†’
⚑
MIST Model Family
8B, 70B, 140B, and Thinking variants. Correct stop tokens and sampling defaults per variant baked in. Local via transformers or hosted via Featherless/Modal.
Explore MIST β†’
🧠
Domain LLMs
LegalPeace β€” fine-tuned Mistral-7B for contract analysis and legal reasoning. Memory-efficient 4-bit inference via unsloth.
Explore LLMs β†’
πŸŽ™οΈ
Speech Architecture
TTS pipeline connecting normalization, diacritization, acoustic model, and vocoder. The NLP front-end is production-ready. Acoustic synthesis is on the roadmap.
Explore Speech β†’
πŸ”Ž
Retrieval
Cross-encoder Reranker for RAG/search pipelines, and a Nigerian-language Embedder for cross-lingual semantic search over Hausa, Yoruba, and Igbo.
Explore Retrieval β†’
πŸ–ΌοΈ
Vision β€” Prism
Lightweight image-to-image models: PrismUpscaler (2x/4x/arbitrary), PrismDenoiser, and PrismSteganography for hiding recoverable messages in images.
Explore Vision β†’
πŸ“Š
Datasets
One-line access to every public olaverse dataset: reranker training pairs, multilingual QG passages, and the DiacBench diacritization benchmark.
Explore Datasets β†’

Install

pip install olaverse
Includes: NLP tools, diacritizers, tokenizers, lightweight LID, PII masking.

pip install olaverse[deeplearning]
Adds: LIDNeural5, MIST local inference (requires GPU).

pip install olaverse[hosted]
Adds: MIST via Featherless, Modal, or any OpenAI-compatible endpoint.

pip install olaverse[legal]
Adds: LegalPeace contract analysis model (requires GPU + unsloth).

pip install olaverse[lid]
Adds: LIDLite25 β€” CPU-only fastText language ID for 25 languages.

pip install olaverse[retrieval]
Adds: Reranker, Embedder (requires sentence-transformers).

pip install olaverse[vision]
Adds: PrismUpscaler, PrismDenoiser, PrismSteganography (requires torch, torchvision, Pillow).

pip install olaverse[data]
Adds: load_dataset β€” every public olaverse dataset on Hugging Face.

pip install olaverse[deeplearning,hosted,legal,lid,retrieval,vision,data]

Quick Start

Language Detection

from olaverse import LIDLite5, LIDNeural5

# Lightweight β€” zero GPU, instant (0.014 ms/sentence)
detector = LIDLite5()
detector.predict("Bawo ni, se daadaa ni?")  # β†’ 'yor'

# Neural β€” 98.96% macro accuracy
neural = LIDNeural5()
neural.load()  # downloads olaverse/lid-neural-5 once
neural.predict("Kedu ka α»‹ mere today?")  # β†’ 'ibo'
neural.predict_proba("How far, wetin dey happen?")
# β†’ {'pcm': 0.991, 'eng': 0.002, ...}

Diacritization

from olaverse import diacritize_yoruba, diacritize_igbo

diacritize_yoruba("Ojo lo si oja lana")
# β†’ 'Γ’jΓ³ lọ sΓ­ ọjΓ  lana'

diacritize_igbo("Kedu ka i mere")
# β†’ 'Kedα»₯ ka α»‹ mere'

MIST β€” Fast (8B, local)

from olaverse import MIST

model = MIST(size="8b")
model.load()
print(model.generate("Explain what makes Yoruba a tonal language."))

MIST β€” Hosted (70B via Featherless)

import os
from olaverse import MIST

model = MIST(size="70b", endpoint="featherless", api_key=os.environ["FEATHERLESS_API_KEY"])
print(model.generate("Write a Python retry decorator with exponential backoff."))

Tokenization

from olaverse import Tokenizer

tok = Tokenizer("yo")  # Yoruba β€” 63% fewer tokens than GPT-4
ids = tok.encode("αΊΈ kΓΊ Γ bọ̀")
tok.decode(ids)  # β†’ 'αΊΈ kΓΊ Γ bọ̀'

Datasets

from olaverse import load_dataset, list_datasets

list_datasets()
# β†’ ['reranker-general-en-llm-judged', 'marco-style-pairs-multi', ...]

bench = load_dataset("diacbench", "yo", split="test")  # requires olaverse[data]
bench[0]
# β†’ {'input': 'Titi di igba ...', 'reference': 'TΓ­tΓ­ di Γ¬gbΓ  ...'}

Supported Languages

Yoruba yor Igbo ibo Hausa hau Nigerian Pidgin pcm English eng
Feature yor ibo hau pcm eng
Language Detection (LIDLite5 / LIDNeural5) βœ… βœ… βœ… βœ… βœ…
Diacritization βœ… βœ… β€” β€” β€”
BPE Tokenizer βœ… βœ… βœ… βœ… via naija
TTS Normalization βœ… βœ… β€” β€” β€”

Model Index

Model Task Size Speed Install
LIDLite5 Language ID (5 langs) 1.1 MB JSON 0.014 ms olaverse
LIDNeural5 Language ID (5 langs) 484 MB 13 ms olaverse[deeplearning]
LIDLite25 Language ID (25 langs) ~5-10 MB <1 ms olaverse[lid]
LIDNeural25 Language ID (25 langs) ~500 MB β€” olaverse[deeplearning]
LIDNeural5_1 Language ID (4 Nigerian langs, no English) ~120 MB β€” olaverse[deeplearning]
MIST-Mini-8B General LLM 15 GB ~63 tok/s olaverse[deeplearning]
MIST-1-70B General LLM 132 GB ~23 tok/s hosted or multi-GPU
MIST-1-140B General LLM 256 GB ~8 tok/s hosted or 2Γ— H200
MIST-Mini-8B-Thinking Reasoning LLM 15 GB ~55 tok/s olaverse[deeplearning]
LegalPeace Legal reasoning 7B (4-bit) β€” olaverse[legal]
DiacNet (5 Yoruba/Igbo variants) Diacritization 1 MB – 503 MB β€” olaverse / [deeplearning]
diacnet-1.0 Diacritization (10 langs) ~300 MB Slow olaverse[deeplearning]
OTK-BPE-50k (5 Nigerian variants) Tokenization ~3 MB each β€” olaverse
OTK-BPE (9 Swahili/Kinyarwanda/merged variants) Tokenization varies β€” olaverse
Reranker (2 sizes) Reranking 23 MB – 150M params β€” olaverse[retrieval]
Embedder Sentence embeddings (ha/yo/ig) ~120 MB β€” olaverse[retrieval]
PrismUpscaler (3 sizes) Image super-resolution ~25K params – small β€” olaverse[vision]
PrismDenoiser Image denoising Small U-Net β€” olaverse[vision]
PrismSteganography Image steganography Small U-Net β€” olaverse[vision]

What's New in v0.1.5

  • 25-language identification β€” LIDLite25 (fastText) and LIDNeural25 (XLM-RoBERTa) extend language detection well beyond the original 5 Nigerian languages; LIDNeural5_1 adds a compact Nigerian-only classifier built on the new mist-encoder-base-ng
  • diacnet-1.0 β€” a single multilingual ByT5 model restores diacritics across 10 languages (Yoruba, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, Italian), added to Diacritizer via lang=
  • OTK-BPE multilingual tokenizer family β€” Swahili, Kinyarwanda, and a merged French/Kinyarwanda/English/Swahili tokenizer, each at 50k/100k/150k vocab, available through the same Tokenizer class
  • New olaverse.nlp retrieval toolkit β€” Reranker (cross-encoder, 2 sizes) and Embedder (cross-lingual Hausa/Yoruba/Igbo sentence embeddings) for RAG/search pipelines
  • New olaverse.vision module β€” PrismUpscaler, PrismDenoiser, and PrismSteganography, general-purpose image-to-image models
  • New extras: olaverse[lid], olaverse[retrieval], olaverse[vision]

Previously, in v0.1.4:

  • MIST wrapper β€” unified interface for all MIST variants with correct stop tokens, sampling defaults, and local/hosted endpoint switching
  • LIDNeural5 moved to olaverse.nlp β€” its correct home alongside LIDLite5 (backward-compat import from olaverse.llm preserved)
  • ExperimentalWarning on speech classes β€” honest signalling that acoustic synthesis is not yet available
  • olaverse[hosted] extra β€” pip install olaverse[hosted] for Featherless/Modal inference