Skip to content

Olaverse β€” small models, sharp focus: LID, DiacNet, MIST, Prism

v0.3.1 β€” Now with Title & Question Generation

The Olaverse SDK

Open-source NLP infrastructure for underrepresented languages

pip install olaverse
Explore Models GitHub Hugging Face

30-Second Quick Start

pip install olaverse

from olaverse.nlp import Diacritizer

d = Diacritizer(model="auto")   # detects the language, routes to the right model

d.restore("Ojo lo si oja lana")     # Yoruba
# β†’ 'Γ’jΓ³ lọ sΓ­ ọjΓ  lana'

d.restore("Kedu ka i mere")         # Igbo
# β†’ 'Kedα»₯ ka α»‹ mere'

Need more languages? diactag-1.0 covers 10 β€” and cannot change your text beyond adding marks. With pip install olaverse[deeplearning]:

d = Diacritizer(model="diactag-1.0", lang="yo")

d.restore("se eranko naa si gbo o?")
# β†’ 'αΉ£Γ© αΊΉranko nÑà sΓ¬ gbọ́ ọ?'

# Leave lang out and it detects the language itself
Diacritizer(model="diactag-1.0").restore("Co ay rat dam dang")
# β†’ 'CΓ΄ αΊ₯y rαΊ₯t Δ‘αΊ£m Δ‘ang'

What is Olaverse?

Olaverse is an open-source multilingual AI infrastructure toolkit for building NLP, speech, retrieval, and language systems for underrepresented languages.

It gives you production-ready APIs for language detection, diacritization, tokenization, embeddings, reranking, and speech-text preprocessing β€” plus the MIST LLM family, domain LLMs, and lightweight vision models β€” all in one Python package.

Why Olaverse?

1. Low-resource language support

Many languages lack high-quality AI tooling. Olaverse ships working models for them today:

  • Language detection β€” 5 to 25 languages, from a 1.1 MB CPU model to neural classifiers
  • Diacritization β€” restore tones and accents across 10 languages
  • Tokenization β€” byte-level BPE tokenizers with 0% OOV, up to 63% fewer tokens than GPT-4 on Yoruba
  • Embeddings & retrieval β€” cross-lingual semantic search for Hausa, Yoruba, Igbo
  • Speech preprocessing β€” TTS text normalization for Yoruba, Igbo, and Nigerian Pidgin

2. Production-ready APIs

Instead of wrestling with AutoModel.from_pretrained(), checkpoints, and generation configs:

Diacritizer(model="auto").restore(text)
LIDLite5().predict(text)
Reranker().rank(query, passages)
Tokenizer("yo").encode(text)

Correct defaults β€” stop tokens, sampling parameters, model routing β€” are baked in.

3. Lightweight deployment

Models are sized for real infrastructure, not just research clusters:

  • LIDLite5 β€” 1.1 MB, 0.014 ms per sentence, pure Python, zero GPU
  • diacnet-yor-viterbi β€” ~7 MB, fast CPU inference
  • diactag-1.0 β€” 38 MB int8, 10 languages, 244 characters/second on one CPU core
  • Reranker (22.7M) β€” 23 MB cross-encoder
  • Prism vision models β€” ~25K-parameter upscalers that run anywhere

Core Capabilities

πŸ—£οΈ
NLP & Tokenization
Diacritization for 10 languages, language detection for 5–25 languages, Byte-Level BPE tokenizers with 0% OOV, PII masking, and TTS text normalization.
Explore NLP β†’
⚑
MIST Model Family
8B, 70B, 140B, and Thinking variants. Correct stop tokens and sampling defaults per variant baked in. Local via transformers or hosted via Featherless/Modal.
Explore MIST β†’
🧠
Domain LLMs
LegalPeace β€” fine-tuned Mistral-7B for contract analysis and legal reasoning. Memory-efficient 4-bit inference via unsloth.
Explore LLMs β†’
πŸŽ™οΈ
Speech Architecture
TTS pipeline connecting normalization, diacritization, acoustic model, and vocoder. The NLP front-end is production-ready. Acoustic synthesis is on the roadmap.
Explore Speech β†’
πŸ”Ž
Retrieval
Cross-encoder Reranker for RAG/search pipelines, and a Nigerian-language Embedder for cross-lingual semantic search over Hausa, Yoruba, and Igbo.
Explore Retrieval β†’
πŸ–ΌοΈ
Vision β€” Prism
Lightweight image-to-image models: PrismUpscaler (2x/4x/arbitrary), PrismDenoiser, and PrismSteganography for hiding recoverable messages in images.
Explore Vision β†’
πŸ“Š
Datasets
One-line access to every public olaverse dataset: reranker training pairs, multilingual QG passages, and the DiacBench diacritization benchmark.
Explore Datasets β†’
Model What it does Size Start here
DiacTag Restores accents & tones in 10 languages, guaranteed not to alter the text 38 MB – 150 MB DiacTag β†’
DiacNet Restores accents, tones & special characters in 10 languages 1 MB – 1.1 GB DiacNet β†’
LID family Language detection, 5–25 languages 1.1 MB – 500 MB Language Detection β†’
OTK-BPE Tokenizers for 8+ languages, 0% OOV ~3 MB each Tokenizers β†’
Reranker / Embedder RAG & cross-lingual search 23 MB+ Retrieval β†’
MIST General LLMs β€” 8B / 70B / 140B / Thinking 15 – 256 GB MIST β†’
Prism Image upscaling, denoising, steganography ~25K params+ Prism β†’

Full catalog with comparison tables: Model Overview β†’

Use Cases

Olaverse components slot into production pipelines for:

  • Speech AI β€” ASR post-processing β†’ normalization β†’ diacritization β†’ TTS front-end
  • OCR correction β€” restore diacritics that scanners drop
  • Search β€” language detection, query normalization, cross-lingual retrieval, reranking
  • Education β€” language-learning tools with correct tone marks
  • Customer support β€” detect and route messages in Nigerian languages and Pidgin
  • Translation pipelines β€” clean, diacritized input for MT systems

See Solutions β†’ for worked examples.

Benchmarks

Model Size Speed Macro F1
LIDLite5 1.1 MB 0.014 ms 98.12%
LIDNeural5 484 MB 13.3 ms 98.96%

All published numbers β€” LID, diacritization, tokenizer efficiency β€” in one place: Benchmarks β†’

Install

pip install olaverse
Includes: NLP tools, diacritizers, tokenizers, lightweight LID, PII masking.

pip install olaverse[deeplearning]
Adds: LIDNeural5, diactag-1.0, diacnet-1.0/1.1, MIST local inference (requires GPU).

pip install olaverse[hosted]
Adds: MIST via Featherless, Modal, or any OpenAI-compatible endpoint.

pip install olaverse[legal]
Adds: LegalPeace contract analysis model (requires GPU + unsloth).

pip install olaverse[lid]
Adds: LIDLite25 β€” CPU-only fastText language ID for 25 languages.

pip install olaverse[retrieval]
Adds: Reranker, Embedder (requires sentence-transformers).

pip install olaverse[vision]
Adds: PrismUpscaler, PrismDenoiser, PrismSteganography (requires torch, torchvision, Pillow).

pip install olaverse[data]
Adds: load_dataset β€” every public olaverse dataset on Hugging Face.

pip install olaverse[deeplearning,hosted,legal,lid,retrieval,vision,data]

Supported Languages

Yoruba yor Igbo ibo Hausa hau Nigerian Pidgin pcm English eng
Feature yor ibo hau pcm eng
Language Detection (LIDLite5 / LIDNeural5) βœ… βœ… βœ… βœ… βœ…
Diacritization βœ… βœ… βœ… (diacnet-1.0) β€” β€”
BPE Tokenizer βœ… βœ… βœ… βœ… via naija
TTS Normalization βœ… βœ… β€” βœ… β€”

Beyond Nigerian languages: LIDLite25/LIDNeural25 detect 25 languages; diacnet-1.0 diacritizes 10 (incl. Vietnamese, Polish, Turkish, Portuguese, Spanish, French, Italian); OTK-BPE tokenizers cover Swahili and Kinyarwanda.

What's New in v0.2.0

  • MISTTitleGenerator β€” short chat titles from a user's first message, wrapping mist-tg-0.3b
  • MISTQuestionGenerator β€” search-style question generation from a passage across 25 languages, wrapping mist-qg-1.5b. Useful as an endpoint, or as a data factory minting (query, positive) pairs to train retrievers
  • diacnet-1.0 no longer truncates long text β€” the model was trained on sentence-length input, so multi-sentence text is now segmented, restored a sentence at a time, and rejoined. Previously a 358-character paragraph came back at 235 characters with its tail dropped. Override with split_sentences=False or your own splitter=
  • Corrected model documentation β€” every documented example was re-run against the real checkpoints, and the claims that didn't hold were fixed (see the changelog)

Previously, in v0.1.5:

  • 25-language identification β€” LIDLite25 (fastText) and LIDNeural25 (XLM-RoBERTa) extend language detection well beyond the original 5 Nigerian languages; LIDNeural5_1 adds a compact Nigerian-only classifier built on the new mist-encoder-base-ng
  • diacnet-1.0 β€” a single multilingual ByT5 model restores diacritics across 10 languages (Yoruba, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, Italian), added to Diacritizer via lang=
  • OTK-BPE multilingual tokenizer family β€” Swahili, Kinyarwanda, and a merged French/Kinyarwanda/English/Swahili tokenizer, each at 50k/100k/150k vocab, available through the same Tokenizer class
  • New olaverse.nlp retrieval toolkit β€” Reranker (cross-encoder, 2 sizes) and Embedder (cross-lingual Hausa/Yoruba/Igbo sentence embeddings) for RAG/search pipelines
  • New olaverse.vision module β€” PrismUpscaler, PrismDenoiser, and PrismSteganography, general-purpose image-to-image models
  • New extras: olaverse[lid], olaverse[retrieval], olaverse[vision]

Previously, in v0.1.4:

  • MIST wrapper β€” unified interface for all MIST variants with correct stop tokens, sampling defaults, and local/hosted endpoint switching
  • LIDNeural5 moved to olaverse.nlp β€” its correct home alongside LIDLite5 (backward-compat import from olaverse.llm preserved)
  • ExperimentalWarning on speech classes β€” honest signalling that acoustic synthesis is not yet available
  • olaverse[hosted] extra β€” pip install olaverse[hosted] for Featherless/Modal inference

Commercial Support

Need custom language support, fine-tuning, or deployment help? We work with teams adding underrepresented-language capability to their products. Learn more β†’