Skip to content
v0.2.1

olaverse-foundry

Small, specialised models from big general ones — even when your language has no training data

pip install olaverse-foundry
Quick Start GitHub PyPI

What is olaverse-foundry?

The normal way to train a model assumes you have data. olaverse-foundry is the pipeline for when you don't: synthesize the training data (MT translation into 400+ languages, LLM query generation, mined hard negatives), distil or contrastively train a small model on it, and prove it head-to-head against mBERT / e5 / LaBSE — one library, one afternoon. Every artifact is a standard HuggingFace directory; production code needs only transformers.

The same machinery covers the data-rich cases too — a compact classifier distilled from a 178M-parameter teacher, an int8 encoder that runs on CPU, a multi-teacher causal-LM student — and it is model-agnostic: pass an HF AutoModel*, or your own module that returns .logits / .last_hidden_state.

Start here: 60-second quickstart · Which trainer do I need? · Concepts & glossary

The flagship walkthrough: a retriever for a language with no training data →

What you can do

🔥
Distillation
Transfer knowledge from one or many teachers into a smaller student. CE+KL for causal LMs, pooled MSE/cosine for embeddings, and token-level hidden-state distillation for encoders. Logit caching and cross-tokenizer alignment included.
Explore Training →
🌍
Synthetic data
No training data in your language? Manufacture it: MT translation into 400+ languages (MADLAD), LLM query generation, and encoder-mined hard negatives — commercially clean, ready for contrastive training.
Explore Synthetic Data →
🔎
Retrieval
Train an embedding model with InfoNCE (the e5/bge recipe) and score it with nDCG / Recall against the strong multilingual baselines — each model encoded with its own pooling and prefixes.
Explore Retrieval →
📚
Pretraining (MLM)
Train an encoder backbone from scratch with masked-language-modeling — your own architecture, your own tokenizer, no teacher required.
Explore MLM →
🎯
Task heads
Fine-tune sequence- and token-classification heads on any base encoder. Full fine-tune or frozen-backbone (train only the head) so many heads share one encoder.
Explore Heads →
📐
Growth & scaling
SOLAR-style depth up-scaling by duplicating layers — native (no external merge tool). The layer prefix is auto-detected, so it works on Llama, BERT, GPT-2, and more.
Explore Growth →
🪶
Quantization (QAT)
Quantization-aware training with int8/int4 fake-quant, plus int8 weight export and a footprint report — keep accuracy on-device.
Explore QAT →
🧩
Skill packs (LoRA)
Detachable LoRA adapters bound to a base-model hash. Snap them onto a frozen base; PEFT-format round-trip included.
Explore Skill Packs →
📦
DataPipeline
One adapter for HF datasets (incl. streaming), text lists, dicts, and numpy. lm / embed modes, reservoir shuffle, and labels for head training.
Explore DataPipeline →
📊
Evaluation
Head-to-head model comparison: fine-tune the same head on each model and print an accuracy / macro-F1 / params table — "better" as a table, not a vibe.
Explore Evaluation →
⚡
Inference
Load any trained model for generation, optionally 4-bit/8-bit quantized, with an optional skill pack merged in.
Explore Inference →
📋
YAML recipes
Pydantic-validated recipe files describing a whole build. Preview the plan before spending a GPU hour.
Explore Recipes →
🖥️
CLI
foundry doctor checks your environment, foundry plan previews a recipe, foundry run / foundry embed execute it.
Explore CLI →

Install

# Core — schema validation, growth planning, recipe parsing (no GPU required)
pip install olaverse-foundry

# Real training + inference (torch, transformers, safetensors, accelerate)
pip install "olaverse-foundry[torch]"

# LoRA skill packs
pip install "olaverse-foundry[torch,lego]"

# HuggingFace dataset streaming
pip install "olaverse-foundry[torch,data]"

# Fast cross-tokenizer alignment (rapidfuzz)
pip install "olaverse-foundry[torch,align]"

# W&B experiment tracking
pip install "olaverse-foundry[torch,logging]"

# Everything
pip install "olaverse-foundry[all]"

Quantized inference additionally needs bitsandbytes; QAT and growth need only [torch].

Trainers at a glance

Trainer Builds Teacher? Notes
TorchDistillTrainer causal LM yes (1+) CE + KL, teachers run every step
CachedDistillTrainer causal LM yes (1+) caches teacher logits; multi-GPU via accelerate
EmbeddingDistillTrainer embedding model yes pooled MSE / cosine for bi-encoders / rerankers
MLMTrainer encoder base no masked-LM pretraining from scratch
EncoderDistillTrainer encoder base yes token-level hidden-state distillation
DistilMLMTrainer encoder base yes distillation + MLM in one loss (DistilBERT objective)
ContrastiveTrainer retrieval embeddings no InfoNCE on pairs; in-batch + hard negatives
SequenceClassificationTrainer classifier head — sequence labels; full or frozen backbone
TokenClassificationTrainer token head (NER) — token labels; full or frozen backbone

Every trainer shares the same production feature set: mixed precision, gradient accumulation, LR scheduler with warmup, reproducible seed, checkpoint save/resume, auto-checkpoint, eval loop, OOM handling, and W&B / TensorBoard logging.

Quick example — distil a causal LM

import torch, numpy as np
from foundry import TorchDistillTrainer, TorchTrainConfig, TeacherRegistry
from foundry.teachers import ToyTeacher

student  = torch.nn.Linear(16, 32)                       # any model with .logits
teachers = TeacherRegistry([ToyTeacher(vocab=32)])       # or HFTeacher(...)
data     = [np.random.randint(0, 32, (4, 16)) for _ in range(50)]

trainer = TorchDistillTrainer(student, teachers, TorchTrainConfig(
    epochs=2, lr_scheduler="cosine", warmup_steps=5,
    save_every=25, save_dir="/tmp/run",
))
result = trainer.train(data)
print(result["losses"][-1])