omnitok

This package documents the chat templates and core tokenizers used in Apertus, and provides utilities for extending LLaMA-3/Apertus text tokenizers with additional vision and audio modalities.

Extension Scripts

# Add vision
python -m omnitok.cli add-modality \
    --input-tokenizer swiss-ai/Apertus-8B-2509 \
    --output-path ./omni_vision \
    --modality vision \
    --vocab-size 131072

# Stack audio on top
python -m omnitok.cli add-modality \
    --input-tokenizer ./omni_vision \
    --output-path ./omni_vision_audio \
    --modality audio \
    --vocab-size 4096

# Add instruct (chat template + SFT sequences)
python -m omnitok.cli add-instruct \
    --base-tokenizer-path ./omni_vision_audio \
    --instruct-tokenizer-path swiss-ai/Apertus-8B-2509-Instruct \
    --output-path ./omni_instruct

Or as a library:

from omnitok import add_modality, create_instruct_tokenizer

add_modality("swiss-ai/Apertus-8B-2509", "./omni_vision", "vision", vocab_size=131072)
add_modality("./omni_vision", "./omni_vision_audio", "audio", vocab_size=4096)
create_instruct_tokenizer("./omni_vision_audio", "swiss-ai/Apertus-8B-2509-Instruct", "./omni_instruct")

Token layout

[0 .. base-1]           text tokens (unchanged)
[base .. base+199]      200 reserved OMNI slots
  slot 0                  boundary marker (never renamed)
  slots 1-7               vision structure tokens
  slots 8-15              audio structure tokens
  slots 16-199            reserved for future modalities
[base+200 .. ]          content tokens (appended per modality in order added)

Reserved slot allocation

Slots	Modality	Tokens
0	--	<|RESERVED_OMNI_000|> (boundary)
1-7	Vision	img_start, img_end, img_token_start, img_end_of_row, img_end_of_frame, img_generation_start, image
8-15	Audio	audio_start, audio_end, stt_transcribe, stt_continue, tts_continue, audio, stt_translate, audio_annotate
16-199	--	Reserved

Token aliases

<image> and <|image|> encode to the same token ID. Same for <audio> / <|audio|>. Handled by the tokenizer's normalizer -- no manual .replace() needed in data loaders.

Known codebook sizes

Tokenizer	Modality	Codebook Size
Emu3.5 (IBQ)	Vision	131,072
Emu3	Vision	32,768
WavTokenizer	Audio	4,096

Validating a deployed model

To check that a served model directory ships the exact canonical tokenizer (chat template, tokenizer, config, special tokens), run validate_model.sh against it. It compares md5 checksums against the manifests under validation/ and exits non-zero on any mismatch.

# Remote (no checkout needed): cd into the model dir, then validate it
cd /path/to/served/model
curl -fsSL https://raw.githubusercontent.com/swiss-ai/apertus-omni-tokenizer/main/validate_model.sh \
  | bash

# To validate the 1.0 tokenizer instead, pass the path slot + model name
#   | bash -s -- . Apertus_1

# Repair mode: download the canonical copy of each missing/mismatched file,
# back the old one up as <file>.bak, then re-validate
#   | bash -s -- --fix

# From a local checkout
bash validate_model.sh /path/to/served/model
bash validate_model.sh --fix /path/to/served/model

Validating Apertus_1p5 tokenizer in: /path/to/served/model

✔ chat_template.jinja
✔ tokenizer.json
✔ tokenizer_config.json
✔ special_tokens_map.json

OK: Apertus_1p5 tokenizer matches the canonical checksums.

Arguments: validate_model.sh [--fix] [MODEL_PATH] [MODEL_NAME] -- MODEL_PATH defaults to the current directory, MODEL_NAME defaults to Apertus_1p5 (pass Apertus_1 to validate the 1.0 tokenizer). With --fix, each missing or mismatched manifest file is re-downloaded from this repo (verified against the manifest md5 before installing; the existing file is saved as <file>.bak, never overwriting earlier backups — repeat runs write <file>.bak.1, .bak.2, ...) and the validation is re-run. generation_config.json is checked field-level and is not auto-fixed. The canonical checksums are regenerated by validation/gen_checksums.sh and kept in sync by CI.

Structure

apertus-omni-tokenizer/
├── README.md
├── pyproject.toml
├── validate_model.sh        # verify a served model dir matches canonical md5s
├── omnitok/
│   ├── __init__.py      # public API exports
│   ├── modalities.py    # ModalityConfig dataclass, built-in VISION/AUDIO configs
│   ├── builder.py       # add_modality() -- the main engine
│   ├── instruct.py      # create_instruct_tokenizer() -- chat template + SFT sequences
│   ├── io.py            # low-level file I/O (rename, alias, detect, save)
│   └── cli.py           # CLI wrapper (python -m omnitok.cli)
├── tests/
│   ├── conftest.py           # shared fixtures
│   ├── test_alias.py         # token alias tests (<image> == <|image|>)
│   ├── test_builder.py       # add_modality tests
│   ├── test_chat_template.py # add chat template test
│   ├── test_task_tokens.py   # task token contract tests
│   └── test_tokenizers.py    # checked-in tokenizers load + special-token IDs
├── examples/
│   └── rename_tool_tokens.py # Script used to add new special tokens in tool calling parsing
├── tokenizers/
│   ├── Apertus_1/            # Instructed tokenizer used for Apertus 1.0
│   │   ├── tokenizer.json
│   │   └── tokenizer_config.json
│   └── Apertus_1p5/          # Instructed tokenizer used for Apertus 1.5
│       ├── tokenizer.json
│       └── tokenizer_config.json
├── chat_templates/
│   ├── Apertus_1/            # Chat template used for Apertus 1.0
│   │   └── chat_template.jinja
│   └── Apertus_1p5/          # Chat template used for Apertus 1.5
│       └── chat_template.jinja
└── validation/
    ├── gen_checksums.sh      # regenerate the manifests below
    ├── Apertus_1.md5         # canonical md5s for the 1.0 tokenizer
    └── Apertus_1p5.md5       # canonical md5s for the 1.5 tokenizer

Adding a new modality = one new ModalityConfig entry in modalities.py.

Author

Yixuan Xu (yixuan.xu@ai.ethz.ch)