Skip to content

Add GPT-5.1 support with o200k_base encoding#468

Open
magkue wants to merge 3 commits intoopenai:mainfrom
magkue:main
Open

Add GPT-5.1 support with o200k_base encoding#468
magkue wants to merge 3 commits intoopenai:mainfrom
magkue:main

Conversation

@magkue
Copy link
Copy Markdown

@magkue magkue commented Nov 21, 2025

Add GPT-5.1 Model Support

Fixes Add GPT-5.1 with the o200k_base encoding in tiktoken/model.py #464

Changes

  • Added GPT-5.1 model mapping: Maps gpt-5.1 to o200k_base encoding in MODEL_TO_ENCODING and MODEL_PREFIX_TO_ENCODING

Impact

Enables tiktoken to properly tokenize text for GPT-5.1 model using o200k_base encoding as requested in #464

@eshaanpathak
Copy link
Copy Markdown

Thank you! @hauntsaninja, can you please review this PR?

@Itsindigo
Copy link
Copy Markdown

Hey @magkue -- could we update this for other models in the 5.x paths, 5.1, 5.2, 5.3, 5.4 etc -- you might need some more generalised pattern matching logic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants