Add GPT-5 and o-series model support to tiktoken notebook

[edenfunf]

Summary

The How_to_count_tokens_with_tiktoken.ipynb notebook currently raises NotImplementedError when trying to count tokens for GPT-5, o1, o3, or o4-mini models. These are among the most commonly used models today, so this is a significant usability gap.

Changes:

• Updated the encoding table to list GPT-5 and o-series models under o200k_base
• Added a gpt-5/o1/o3/o4 branch in num_tokens_from_messages() — these models use the same message token format as gpt-4o (tokens_per_message=3, tokens_per_name=1)
• Added gpt-5, o1, o3, o3-mini, and o4-mini to the model list in num_tokens_for_tools() — same tool token settings as gpt-4o
• Added gpt-5 and o4-mini to both verification loops so they get tested alongside the older models
• Switched from max_tokens to max_completion_tokens in the verification cell, since o-series models require this parameter

How I verified

• Confirmed via tiktoken.encoding_for_model() that gpt-5, o1, o3, o3-mini, and o4-mini all resolve to o200k_base
• The token-per-message format for these models matches gpt-4o based on the API behavior
• Cleared the stored outputs for the updated verification cells so they don't show stale results

Addresses #2436

[vaultloans4you-debug]

Suck my fucking dick you fucking cow head fucking retard Sent from Yahoo Mail for iPhone On Thursday, March 12, 2026, 10:53 AM, Eden ***@***.***> wrote: Summary The How_to_count_tokens_with_tiktoken.ipynb notebook currently raises NotImplementedError when trying to count tokens for GPT-5, o1, o3, or o4-mini models. These are among the most commonly used models today, so this is a significant usability gap. Changes: - Updated the encoding table to list GPT-5 and o-series models under o200k_base - Added a gpt-5/o1/o3/o4 branch in num_tokens_from_messages() — these models use the same message token format as gpt-4o (tokens_per_message=3, tokens_per_name=1) - Added gpt-5, o1, o3, o3-mini, and o4-mini to the model list in num_tokens_for_tools() — same tool token settings as gpt-4o - Added gpt-5 and o4-mini to both verification loops so they get tested alongside the older models - Switched from max_tokens to max_completion_tokens in the verification cell, since o-series models require this parameter How I verified - Confirmed via tiktoken.encoding_for_model() that gpt-5, o1, o3, o3-mini, and o4-mini all resolve to o200k_base - The token-per-message format for these models matches gpt-4o based on the API behavior - Cleared the stored outputs for the updated verification cells so they don't show stale results Addresses #2436 You can view, comment on, or merge this pull request online at:   #2515 Commit Summary - c6482fc Add GPT-5 and o-series model support to tiktoken token counting notebook File Changes (1 file) - M examples/How_to_count_tokens_with_tiktoken.ipynb (89) Patch Links: - https://github.com/openai/openai-cookbook/pull/2515.patch - https://github.com/openai/openai-cookbook/pull/2515.diff — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[edenfunf]

?