Skip to content

Use gpt-4o's tokenizer #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 18, 2025
Merged

Use gpt-4o's tokenizer #258

merged 3 commits into from
Jun 18, 2025

Conversation

AmgadHasan
Copy link
Contributor

@AmgadHasan AmgadHasan commented Apr 11, 2025 •
edited by filipchristiansen
Loading

Use gpt-4o tokenizer (o200k_base) instead of the old gpt-4 tokenizer (o100k_base).

@AmgadHasan
Copy link
Contributor Author

cc @cyclotruc

@filipchristiansen filipchristiansen marked this pull request as draft June 17, 2025 11:35
@coderamp-labs coderamp-labs deleted a comment from AmgadHasan Jun 18, 2025
@filipchristiansen filipchristiansen marked this pull request as ready for review June 18, 2025 08:51
@filipchristiansen filipchristiansen self-requested a review June 18, 2025 09:14
@filipchristiansen
feat: bump tiktoken to ≥0.7.0 and drop Python 3.7 support
1130b55 
Context
-------
The previous commit switched token counting from **cl100k_base** to **o200k_base**.
That encoding is available only in **tiktoken ≥ 0.7.0**, whose wheels require
Python 3.8+.  We already exclude 3.7 from the CI matrix, so this raises the
documented minimum Python version without affecting tested platforms.

Changes
-------
* **README.md** – “Python 3.7+” → **“Python 3.8+”**
* **pyproject.toml**
  * `tiktoken` → `tiktoken>=0.7.0`  # o200k_base support
  * Remove classifier *Programming Language :: Python :: 3.7*
* **requirements.txt** – `tiktoken` → `tiktoken>=0.7.0`
* No code changes beyond the earlier one-liner already in this PR.

Impact
------
* **Breaking** for users pinned to Python 3.7; they must upgrade to ≥ 3.8.
* No new runtime dependencies—just a higher floor for *tiktoken* and Python.
@filipchristiansen
Copy link
Contributor

PR overview

This PR does three tightly-related things:

  1. Token counter – changes

    tiktoken.get_encoding("cl100k_base")  ➜  tiktoken.get_encoding("o200k_base")

    so counts match GPT-4o / 4o-mini.

  2. Dependency floor – sets tiktoken>=0.7.0, the first wheel that ships the o200k_base vocab.

  3. Runtime floor – removes Python 3.7 from docs, classifiers, and requirements because tiktoken≥0.7.0 needs Python 3.8+ (CI already runs 3.8-3.13).

Files touched

  • src/gitingest/output_formatters.py
  • pyproject.toml
  • requirements.txt
  • README.md

Breaking changes & actions

Who’s affected Required action
Users still on Python 3.7 Upgrade to Python 3.8+
Environments pinning tiktoken==0.6.* or older pip install -U "tiktoken>=0.7.0"

No new runtime libraries are introduced—just the higher version floor.

@filipchristiansen filipchristiansen merged commit 2dea7c8 into coderamp-labs:main Jun 18, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@filipchristiansen filipchristiansen filipchristiansen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AmgadHasan @filipchristiansen