tokfit

Byte-level BPE token counting and budget trimming for OpenAI’s o200k_base encoding.

Pure Gleam on both the Erlang and JavaScript runtimes. No NIFs, no FFI, no network access, and no filesystem access at run time.

Types

Tokenizer

opaque </>

A loaded o200k_base vocabulary.

Build one with tokenizer/0 and keep it. Every call on a fresh handle reparses 199,998 entries, which costs seconds, so a loop that builds a handle per call spends all its time in the parser.

Gleam has no module-level mutable state and this package takes no target-specific dependency, so a plain function cannot cache the table across calls. Holding the handle is the only way to pay for it once.

pub opaque type Tokenizer

Trimmed

</>

A prefix that fits inside a token budget, and what it cost.

pub type Trimmed {
  Trimmed(
    text: String,
    tokens: Int,
    characters: Int,
    dropped_pieces: Int,
  )
}

Constructors

```
Trimmed(
  text: String,
  tokens: Int,
  characters: Int,
  dropped_pieces: Int,
)
```
Arguments

text

The retained prefix of the input.

tokens

Its exact token count.

characters

Its length in grapheme clusters.

dropped_pieces

How many trailing pre-tokens were discarded.

Values

count

</>

pub fn count(tokenizer: Tokenizer, text: String) -> Int

The exact number of o200k_base tokens in text.

The empty string is zero tokens. Special tokens such as <|endoftext|> are not recognised and tokenize as ordinary text, which is what OpenAI calls encode_ordinary.

encode

</>

pub fn encode(tokenizer: Tokenizer, text: String) -> List(Int)

The exact o200k_base token ids for text.

encoding_name

</>

pub fn encoding_name() -> String

The name of the encoding this package implements.

tokenizer

</>

pub fn tokenizer() -> Tokenizer

Load the vocabulary. Seconds, once per handle, and by far the larger half of the cost of a first call.

trim_to_budget

</>

pub fn trim_to_budget(
  tokenizer: Tokenizer,
  text: String,
  budget: Int,
) -> Trimmed

The longest prefix of text whose token count is at most budget.

Cuts happen at pre-token boundaries and never inside one, so the tokens of a result always equal count of that result’s text. The cost of that guarantee is that the result can sit well under budget: the first pre-token that does not fit stops the trim, so a leading pre-token larger than the whole budget yields the empty string. A budget of zero or less always yields an empty result.

Constructors

Arguments