All terms
Models and requests
Token
A token is a chunk of text a language model processes, such as part of a word, a word, punctuation, or code symbols. Token boundaries are chosen by a tokenizer and do not always match words.
Example
The code `user.id` may be represented as several tokens before the model predicts the next one.