A token can range from individual letters to several words in a stock phrase, from what little I understand. It’s a piece of whatever they’re doing behind the scenes leaking into the public-facing terminology.
According to my SO, it’s safe enough to just say “a token is a word” as long as you remember this is only an approximation.
A token can range from individual letters to several words in a stock phrase, from what little I understand. It’s a piece of whatever they’re doing behind the scenes leaking into the public-facing terminology.
According to my SO, it’s safe enough to just say “a token is a word” as long as you remember this is only an approximation.