From the chat APIs alone, we can extract two useful oracles:
Token length oracle: Given any string s, return len(tokenize(s)).
Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s).
The first oracle (token count) is straightforward. The second oracle's main purpose is to resolve ambiguities in merge order. For example, without it we couldn’t distinguish between the segmentations (a, bc) and (ab, c). By leveraging the model’s ability to repeat text in a controlled way, we can construct this second oracle from the regular chat interface.
With only these two oracles, is it possible to reconstruct an equivalent tokenizer — one that produces the exact same token boundaries/segmentation on any input (up to token ID remapping)?
From the chat APIs alone, we can extract two useful oracles: Token length oracle: Given any string s, return len(tokenize(s)).
Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s).
The first oracle (token count) is straightforward. The second oracle's main purpose is to resolve ambiguities in merge order. For example, without it we couldn’t distinguish between the segmentations (a, bc) and (ab, c). By leveraging the model’s ability to repeat text in a controlled way, we can construct this second oracle from the regular chat interface.
With only these two oracles, is it possible to reconstruct an equivalent tokenizer — one that produces the exact same token boundaries/segmentation on any input (up to token ID remapping)?