feat: add HF tokenizer version

Manny thanks to

@Xenova
for pointing out, how to convert the tiktoken one to HF, see here: https://huggingface.co/karpathy/nanochat-d32/discussions/1#68fbd549bd6f08353a95125e

Files changed (1) hide show

tokenizer.json +0 -0

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff