stefan-it
/

nanochat-german-tokenizer

Model card Files Files and versions

nanochat-german-tokenizer

9.83 MB

1 contributor

History: 12 commits

stefan-it's picture

feat: add model config

0049e29 verified about 1 month ago

.gitattributes

1.52 kB

initial commit about 2 months ago
README.md

2.28 kB

docs: add new section with tokenizer stats about 1 month ago
config.json

736 Bytes

feat: add model config about 1 month ago
token_bytes.pt
Detected Pickle imports (3)
- "collections.OrderedDict",
- "torch.IntStorage",
- "torch._utils._rebuild_tensor_v2"
What is a pickle import?
264 kB
xet

feat: add newly trained tokenizer (final one) about 1 month ago
tokenizer.json

8.68 MB

feat: add HF tokenizer version about 1 month ago
tokenizer.pkl
Detected Pickle imports (1)
- "tiktoken.core.Encoding"
How to fix it?
889 kB
xet

feat: add newly trained tokenizer (final one) about 1 month ago
tokenizer_config.json

510 Bytes

feat: add tokenizer config about 1 month ago