Vietnamese Tokenizer
This repository contains a ByteLevel BPE tokenizer trained from scratch specifically for the Vietnamese language, designed for decoder-only language model pretraining.
🚀 Usage
Load tokenizer
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"tranhuyHoang/mini_VN_decoder_tokenizer",
use_fast=True
)
- Downloads last month
- 39
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support