Questions related to inputs and outputs
When we print text after the processor.apply_chat_template() it gives something like:
['<｜hy_begin▁of▁sentence｜><｜hy_place▁holder▁no▁3｜><｜hy_place▁holder▁no▁100｜><｜hy_place▁holder▁no▁102｜><｜hy_place▁holder▁no▁101｜>Detect and recognize text in the image, and output the text coordinates in a formatted manner.<｜hy_User｜>']
What is the purpose of "<｜hy_place▁holder▁no▁3｜>" ?
it is not documented anywhere.
Also we would expect "<｜hy_User｜>" before the prompt (eg just after BOS), so why is it at the end of the prompt where we would expect the assistant token?
Now for the output, you use skip_special_tokens=False, which eliminates a bunch of special tokens:
<｜hy_place▁holder▁no▁110｜>
<｜hy_place▁holder▁no▁111｜>
<｜hy_place▁holder▁no▁112｜>
<｜hy_place▁holder▁no▁113｜>
I don't see anything in the doc or code to explain their meaning.
Thanks for the feedback.
Thanks for the feedback. You may find it in the chat template that "<｜hy_place▁holder▁no▁3｜>" serves as the separator between system prompt and user prompt. In this sepcific model we put system prompt as blank. And "100", "101", "102" serve as image start, end, image itself repectively.
For the "meaningless" special tokens, we are considering putting the actual meaning in the tokenizer in the next update. They might seem useless in HunyuanOCR, but useful in other Hunyuan Series models.
br
ALSO:
is it on purpose that in the chattemplate (in tokenizer_config.json):
"<｜hy_User｜>" comes AFTER the content of the user
eos_token is "<｜hy_Assistant｜>"
Pad token is supposed to be "<｜hy_▁pad▁｜>" = 120002
but there is also 120817 (-1)
finally I am seeing nowhere im_newline_id , is it actually used ?
Obviously, looking at tokenizer_config.json, this "<｜hy_Assistant｜>" token seems to be used as eos_token as well. If this is the case this needs to be part of the config.json so that th emodel understand both the regular eos_token and this one as eos_tokens.