Thank you for your great work !
I tried to convert DreamTokenizer into fast one but I couldn't because these errors occured.
Traceback (most recent call last):
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1867, in convert_slow_tokenizer
).converted()
^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1679, in converted
tokenizer = self.tokenizer()
^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1672, in tokenizer
vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1648, in extract_vocab_merges_from_model
bpe_ranks = load_tiktoken_bpe(tiktoken_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tiktoken/load.py", line 171, in load_tiktoken_bpe
raise ValueError(f"Error parsing line {line!r} in {tiktoken_bpe_file}") from e
ValueError: Error parsing line b'{' in /app/dream_v0_instruct_7b_slow/vocab.json
Traceback (most recent call last):
File "/app/debug3.py", line 18, in
tokenizer_fast = convert_slow_tokenizer(tokenizer_slow)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1869, in convert_slow_tokenizer
raise ValueError(
ValueError: Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: ['AlbertTokenizer', ...]
Does anyone know solution for this error ?
Thank you for your great work !
I tried to convert DreamTokenizer into fast one but I couldn't because these errors occured.
Traceback (most recent call last):
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1867, in convert_slow_tokenizer
).converted()
^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1679, in converted
tokenizer = self.tokenizer()
^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1672, in tokenizer
vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1648, in extract_vocab_merges_from_model
bpe_ranks = load_tiktoken_bpe(tiktoken_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tiktoken/load.py", line 171, in load_tiktoken_bpe
raise ValueError(f"Error parsing line {line!r} in {tiktoken_bpe_file}") from e
ValueError: Error parsing line b'{' in /app/dream_v0_instruct_7b_slow/vocab.json
Traceback (most recent call last):
File "/app/debug3.py", line 18, in
tokenizer_fast = convert_slow_tokenizer(tokenizer_slow)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1869, in convert_slow_tokenizer
raise ValueError(
ValueError: Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: ['AlbertTokenizer', ...]
Does anyone know solution for this error ?