Skip to content

TokenizerFast is not available. #81

Description

@DieVerwandlungTK

Thank you for your great work !
I tried to convert DreamTokenizer into fast one but I couldn't because these errors occured.

Traceback (most recent call last):
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1867, in convert_slow_tokenizer
).converted()
^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1679, in converted
tokenizer = self.tokenizer()
^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1672, in tokenizer
vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1648, in extract_vocab_merges_from_model
bpe_ranks = load_tiktoken_bpe(tiktoken_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/tiktoken/load.py", line 171, in load_tiktoken_bpe
raise ValueError(f"Error parsing line {line!r} in {tiktoken_bpe_file}") from e
ValueError: Error parsing line b'{' in /app/dream_v0_instruct_7b_slow/vocab.json

Traceback (most recent call last):
File "/app/debug3.py", line 18, in
tokenizer_fast = convert_slow_tokenizer(tokenizer_slow)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1869, in convert_slow_tokenizer
raise ValueError(
ValueError: Converting from SentencePiece and Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast converters: ['AlbertTokenizer', ...]

Does anyone know solution for this error ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions