Skip to content

Commit fa1bddd

Browse files
authored
fix: start of word tokens for added (#351)
1 parent 4fb4af1 commit fa1bddd

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

model2vec/tokenizer/tokenizer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ def clean_and_create_vocabulary(
4646
added_tokens_to_add: list[str] = []
4747
seen_added = set()
4848
for token in vocabulary_to_add:
49-
preprocessed = preprocessor.preprocess(token)
49+
preprocessed = preprocessor.preprocess(token, had_initial_subword_prefix=True)
5050
if len(preprocessed) < 1:
5151
logger.warning(f"Token '{token}' was empty after preprocessing.")
5252
n_empty += 1

0 commit comments

Comments
 (0)