useTokenizer(
tokenizerPath,options?):object
Defined in: hooks/useTokenizer.ts:21
React hook to load and run a tokenizer.
This hook manages downloading (if remote URLs are provided) and loading the tokenizer configuration files, tracking download progress and load errors, and releasing native memory when the component unmounts or the configuration changes.
For imperative usage, see createTokenizer.
string
A remote URL or local path to a tokenizer.json file.
Load and caching options. See ResourceOptions.
object
The same object as nlp.Tokenizer (without dispose),
combined with loading state and download progress.
decode: (...
args) =>Promise<string> |undefined=model.decode
downloadProgress:
number
encode: (...
args) =>Promise<Int32Array<ArrayBufferLike>> |undefined=model.encode
error:
Error|undefined
getVocabSize: () =>
number|undefined=model.getVocabSize
() => number
Returns the total vocabulary size.
number
The total number of tokens in the vocabulary.
With code RESOURCE_BUSY if the tokenizer is
in use, or RESOURCE_DISPOSED if disposed.
undefined
idToToken: (
id) =>string|undefined=model.idToToken
(id) => string
Converts a numeric token id to its string piece representation.
number
The token id to look up.
string
The string piece for the given token id.
With code EXECUTION_FAILED if id lookup
fails, RESOURCE_BUSY if the tokenizer is in use, or RESOURCE_DISPOSED
if disposed.
undefined
isReady:
boolean=!!model
resource:
string|undefined
tokenToId: (
token) =>number|undefined=model.tokenToId
(token) => number
Converts a string piece token to its numeric token id.
string
The token piece string to look up.
number
The numeric id for the given token piece.
With code EXECUTION_FAILED if token lookup
fails, RESOURCE_BUSY if the tokenizer is in use, or RESOURCE_DISPOSED
if disposed.
undefined