Migrate ByteTokenizer to PyGrain

[Nikhilyammani007]

Description of the change

This PR migrates ByteTokenizer to support Keras 3 pure-Python execution, enabling compatibility with PyGrain and non-TensorFlow backends for data loading.

Key Implementation Details:

Standard PyGrain Routing: Followed the established pattern (e.g. from byte_pair_tokenizer.py) by adding _allow_python_workflow and in_tf_function() guards to gracefully route between _tokenize_tf and the new _tokenize_python fallbacks.

Text Normalization: Ensured the Python fallback respects normalization_form by utilizing Python's native unicodedata.normalize.

Padding & Truncation: Ensured the Python fallback correctly respects sequence_length, padding and truncating outputs identically to the TF backend.

Native Byte Processing: Replicated tf.strings.unicode_transcode natively. Added a fast loop using UnicodeDecodeError to properly support errors="replace", errors="strict", and custom replacement characters (e.g. U+FFFD).

Optimized Unpadding: Leveraged native C-level byte replacements (bytes().replace(b"\x00", b"")) inside Python detokenization for maximum speed.

Fixed Graph Tracing Issues: Moved static tensor instantiation (self._char_lst_tensor) to init to prevent memory leaks during tf.function compilation.

Metric/PyTree Compatibility: Used keras.tree.map_structure to safely iterate over nested array inputs (such as those generated during BleuTest evaluation).

Docstring Updates: Updated the doctests to correctly reflect standard Python list/NumPy array outputs (dropping the dtype=int32 suffix where eager evaluation occurs).

Reference

#2949

Colab Notebook

ByteTokenizer_Tested_Colab

Checklist

• I have added all the necessary unit tests for my change.
• I have verified that my change does not break existing code and works with all backends (TensorFlow, JAX, and PyTorch).
• My PR is based on the latest changes of the main branch (if unsure, rebase the code).
• I have followed the Keras Hub Model contribution guidelines in making these changes.
• I have followed the Keras Hub API design guidelines in making these changes.
• I have signed the Contributor License Agreement.

[gemini-code-assist]

Code Review

This pull request introduces a pure Python workflow for the ByteTokenizer class, enabling tokenization and detokenization without relying on TensorFlow. It also adds a custom decoding helper with replacement support and includes corresponding unit tests to verify workflow parity. The reviewer recommends caching the tf.constant created from self._char_lst in the TensorFlow detokenization path to prevent redundant tensor creation and potential graph bloat during tracing.

[gemini-code-assist]

Creating a tf.constant from self._char_lst on every call to _detokenize_tf is inefficient and can lead to graph bloat when executing in a tf.function or under tracing. Since self._char_lst is static, we should cache the created tf.Tensor on the instance so it is only created once.

        if not hasattr(self, "_char_lst_tensor"):
            self._char_lst_tensor = tf.constant(self._char_lst)
        _char_lst_tensor = self._char_lst_tensor