Description

tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.

Executive Summary

Technical Details

Mitigations

References

Additional Metadata

{
  "score": "6.5",
  "pubDate": "2026-09-04T15:17:44.993Z",
  "pubdate": "2026-09-04T15:17:44.993Z",
  "executiveSummary": "The Hugging Face tokenizers library is susceptible to a critical denial-of-service (DoS) vulnerability originating from an out-of-bounds buffer access within the BpeBuilder::build function.\nThis vulnerability is triggered during the deserialization of a tokenizer.json file, allowing an unauthenticated attacker to supply a maliciously crafted configuration file that forces the application to abort.\nThe flaw stems from improper buffer sizing logic when processing Byte Pair Encoding (BPE) merge rules, which fails to account for cases where concatenated token lengths exceed the expected vocabulary maximum.\nAdditionally, a secondary vulnerability involving usize underflow exists when the 'continuing_subword_prefix' is utilized, which can lead to memory corruption in release builds or panics in debug builds.\nBecause the vulnerability is triggered during the tokenizer initialization process (Tokenizer::from_file or Tokenizer::from_str), no actual encoding operations are required for exploitation, making it a low-complexity attack vector.\nThe impact is significant for services that accept user-supplied tokenizer configurations, as the resulting panic leads to an immediate process abort, effectively denying service to legitimate users.",
  "technicalDetails": "The root cause of the vulnerability resides in the BpeBuilder::build function within tokenizers/src/models/bpe/model.rs. During the BPE model construction, the builder allocates a scratch buffer based on the length of the longest vocabulary key. Subsequently, the system attempts to write concatenated merge rules into this pre-allocated buffer.\nThe logic fails to validate whether the concatenated merge rule exceeds the dimensions of the buffer. When a crafted tokenizer.json contains merge rules resulting in a concatenated string longer than the longest vocabulary key, an out-of-bounds write occurs. In Rust, this access violation triggers a panic, which, in the context of FFI (Foreign Function Interface) embeddings, leads to a process-wide abort.\nThe attack flow follows a straightforward progression: 1) The attacker crafts a malicious tokenizer.json file containing specially designed BPE merge rules. 2) The target application invokes Tokenizer::from_file or Tokenizer::from_str to process this file. 3) The BpeBuilder::build function initializes a buffer based on the expected maximum key length. 4) During the iteration over merge rules, the concatenation operation causes the buffer write index to exceed the allocated boundary. 5) The Rust runtime encounters a bounds-check failure and triggers a panic, causing the entire process to exit abruptly.\nA secondary, highly critical defect involves the 'continuing_subword_prefix' parameter. When this prefix is active, the builder performs arithmetic on token lengths. If a merge token is shorter than the defined prefix, a usize underflow occurs during the length calculation. In debug builds, this results in a panic; however, in release builds, this underflow can lead to wrap-around memory access, creating potential for memory corruption or arbitrary memory access patterns.\nThe vulnerability affects version 0.23.1 and does not require elevated privileges or authentication. It is exposed to any application segment that parses untrusted tokenizer configuration files, presenting a high risk to systems utilizing Hugging Face tokenizers as part of a pipeline that processes externally sourced model configurations."
}