repository icon
huggingface / tokenizers Project repository

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Submit a report

FIRST INTERACTION

WITHINN/A DAYS

REVIEW

WITHIN45 DAYS

FIX

WITHINN/A DAYS

Denial of Service in tokenizers BPE model deserialization via unchecked length a...
hocinek
self closed
TemplateProcessor todo!() Panic — Unhandled Encoding Count in process_encodings...
harrythebountyhunter2026-max
duplicate
Medium
BPE deserializer: from_utf8_unchecked on possibly-invalid UTF-8 (UB) + usize und...
nahldi
self closed
Multiple unauthenticated DoS panics in huggingface/tokenizers via crafted tokeni...
lewiswigmore
self closed
Multiple unauthenticated DoS panics in huggingface/tokenizers via crafted tokeni...
lewiswigmore
informative
High
ReDoS via user-supplied regex in Split pre-tokenizer and Replace normalizer (Oni...
allen930311
informative
Medium
Malformed BPE continuing_subword_prefix Can Panic or Abort During Tokenizer Load
surrealgrain
informative
Medium
Reachable panics in Strip decoder via crafted tokenizer.json (decoders/strip.rs)
cotu23
informative
Medium
`BpeBuilder::build()` soundness bug: `from_utf8_unchecked` on invalid UTF-8 caus...
11x-x11
duplicate
High
ReDoS via user-supplied regex in pre_tokenizers Split / normalizers Replace — ma...
lihfdgjr
informative
Medium
BPE builder: attacker-controlled continuing_subword_prefix wraps b.len() - prefi...
hohk0923
duplicate
Medium
High — CWE-400: Algorithmic Resource Amplification DoS via Unbounded added_token...
objectverseyoutubechannel-sys
informative
Medium
Current-working-directory import hijack in convert.py leads to arbitrary code ex...
studentcodinguser
informative
High
Multiple reachable panics in tokenizer deserialization/encode/decode cause denia...
snakeyworm
duplicate
None
Multiple reachable panics in tokenizer deserialization/encode/decode cause denia...
snakeyworm
informative
None
Multiple reachable panics in tokenizer deserialization/encode/decode cause denia...
snakeyworm
informative
None
Crafted precompiled_charsmap in tokenizer.json Causes Out-of-Bounds Array Access...
pkmdev-sec
informative
Medium
BPE Model Deserialization Panics When continuing_subword_prefix Exceeds Merge To...
pkmdev-sec
informative
Medium
[Integer Overflow] Memory Safety Violation via Integer Underflow in BPE Merge Ma...
anlor1002-alt
informative
Medium
Panic-on-load DoS in `BPE::build` via malformed `tokenizer.json`
r0h1th2025
duplicate
Medium
Unsound `from_utf8_unchecked` on attacker-controlled bytes in `BPE::build`
r0h1th2025
duplicate
High
Non-atomic test fixture downloads can leave corrupted files and cause misleading...
adithyanjkr
self closed
Integer overflow/wraparound in BPE trainer pair counting can corrupt merge stati...
adithyanjkr
informative
Medium
ReDoS via attacker-controlled regex pattern in tokenizer.json causes CPU exhaust...
snakeyworm
informative
Medium
Stack Overflow via Recursive Trie Drop in Unigram Model
remiboudreau
self closed
Stored XSS via Unescaped Annotation Label in EncodingVisualizer HTML Output
javitoia
pending
ReDoS via attacker-controlled regex in Split/Replace normalizers deserialized fr...
elliottower
informative
Medium
ReDoS via Malicious Regex in Tokenizer JSON Pre-tokenizer/Normalizer
lihfdgjr
informative
Medium
Unvalidated Regex in Split/Replace Causes Uncatchable PanicException via ReDoS
marwan9696
informative
High
ReDoS via Unvalidated Regex Patterns in Split Pre-tokenizer and Replace Normaliz...
davidaroca27
informative
High
Truncation Stride Assert Panic Crashes Python Process via Crafted tokenizer.json
ryanhugh
duplicate
Medium
Oniguruma ReDoS Panic via Malicious tokenizer.json
ryanhugh
duplicate
Medium
Precompiled Normalizer Out-of-Bounds Panics via Crafted tokenizer.json
ryanhugh
duplicate
Medium
Incomplete Fix: Path Traversal in Model.save() Allows Arbitrary File Overwrite v...
ibfavas
duplicate
High
Arbitrary Code Execution via Pickle Deserialization of Tokenizer Objects
caoxuyang
informative
High
Denial of Service via ReDoS Panic in tokenizer.json Split/Replace Regex Patterns
caoxuyang
duplicate
Medium
ReDoS via Unprotected Regex Compilation in SplitPattern::Regex Deserialization (...
nhomyk
informative
Medium
Exponential memory amplification via chained Replace normalizers in tokenizer.js...
k-rintaro
duplicate
High
Unrecoverable process crash via malicious regex in Split pre-tokenizer loaded fr...
kodareef5
informative
Medium
Silent u64-to-u32 Truncation in BPE Vocab Loading Causes Token ID Collision
javitoia
informative
Medium
Integer Overflow in NumPy Array Buffer Extraction via Unchecked from_raw_parts
javitoia
informative
Medium
Unsound unsafe impl Send+Sync on RefMutContainer with Raw Pointer — Use-After-Fr...
javitoia
pending
ReDoS via Malicious Regex Pattern in tokenizer.json Configuration (Split/Replace...
javitoia
duplicate
Medium
Denial of Service via Rust Panic in Precompiled Normalizer — .expect() crashes p...
mscgo
self closed
DoS via Unhandled Regex Panic in Split Pre-Tokenizer / Replace Normalizer
snailsploit
informative
High
Truncation Bypass via Integer Underflow in Post-Processor Token Calculation
w0termelon
informative
None
Stride Assertion Panic on Pair Encoding with LongestFirst Truncation
w0termelon
duplicate
None
Normalizer Chain Exponential Expansion Causes OOM Crash
w0termelon
informative
None
BPE Model Crashes on Load with Oversized Prefix
w0termelon
informative
None
Strip Decoder Integer Underflow Crashes Process on Decode
w0termelon
informative
None
Reachable todo!() Panic in Template Processing with Chained Post-Processors
w0termelon
informative
None
Template Post-Processor Validation Bypass Causes Multiple Crashes
w0termelon
informative
None
Process Crash from FixedLength Pre-Tokenizer with length=0
w0termelon
informative
None
Tokenizer Crash on Encode Due to Invalid Truncation Configuration
w0termelon
duplicate
None
ReDoS via malicious regex patterns in tokenizer.json allows denial of service on...
eistee82
informative
High
ReDoS Pattern in Tokenizer Config Crashes Process
w0termelon
duplicate
Medium
Denial of Service via Long Added Token in Tokenizer Configuration
w0termelon
duplicate
None
Uncatchable Crash When Loading Tokenizer with Invalid Precompiled Normalizer
w0termelon
duplicate
None
Process Crash from Malformed Precompiled Normalizer Data
w0termelon
duplicate
High
Reachable Panic in Precompiled Normalizer Crashes Python via Uncatchable PanicEx...
tulgaaaaaaaa
duplicate
Medium
Symlink following in save() allows arbitrary file overwrite
w0termelon
informative
Medium
Vocab file line injection via newlines in token names (WordPiece save)
w0termelon
informative
Medium
Process crash from uncatchable panic when stride equals max_length
w0termelon
informative
High
OOM crash from unbounded padding.Fixed value in tokenizer config
w0termelon
informative
Medium
Process crash from malicious regex in tokenizer.json (uncatchable panic)
w0termelon
informative
Critical
Path Traversal in save_model() prefix parameter
w0termelon
informative
Medium
Process Panic on Loading Malformed Precompiled Normalizer (Commented-Out Error H...
jdhart81
duplicate
None
Process Panic via Attacker-Controlled Regex in Normalizer (Oniguruma Retry-Limit...
jdhart81
informative
High
Stored XSS via Malicious Special Token Names in EncodingVisualizer
s3zer0
pending
Workflow_run artifact poisoning in upload_pr_documentation.yml deploys attacker...
vera-platform
pending
Byte/Char Length Confusion in NormalizedString::lrstrip Causes Panic on Multi-by...
wernerina
informative
High
Integer Overflow in BPE Trainer Pair Count via u64→i32 Truncation
wernerina
informative
High
OS Command Injection in HuggingFace Tokenizers Python Bindings
aybanda
self closed
Arbitrary File Write via Path Traversal in HuggingFace Tokenizers Python Binding...
aybanda
informative
Critical
ReDoS via User-Controlled Regex in `tokenizer.json` Causes Process-Killing Panic...
dnyaneshmoh
duplicate
High
HuggingFace tokenizers Library Unsafe Regular Expression Handling Leads to Regul...
mia-718ai
informative
Medium
TemplateProcessing::process_encodings Incomplete Pattern Matching Leads to Panic...
mia-718ai
informative
Medium
HuggingFace tokenizers Encoding.pad() Memory Exhaustion Denial of Service
mia-718ai
informative
High
Integer Overflow in PyArrayUnicode::extract_bound Leading to Out-of-bounds Read...
mia-718ai
informative
High
HuggingFace tokenizers PyArrayUnicode NULL pointer dereference leading to Denial...
mia-718ai
informative
Medium
Deterministic Denial of Service via poisoned RwLock unwrap in Node.js bindings
eljout
informative
High
Remote Denial of Service (DoS) in UnigramTrainer via Unchecked Panic on Large Da...
ub1cu0
informative
Medium
Denial of Service (DoS) via Excessive Memory Consumption with Long Text Input
stevenjulian1528
informative
High
Out-of-Bounds Access in NormalizedString::transform_range Leading to Memory Safe...
hyperps1
informative
High
Insecure deserialization and arbitrary file creation leading to memory exhaustio...
sonw-vh
informative
Critical
Critical Panic-based Denial of Service via Serialization Double Unwrap Pattern
gokul-labb
informative
Critical
Path Traversal vulnerability on Unigram Model
h3c-t0r
informative
High
SSRF in extractor.py
hohky
informative
High
Path Traversal in extractor.py
hohky
informative
Medium
huntr: huggingface/tokenizers