GCVE Workshop - 22 September 2026 (14:00-18:00), Luxembourg Before The Vulnopticon Conference - Registration

PYSEC-2026-3906

Vulnerability from pysec - Published: 2026-09-10 09:44 - Updated: 2026-09-10 11:02
VLAI
Details

Problem description

Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego._urlpattern._URLPattern._prepare_pattern_for_regex(). Every * in the directive value is translated into a lazy .*? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.

Impact

Parsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.can_fetch() results in the latter call not returning for a period dependent on the length of the URL.

Proof of concept

from protego import Protego

robotstxt = f"""
User-agent: *
Disallow: /{"*1" * 12}*Z
"""
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot")  # freezes
Severity
7.1 (High)
CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Impacted products
Name purl
protego pkg:pypi/protego
Aliases
To clipboard 

{
  "affected": [
    {
      "package": {
        "ecosystem": "PyPI",
        "name": "protego",
        "purl": "pkg:pypi/protego"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0"
            },
            {
              "fixed": "0.6.2"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ],
      "versions": [
        "0.1",
        "0.1.12",
        "0.1.14",
        "0.1.15",
        "0.1.16",
        "0.1.dev0",
        "0.2.0",
        "0.2.1",
        "0.3.0",
        "0.3.1",
        "0.4.0",
        "0.5.0",
        "0.6.0",
        "0.6.1"
      ]
    }
  ],
  "aliases": [
    "CVE-2026-55520",
    "GHSA-wjmf-p669-5m5p"
  ],
  "details": "### Problem description\n\nProtego constructs regular expressions to match URLs against `robots.txt` `Allow:` and `Disallow:` directives, see `protego._urlpattern._URLPattern._prepare_pattern_for_regex()`. Every `*` in the directive value is translated into a lazy `.*?` regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.\n\n### Impact\n\nParsing a specially crafted `robots.txt` with `protego.Protego.parse()` and then trying to match an URL with `protego.Protego.can_fetch()` results in the latter call not returning for a period dependent on the length of the URL.\n\n### Proof of concept\n\n```python\nfrom protego import Protego\n\nrobotstxt = f\"\"\"\nUser-agent: *\nDisallow: /{\"*1\" * 12}*Z\n\"\"\"\nrp = Protego.parse(robotstxt)\nurl = \"/\" + \"1\" * 60\nrp.can_fetch(url, \"mybot\")  # freezes\n```",
  "id": "PYSEC-2026-3906",
  "modified": "2026-09-10T11:02:29.238962Z",
  "published": "2026-09-10T09:44:58.501183Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/security/advisories/GHSA-wjmf-p669-5m5p"
    },
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/commit/785940181659bf440ba82f1da148fade5087e858"
    },
    {
      "type": "PACKAGE",
      "url": "https://github.com/scrapy/protego"
    },
    {
      "type": "WEB",
      "url": "https://github.com/scrapy/protego/releases/tag/0.6.2"
    },
    {
      "type": "PACKAGE",
      "url": "https://pypi.org/project/protego"
    },
    {
      "type": "ADVISORY",
      "url": "https://github.com/advisories/GHSA-wjmf-p669-5m5p"
    },
    {
      "type": "ADVISORY",
      "url": "https://nvd.nist.gov/vuln/detail/CVE-2026-55520"
    }
  ],
  "severity": [
    {
      "score": "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N",
      "type": "CVSS_V4"
    }
  ],
  "summary": "Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching"
}



Log in or create an account to share your comment.




Tags
Taxonomy of the tags.





Forecast uses a logistic model when the trend is rising, or an exponential decay model when the trend is falling. Fitted via linearized least squares.

Sightings

Author Source Type Date Other

Nomenclature

  • Seen: The vulnerability was mentioned, discussed, or observed by the user.
  • Confirmed: The vulnerability has been validated from an analyst's perspective.
  • Published Proof of Concept: A public proof of concept is available for this vulnerability.
  • Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
  • Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
  • Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
  • Not confirmed: The user expressed doubt about the validity of the vulnerability.
  • Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.


Detection rules are retrieved from Rulezet.