GCVE Workshop - 22 September 2026 (14:00-18:00), Luxembourg Before The Vulnopticon Conference - Registration

AVID-2023-V026

Vulnerability from avid – Published: 2023-03-31 – Updated: 2023-03-31 LLM Evaluation
Summary
When prompting ChatGPT with lexical constraints, e.g. "Generate a text without the letter "e" in it", ChatGPT almost always fails to follow these constraints.
Risk domain
Performance
SEP view
P0204: Accuracy
Lifecycle
L02: Data Understanding, L04: Model Development, L05: Evaluation, L06: Deployment
Organisations
OpenAI (deployer), OpenAI (developer)
Affected artifacts
Artifact Type
ChatGPT System
References
URL Label
https://www.gwern.net/GPT-3#bpes Gwern's analysis of lexical constraints and ChatGPT
https://paperswithcode.com/paper/most-language-mo… Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio
To clipboard 

{
  "affects": {
    "artifacts": [
      {
        "name": "ChatGPT",
        "type": "System"
      }
    ],
    "deployer": [
      "OpenAI"
    ],
    "developer": [
      "OpenAI"
    ]
  },
  "credit": [
    {
      "lang": "eng",
      "value": "Allen Roush, Oracle Corporation"
    }
  ],
  "data_type": "AVID",
  "data_version": "0.2",
  "description": {
    "lang": "eng",
    "value": "When prompting ChatGPT with lexical constraints, e.g. \"Generate a text without the letter \"e\" in it\", ChatGPT almost always fails to follow these constraints. "
  },
  "impact": {
    "avid": {
      "lifecycle_view": [
        "L02: Data Understanding",
        "L04: Model Development",
        "L05: Evaluation",
        "L06: Deployment"
      ],
      "risk_domain": [
        "Performance"
      ],
      "sep_view": [
        "P0204: Accuracy"
      ],
      "taxonomy_version": "0.2"
    }
  },
  "last_modified_date": "2023-03-31",
  "metadata": {
    "vuln_id": "AVID-2023-V026"
  },
  "problemtype": {
    "classof": "LLM Evaluation",
    "description": {
      "lang": "eng",
      "value": "ChatGPT fails to follow lexical constraints"
    },
    "type": "Advisory"
  },
  "published_date": "2023-03-31",
  "references": [
    {
      "label": "Gwern\u0027s analysis of lexical constraints and ChatGPT",
      "type": "source",
      "url": "https://www.gwern.net/GPT-3#bpes"
    },
    {
      "label": "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio",
      "type": "source",
      "url": "https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai"
    }
  ],
  "reports": [
    {
      "name": "ChatGPT fails to follow lexical constraints",
      "report_id": "AVID-2023-R0002",
      "type": "Advisory"
    }
  ]
}



Log in or create an account to share your comment.




Tags
Taxonomy of the tags.





Forecast uses a logistic model when the trend is rising, or an exponential decay model when the trend is falling. Fitted via linearized least squares.

Sightings

Author Source Type Date Other

Nomenclature

  • Seen: The vulnerability was mentioned, discussed, or observed by the user.
  • Confirmed: The vulnerability has been validated from an analyst's perspective.
  • Published Proof of Concept: A public proof of concept is available for this vulnerability.
  • Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
  • Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
  • Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
  • Not confirmed: The user expressed doubt about the validity of the vulnerability.
  • Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.


Detection rules are retrieved from Rulezet.


Related by attack behaviour

Vulnerabilities whose description is nearest to this one in the vector space of the CIRCL/vulnerability-attack-technique-biencoder model. This is a similarity search over the bi-encoder space (plain cosine), not a classification, and it has no measured accuracy.