github - ghsa-mvj3-qrqh-cjvr

GHSA-MVJ3-QRQH-CJVR

Vulnerability from github – Published: 2023-07-05 21:33 – Updated: 2023-07-18 19:15

Summary

CometBFT PeerState JSON serialization deadlock

Details

Impact

An internal modification to the way struct PeerState is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places:

Via logs
- Setting the consensus logging module to "debug" level (should not happen in production), and
- Setting the log output format to JSON
Via RPC dump_consensus_state

Case 1 above, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.

In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.

The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one PeerState struct. If dump_consensus_state is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different PeerState structs) and the chain will progress again. Then, the same process can be repeated.

As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with 2 nodes decreases. Only the first PeerState struct will deadlock, and not the others (RPC dump_consensus_state accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).

This regression was introduced in versions v0.34.28 and v0.37.1, and will be fixed in v0.34.29 and v0.37.2.

Patches

The PR containing the fix is here, and the corresponding issue is here

Workarounds

For case 1 (hitting the deadlock via logs) * either don't set the log output to "json", leave at "plain", * or don't set the consensus logging module to "debug", leave it at "info" or higher.

For case 2 (hitting the deadlock via RPC dump_consensus_state) * do not expose dump_consensus_state RPC endpoint to the public internet (e.g., via rules in your nginx setup)

References

Issue that introduced the deadlock
Issue reporting the bug via logs

Severity ?

5.3 (Medium)


                  
                    CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Show details on source website

JSON

To clipboard

{
  "affected": [
    {
      "package": {
        "ecosystem": "Go",
        "name": "github.com/cometbft/cometbft"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0.34.28"
            },
            {
              "fixed": "0.34.29"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ],
      "versions": [
        "0.34.28"
      ]
    },
    {
      "package": {
        "ecosystem": "Go",
        "name": "github.com/cometbft/cometbft"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "0.37.1"
            },
            {
              "fixed": "0.37.2"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ],
      "versions": [
        "0.37.1"
      ]
    }
  ],
  "aliases": [
    "CVE-2023-34450"
  ],
  "database_specific": {
    "cwe_ids": [
      "CWE-401",
      "CWE-770"
    ],
    "github_reviewed": true,
    "github_reviewed_at": "2023-07-05T21:33:48Z",
    "nvd_published_at": "2023-07-03T17:15:09Z",
    "severity": "MODERATE"
  },
  "details": "### Impact\nAn internal modification to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places:\n\n1. Via logs\n    * Setting the `consensus` logging module to \"debug\" level (should not happen in production), and\n    * Setting the log output format to JSON\n2. Via RPC `dump_consensus_state` \n\nCase 1 above, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.\n\nIn case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node\u0027s peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.\n\nThe theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.\n\nAs the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with 2 nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that \"for\" loop to never be reached).\n\nThis regression was introduced in versions `v0.34.28` and `v0.37.1`, and will be fixed in `v0.34.29` and `v0.37.2`.\n\n### Patches\nThe PR containing the fix is [here](https://github.com/cometbft/cometbft/pull/865), and the corresponding issue is [here](https://github.com/cometbft/cometbft/pull/863)\n\n### Workarounds\nFor case 1 (hitting the deadlock via logs)\n* either don\u0027t set the log output to \"json\", leave at \"plain\",\n* or don\u0027t set the consensus logging module to \"debug\", leave it at \"info\" or higher.\n\nFor case 2 (hitting the deadlock via RPC `dump_consensus_state`)\n* do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in your nginx setup)\n\n### References\n\n* [Issue](https://github.com/cometbft/cometbft/pull/863) that introduced the deadlock\n* [Issue](https://github.com/cometbft/cometbft/pull/524) reporting the bug via logs\n",
  "id": "GHSA-mvj3-qrqh-cjvr",
  "modified": "2023-07-18T19:15:58Z",
  "published": "2023-07-05T21:33:48Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr"
    },
    {
      "type": "ADVISORY",
      "url": "https://nvd.nist.gov/vuln/detail/CVE-2023-34450"
    },
    {
      "type": "WEB",
      "url": "https://github.com/cometbft/cometbft/pull/524"
    },
    {
      "type": "WEB",
      "url": "https://github.com/cometbft/cometbft/pull/863"
    },
    {
      "type": "WEB",
      "url": "https://github.com/cometbft/cometbft/pull/865"
    },
    {
      "type": "PACKAGE",
      "url": "https://github.com/cometbft/cometbft"
    }
  ],
  "schema_version": "1.4.0",
  "severity": [
    {
      "score": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L",
      "type": "CVSS_V3"
    }
  ],
  "summary": "CometBFT PeerState JSON serialization deadlock"
}

CVE-2023-34450 (GCVE-0-2023-34450)

Vulnerability from cvelistv5 – Published: 2023-07-03 16:36 – Updated: 2024-10-29 13:49

Title

CometBFT PeerState JSON serialization deadlock

Summary

CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the `consensus` logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPC `dump_consensus_state`. Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node. In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded. The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated. As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached). This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC `dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in one's nginx setup).

Severity ?

3.7 (Low)


                        
                          CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:L

CWE

CWE-401 - Missing Release of Memory after Effective Lifetime
CWE-770 - Allocation of Resources Without Limits or Throttling

Assigner

GitHub_M

References

URL

Tags

	https://github.com/cometbft/cometbft/security/adv…	x_refsource_CONFIRM
	https://github.com/cometbft/cometbft/pull/524	x_refsource_MISC
	https://github.com/cometbft/cometbft/pull/863	x_refsource_MISC
	https://github.com/cometbft/cometbft/pull/865	x_refsource_MISC

Impacted products

	Vendor	Product	Version
	cometbft	cometbft	Affected: = 0.34.28 Affected: = 0.37.1

Show details on NVD website

JSON

To clipboard

{
  "containers": {
    "adp": [
      {
        "providerMetadata": {
          "dateUpdated": "2024-08-02T16:10:06.972Z",
          "orgId": "af854a3a-2127-422b-91ae-364da2661108",
          "shortName": "CVE"
        },
        "references": [
          {
            "name": "https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr",
            "tags": [
              "x_refsource_CONFIRM",
              "x_transferred"
            ],
            "url": "https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr"
          },
          {
            "name": "https://github.com/cometbft/cometbft/pull/524",
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/cometbft/cometbft/pull/524"
          },
          {
            "name": "https://github.com/cometbft/cometbft/pull/863",
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/cometbft/cometbft/pull/863"
          },
          {
            "name": "https://github.com/cometbft/cometbft/pull/865",
            "tags": [
              "x_refsource_MISC",
              "x_transferred"
            ],
            "url": "https://github.com/cometbft/cometbft/pull/865"
          }
        ],
        "title": "CVE Program Container"
      },
      {
        "metrics": [
          {
            "other": {
              "content": {
                "id": "CVE-2023-34450",
                "options": [
                  {
                    "Exploitation": "none"
                  },
                  {
                    "Automatable": "no"
                  },
                  {
                    "Technical Impact": "partial"
                  }
                ],
                "role": "CISA Coordinator",
                "timestamp": "2024-10-28T19:22:20.977405Z",
                "version": "2.0.3"
              },
              "type": "ssvc"
            }
          }
        ],
        "providerMetadata": {
          "dateUpdated": "2024-10-29T13:49:01.399Z",
          "orgId": "134c704f-9b21-4f2e-91b3-4a467353bcc0",
          "shortName": "CISA-ADP"
        },
        "title": "CISA ADP Vulnrichment"
      }
    ],
    "cna": {
      "affected": [
        {
          "product": "cometbft",
          "vendor": "cometbft",
          "versions": [
            {
              "status": "affected",
              "version": "= 0.34.28"
            },
            {
              "status": "affected",
              "version": "= 0.37.1"
            }
          ]
        }
      ],
      "descriptions": [
        {
          "lang": "en",
          "value": "CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the `consensus` logging module to \"debug\" level (should not happen in production), and setting the log output format to JSON. The second is via RPC `dump_consensus_state`.\n\nCase 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.\n\nIn case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node\u0027s peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.\n\nThe theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.\n\nAs the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that \"for\" loop to never be reached).\n\nThis regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don\u0027t set the log output to \"json\", leave at \"plain\", or don\u0027t set the consensus logging module to \"debug\", leave it at \"info\" or higher. For case 2 (hitting the deadlock via RPC `dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in one\u0027s nginx setup)."
        }
      ],
      "metrics": [
        {
          "cvssV3_1": {
            "attackComplexity": "HIGH",
            "attackVector": "NETWORK",
            "availabilityImpact": "LOW",
            "baseScore": 3.7,
            "baseSeverity": "LOW",
            "confidentialityImpact": "NONE",
            "integrityImpact": "NONE",
            "privilegesRequired": "NONE",
            "scope": "UNCHANGED",
            "userInteraction": "NONE",
            "vectorString": "CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:L",
            "version": "3.1"
          }
        }
      ],
      "problemTypes": [
        {
          "descriptions": [
            {
              "cweId": "CWE-401",
              "description": "CWE-401: Missing Release of Memory after Effective Lifetime",
              "lang": "en",
              "type": "CWE"
            }
          ]
        },
        {
          "descriptions": [
            {
              "cweId": "CWE-770",
              "description": "CWE-770: Allocation of Resources Without Limits or Throttling",
              "lang": "en",
              "type": "CWE"
            }
          ]
        }
      ],
      "providerMetadata": {
        "dateUpdated": "2023-07-03T16:36:34.419Z",
        "orgId": "a0819718-46f1-4df5-94e2-005712e83aaa",
        "shortName": "GitHub_M"
      },
      "references": [
        {
          "name": "https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr",
          "tags": [
            "x_refsource_CONFIRM"
          ],
          "url": "https://github.com/cometbft/cometbft/security/advisories/GHSA-mvj3-qrqh-cjvr"
        },
        {
          "name": "https://github.com/cometbft/cometbft/pull/524",
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/cometbft/cometbft/pull/524"
        },
        {
          "name": "https://github.com/cometbft/cometbft/pull/863",
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/cometbft/cometbft/pull/863"
        },
        {
          "name": "https://github.com/cometbft/cometbft/pull/865",
          "tags": [
            "x_refsource_MISC"
          ],
          "url": "https://github.com/cometbft/cometbft/pull/865"
        }
      ],
      "source": {
        "advisory": "GHSA-mvj3-qrqh-cjvr",
        "discovery": "UNKNOWN"
      },
      "title": "CometBFT PeerState JSON serialization deadlock"
    }
  },
  "cveMetadata": {
    "assignerOrgId": "a0819718-46f1-4df5-94e2-005712e83aaa",
    "assignerShortName": "GitHub_M",
    "cveId": "CVE-2023-34450",
    "datePublished": "2023-07-03T16:36:34.419Z",
    "dateReserved": "2023-06-06T16:16:53.558Z",
    "dateUpdated": "2024-10-29T13:49:01.399Z",
    "state": "PUBLISHED"
  },
  "dataType": "CVE_RECORD",
  "dataVersion": "5.1"
}

Sightings

Author	Source	Type	Date

Nomenclature

Seen: The vulnerability was mentioned, discussed, or observed by the user.
Confirmed: The vulnerability has been validated from an analyst's perspective.
Published Proof of Concept: A public proof of concept is available for this vulnerability.
Exploited: The vulnerability was observed as exploited by the user who reported the sighting.
Patched: The vulnerability was observed as successfully patched by the user who reported the sighting.
Not exploited: The vulnerability was not observed as exploited by the user who reported the sighting.
Not confirmed: The user expressed doubt about the validity of the vulnerability.
Not patched: The vulnerability was not observed as successfully patched by the user who reported the sighting.

Detection rules are retrieved from Rulezet.

Action not permitted

GHSA-MVJ3-QRQH-CJVR

Impact

Patches

Workarounds

References

CVE-2023-34450 (GCVE-0-2023-34450)

Tags

Sightings

Nomenclature