Affected: Maven: io.netty:netty-codec-http3

### Summary Netty's Http3FrameCodec buffers incoming data for HTTP/3 reserved frame types up to the specified payload length without any limits. The payload length is read directly from the wire and trusted without validation. A bad actor can send a reserved frame with a payload length of up to Integer.MAX_VALUE, causing the server to buffer the data in memory. This leads to an OOM and a gradual Denial of Service due to memory exhaustion as multiple streams are opened.

### Details `io.netty.handler.codec.http3.Http3FrameCodec#decodeFrame` handles reserved frame types as follows:

```java // Handling reserved frame types // https://tools.ietf.org/html/draft-ietf-quic-http-32#section-7.2.8 if (in.readableBytes() < payLoadLength) { return 0; } ```

The `payLoadLength` is read directly from the wire and trusted implicitly. Since `payLoadLength` can be up to Integer.MAX_VALUE and there is no maximum payload length enforcement for reserved frames, the decoder will accumulate bytes in memory until the wire-provided length is reached.

This allows a bad actor to exhaust server memory by opening multiple QUIC streams and sending reserved frames with large payload lengths, followed by a small amount of data (e.g., up to the defined limit) on each stream.

### PoC

```java @Test public void test() throws Exception { EventLoopGroup group = new MultiThreadIoEventLoopGroup(1, NioIoHandler.…