Http-Desync-Guardian – Analyze HTTP Requests To Minimize Risks Of HTTP Desync Attacks

Http-Desync-Guardian is to Analyze HTTP Requests To Minimize Risks Of HTTP Desync Attacks.

HTTP/1.1 went through a long evolution since 1991 to 2014:

• HTTP/0.9 – 1991
• HTTP/1.0 – 1996
• HTTP/1.1
  • RFC 2068 – 1997
  • RFC 2616 – 1999
  • RFC 7230 – 2014

This means there is a variety of servers and clients, which might have different views on request boundaries, creating opportunities for desynchronization attacks (a.k.a. HTTP Desync).

It might seem simple to follow the latest RFC recommendations. However, for large scale systems that have been there for a while, it may come with unacceptable availability impact.

http_desync_guardian library is designed to analyze HTTP requests to prevent HTTP Desync attacks, balancing security and availability. It classifies requests into different categories and provides recommendations on how each tier should be handled.

It can be used either for raw HTTP request headers or already parsed by an HTTP engine. Consumers may configure logging and metrics collection. Logging is rate limited and all user data is obfuscated.

If you think you might have found a security impacting issue, please follow our Security Notification Process.

Priorities

• Uniformity across services is key. This means request classification, logging, and metrics must happen under the hood and with minimally available settings (e.g., such as log file destination).
• Focus on reviewability. The test suite must require no knowledge about the library/programming languages but only about HTTP protocol. So it’s easy to review, contribute, and re-use.
• Security is efficient when it’s easy for users. Our goal is to make integration of the library as simple as possible.
• Ultralight. The overhead must be minimal and impose no tangible tax on request handling (see benchmarks).

Supported HTTP versions

The main focus of this library is HTTP/1.1. See tests for all covered cases. Predecessors of HTTP/1.1 don’t support connection re-use which limits opportunities for HTTP Desync, however some proxies may upgrade such requests to HTTP/1.1 and re-use backend connections, which may allow to craft malicious HTTP/1.0 requests. That’s why they are analyzed using the same criteria as HTTP/1.1. For other protocol versions have the following exceptions:

• HTTP/0.9 requests are never considered Compliant, but are classified as Acceptable. If any of Content-Length/Transfer-Encoding is present then it’s Ambiguous.
• HTTP/1.0 – the presence of Transfer-Encoding makes a request Ambiguous.
• HTTP/2+ is out of scope. But if your proxy downgrades HTTP/2 to HTTP/1.1, make sure the outgoing request is analyzed.

See documentation to learn more.

Overview

This page contains request classification tiers and reasons as well as mitigations, with explanations for some non-trivial cases.

Request classification

http_desync_guardian is a library for analyzing and classifying HTTP/1.x requests to provide customers security balanced with necessity to serve traffic for legacy or proprietary systems (not always RFC compliant).

• Compliant – RFC compliant requests (*)
• Acceptable – non RFC compliant requests, but which do not represent security risks
• Ambiguous – requests that might be treated differently by different HTTP servers and therefore may lead to HTTP Desync issues (and request splitting/smuggling as a possible consequence)
• Severe – either malformed or highly likely crafted to trick HTTP parsers and cause HTTP de-synchronization.

Recommended http_desync_guardian Modes

Classification	Defensive mode	Strictest mode
Compliant	Allowed	Allowed
Acceptable	Allowed	Blocked
Ambiguous	Allowed¹	Blocked
Severe	Blocked	Blocked

¹ Route the requests but closes the client and target connections.

For Blocked requests the client connection must be closed.

If you are concerned about potential impact, Monitoring mode offers a metrics-only approach to assess prior to switching.

Classification Reasons

• Compliant
  • Compliant – a compliant request
• Acceptable
  • NonCompliantHeader – non-essential header containing a non-ASCII or control characters (CTL) – i.e. special invisible characters.
  • SpaceInUri – unescaped space in the URI
  • NonCompliantVersion – version which contains extra spaces, missing (i.e. HTTP/0.9) or matches HTTP/1.[2-9]
  • GetHeadZeroContentLength – GET/HEAD request with a “Content-Length: 0” header
• Ambiguous
  • EmptyHeader – if there is an empty header or a line with whitespaces only in the request
  • AmbiguousUri – an URI containing CTL characters
  • UndefinedContentLengthSemantics – Content-Length for GET/HEAD requests
  • UndefinedTransferEncodingSemantics – Transfer-Encoding for GET/HEAD requests
  • DuplicateContentLength – duplicated Content-Length header (same value)
  • BothTeClPresent – both Transfer-Encoding and Content-Length are present in the request
  • SuspiciousHeader – a header that can be normalized to Transfer-Encoding or Content-Length using common text normalization techniques (sanitation, case normalization, delimiters normalization).
• Severe
  • BadHeader – header containing null-character or CR
  • BadUri – URI containing null-character or CR
  • BadVersion – malformed version
  • MultipleContentLength – different Content-Length headers
  • BadContentLength – a non-parseable value or an invalid number
  • MultipleTransferEncodingChunked – multiple Transfer-Encoding: chunked headers
  • BadTransferEncoding – unknown Transfer-Encoding value
  • BadMethod – malformed method
• Parsing raw-requests
  • NonCrLfLineTermination (Acceptable) – allowing “\n” line termination (similar to Nginx).
  • MultilineHeader (Ambiguous) – multi-line headers are non RFC compliant (except Content-Type)
  • PartialHeaderLine (Ambiguous) – if a header line was not terminated
  • MissingLastEmptyLine (Ambiguous) – there is no empty line at the end of request
  • MissingHeaderColon (Ambiguous) – header line doesn’t have colon separator
  • MissingUri (Ambiguous) – there is no URI in the request line

Details on certain classifications

Undefined Content-Length/Transfer-Encoding Semantics (UndefinedTransferEncodingSemantics, UndefinedContentLengthSemantics)

A payload within a GET/HEAD request message has no defined semantics. https://tools.ietf.org/html/rfc7231#section-4.3

https://medium.com/@knownsec404team/protocol-layer-attack-http-request-smuggling-cc654535b6f 3.1 GET Request with CL != 0

https://portswigger.net/web-security/request-smuggling/exploiting See “Capturing other users’ requests” https://www.cgisecurity.com/lib/HTTP-Request-Smuggling.pdf see “EXAMPLE #3”

Both Content-Length and Transfer-Encoding are present (BothTeClPresent)

If a request containing both Content-Length and Transfer-Encoding was received, it means that the sender didn’t follow RFC, and thus there is a chance that request boundaries might be out of sync with the sender.

If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling (Section 9.5) or response splitting (Section 9.4) and ought to be handled as an error. A sender MUST remove the received Content-Length field prior to forwarding such a message downstream.

https://tools.ietf.org/html/rfc7230#section-3.3.2

Multi-line headers (MultilineHeader)

Multi-line headers have been deprecated in RFC 7230, and different engines may either support it or not, which provides malicious actors a toolkit to trick parser to “see” headers that are not there or vice versa. That’s why we mark requests containing multi-line headers as Ambiguous (except the Content-Type header).

Historically, HTTP header field values could be extended over multiple lines by preceding each extra line with at least one space or horizontal tab (obs-fold). This specification deprecates such line folding except within the message/http media type (Section 8.3.1). A sender MUST NOT generate a message that includes line folding (i.e., that has any field-value that contains a match to the obs-fold rule) unless the message is intended for packaging within the message/http media type.

https://tools.ietf.org/html/rfc7230#section-3.2.4

Multiple Transfer-Encoding Chunked (MultipleTransferEncodingChunked)

A sender MUST NOT apply chunked more than once to a message body

https://tools.ietf.org/html/rfc7230#section-3.3.1

Multiple Content-Length Headers (MultipleContentLength, DuplicateContentLength)

If there are multiple different Content-Length headers (different values) the request is marked as Severe. In the case of multiple but same values, it falls into DuplicateContentLength category (marked as Ambiguous).

If a message is received that has multiple Content-Length header fields with field-values consisting of the same decimal value, or a single Content-Length header field with a field value containing a list of identical decimal values (e.g., “Content-Length: 42, 42”), indicating that duplicate Content-Length header fields have been generated or combined by an upstream message processor, then the recipient MUST either reject the message as invalid or replace the duplicated field-values with a single valid Content-Length field containing that decimal value prior to determining the message body length or forwarding the message.

https://tools.ietf.org/html/rfc7230#section-3.3.2

Suspicious headers (SuspiciousHeader)

There is a range of attacks to masquerade Transfer-Encoding and Content-Length headers, so some engines in the chain will see them while others won’t. For example:

Transfer-Encoding : chunked
Content-Length: 100

In this case, some engines may reject this request, as it’s not RFC compliant. Some may sanitize the space before the colon and treat as it has “Transfer-Encoding: chunked” while some may see “Transfer-Encoding[space]” and ignore it. It is the simplest case to illustrate this idea, but there are many others. For instance:

Transfer-Encodıng: chunked

Small Dotless I becomes ASCII “I” on upper-case transformation which may trick some engines. Or have a CTL character:

Transfer_Encoding: chunked
\x01Transfer-Encoding: chunked
Transfer-Encoding\b: chunked

Some engines may normalize delimiters or non-letters, e.g., using regular expressions, etc. (especially using standard string trimming routines that may have different behaviors in different platforms).

To mitigate these risks, we determine the similarity of headers to Transfer-Encoding and Content-Length and mark requests as Ambiguous if any of these deviations are detected.

Mitigations

There are two types of mitigations:

• Reject request with 400 and close the connection
• Serve the request but disable connection re-use on both front-end and back-end.

Why connection is closed after a Severe request

In this case, we cannot establish request boundaries and tell when the next request starts.

Why connections are both FE/BE connections closed after an Ambiguous request?

Let’s start with not re-using a backend connection example:

1. Attacker sends a request, such as Proxy only sees POST /foo while backend also sees GET /poison
2. However Proxy marks the request as Ambiguous
3. Proxy closes the connection after the response
4. The /poison response is dropped as the connection is not going to be re-used.

This seems to be efficient, but falls short if there is a layer in front of the proxy:

1. In this case let’s assume Desync happens between CDN and the Proxy and Proxy marks the request as Ambiguous
2. While Proxy closes the BE connection, it’s not helpful
3. The /poison response is still served via re-used front-end connection

But if both FE/BE connections are closed, then HTTP Desync is prevented:

1. Same as in the previous example, let’s assume Desync happens between CDN and the Proxy and Proxy marks the request as Ambiguous
2. Now Proxy closes both FE/BE connections.
3. The /poison response is dropped.

Usage from C

This library is designed to be primarily used from HTTP engines written in C/C++.

1. Install cbindgen: cargo install --force cbindgen
2. Generate the header file:
   • Run cbindgen --output http_desync_guardian.h --lang c for C.
   • Run cbindgen --output http_desync_guardian.h --lang c++ for C++.
3. Run cargo build --release. The binaries are in ./target/release/libhttp_desync_guardian.* files.

Learn more: generic and Nginx examples.

include “http_desync_guardian.h”
http_engine_request_t – already parsed by the HTTP engine
*/
static int check_request(http_engine_request_t *req) {
http_desync_guardian_request_t guardian_request = construct_http_desync_guardian_from(req);
http_desync_guardian_verdict_t verdict = {0};
http_desync_guardian_analyze_request(&guardian_request, &verdict);
switch (verdict.tier) { case REQUEST_SAFETY_TIER_COMPLIANT:
// The request is good. green light
break;
case REQUEST_SAFETY_TIER_ACCEPTABLE:
// Reject, if mode == STRICTEST
// Otherwise, OK
break;
case REQUEST_SAFETY_TIER_AMBIGUOUS:
// The request is ambiguous.
// Reject, if mode == STRICTEST
// Otherwise send it, but don’t reuse both FE/BE connections.
break;
case REQUEST_SAFETY_TIER_SEVERE:
// Send 400 and close the FE connection.
break;
default:
// unreachable code
abort();
}
}