STEWS : A Security Tool For Enumerating Web Sockets

STEWS is a tool suite for security testing of Web Sockets

This research was first presented at OWASP Global AppSec US 2021

Features

STEWS provides the ability to:

• Discover: find WebSockets endpoints on the web by testing a list of domains
• Fingerprint: determine what WebSockets server is running on the endpoint
• Vulnerability Detection: test whether the WebSockets server is vulnerable to a known WebSockets vulnerability

The included whitepaper in this repository provides further details of the research undertaken. The included slide deck was presented at OWASP AppSec US 2021.

Complementary respositories created as part of this research include:

• The Awesome Web Socket Security repository, which compiles Web Sockets security information for future researchers
• The Web Sockets-Playground repository, which is a script to easily jump start multiple local Web Socket servers in parallel

Installation & Usage

Each portion of STEWS (discovery, fingerprinting, vulnerability detection) has separate instructions. Please see the README in each respective folder.

Web Socket Discovery

STEWS Discovery Tool

The STEWS (Security Tool for Enumerating WebSockets) discovery tool uses a custom fork of ZGrab2 to test URLs for WebSocket support by sending the first part of a WebSocket connection handshake. If the server responds to this WebSocket connection request with a HTTP 101 “Switching Protocols” response, it can be assumed that the server supports WebSockets. The approach used for WebSocket endpoint discovery is a brute-force approach that relies on a wordlist. This is because WebSockets may only be accessible at a specific path of a server. By sending out large numbers of these WebSocket handshake requests and filtering for servers that respond with a 101 status code, many WebSocket endpoints can be discovered. However, there are some weaknesses to this approach:

• Specific URL paths are tested, which means that a WebSocket endpoint not at this location will not be detected.
• ZGrab2 is a work in progress, with some key PRs for improved HTTP support merged in the last year or two
• ZGrab2 was not originally designed to operate at the HTTP layer, but at the TCP/IP layer. Therefore, ZGrab2 doesn’t solve high throughput DNS lookups and this can be a problem point depending on your configuration. See the following DNS tip for the current recommended approach.

DNS tips

If you have used a common web fuzzer or URL brute force tool such as gobuster or ffuf, you have likely used this tool against a single domain. Because the STEWS discovery process is testing many different domains, a large number of DNS requests will occur. The DNS lookup process can take just as much time, if not more, than sending the actual WebSockets request. If you are using your ISP’s default DNS server, you will likely reach the lookup rate limit and start encountering DNS errors that can cause missed Web Socket endpoints.

The approach used for testing on a vanilla Ubuntu system that is relying on the /etc/resolv.conf file for DNS server is to add several well-known public DNS servers, such as Google (8.8.8.8 and 8.8.4.4), Quad9 (9.9.9.9), and Cloudflare (1.1.1.1 and 1.0.0.1) to the /etc/resolve.conf file. When your system is performing the DNS lookups and does not get a response from the first DNS nameserver, it will use other DNS servers in the /etc/resolv.conf, which can help balance the DNS request load in case the rate limit has been hit on other nameservers in the /etc/resolv.conf file.

There are optimizations that can speed up discovery beyond the approach described above. For example, zgrab2 accepts input files that contain the IP of the domain, in the format IP,domain, to allow zgrab2 to skip the DNS lookup step. This approach saves time if many URL paths are being tested (1 DNS lookup per domain rather than a DNS lookup per domain for each URL path tested).

If you aren’t discovering any WebSockets endpoints and suspect DNS lookups may be the issue, you can use Wireshark or tcpdump to troubleshoot the issue.

Domain list tips

There are many ways to get a long list of domains to test for WebSockets.

• If you want to manually find endpoints to discover new URL paths where WebSockets may exist beyond what is listed in the sample discovery results table, there aren’t many known shortcuts beyond manual browsing. Finding repositories on GitHub that contain many WebSockets endpoints (such as this cryptofeed repo).
• If you are focused on testing a specific domain or set of domains, you can use a list of the domains and subdomains in scope.
• If you are scanning the web, you can either search on your favorite search engine for “top million domains” or “top 100 million domains”. Lastly, for a more comprehensive list of domains, you can request access to the same source that top level DNS servers use, zone files. You can submit a request to ICANN for these zone files. As a warning, the .com zone file is a 21+ GB text file and the .org zone file is 1.5+ GB. Additionally, the zone files contain many domains that resolve to 0.0.0.0, internal IPs, etc. that could be cleaned or minified before using.

Usage and dependencies

The STEWS-discovery.sh script is a bash script tested on Linux. The only dependencies are jq and a zgrab2 binary from the custom Palindrome Technologies zgrab2 fork (a working binary can be downloaded from here). This zgrab2 fork makes the following changes (as of Nov 2021):

• The DynamicOrigin flag is added to set the “Origin” header to the target domain without path (in case Origin is checked for CSWSH mitigation)
• To simplify the Web Sockets handshake HTTP request, the User Agent header, the Accept-Encoding header, and the Accept header are all removed, the latter using a new RemoveAcceptHeader flag
• The Endpoint flag is removed because the endpoint path is included in the URL list provided as input

The script uses the known-endpoints.txt by default (these known Web Sockets servers are part of bug bounty programs), but any text file of domains can be provided as input.

The STEWS-discover.sh script can be modified to view additional information about each server. For example, adding .data.http.result.response.headers to the values provided to jq will output the headers from each server.

Sample discovery results

From a sample size of ~3 million domains tested in Nov 2021, the following table illustrates the number of servers discovered that supported WebSockets for each URL pattern. The xxx characters imply a variety of TLDs were tested.

URL	WebSocket servers found
domain.xxx	2281
domain.xxx/ws	1991
domain.xxx/ws/v1	1605
domain.xxx/ws/v2	1606
domain.xxx/socket.io/?EIO=3&transport=websocket	1389
domain.xxx/stream	448
domain.xxx/feed	452
www.domain.xxx	1582
ws.domain.xxx	891
stream.domain.xxx	574
Total	12819

Web Socket Fingerprinting

STEWS Fingerprinting Tool

The STEWS (Security Tool for Enumerating WebSockets) fingerprint tool uses implementation-level differences in popular WebSocket implementations to try to identify running WebSocket servers. The STEWS fingerprinting tool uses server features both in the WebSocket handshake (using the HTTP protocol) and in the WebSocket protocol frames (using the WebSocket protocol), requiring the tool to handle two different protocols.

Web Sockets Fingerprint Tests

In the process of testing different WebSocket servers, differences in implementation were found that helped identify different WebSocket servers. These differences could allow a user to identify a server by sending crafted messages that triggered the servers to respond with their identifying features. Some identifiers were found to be better (or higher signal-to-noise for identification) than others.

For example, the capitalization of HTTP headers in the WebSocket handshake response can be modified by a reverse proxy or other intermediate network element, regardless of the WebSocket implementation’s source code. Similarly, no major differences were found in server responses when different masking keys were used to send messages to servers. To examine other possible fingerprinting client requests, it can be helpful to examine the format of the WebSocket data frame from RFC6455.

Basic Usage

First, make sure you have the necessary Python 3 dependencies installed using pip3 install -r requirements.txt. Then if you run python3 STEWS-fingerprint.py -h you will be greeted by the following options:

usage: STEWS-fingerprint.py [-h] [-v] [-d] [-u URL] [-f FILE] [-n] [-k]
[-o ORIGIN] [-g] [-a] [-1] [-2] [-3] [-4] [-5]
[-6] [-7]
Security Testing and Enumeration of WebSockets (STEWS) Fingerprinting Tool
optional arguments:
-h, –help show this help message and exit
-v, –verbose Enable verbose tracing of communications
-d, –debug Print each test case to track progress while running
-u URL, –url URL Provide a URL to connect to
-f FILE, –file FILE Provide a file containing URLs to check for valid
WebSocket connections
-n, –no-encryption Connect using ws://, not wss:// (default is wss://)
-k, –nocert Ignore invalid SSL cert
-o ORIGIN, –origin ORIGIN
Set origin
-g, –generate-fingerprint
Generate a fingerprint for a known server
-a, –all-tests Run all tests
-1, –series-100 Run the 100-series (opcode) tests
-2, –series-200 Run the 200-series (rsv bit) tests
-3, –series-300 Run the 300-series (version) tests
-4, –series-400 Run the 400-series (extensions) tests
-5, –series-500 Run the 500-series (subprotocols) tests
-6, –series-600 Run the 600-series (long payloads) tests
-7, –series-700 Run the 700-series (hybi and similar) tests

Each series of tests enumerates a specific part of the Web Socket protocol. If you want to see how the tool works, try running a single series of test first, such as the 500 series tests. It is useful to add the debug flag, -d, to observe the progress as test cases are being run. If you have a server running on local port 8080 and want to test the 500 series of test cases, you might use:

python3 STEWS-fingerprint.py -5 -d -n -u 127.0.0.1:8080

If instead you wish to test a public server using TLS and do not want to see the verbose debug info, you might use:

python3 STEWS-fingerprint.py -1 -k -u streamer.finance.yahoo.com

Running all test cases with the -a flag provides the most precise fingerprint matching, but it can also take a lot of time and require sending a lot of data to the endpoint being fingerprinted. The series 600 test cases in particular send very long payloads to the server.

If you have several Web Socket endpoints that you want to fingerprint, you can use the -f flag to provide a file of Web Socket endpoints for testing.

How it works

The WebSocket connection process can be split into two main parts:

• HTTP communication (WebSocket handshake with a HTTP 101 response)
• WebSocket communication

Both portions of the WebSocket connection can allow for fingerprinting.

The HTTP communication contains several fields that may provide information about the supported features on the server, including:

• Sec-WebSocket-Version (series 300)
• Sec-WebSocket-Extensions (series 400)
• Sec-WebSocket-Protocol (series 500)

The WebSocket communication contains several fields that can determine the server-size supported features, including:

• Opcodes (series 100)
• Reserved bits (series 200)
• Maximum payload length (series 600)
• Older WebSocket hybi draft protocol features support (series 700)

By sending unexpected or edge case inputs to the WebSocket server, the STEWS fingerprinting tool can receive different responses from the WebSocket server depending on what server is running. For example, compare the server responses to test case 200 between three different WebSocket implementations:

• Faye: One or more reserved bits are on: reserved1 = 0, reserved2 = 0, reserved3 = 1
• Gorilla: unexpected reserved bits 0x10
• Java Spring Boot: The client frame set the reserved bits to
• Ratchet: Ratchet detected an invalid reserve code

By collecting many such varying responses, the STEWS finger printer can compare any Web Socket server’s fingerprint against database of known Web Socket servers to attempt to identify it. The current fingerprint database was created using the Web Sockets-Playground repository, which simplifies the process of starting multiple local Web Sockets servers. The current fingerprint matching algorithm is very basic and assigns one or two points to each test case to weight some test case results more heavily than others. The fingerprint in the STEWS finger printer database with the smallest points delta is considered the top candidate for an identification match.

Adding New Web Socket Server Fingerprints

Please submit a pull request (PR) if you have a Web Socket fingerprint to add. The fingerprint definition for a WebSocket server is created in the form of a list. You can generate this list for a known Web Socket server using the -g or --generate-fingerprint flag of the STEWS fingerprint tool. Manual edits of the automatically generated fingerprint are recommended based on the test case. For example, the test cases in the “contain Cases” array (currently test cases 104-206) use a string find test rather than an exact match to determine the fingerprint delta.

False Positives and Errors

If testing a Web Socket server over the internet, additional network elements can interfere with the fingerprinting results. Fingerprinting identifiers from the handshake process, which happens over HTTP, may be modified by a reverse proxy or WAF. It is better to focus on using the Web Socket post-connection fingerprinting identifiers in this situation (see the categorization in the How it works section), especially error messages that are usually unique to specific Web Socket servers.

Web Socket Vulnerability Detection

STEWS Vulnerability Detection Tool

The STEWS (Security Tool for Enumerating WebSockets) vulnerability detection tool allows users to test whether a WebSockets endpoint is vulnerable to known CVEs or other WebSockets vulnerabilities.

The tool currently supports tests for vulnerabilities including:

• CSWSH (Cross-Site WebSocket Hijacking)
• CVE-2020-27813 (Gorilla DoS Integer Overflow)
• CVE-2020-7662 & CVE-2020-7663 (faye Sec-WebSocket-Extensions Regex DoS)
• CVE-2021-32640 (ws Sec-Websocket-Protocol Regex DoS)

A more complete list of CVEs that this tool might support in the future can be found in the Awesome WebSocket Security repository.

Basic Usage

First, make sure you have the necessary Python 3 dependencies installed using pip3 install -r requirements.txt. Then if you run python3 STEWS-vuln-detect.py -h you will be greeted by the following options:

usage: STEWS-vuln-detect.py [-h] [-v] [-d] [-u URL] [-f FILE] [-n] [-k] [-o ORIGIN] [-1] [-2] [-3] [-4]
Security Testing and Enumeration of WebSockets (STEWS) Vulnerability Detection Tool
optional arguments:
-h, –help show this help message and exit
-v, –verbose Enable verbose tracing of communications
-d, –debug Print each test case to track progress while running
-u URL, –url URL URL to connect to
-f FILE, –file FILE File containing URLs to check for valid WebSocket connections
-n, –no-encryption Connect using ws://, not wss:// (default is wss://)
-k, –nocert Ignore invalid SSL cert
-o ORIGIN, –origin ORIGIN
Set origin
-1 Test for generic Cross-site WebSocket Hijacking (CSWSH)
-2 Test CVE-2021-32640 – ws Sec-Websocket-Protocol Regex DoS
-3 Test CVE-2020-7662 & 7663 – faye Sec-WebSocket-Extensions Regex DoS
-4 Test CVE-2020-27813 – Gorilla DoS Integer Overflow

Test 1 provides a generic CSWSH test. This can be used in combination with the -o flag to specify a specific origin to attempt to bypass any server-side checks.

Tests 2, 3, and 4 check for specific CVEs. The test cases for these were created based on the PoC code published as part of the discovery of these CVEs. For example, to run test 4 on a local server on port 8084, you can run: python3 STEWS-vuln-detect.py -4 -n -u 127.0.0.1:8084

Why this tool?

WebSocket servers have been largely ignored in security circles. This is partially due to three hurdles that have not been clearly addressed for WebSocket endpoints:

• Discovery
• Enumeration/fingerprinting
• Vulnerability detecting

STEWS attempts to address these three points. A custom tool was required because there is a distinct lack of support for manually configured WebSocket testing in current security testing tools:

• There is a general lack of supported and scriptable Web Socket security testing tools (for example, NCC’s unsupported wssip tool, nuclei’s lack of Web Socket support, and nmap’s lack of Web Socket support)
• Burp Suite lacks support for Web Socket extensions (for example, see this Port Swigger forum thread and this one).
• There is a lack of deeper Web Socket-specific security research (the Awesome Web Socket Security repository lists published Web Sockets security research)
• The proliferation of Web Sockets around the modern web (as seen in the results of the STEWS discovery tool)