Sauce Connect is an integrated HTTP proxy server that is a secure gateway for accessing local environments behind a corporate firewall while running tests on Sauce Labs. This added security layer prevents exposure of internal data to the public internet, enabling users to test their web and mobile applications safely.
Sauce Connect 5.0 features significant enhancements in performance, security, and integration. This new version boasts up to five times faster test speeds and 50 times less memory usage than its predecessor, allowing for lower latency and more efficient test executions, especially when multiple sessions are in use. Transitioning from the proprietary KGP protocol to the widely adopted HTTP/2 protocol with SOCKS5 and websocket support simplifies security approvals and strengthens connections. The streamlined onboarding process facilitates easier integration into CI/CD environments, while a new Prometheus metrics dashboard provides in-depth performance insights. Overall, Sauce Connect 5.0 represents a significant upgrade, offering robust security and improved functionality for testing.
We’re pleased to announce the release of Sauce Connect v5.2.0, which offers significant performance enhancements, including up to a 32.5% improvement in network throughput and an impressive 850% increase in request handling compared to earlier versions. These upgrades contribute to faster, more secure, and more reliable testing.
This article will walk you through the key enhancements in Sauce Connect v5.2.0, share benchmark results highlighting the performance gains, and explain why now is the perfect time to migrate from v4 to the latest version.
What’s New in Sauce Connect v5.2.0?
In Sauce Connect v5.2.0, we've introduced several enhancements designed to significantly improve performance, security, and usability. Here's a closer look at what's new:
Reverting to Resigning Domains by Default In Sauce Connect 5.0.0, we shifted the default mode to non-resigning, prioritizing lower CPU usage and efficiency for specificcertain use cases. However, after extensive benchmarking in version 5.2.0, we’ve concluded that resigning mode provides superior performance for latency-sensitive workloads, particularly under heavy HTTP/2 traffic. While non-resigning mode does use slightly less CPU, the difference is minimal compared to the significant improvements in latency and request handling offered by resigning mode. As a result, we’re reverting to resigning domains by default in 5.2.0, ensuring that customers benefit from optimal performance without compromising efficiency. See the benchmark results below for a detailed comparison.
Increased Tunnel Capacity A significant update in v5.2.0 is the improvement in tunnel capacity. The tunnel now establishes 4 connections per CPU core, with a maximum of 16 connections. This is a notable increase from the previous constant of 2 connections. Each connection supports up to 256 concurrent requests, meaning that in optimal conditions, users can now handle up to 4,096 concurrent requests. This addresses the slowness some clients experienced in earlier versions due to the previous limit of 1024 concurrent requests.
Support for SSLKEYLOGFILE We’ve added support for SSLKEYLOGFILE, which allows you to inspect encrypted traffic using tools like Wireshark. This feature is especially useful for troubleshooting and security audits, as it enables detailed inspection of secure traffic flows. For more information on how to use this feature, check out our documentation.
Automatic Client Metrics Collection: Version 5.2.0 now automatically collects client metrics and streams them to our servers for statistical purposes. This feature streamlines the issue resolution process, as our support team will have access to the necessary data in real time.
Improved Log Visibility To make debugging and monitoring easier, we’ve enhanced the logging capabilities in v5.2.0. The new logs provide better visibility into tunnel operations, making it simpler to diagnose issues and optimize performance.
Performance Benchmarks: Sauce Connect 5 vs. 4
To ensure that Sauce Connect v5.2.0 delivers tangible performance improvements, we conducted a series of benchmarks that closely mirror real-world usage scenarios. Using the same tools and setups our customers typically employ, we’ve made it easy for anyone to reproduce these benchmarks and verify the results.
Benchmark Methodology
For our testing, we employed two distinct benchmarking setups to measure different aspects of Sauce Connect’s performance:
Streaming capabilities testing with Fast.com We utilized Fast.com to gauge the tunnel's capacity under various conditions. This tool allowed us to test how well Sauce Connect streams large volumes of data, reflecting scenarios where users require high throughput for their testing needs.
Latency testing with Vegeta (HTTP Loader) To accurately measure Sauce Connect’s performance under heavy load, we employed Vegeta, a high-performance HTTP load testing tool. This allowed us to test how many requests per second the tunnel could handle while maintaining low latency, specifically focusing on p99 latency (the time within which 99% of requests are completed).
The Sauce Connect client can operate in two distinct modes, each of which affects tunnel behavior and performance:
TLS Resign mode (default since v5.2.0): In that mode, Sauce Connect decrypts encrypted HTTPS traffic at the start of the tunnel and re-encrypts it at the tunnel's end. While this adds some processing overhead, it allows for more efficient use of HTTP/2 connections by enabling the tunnel to handle multiple streams simultaneously. This results in better performance when dealing with large volumes of concurrent requests. More details can be found in the documentation.
TLS Passthrough mode: In this mode, the Sauce Connect client passes TLS traffic through without terminating it, thus avoiding the processing overhead associated with re-encryption. However, this mode has its own limitations: it utilizes only one HTTP/2 stream per proxy connection and can hold it for a very long time, which can reduce the overall efficiency of tunnel connections.
More details can be found in the documentation.
To ensure consistency and eliminate bottlenecks unrelated to Sauce Connect, we ran our benchmarks on a robust infrastructure setup:
Google Cloud VM (GCE VM) We hosted the Sauce Connect client on a Google Cloud VM (GCE VM), specifically the e2-standard-8 instance with ubuntu-os-cloud/ubuntu-2204-lts image. This instance features 8 CPU cores, 32GB of memory, and a high-bandwidth network connection, ensuring that our test environment was not a limiting factor.
Fast.com Benchmark:
To evaluate the data throughput capacity of Sauce Connect, we used fast.com, a popular tool for measuring internet speed. However, we recognized that the results can sometimes be inconsistent due to the nature of its estimation algorithms. To ensure a more accurate and stable measurement, we supplemented these results by collecting network metrics directly from the GCE VM. These metrics provided a reliable and unanimous view of the traffic load, allowing us to better assess the performance of Sauce Connect in both TLS resigning and non-resigning modes.
The benchmark was conducted using the following setup:
1. Test VM in Sauce Labs | - A test virtual machine in Sauce Labs was used to load the fast.com website via Playwright. - The script would download files for 5 minutes, using 16 connections (fast.com settings). |
2. Sauce Connect Server (sc-server) | - The sc-server was set up to manage the connections between the test VM and the Sauce Connect client. |
3. Sauce Connect Client (sc-client) | - The sc-client was running in a GCE VM - It was given 2 CPUs. |
4. Fast.com | - The fast.com servers were accessed through the Sauce Connect tunnel. |
The fast.com servers that were accessed through the Sauce Connect tunnel.
Results:
Key Findings:
Version 4.x: The average transfer rate ranges from 58 MB/s to 65 MB/s, reflecting modest performance levels.
Version 5.x: Upgrading to the 5.x versions leads to a notable improvement, with average transfer rates climbing to 78 MB/s to 85 MB/s. This represents an increase of up to 32.5% in network throughput.
Impact of TLS Resigning Mode: In the 4.x series, enabling TLS resigning mode makes a significant difference in performance. However, in the 5.x series, the effect is negligible, indicating that the 5.x versions benefit from stronger base optimizations.
Latency Under Load Benchmark:
We employed Vegeta, a powerful HTTP load testing tool, to measure how many requests the tunnels could handle while maintaining a 99th percentile (p99) latency below 100 milliseconds. This benchmark was designed to push Sauce Connect to its limits under realistic, high-load conditions.
The benchmark was conducted using the following setup:
1. Multiple Test VMs with Vegeta | - Several test VMs within Sauce Labs were configured to run Vegeta as a pre-run executable. - Each Vegeta instance was sending 300 HTTPS GET requests per second with up to 20 connections over the tunnel to the gnet server. |
2. Sauce Connect Server (sc-server) | - The sc-server managed the incoming requests from Vegeta, passing them through the Sauce Connect tunnel. |
3. Sauce Connect Client (sc-client) on GCE VM | - The sc-client was running on a Google Cloud Engine (GCE) VM. - It was given 2 CPUs |
4. Envoy on GCE VM | - Envoy was deployed on the very same GCE VM to handle TLS termination, converting HTTPS requests into plain HTTP for the Gnet server. - It was given 4 CPUs. |
5. Gnet on GCE VM | - The Gnet server received plain HTTP requests and processed them with high efficiency, providing a clear measure of how well Sauce Connect could manage a high number of requests under low latency. - We used Gnet as our HTTP server because it was the top performer in the TechEmpower Benchmark Round 22, excelling in the "best plaintext responses per second" category. However, Gnet does not natively support HTTPS, so we adapted the setup with Envoy. - It was given 2 CPUs. |
Key findings:
Version 4.x Performance: On average, versions 4.x can handle about 1,000 requests per second (req/s) before experiencing a 99th percentile (p99) latency of 100 milliseconds (ms). Notably, version 4.9.2 stands out by achieving around 2,500 req/s.
Version 5.1.3 Enhancements: This version shows significant improvements, reaching up to 6,500 req/s in non-resigning mode and up to 8,500 req/s in resigning mode. This represents a remarkable 160% improvement over version 4.9.2 and a staggering 750% increase compared to the 4.x versions when resigning is enabled.
Version 5.2.0 Advancements: With this version, the performance further escalates, handling up to 7,500 req/s in non-resigning mode and 9,500 req/s in resigning mode, still staying below the critical 100 ms p99 latency threshold. This marks a 200% improvement over 4.9.2 and a 15% increase over version 5.1.3 in non-resigning mode. In resigning mode, it boasts an 850% improvement over 4.9.2 and a 12% enhancement over version 5.1.3.
HTTP/2 Stream Handling: Both versions 5.1.3 and 5.2.0 excel in managing HTTP/2 streams in TLS resigning mode, allowing for significantly higher request rates while maintaining latency below 100 ms.
Upgrade to Sauce Connect 5.2.0 Today
With its enhanced performance and capacity, now is the ideal time to upgrade to Sauce Connect v5.2.0. Experience the benefits firsthand and streamline your testing process. To begin your migration, follow our guide today!
