Code.org is a non-profit organization that provides computer science and AI education to students worldwide. To make this mission a reality, they rely on a web-based programming environment and interactive games to support teachers and engage a diverse student population, while ensuring compatibility across various devices and browsers.
The Challenge: Ensuring Reliable, Accessible Learning at Scale
Capturing and holding students’ attention is hard enough. Code.org aims to make learning easy for teachers and students alike, which means their technology can’t fail mid-class. To help their engineers spend less time troubleshooting, and their users spend more time learning, Code.org wanted to address several key challenges to ensure quality and accessibility across its educational platform, including:
Maintaining stability across diverse environments: School technology resources vary widely, with some classrooms relying on older Chromebooks, tablets, and outdated browser versions (e.g., Android 8, iOS 16.2). Supporting such a broad range of devices makes thorough testing complex and resource-intensive.
Preventing regressions in a large, evolving codebase: Code.org's 12-year-old codebase is maintained by many engineers and volunteers, so it wanted to prevent regressions when making updates.
Ensuring continuous delivery with quality: The team deploys to production daily and requires an efficient, reliable testing process to keep pace with its development velocity.
Optimizing testing efficiency for remote teams: As a fully remote team, maintaining an in-house fleet of test devices is impractical, so cloud-based testing tools are necessary.
Delivering a reliable user experience in the classroom: Even minor bugs or issues can significantly disrupt a 40-minute class period. Code.org needed better testing tools to improve reliability and maintain a quality learning experience.
The Solution: Smart CI/CD Testing for Fast, Reliable Releases
The Code.org team uses Sauce Lab’s automated testing, incorporated into their CI pipelines, and Sauce Lab’s visual comparison testing tool to catch unintentional page shifts or design changes. On the backend, Code.org primarily uses Selenium for its automated browser testing, employing a two-tiered testing approach to balance efficiency and thoroughness:
1. Continuous Integration (CI) Testing: For feature branches, tests run first against a local Chrome driver. If a local test fails, it’s automatically retried using Sauce Labs to determine if the issue is localized. This initial pass optimizes efficiency and manages Sauce Labs API usage. Additionally, the team uses partial testing in CI based on changed files to reduce the overall test cycle time.
2. Continuous Deployment (CD) Testing: Before shipping to production, Code.org runs a full UI test suite in Sauce Labs across a wide array of browsers—typically generating in 6 to 12 test builds daily. Tests are configured via a JSON file to run on various platforms, including:
iPhone simulators with Safari on iOS 16.2 (upgrading soon from 14.5)
Android 8, targeting a significant portion of their global users while excluding newer U.S. traffic
The Impact: Seamless and Reliable Learning Experiences at Scale
Since partnering with Sauce Labs, Code.org has seen measurable improvements across its development and delivery pipeline, helping the non-profit ensure a smooth, frustration-free experience for millions of students and teachers worldwide. These benefits include:
Improved efficiency: Code.org's remote team no longer needs to manage an in-house fleet of test devices, saving substantial engineering time. The team estimates that Sauce Labs’ testing tools reduce the time it spends maintaining systems or waiting for tests by 20%.
Enhanced reliability: Code.org can run comprehensive UI tests across a wide range of real and emulated devices and browser versions, helping catch regressions before they reach the classroom and ensuring equitable access to their computer science education platform.
Fewer bugs: Even small bugs can derail a classroom lesson. Sauce Labs helps Code.org catch issues early with thorough testing, ensuring every deployment delivers a smooth, reliable experience for both students and teachers.
Increased deployment velocity: With confidence in their testing processes, Code.org’s engineering teams can move quickly on new features without worrying about regressions.
“Thanks to Sauce Labs, we don't have to spend a lot of effort gathering tests and devices for different platforms. We have a fully remote team, so even if we wanted to maintain a fleet of different devices in-house, we don't have a place to do that. Being able to run and just configure a JSON file that runs a certain device on a certain OS is super valuable to support the huge variety of users that we have."
-Darin Webb, Software Engineering Manager, Infrastructure at Code.org
Smarter Testing with Sauce Labs to Keep Classrooms on Track
By bringing Sauce Labs into its CI/CD pipeline, Code.org is improving engineering workflows and creating a more reliable, accessible experience for every student and teacher. Automated UI testing with Sauce Labs has helped the team catch issues earlier, reduce bugs and regressions, and maintain a fast development cycle. Most importantly, it gives them the confidence to keep innovating and making computer science education available to all, no matter what device a student has in hand.
