Online Program

Friday, February 22
PS2 Poster Session 2 (with refreshments) Fri, Feb 22, 4:45 PM - 6:15 PM
Napoleon Ballroom

Performance Testing of the Firefox Web Browser

*Christina Choi, Mozilla 
Joseph Kelly, Harvard University 

Keywords: browser, internet, software, testing, analysis, design, multiple comparisons, hypothesis testing

At Mozilla, developers drive open web innovations by introducing new features that will enhance users’ experience of Firefox. Due to the large volume of code changes pushed to the code tree everyday, developers rely on automated in-house tests to identify whether a specific change in the code has introduced a bug; a regression. These tests are intended to measure key aspects of the browser such as its functionality, responsiveness, and performance; however, many of these tests were unable to detect significant regressions due to a lack of explicit test design. To improve this process, a group of statisticians have been working with engineers to integrate more systematic experimental design and analyses in Mozilla’s testing environment. This includes blocking on certain factors to reduce irrelevant variation, the implementation of principled statistical analysis to control for multiple hypothesis testing via a false discovery rate control method and the introduction of smoothing to remove the systematic variation present in the testing environment. These changes have increased the ability to detect regressions in the codebase and opened doors for more scientific testing schema at Mozilla.