Project Iris Update – H1 2019 Review

About this Report

This is a report on the progress made in the Mozilla Iris project through the first half of 2019.

Mozilla Iris is a new test framework developed by Mozilla’s Product Integrity group. Unlike other automated test frameworks, Iris aims to visually validate user interaction thereby freeing up manual testers from completing the mundane but very necessary tasks performed each and every product release.

Mozilla Iris running on MacOS

You can find out more by visiting the Mozilla Iris page on Github, dropping by the #qa-automation channel on Slack or by emailing us at iris@mozilla.com.

Project Stats

Iris stats for first half of 2019
  • 942 total PRs in the last 6 months
  • 5.23 PRs per day on average
  • March was the busiest month for Firefox with 27% of the total PRs
  • May was the busiest month for Core with 38% of the total PRs

Key Accomplishments in the past 6 months

  • We now have 502 tests and growing
  • Iris 2.0 shipped
  • Refactored the entire Iris codebase to Python 3
  • Split Iris into a product-agnostic Core and reference PyTest plugin for Firefox

What’s Next

The Iris team recently welcomed Matt Brandt and Kimberley Sereduck to the team bringing with them their expertise from the Services QA group and the recent relaunch of the Bughunter project (now Tantalus). We had a very productive All-hands in Whistler envisioning what this new team could accomplish together over the next six months.

These discussions included:

  • Expanding test coverage to 650 desktop Firefox tests
  • Improving accessibility and usability of the control centre
  • Evaluating where Tantalus and Iris overlap, and where they provide unique value
  • Supporting a tier-1 set of tests running against Nightly builds in CI
  • Moving Release QA test infrastructure into a scalable virtualized environment
  • Spreading Iris across Mozilla to include Mobile, Dev Tools, and Web Compat testing
  • Spreading Iris outside Mozilla

The rest of 2019 looks positive and ambitious. All of us on the team are looking forward to digging in and seeing how far we can push our tools. We’re confident that the tools we’ve developed will lower the bar to participating in test automation, broaden the scope of what’s possible to be tested, and enable Mozilla to ship higher quality products faster.