What I Learned @ Sauce Con 2017 – Testing Conference

I had the privilege of attending the first ever Sauce Labs user conference – SauceCon in San Francisco the week of June 5th, 2017 with my two technology leads: Lin, Mao (Albert) and Ross, Derek.  From prior experience attending product sponsored conferences, I was skeptical of the value and was expecting to be inundated with product sales.  I was pleasantly surprised that the focal point of the conference was bringing the Sauce Labs user community together in an intimate setting in the heart of San Francisco and not pushing sales of the product or services.

Here were the key take aways from the sessions I attended and the discussions in between.  At the bottom of the post you’ll find links to the SauceCon presentations.  Hope this is useful!  And don’t hesitate to reach out if you want to explore any of the takeaways in depth.

Key Take Aways:

  • No risk?  No test!  
    • We often times forget one of the main purposes of software testing – to reduce the risk of change associated to any software change.  If there isn’t any risk then why waste time and energy testing it?  As a by product of not understanding and managing risk appropriately, we over test.  There is so much fear in allowing defects to escape QA and enter into production that test every possible condition we can think of and just to be safe, we do it multiple times.  Instead, we should define our risk factors and threshold levels and manage our testing to achieving those levels to ensure we balance speed of delivery and risk of delivery.
  • Emphasize speed over completeness in the early stages of your delivery process (or pipeline) to ensure fast feedback.
    • It’s more important to get fast feedback in seconds and minutes early in your development process over thoroughness and completeness of your testing.  The more you shift left, the more your decisions should be based on and constrained by speed (seconds/minutes) versus as you slide right of midpoint in your pipeline it should be balanced with thoroughness and completeness keeping speed in mind.  For example, linting your code, writing unit test and other tactics provide much faster feedback and are more cost effective than having to push every change to an integrated test environment.  Another great example is using emulators/simulators to test your mobile or hybrid app for early testing over securing real devices.  Emulators are much more cost effective and provide faster feedback.
  • Test stability and reliability of your tests.
    • Test your tests.  Sounds funny right?  But if you think about how important your automated tests are to protecting your CI/CD pipeline, then it isn’t funny at all, it’s what you have to do.  Treat your test code just like your app code and build a test pipeline to ensure standards, best practices, quality and reliability are built in and enforced.  Flaky tests or any test that isn’t trusted will jeopardize your pipeline and the confidence teams have in it.  If trust is nonexistent and confidence is low then you will not be able to gate your pipeline.
    • Monitor your tests.  Track the results and performance of your tests.  If you see a particular test showing signs of “flakiness” then pull it out of the suite and quarantine the test until you can apply appropriate treatments to bring it back into your pipeline.  Remember, your pipeline is built on trust and confidence, flaky test lower trust and confidence.
    • Got a lot of unstable tests?  Host a “Test Stability Bash” to squash those unstable tests.  =)
  • Build test APIs to make your application or solution more testable.  
    • We talk about the importance of building atomic and autonomous tests but we struggle to find techniques and patterns to make that happen.  Instead, we complain about how complicated our applications and solutions are and how intertwined they are and end up throwing our hands up and saying, we can’t write atomic and autonomous tests.  The idea of writing test APIs for internal testing use is brilliant and provides test engineers with the tools to to easily anchor into a particular application state for thinly sliced automation tests versus the traditional chained / dependent tests (every test has to go to the login screen).  Spotify is one of the key companies that embraces this idea of building test APIs for their test engineers and has provided them the ability to deliver fast while maintaining high quality.
  • Stop trying to be perfect in preventing issues, instead get better at recovering from issues.
    • What’s the harm of making mistakes if no one notices or there isn’t any negative impact?  If we are able to quickly recover from production issues with minimal customer impact than what’s the big deal?  No harm, no foul.  If we can embrace this idea of getting better at recovering from production issues than the more we can experiment, move fast and truly learn.  So instead of creating processes on top of processes to try and prevent issues, let’s focus on how quickly we recover with grace and ease.  This enables us to experiment in production like Netflix, Amazon and other modern technology companies.  More importantly, this promotes continuous learning through responsible experimentation.  A good analogy to this idea/concept is Amazon’s culture around decision making.  They value fast decisions over getting the decision right the first time.  If a decision can quickly be reversed with minimal impact, then make the decision and move forward.  Speed is more important than getting the decision right the first time.


Recommended Sessions:

(See playlist link below)

  • Day 1 – Keeping Your Tests Lean: Presented by Meaghan Lewis (QA Engineer) @ Lever.
  • Day 1 – Building Your CD Pipeline with Testing in Mind (Solutions Architect) @ Sauce Labs.
  • Day 2 – Key Note – Cognitive Bias and It’s Impact on Continuous Improvement: Presented by Jason Hand (DevOps Evengelist) @ VictorOps.
  • Day 2 – Release Code with Confidence: Presented by Shivani Sharma (Sr. Manager) @ Slack.
  • Day 2 – Test.AllTheThings() – Mobile Edition: Presented by Asaf Saar + Neil Manvar @ Sauce Labs.
  • Day 2 – Making Your Mobile Apps Automatable: Presented by Dan Cuellar (Principal Development Manager) @ FOODit.

Links to SauceCon Materials