What I Learned @ Sauce Con 2017 – Testing Conference

I had the privilege of attending the first ever Sauce Labs user conference – SauceCon in San Francisco the week of June 5th, 2017 with my two technology leads: Lin, Mao (Albert) and Ross, Derek.  From prior experience attending product sponsored conferences, I was skeptical of the value and was expecting to be inundated with product sales.  I was pleasantly surprised that the focal point of the conference was bringing the Sauce Labs user community together in an intimate setting in the heart of San Francisco and not pushing sales of the product or services.

Here were the key take aways from the sessions I attended and the discussions in between.  At the bottom of the post you’ll find links to the SauceCon presentations.  Hope this is useful!  And don’t hesitate to reach out if you want to explore any of the takeaways in depth.

Key Take Aways:

  • No risk?  No test!  
    • We often times forget one of the main purposes of software testing – to reduce the risk of change associated to any software change.  If there isn’t any risk then why waste time and energy testing it?  As a by product of not understanding and managing risk appropriately, we over test.  There is so much fear in allowing defects to escape QA and enter into production that test every possible condition we can think of and just to be safe, we do it multiple times.  Instead, we should define our risk factors and threshold levels and manage our testing to achieving those levels to ensure we balance speed of delivery and risk of delivery.
  • Emphasize speed over completeness in the early stages of your delivery process (or pipeline) to ensure fast feedback.
    • It’s more important to get fast feedback in seconds and minutes early in your development process over thoroughness and completeness of your testing.  The more you shift left, the more your decisions should be based on and constrained by speed (seconds/minutes) versus as you slide right of midpoint in your pipeline it should be balanced with thoroughness and completeness keeping speed in mind.  For example, linting your code, writing unit test and other tactics provide much faster feedback and are more cost effective than having to push every change to an integrated test environment.  Another great example is using emulators/simulators to test your mobile or hybrid app for early testing over securing real devices.  Emulators are much more cost effective and provide faster feedback.
  • Test stability and reliability of your tests.
    • Test your tests.  Sounds funny right?  But if you think about how important your automated tests are to protecting your CI/CD pipeline, then it isn’t funny at all, it’s what you have to do.  Treat your test code just like your app code and build a test pipeline to ensure standards, best practices, quality and reliability are built in and enforced.  Flaky tests or any test that isn’t trusted will jeopardize your pipeline and the confidence teams have in it.  If trust is nonexistent and confidence is low then you will not be able to gate your pipeline.
    • Monitor your tests.  Track the results and performance of your tests.  If you see a particular test showing signs of “flakiness” then pull it out of the suite and quarantine the test until you can apply appropriate treatments to bring it back into your pipeline.  Remember, your pipeline is built on trust and confidence, flaky test lower trust and confidence.
    • Got a lot of unstable tests?  Host a “Test Stability Bash” to squash those unstable tests.  =)
  • Build test APIs to make your application or solution more testable.  
    • We talk about the importance of building atomic and autonomous tests but we struggle to find techniques and patterns to make that happen.  Instead, we complain about how complicated our applications and solutions are and how intertwined they are and end up throwing our hands up and saying, we can’t write atomic and autonomous tests.  The idea of writing test APIs for internal testing use is brilliant and provides test engineers with the tools to to easily anchor into a particular application state for thinly sliced automation tests versus the traditional chained / dependent tests (every test has to go to the login screen).  Spotify is one of the key companies that embraces this idea of building test APIs for their test engineers and has provided them the ability to deliver fast while maintaining high quality.
  • Stop trying to be perfect in preventing issues, instead get better at recovering from issues.
    • What’s the harm of making mistakes if no one notices or there isn’t any negative impact?  If we are able to quickly recover from production issues with minimal customer impact than what’s the big deal?  No harm, no foul.  If we can embrace this idea of getting better at recovering from production issues than the more we can experiment, move fast and truly learn.  So instead of creating processes on top of processes to try and prevent issues, let’s focus on how quickly we recover with grace and ease.  This enables us to experiment in production like Netflix, Amazon and other modern technology companies.  More importantly, this promotes continuous learning through responsible experimentation.  A good analogy to this idea/concept is Amazon’s culture around decision making.  They value fast decisions over getting the decision right the first time.  If a decision can quickly be reversed with minimal impact, then make the decision and move forward.  Speed is more important than getting the decision right the first time.


Recommended Sessions:

(See playlist link below)

  • Day 1 – Keeping Your Tests Lean: Presented by Meaghan Lewis (QA Engineer) @ Lever.
  • Day 1 – Building Your CD Pipeline with Testing in Mind (Solutions Architect) @ Sauce Labs.
  • Day 2 – Key Note – Cognitive Bias and It’s Impact on Continuous Improvement: Presented by Jason Hand (DevOps Evengelist) @ VictorOps.
  • Day 2 – Release Code with Confidence: Presented by Shivani Sharma (Sr. Manager) @ Slack.
  • Day 2 – Test.AllTheThings() – Mobile Edition: Presented by Asaf Saar + Neil Manvar @ Sauce Labs.
  • Day 2 – Making Your Mobile Apps Automatable: Presented by Dan Cuellar (Principal Development Manager) @ FOODit.

Links to SauceCon Materials


The Benefits of Pair Programming – Why is it important?

What is it?

Pair programming is two developers working together on a common task on a single machine, ideally with dual monitors, keyboards and mice.  There are two roles in pairing: the driver and the navigator.  The driver is focused on the tactical aspects (i.e. writing the code to get the task done).  The navigator is focused on more strategic aspects (i.e. improvements in the algorithm, potential impacts to other components, etc…).

Why is it important?

Benefits include:

  • Improved software quality
  • Reduction in software maintenance cost
  • Improved productivity
  • Improved business continuity (in other words, knowledge is spread across more people within your team and therefore if someone is out sick, on vacation or quits you are not as vulnerable)

How are those benefits realized?

I hate to start off by answering this question with a cliché but I can’t resist.  Knowledge is power.  One of the primary constraints we face in software development is our own ignorance.  For us to be successful we have to manage it.  The more we know the better we can perform.  Consider the cone of uncertainty and how it represents the variability in our software estimates through the lifecycle of a project.  We are at our smartest at the end of the project when we have the most information.  Therefore, whatever we can do to reduce our ignorance by discovering the information we would normally uncover at the end of a traditional waterfall project earlier will allow us to be that much more successful in delivering quality software.  In other words the act of obtaining and sharing knowledge allows us to achieve better quality software, improved productivity and business continuity.  I will caveat my statement by acknowledging that it’s impossible to know everything and discovery everything upfront (that’s what the traditional waterfall approach attempts to do) but I do believe we can make more use of techniques such as “tracer bullets” to deliberately discover gaps in our knowledge.

Pairing allows two developers either of the same level or mixed to bring their knowledge and understanding of the functionality and the technology into a common medium, normally in the form of code.

This also addresses the 3rd benefit, improved business continuity.  Knowledge sharing in a traditional waterfall organization occurs at the end of a phase or project when all the code has been written.  Pairing promotes real time engagement and knowledge sharing in a more intimate setting.  It also touches on the 3 commonly recognized learning styles (Fleming’s VAK model):

  • Visual Learners – Those that learn through seeing pictures, handouts and other materials.  They also benefit from observing demonstration.
  • Auditory Learners – Those that learn through hearing information in the form of conversations (face to face or over the phone), podcasts, etc…
  • Kinesthetic Learners – Those that learn through hands on activities.  They learn best through experience.

A good pairing session will have a balance of conversation, observation and “doing”.

Instead of a 4 hour code review or presentation in front of 10 or so developers, it’s focused one on one time.  It’s important to realize that the goal and objective of business continuity isn’t that everyone within the team can do everything or will know everything but to ensure “enough” people know to reduce the impacts of natural attrition and absences that occur within organizations.

The 2nd benefit, improved productivity may not be realized initially.  In fact it may feel like less work is being done but it is part of the natural progression.   Pairing is a skill.  People don’t magically become great at pairing.  It takes practice and time to develop like any other skill.  What really drives the productivity is the learning that takes place as they engage on a common task.  That learning can then be applied to future work.

In conclusion, pairing is a technique that can used to help manage knowledge within an organization and it is through the acquisition of knowledge and application of it that leads to higher productive teams.

Give Credit Where Credit is Due:

I want to clearly state that most of what I have mentioned above (if not everything) is not an original thought of my own but my own personal understanding through experiences ranging from presentation and seminars I’ve attended, articles I’ve read (see below) and developers I’ve spoken with.  My intention in writing this is to help flesh out my own understanding and provide a means to share the knowledge I’ve gained through those experiences with my team and colleagues.  The resources below are among many that have helped shape my perspective on this topic.  I encourage all of you to reach them.  =)




“THE LAWS OF SOFTWARE PROCESS A New Model for the Production and Management of Software”, Armour, Phillip G., 2004