What I Learned @ Sauce Con 2017 – Testing Conference

I had the privilege of attending the first ever Sauce Labs user conference – SauceCon in San Francisco the week of June 5th, 2017 with my two technology leads: Lin, Mao (Albert) and Ross, Derek.  From prior experience attending product sponsored conferences, I was skeptical of the value and was expecting to be inundated with product sales.  I was pleasantly surprised that the focal point of the conference was bringing the Sauce Labs user community together in an intimate setting in the heart of San Francisco and not pushing sales of the product or services.

Here were the key take aways from the sessions I attended and the discussions in between.  At the bottom of the post you’ll find links to the SauceCon presentations.  Hope this is useful!  And don’t hesitate to reach out if you want to explore any of the takeaways in depth.

Key Take Aways:

  • No risk?  No test!  
    • We often times forget one of the main purposes of software testing – to reduce the risk of change associated to any software change.  If there isn’t any risk then why waste time and energy testing it?  As a by product of not understanding and managing risk appropriately, we over test.  There is so much fear in allowing defects to escape QA and enter into production that test every possible condition we can think of and just to be safe, we do it multiple times.  Instead, we should define our risk factors and threshold levels and manage our testing to achieving those levels to ensure we balance speed of delivery and risk of delivery.
  • Emphasize speed over completeness in the early stages of your delivery process (or pipeline) to ensure fast feedback.
    • It’s more important to get fast feedback in seconds and minutes early in your development process over thoroughness and completeness of your testing.  The more you shift left, the more your decisions should be based on and constrained by speed (seconds/minutes) versus as you slide right of midpoint in your pipeline it should be balanced with thoroughness and completeness keeping speed in mind.  For example, linting your code, writing unit test and other tactics provide much faster feedback and are more cost effective than having to push every change to an integrated test environment.  Another great example is using emulators/simulators to test your mobile or hybrid app for early testing over securing real devices.  Emulators are much more cost effective and provide faster feedback.
  • Test stability and reliability of your tests.
    • Test your tests.  Sounds funny right?  But if you think about how important your automated tests are to protecting your CI/CD pipeline, then it isn’t funny at all, it’s what you have to do.  Treat your test code just like your app code and build a test pipeline to ensure standards, best practices, quality and reliability are built in and enforced.  Flaky tests or any test that isn’t trusted will jeopardize your pipeline and the confidence teams have in it.  If trust is nonexistent and confidence is low then you will not be able to gate your pipeline.
    • Monitor your tests.  Track the results and performance of your tests.  If you see a particular test showing signs of “flakiness” then pull it out of the suite and quarantine the test until you can apply appropriate treatments to bring it back into your pipeline.  Remember, your pipeline is built on trust and confidence, flaky test lower trust and confidence.
    • Got a lot of unstable tests?  Host a “Test Stability Bash” to squash those unstable tests.  =)
  • Build test APIs to make your application or solution more testable.  
    • We talk about the importance of building atomic and autonomous tests but we struggle to find techniques and patterns to make that happen.  Instead, we complain about how complicated our applications and solutions are and how intertwined they are and end up throwing our hands up and saying, we can’t write atomic and autonomous tests.  The idea of writing test APIs for internal testing use is brilliant and provides test engineers with the tools to to easily anchor into a particular application state for thinly sliced automation tests versus the traditional chained / dependent tests (every test has to go to the login screen).  Spotify is one of the key companies that embraces this idea of building test APIs for their test engineers and has provided them the ability to deliver fast while maintaining high quality.
  • Stop trying to be perfect in preventing issues, instead get better at recovering from issues.
    • What’s the harm of making mistakes if no one notices or there isn’t any negative impact?  If we are able to quickly recover from production issues with minimal customer impact than what’s the big deal?  No harm, no foul.  If we can embrace this idea of getting better at recovering from production issues than the more we can experiment, move fast and truly learn.  So instead of creating processes on top of processes to try and prevent issues, let’s focus on how quickly we recover with grace and ease.  This enables us to experiment in production like Netflix, Amazon and other modern technology companies.  More importantly, this promotes continuous learning through responsible experimentation.  A good analogy to this idea/concept is Amazon’s culture around decision making.  They value fast decisions over getting the decision right the first time.  If a decision can quickly be reversed with minimal impact, then make the decision and move forward.  Speed is more important than getting the decision right the first time.

 

Recommended Sessions:

(See playlist link below)

  • Day 1 – Keeping Your Tests Lean: Presented by Meaghan Lewis (QA Engineer) @ Lever.
  • Day 1 – Building Your CD Pipeline with Testing in Mind (Solutions Architect) @ Sauce Labs.
  • Day 2 – Key Note – Cognitive Bias and It’s Impact on Continuous Improvement: Presented by Jason Hand (DevOps Evengelist) @ VictorOps.
  • Day 2 – Release Code with Confidence: Presented by Shivani Sharma (Sr. Manager) @ Slack.
  • Day 2 – Test.AllTheThings() – Mobile Edition: Presented by Asaf Saar + Neil Manvar @ Sauce Labs.
  • Day 2 – Making Your Mobile Apps Automatable: Presented by Dan Cuellar (Principal Development Manager) @ FOODit.

Links to SauceCon Materials

Should scrum masters remove impediments or empower their teams to?

Where do you draw the line?  How do you make sure the scrum master doesn’t become a dumping ground for work?

Here are my thoughts:

This is an interesting question that I’ve been pondering and I don’t believe the answer is a simple yes or no (maybe it is and I’m just not looking at it from the right perspective).  I think it depends on the situation and where you draw the line between impediments and team responsibilities.  It’s always important to keep responsibilities in mind when examining impediments because what the impediment is and what you need to resolve as a scrum master could be two different things.  You don’t want to overstep your bounds.

I’ll illustrate with an example I’ve seen in the past.  The team identifies they need test data in order to complete verification of a user story and raises this as an impediment.  The scrum master then spends time with the team to understand what test data is needed and why without it the story can’t be verified.  He or she then works through whatever enterprise process is required to request test data.

My issue with this example is that I believe it’s the team’s responsibility to get the necessary test data to verify and complete the story.  If the scrum master does it for them then he or she is taking away their responsibility and not maintaining accountability with the team.  The scrum master needs to address the teams lack of knowledge and understanding on how to request test data.  That’s the underlying impediment the scrum master should be solving.

Obviously removing impediments is part of a scrum master’s duties but it shouldn’t be done at the expense of removing accountability and responsibility from the team (as outlined in my example above).  It’s a delicate balance and one that people need to be careful of.

The Benefits of Pair Programming – Why is it important?

What is it?

Pair programming is two developers working together on a common task on a single machine, ideally with dual monitors, keyboards and mice.  There are two roles in pairing: the driver and the navigator.  The driver is focused on the tactical aspects (i.e. writing the code to get the task done).  The navigator is focused on more strategic aspects (i.e. improvements in the algorithm, potential impacts to other components, etc…).

Why is it important?

Benefits include:

  • Improved software quality
  • Reduction in software maintenance cost
  • Improved productivity
  • Improved business continuity (in other words, knowledge is spread across more people within your team and therefore if someone is out sick, on vacation or quits you are not as vulnerable)

How are those benefits realized?

I hate to start off by answering this question with a cliché but I can’t resist.  Knowledge is power.  One of the primary constraints we face in software development is our own ignorance.  For us to be successful we have to manage it.  The more we know the better we can perform.  Consider the cone of uncertainty and how it represents the variability in our software estimates through the lifecycle of a project.  We are at our smartest at the end of the project when we have the most information.  Therefore, whatever we can do to reduce our ignorance by discovering the information we would normally uncover at the end of a traditional waterfall project earlier will allow us to be that much more successful in delivering quality software.  In other words the act of obtaining and sharing knowledge allows us to achieve better quality software, improved productivity and business continuity.  I will caveat my statement by acknowledging that it’s impossible to know everything and discovery everything upfront (that’s what the traditional waterfall approach attempts to do) but I do believe we can make more use of techniques such as “tracer bullets” to deliberately discover gaps in our knowledge.

Pairing allows two developers either of the same level or mixed to bring their knowledge and understanding of the functionality and the technology into a common medium, normally in the form of code.

This also addresses the 3rd benefit, improved business continuity.  Knowledge sharing in a traditional waterfall organization occurs at the end of a phase or project when all the code has been written.  Pairing promotes real time engagement and knowledge sharing in a more intimate setting.  It also touches on the 3 commonly recognized learning styles (Fleming’s VAK model):

  • Visual Learners – Those that learn through seeing pictures, handouts and other materials.  They also benefit from observing demonstration.
  • Auditory Learners – Those that learn through hearing information in the form of conversations (face to face or over the phone), podcasts, etc…
  • Kinesthetic Learners – Those that learn through hands on activities.  They learn best through experience.

A good pairing session will have a balance of conversation, observation and “doing”.

Instead of a 4 hour code review or presentation in front of 10 or so developers, it’s focused one on one time.  It’s important to realize that the goal and objective of business continuity isn’t that everyone within the team can do everything or will know everything but to ensure “enough” people know to reduce the impacts of natural attrition and absences that occur within organizations.

The 2nd benefit, improved productivity may not be realized initially.  In fact it may feel like less work is being done but it is part of the natural progression.   Pairing is a skill.  People don’t magically become great at pairing.  It takes practice and time to develop like any other skill.  What really drives the productivity is the learning that takes place as they engage on a common task.  That learning can then be applied to future work.

In conclusion, pairing is a technique that can used to help manage knowledge within an organization and it is through the acquisition of knowledge and application of it that leads to higher productive teams.

Give Credit Where Credit is Due:

I want to clearly state that most of what I have mentioned above (if not everything) is not an original thought of my own but my own personal understanding through experiences ranging from presentation and seminars I’ve attended, articles I’ve read (see below) and developers I’ve spoken with.  My intention in writing this is to help flesh out my own understanding and provide a means to share the knowledge I’ve gained through those experiences with my team and colleagues.  The resources below are among many that have helped shape my perspective on this topic.  I encourage all of you to reach them.  =)

Resources:

http://johannesbrodwall.com/2011/02/02/pair-programming-research-misses-the-most-important-point/

http://www.worldwidelearn.com/education-articles/how-do-you-learn.htm

“THE LAWS OF SOFTWARE PROCESS A New Model for the Production and Management of Software”, Armour, Phillip G., 2004

http://dannorth.net/2010/08/30/introducing-deliberate-discovery/

Should story points be a measure of complexity or effort?

In the last affinity estimation session I was in we had an interesting debate on whether story points should be a reflection of complexity (how difficult something is to develop?) or level of effort.  I realized that for many starting out in Scrum it can be very confusing why story points are used instead of hours, which is what most traditional development shops tend to use and what they represent.  In this post, I’d like to address two commonly asked questions regarding story points:

  1. What are story points?  (measure of complexity or effort?)
  2. Why are they used instead of hours?

What are story points? Story points are arbitrary measures for the amount of work to be done to complete a story (or feature).  They are usually unique to the team assigning them, meaning you can’t compare points from one team to another.  Note that it is certainly possible for two or more teams to have similar point values by using the same “anchor story” to base their points on but even if this was done, it is not advisable to compare points and velocity (how many points a team can complete within a given sprint or iteration) for many reasons (See Misuse of Velocity in Agile Projects).  When coupled with a team’s known or projected velocity, it can be used to help inform the product owner and key stakeholders on when features could potentially be delivered.  The most important question story points help answer is, “When can I get it?”.  Knowing the complexity of the story doesn’t answer that question.  Although, understanding complexity is important.  Typically highly complex features tend to have associated risks in the form of performance, security, testing, etc… that need to be actively managed.  But when it comes to story points, it’s effort and not complexity.

Why are story points used instead of ideal hours? Before I answer, I’m assuming that you are familiar with the cone of uncertainty in software estimates.  If you are not then I suggest you read this article (Software Estimation and the Cone of Uncertainty) or do some googling.  =)  Also keep in mind that this is a heavily debated topic with leaders in the Scrum community and that there are books solely dedicated to agile estimating so take this for what it is, a simple blog post.

As mentioned above, story points are arbitrary measurements of work.  Hours are actual measurement of time.  How many times have you been in meetings where developers argue over the amount of time it takes to perform a development task?  Too many to count right?  Points are an easy and effective way to size the amount of “work” required for a story irrespective of time.  How quickly can it be done is not a question of size but a question of velocity. Because hours are a unit of time, it makes it difficult to use as a measurement for size.  Where do hours come into play?  The teams I manage assign hours to their tasks during sprint planning.  This helps them track their progress during the sprint by indicating how much work is left to complete.  Why use hours and not points for story tasks?  I’ll answer that simply by saying at this granular of a level in planning it’s easier to use a meaningful measurement like hours given that the teams have a much better understanding of the work.

Hope this helps!  Happy sizing!

For more information on story points and hours, check out Mike Cohn’s blog post – “It’s Effort, Not Complexity“.