Ideally, test suites should be faster than the speed of light, more stable than Shawn Michaels at Royal Rumble 1995, and easy to maintain like.. well.. something that is easy to maintain.
In real life things are a bit different. Tests can be shaky – fail, then pass, then fail, unreadable, and slow. Very slow. Slow up to the point that you can here people complaining that the tests slow them down.
In this post I will claim (and hopefully convince) that making an effort to make the tests SUPER FAST will improve also their stability and readability.
Imagine that your development pipeline looks like this
Now, components add up to the system, and more tests for them. Then more component, then more tests, and so on.
When tests add up, they naturally take more and more time to run. It won’t be long until people will start to complain.
Not only it takes time to write them, it takes forever to run them.
Some developer, somewhere
To fix this situation, one can choose to keep only a selected few tests in the development pipeline, cancel the tests completely (reduces the run time indeed..) or maybe to invest some time making them faster.
When I started writing this post I thought it would be relevant for end to end tests only – ones that are testing the system with a browser and are a bit isolated from the system’s code. But on a second thought, it may be relevant also for unit tests or integration tests of other kinds, backend or front end.
In this post I will share some non-trivial reasons that one should consider invest in reducing the run time of the tests.
Reason 1: Stop wasting my time
This is the obvious reason I described above – developers complaining that it takes to long for their new code to be deployed and tested. If there’s a word on the street that the tests are slowing down the development, it’s a slippery slope before cancelling them, taking them out of the pipe line (sometime it’s not a bad option), or having fear of adding new ones.
Reason 2: Makes the tests more focused and readable
When focusing on making tests fast (and test the right thing of course), it can make them be more focused. Test only one thing in each run, avoid temptations such as “I already entered this place in the system, maybe we will add another test scenario on the way”.
Declaring on speed as a parameter all tests puts the developers’ mindset in a very commando like mode – go in the system, test what I need, get out as fast as possible. Like a game, I like it.
Reason 3: Helps find the real slow spots in the system
Lets say that the tests suites are well optimized, everything is stable, but slow. After some refactoring and hard work, it is now also super fast. Nice!
After having a victory celebration with beer and your keyboard, you notice a single test suite for the “add 4 items to cart” flow, that is still very slow.
Well, it might just be that you found a slow spot in the system. It’s not the tests, it’s the system. Perhaps the login phase of each test takes 10 seconds, maybe the time to get to the cart page after clicking add to cart is very slow. NOW you have the testing team helping spotting weak spots that have influence on the client. not only bugs! This might just be the first step in intimating the technical talks between test and dev guys.
Reason 4: Bypass non related false negatives such as network hickups
This is relevant to end to end tests mainly, but can be paraphrased to unit tests as well. Longer tests tend to be less stable then short ones.
Tests that test the system through the browser must assume many things before they start: that the internet is working and stable, that the servers are up, and much much more. These are things that change during the time (server can be rebooted after installation for example) and may happen even during a test, which will make it not too stable.
Probability to the rescue. Imagine that a test lasts 2 minutes, the probability of a server reboot during that time is X. If the run time is reduced magically to 30 seconds, the probability for a reboot for the same test is now x/4, nice!
Reason 5: Fast -> Less steps -> Bugs are easier to reproduce
A bug description with 20 steps can be a nightmare for the one who needs to reproduce it. Not only it takes time, but it does not necessarily helps spotting the real bug. A faster test tend to have fewer steps, so it’s easier to reproduce and investigate (if it fails).
Also, if it’s fast you can run it 100 time in a row in no time to help the dev team understand what happened if it doesn’t reproduce easily.
Reason 6: Less storage for the test videos
Video speaks better than 100 words (on a bug description), and also – no one can argue with a video of a button that cannot be clicked. Many frameworks suggest a way to keep a video of the tests. If the tests take more time, more storage is needed.
To conclude, optimizing the run time is fun, because it has a great benefit, and it is usually an achievable goal that one can learn a lot from.
Hope you found this interesting 🙂