Aug 21

What and how to test? Software testing tutorial

Test-driven development (TDD) is presumably 20 years old, and testing itself is even older. But still, it’s a complicated process. Fortunately, some helpful guidelines can help you write the proper tests.

Be first!

So, the first question you should always ask yourself is: what should my tests be like? And the answer is simple: tests should be FIRST.

This acronym is probably very well known, but in my opinion, it’s always worth going through it again from time-to-time, to see if we remember all the rules:

  1. ‘F’ stands for ‘fast,’ and it’s easy to know if your tests are fast enough. You shouldn’t have to wait to run your test. Also, tests shouldn’t distract your development process. You should be able to make a simple change, then run the test and see the score in a few seconds.
  2. ‘I’ stands for ‘isolated,’ which means two things. Firstly, you should be able to run a single test, from any test suite that you have in your code. You shouldn’t have to run many tests in defined ordered to make them work.
    The other thing is that tests should have only one reason to fail. If you can find two reasons for your tests to fail, you should probably split your test case.
  3. ‘R’ stands for ‘repeatable.’ You should get the same result every single time – doesn’t matter how many times you run your test. You shouldn’t have to rely on any system-like state (for example, date) that varies every time you run your tests. 
  4. ‘S’ stands for ‘self-validating.’ People used to run tests and check if the score at the end is satisfying because there were no test frameworks, and they didn’t have assertions. And that’s precisely why we have the self-validating rule in this acronym.
  5. ‘T’ stands for ‘timely,’ meaning you should write your test at the same time you’re writing the code – not separately. It’s particularly important for two reasons: firstly because there is a risk that you will be committing a code that’s not proven to be working correctly, and secondly, because the truth is that if you don’t write your test straight away, you won’t do it at all – the moment you commit one code, you will start to write another.

Always use the right bicep(s)

So making your tests FIRST was the easy part. Now we go to the second part: what to test? And here we have quite a funny acronym to help us, called RIGHT BICEP.

  • ‘Right,’ meaning you should know what you want to test. Also, your tests should be right, meaning they should test the right thing. You shouldn’t test something that will never happen. You should know what you want to test.
  • Correct ‘boundary conditions’ requires that we take a look at another acronym:

  1. ‘C’ for ‘conformance,’ meaning check if the value conforms to the expected format. So, for example, if you have something like an ID number or a card number, you should validate and test that it’s in an expected format. You should also check what happens if it is not.
  2. ‘O’ for ‘ordering.’ Firstly, if you are returning some collection, you should check if it’s ordered or not, according to your expectations. If you’re putting a collection as an argument to a method, you should check how this method will perform if you inverse or randomize the order in that collection. It might make the test fail.    
  3. ‘R’ for ‘range.’ You should check for the value of the expected range. Maybe -1 or 2000 will not work.
  4. Second ‘R’ for ‘reference.’ If your code is referencing to anything external, check how the test will work when the external state is as expected, and how it will work if it is changed. So, for example, if you are relying on system time, check what will happen if you set a different time. 
  5. ‘E’ for ‘existence,’ meaning does the value that you’re passing to the method exist, or not. If it doesn’t, you should maybe throw some exception.
  6. ‘C’ for ‘cardinality.’ If you are putting collections into the methods, check how your code will act with different collection sizes: one element? Zillion elements? Or maybe an empty collection?
  7. ‘T’ for ‘time.’ You should check for the order of actions. For example: if you’ve got an issue, and you want to change its status to ‘in progress,’ and later on change it again, but this time to ‘closed,’ everything should be fine. But what will happen if you reorder the state transmissions? If you have a ‘closed’ issue and later on you want to change its status to ‘in progress’? You should examine it, and see if your test fails or passes, and be sure that you are aware of all those boundary conditions.

Now, going back to RIGHT BICEP:

  • Checking the inverse relationship is quite a funny concept. Imagine that you are testing division and you’ve got 3 divided by 2 – how can you check if it works? Multiply the score by 2, and if you get 3, it means it works.
    Another example: We have a set of integers, e.g. (1,2,3,4), and we want to write a function that filters these elements according to a rule that requires numbers to be even. How can we test if a function works? We review if the list contains elements 2 and 4 and filter them by creating the ‘A’ collection. We should also check that it doesn’t contain elements 1 and 3 and filter them by creating the ‘B’ collection. Finally, if we add ‘A’ and ‘B,’ we should get a full collection of elements (1,2,3,4).
  • Cross-checking using other means. For example: if you’re writing a super-fast sorting algorithm, how can you test it? You can use the standard algorithm to check if results are correct. If you are writing some mathematical library, you can use standard JAVA mathematical functions to check if the library is correct. 
  • Forcing error conditions. If you’re writing some recursive functions, remember to test extreme cases. If you test code that relies on some connection, mock it and check what will happen if you terminate the connection in the middle. If you are testing some FileReader, give that FileReader the biggest file you can. You can even add some assumptions, like “if I have a file bigger than something, the process will fail” – and that’s fine, as long as you make sure that execution will fail. 
  • Performance characteristics. People used to write tests that required invoking methods numerous times. Tests would fail if their average invocation time were above some fixed value. But this didn’t make much sense since many factors can slow execution down. That’s why we have predefined tools like JMeter, JUnitPerf, JProfiler or Gattling, and we can check if our response time is within some bounds. If it’s not, it still doesn’t have to mean that something is wrong. But you should be at least aware that it’s not within bounds and decide that you should do something about it. 

Things to remember

About The Author

Marcin Haręza - Software Developer and Team Leader at SoftwarePlant