Test-Driven Development: The bug killer
In this post, we'll explore the application of Test-Driven Development (TDD) in the world of embedded devices. We'll discuss how TDD can address the challenges of the traditional debug-driven approach, and the benefits and potential drawbacks of this methodology. We will not disscuss, however, how to write high-quality tests or how to set the testing environment up. That may come in a future post.
A (De)bug-Driven approach
Traditionally, software development for embedded devices has been largely debug-driven. This approach involves writing code and then spending significant time debugging and performing manual tests to ensure it functions as expected (don’t look at me, you know you’ve also done it). This method, still widely used across teams, leads to long and unpredictable development time, as the debugging phase usually takes longer than development itself, only finishing when enough time has passed without detecting a bug.
This approach is also frustrating for developers. Instead of focusing on solving real problems and creating innovative solutions, they often find themselves swamped in bug fixing, decreasing productivity and job satisfaction.
On average, developers spend over 40% of their time debugging software, according to Beningo Embedded Group (https://www.beningo.com/5-strategies-for-minimizing-debug-time/). Other sources suggest that embedded software engineers spend between 20% to 40% of their time debugging.
These figures indicate that debugging can consume a substantial part of the development cycle in embedded software teams.
TDD as an alternative
But, what if we had a way out of this recurring issue? that’s where TDD comes in. It's a development methodology where tests are written before the actual code. The process follows a simple cycle: write a test and watch it fail, write the minimum code necessary to pass the test, and then refactor the code for optimization. That’s why it’s also known as Red, Green, Refactor.
This approach ensures that every piece of code is tested, leading to robust and mostly* bug-free reliable software, which is critical for embedded systems. By focusing on testing from the beginning, TDD will reduce the time spent on debugging and allow developers to focus on solving real problems.
*This doesn’t mean TDD will get rid of all bugs, but ensures you get rid of most of them in a very early stage of development.
Pros & Cons of TDD
TDD has its advantages and disadvantages. On the positive side, it leads to improved code quality, reduces the likelihood of bugs, and makes refactoring safer and easier, while also encouraging simpler designs and higher modularity, a must for maintaining and scaling the code.
However, TDD also has its challenges. It requires a shift in mindset, as writing tests before code is not a common practice. It can also be time-consuming, especially in the initial stages. Moreover, writing meaningful tests for embedded systems can be complex due to hardware dependencies and real-time constraints. Keep in mind on-target testing will almost double your binary size, so you either need memory space available for tests or to test your system in small chunks, which is not always possible.
Generally speaking, I would say it really pays off, both during development and especially in the later stages of your project, where every bug is expensive.
TDD in Practice
Implementing TDD in embedded systems development involves a few key steps:
First, write a test for the smallest possible functionality. This test should fail initially because the functionality has not been implemented yet.
Next, write the minimum amount of code required to pass the test.
Once the test passes, the code can be refactored to improve its structure and efficiency, while ensuring that it still passes the test.
This cycle is repeated for each new functionality, gradually building up the software while ensuring that all code is thoroughly tested. This approach leads to more predictable code and timeline, as the time spent on debugging is significantly reduced (remember this was the biggest time and effort investment in a generic project).
Dual targetting TDD
Since the main objective of TDD is to catch as many bugs as early as possible, using multiple compilers will be beneficial. You can quickly iterate tests on desktop and after a feature is complete or after some time, test on target.
Even if the optimal environment for unit testing is a host machine or a simulator (enables full register control, even read/write only regions, etc.), testing on target can give more accurate info (sizeof(), endianess) if the circumstances allow it.
This workflow exploits the strengths of dual targetting TDD. Testing on the host machine means the test cycle is fast, and using the target compiler ensures we don’t use features only available in our host compiler.
Example - Circular buffer
In this simple example, we’re going to create a circular buffer using dual targetting TDD, with standard MinGW for Windows and the arm-gcc compiler with a STM32F4 board. We’ll also use Unity as our testing framework and the PlatformIO extension for VSC as our IDE of choice, but feel free to use the IDE or toolchain your heart desires.
Disclaimer: I chose PlatformIO because it’s open source and provides tools for quickly and efficiently setting up a project. However, I will not dive into how to set up PlatformIO itself (check reference links). This is just an example to showcase how easy it is to develop using dual targetting TDD.
Set up
For dual targetting TDD, we’ll need 2 environments in platformio.ini:
Windows native (MinGW)
STM32F401RE (arm_gcc)
You don’t have to include the HAL drivers for this example.
We also need to separate the tests folder from the actual code; while testing, you don’t want to run your main code, but the tests you created:
For setting up Unity, you need to include set up and teardown functions in your main test file (test_circular_buffer). They will be called at the beginning and end of each test. You also need a main function to start and close the unity framework, and run your tests.
Our circular buffer will have the following requirements:
It will have a size of 10 elements
The elements will be of type uint8_t
When the buffer is full, adding a new element should overwrite the oldest element in the queue
The functionality must include:
Initialization of a circular buffer
Adding a new element to a circular buffer
Getting an element from a circular buffer
Reporting the number of elements in the buffer
Reporting if a circular buffer is empty
Reporting if a circular buffer is full
Cleaning the circular buffer
With setup out of the way, it’s time to start testing!
2. TDD on native
Using platformIO for native testing is pretty straightforward. Just remember that drivers, HAL and HW interactions are out of the table, unless you want to mock them. You should focus here on your business/app layer, which should be completely decoupled from the HW itself. Remember the process:
1 - Create a failing unit test
2 - Write the minimum code necessary for it to work
3 - Refactor if needed
Our buffer will have this structure
These will be the functionalities to implement
T1 - Buffer initialization
First, we need to test if the circular buffer has been correctly initialized. This means that the number of elements in it after initialization will be 0.
We need defaults functions for initializing it and checking if the buffer is empty, so our program actually compiles.
Our first test
The default functions
As expected, the test failed. Now, let’s populate the default functions with the minimum amount of code needed for the test to pass.
Minimal functionality for the test to pass
Now the test passed! Let’s continue with more tests until we get all functionality done.
T2 - Adding an element
Same way as before, we create a test to check if adding an element works as expected. It should make the buffer be recognized as not empty anymore.
After running this test, it fails. Now it’s time to complete the the add_element function.
Now, our test passes!
T3 - Oldest element overwrite
For our next test, we will make sure that, when we add an element and the buffer is full, it should overwrite the oldest element.
As expected, the test fails. Now, let’s add some new functionality to make sure all of them pass:
Aaaand, it passes all tests!
Now that we covered some basics of testing, the rest of the implementation will be left to the viewer. The complete project can be found on the repo provided at the end of the article. I’ll add all the test cases I tried so you can try them by yourself before taking a look at the repo:
This is how your output should look like after completing all the tests
No that we have our tests running on the native platform, it’s time to run them on target board.
3. TDD on target
For testing on target, we need a way to collect the results into the host machine. In my case, I set up the UART2 since in my nucleo board it’s already connected to the debug channel. My host laptop recognizes said channel as a COM port.
For Unity to recognize our tests, we need to create and populate these macros in the unity_config.c/.h files:
UNITY_OUTPUT_START() Should initialize the selected comms channel
UNITY_OUTPUT_CHAR(c) Should transmit one character through the comms channel
UNITY_OUTPUT_FLUSH haven’t implemented, don’t really need it
UNITY_OUTPUT_COMPLETE Should end the comms channel
Again, the full implementation can be visualized in the references repo.
There is a slight difference in our code compared with the tests on native environment: we need to trigger our board to start receiving the data. Since my Nucleo_f401re doesn’t provide a soft reset functionality, we need to include a 2s delay for Unity to catch up. This is just a ballpark estimate, and it will depend on your board.
After implementing these changes, you should see this output. Don’t forget connecting your board!
If you do, congratulations! you just created your very first TDD project in a dedicated hardware board! For any problems or bugs, please refer to the repo link on the references page.
TL;DR
Shifting from a (de)bug-driven to a test-driven approach in embedded software development will lead to more efficient and predictable development processes, and higher job satisfaction among developers. While TDD does have its challenges, the benefits it offers make it a worthy consideration for any professional embedded software project.
In a future article, we’ll combine TDD with CI/CD techniques to create an automated testing and deployment pipeline that will greatly speed up our development process.
