Saturday, August 31, 2019
When a machine is not working properly, it can sometimes be difficult to know who to assign to troubleshooting. I spent the better part of a day trying to remotely help a customer diagnose a problem he was having. I could tell there was a bad signal coming from a part sensor but he couldn't find a reason why. He had replaced the sensor and checked every wire connection.
I flew out that week to help them in person. I quickly proved that it was a signal problem which was causing unexpected behavior. That behavior changed the speed at which a third party piece of equipment was running which made it difficult to troubleshoot. While my software wasn't causing the problem. My program wasn't helping to diagnose the software.
I altered my program to be more forgiving of the signal quality we were seeing. As soon as we did that we were able to focus troubleshooting on that 3rd party machine. This led to the discovery of some bad components (relay, potentiometer) and ultimately to find that their wiring harness had been rubbed raw and wires were shorting out.
Trips like this teach me that I need to be more mindful of edge cases when writing control software and to provide better options for remote troubleshooting. I was glad to be able to restore some control to the situation which proved helpful to their very capable maintenance person.
I felt like what I was adding was easy enough in my mind and started, by habit, writing in a solution in spot. I stopped and thought back on an experience the week before where I was able to use a new function to improve old code. I reset my file (using git) and set up a new test environment instead.
In this development environment, I write my code in Structured Text (IEC 61131-3 standard) and my tests have to be written in c. So I started with a test that couldn't compile and then added the function it was calling without any behavior. I did only what I needed to get to green until I had all of the tests in my inbox (as J.B. Rainsberger teaches in his World's Best Intro to TDD Course: https://online-training.jbrains.ca/p/wbitdd-01)
I was able to write it test first and do final integration tests by hand in simulation before every putting the change on their machine.
I installed and ran it with confidence while the customer watched. I had no doubts when we ran off the first part to his specifications.
In the past I would've had something ready to test sooner but would've had to debug it in front of the customer. I would've ended up adding to the section of code with a procedural solution instead of having a few useful functions .
It felt slower up front but the confidence it gave me when I was ready to install it on the machine was fantastic. I suspect it would've taken at least as long if I'd done it on the machine but it would've taken time away from the operator being able to run the machine while I was working it out. So I'd say this was a force multiplier as it gives the customer confidence to see that we can make a change confidently.