SMACNA Contractor Spotlight: Hermanson

SMACNA Contractor Spotlight: Hermanson from SMACNA on Vimeo.

Customer Testimonial - Walker White Mechanical

https://vimeo.com/mestekmachinery/review/86334045/e8bdfa042e

Branching out to Extend Your Technical Perspective

What's in your domain?

I appreciate how European developer conferences like DevTernity help me to gain a wider range of technical perspectives than I have available closer to home. (I live in Eastern Iowa, USA)

This conference had over 700 attendees from over 40 countries with my favorite speaker being J. B. Rainsberger with his "The Well-Balanced Programmer" presentation.

I was also glad I attended his "Surviving Legacy Code" workshop yesterday which he ended with an introduction to his "Universal Architecture" version of clean architecture and the value of having it.

This was my second European developer conference and I'm still impressed. The first was Domain-Driven Design Europe which was packed with people who take code quality seriously.

In case you're wondering, all the presentations at each of the conferences were done in English.

If you have a passion for software and the opportunity, I'd encourage you to try one of these. Getting outside your normal circle helps you to see other ways of addressing the challenges we all face in writing and maintaining software that can meet the needs of those who depend on us daily.

One of these is a great opportunity to combine it with a European vacation too. My wife sat this one out but we had a nice visit to Amsterdam together with an brief visit to London on our DDD EU trip.

Why We Support Wills Dady Homeless Services

How I Work

I've recently been doing a lot of factory tours and explaining what I'm looking for in hiring somebody to join me as a team.

In explaining my work, I'm finding that it helps me process some of the software development goals I have.

Knowing that the visitors have little familiarity with the type of work I do helps me stay at a more abstract and easier to understand level. I find that reflecting on recent tours helps me reason about my own work through fresh eyes.

A recent twitter musing from GeePaw Hill got my attention:
https://twitter.com/GeePawHill/status/1198354786049019905?s=20

This was especially timely as I'm preparing to head out this afternoon to help a customer with a machine being controlled by another programmer. I'm not familiar with how he solves problems and I'm not familiar with this older version of machinery. But in preparing, I have to map out a plan of attack as I'd rather not have this turn into a holiday ruining trip.

This is also a good example of how I take on a new machine control project. My process is similar for both.

In the case of helping to troubleshoot a machine being controlled by unfamiliar code, it's best to have that code available. There are two parts to most machine controls. There is the part that connects to the machine through sensors, drives, and actuators and there is the part that humans can interact with (Human to Machine Interface or HMI). The HMI can be screens, keyboards, mice, operator control buttons and switches. These are ways for the operator to tell the program what they are trying to do and a way for the program to communicate back to the operator with screens or lighted buttons.

When reasoning about a new machine control problem set, I tend to go outside in. All I know for sure is that things need to be told to move and moving things need to be sensed or measured. I can see these things and wires can be traced (hopefully they are labeled). I tend to pick one area of the machine that I find interesting. Sometimes it's the simplest thing and sometimes it's the most critical operation but I pick one.

I then look to the physical outputs. That thing can't do what it does without being connected to something. Once I find out what that something is, I can move on to identify wire numbers or relays or I/O (input/output) modules that are involved. I then find these on the the wiring diagram (hopefully there is one). I'm looking for some nomenclature that identifies what this thing is and how the things it connects to are known to the system.

Is this a "shear", a "clamp", a "beam", a "pusher", or something else? If it connects to a relay, what else connects to that relay? I follow the wires until they get to something the computer can control directly. I then focus on the labels of that device. I look at any indicator lights and see how those are labeled. I look at the brand of hardware and model numbers. Anything I can search for to find more information.

If there is a computer screen, I will look at that to see what names were provided for these things. If there is an I/O monitoring screen, I will pay special attention to that and focus on the hardware that interested me.

If the machine is operational, I'll want to see it run. Even if it's not working correctly, I'd like to see how it does work if that's possible. I'll want to have a sense for sequences of operations. I want to know the order in which things happen.

With all of that noted and in mind, I can take a look at the code. I have to first make sure that I can preserve the initial state of the code. This particular project wasn't under version control. I fixed that first. I need to be able to always see how it looked before I touched it. These days I use git through posh-git, and GitKraken.

The next thing I do is explore evidence of IDE (Integrated Development Environment) version numbers. I was familiar with both of the IDE's being used in this project so that helped. I could sniff around a few files to find version numbers. Once was Visual Studio 2013. I'm not reloading that unless I have to. I don't have it on a virtual machine and I don't want to take the time. I'm reasonably confident that my Visual Studio 2019 will work just fine with it.

The other project is with B&R Automation Studio. I'm not as confident with that and because of that I have virtual machines for several major versions. I'm on 4.5 and this project was done with 4.2. I'll start with opening that in 4.2 through the virtual machine. I don't want to introduce upgrade problems while I'm exploring.

Now that:

• I know what I'm focused on (that interesting thing and the devices that it connects to the computer through)  
• I have a way to get back to square one  
• I have a plan for how to explore these things

I can begin to crack open the code and look for things that look like they might have something to do with what I'm interested in.

Again, I will start with the outside in. I start with the PLC (Programmable Logic Controller) project and look for ways to identify the I/O modules (or drives) that correspond to my notes. Once I find those, I can look for the way variables are mapped to those hardware devices. Some programmers bring the hardware addresses all the way into the program. Some map a variable to the device. I prefer separation and so I map variables. I won't know what this other programmer prefers until I get to that point.

The HMI code might be a mix between the PLC project and the Visual Studio Project. The fact that both of these are involved (a fact that I was aware of already... which helps) tells me that there must be some sort of communication between the screens and the PLC. The PLC will know about operator controls (physical buttons, switches, and lights).

I happen to know that this developer is using OPC (Open Platform Communications). This gives me a clue as to where to find the correlation between the two systems. The PLC will assign variables to the OPC system and the PC will do the same. There will be a common point of communication for each variable. In this type of system, outside in is in two contexts as there are two computers. The PLC has an outside edge that is shared by the PC. That touching point is OPC in this case.

Once I find all of these outside points where the software comes into contact with the human or the hardware, I can begin to look for code that use these variables or hardware addresses in logic or code that assigns value to them. This can generally be done with IDE search tools in order to get a sense for how prolific these points of interest are. If they are too common and return too many results, I may lose interest and focus on something more unique about this machine (if that applies). I don't want to wade through too many false positives if I can avoid it.

Having identified what the computer can "see" (sense from machinery or operator) and what the computer can "do" (cause motion or change indicators/screen information), I can then begin to read code in the context of those points of interest. I tend to start without outputs over inputs. I'm not sure why. I guess in my mind the outputs are what the machinery is an extension of. I want to know what logic leads to the something moving.

That feels like enough for now. I'll leave this at that. I might get back to this later if there is any interest in it.

Put My Shingle Out

As it turns out, Steve, the guy who came out to install my sign, was also the one who installed the original sign which was for Iowa Precision Industries (fka Welty Way).

He knew my dad and much about my family history back in the day. Super nice guy and we both enjoyed chatting and getting to know a bit about each other. 

Interview with Maplesoft

John Welty, Owner of Welty Automation and Product System Support of Mestek Machinery Inc., discusses his experience with MapleSim.

Machine Control Programming Adds Layer of Puzzle Solving to Software Development

I like that machine control programming has an added layer or two of puzzle solving when diagnosing machine behavior. The operator can't see my code executing. They see the machine behaving. My program watches inputs and controls outputs. The actual behavior has electrical and mechanical dependencies.

When a machine is not working properly, it can sometimes be difficult to know who to assign to troubleshooting. I spent the better part of a day trying to remotely help a customer diagnose a problem he was having. I could tell there was a bad signal coming from a part sensor but he couldn't find a reason why. He had replaced the sensor and checked every wire connection.

I flew out that week to help them in person. I quickly proved that it was a signal problem which was causing unexpected behavior. That behavior changed the speed at which a third party piece of equipment was running which made it difficult to troubleshoot. While my software wasn't causing the problem. My program wasn't helping to diagnose the software.

I altered my program to be more forgiving of the signal quality we were seeing. As soon as we did that we were able to focus troubleshooting on that 3rd party machine. This led to the discovery of some bad components (relay, potentiometer) and ultimately to find that their wiring harness had been rubbed raw and wires were shorting out.

Trips like this teach me that I need to be more mindful of edge cases when writing control software and to provide better options for remote troubleshooting. I was glad to be able to restore some control to the situation which proved helpful to their very capable maintenance person.

Great experience using TDD this week

I had a great experience with using TDD (Test Driven Development) to implement a new feature this week while at a customer site. I started by going into the location the code would need to execute. I did this to gather available information that would be needed for the feature.

I felt like what I was adding was easy enough in my mind and started, by habit, writing in a solution in spot. I stopped and thought back on an experience the week before where I was able to use a new function to improve old code. I reset my file (using git) and set up a new test environment instead.

In this development environment, I write my code in Structured Text (IEC 61131-3 standard) and my tests have to be written in c. So I started with a test that couldn't compile and then added the function it was calling without any behavior. I did only what I needed to get to green until I had all of the tests in my inbox (as J.B. Rainsberger teaches in his World's Best Intro to TDD Course: https://online-training.jbrains.ca/p/wbitdd-01)

I was able to write it test first and do final integration tests by hand in simulation before every putting the change on their machine.

I installed and ran it with confidence while the customer watched. I had no doubts when we ran off the first part to his specifications.

In the past I would've had something ready to test sooner but would've had to debug it in front of the customer. I would've ended up adding to the section of code with a procedural solution instead of having a few useful functions .

It felt slower up front but the confidence it gave me when I was ready to install it on the machine was fantastic. I suspect it would've taken at least as long if I'd done it on the machine but it would've taken time away from the operator being able to run the machine while I was working it out. So I'd say this was a force multiplier as it gives the customer confidence to see that we can make a change confidently.

Focusing experience on a customer service call

I shared this with some colleagues who found it inspiring and I thought it might also be generally interesting to others:

---

On a recent trip to Dallas, I arrived at the customer’s business on a Monday morning. They’d been down in production for more than a day the week before. As I walked in, everyone in the shop looked relieved; their shoulders relaxing. They were a smaller shop that depended on their processor for making blanks to run their plasma tables and to make rectangular duct. So, the entire shop was out of commission while that necessary machine was down and essentially inoperable.

I quickly connected with my laptop and let them run the machine for a couple of hours off that. They were elated to learn that this was an available service option. Within just a couple of hours they’d built up enough blanks and ran enough duct to get the rest of the shop back up and running and profitable again.

Those were hardworking people. They were on the floor busting their butts every day in a hot dirty duct shop suffering the elements and rewarded with reduced hours. Not a ballistic owner present voicing concern over profit margins, but real people who share in those ventures. In fact, the operator’s wife was pregnant and actually went into labor towards the end of my trip and he couldn’t make it in that last day. The magnitude of lost work falls to those that are the most vulnerable; the family.

So, on this trip I gained a better perspective of just how important our machinery, controls, and software are, both personally and professionally. I saw a dozen lives with families and mortgages and car payments tightly tied to our ability to provide quality equipment and expedient service.

I used this as a wake-up call for myself and I put together documentation while waiting for my flight back. I wanted to make sure that no one ever has to be affected like this again because of a lack of insight. A point in fact, this document proved immediately helpful with another customer having a similar problem but then being able to function from their own laptop thanks to that document.

I’m sharing these notes to encourage each and every one of us to keep the thousands of lives that depend on us each day in mind. It’s more than just those of us here at the factory and our lives and families. We have many hundreds of installations which multiply that demand on us to continually look for better ways to communicate to one another and make sure there are no roadblocks to continuously improving our products, services, and customer experience.

Best Regards,

John Welty

Joined Toastmasters yesterday

Looking forward to working with the Marion Sunrise group in Iowa. 

I'm looking to improve my communication skills and ultimately plan to do some technical presentations while also working on my leadership skills. 

https://sunrisemarion.org/

SQL Saturday

I had a great day at

SQL Saturday yesterday in Iowa City, Iowa.

This was hosted by the I-380 PASS SQL Server User Group and they always do a fantastic job.

There are SQL Saturday events all over the world and if you haven't been to one and have an interest in all things related to SQL Server and Azure Data Services, you really need to look into these free learning events. The volunteers do a great job and I dipped my toe in the volunteer water by being a room monitor for one of my sessions on CosmosDB.

The icing on the cake was winning an Ember Temperature Controlled Ceramic Mug in the raffle at the end of the day.

Getting Started with Getting Things Done

I came late to GTD and was introduced to it by J.B. Rainsberger.

GTD (Getting Things Done) is a system from David Allen published in his book, "Getting Things Done"

I began slowly with the 2-Minute Rule and started using the FollowUpThen.com service. These two things alone proved to provide immediate help with dealing with my e-mail inbox and staying on top of commitments.

The 2-Minute rule goes like this, "When work arrives at your door, if you can’t avoid it and you can do it in two minutes or less, then do it now; otherwise, schedule it for later."

So the first part is easy... if I can do it in 2 minutes, knock it out and be done with it. But the second part is where I needed better tooling. I've always found it difficult to keep track of things I need to do later. I can focus on one thing for days on end and forget everything else that needs my attention. This was a problem I knew I had but not one that I was trying to solve. I was too busy trying to cope with the problem. It was consuming me.

Enter FollowUpThen.com (or fut.io for short).

I learned about this tool after receiving a post from J.B. called, "Change Your Life One Habit at a Time". In this article, he explains how he used this free service to help him establish a new habit into his routine. I found this to be quite intriguing and signed up for the service.

I found it very useful for exactly the situation J.B. brings out in his article but as I explored the use of this simple tool, I saw it had far reaching benefits for me. When I found something in my inbox that I couldn't deal with in 2 minutes, I would forward it to tomorrow@followupthen.com and sure enough, I would get that email back to me the next day with clickable links to postpone it if necessary. If that reminder was timely, I would take care of it and that would be the end of it.

FollowUpThen gives you the option to use many version of the email address to get nearly anything you want. I can forward to 3pm@fut.io for instance and get that email back at 3pm either today or tomorrow depending on what time I send it.

I can also add one of these in my BCC field on an email. Let's say you need my help and send me an email. I reply, "Thanks for the email. I'll get back to you later today on this." I don't want to forget that I told you that so I add a BCC to 2hours@fut.io and BAM... two hours later... after I'd long forgotten that I'd promised to get back to you, I have a reminder.

This has removed a lot from the back of my mind which would nag me at night and it has also made me more reliable with people I make commitments too. This builds trust and confidence. I can't stress the importance of this enough.

If you want to remind both yourself and the person you are sending the email to, you can put it in the CC field instead. So if I send you an email saying, "Let's talk about this tomorrow at around 3", I can add a CC address for tomorrow2pm@fut.io and we'll both get a reminder in our inboxes an hour before that time.

So the next challenge was J.B.'s post on "Get Stuff Out of Your Head: Now!" This was something I'd seen him demonstrate in his "Worlds Best Intro to TDD" course. I experimented with best ways to accomplish this and have always had a problem with pens and paper. Keeping them handy was always a problem for me and the more I come into contact with paper the less it likes to work when I want to write on it. This has always been an issue for me and must have something to do with natural oils in my skin or the type of paper that I tend to have... maybe the pen. I've tried many things and have always been frustrated by that.

The solution I came up with was the Remarkable Paper eInk Tablet. While this was pricey, it proved to be easy to keep with me, quick to use, never ran out of paper, and I could use it to get things out of my head quickly so that I could review and process those notes (later to become one of my GTD inboxes) later.

Any and all of these things could prove helpful to you if you are wrestling with any of the same issues. When I have some time I'll write up my experience with the David Allen book, "Getting Things Done"

This was when these first few helpful steps were kicked into high gear and started freeing me up to think about what else I could be doing better.

Stay tuned!

My notes on J.B. Rainsberger's "Practical Tools for Playing Well with Others"

Presented by J.B. Rainsberger at www.oredev.org November 7th 2013

https://vimeo.com/78917211

Conversations begin with Intake - See, hear, read

This can go wrong. We can hear wrong, misinterpret, misread.

This is an area we tend to assume isn't a problem but it often is. The gorilla test proves that. While we are focused on one thing we miss other things.

After Intake, conversations progress with Meaning - We decode the signals/symbols.

We have issues with meaning all the time. Same words can mean multiple things to different people and in different contexts.

This leads to arguing with people you agree with.

It's important to clarify meaning.

"Normally when somebody says xxx, they mean this, this, and that…, What did you mean when you said xxx just now?"

We need to establish how close we are to the meaning of the words, identify potential differences and see what we can do about it.

After Meaning we have Significance - Why did this just happen?

Significance has to do with the deeper meaning. The meaning behind the meaning.

Why is the other person behaving this way. Figuring out how to interpret the message.

The last part of the model is Response - After all the first three happen, we have to figure out how to respond to that.

Checking on perception before responding can help. Check intake.

Checking on misunderstanding before responding can help. Check Meaning.

Checking on misinterpretation before responding can help. Check Significance.

What did we see or hear that leads us to conclude our choice of response?

The path between Significance and response is REALLY hard to control.

It's best to assume that this path for others is hardwired and isn't going to change.

Timestamp: 24:50

One of the ways that you can improve your interactions with people is by assuming that if somebody responded in an inappropriate, unexpected, threatening, unusual way; that that reaction was perfectly sensible, reasonable, and defensible given how they interpreted, understood, and perceived the situation.

Being hardwired, there is nothing we can do about it and there is nothing we ought to do about it. Let's fix the other areas instead.

Misperception is a difference of perception. Misperception is a wrong assumption that our way of perceiving things is the only right way of perceiving them. When there is a difference, it's best not to make a value judgment.

Same with misunderstanding. This is a difference of understanding.

Same with misinterpretation. This is a difference of interpretation.

Because these paths are so quick, at first all we can do with this model is debug a conversation after it happened. But that's a start. It can help to work from the bottom up.

How can we figure out why they interpreted it the way they did? Or understood it the way they did… or what they may have perceived that wasn't intended.

Timestamp: 31:25

It gets even more fun when you are so practiced at this that you can use this to pre-bug a conversation. TDD for conversations.

How might one interpret what I'm about to say?

Maybe I better choose different words.

How well do I know whether this persons understands the word I'm about to use?

Maybe I should try a different word.

Am I speaking clearly enough?

Maybe I'd better speak up or check my phone connection.

Timestamp: 32:40

Things really get interesting as we consider how your response leads to their intake and their response leads to your intake. This can spiral in terrifying ways.

https://www.infoq.com/articles/satir-communication-model-teams/

Timestamp: 41:00

A Few Helpful Tricks

Think of three ways to interpret what just happened… before you respond if you can.

Ask, "What did you intend by that [surprising comment, question, action…]?"

Communicate in E-Prime to reduce judgmental perception. (https://en.wikipedia.org/wiki/E-Prime)

Warn the other person when you need to say something uncomfortable.

Further reading:

Jerry Weinberg, "The Secrets of Consulting"

Mark Goulston, "Just Listen"

Stone, Patton, Heen, "Difficult Conversations"

Slide Deck: http://link.jbrains.ca/ptfpwwo-oredev-2013

Interested in Exploring Software Development?

Glad to hear about your interest.

Here’s a link to some free training courses on Pluralsight:
https://learn.pluralsight.com/resource/free-course/free-weekly-course

This is a good resource too:
https://developer.microsoft.com/en-us/

This one from Microsoft Virtual Academy is free (and perhaps a bit dry): https://mva.microsoft.com/en-us/training-courses/c-fundamentals-for-absolute-beginners-16169

Udemy.com is a good resource I’m told. Here’s a link to their Development section: https://www.udemy.com/courses/development/

My son vouches for Mosh Hamedani's C# for Beginners course: https://www.udemy.com/user/moshfeghhamedani/
He also operates his own site: https://programmingwithmosh.com/

I tend to use Pluralsight.com which comes with a free month to explore it. Scott Allen has authored great courses on Microsoft stuff in Pluralsight. You could search for him on that site and find plenty. He’s got a good teaching style. He’s got a blog site here: https://odetocode.com/

I would also encourage you to become familiar with using git (Version Control Software).

A great tutorial on git can be found here: http://gitimmersion.com/ (It talks about using the Ruby programming language but you don’t need to install Ruby or be familiar with it… they just use Ruby source code files as examples).

There is an excellent free eBook on git too: https://git-scm.com/book/en/v2

Git is something you’ll want to use for a lot of things… it’s quite a powerful tool and I’m finding more and more value in it week after week of using it. Quite impressive.

I like to use Git Extensions for Windows: https://gitextensions.github.io/

I believe that will walk you through everything you need but it will depend on git and something like kdiff being installed. I just use their defaults and follow their installation recommendations.

If you run into problems with any of these let me know. I'm happy to help.

Exploring Virtual Commissioning (AKA Digital Twins)

We're seeing tremendous potential value in Virtual Commissioning.

In the past, I was able to improve my control software with a focus on the handling of part passing through the line from coil to finished product. I was able to do this by writing up my own 2D simulator from scratch which we showcased at the Manufacturing Day we hosted at Mestek Machinery in 2012.

Showing this to our state senator (Charles Grassley)

Video we played to show the visitors 

on tour about some of the things 

we were using technology for

This project was a big success and greatly reduced problems we were seeing in the field for those who we upgraded with the changes.

I wanted to take that to the next level to enable better tuning and refinement of my control software at the machine level (instead of just at the line level).

I began generating a virtual 3D ductline within the Unity 3D game engine which is used in many game development environments. I replicated my 2D line simulation within Unity (using cubes as stand-ins for detailed models) and drove it by writing a networked interface between the game and the inputs/outputs from my actual machine control program running on actual hardware (in my office). This allowed us to interact in a very natural way with the parts on the line while seeing how the real-world program responded.

This showed a lot of promise but at that time we had little of the product line modeled in solids.

Times have changed since then and I was ready to re-visit these efforts when I found out about a fully integrated solution being provided by our hardware vendor. This takes things to a whole new level and promises huge payback once implemented. This enables the virtual commissioning of machinery and automation control software through something being called a "Digital Twin". Exciting times!

Brief History with Video Examples

Welty Automation contracts full time with Mestek, Inc. and works out of the engineering department in Cedar Rapids, Iowa factory location of Mestek Machinery. 

I grew up around automated sheet metal and coil processing equipment design and manufacturing. I first helped my father on a design project at around 16 years old; by programming a PLC to control a simple robot designed for use on a machine that produces downspout elbows. 

The robot took the elbows out of the machine, crimped the end, and drop the crimped elbow into a box positioned next to the machine to reduce operator repetitive motion strain.

This is a video of the newer version of that machine and robot which improved on my original design:

I started working for Iowa Precision back in 1987 as a drafter after my father's adjustable elbow machine company was acquired by them. I helped usher in the use of CAD shortly after that and later wrote the software to replace their mainframe based business management software. 

Sometime around 2007 I was asked if I could take over the control software development for a new machine which I was able to name: The DieMation

This was the project that launched my career in automated control software:

I was then asked to get involved with the controls on their flagship productline and have been writing the control software for new shipments of that product line since 2008. 

Here are some sample videos of projects related to that product line:

Customer Testimonial for Pro-Fabriduct line with partial width inline plasma table:

Residential Ductline:

Compact Ductmaker:

Working remote

One of the challenges when being out on the road at customer installations is finding a good place to work. This is my desk at times which works pretty good. 

These bad boys pay the bills

--both the machine and the laptop--

Operator Training Videos in Production!

We're currently working to create video training content for users of new ductlines. This is an exciting project I've been trying to get traction on for quite some time. I went out to meet a colleague who does video production at a graciously hospitable customer site to get screen recordings and video footage to put together with audio.

My goal is to target short video segments to cover targeted areas of focus and key aspects of the control software that I always ensure new operators understand. These videos will ideally be easy to access, easy to digest, and short enough that people won't feel that they have trouble finding the spot they want to refresh themselves on.

A first draft video showing a technique for aligning fiberglass insulation in an Insulmatic:

Welcome to my blog!