Paralysis by Analysis in Engineering (and in life)

I'd like to start this blog by saying that sometimes it is difficult to choose one option. However, in my case, it's always challenging to make a decision when we have a plethora of options. Everyone wants to select the best one, which requires thorough investigation of a good sample of alternatives. As engineers, though, we always face deadlines, so there isn’t enough time to explore the universe of options. So, what do we do?

First, I make a list of my options. Let's say you have to choose a tool to implement an orchestration; I try to build a rubric with those options, adding as columns the features I like from each tool. After that, I add check marks where tools have the feature. Then I just count; sometimes I do a weighted average, especially when there are features more important than others. Another important thing I do is read blogs about the tools and watch some YouTube videos. I like to know beforehand the opinions of others who have used the tool. Most of the time, I have a preferred tool after finishing the aforementioned steps. But even with that hunch, I always schedule some time to do a Proof of Concept. I need to get my hands dirty with the tool before telling my higher-ups that we are ready to go.

In my decision rubric, I include:

• Documentation
• Cadence of updates
• Support teams or community
• Expenses
• Integration with my current stack
• Learning curve

Being paralyzed is not an option in engineering; at the end, we have to make a choice. However, in normal life, it's tougher; we can choose to do nothing, and that's in most cases the worst outcome. So, in my case, I have opted to include my 'Do Nothing' option within the rubric. Seeing numbers helps my brain to not get stuck and move on. "Algorithms to Live By" is a great book to learn more about this topic in real life.

My next blog will be about my selection of Dagster over Airflow as the orchestration tool in a migration project. Stay tuned! ✌️