the blind side of seniority

I was raised by a father who loved underdogs. Kelly's Heroes and the stories of the Black Sheep squadron, were some of his favorites to watch and tell. These were stories of outcasts, thrown together because of their inability to fit the mold required of them, who rose to heroic status as they accepted each other and capitalized on their individual strengths to build unbeatable teams.

Because of that, I have always looked at building teams as finding the right mix of strengths and weaknesses.

I suppose we could get esoteric here and consider the idea that we as humans are all incomplete, and only through unified efforts can we become whole. But let's leave that for another day.

Instead, let's talk about how each of us has weaknesses that leave us with unique blind sides that we are both aware of, and oblivious to.

A while back, I was talking with one of my team leads, a very experienced software engineer. He has architected large systems, worked on large systems, and worked on small ones. We got onto how having a lot of experience is both a blessing and a curse to software engineers, but also to all of us. As an experienced software engineer, you've run into lots of problems, so you can spot them quickly. This is what makes you valuable because you can guide decisions that avoid and accommodate those edge cases.

But it also creates a blind side, because sometimes you fixate on those edge cases and neglect the naive solution that would solve most of what you need to do in order to have impact. Because of your experience and all the things you have seen, it is hard to ignore the things that you learned the hard way.

Two Event Buses

We were designing an event bus at the time, and before you go "Ahhhh! Evented systems are hard to troubleshoot, and harrrrdddd! Why!?", we were in a space where most communications with third-party systems happened asynchronously. Add to that that many of the partners we integrated with were not particularly mature in their architectures (think blank stares when you mention webhook retry, and only allowing one webhook URL per event type), and you have a real case where an event bus can simplify things architecturally. You add some complexity to simplify the ecosystem's developer experience.

One of the ways we made sure it actually got out the door was by focusing on a single problem statement. It would only manage inbound webhooks and nothing else. Anything more, we'd evaluate after we solved that first problem.

That helped us avoid the edge cases we knew were there, because we were focused on a single problem space. We weren't trying to solve all of our asynchronous communication needs, internal and external. We didn't have to think about the edge cases for every use case for an event bus or how every message should be structured. We could say we were going to take in webhooks with these few payload structures, and our subscribers were going to respond to them in this specific way. That narrowed the focus enough to execute and deliver, instead of spinning our wheels on a perfect solution for every possible use of an event bus, most of which we might never need. We might only ever use it for this one thing. If so, great. It would still provide value.

We both knew what happens otherwise, because we'd lived it. He and I had worked together at a previous company, and we tried to ship an event bus there too. That one was going to solve every possible use case for an event bus across the whole company. We spent a lot of time on it. It never launched.

Fred Brooks has a name for a close cousin of this in The Mythical Man-Month: the second-system effect. The second time you build something, you pour in every idea and every safeguard you held back the first time, and the system sinks under the weight of all of it. Nobody on that team was short on experience. Experience is what sank it.

The Blessing

None of which makes experience the problem. Most of the time it's the whole point. I've argued before that mētis, the practical, local knowledge you earn by getting burned, is what separates senior engineers from junior ones, and that it's worth protecting because no metric can see it. I still believe that.

Everybody who has worked with database migrations knows the fundamentals. But some things you only learn at scale. Add a column with a default value to a small table and it's done in a few seconds. Do the same thing on a table with millions of rows, and on plenty of databases, older Postgres and MySQL among them, that change backfills every row while holding a lock. Now you're talking minutes. Even a brief pause like that can drag down the performance of the whole application, potentially taking the system down.

If you've never seen that happen at scale, you've probably never traced it back to the root cause. So you don't worry about it. Plenty of people who have been in the industry a long time add columns without a second thought.

If you have seen it, you know the moment you read the migration. "Cool, we're adding a column with a default value. That's going to backfill the entire table, so anything else that touches it while the migration is running is going to have performance problems." Whether that matters is something you evaluate before you decide what to do about it. But you can't evaluate a problem you don't know exists.

The fix is cheap once you know. Add the column with no default. Backfill the value you want in batches, a thousand or a hundred thousand rows at a time, during normal operation of the application. Then set the default. The expensive work moves out of the migration, and the performance problem all but disappears.

That's the blessing. You see the problem before it becomes one.

The Curse

The trouble is that mētis doesn't come with an off switch. Scar tissue is a reflex, and reflexes fire whether or not the situation calls for them. The same reflex that warns about the migration lock also warns about every way an event bus has ever gone wrong, in a system that only needed to route some webhooks. Each of those problems is real. Each one was learned the hard way. And each one pulls the design a little further from shipping.

Psychologists have been documenting this since at least 1942, when Abraham Luchins gave people a series of water jar puzzles that could all be solved with the same multi-step formula. Once they learned it, most kept using it on later puzzles that had a much simpler solution. When only the simpler solution could solve the puzzle many failed entirely. He called it the Einstellung effect: once you know a solution, the knowing blocks you from seeing a better one.

In 2008 Merim Bilalić, Peter McLeod, and Fernand Gobet tested it on chess players. When a board contained a familiar winning pattern, strong players found it and missed a shorter win that was also present. Only the very strongest players were able to shake it off. In a follow-up, eye tracking showed that while players said they were searching for something better, their eyes kept going back to the squares of the solution they already had. ¹

We've all experienced something like that at some point in our lives.

I got a dose of this while writing this piece. In the original first draft, the migration example said that adding a column with a default backfills the table on every database. I was sure of it, because it had many databases that I've worked on. The AI editor I was working with flagged it. I pushed back and asked for my facts to be checked, I didn't believe it or trust it. I was wrong though. Postgres has handled a constant default without rewriting the table since version 11, and MySQL since 8.0.12.

My scar tissue was real, but it was out of date. And I would never have gone looking, because experience doesn't tell you when it is no longer relevant. It took something without my history to ask whether it was still true.

Bring the Naive Ones

Which brings me back to the Black Sheep. A team made only of people who have been burned will spend its time building firebreaks. A team of people who have already seen all the skeletons in the closet will forget that they can throw out the skeletons.

You can't think your way out of a blind side, because you can't see it. You need someone whose eyes aren't trained to snap back to the familiar solution. That's what sharp junior and mid-level engineers bring. They have very little scar tissue, and they're free to see the simple solution. In fact that may be the only solution they can see. They can drive the team into the solutions our experience has trained us to skip, and force us to update our scar tissue.

This can only work when we give their voices the weight they need,despite their lack of experience. Even though they'll miss things that senior people won't. If the most experienced person in the room always wins, you've built a team with one point of view and a lot of headcount.

And if you're the senior one, you have to manage your the blind sides. Yes, your experience is a big part of why you're valuable. Knowing when it's steering instead of you, is a big part of being good at building software.

Go find the naive ones on your team and give them the floor, they'll learn and grow, and so will you.