Is this person helping the team ship valuable software to users? If they spent a week reducing the program's memory usage by 10% and users don't care at all, then the answer is "no." —Internet comment

Before embarking on the strange software path I'm on now, I was a backend developer at a B2B company. I was primarily responsible for the core Rails application, infrastructure, and things like that. I did some very useful things and some stupid things, and I know I wasn't the best team player. To this day I'm still good friends with a few people who still work there. I've been thinking about one of my past projects all week. Because it was so long ago, my memory of the project is incomplete, and I've adjusted some details for the sake of narrative, so it's not 100% accurate.

One day in 2016, I received an urgent task: our data scientist's database queries were unresponsive, and I needed to investigate. I restarted the "data warehouse" (an AWS server running Postgres), tweaked some parameters, and asked the data scientist to re-run the query.

"Did it time out?"

"It's still running, but we don't know yet."

"When will you know?"

"Tomorrow."

The average query took an entire day!

Nobody thought there was anything wrong with this. That's just how things were. But it deeply bothered me. We only had a few terabytes of data—should it really take this long? I decided to investigate and see if I could make it faster.

(One of my primary motivations at the time was "intrinsic joy." I wasn't very interested in serving end customers—I was more interested in serving internal people, like data scientists.)

Eventually, I concluded that the problem was our infrastructure. We shouldn't be running on a single Postgres machine—we should have a real data warehouse! Since we were deployed on AWS, I chose to switch to Redshift. I designed the migration plan and architecture so that every query would be much faster! Then I presented this proposal to management.

"Does this help students? Teachers? Administrators? Company revenue?"

"No, it would just make things easier for the data scientists. I think."

The CTO pulled me aside and explained one of the most important business rules: Do things that deliver value to customers. If end users don't care, don't do it.

But I was an arrogant young programmer at the time, so I decided to push forward anyway. I used all my spare time to improve the existing architecture. I dove deep into AWS infrastructure, learned how to organize Redshift tables and use schema features. I taught myself how to optimize queries using window functions and CTEs.

Midway through, another colleague took a look at the timeout problem and solved it in an hour. The Postgres database was on an RDS instance. He simply upgraded the RDS instance's performance and disk space. Problem solved. The data scientists were happy again.

But I wasn't happy, so I kept working. I built all the infrastructure and migrated the event logs over to it. I spent days tuning the database, setting sort keys, and running tests. A month or two after starting, I proudly deployed it to production. Through testing and countless hours of sweat, I achieved over a 1000x speedup! Queries that used to take a full day now completed in under two minutes.

In most cases, this story should end as a cautionary tale about how naive I was. But not this time. Everyone loved it. Data scientists could run iterative queries at any time. Sales staff could run ad-hoc queries and get useful results. Engineers found the new system more stable and easier to modify. And using a proper data warehouse saved us $60,000 per year. The accountants certainly loved that. Until they actually used it, nobody knew they wanted a real data warehouse.

That's right—customers know most of their needs better than we do. But they don't know all of their needs better than we do. They are domain experts, but so are we. They may not realize how much happier they would be if our product were faster, more secure, and more stable. They can't see how the internal work we do has downstream effects, making it easier for us to help them better. Because they aren't as well-versed in software engineering as we are, they don't know what we could do to make things better. They may not even realize things could be better until we've already improved them.

This isn't to say that all or most of our ideas should be acted upon. We're still not the customer, and we shouldn't assume we know their needs better than they do. Yet this project was a stunning success. There were other similar success stories too, which convinced me that people don't always know what they truly need. At the same time, I have plenty of failures—projects I wish I'd done better, changes that turned out to be wrong. In short, it's not easy! It's like a pendulum. We swing between two extremes—first thinking we know better than the customer, then thinking they know better than us. Neither we nor the customer can fully understand the true needs, but we can iterate together and validate them—what the customer wants is not always what the customer needs.