As always, reality is smashing my face, and I have to adapt to it instead of sticking to my plans. When I was thinking about this post I was trying to organize it like the post before (The Nature of Software Development review): my sensations, what I got from it, and what other people can get. But after reflecting I’ve concluded that it was silly.

Why? Because my knowledge about these topics is so different. While on the software side I feel strong and confident, on product development I feel lost. I don’t have references, I have not read more than articles, and I don’t even know what I don’t know. After reading my first book about this subject, I think it is normal that my list of what I’ve learned is quite long. Accordingly, I believe that I’m not able to advise others about what they can get out of it.

In short, this will be a list of what I have learned.

The context

The context almost always determines us, it is indisputable. Some books like “The Art of War” or the “The Little Prince” bring us different knowledge depending on our age or circumstances when we read it. I will write down some concepts because they are new for me, not because they are disruptive nor essential. Or maybe I will do it because they resonate with me because of our failures or lessons learnt.

To understand my takeaways, I will try to summarize some of the keys (in my opinion) of my context: As I said, the first circumstance is that I’m new to the product development area. So, except ideas that derive from or converge with Agile, the rest was unknown. The second is the feeling that in the Discovery team something was wrong. I couldn’t put my finger on what, but I felt that many things prevented us from succeeding. Maybe the first was that I was unsure what success was, but this is another story. The third is that after a couple of months being familiar with the product area, I started to disagree about how we were validating our progress in some cases. And the last is that I disagreed with how we listened to our users and how we were getting feedback. Regardless, if I was right about my sensations, it was how I felt about what was happening.

The big ideas

Although I have learnt many things, there are a few core ideas that marked me.

Without a clear hypothesis it is not possible to validate

The first time that I read this assertion, I was shocked. I thought it is not true, I may not have any hypothesis but see how revenue increases or my user base improves. But then, the question of The Nature Of Software Development came to my mind. What is value?. And then it hit me. One of the most important things you can get in an uncertain scenario is clarity and knowledge.

Also, I have to say that gaining clarity or knowledge is not an excuse for the flawed execution of a plan. Sometimes people use the sentence: “Well at least we learnt something”, I beg to differ. How will we know that we learnt something if we don’t know what we are searching for? It is not easy to learn from uncertainty and we can easily fall into fallacies or wrong conclusions.

Even if we succeed, to improve without understanding the whys, is like playing roulette. If we’re not aware of the context, assumptions, and actions regarding them, when we try to replicate the successes, we probably won’t be able to because maybe the context has changed or the assumptions now are wrong. It’s essential to be explicit about the premises, what we will try to achieve with the action and how we will measure it.

And no, I’m not saying that ideas could not come from beliefs or hunches. I believe there are people with amazing intuition to know what people may need or want. But these assumptions must be validated. And the best way to validate something is by using the scientific method: observing, formulating the hypothesis, testing it and then through the data drawing conclusions. Anything else is just throwing the dice and being a shaman.

Vanity metrics and success theatre

Ok, after accepting the point before, and tacking on that we need metrics, can we measure anything? The answer is nope. The book talks about how dangerous it is to measure the wrong things. It points out that the metrics used in an established company are not useful for a startup; for example, the total number of users. Suppose we see the total number of users is growing week after week. And this is done through marketing or growth hacking strategies. But, what happens if our product is not cool enough or useful to make them come back and the retention is extremely low? Someday we will run out of users to engage.

This situation is hazardous because it leads to a leaky bucket scenario. To be in this situation is not good, but it’s worse not to know it. Starting from the premise that we are working on innovation and don’t know many things, only being aware of it is a good step. We can work on our retention or pivot. The sooner we know it, the sooner we will do something.

But this is not easy at all. When people see the user base growing, and everything seems to be going well, who’s the jinx that raises their hand and suggests that maybe this metric is not as good as they believe? Are we sure that we want to insinuate that they are mistaken while celebrating success? And even harder, are we going to challenge the metrics that are being taught to the investors? Those metrics that maybe give them one more investment round? This situation is what Eric labels as __ success theatre__, when all the efforts go towards pretending that everything is going well instead of trying to create something sustainable and real.

Eric dedicates a full chapter to this topic. I highly recommend reading it. When writing this, I don’t precisely remember many concepts like cohort analysis or innovation accounting. As I don’t currently need them, I will go back to them when required.

Value hypothesis and growth hypothesis

It can be something evident for people who are used to working in the product area, but for me, a newbie, it was meaningful learning: validating that an idea is liked by potential customers, is not the same as validating that it can become a business.

We should answer this question in the value hypothesis: is this product valuable to potential customers?. To answer, we will rely on experiments and metrics to measure things like NPS, retention, or pay for the product. Although they are not perfect, they give us some (cheap) feedback to start. At this point, we can succeed or fail. If we fail, we can keep asking and trying to understand why. Does it really cover a user need? Or is it usable enough for our target user? If we succeed, the second part comes next.

Ok, we found something that lures people. How will the users find our product? This is the question to answer in the growth hypothesis. Do we know how we will grow? Let’s test it. And if we are right, fantastic, we are on a path that will probably take us to success. If not, we go back again to our assumptions. Is this the correct way to reach potential users? Or even crazier, are we wrong about our target user? Let’s think about it, decide and test it again.



I link here the notes I got from the book. They are in Catalan; if you are interested, I can translate them.

If you’ve read the book and have a different opinion or have learnt something else, don’t hesitate to share your thoughts. I will be happy to hear about it 😄.