THE MERIDIAN POINT with Kumar Dattatreyan
Why 80% of Features Go Unused - Gary Cohen on Building the Right Thing
Guest: Gary Cohen, Principal Consultant, Practical Agility
Host: Kumar Dattatreyan, Agile Meridian
------------------------------------------------------------
Kumar Dattatreyan: Hey, everyone. Kumar Dattatreyan here with The Meridian Point. Today we're joined by Gary Cohen, principal consultant at Practical Agility and an engineering and product leader who spent twenty years building a delivery process that could ship software on schedule almost every time. Here's the problem he ran into. While the delivery execution worked, and many of the features were happily used by customers, it didn't always move the needle in terms of revenue or customer growth. With a top-notch UX group and a talented product management team, they were seemingly doing everything right, but they didn't always get the full impact they were hoping for. That sent him chasing a harder question than "are we on track to deliver against our roadmap?" It was the question of whether we're building the most impactful features for our customers and for our business. He's here to share his thoughts on why it's important to embrace uncertainty in product development, and to share ideas on how to improve the executive status meetings that go along with it. So without further ado, let me pull Gary on stage. Gary, so happy to have you here. Welcome to the show.
Gary Cohen: Thank you very much for having me. It's great to be here.
Kumar Dattatreyan: It's interesting. When I read your intro, I was struck by how similar it is to the pitch for our Product Mindset course. It's about improving the impact on both the customer, the outcome for the customer, and the impact on the business. It's funny how we've traveled the last twenty years trying to crack that same nut. In your case, you spent a long time building teams that could deliver, almost predictably delivering what they commit to every sprint, every iteration. Then, after delivering, you found it didn't quite hit the mark. Tell me about how that changed your thinking and what you started to look for instead.
Gary Cohen: Honestly, it took a long time to get the delivery portion down to a smooth-running process, so I don't want to shortchange that. But yes, we worked on that, we focused on that. And our heavy-duty customers liked the software. They liked the stuff we were building. They participated in everything. But at the end of the day, did it drive a whole bunch of new customers in? Sometimes, but sometimes not. Did it increase our revenue? No. The moment that really hit home for me, that maybe we were sub-optimizing on delivery instead of impact, was when we were in our private-equity-owned stage. One of the principals from the PE company, who was also a board member, asked me, "So what additional revenue do we anticipate from the investments in all these features?" In some ways it makes perfect sense, but in some ways it was a brand-new world for me. I thought we just continued to build because that's what customers pay us for. It's a SaaS company. We assumed that by building more features it would result in more usage and more revenue from both existing and new customers. As it turns out, if you're not looking out for that to begin with, the ideas you generally wind up working on are very incremental and not big enough to move the needle.
Kumar Dattatreyan: It's an interesting insight. For most companies, the distance between the people building the thing and the people using the thing is quite far. You're not sitting in the customer's shoes every day. You don't know how they use these products or services. Do you think that has a bearing on the inability, or the lack of empathy, for what customers go through when you're building them these new features?
Gary Cohen: In some places that's very true. They don't see the customer at all. Here, we actually spent a lot of time and money to bring in customers, sometimes going to their place to see what they were trying to do. Some of it is just understanding their job. The other part is, okay, we've put together something we think will help you do your job better, now tell us what you think of it. We had packed sprint reviews every two or three weeks. Literally, it was appointment television for some of our customers. So we were good with them. But one problem was that we weren't out in the general marketplace. We weren't sitting with people who weren't already our customers. In our case, a lot of the time our competition isn't other software companies, it's Microsoft Excel or Project or Word. We were definitely missing those folks. And because we were so focused on incremental change for our existing customers, if the board had said, "Hey, we need to increase revenue by twenty percent over the next year," I don't know if I could have done it. But if that had been put in front of me, I certainly would have entertained a lot of different ideas at a bigger scale than what we were doing.
Kumar Dattatreyan: So do you think it was the language that was used? Until this investor asked, "What's the ROI for these features?" it was more about just delivering. They weren't using those terms, they weren't asking for those things. And so the teams got really good at delivering things that incrementally improved the experience, but not enough to move the needle.
Gary Cohen: We were not thinking big enough. And it's not just the product team or the engineering team. The royal we, all of us, weren't thinking big enough. The whole company. As a product company with a sizable customer base, we wanted to protect that customer base. Sometimes we lost the instinct to go out and talk to people who aren't using anything, to see how they're getting along, what their biggest pains are, and how we can try to solve them.
Kumar Dattatreyan: I think part of it is that if you know what's expected of you is twenty percent year-over-year revenue growth, there are only certain ways you can go about doing that. You need to venture a little bit further out. That part was definitely missing. Was it a startup?
Gary Cohen: We were part of a bigger company. We were the only software product. Then we got sold to a private equity company and spun out into our own company. But again, it's a little bit of the innovator's dilemma. You have your existing customers, you have a reputation in the marketplace, and you want to protect that. Sometimes when you do that, you get a little bit too incremental.
Kumar Dattatreyan: That's a good lesson to learn. One of the things you said in our prep conversation that stuck with me is that you tell leaders to embrace uncertainty. What does that look like for you when you work with teams today?
Gary Cohen: I think embracing uncertainty is incredibly important because...
Kumar Dattatreyan: Uncertainty. That's the word you use. You use the word uncertainty for leaders.
Gary Cohen: Uncertainty is incredibly important. Sorry if I said certainty. It's that presence of uncertainty that creates the opportunity for differentiation in the marketplace. If there's no uncertainty, then the only differentiation comes from going from plan to delivery, from who can execute the best. And if that's true, maybe you're in a commodity business with shrinking profit margins. But when there's a high level of uncertainty, the organizations that can reduce that uncertainty iteratively, systematically, and most efficiently will wind up finding the best ideas, learning how to implement them, and doing so faster than their competitors.
Kumar Dattatreyan: So when things are uncertain, you shouldn't be making big investments in product. You should be making more exploratory investments to see if the idea is viable before you invest a lot of money in something uncertain. Do you agree with that?
Gary Cohen: Yeah, you have to get very good at learning. How do I reduce uncertainty? What's the one thing that can doom this opportunity or this potential solution? And then what's the quickest way for me to either validate that it's okay to go ahead, or invalidate it and say, no, this will never work, let me find another idea.
Kumar Dattatreyan: So you pivot and move to something else. You don't pivot if you've proven your hypothesis, but if you haven't, you move to something else. That makes a lot of sense. But it's a hard thing. A lot of times product leaders, or leaders in general, will commit to something when they don't know enough about whether it's viable or feasible. How do you change that mindset? What's been your experience?
Gary Cohen: One way is to say, hey, we've done it this way in the past, and while we may have gotten decent results, have we set the world on fire in a positive way? If whatever you're doing works spectacularly, then I'm not here to tell you to change. But for most people, the way product development usually works is we do a bunch of upfront discovery work. We talk to people in the marketplace, customers, and so on, and we come up with the one best idea. There are a few problems with that. One is that the beginning, when you just start working in an area, is when you know the least about it. So you're making a decision way too early. Number two, once you have that initial idea and roadmap, it's just a question of degree and how quickly you get through the roadmap, even in really agile places. Whatever's on the roadmap, that's how you're going about it. I'm not sure if you're a poker player.
Kumar Dattatreyan: I'm not really a poker player.
Gary Cohen: It's basically going all in on your one hand. And if your hand loses, you've got problems, especially because these roadmaps, even done iteratively, still take months and even a year or two.
Kumar Dattatreyan: Years, yeah. I've certainly been there.
Gary Cohen: So the idea is to start with the business objective we're hoping to achieve, and then ask what are the big-enough opportunities in the marketplace, the customer wants or needs that are significant enough that if we solve them we'll at least be in the right ballpark for our business outcomes. Then, for each of those opportunities, you say, okay, we have a few ideas for solutions that might address it. Now you have multiple potential solutions, which in the old world kind of frightened us. Oh no, we need to keep moving. But when you're truly in an uncertain world and you're shooting for big market changes, you need to continuously consider several different solutions and identify the critical assumptions that have to be true for each one. And, as you were saying before, at that stage it's more about learning than about writing a whole lot of code. This is a good time to say I don't want to take credit for some of this terminology. A lot of it comes from Teresa Torres and her work on opportunity solution trees and continuous discovery habits. It's a wonderful book if people haven't read it. A lot of what she says resonates with me, having gone through the experience of doing it the traditional way.
Kumar Dattatreyan: It makes a lot of sense, and I have read her book. It's been a little while, but I have read it, and it influenced, maybe subliminally and maybe more than that, the creation of the Product Mindset course I mentioned. A lot of that language and thinking is in there. It was a great joy to create, and the fruits of it have been proven. A group of students just went through it, and many of the comments they left were like, "Oh my God, this is so life-changing. No one's ever really asked me about the impact on the business, or the outcomes the customers would experience. We just go do it." A lot of companies are just feature factories. Just go deliver features because that's what you're expected to do. There isn't enough of a feedback loop to say, okay, what we are delivering, is it actually moving the needle? Is it making an impact on our business? Is it providing the outcomes we think it should for our customers? And how do we know that? How were they feeling before, compared to how they feel now about the products and services they use? So this all resonates with me very strongly.
Gary Cohen: Sounds like a good class.
Kumar Dattatreyan: It was great. The last session was about two weeks ago, and the results are still coming in. It's a six-week program, and they build a capstone project and have to present it. Some people were late, so it's still coming in, but the feedback has been amazing. We just revamped it and added some AI into the course. All the thinking we do in our live sessions is with our human minds, no assistance from AI. But in the asynchronous sessions, they can refine what we started in the live sessions with the help of an AI agent, a product coach that helps them refine their thinking. The combination of the two was really useful for the class. It was very neat.
Kumar Dattatreyan: All right, let's shift a little. The second part of your intro is about the status meeting and how to improve it. Where's the connection? You've got product teams delivering on time but not necessarily delivering things that make an impact. How does that show up in an executive status meeting?
Gary Cohen: In traditional ones, it doesn't. And that's usually not even the biggest problem. You mentioned feature factories, and they get a bad name, but there are plenty of development organizations that would love to at least get to being a feature factory. At least they're delivering on a regular basis. The next stage is, how do we deliver the stuff that really matters and moves the needle? We've all been in the red, yellow, green status meetings. They're problematic for a few reasons. One is that all you're really concerned about is delivery risk and schedule risk, which doesn't tell you if you're building the right thing. The other big problem is that people are scared to death in those meetings. They're very defensive. Everybody reports that their project is green, or the ever-popular yellow but trending back to green. That's about as far as people are willing to go. So you get these watermelon projects, which are green on the outside but really red inside. It's essentially theater. As a leader, these meetings are almost useless. Not only are you only talking about delivery risk, you're not even getting accurate information. Everybody hates it. So I think there's a different way we could do it, which gets to asking the right questions. We talked about opportunity solution trees. Imagine a team going into a status meeting with an executive and bringing an opportunity solution tree. All of a sudden, there's information on the table that wasn't there before. Which opportunity are we focused on? What are the different ways we could take advantage of that opportunity, the different potential solutions? What are the critical assumptions that have to be true for each solution to let us hit our business outcomes, which is what the executives really care about? And what have we validated or invalidated already? With that information, think about the questions executives can ask. What potential solutions did we kill because we learned something that invalidated a critical assumption? I know it sounds bad to kill ideas, but the faster you do that, the more chance you have of getting to the right solution. What have we learned since our last meeting that validates the critical assumptions in the solutions we're actively looking at? What's the most important thing for us to learn before our next meeting? Have we identified any new potential solutions? And here's a big one, are we still convinced that this is the opportunity we should be focused on? You definitely don't hear that in your everyday status meeting. None of these questions are accusatory. They're more collaborative.
Kumar Dattatreyan: They're more generative and collaborative. You're right. It seems like a lot of big companies, and I don't know what your experience has been, but I haven't seen this type of meeting all that often. I wonder why that is. Is it because leaders want to know about the one solution, and they're overwhelmed by having multiple options in front of them, so they just don't have time to engage? What's your sense of the appetite for this? It seems a lot more useful, but I don't know how it would work in a large bureaucratic company where the culture is built around these status meetings. Even though everyone would tell you it's a waste of time, it still persists. Why is that?
Gary Cohen: I think there are a couple of reasons. First, if you're doing internal software, for example, you don't really have to differentiate. You just have to get the job done. You're not trying to find the optimal solution, any solution will do, so delivery becomes the big thing. We just need a new billing system that can do X. Second, it's incentives. A lot of times, within the organization, we form our financial incentives around delivering a particular feature or project. That again points to the assumption that we've figured out the right way to go, and all we need to do is deliver it. But one thing that might be an opening is this. If you're a senior or executive leader who has to go in front of the CEO or the board regularly and commit to things, especially when the board is talking about business outcomes, you have to report out. You might not always be able to commit to something, but you at least want confidence that whatever you're sharing with the board is accurate. Going back to the red, yellow, green meetings, you should have no confidence that you have any certainty about what's going on. And you only have one hand to play in terms of saying, this will help us hit our business outcomes. That's the way it's traditionally done. But having been in that situation, having to report to the board, I would feel a whole lot better talking about the more generative things we just mentioned, and having multiple ways to get to the intended business outcome. You can say, hey, we're basing this on these assumptions that we validated in this particular way. There's still some inherent risk, and that's why we also have this other approach in our back pocket that takes a slightly different tack. In certain circumstances, that's a much more defensible thing to say. So there are a lot of things holding the status quo together.
Kumar Dattatreyan: That's a great summary of how to shift the narrative. For a listener who runs these meetings, maybe like an interrogation because that's just the culture, "What's the status of this? Why is it not green? What are you doing to fix it?", what's the first thing they can do to shift the conversation from policing delivery to more discovery, more of that generative, collaborative type of conversation?
Gary Cohen: Short answer, there are several things they could do fairly easily to change the tenor of the meeting. One obvious thing is to ask each team, when it's their turn, "How can I help?" I'm not sure if you're familiar with an NBC show called New Amsterdam, one of those hospital shows. The fictional medical director, Max Goodwin, said that first to anybody who came up to him. "How can I help?" I know it sounds Pollyannish, but it changes the tone from "what can you do for me," or being on the defensive, to "what can I do for you." It nudges everyone toward sharing information rather than framing it, because if you're offering to help me, then to take you up on it I have to offer up information about what I need help with.
Kumar Dattatreyan: It changes the narrative completely, doesn't it? It sets the frame for more safety in the room, just by saying those words.
Gary Cohen: Right. That changes the tenor, but it doesn't change the questions being asked. So another thing is, before I ever ask if things are on schedule, I might say, "As a reminder, here are the business outcomes we're shooting for on this effort. How comfortable are we that this is still the best way to accomplish them? And what have we learned since last time that either confirms or casts doubt on that?"
Kumar Dattatreyan: I love it. Those are all simple things, but they can have a profound impact on the tenor, the feeling, the vibe, and the meaning. So, I don't think I can end any of these interviews without asking about AI. I already mentioned that I use it in the course I revamped, embedding an AI coach and agent in each of the modules as they go from concept to build. It proved very helpful for the people in the class. That's a small example of how AI can augment people's understanding, in this case understanding how to build products more collaboratively. But AI is also getting really good at the build itself, at writing code, shipping code, monitoring pipelines, and making sure the tests execute. A lot of that can be automated, if an organization chooses, with humans there to watch the edges and catch things that fall out. So where does product management go with all these tools that let you ship faster, and maybe still ship a bunch of stuff people don't need?
Gary Cohen: We can ship stuff people don't want a lot faster.
Kumar Dattatreyan: Yeah.
Gary Cohen: I think it only makes product discovery more important when something becomes cheaper. In this case, writing and deploying code. When something gets cheaper, people do more of it. So now there's going to be more competition in the marketplace. I read an article somewhere that on Google Play, or whatever app store it was, there are hundreds of thousands of new apps out there. So how do you stand out? If you're a SaaS company, a product company, a lot of your former customers may decide they can just build something themselves that does a good-enough job. So differentiation becomes that much more difficult and that much more critical. The only two levers I know of that will help you with that are, one, having a data moat, some data that nobody else has, because in the age of AI that data is really important. And two, can you do product discovery? Now that the bar is higher, can you find the things that are really valuable that nobody else can do? It's a lot harder. But I'd also say it's not all doom and gloom. Remember we were talking about opportunity solution trees and considering multiple solutions to an opportunity. Considering multiple solutions becomes much easier when the cost and speed of developing them drops significantly. You can get to a prototype very fast and start evaluating options much faster. And in the past, you may have discarded a potential solution because some part of it wasn't feasible, or was too costly. Now, with AI, you might want to revisit those assumptions, because things have changed. So product discovery becomes not only knowing your customers, the market, and the opportunities, but a continuous activity, always rethinking what the best opportunities are.
Kumar Dattatreyan: Really good point. All right, we're going to end with some lightning round questions. First one. You led the agile transformation for Google's customer support tooling, right?
Gary Cohen: For one of their customer support groups, yep.
Kumar Dattatreyan: So, one thing about how Google builds products that they get right and everybody else gets wrong. What's one thing?
Gary Cohen: One thing is they shoot really big.
Kumar Dattatreyan: So they shoot big, as in their vision is very aspirational. Is that what you mean?
Gary Cohen: Yeah. They're looking at very complex problems, using advanced technology. They're very smart people with big ideas, and even their incentive system encourages them to try to build big things.
Kumar Dattatreyan: Gotcha. Do you think they're more focused on customer outcomes, business impact, or both equally?
Gary Cohen: Again, it's been a few years since I worked with them, and it was one group there. This is my own personal opinion, but I'd say they're more of the "here's the technology, let's build something great with it" mindset. Yes, they have reasons for building certain things for customer support. But I think a lot of it is driven more by "this technology could be used for this," rather than "we think there's this huge opportunity, let's figure out what technology is best."
Kumar Dattatreyan: Gotcha. So they've got the heft and the money to experiment with lots of different technologies, put them to use, and learn from it. And it's okay if eight out of ten products fail, because it doesn't matter.
Gary Cohen: Right. When you have your search business and your ads business, you have a lot of room to experiment.
Kumar Dattatreyan: Okay, cool. Complete this sentence. The most useless artifact in a product status meeting is what?
Gary Cohen: Whatever has red, yellow, and green on it.
Kumar Dattatreyan: Whatever has red, yellow, and green. Okay. Now I'll grade this, A through F. How well has the agile industry actually learned to do discovery?
Gary Cohen: The agile industry?
Kumar Dattatreyan: Mm-hmm.
Gary Cohen: Probably a C minus or D. I've seen it done fairly well at a few places. If you said agile organizations, I might give a slightly higher grade. But at all the agile conferences, they might have fifty sessions, and only three or four of them are on product discovery.
Kumar Dattatreyan: That's why I asked, because I don't think we pay enough attention to it.
Gary Cohen: Right.
Kumar Dattatreyan: One book, and not Teresa Torres, that changed how you think about building products, because we know you love that one.
Gary Cohen: That's a good question. There are a couple that stick out. Lean Startup is one that stuck with me for a long time. That's probably the one that influenced me the most.
Kumar Dattatreyan: One last one. You attend Jardena London's leadership circle regularly. What's one of the best ideas you walked out of that room with?
Gary Cohen: These are tough questions, because there are so many to choose from. Usually it's a sense of nuance more than anything else, related to how organizations work and how change gets absorbed through them. One thing I remember from a while back is top-down organizational change versus bottom-up, or big bang versus organic. People have different opinions, and you come away with a more nuanced view. As I like to kid around with Jardena, since she's the one who introduced me to polarities, it's a polarity. You need a little bit of both. You can't get there just by doing top-down and forgetting about organic change, and vice versa. So you have to manage the balance. At certain times you may want to go more top-down, and at other times you may want to structure the organization so organic change is happening and you're constantly evolving.
Kumar Dattatreyan: I love it. Anything I haven't asked you, Gary, that you'd like to share?
Gary Cohen: No, lots of good questions here. I enjoyed chatting with you about it.
Kumar Dattatreyan: So did I. I hope all of you in the audience enjoyed it as well. Gary, we'll put information in the show notes on how people can contact you, LinkedIn and so on, so if they want to learn more about product leadership, they know who to reach out to.
Gary Cohen: Great. I appreciate that. I'd love to hear from people.
Kumar Dattatreyan: All right. Thanks so much. See you all in a couple of weeks. Bye.
Gary Cohen: Thanks.
