Product Discovery

Description

Now that we have discussed the overall concept of product management and why it is important, and how product teams are formed, we can turn more specifically to the topics of product discovery and product design; see Product Design. We have previously discussed the overall digital business context, as a startup founder might think of the problem. But the process of discovery continues as the product idea is refined, new business problems are identified, and solutions (such as specific feature ideas) are designed and tested for outcomes.

This guidance favors the idea that products are “discovered” as well as “designed”.

The presence of a section entitled “product discovery” in this document is a departure from other IT management textbooks. “Traditional” models of IT delivery focus on projects and deliverables, concepts we touched on previously but that we will not explore in depth until later in the document. However, the idea of “product discovery” within the large company is receiving more and more attention. Even large companies benefit when products are developed with tight-knit teams using fast feedback.

For our discussion here, the challenge with the ideas of projects and deliverables is that they represent approaches that are more open-loop, or at least delayed in feedback. Design processes do not perform well when feedback is delayed. System intent, captured as a user story or requirement, is only a hypothesis until tested via implementation and user confirmation.

Formalizing Product Discovery

In Application Delivery, we needed to consider the means for describing system intent. Even as a bare-bones startup, some formalization of this starts to emerge, at the very least in the form of test-driven development; see Product Discovery Tacit.

Figure 1. Product Discovery Tacit

But, the assumption in our emergence model is that more formalized product management emerges with the formation of a team. As a team, we now need to expand “upstream” of the core delivery pipeline, so that we can collaborate and discover more effectively. Notice the gray box in Product Discovery Explicit.

Figure 2. Product Discovery Explicit

The most important foundation for your newly formalized product discovery capability is that it must be empirical and hypothesis-driven. Too often, product strategy is based on the Highest Paid Person’s Opinion (HiPPO).

The problem with relying on “gut feeling” or personal opinions is that people – regardless of experience or seniority – perform poorly in assessing the likely outcome of their product ideas. A well-known case is the initial rejection of Amazon shopping cart suggestions [Linden 2006]. Some well-known research on this topic was conducted by Microsoft’s Ronny Kohavi. In this research, Kohavi and team determined that “only about 1/3 of ideas improve the metrics they were designed to improve” [Kohavi et al. 2009]. As background, the same report cites that:

“Netflix considers 90% of what they try to be wrong”
“75% of important business decisions and business improvement ideas either have no impact on performance or actually hurt performance” according to Qualpro (a consultancy specializing in controlled experiments)

It is, therefore, critical to establish a strong practice of data-driven experimentation when forming a product team and avoid any cultural acceptance of “gut feelings” or deferring to HiPPOs. This can be a difficult transition for the company founder, who has until now served as the de facto product manager.

A useful framework, similar to Lean Startup, is proposed by Spotify™ in the “DIBB” model:

Data
Insight
Belief
Bet

Data leads to insight, which leads to a hypothesis that can be tested (i.e., “bet” on – testing hypotheses is not free). We discuss issues of prioritization further in Work Management, in the section on cost of delay.

Don Reinertsen (see Competency Area 5 for more details) emphasizes that such experimentation is inherently variable. We cannot develop experiments with any sort of expectation that they will always succeed. We might run 50 experiments, and only have two succeed. But if the cost of each experiment is $10,000, and the two that succeeded earned us $1 million each, we gained:

$ 2,000,000
$ - 500,000
-----------
$ 1,500,000

Not a bad return on investment! See Product Management for a detailed, mathematical discussion, based on options and information theory [Reinertsen 2009]. Roman Pichler, in Agile Product Management with Scrum, describes “old-school” versus “new-school” product management as in Old School versus New School Product Management [Pichler 2010].

Table 1. Old School *versus* New School Product Management
Old School	New School
Shared responsibility	Single product owner
Detached/distant product management	Product owner on the Scrum team
Extensive up-front research	Minimal up-front work to define rough vision
Requirements frozen early	Dynamic backlog
Late feedback due to lengthy release cycle	Early and frequent releases drive fast feedback, resulting in customer value

Product Discovery Techniques

There are a wide variety of techniques and even “schools” of product discovery and design. This section considers a few representatives. At the team level, such techniques must be further formalized. The techniques are not mutually-exclusive; they may be complementary. User Story Mapping was previously mentioned. In product discovery terms, User Story Mapping is a form of persona analysis. But that is only one of many techniques. Roman Pichler mentions “Vision Box and Trade Journal Review” and the “Kano Model” [Pichler 2010]. Here, let us discuss:

“Jobs to be Done” analysis
Impact mapping
Business analysis and architecture

Jobs To Be Done

The “Jobs to be Done” framework was created by noted Harvard professor Clayton Christensen, in part as a reaction against conventional marketing techniques that “frame customers by attributes – using age ranges, race, marital status, and other categories that ultimately create products and entire categories too focused on what companies want to sell, rather than on what customers actually need” [Christensen Institute].

In other words, “some products are better defined by the job they do than the customers they serve” [Traynor]. This is in contrast to many kinds of business and requirements analysis that focus on identifying different user personas (e.g., 45-55 married woman with children in the house). Jobs to be Done advocates argue that “the job, not the customer, is the fundamental unit of analysis” and that customers “hire” products to do a certain job [Christensen at al. 2006].

To apply the Jobs to be Done approach, Des Traynor suggests filling in the blanks in the following:

People hire your product to do the job of ------— every --------— when ----------. The other applicants for this job are --------, --------, and --------, but your product will always get the job because of --------.

Understanding the alternatives people have is key. It is possible that the job can be fulfilled in multiple different ways. For example, people may want certain software to be run. This job can be undertaken through owning a computer (e.g., having a data center). It can also be managed by hiring someone else’s computer (e.g., using a cloud provider). Not being attentive and creative in thinking about the diverse ways jobs can be done places you at risk for disruption.

Impact Mapping

Understanding the relationship of a given feature or component to business objectives is critical. Too often, technologists (e.g., software professionals) are accused of wanting “technology for technology’s sake”.

Showing the “line of sight” from technology to a business objective is, therefore, critical. Ideally, this starts by identifying the business objective. Gojko Adzic’s Impact Mapping: Making a Big Impact with Software Products and Projects [Adzic 2012] describes a technique for doing so: “An impact map is a visualization of scope and underlying assumptions, created collaboratively by senior technical and business people.”

Starting with some general goal or hypothesis (e.g., generated through Lean Startup thinking), build a “map” of how the goal can be achieved, or hypothesis can be measured. A simple graphical approach can be used; see Impact Map.

Figure 3. Impact Map

Impact mapping is similar to mind mapping, and some drawing tools come with “Mind Mapping” templates.

The most important part of the impact map is to answer the question “why are we doing this?”. The impact map is intended to help keep the team focused on the most important objectives, and avoid less valuable activities and investments.

For example, in Impact Map, we see that a bank may have an overall business goal of customer retention. (It is much more expensive to gain a new customer than to retain an existing one, and retention is a metric carefully measured and tracked at the highest levels of the business.)

Through focus groups and surveys, the bank may determine that staying current with online services is important to retaining customers. Some of these services are accessed by home PCs, but increasingly customers want access via mobile devices.

These business drivers lead to the decision to invest in online banking applications for both the Apple and Android™ mobile platforms. This decision, in turn, will lead to further discovery, analysis, and design of the mobile applications.

The Business Analysis Body of Knowledge^® (BABOK^®)

One well-established method for product discovery is that of business analysis, formalized in the Guide to the Business Analysis Body of Knowledge [BABOK Guide] from the International Institute of Business Analysis.

The BABOK defines business analysis as:

The practice of enabling change in the context of an enterprise by defining needs and recommending solutions that deliver value to stakeholders.

The BABOK is centrally concerned with the concept of requirements, and classifies them as follows:

Business requirements
Stakeholder requirements
Solution requirements
- Functional requirements
- Non-functional requirements
Transition requirements

The BABOK also provides a framework for understanding and managing the work of business analysts; in general, it assumes that a Business Analyst capability will be established and that maturing such a capability is a desirable thing. This may run counter to the Scrum ideal of cross-functional, multi-skilled teams. Also as noted above, the term “requirements” has fallen out of favor with some Agile thought leaders.

Product Design

Once we have discovered at least a direction for the product’s value proposition, and have started to understand and prioritize the functions it must perform, we begin the activity of design. Design, like most other topics in this document, is a broad and complex area with varying definitions and schools of thought.

Design is an ongoing theme throughout the humanities, encountered in architecture (the non-IT variety), art, graphics, fashion, and commerce. It can be narrowly focused, such as the question of what color scheme to use on an app or web page. Or it can be much more expansive, as suggested by the field of design thinking. We will start with the expansive vision and drill down into a few topics.

Design Thinking

Design thinking is a recent trend with various definitions, but in general, combines a design sensibility with problem solving at significant scale. It usually is understood to include a significant component of systems thinking.

Design thinking is the logical evolution of disciplines such as user interface design when such designs encounter constraints and issues beyond their usual span of concern. Although it has been influential on Lean UX and related works, it is not an explicitly digital discipline.

There are many design failures in digital product delivery. What is often overlooked is that the entire customer experience of the product is a form of design.

Consider, for example, Apple. Their products are admired worldwide and cited as examples of “good design”. Often, however, this is only understood in terms of the physical product; for example, an iPhone^® or a MacBook Air^®. But there is more to the experience. Suppose you have technical difficulties with your iPhone, or you just want to get more value out of it. Apple created its popular Genius Bar support service, where you can get support and instruction in using the technology.

Notice that the product you are using is no longer just the phone or computer. It is the combination of the device PLUS your support experience. This is essential to understanding the modern practices of design thinking and Lean UX.

Design Thinking Key Characteristics, condensed from Lotta Hassi and Miko Laakso [Hassi & Laakso 2011], provides a useful overview of design thinking.

Table 2. Design Thinking Key Characteristics
Practices	Thinking Styles	Mentality
• Human-Centered Approach; e.g., people-based, user-centered, empathizing, ethnography, observation • Thinking by Doing; e.g., early and fast prototyping, fast learning, rapid iterative development cycles • Combination of Divergent and Convergent Approaches; e.g., ideation, pattern finding, creating multiple alternatives • Collaborative Work Style; e.g., multi-disciplinary collaboration, involving many stakeholders, interdisciplinary teams	• Abductive Reasoning; e.g., the logic of “what could be”, finding new opportunities, urge to create something new, challenge the norm • Reflective Reframing; e.g., rephrasing the problem, going beyond what is obvious to see what lies behind the problem, challenge the given problem • Holistic View; e.g., systems thinking, 360 degree view on the issue • Integrative Thinking; e.g., harmonious balance, creative resolution of tension, finding balance between validity and reliability	• Experiemental & Explorative; e.g., the license to explore possibilities, risking failure, failing fast • Ambiguity Tolerant; e.g., allowing for ambiguity, tolerance for ambiguity, comfortable with ambiguity, liquid and open process • Optimistic; e.g., viewing constraints as positive, optimism attitude, enjoying problem solving • Future-Oriented; e.g., orientation towards the future, vision versus status quo, intuition as a driving force

Hypothesis Testing

The concept of hypothesis testing is key to product discovery and design. The power of scalable cloud architectures and fast continuous delivery pipelines has made it possible to test product hypotheses against real-world customers at scale and in real time. Companies like Netflix and Facebook have pioneered techniques like “canary deployments” and “A/B testing”.

In these approaches, two different features are tried out simultaneously, and the business results are measured. For example, are customers more likely to click on a green button or a yellow one? Testing such questions in the era of packaged software would have required lab-based usability engineering approaches, which risked being invalid because of their small sample size. Testing against larger numbers is possible, now that software is increasingly delivered as a service.

Usability and Interaction

At a lower level than the holistic concerns of design thinking, we have practices such as usability engineering. These take many forms. There are many systematic and well-researched approaches to:

Usability, interaction design (Cooper et al. 2007, Isaacs & Walendowski 2001, Tidwell et al. 2020, Bank & Cao 2016)
Visualization (Card et al. 1999, Tufte 2001)

All such approaches, however, should be used in the overall Lean Startup/Lean UX framework of hypothesis generation and testing. If we subscribe to design thinking and take a whole-systems view, designing for ease of operations is also part of the design process. We will discuss this further in Digital Operations. Developing documentation of the product’s characteristics, from the perspective of those who will run it on a day-to-day basis, is also an aspect of product delivery.

Product Discovery versus Design

Some of the most contentious discussions related to IT management and Agile come at the intersection of software and systems engineering, especially when large investments are at stake. We call this the “discovery versus design” problem.

Frequent criticisms of Lean Startup and its related digital practices are:

They are relevant only for non-critical Internet-based products (e.g., Facebook and Netflix)
Some IT products must fit much tighter specifications and do not have the freedom to “pivot” (e.g., control software written for aerospace and defense systems)

There are two very different product development worlds. Some product development (“cogs”) is constrained by the overall system it takes place within. Other product development (“flowers”) has more freedom to grow in different directions – to “discover” the customer.

The cog represents the world of classic systems engineering – a larger objective frames the effort, and the component occupies a certain defined place within it. And yet, it may still be challenging to design and build the component, which can be understood as a product in and of itself. Fast feedback is still required for the design and development process, even when the product is only a small component with a very specific set of requirements.

The flower represents the market-facing digital product that may “pivot”, grow, and adapt according to conditions. It also is constrained, by available space and energy, but within certain boundaries has greater adaptability.

Neither is better than the other, but they do require different approaches. In general, we are coming from a world that viewed digital systems strictly as cogs. Subsequently, we are moving towards a world in which digital systems are more flexible, dynamic, and adaptable.

When digital components have very well-understood requirements, usually we purchase them from specialist providers (increasingly “as a service”). This results in increasing attention to the “flowers” of digital product design since acquiring the “cogs” is relatively straightforward (more on this in the section on sourcing).

Evidence of Notability

Product discovery techniques are widely discussed in the product management community and are frequent topics of presentation at notable industry events such as Agile Alliance conferences.

Limitations

In organizations that are primarily purchasing software and not building it, product discovery techniques may be less applicable. However, internal “products” understood as business capabilities may still benefit from a design/discovery approach, even if they are based on, for example, a SaaS offering.

Related Topics