i

Description

Coca-Cola bottles

Source

Rowanswiki · original page

Licence

CC0 Public domain

Original photo, unretouched. The background is a version of it stylised by the site.

Need confirmation bias

We think we are gathering a need, when we are really seeking approval for the solution we already have in mind.

In brief

• What: a requirements-gathering exercise that actually serves to validate a solution already chosen.
• Why: it is a form of confirmation bias; the wording of the questions and the way we listen do the sorting before any analysis.
• Detect: closed questions (“would you like this button?”), the solution mentioned before the problem, all the feedback is favorable.
• Avoid: describe the problem before the solution, ask open questions, observe real work, look for what refutes the hypothesis.

On this page

1. Definition
2. The mechanism
3. Where it shows up
   1. In project management
   2. In management
4. Famous cases
   1. New Coke: they tested the taste, not the need 1985
   2. The “2-4-6” task 1960
5. Warning signs
6. Avoidance reflexes
7. With AI: detect, remedy, mitigate
   1. Detect
   2. Fix
   3. Mitigate
   4. Prompts to build into your routines
   5. How to build them in
   6. Precautions
8. Limits
9. Sources and verification
10. See also
    1. Combines with, or reinforces
    2. Related biases
    3. Counterweights and related reflexes

Definition#

Fictitious illustration

Need confirmation describes the way confirmation bias settles into a requirements-gathering exercise: we interview users, clients or sponsors while already having a solution in mind, we ask questions that favor it, and we retain from the answers whatever supports it. The result is a specification that seems validated by the field, when it has been neither tested nor refuted.

It is not a distinct bias in psychological terms: it is the confirmation bias described by Peter Wason (1960) and reviewed by Raymond Nickerson (1998), applied to a specific situation in project management. We treat it separately because its remedies are specific to the trade: interview method, observation, the wording of questions.

The mechanism#

• Looking for what confirms. In Wason's “2-4-6” task, participants test sequences that satisfy their hypothesis rather than sequences that could refute it; they are therefore reinforced in a false rule. An interview that tests “do you want a dashboard?” works the same way.
• Leading questions. A closed or suggestive question (“it would be more convenient with a mobile app, wouldn't it?”) almost always gets a polite “yes”. The respondent answers the question, not the need.
• Selective listening. Favorable remarks are noted down; reservations are read as special cases or as “resistance to change”.
• Attachment to the idea. The more we have invested in a solution (mock-ups, estimate, promise to management), the more costly it is to see it contradicted (see the IKEA effect).
• The respondent's politeness. A client or user approves out of courtesy, or because they cannot picture real use before getting their hands on it.

Where it shows up#

In project management#

• Requirements gathering: the analyst arrives with an architecture diagram or a mock-up and asks “does this cover your need?”. The user, who has nothing to compare it with, approves.
• Workshops: the facilitator presents the target solution at the opening; participants discuss its details instead of setting out the problem.
• A need already phrased as a solution: “we need an Excel export” is a solution, not a need. The team builds it without asking what the export is meant to enable.
• Proof of concept and demos: we show what works to those who are supportive; we avoid critical users, or see them last.
• Specification: we keep the requirements that agree with the chosen architecture and file the others under “out of scope”.

In management#

• Internal survey or barometer: the questions are written to obtain buy-in for a reorganization that has already been decided.
• Team feedback on a change of tool or process: we interview those we know will agree, or we ask “do you have any objections?” in front of everyone.
• Annual reviews: the manager who already has an opinion about an employee collects the facts that confirm it.
• Field consultation by headquarters: headquarters presents the target arrangement and collects opinions on its details; the question of whether it meets a real need is never asked.

Famous cases#

Warning signs#

• The solution is named before the problem (“we're going to build a portal, what do we put in it?”).
• Questions that end with “right?”, “don't you think?”, or that allow only “yes” or “no”.
• The feedback is unanimously favorable, with no reservations, no unforeseen use cases.
• The needs expressed are technical functions (“an export”, “a filter”), not tasks to accomplish.
• The users interviewed are the ones we already know or who are supportive.
• Critical remarks are filed as “resistance to change” or “out of scope” before they have been examined.
• None of the interviews took place where the work is done: we have only heard about the work.

Avoidance reflexes#

1. The problem before the solution. Open the workshop with the current situation: what the user does today, what costs them time, what fails. The solution is presented only afterwards, as a hypothesis.
2. Open questions about the past. “Tell me about the last time you…” rather than “would you like…”. Past facts are less subject to politeness than future intentions (see Rob Fitzpatrick's The Mom Test).
3. Observe real work. Spending half a day next to the user teaches what they do not say, because they have forgotten it or find it too obvious.
4. Look for what refutes. Before the interview, write down what would make you abandon or change the idea, then look for it: “what would make this solution useless to you?”
5. Also interview those who disagree. Make a list of “difficult” stakeholders (critical users, support, operations, a competing business unit) and see them first.
6. Translate a requested solution into a need. Faced with “we need an Excel export”, ask “what do you do with it afterwards?” until you reach the real task.
7. Appoint a devil's advocate in the team: at each readout, they prepare the three reasons why the retained need could be wrong.
8. Test with a behavior, not an opinion. A rough prototype used in context, or a commitment (time, data, pre-order), teaches more than verbal approval.

With AI: detect, remedy, mitigate#

Prompts to build into your routines#

1. Reviewing an interview guide. Before requirements-gathering interviews.

Here is the interview guide I plan to use to understand the work of [type of users]. Do not concern yourself with the solution being considered.
1. Identify each question that is closed, leading (it contains the expected answer), double-barreled or oriented toward a solution.
2. Rephrase them as open questions about past facts (“tell me about the last time you…”).
3. Add five questions that would try to show that the need does not exist, or that it can be met another way.
4. Say, in one sentence, what this guide would let us avoid hearing.

Guide: [text]

2. Separating needs, solutions and reservations. On workshop or interview notes, anonymized.

Here are the raw notes from a requirements-gathering workshop (names removed). Without knowing the solution we are considering:
- classify each statement into three columns: “task or problem”, “requested solution”, “reservation or objection”;
- for each requested solution, propose the real task it hides, as a question to ask;
- list all the reservations and objections, including those the official minutes did not retain;
- conclude in one sentence: do the notes contain grounds to challenge the initial idea?

Notes: [text]

How to build them in#

• Before: prompt 1 serves as a review of the guide, with a colleague playing devil's advocate.
• After each interview: drop the raw notes into prompt 2 before writing the minutes.
• At the readout: present the reservations next to the confirmed needs, not in an appendix.

Precautions#

• Do not reveal the solution to the tool before it has sorted the notes: it would start confirming it.
• AI does not replace observing real work: it only sees what has been written.
• Interview notes often contain information about people: anonymize before pasting them into an unauthorized tool.
• Check the references or “best practices” it cites: they may be invented.

Limits#

• Having a hypothesis is not a fault: no one conducts an interview without a starting idea. The flaw is treating it as established and looking only for what supports it.
• Open questions and observation take time. For a simple, well-known need, a quick validation may be enough.
• Users do not always know how to say what they need; observing and prototyping compensate, but do not remove the uncertainty.
• The New Coke example is famous but often told in a simplified way: other factors (brand, nostalgia, media) played a part, and the episode even, according to some analysts, revived the brand's sales. The lesson about the badly posed question remains valid.

Sources and verification#

• Wason (1960), “On the failure to eliminate hypotheses in a conceptual task”, Quarterly Journal of Experimental Psychology, 12(3) — the “2-4-6” task.
• Nickerson (1998), “Confirmation bias: A ubiquitous phenomenon in many guises”, Review of General Psychology, 2(2) — review.
• Fitzpatrick (2013), The Mom Test: How to talk to customers and learn if your business is a good idea when everyone is lying to you, CreateSpace.
• Torres (2021), Continuous Discovery Habits, Product Talk LLC.
• The Coca-Cola Company, “New Coke: The Most Memorable Marketing Blunder Ever?” — dates, taste tests, return on July 11, 1985.
• ABC7, “Beverage Backlash: History of New Coke” — repeats the dates and the figure of nearly 200,000 consumers.

See also#