Workflow: Competing Hypotheses Investigation

Type: Debugging | Complexity: High | Team Size: 3-5 teammates

Parallel Debugging

When the root cause is unclear, spawn multiple teammates to investigate different theories in parallel. They actively try to disprove each other's hypotheses, preventing anchoring bias and converging on the real cause faster.

When to Use

• Mysterious bugs with unclear root cause
• Performance issues requiring investigation
• User reports that don't match observations
• Multi-layered systems where cause could be anywhere
• Time-sensitive issues where speed matters

When NOT to Use

• Obvious bugs with clear cause
• Simple issues where one developer suffices
• Non-technical problems (infrastructure, deployment)

The Competing Hypotheses Advantage

Traditional debugging (single investigator):

1. Investigator finds one plausible theory
2. Confirms that theory with biased evidence collection
3. Stops investigating once theory "explains" symptoms
4. Often misses the real root cause

Competing hypotheses (multiple investigators):

1. Each investigator proposes different theory
2. All actively try to disprove others' theories
3. Debate forces examination of evidence
4. Theory that survives scrutiny is likely correct

Workflow Overview

Step-by-Step Guide

Step 1: Define the Symptoms

Write clear, observable symptoms:

App crash report:
- User sends first message
- App crashes immediately
- Error: "Cannot read property 'id' of undefined"
- Happens in 100% of cases
- Occurred after deploy of commit abc123

Step 2: Generate Initial Hypotheses

Think of 3-5 plausible explanations:

1. Memory leak - Growing memory use kills process
2. Database connection - First query hangs or fails
3. Message queue - Queue service unavailable
4. Race condition - Timing issue in initialization
5. Dependency bug - New version of library broke it

Step 3: Create Team with Adversarial Setup

Users report the app crashes immediately when sending the first message.
Error: "Cannot read property 'id' of undefined"
Happened after the latest deploy.

Create 5 agent teammates to investigate competing hypotheses.
Each teammate takes a different theory and ACTIVELY TRIES TO DISPROVE
the others' hypotheses through evidence and testing:

- Teammate 1: "It's a memory leak causing OOM"
- Teammate 2: "Database connection is failing"
- Teammate 3: "Message queue service is down"
- Teammate 4: "There's a race condition in startup"
- Teammate 5: "A dependency was updated and broke something"

Have them talk to each other, challenge each other's findings,
and converge on the most likely root cause.

Step 4: Direct the Investigation

Each teammate works on their hypothesis:

Teammate 1: "Check memory usage patterns. Show me heap snapshots from before/after deploy."
Teammate 2: "Try connecting to the database directly. What happens?"
Teammate 3: "Verify message queue service is running. Check logs."
Teammate 4: "Add debug logging to startup sequence. Identify race conditions."
Teammate 5: "Run 'npm audit' and check dependency changes. Test with previous version."

Step 5: Foster Active Debate

Encourage challenge and cross-examination:

Teammates should challenge each other's findings.
Teammate 1: "How do you explain normal CPU usage if there's a memory leak?"
Teammate 2: "Where's the error trace showing database failure?"
Etc.

Update your findings doc with what you discover and how you'd respond to challenges.

Lead might also inject skepticism:

Teammate 3: Your message queue theory doesn't explain why this started
IMMEDIATELY after deploy. Before, users could send messages successfully.
How does that fit your theory?

Step 6: Converge on Most Likely Cause

Based on all investigations, which hypothesis survives the most scrutiny?
What's the single most likely root cause?
Update findings doc with:
1. Root cause hypothesis
2. Evidence that supports it
3. Evidence that contradicts other hypotheses
4. How to verify
5. How to fix

Step 7: Verify and Document

Let's verify the root cause with a test:
[Specific test to confirm]

If confirmed, update findings with:
- Root cause confirmed
- Why this wasn't obvious
- Fix recommendation
- Steps to prevent in future

Step 8: Clean Up

Clean up the team

Real-World Example

Scenario: Post-Deploy Crash

Reported Issue:

After pushing commit abc123, users report app crashes
immediately when trying to send a message.
Error: Cannot read property 'id' of undefined
100% reproducible

Team Investigation:

Hypothesis 1 - Memory Leak (Teammate A)
Investigation: Monitor memory, find growth pattern
Finding: Memory stable, not leak
Evidence: heap snapshot shows normal memory

Hypothesis 2 - Database Connection (Teammate B)
Investigation: Test DB connection, query directly
Finding: Connection hangs on user lookup query
Evidence: Query takes 30+ seconds, then times out

Hypothesis 3 - Message Queue Down (Teammate C)
Investigation: Check service status, test manually
Finding: Queue is running, accepts messages
Evidence: Manual publish/subscribe works fine

Hypothesis 4 - Race Condition (Teammate D)
Investigation: Add timing debug logs to startup
Finding: User service loads after message service
Evidence: Message handler tries to call user.id before user loads

Hypothesis 5 - Dependency Bug (Teammate E)
Investigation: Check dependency versions, test rollback
Finding: New Express version changed middleware order
Evidence: Rollback to previous version fixes issue

Debate:

A: Database query hangs, but why only after deploy?
B: Good point. Let me check git diff...
B: Ah! We added a JOIN that's inefficient. That explains the hang.
D: But even if query hangs, that shouldn't crash with "property undefined"
B: Right. So database delay triggers something else...
E: What if the middleware change causes timeout before user data loads?
D: YES! New middleware must have changed execution order!

Converged Finding:

Root Cause: Express middleware order changed in v4.18.0
- Old: Load user middleware → Message handler
- New: Message handler → Load user middleware
Result: Message handler tried to use user.id before it was loaded
Fix: Reorder middleware in app.js or downgrade Express

Advanced Techniques

Multiple Investigation Tracks

Instead of 5 single investigators, use pairs:

Pair 1 (Memory + Resource): Investigate memory leak and resource exhaustion
Pair 2 (Database): Investigate database failures and queries
Pair 3 (External Services): Investigate message queue, cache, external APIs

Pairs challenge each other while collaborating within pair.

Staged Hypothesis Refinement

Start broad, refine:

Stage 1 - Broad categories:

1. Client-side issue
2. Server-side issue
3. Infrastructure issue

Stage 2 - Narrow down:

1. Server startup issue
2. Request handling issue
3. Database issue

Stage 3 - Specific causes:

1. Middleware initialization order
2. Race condition in pool connection
3. Missing environment variable

Root Cause Analysis Document

After finding cause, document:

# Root Cause Analysis: Post-Deploy Crash

## Symptoms
- App crashes immediately when sending first message
- Error: Cannot read property 'id' of undefined
- 100% reproducible
- Started after commit abc123

## Root Cause
Express middleware order changed in upgrade to v4.18.0
- Message handler tried to access user.id before user middleware loaded

## Why Not Obvious
- Worked in dev (single instance, different load)
- Only manifests under certain timing conditions
- Error message didn't indicate middleware issue

## How It Should Work
1. Request arrives
2. User middleware loads user data
3. Message handler accesses user.id
4. Message processed

## What Went Wrong
1. Middleware order changed
2. Message handler now runs first
3. user.id doesn't exist yet
4. Crash

## Fix
Revert to Express v4.17.x OR explicitly set middleware order in app.js

## Prevention
- Test all dependencies before deploying
- Monitor error rates immediately after deploy
- Have quick rollback process

Verification Checklist

✓ Team investigating simultaneously

• All 3-5 hypotheses being tested in parallel
• Not waiting on each other
• Active investigation happening

✓ Debate happening

• Teammates challenging each other
• Evidence being presented
• Assumptions being questioned

✓ Progress tracked

• Finding doc being updated
• Evidence accumulating
• Hypotheses being tested/ruled out

✓ Convergence reached

• Team agrees on most likely cause
• Evidence strongly supports it
• Other hypotheses ruled out

✓ Verification complete

• Root cause confirmed with test
• Fix is clear
• Prevention steps identified

Troubleshooting

Q: Investigation going in circles A:

1. Refocus: "What specific evidence would disprove your hypothesis?"
2. Set timeline: "We have 30 minutes to gather evidence"
3. Escalate to specific test

Q: One hypothesis dominating A:

1. Actively amplify minority views
2. Ask "What if that hypothesis is wrong? What evidence would change your mind?"
3. Swap teammates' hypotheses

Q: Can't reproduce the issue A:

1. Get exact reproduction steps
2. Try in different environments
3. May need to investigate "why sometimes" rather than "why always"

Q: Root cause is combination of factors A: This is realistic! Document:

Root Cause: Combination of three factors
1. Middleware order change (necessary condition)
2. High load in prod vs dev (trigger condition)
3. Database query timeout (cascading failure)
All three needed for crash; fix any one prevents it.

Performance Metrics

Track investigation effectiveness:

Metric	Single Investigator	Competing Hypotheses
Time to root cause	4 hours	1-2 hours
Hypotheses tested	1-2	5+
False leads pursued	2-3	0-1
Confidence in cause	Medium	High

Competing hypotheses typically finds real cause 3-4x faster by preventing anchoring bias.

See Also

• SOP-002: Create Your First Team
• Parallel Code Review Workflow
• Plan Approval Workflow
• Troubleshooting Guide