prodia.dev
Start Free
Falsification Log

The system learns from what failed

Prodia does not hide failures. Every falsified hypothesis is recorded, root-caused, and converted into a policy. Failure is the substrate of trust.

Failures · this week
14
▼26% vs. last
Rejected before deploy
87%
▲caught pre-prod
Mean recovery · p50
47s
•p95 184s · p99 312s
Policies authored from failures
142
▲all-time

Falsified hypotheses

The mutation, what broke, why, and what we learned. Each entry is a public commitment to honest engineering.

  1. FX-44212026-05-29 13:41falsified
    Hypothesis
    Advisory lock prevents duplicate refunds w/o throughput loss
    Attempted mutation
    Add pg advisory_xact_lock(refund_id) in /refunds/issue
    Validation failure
    p99 latency +480ms under burst; 3 contended deadlocks/min
    Root cause
    Lock acquired before idempotency check → serialised all refunds per merchant
    Impact
    Staging only · 0 customer impact · auto-blocked deploy
    Resolution
    Move lock past idempotency short-circuit; scope to (merchant_id, idempotency_key)
    Learning outcome
    Locks must follow idempotency, not precede it; codified in policy POL-RFND-09
  2. FX-44202026-05-29 11:08falsified
    Hypothesis
    Switching to HTTP/3 cuts TTFB on edge by ≥20%
    Attempted mutation
    Enable QUIC on 2 canary PoPs
    Validation failure
    TTFB +6ms regional; 0.4% client handshake failures (older Android)
    Root cause
    Client base skews older; QUIC fallback path slower than warm TCP
    Impact
    0.4% canary users · auto-rolled back in 92s
    Resolution
    Restrict QUIC to clients announcing h3 in Alt-Svc cache; revisit Q3
    Learning outcome
    Protocol upgrades require client-mix telemetry, not just synthetic benchmarks
  3. FX-44192026-05-29 09:52falsified
    Hypothesis
    Removing the legacy /v1/users endpoint is safe (0 traffic 30d)
    Attempted mutation
    Return 410 Gone on /v1/users/*
    Validation failure
    1 internal cron (data-warehouse-sync) broke silently
    Root cause
    Cron used VPC-internal route not captured in edge traffic logs
    Impact
    1 internal job failed · no customer surface
    Resolution
    Restore endpoint; add deprecation header; require dual-source traffic proof before removal
    Learning outcome
    Deprecation requires edge + VPC traffic confirmation, not edge alone
  4. FX-44182026-05-28 22:14falsified
    Hypothesis
    Re-encoding hero video to AV1 cuts page weight ≥40%
    Attempted mutation
    Swap mp4→av1 on /pricing
    Validation failure
    Safari <16 cannot decode AV1; LCP +1.2s on 6% of traffic
    Root cause
    Browser support matrix not consulted in mutation planning
    Impact
    6% of pricing-page sessions · auto-rolled back
    Resolution
    Multi-source <video>; AV1 for capable clients, h264 fallback
    Learning outcome
    Asset format changes must run through compat-matrix policy POL-WEB-12
  5. FX-44172026-05-28 17:33falsified
    Hypothesis
    Async webhook delivery improves checkout p95
    Attempted mutation
    Detach Stripe webhook ack from response cycle
    Validation failure
    Stripe retries spiked; double-charge prevention guarantee weakened
    Root cause
    Stripe expects ack within HTTP cycle; async ack interpreted as failure
    Impact
    Staging only · 0 customer impact
    Resolution
    Keep sync ack; offload only post-ack side effects
    Learning outcome
    Vendor contracts override generic perf heuristics; added to vendor-policy registry

Hiding failures is not engineering — it is theatre. Prodia records every falsification so the system, and its operators, can be trusted with autonomy.