A multi-pillar framework for synthetic behavioral research, built around a Three-Run Replication Architecture, that catches systematic friction in linear flows fast and cheap — as a complement to deep human research, not a replacement for it.
Canary is a friction sweeper / heuristic detector for linear, decision-heavy flows: checkout, signup, pricing, onboarding, account setup. It does not claim to match or exceed the trustworthiness of traditional human-subject research, and it is not a replacement for generative discovery, ethnography, accessibility audits with assistive-tech users, or culturally-bound research. It is a complement to that work.
Traditional user research is slow for structural reasons. You must recruit real people, schedule them, compensate them, talk to them one at a time, transcribe and code the output, and synthesize across a small sample. A competent mixed-methods study takes 4–8 weeks and $12K–$30K. The median business-to-business (B2B) Software-as-a-Service (SaaS) product team cannot run more than 2–4 of these per year.
Most product decisions are made without research at all.
Synthetic research uses large language models to simulate a population of research participants with specified demographic, psychographic, and behavioral characteristics — and then runs interviews, surveys, card sorts, concept tests, and usability walkthroughs against that synthetic population. The output is a research report.
Done naively, synthetic research is worse than no research — it launders a single model's biases into what looks like evidence. Done with rigor, it is a legitimate instrument for a well-defined class of product questions.
The Canary Method is our formalization of "done with rigor." It draws on published validation literature (Horton 2023 on homo silicus, Argyle et al. 2023 on demographic-conditioned Large Language Models (LLMs), Dillion et al. 2024 on moral judgment parity, Aher et al. 2023 on Turing experiments) and on 15 years of practical field-research operations.
These are the commitments every Canary engagement honors. Violations are disqualifying. No report leaves our door without all nine in place.
Canary's foundational rigor mechanism. Each audit runs the same five personas through three independent model families, with a separate concordance check on top.
| Run | Model | Role |
|---|---|---|
| Primary | Claude Sonnet 4.5 + extended thinking | Full audit: 5 personas, synthesis, full report draft |
| Replication 1 | GPT-5 | Lean re-audit: same 5 persona prompts, finding list only |
| Replication 2 | Gemini 2.5 Pro | Lean re-audit: same 5 persona prompts, finding list only |
| Concordance check | Gemini 2.5 Pro (separate call) | Tags each Primary finding Concordant-3 / Concordant-2 / Singleton-1 |
The five personas are differentiated by prompt engineering — different system prompts, different goals, different decision rules — inside each run. We do not route different personas to different models; that conflates persona variance with model variance and makes results uninterpretable. Each run sees all five personas; the variance we measure is across model families, holding personas constant.
Concordance tagging and how it ships:
Why it matters: Canary already replicates against itself three times before any external validity comparison. Concordance is the rigor signal a client can read off the page. A finding tagged Concordant-3 has survived three independent model-family runs; that is something neither a single-model AI tool nor a single-reviewer heuristic eval can claim.
The most common reviewer challenge to Canary's methodology is variance: how can 5 personas produce significant results? The answer is that significance comes from depth and replication, not from sample size. Canary's defensibility rests on three pieces working together — persona depth (this section), Three-Run Replication (Pillar #1 above), and Behavioral Architect sign-off (Pillar #3 below). High-N synthetic studies that skip persona architecture are louder, not stronger.
Each of the 5 personas in a Canary audit is built on a seven-part loadout — not a character sketch. The full loadout is documented per audit and shipped with the report so the client can audit who walked their product.
| Loadout component | What it specifies |
|---|---|
| Demographic profile | Age, role, tech literacy, native language, accessibility considerations. |
| Cognitive bias loadout | The 2–3 specific biases that dominate this persona's decision-making (e.g., loss aversion, status quo bias, anchoring, ambiguity aversion). |
| Domain knowledge level | Novice, intermediate, or expert in the product category — calibrated to a real distribution of likely visitors. |
| Decision rules | Explicit triggers for purchase, abandonment, and override (e.g., "abandons if asked for credit card before seeing pricing"). |
| Goal stack | Primary goal, secondary goal, fallback goal — ranked, not just listed. |
| Emotional state baseline | Stressed, curious, skeptical, or motivated at the moment of arrival. |
| Time-pressure profile | How long the persona is willing to spend before giving up; how interruptions reshape the walk. |
Canary does not draw 5 personas from a uniform distribution. The 5 are selected to triangulate the user space of the specific product under audit, typically:
Persona composition is documented per audit, justified against the product's likely visitor distribution, and shipped with the report. A client who disagrees with the composition can ask for a re-run with a substituted persona — that diff is logged.
Shallow Large Language Model (LLM) personas — "imagine you are a 32-year-old marketing manager in Denver" — converge to the same response distribution because they share latent space. Running 500 of them produces a loud average, not 500 independent walks. Five personas with explicit divergent loadouts — different bias stacks, different decision rules, different goal stacks — produce genuinely different walks of the same product. Re-running those five walks across three model families (Three-Run Replication) is the significance test: does the friction reproduce when the same persona is voiced by a different model? That is a stronger test than averaging 500 shallow personas inside one model.
Methodological lineage. Canary's persona architecture is informed by established behavioral-science persona-construction conventions — Alan Cooper's primary/secondary persona model, Christensen's jobs-to-be-done framework, and the cognitive-bias taxonomies of Kahneman and Tversky. It is not invented from scratch. It is the standard human-research persona-design discipline, ported to a synthetic medium and pressure-tested by Three-Run Replication.
Before a single token is generated, we write the research questions, the hypotheses, the operational definitions, the measurement approach, and the decision criteria. This document is timestamped and delivered to the client. Findings that contradict hypotheses are reported with equal weight to confirming findings.
Why it matters: The single biggest critique of synthetic research is post-hoc cherry-picking — running studies until you get the answer the client wants. Pre-registration is the only defensible answer. It is the same standard major peer-reviewed behavioral journals now require of human-subject studies.
No audit reaches a client or a cold-outbound prospect without human review. The reviewer can modify, remove, or add findings; that diff is logged in the audit review log. Every outbound email includes the line: "A behavioral architect reviewed this audit before sending. — Rob"
Why it matters: Three-Run Replication catches model-family bias. Concordance tagging catches singleton hallucinations. But neither catches "all three models confidently arrived at the same wrong thing." Named-human review is the third leg of the stool. The signature is non-anonymous on purpose — accountability sits on a real person.
Before any synthetic finding is reported, we search for published human-subject studies on related behaviors. Where real-world data exists, we compare. Where the synthetic finding diverges from established literature, we flag it and either investigate the divergence or downgrade the finding's confidence.
Why it matters: Synthetic populations are trained on human-generated text — they should broadly reproduce human findings on well-studied questions. When they don't, something is wrong with the prompt, the population, or (occasionally) the literature. Either is worth knowing.
Every report includes an appendix with the complete prompt chain used, the persona specifications sampled, the models invoked, the seed parameters, and the raw model outputs. A client (or a skeptical reviewer) can re-run the study at any time. Nothing is black-boxed.
Why it matters: Reproducibility is the cornerstone of credible research. Black-box synthetic research is unreviewable and therefore unfalsifiable. We hold ourselves to the same reproducibility standard as peer-reviewed behavioral science.
Before any finding enters the client report, we run it through a dedicated adversarial critique prompt: "You are a skeptical behavioral scientist. Identify every way this finding could be wrong, biased, or overreaching." Findings that survive the critique are reported. Findings that don't are revised or downgraded.
Why it matters: The hardest failure mode of LLM-assisted research is plausibility-without-truth. Findings look correct because they are written correctly. Adversarial prompting is the one technique that reliably surfaces hidden weaknesses.
A blanket sign-off rule is either too strict (caps volume on low-stakes artifacts) or too loose (lets high-stakes studies ship unreviewed). Canary uses three disclosed tiers, matched to what the deliverable is actually being used for. The line between automated and human-verified is binary — there is no '1-in-5' spot-check, no hidden sampling, no ambiguity.
Why it matters: The three tiers make the moat visible in the price. The $149 tier competes with commodity AI audits on speed and Consensus rigor. The $249 and $2,500+ tiers add what no commodity tool can: a named human accountable for every finding. Buyers self-select by how much human judgment their decision warrants. Honest tiering beats a vague blanket claim every time.
Canary's external validity claim rests on three continuously-running proofs. None of them require us to design self-funded human studies that would be biased toward validating our own method.
Why it matters: The question "does this work?" deserves an honest answer, and the honest way to answer it is with real data from real clients and published canonical research — not self-funded human-subject studies designed to validate our own method. All three phases above accumulate credibility with every study we ship, instead of requiring quarterly research sprints that cost weeks of analyst time we do not have.
Every Canary scope document includes a section titled "What this study cannot tell you." This is not a disclaimer buried in fine print — it is a centerpiece of the contract. If a client question lives in our honest-limits zone, we decline the engagement or scope it differently.
Why it matters: The firms that blow up will be the ones that overpromise. We would rather say "this question requires human subjects, here is a partner who can run that" than take money for a question we can't answer. Long-term, this is the only defensible way to operate.
Academic validation of LLM-based behavioral simulation is maturing rapidly. The most rigorous published studies we draw on:
| Study | Finding | Relevance |
|---|---|---|
| Argyle et al. (2023) — PNAS | GPT-3 conditioned on demographic profiles reproduces voting behavior with fidelity comparable to traditional polling | Demographic conditioning is a valid technique for simulating population-level preferences |
| Dillion, Tandon, Gu, Gray (2024) — Trends in Cognitive Sciences | GPT-4 moral judgments correlate with human judgments at r = 0.95 across 464 scenarios | LLMs can substitute for human participants in many moral / preference tasks |
| Horton (2023) — homo silicus | LLMs reproduce classic behavioral economics findings including loss aversion, social preferences, and fairness | Core behavioral phenomena replicate in synthetic populations |
| Aher, Arriaga, Kalai (2023) — ICML | LLMs reproduce Milgram, Ultimatum, and Wisdom-of-Crowds experiments quantitatively, with some directional drift | Magnitudes are less reliable than directions — a principle baked into our reporting standard |
| Park et al. (2023) — Generative Agents | LLM agents in a simulated town produce emergent social behavior reviewers find plausible | Multi-agent synthetic populations can produce interaction dynamics useful for qualitative insight |
The literature is clear: LLMs are directionally reliable on many behavioral outcome classes and magnitude-unreliable on most. Our methodology is calibrated to this reality. We report directions confidently and magnitudes as ranges.
Canary commitment: We publish two live validity metrics on our public dashboard — our literature-correlation rate (Canary findings vs. independently published UX critiques on well-studied products) and our Implementation Outcome rate (the percentage of voluntary 30-day client check-ins reporting improvement). Both numbers update as we ship. If either declines, clients see it. We trust continuous calibration on real work more than quarterly calibration on contrived work.
What Canary is. A friction sweeper / heuristic detector for linear, decision-heavy flows: checkout, signup, pricing pages, onboarding, account setup. On those flows, against canonical UX findings, Canary surfaces a high share of known issues fast and cheap. Three-Run Replication, concordance tagging, and named-human review are how we make that claim defensible.
When we encounter a client question in this zone, we say so, and (where possible) refer to a partner who does human-subject research. The referral is our credibility asset.
Every Canary study ships as a single document with the same structure. Clients know what they are getting. Reviewers can audit our process.
Canary operates under research-ethics principles consistent with our Tri-Council (TCPS2) and CITI Human Subjects Research training:
Our analysts came to this work through behavioral science, experimental design, and field operations where bad data killed projects and careers. We have an operational allergy to methodological shortcuts.
Synthetic research done sloppily is worse than no research. It wears the costume of rigor while laundering a single model's biases into an expensive-looking PDF. We will not build a business on that.
Canary Research is our bet that synthetic friction-detection done correctly — replicated three ways across model families, concordance-tagged, landmark-replicated against Baymard and NN/g, named-human reviewed before delivery, and honest about its limits as a complement to deep human research — is a useful, defensible instrument for the class of friction questions where heuristic evaluation already works. We intend to be the firm that ships it that way.
Every report we ship is one more piece of evidence that we're right. Or one more lesson that we're not. Either is better than not knowing.
Canary Research · Methodology Team
Okanagan, BC, Canada · April 2026