Stop health hype at its source.

Most health hype is written before a journalist ever sees it. A third of university press releases claim more than their study found, and most news repeats them. HypeCheck checks every sentence against the paper before the release goes out, has Claude rewrite what overclaims, and checks the rewrite again.

See it on 85 real NIH releases
  • 6.7% false alarms, against 30–45% for Claude asked the same question
  • ≈219,000 people stopped a heart medicine after one news storm. The hype starts in the press release
  • Run on text nobody chose for us: 85 real NIH releases and 170 AI summaries
Replay of a real check, from the research data
What the paper concluded
“Use of the urgent referral pathway could be efficacious.”
What the press release says
Use of the urgent referral pathway is saving lives, according to a study in The BMJ.

One overclaim travels from a press office to millions of readers.

The research is clear about where health hype starts, and that fixing it at the release works. HypeCheck sits at the one point where a single correction changes everything downstream.

At the end of this chain is someone at a kitchen table who reads that their pills might do more harm than good and quietly stops taking them. Fix the sentence before it leaves the press office, and that reader never sees it.

  1. The paper
    “could be”

    The study finds an association or a possible effect.

  2. HypeCheck checks here
    The press release
    33%

    of 462 university releases turned correlation into causation.

    is saving lives
  3. The news
    81%

    of news stories repeated the causal claim when the release made it, vs 18% when it didn't.

  4. The public
    ≈219,000

    UK patients estimated to have stopped statins after one news storm.

Fixing the release fixes the news, and costs nothing.

A randomised trial in nine real press offices aligned releases with the evidence: news headlines matched the evidence far more often, and coverage did not drop (Adams et al., BMC Medicine 2019). The free human service that reviewed releases closed in 2018; nothing has automated it until now.

A small model judges. A large model writes.

Claude can be talked out of a “no”; a decision model can't. It only returns a probability, and plain code decides what happens next.

  1. Claude on Amazon Bedrock

    Tries to publish

    The agent calls its publish_release tool with the draft.

  2. Strands Decider 2B

    Rates every claim

    no link → linked → may cause → causes, for the paper and each sentence.

  3. Claude on Amazon Bedrock

    Rewrites what's blocked

    Only the flagged sentences go back, with the reason.

  4. Strands Decider 2B

    Re-checks, then publishes

    The rewrite faces the same gate. After 3 tries a person decides.

See the AWS architecture

Amazon CloudFront in front of an EC2 instance running the web app and Strands Decider; the agent runs on Amazon Bedrock AgentCore Runtime and calls Claude through Amazon Bedrock.

Browser to CloudFront to the EC2 web app; the agent on AgentCore Runtime calls Claude on Bedrock and Decider on EC2 through a key-guarded endpoint whose key is in SSM Parameter Store.

Watch it catch an overclaim, live.

Pick a real press release from the research data, or paste your own with the paper's conclusion.

or
Where the agent runs
Block Review OK score

Results appear here, sentence by sentence.
Start with an example on the left.

A gate that cries wolf gets switched off.

On 563 expert-labelled press releases we did not write, every Claude model asked to judge flagged far more honest releases than Decider with a calibrated threshold. All experiments were pre-registered.

Honest press releases wrongly flagged as exaggerated

345 releases experts judged faithful to their paper · lower is better · hover a bar for detail

0%20%40%60%
Every 1,000 honest sentences: Claude Sonnet wrongly blocks 296 and costs $0.97. HypeCheck wrongly blocks 67 and costs $0.18 ($0 on your own GPU).
7.7% vs ≤10% promised

Calibrated threshold, checked on 441 pairs it had never seen.

0.498 best published 0.474 (a tie)

Overall accuracy (macro-F1), with no training on the task.

0.72 vs 0.50 for Bedrock Guardrails

Amazon's own grounding check ranks exaggerations at chance on these pairs; it catches invented facts, not inflated ones.

0.84

Spotting advice stronger than the paper, on 116 real 2011 releases (AUROC).

Full results table (all numbers, including cost and speed)
What we measuredHypeCheck (Decider 2B)Comparison
Honest releases wrongly flagged6.7%Haiku 4.5 44.9% · Sonnet 4.6 29.6% · Opus 4.6 40.0%
Promise vs reality (conformal)≤10% promised → 7.7% measured95% CI 5.6–10.6%
Exaggerations caught at that strictness39%Haiku at a similar false-flag level: 18%
Overall accuracy (macro-F1)0.498best published 0.4735 · Haiku 0.358 · Sonnet 0.401 · Opus 0.460
Ranking quality (AUROC)0.716Haiku 0.570 · Sonnet 0.580 · Opus 0.663
Amazon Bedrock Guardrails grounding check, same pairs (AWS-native baseline)AUROC 0.716Guardrails AUROC 0.503 (chance) · recall 13.9% vs our 38.5%
Advice stronger than the paper, 116 real releasesAUROC 0.84—
Speed per decision160 ms (laptop GPU) · ~0.9 s (this CPU server)Haiku ~2.0 s · Sonnet ~0.9 s · Opus ~5.3 s
Cost per sentence checked$0.00018 (CPU server) · $0 (own GPU)Haiku $0.00051 · Sonnet $0.00097 · Opus $0.0056

Limits we measured

  • At the strict threshold it catches 39% of exaggerations (95% CI 30–47%): a safety net, not an editor.
  • The animal-to-human hint is weak (AUROC 0.61 on 45 releases), so it never blocks.
  • 563 test pairs give wide intervals: accuracy is a tie with the published best (0.4735) and with Opus. The false-alarm gap to every Claude model is far outside chance.
  • This demo server runs Decider on a CPU, about six times slower than a GPU.

Then we ran it on text nobody chose for us.

A benchmark can flatter a model. So the same check runs every morning on real press releases from the US National Institutes of Health, and we pointed it at AI-written summaries of the same papers. In 2014, Ben Goldacre proposed linking every press release to its paper to create “an information trail, and accountability among peers and the public”. HypeWatch automates part of that.

HypeWatch: real NIH releases, checked against their papers

Every NIH research release since 2023 that cites its paper by DOI, with the paper's own words, Claude's suggested rewrite and Decider's re-check beside every flag.

  • 85 releases checked, 615 sentences scored
  • 15% had a sentence the gate would block
  • New releases are checked automatically each morning
Open HypeWatch

AI summaries of science overclaim too

AI summaries were nearly five times as likely as expert-written ones to state findings more broadly than the paper . We asked Claude to summarise the same 85 papers.

Block“…expanding access to opioid use disorder medications in jails can save lives…” Claude Sonnet 4.6
OK“…may be associated with reduced risks of overdose, death, and reincarceration…” after the gate

The paper: treatment in jail “was associated with” lower risks.

  • 1 in 9 Claude summaries had a sentence the gate blocks
  • 9 in 10 of those were fixed by Claude's rewrite and Decider's re-check
  • 1 of 4 pre-registered predictions missed, and reported
Method and all 170 summaries

Drop the same gate into any agent.

Anything that must not say more than its source: summaries, RAG answers, patient leaflets, release notes. Use the open-source gate below, call the free REST API, connect over MCP, calibrate on your own data, or deploy into your own AWS account. Get a free API key →

# pip install strands-agents · strands-decider serve … --port 8099 from decision_pattern import Decider, ClaimFidelityGate, load_threshold gate = ClaimFidelityGate( Decider(), source=lambda: paper_conclusion, # what the text must not exceed tool_name="publish_release", text_arg="text", threshold=load_threshold("calibration.json", "0.1"), # BLOCK → Guide (rewrite) review_threshold=load_threshold("calibration.json", "0.2"), # REVIEW → listed, not blocked ) agent = Agent(model="us.anthropic.claude-sonnet-4-6", tools=[publish_release], interventions=[gate])

Questions, answered plainly.

What is Strands Decider 2B?

A small, open decision model from AWS Strands Labs (Apache-2.0). It doesn't write text: give it text and a typed question (yes/no, pick one, or a score) and it returns an answer with a calibrated probability in a fraction of a second, on hardware you already own.

What does Amazon Bedrock do here?

Bedrock serves Claude, which drives the agent and rewrites flagged sentences. The agent itself runs on Amazon Bedrock AgentCore Runtime.

Why not just ask Claude whether the release exaggerates?

We measured it: on the same 563 pairs, Claude flagged 29.6–44.9% of faithful releases depending on the model, Opus included. Decider with a calibrated threshold flagged 6.7%. Claude also can't be trusted to check its own rewrite, so Decider does.

What is “conformal calibration”?

A standard statistical recipe for turning model scores into a cut-off with a guarantee: set it on known-faithful examples so at most 10% would be flagged. If new text resembles the calibration set, the same rate holds; we measured 7.7%.

What do BLOCK, REVIEW and OK mean?

BLOCK: above the strict threshold; Claude must rewrite. REVIEW: between the two thresholds; not blocked, but listed for an editor. OK: below both.

Where does it run on AWS?

The page and Strands Decider run on Amazon EC2 behind Amazon CloudFront. The agent runs on Amazon Bedrock AgentCore Runtime (or in-process on EC2 if you choose), calls Claude through Amazon Bedrock, and reaches Decider through a key-protected endpoint whose key lives in AWS Systems Manager Parameter Store.

Is my text stored?

Only a published release is written to the server's outbox; drafts are not kept.