Can AI Beat Polymarket? A Time-Locked Trading Experiment

“Can AI beat Polymarket?” sounds like one question. It is actually at least three:

Can an AI produce better-calibrated probabilities than the market price?
Can it identify differences large enough to survive spread, slippage, and fees?
Can a fixed process do that repeatedly on information available at the time?

A screenshot of one winning forecast answers none of them. Neither does asking a modern model to revisit an old market with unrestricted web access. The model may know the outcome, quote a later article, or benefit from training data published after the decision. That is not forecasting; it is answer leakage.

This article defines the experiment before running it. There are no claimed results here. The objective is a protocol another researcher could reproduce, criticize, and rerun.

The hypothesis must be narrow

Start with a testable statement:

Using only a fixed evidence packet available at checkpoint time, the AI produces probabilities that are better calibrated than a defined Polymarket price benchmark, and any simulated trading edge remains after documented execution costs.

“Better” needs pre-registered measurements. “The AI made more money” is too weak on its own because a few concentrated bets can dominate P&L. “The AI was right more often” is also weak because a forecaster can say 51% for every eventual Yes and ignore confidence quality.

Use forecast metrics as the primary result. Treat simulated P&L as a secondary, assumption-sensitive result.

1. Freeze the market sample before seeing outcomes

Write inclusion rules first. For example:

binary Yes/No markets with unambiguous resolution rules;
a published primary resolution source;
a minimum history of usable price and order-book data;
checkpoints at least a specified time before scheduled resolution;
no markets whose outcome was already effectively certain at intake;
one defined category and date range.

Then publish the selected market IDs and exclusion reasons. Do not quietly delete a market because the model misunderstood it or add an easy winner after seeing the outcome.

Polymarket’s Gamma API is the primary public API for discovering markets and events. Archive each market’s full question, description, resolution source, end date, outcome labels, and token IDs—not merely its headline.

The contract is part of the forecast target. Polymarket’s resolution documentation says rules define the source, end date, and handling of edge cases. It also describes how proposals can be disputed and rare clarifications can be published. Record the rules known at each checkpoint and separately record later changes.

2. Define point-in-time checkpoints

Choose checkpoints mechanically, such as 30 days, seven days, 24 hours, and one hour before the market’s scheduled end. Event-driven checkpoints can also work, but the trigger must be defined in advance.

At each checkpoint, create a sealed bundle:

/market.json
/rules.html
/book-yes.json
/book-no.json
/price-history.json
/evidence/index.json
/prompt.txt
/model-config.json
/manifest.sha256

Each source in evidence/index.json needs a publication time, retrieval time, canonical URL, publisher, and local content hash. If a source cannot be shown to have existed before the checkpoint, exclude it.

Polymarket’s public CLOB order-book and price-history methods provide token-level bids, asks, spreads, and historical price points. A current price-history response is not automatically a perfect historical order-book reconstruction. If you did not archive depth at the checkpoint, say so; do not infer it with fake precision.

3. Prevent leakage at every layer

Leakage can enter through more than news search:

the model may have the result in its training data;
a webpage may have been updated after resolution;
a URL, snippet, metadata field, or filename may reveal the winner;
Polymarket market records may now contain winning outcome fields;
the experimenter may tune prompts after reviewing failures;
a tool may silently fetch current search results.

The strongest design uses genuinely forward-collected forecasts: run the model before outcomes occur and timestamp the sealed output. For historical replay, use a model and tool environment that cannot access the open web, remove resolution fields, and include only archived pre-checkpoint material. Historical replay remains less convincing than prospective collection because model pretraining contamination may be impossible to rule out completely.

Pre-register the prompt, model identifier, temperature, tool access, retry policy, and what happens when output is invalid. One forecast per checkpoint is cleaner than sampling repeatedly and selecting the most persuasive answer.

4. Require a probability—not a pick

The model should return a machine-readable record:

{
  "p_yes": 0.61,
  "range_low": 0.48,
  "range_high": 0.72,
  "decision": "abstain",
  "reasons": ["..."],
  "counterevidence": ["..."],
  "unknowns": ["..."]
}

This example is a format, not a real forecast. Reject probabilities outside 0–1 and outputs that do not cite evidence IDs from the sealed packet. Do not let the model rewrite the contract. If it cannot explain the resolution rule in plain language, mark the forecast invalid or require abstention under a pre-registered policy.

5. Compare against a fair market benchmark

Decide what “the Polymarket probability” means. The prices and order-book guide explains that the displayed price is normally the midpoint between best bid and ask, but the last traded price may be displayed when the spread is wide. Those are not interchangeable.

For forecast scoring, use a documented checkpoint benchmark such as the contemporaneous midpoint when both sides are available. For simulated execution, use the actual visible bid/ask depth—not the midpoint. If the best Yes bid is 0.54 and ask is 0.60, the market forecast benchmark might be 0.57, while a Yes buyer’s starting cost is 0.60.

Also define stale-book handling. If the snapshot is too old or one side is absent, the checkpoint may be unscorable for trading even if it remains usable for forecast analysis.

6. Score accuracy with Brier score and calibration

For a binary outcome, the Brier score is:

Brier = (forecast probability - outcome)²

The outcome is 1 for Yes and 0 for No. Lower is better. If a forecast assigns 0.80 to Yes and Yes occurs, its score is 0.04. If No occurs, its score is 0.64. These are mathematical examples, not experiment results.

Report mean Brier score for the AI and the pre-declared market benchmark over the same eligible checkpoints. Use paired uncertainty estimates or bootstrap intervals rather than declaring victory from a tiny numerical difference.

Calibration asks a different question: when the agent says 70%, do roughly 70% of those cases resolve Yes? Group forecasts into pre-declared probability bins, then show predicted probability against observed frequency. Use enough observations per bin and publish counts; a beautiful calibration line based on three cases is not persuasive.

Sharpness also matters. An always-50% forecaster can look cautious but adds little information. Report the distribution of forecasts alongside calibration.

7. Treat abstention as a real output

An agent should be allowed to say “I do not have enough reliable evidence.” Pre-register abstention rules and score:

coverage: share of eligible checkpoints receiving a forecast or trade;
accuracy and Brier score on covered cases;
abstention rate by reason;
performance at several minimum-edge thresholds.

Do not let abstention become a way to erase losing forecasts after the fact. The decision must be committed before resolution. Compare systems at similar coverage, or show the full coverage-versus-quality curve.

8. Simulate execution without pretending it happened

Only generate a paper trade when the agent’s estimated advantage exceeds a fixed buffer. Deterministic code should choose side and size, with position caps defined before the test. Our paper-trading build guide shows the safe data flow.

Walk the visible book to estimate a volume-weighted fill. Apply the market’s documented fee parameters; Polymarket’s fee guide states that fee status and rates vary by market. Record latency scenarios and reject any size that exceeds your visible-depth rule.

For a simulated Yes purchase, an eventual binary payoff model can be summarized as:

paper P&L = shares × (resolved value - simulated fill price)
            - simulated fees - other declared costs

That formula is not evidence that the order could have filled. Passive limit orders have unknown queue placement. Use conservative, base, and optimistic scenarios, and label them. See Spread, Slippage and Partial Fills for the execution audit.

9. Publish the failures, not just the leaderboard

A credible result package includes:

the pre-registration and complete market list;
all sealed input manifests;
prompts, model versions, and raw outputs;
invalid responses and abstentions;
market benchmark calculations;
Brier and calibration code;
every fill assumption and fee parameter;
sensitivity tests for size, latency, and minimum edge;
known sources of contamination or missing data.

Useful failure categories include wrong contract interpretation, stale evidence, overconfidence, underconfidence, unsupported causal stories, and forecast edge erased by execution. Those diagnoses are more valuable than a single P&L total.

What would count as evidence?

One market proves almost nothing. A result becomes interesting when it survives a sufficiently broad prospective sample, fixed prompts, honest coverage reporting, paired comparison with contemporaneous odds, and realistic cost sensitivity. It becomes stronger when an independent party can reproduce the calculation from the archived bundle.

Even then, “beat Polymarket” should be bounded by the tested markets, period, model, size, and execution assumptions. It does not mean the system will work in a new category or at larger capital.

For venue background, read our Polymarket exchange review. For related methods, visit Technical Analysis and our comparison of crypto charts versus prediction-market odds.

AI versus Polymarket FAQ

Can AI beat Polymarket?

It is an empirical question that requires a broad prospective sample, sealed point-in-time inputs, fixed systems, honest coverage and realistic execution costs.

Why freeze the market sample before outcomes?

Pre-selection prevents researchers from choosing only easy, interesting or successful markets after knowing how events resolved.

What is a time-locked forecast?

It is a probability and evidence record committed at a declared checkpoint before later news or the outcome can enter the system.

How can answer leakage enter an AI forecast test?

Leakage can come from revised webpages, search snippets, model training cutoffs, resolved API fields, cached summaries or later rule clarifications.

Why require a probability instead of Yes or No?

Probabilities reveal confidence and support calibration and proper scoring. A forced binary pick hides whether the model estimated 51% or 99%.

What is the fair Polymarket benchmark?

Define it in advance from contemporaneous executable or displayed data, accounting for whether the interface showed midpoint or last trade and preserving spread.

What is the Brier score?

For a binary event it is the squared difference between forecast probability and outcome, rewarding accurate, well-calibrated probabilities and penalizing confident errors.

How should abstentions be evaluated?

Commit them before resolution, record reasons, report coverage and compare forecast quality at similar coverage or across a coverage-quality curve.

Does a good forecast imply a profitable Polymarket trade?

No. Spread, fees, slippage, latency, visible depth, queue uncertainty and position sizing can erase a forecasting advantage.

What evidence would support an AI edge?

The result should survive many prospective markets, fixed prompts, paired comparison with contemporaneous odds, cost sensitivity and independent reproduction from the archive.

Sources and further reading

Risk disclosure: This is an experimental design, not a claim of profitable results or investment advice. Historical replays can contain hidden look-ahead bias, archived data can be incomplete, model outputs can be wrong, and simulated fills can materially overstate real execution. Prediction markets involve loss, technology, contract-resolution, liquidity, and legal-access risks. Do not risk funds based on an unvalidated AI forecast.