Is ChatGPT Accurate on BTC? How to Test Direction Calls

1. How to Design the Test #

Record the model name or version, execution time, complete prompt, input files, whether browsing was enabled, and any material returned by browsing. If the model cannot access current prices, writing “as of today” does not supply the missing data. Save source-linked market material with a clear cutoff so another person can see what the model actually had.

To test the next 48 hours, save the output before the outcome exists and score it after the window closes. Asking about a past date may leak later events already present in the model; disabling browsing does not guarantee that this historical leakage disappears. Fix the prompt and thresholds on earlier data, then stop adjusting them during the evaluation period. The TimeSeriesSplit documentation illustrates ordered train/test splits; a real design must also handle overlap between adjacent 48-hour label windows.

A record you can save:

Record ID: {ID}
Data cutoff: {UTC time}
Materials: {market data, source URLs, retrieval time}
Task: using only the attached material, classify the next 48 hours as up / down / range.
Rule: return > 2% is up; return < −2% is down; otherwise range.
Output: one class, reasons, and material IDs; state clearly when evidence is insufficient.
Do not add real-time prices, news, or on-chain figures that are absent from the material.

Before scoring, fix the source, pair, quote type, and sampling times for the starting price P₀ and the price 48 hours later P₄₈. Compute r = (P₄₈ / P₀ − 1) × 100%. The 2% threshold here is only a teaching choice, not an optimized trading parameter:

• r > 2%: up.
• r < −2%: down.
• −2% ≤ r ≤ 2%: range, including both boundaries.
• Track refusals, invalid formats, and fact-check outcomes separately. Do not remove refusals and report accuracy only on the remainder; report total requests, valid classifications, and coverage.

If both the start and end price are 100, the actual class can only be range. A rule that also gives credit to “up” and “down” would let three contradictory calls score as correct on the same record, making the resulting accuracy meaningless.

2. Score 200 Synthetic Records #

Assume 200 valid classifications, with predicted classes in rows and actual classes in columns. The nine cell counts below were constructed by hand; they do not come from model calls. A real test also needs a reviewable row-level log and a count of invalid outputs.

The diagonal is 58 + 23 + 25 = 106, so overall accuracy is 106 / 200 = 53%. Of the 97 “up” predictions, 58 are correct: 59.8%. The corresponding figures are 23 / 62 = 37.1% for “down” and 25 / 41 = 61.0% for “range.” These are per-predicted-class precision values, not recall. Up recall uses all 85 actual-up cases as its denominator: 58 / 85 = 68.2%.

3. What Should 53% Be Compared With? #

There are three classes, so a 50% coin-flip baseline does not apply. One simple benchmark can be declared before evaluation: always predict up. In this synthetic table it gets 85 records right, or 42.5%. If the benchmark predicts the most frequent class in training data, select that class on the training set and carry it unchanged into the evaluation window. Do not look at the test answers and then choose the best constant prediction. The DummyClassifier documentation defines these simple strategies.

The example's 53% is 10.5 percentage points above that benchmark. This states only an arithmetic relationship inside the table; with no real observations, it says nothing about model effectiveness. A real report also needs its date range, class distribution, and errors. Repeated prompts on the same day and adjacent 48-hour windows share information, so 200 records are not automatically 200 independent samples. Do not attach a confidence interval for independent trials without testing that assumption.

Repeated prompts measure stability. Consider a separate teaching example: 25 groups of the same input, five runs per group. If 18 groups produce the same direction at least four times, the stated consistency rate is 18 / 25 = 72%. The remaining 28% is only the share of groups below that threshold; it is not the probability that the fifth answer flips. “Up, up, up, up, down” meets the four-of-five rule even though the fifth answer differs. Five identical wrong answers would still be highly consistent.

Direction classification and trading profit are separate questions. This table has no entry or exit price, position size, fees, funding, or slippage, so it cannot produce any return estimate.

4. Where to Verify Seven Fact Types #

This is a verification checklist, not a set of errors observed in 200 experiments, and it is not ranked by frequency. Breaking the output into individual factual claims makes it easier to review than assigning one “trustworthy” label to an entire answer.

If no source is available, mark the claim “unverified.” Do not assume it is true, and do not claim an event never happened merely because one search found nothing. When a source conflicts with the output, preserve the original sentence, source, and conflicting field, then label it a factual error so another person can reproduce the decision.

5. How to Verify the Output #

First, check whether the source supports the exact sentence. A page loading successfully is not enough; dates, units, and entities must match. If a news story cites another report, follow the chain to the announcement, data table, or original remarks where possible.

Next, separate fact from inference. “This address transferred funds” is a checkable record. “The holder is preparing to sell” is an interpretation. A correct first sentence does not verify the second.

Finally, recompute anything calculable. Percentages, moving averages, and position changes should retain their inputs and formulas. Repeated prompts can test output stability, but agreement between answers is not independent evidence.

6. Keep a Reproducible Analysis Record #

This method example does not provide ChatGPT's real accuracy for predicting BTC. Answering that question requires running and preserving a defined experiment, then checking for a reproducible difference from a benchmark chosen in advance. One correct direction call also does not establish that every factual claim in the answer was correct.

For everyday research, ask the model to use the format below and then verify each item. The format makes evidence easier to locate; this article did not measure how much it reduces errors. For other task formats, see the Trading Prompt Library.

Materials: [attach each source, cutoff, and data item]
Organize the answer in three columns: facts in the material / your inference / missing evidence.
Give the material ID for every factual claim; do not add current numbers absent from the material.
Show inputs and formulas for calculations.
For every inference, state what evidence would falsify it.
Write “cannot be confirmed from the supplied material” when evidence is insufficient.

Keep the input, raw answer, and review notes together. When you change models or prompts later, you can compare them under the same scoring method instead of deciding from two screenshots with different writing styles.

View the Full Prompt Library →

— PromptDeck, revised 2026-09-24