AI Auto-Execution vs Manual Orders: A 30-Day Evaluation Method
A fair comparison starts by freezing the same signals, position rules, and cost treatment, then retaining real order logs from both execution paths. This article gives a practical 30-day logging and comparison method.
1. Freeze the comparison rules #
Before day one, assign every signal a signal_id, UTC timestamp, and strategy version. Both paths must receive the same market, side, order type, quantity, target price, stop, take-profit, expiry, and cancellation rule at the same time. Starting equity, leverage, margin mode, fee tier, and trading hours must also match. Report spot and futures separately.
| Field group | Freeze in advance |
|---|---|
| Signal | ID, UTC time, strategy version or file hash |
| Order | Market, symbol, side, type, quantity, target price |
| Risk | Stop, take-profit, leverage, expiry, partial-fill rule |
| Failures | Retry limit, rejection, disconnect, and manual-override rules |
| Costs | Actual fees, funding, and any other recorded costs |
If the system merely routes fixed instructions, call it an automated executor. That keeps signal quality separate from order-routing performance.
2. Required order logs #
Keep a row for every attempt, including rejections and unfilled orders. Record signal_id; received, submitted, and exchange-acknowledged times in UTC; order ID; side and type; requested price and quantity; filled quantity; VWAP; status or rejection code; retry count; cancellation time; disconnect interval; fee asset and amount; funding; any manual override and reason; and the entry and exit benchmarks.
An expired signal with no order is missed execution. A rejected order is an execution failure. Calculate a partial fill only on filled quantity. If order ID, fill price, or costs are missing, do not backfill them with a candle close. Exclude that signal from net-PnL comparison if necessary, but keep it in missing-data and failure rates.
3. Compare signal by signal #
| Metric | Treatment |
|---|---|
| Coverage | Eligible, submitted, acknowledged, and filled signal counts |
| Latency | Submit time minus receive time; report median and 95th percentile |
| Fill quality | Median and 95th-percentile slippage, partial-fill and rejection rates |
| Costs | Actual fees, funding, and other recorded costs |
| Net outcome | Calculate from actual fills; pair only signals with complete data on both paths |
Use the same valuation frequency and timezone for drawdown. Minute-by-minute unrealized PnL on one path cannot be compared with end-of-day balances on the other.
4. Slippage and failed orders #
Freeze a reproducible benchmark, such as the market midpoint recorded when the signal entered the system. Buy slippage is (VWAP − benchmark) ÷ benchmark; sell slippage is (benchmark − VWAP) ÷ benchmark. Multiply by 10,000 for basis points; a positive number means a worse fill.
Automation does not remove slippage. Market movement, liquidity, and order size can move the fill away from the expected price, as the Binance Academy slippage glossary explains. Log disconnect start and end without assuming a fill, retain rejection codes, and set an identical retry limit before the test.
5. A reproducible example #
These are constructed inputs, not account records or a Binance fee schedule: a 0.01 BTC linear-contract long, entry benchmark 60,000 USDT, actual entry VWAP 60,030, exit VWAP 60,600, entry fee 0.60 USDT, exit fee 0.61 USDT, and funding 0.20 USDT.
- Entry slippage:
(60,030 − 60,000) ÷ 60,000 × 10,000 = 5 bps. - Gross PnL:
0.01 × (60,600 − 60,030) = 5.70 USDT. - Net PnL:
5.70 − 0.60 − 0.61 − 0.20 = 4.29 USDT.
Apply the same formula to each path, then pair by signal ID. This example establishes the calculation only; it does not show that either execution method is more profitable.
6. Limits on conclusions #
After 30 days, inspect completeness first. If too few signals are comparable, symbols or fee tiers differ, or one path has many missing fields, report the limitation and do not declare a winner. Even a complete month describes only that period, signal set, and execution policy.
Automation may reduce latency, but software, permissions, or network faults can also miss orders. A person may catch an anomaly or submit late. Only paired real records can show which process is more reliable; discipline, intelligence, and profit are not preset outcomes.
— AI Trade Lab, revised Sep 24, 2026