Treat AI Lab as an experiment space
AI Lab helps organise research around a strategy and a baseline result. Choose an explainable experiment: understanding losses, studying a variant or comparing a market context when those actions are available.
A proposal deserves testing when it answers a specific question. An AI suggestion or the best result on one period is not, by itself, demonstrated improvement.
Prepare a usable baseline
Check that the strategy is saved, data is associated and the baseline test corresponds to the intended version. Review dates, costs and trade count before requesting a comparison.
If the strategy changed after the last test, update the baseline first. Otherwise, a different starting state can be mistaken for the effect of the variant.
Define the objective first
| Objective | Example question | Trade-off to inspect |
|---|---|---|
| Understand losses | Are losses concentrated in a context? | Observations per group |
| Reduce drawdown | Does a filter reduce losing sequences? | Gains and opportunities also removed |
| Simplify | Does a condition contribute useful information? | Behaviour on another period |
| Study costs | Does the signal remain useful with higher costs? | Difference from baseline |
Choose a primary criterion and a few control measures. Trying to maximise every metric makes the decision difficult to explain.
Design a clear comparison
- Name the hypothesis and baseline version.
- Define one change or a limited set of values.
- Keep data and costs identical for the initial comparison.
- Record net result, drawdown and trade count.
- Inspect the operations responsible for the difference.
- Check the selected variant on a separate period.
Suggest an experiment to reduce concentration of losses.
Keep the baseline unchanged and describe one variant.
State the change, expected effect and rejection criterion.
Compare net result, drawdown, trade count and distribution of losses.
Separate measured results from work still to test.Interpret an apparent improvement
A variant can make more profit with more risk, or reduce drawdown with far fewer operations. Describe that trade-off explicitly. A single metric cannot make the decision for you.
A precise setting that works only at one value and deteriorates nearby needs investigation. Trying neighbouring values can show a narrow observation. It does not establish that the effect will persist on new data.
Keep experiments and decisions
Hypothesis: session filter.
Decision: retain / reject / insufficient information.
Reason: report values and observed trade-off.
Remaining check: another period, costs or a larger sample.Use Stress testing for less favourable assumptions and Analytics to deepen your reading of results.