Skip to content
Guides
FROpen Studio

Test and understand

Compare in AI Lab

Evaluate a change against a baseline using criteria chosen in advance.

3 min read · 6 sections
Contents of this guide

Treat AI Lab as an experiment space

AI Lab helps organise research around a strategy and a baseline result. Choose an explainable experiment: understanding losses, studying a variant or comparing a market context when those actions are available.

A proposal deserves testing when it answers a specific question. An AI suggestion or the best result on one period is not, by itself, demonstrated improvement.

Prepare a usable baseline

Check that the strategy is saved, data is associated and the baseline test corresponds to the intended version. Review dates, costs and trade count before requesting a comparison.

If the strategy changed after the last test, update the baseline first. Otherwise, a different starting state can be mistaken for the effect of the variant.

Define the objective first

ObjectiveExample questionTrade-off to inspect
Understand lossesAre losses concentrated in a context?Observations per group
Reduce drawdownDoes a filter reduce losing sequences?Gains and opportunities also removed
SimplifyDoes a condition contribute useful information?Behaviour on another period
Study costsDoes the signal remain useful with higher costs?Difference from baseline

Choose a primary criterion and a few control measures. Trying to maximise every metric makes the decision difficult to explain.

Design a clear comparison

  1. Name the hypothesis and baseline version.
  2. Define one change or a limited set of values.
  3. Keep data and costs identical for the initial comparison.
  4. Record net result, drawdown and trade count.
  5. Inspect the operations responsible for the difference.
  6. Check the selected variant on a separate period.
Example prompt
Suggest an experiment to reduce concentration of losses.
Keep the baseline unchanged and describe one variant.
State the change, expected effect and rejection criterion.
Compare net result, drawdown, trade count and distribution of losses.
Separate measured results from work still to test.

Interpret an apparent improvement

A variant can make more profit with more risk, or reduce drawdown with far fewer operations. Describe that trade-off explicitly. A single metric cannot make the decision for you.

A precise setting that works only at one value and deteriorates nearby needs investigation. Trying neighbouring values can show a narrow observation. It does not establish that the effect will persist on new data.

Keep experiments and decisions

Text
Hypothesis: session filter.
Decision: retain / reject / insufficient information.
Reason: report values and observed trade-off.
Remaining check: another period, costs or a larger sample.

Use Stress testing for less favourable assumptions and Analytics to deepen your reading of results.

Search the guides