If you’ve already tried designing your strategy you’ll have noticed a small check-box in the historical tester of the jforex platform named optimization. You can find a little bit about this option on the dukascopy official forums but what is optimization exactly and how does it work? If I optimize my strategy will it necessarily work better in the future? These are some questions I want to tackle in this short article because optimization can improve a strategy or render it worthless while falsely making it look good on back-tests.
Over-learning is the enemy of optimization and happens very easily when optimizing a strategy. An over-learned strategy will have a magnificent back-test but will be worthless in the future.
Optimizing a strategy means tweaking its parameters to obtain better results. Especially when a strategy has many such parameters, it is very easy to optimize it on any period in time, even very long. Why is that? It’s because the strategy learns the time period by heart. It can tell when an investment is going to be juicy because it has seen it before! This is called over-learning because your strategy adapts to the time-period it’s optimized on too well. The only worthwhile back-test is done on a period that was not looked at during optimization.
So when and if you buy a strategy on the internet that shows amazing results on back-tests, you should always ask on what period and instruments this strategy was optimized. And this is always true, whether the optimization was done by a computer algorithm or a human brain.
How to accurately estimate future profits
Below is an idealized graph of what happens when you optimize a strategy. Each point on the x-axis is a strategy produced by an optimization process trying to maximize profit. The y-axis gives the performance of the strategy (profit over a specific period of time).
The green line shows the performance on the training data, the time-period on which the strategy is optimized. The red line shows the performance on the validation data, a separate time-period.
When optimizing or “training” many types of systems, the performance on the training period will rise to a maximum but if we look at the performance of the systems on a different period we see that the optimum is reached earlier (dotted line). On the right of this line, the system is over-learning. This occurs because of bias, a type of error that occurs when the systems learn the information specific to the training data that is different from other sets of similar data. By keeping track of the systems performances on a validation period, one can pin-point the best system. We still need to test the system on a separate test period to have an idea of expected profits. This cannot be done on the validation period since we kept track of the results on it and thus we would be doing statistical cherry picking.
This example is simply to explain that optimization, whatever the type of optimization process used, must be used properly. Next is another way to estimate your profits that allows you to use all of your data.
Cross-validation is a technique used especially when the amount of data, or in this case trades, is small and we don’t want to use half of our data on a test set.
What you need is several sets of data, for example ten different instruments. Optimize your strategy on 9 of them and test it on the remaining instrument. Repeat the process 9 more times, each time isolating a different instrument for testing, and sum the ten resulting profits. This will give you an estimate of the profits you can expect when you eventually optimize over all instruments at once.
Note that this is independent of the earlier validation process that prevents over-learning but you can use both tactics together.
When we optimize a strategy, it isn’t wise to optimize on profit alone. Imagine a strategy that would create huge profits and equally huge drawdowns. You can end a given time period with a good profit but it may just as well have ended on a drawdown had the time period been slightly longer or shorter. The two optimization functions I often use are the ratio of profits over drawdowns and the ratio of mean profit per trade over the standard deviation. I discovered the later in an article from the august 2011 contest. The title is Sharpe’ning up your strategy by randomwraith. For these values to have real meaning you must impose a certain amount of trades, instinctively I use somewhere around two hundred trades minimum but there are statistical methods to calculate the exact number necessary to determine a confidence interval for a certain degree of confidence. Here is a link that explains all this and provides a calculator for the confidence interval given the size of your population; http://www.surveysystem.com/sscalc.htm.
Thank you for reading, In my next article I will introduce neural networks and especially the recurrent neural networks like the Long Short Term Memory model that is state of the art and that can be applied to the market with surprising performances.