Walk Forward testing – some thoughts

The numbers below are from a weekly system I have been playing with. It’s not really a complex system, although it does use ranking of stocks potentially to boost results – something I am still contemplating as a strategy.

The buy signal initially only used one indicator – the one optimised in this test. I then added in extra criteria, but only when the index was bearish as determined by a moving average. I then added a ranking criteria based on momentum – the system takes the strongest stocks in terms of momentum. So the breakout criteria optimised in these tests is a major component since when the market is bullish it is the only buy criteria used.

The table below shows out of sample only results for 3 walk forward simulations. The walk forward used optimisation of a parameter used in an indicator, a dummy parameter with the real parameter fixed throughout the test and, in the final test the indicator that used the real parameter was deleted from the buy signal altogether. The optimisation metric used was profit factor in all tests, although it’s sort of irrelevant if the test is optimising a dummy value.



Working on the hypothesis that the in-sample tests are irrelevant, and to save words, I have left the in-sample results out of the above table.

The orange colored rows are using a dummy walk forward optimisation variable with the parameter / indicator removed altogether from the buy signal, instead of being a fixed value as in the white rows, or the optimised parameter in the yellow rows.

I also graphed CAR% below to get a better picture of the results.


So - the questions are;
1. Does the buy parameter / indicator being tested make any difference?
2. Is the system robust?
3. How valuable has the walk forward process been?

Q1 Whilst at first glance it doesn't make much difference look at the Maximum System Drawdown column, especially the last row when drawdown hit 57%. Whilst the parameter often doesn’t do a lot it was certainly worth having in the buy signal during the Global Financial Crisis. Exposure to the market was also often higher than the other tests when the indicator was removed, as was the number of trades taken.

Q2 The system's robustness is something that I would test using Monte Carlo analysis for further verification, although the system performed well over a range of different market conditions. I would like to understand what path the ranking is leading me on through the range of possible paths through the market.

Q3 I get a lot out of walk forward, but not in the sense that some people might. I am not keen on changing indicator parameters on a regular basis, but I have no real proof that my lack of keenness is warranted. But as a method of testing a strategy over a range of different market conditions and portfolio start-up dates it certainly is useful.

Without going into a lot of statistics I would be happy to say that there is no significant difference HHV optimised and changed every walk forward period and HHV "intelligently" fixed, however taking the parameter out altogether….no.

stevo