Tuesday, November 24, 2009

Walk Forward testing – some thoughts

The numbers below are from a weekly system I have been playing with. It’s not really a complex system, although it does use ranking of stocks potentially to boost results – something I am still contemplating as a strategy.

The buy signal initially only used one indicator – the one optimised in this test. I then added in extra criteria, but only when the index was bearish as determined by a moving average. I then added a ranking criteria based on momentum – the system takes the strongest stocks in terms of momentum. So the breakout criteria optimised in these tests is a major component since when the market is bullish it is the only buy criteria used.

The table below shows out of sample only results for 3 walk forward simulations. The walk forward used optimisation of a parameter used in an indicator, a dummy parameter with the real parameter fixed throughout the test and, in the final test the indicator that used the real parameter was deleted from the buy signal altogether. The optimisation metric used was profit factor in all tests, although it’s sort of irrelevant if the test is optimising a dummy value.



Working on the hypothesis that the in-sample tests are irrelevant, and to save words, I have left the in-sample results out of the above table.

The orange colored rows are using a dummy walk forward optimisation variable with the parameter / indicator removed altogether from the buy signal, instead of being a fixed value as in the white rows, or the optimised parameter in the yellow rows.

I also graphed CAR% below to get a better picture of the results.


So - the questions are;
1. Does the buy parameter / indicator being tested make any difference?
2. Is the system robust?
3. How valuable has the walk forward process been?

Q1 Whilst at first glance it doesn't make much difference look at the Maximum System Drawdown column, especially the last row when drawdown hit 57%. Whilst the parameter often doesn’t do a lot it was certainly worth having in the buy signal during the Global Financial Crisis. Exposure to the market was also often higher than the other tests when the indicator was removed, as was the number of trades taken.

Q2 The system's robustness is something that I would test using Monte Carlo analysis for further verification, although the system performed well over a range of different market conditions. I would like to understand what path the ranking is leading me on through the range of possible paths through the market.

Q3 I get a lot out of walk forward, but not in the sense that some people might. I am not keen on changing indicator parameters on a regular basis, but I have no real proof that my lack of keenness is warranted. But as a method of testing a strategy over a range of different market conditions and portfolio start-up dates it certainly is useful.

Without going into a lot of statistics I would be happy to say that there is no significant difference HHV optimised and changed every walk forward period and HHV "intelligently" fixed, however taking the parameter out altogether….no.

stevo

Saturday, November 14, 2009

Rubber glove bagpipe

Wonderful stuff - you should see what he does with a condom.



Linsey has a website as well.

stevo

Thursday, November 12, 2009

TradeSim Parametric Testing - good stuff!

I always thought it would be nice to optimise on position size criteria. It can be done in Amibroker, however TradeSim has really excelled with the recent upgrades to their software as shown by the following charts. It is clear that, with the system below, there is not a lot to be gained from increasing risk much beyond 1% in terms of net profit. Drawdown will increase, with some decent size outliers to spoil the party. I am sure I would manage to become an outlier (in the wrong direction) in actual trading!





These tests were run using TradeSim Enterprise Edition. I used Amibroker to generate the trades without having made any changes to the setup I had in AB for the previous version of TradeSim. The charts above do not do justice to the quality of the output - although the TradeSim site has better examples.

The charts above are from a monthly system I am using, as mentioned below.

regards

stevo

Wednesday, November 11, 2009

Monte Carlo and spreadsheets



I couldn't find any posts I had put up on this technique. I run Monte Carlo simulations (as many as you have the patience for - I did 200 runs) in Amibroker and then export the data to a spreadsheet.

The graphs shown above were done using Open Office, but any decent spreadsheet software will do (although OpenOffice is free for those that are frugal with their dollars :) ). I have done the same thing using Excel as well. I didn't generate the data for these graphs but the technique is quite easy once you master how to work out how to auto generate the "Bins" for the frequency distribution.

I put this up on ASF in response to a question - I haven't visited ASF for nearly a year.

regards

stevo

BTA & CTX - A couple of trades

I closed a couple of system trades over the last couple of weeks. CTX didn't do much, except to lose some dollars for me.


BTA was a more interesting ride. I remember taking the entry on this stock because it had closed at $0.87 on the Friday and jumped to $1.405 on the opening the following Monday when I was about to take the trade. I usually struggle to remember an individual trade after a couple of weeks. It peaked at at $1.79 for the week whilst I entered at $1.61! The volatility forced me to scale back the position substantially to reduce risk. I got out at $2.72.

I am wondering if I could consider recalculating risk on this sort of trade a couple of weeks in and add to the position size - something to ponder (but not lose sleep over).



The dog is looking at me like I should feed him so I better go - I sometimes wonder who is wearing the leash! According to accepted wisdom my dog is 90 human years old, although I haven't seen too many 90 year old men chasing possums (younger women maybe, but not possums)

regards
stevo