Walk Forward testing – some thoughts

The numbers below are from a weekly system I have been playing with. It’s not really a complex system, although it does use ranking of stocks potentially to boost results – something I am still contemplating as a strategy.

The buy signal initially only used one indicator – the one optimised in this test. I then added in extra criteria, but only when the index was bearish as determined by a moving average. I then added a ranking criteria based on momentum – the system takes the strongest stocks in terms of momentum. So the breakout criteria optimised in these tests is a major component since when the market is bullish it is the only buy criteria used.

The table below shows out of sample only results for 3 walk forward simulations. The walk forward used optimisation of a parameter used in an indicator, a dummy parameter with the real parameter fixed throughout the test and, in the final test the indicator that used the real parameter was deleted from the buy signal altogether. The optimisation metric used was profit factor in all tests, although it’s sort of irrelevant if the test is optimising a dummy value.



Working on the hypothesis that the in-sample tests are irrelevant, and to save words, I have left the in-sample results out of the above table.

The orange colored rows are using a dummy walk forward optimisation variable with the parameter / indicator removed altogether from the buy signal, instead of being a fixed value as in the white rows, or the optimised parameter in the yellow rows.

I also graphed CAR% below to get a better picture of the results.


So - the questions are;
1. Does the buy parameter / indicator being tested make any difference?
2. Is the system robust?
3. How valuable has the walk forward process been?

Q1 Whilst at first glance it doesn't make much difference look at the Maximum System Drawdown column, especially the last row when drawdown hit 57%. Whilst the parameter often doesn’t do a lot it was certainly worth having in the buy signal during the Global Financial Crisis. Exposure to the market was also often higher than the other tests when the indicator was removed, as was the number of trades taken.

Q2 The system's robustness is something that I would test using Monte Carlo analysis for further verification, although the system performed well over a range of different market conditions. I would like to understand what path the ranking is leading me on through the range of possible paths through the market.

Q3 I get a lot out of walk forward, but not in the sense that some people might. I am not keen on changing indicator parameters on a regular basis, but I have no real proof that my lack of keenness is warranted. But as a method of testing a strategy over a range of different market conditions and portfolio start-up dates it certainly is useful.

Without going into a lot of statistics I would be happy to say that there is no significant difference HHV optimised and changed every walk forward period and HHV "intelligently" fixed, however taking the parameter out altogether….no.

stevo

System design & testing

This post refers to a monthly time-frame system I mentioned a couple of posts ago.

To design this system I used ASX data for around 500 stocks from January 1999 up to the end of 2004. The initial ideas were coded and then optimised. The buy signal is made up of 2 variables and the exit is a simple ATR trailing stop plus a simple, rarely triggered, profit exit.

I optimised the variables, fixed them and forward tested up to September 2009. The whole exercise was completed over a weekend. It's not a question of what time-frame to use, trading is about what works for the trader.

The security profit distribution shown below has the top 50 winning trades excluded to see if there are any big winners impacting on the results. I excluded the top 50 since there were over 1000 possible trades over the 10 year period.


Yearly profit is shown below ($500,000 capital kept constant) again with the top 50 trades excluded.


Yearly profit without trades excluded.


I am ranking the trades, not so much to improve the performance of the system, then to stop me wondering which stock to pick out of the handful presented.

I also did some runs where I started the system up at the beginning of every year from 2000 through to 2009. There are some difficult startups based on these runs, but nothing that a trader that understands the system could not ride out. I am not starting from scratch anyway since the portfolio is already close to fully in the market.

regards

stevo

Even a monkey could write a great novel

I have been reading Fooled by Randomness by Nassim Nicholas Taleb. It’s a challenging read, but worth the effort by anyone that is into system design. I haven’t finished it yet.

To quote p.163;
“Beset with insomnia, the computerized day traders become night testers plowing the data for some of its properties. By dint of throwing monkeys on typewriters, without specifying what book they want their monkey to write, they hit upon hypothetical gold somewhere. Many of them blindly believe in it.”

The reference to “monkeys on typewriters” relates to the theory that if enough monkeys are put in front of typewriters eventually one of them will write War and Peace. I strongly suspect that the number of monkeys required would be impossible to find, let alone feed!

Any results need a good dose of scepticism – even a monkey can write a great novel!?

Trading is not just about making the maximum profit. “The greater the profit the greater the chance of going broke. “ (stevo 2008). Well maybe – I am not really into hard and fast rules. If I try for 100% plus returns using leverage then I might make it 3 out of 4 years, but the 4th year will wipe me out. If I plug for 20% I am less likely to self destruct.

Then there is the ride taken. I did some work on a fairly simple system. The equity curves are shown below. The system was developed using data from 1999 to 2003 so the walk forward equity curve looks pretty good.

Figure 1 Basic System

The first curve is the basic system from 1999 to March 2008. It looks pretty good considering I am not pyramiding profits into the system, just trading with a flat amount. The second equity curve has an index filter added to turn the buy signals off if the market looks weak. The equity curve is still pretty good, but total profits are down compared to the system without the index filter.

Figure 2 With index filter

But when I look at the period from Jan 2007 to current a different picture is painted. Without the index filter the equity curve is south of the zero line for most of the period, whilst the index filter version looks much more tradeable. . I always remember starting up a system back in 2002 and immediately losing money. Startup is always hard! Starting this system up at the beginning of 2007 would have been tough.

Figure 3 Basic system from 2007


Figure 4 With index filter from 2007. Note that a lot of the money was made in just a couple of trades, but at least it wasn’t given back as in the basic system.

Over the last 5 years from 2003 to 2008 turning off a system when the occasional market dip occurred didn’t really help make money. It was better to be in the market all the time.

But it’s not only about CAR (compound annual return) it’s about the ride taken. Measuring the ride can be difficult, but the equity “curve” paints a picture that most people can understand

stevo

AmiBroker testing and Excel

Many system testers focus on optimising variables like moving averages. But it is also possible to get a better understanding of position size, risk and other trade parameters.

The following charts were generated from Amibroker optimise runs. I optimised a number of criteria, %Risk, minimum position size $, capital and maximum number of trades held - all at the same time. What results is a lot of data that I then transfered to Excel. Using Autofilters and pivot tables in Excel I can fix some parameters, as well as exclude others, and create 3D charts as shown below.


I had some fun with colours - the default ones set on my version of Excel didn't really work for me.


So in the one spreadsheet I have, amongst other things, a large number of 3D combinations to consider. I lose the ability to rotate the charts but I did customise the colours to my liking. It's a little time consuming generating pivot tables and determining what values to include / exclude, what parameters to fix and what value to fix them at but I feel I am only scratching the surface with Excel.

regards
stevo

3D charts & Trade-offs

The 3d plots are from a new system I have been working on and show a 2 variable optimise. The first chart plots Compound Annual Return, whilst the 2nd chart looks at drawdown. I find the 3d charts feature of AmiBroker a very useful tool - that's why I am posting these charts here.

What would be nice is if we could have 4 or 5 dimensional charts! I tried using Excel surface charts to get more insight but the 3d Amibroker charts are more convenient.


One variable (ATR multiplier) didn't have much impact over the range I tested - I could choose pretty much any value in the relatively tight range I tested. This suits me.


As is often the case using the values that give the best draw down result in lower CAR and visa versa. I could trade off CAR for lower draw down.

What was good is that I finally coded up profit target code. The following code was posted on the Yahoo AmiBroker site courtesy of Tomasz Janeczko;

/* a sample low-level implementation of Profit-target stop in AFL: */
Buy = Cross( MACD(), Signal() );

priceatbuy=0;

for( i = 0; i < BarCount; i++ )
{
if( priceatbuy == 0 && Buy[ i ] )
priceatbuy = BuyPrice[ i ];

if( priceatbuy > 0 && SellPrice[ i ] > 1.1 * priceatbuy )
{
Sell[ i ] = 1;
SellPrice[ i ] = 1.1 * priceatbuy;
priceatbuy = 0;
}
else
Sell[ i ] = 0;
}

All I had to do was modify it so that I also tested for a trailing stop. I initially tested using APPLYSTOP code but to set up the system I was more comfortable coding up the profit exit. Because the Sell is dependent on the buy what sounds quite easy does require something like the code above. GP's document on looping was also of value - it's in the files section of the Amibroker Yahoo site. The Yahoo site is a wonderful resource.

I will run some TradeSim tests before I take the system I have been working on live. As usual for me it's a longer term weekly system. I think I will call it Chaos, although Chicken & Chips is tempting! There is a small restuarant that does wonderful chicken & chips at Crows Nest and I do like my food. Why not name a system after food I like?

regards
Stevo