Thursday, 1 November 2018

Evaluating and Backtesting your system

This is the nitty-gritty part of evaluating and backtesting: How to read the results and determine if you have a good system with a positive expectancy. A positive expectancy is needed to determine if a trader has an edge; without an edge one should not be trading.
Total net profit is the overall bottom line of a system: Does it make money? If the results had been negative, it would be back to the drawing board, as you couldn’t expect to make money by using the system. Total net profit is something all people will look at when evaluating a system, but by itself it is not that great of an indication. Yes, you want a system that has a net profit and not a loss, but you also need to know how many trades it made, how big the swings were, how large a drawdown there was, what the average trade made, and so on.
A trader may be tempted to jump into a system that shows a R50,000 return rather than one that shows only a R10,000 return, but the second may be a better system. Maybe the first one made 1000 trades in a year with a drawdown of R35,000; maybe some months had large losses and others had large wins. Meanwhile the second system made only 50 trades with a drawdown of only R3000 and showed a small profit every month. In that case the second system is a better, safer system even though it returned less. You need to decide what is more important: large potential profits or safe, steady returns. A smart trader will always choose the latter.

Total Number of Trades
In choosing between two systems that have similar results, stick to the one that has a lower number of trades in the same time period. This will reduce the damage done by slippage and commissions, which can be substantial. A system that trades less may be boring for some people, but if you can get the same results, always take the one with fewer trades. Not only does System 1 have a lower net profit, it makes 3 times as many trades as System 2. This means you are working harder to get worse results. Though making fewer trades is desirable, make sure your test has at least 30 trades in it or there is a decent chance that your results were due to chance. If you don’t have 30 trades, you will need to get more data to test it properly.
Percent Profitable
This number means so little, yet people take so much notice of it. The best traders make money only 50 percent of the time, yet in real life people see 50 percent as failure. If people see a system with a 40 percent win/loss ratio, they instinctively think it is a loser because they are conditioned to think of it that way. Don’t pay much heed to percent profitability, but some traders may not feel comfortable using a system with only a 40 percent win/loss ratio. It doesn’t matter if a system is right 30 percent, 40 percent, or 60 percent of the time. What is important is how big the average loser is compared to the average winner. If one has good risk management skills, even a system with a 30 percent win/loss ratio can be quite robust. Both of the systems here have percent profitability numbers in the low 40s, and this is what I typically shoot for.

Largest Winner Versus Largest Loser
This is something that to look for to see how valid a system is. First, look to see if the total profit of the system was due to just one or two trades. In System 1, the total profit was R7025 and the largest winner was R7000. Take away this one trade and the system is profitable by R25 over 182 trades, which is not very good. If a system doesn’t perform well after you take one or two of the largest trades away, it is not very reliable. The other thing to look for is that the largest losers are not bigger than the winners. The key to trading is to keep your losers smaller than your winners; the opposite situation can lead to disaster. If the largest losers are too big, you need to work on your exits and stops some more. I look for at least a 2:1 or 3:1 ratio between the largest winner and the largest loser but will settle for a 1.5:1 ratio before I’ll consider trading a system. The same ratios hold for average winning trade versus average losing trade; unless the average winning trade is bigger than the average losing trade, I won’t use the system. I also look to hold my losers shorter than my winners, so I like to look at the average number of bars in winners versus those in losers to make sure the system fits my trading methodology.

Consecutive Losers
How many losers in a row were there? Many traders can’t handle 10 bad trades in a row and will abandon a system before it has a chance to work. By knowing how many losers in a row this system has had, you can decide not to trade the system because it may be more than you can stomach. Or if you are trading it and are going through a losing streak, you may stick with it because you know that it is common to have a six-trade losing streak. If you don’t know what the largest consecutive losing streak was, you may abandon a system after four straight losses even though this is considered normal for the system.

Average Trade
This is one of the more important numbers to look at as it can compare two systems to each other or one system to itself when changes are made. The average trade measures how the system does on a per-trade basis. This is the number that tells you how much on average you will make or lose every time you trade with the system. System 1 has an average trade of R38.60 (and this is before commissions), while System 2 has an average trade of R564.66. It doesn’t take a genius to realize that it is more profitable on average to take a trade on the second system. If the average trade is negative, don’t trade with the system; that’s easy enough to figure out. What’s harder is determining when the average trade is positive but too small to be worth using a system. Each trader must find an average trade he is comfortable having and then not trade systems with a lower amount.

Probably the most important factor in a system is the drawdown. The drawdown will tell you how much money you will need to start trading with this system in a particular market and give you a basis from which to measure risk. It tells you how much it would have cost you at its worst to use this system. This number will give you an estimate of how much starting capital is needed for each stock or market traded. You may think you have something good, but when you test it over extensive data, you may find that it would have lost R25,000 during its worst period. Don’t think it won’t happen again; the worst losing streak is always just around the corner. If you can’t afford to go through a losing streak twice as long as the biggest drawdown, you shouldn’t trade with the system.
Risk-averse traders are more likely to look at the drawdown than at any other statistic. If they can’t stomach the drawdown they’ll abandon the system or make changes to limit the losses. Money management plays a huge role in trading and should be considered in every aspect of trading. If two systems have similar results and one has a smaller drawdown than the other, it is probably less risky. If a system is too risky, one should avoid it.

Profit Factor
The profit factor is total gains divided by total losses. It tells you the amount that will be made for every dollar that is lost. If the profit factor is 1, you are breaking even. To be on the safe side, the profit factor should be at least 1.5. If you have a profit factor of more than 2, you have a very good system. In System 1 the profit factor is barely 1, and so this is a system that should be avoided.  The profit factor of 1.64 in System 2 is considered decent, and so one could feel comfortable trading with it.

Distribution of Returns
Finally, one needs to see how volatile a system is. How did the system make money? Was it a steady flow of good returns, or were there wild swings in the system’s equity over time? If you have enough data, look for steady monthly performance; if you are testing an intraday system, see how it does on a daily basis. A system with a steady positive return day after day with little variance from the mean is better than one that makes more but has wild swings. If the standard deviation is too wide, this may not be a good system to use, as the drawdowns could be large. If too many trades, days, or months fall more than 2 standard deviations outside the mean on the losing side, be cautious about using the system. Stick to systems that have smaller equity swings, as they are more reliable.
You also can use Excel or another statistical program to get a month-by-month or trade-by-trade breakdown, and so you can see if the performance was steady. It’s not a simple task, but it’s one that a serious trader wants to do to make sure his system does not have wild swings.

Commissions and slippage
No one likes to talk about commissions and slippage, but these two items can really make the difference between a winning and a losing system and a winning and a losing trader. One thing to consider when developing a trading style is that every trade that’s made—win, lose, or break even—will cost a trader some money. These are costs a trader can’t do anything about, and they need to be considered in his system design or his systems will be unrealistic. First, one needs to include every trader’s favorite thing, commissions. The second cost, slippage, is something some traders tend to (or would like to) forget about when considering their trading decisions.
Basically, slippage is the cost associated with paying more for or selling for less than what a trader intended. It may be caused by the market moving away or may be due to the difference in the spread between the bid and ask prices. Ideally, a trader would like to buy on the bid and sell on the offer. Unfortunately, he tends to do the opposite, putting himself into a losing position from the start. On a per trade basis, commissions and slippage are trivial, but when added up over the course of time, they can be astronomical and can have a huge impact on a trader’s profit and loss (P&L) statement. They can easily turn seemingly profitable trades into losing ones. When designing a system a trader needs to make sure it will cover his trading costs or else it will become a losing system.

Become a better trader

To become a better trader you must backtest your trading ideas and systems properly before you trade them. If you do not do this, you will never know if you are trading with a sound system until you are risking real money. If your system is not profitable, you don’t want to find out by losing money. You are better off learning this by spending the time to backtest it. To backtest a system properly
you will need to have enough data to give you 30 sample trades and cover all the different market conditions. You don’t want to test a system just on a trending market, not knowing how it will react in a choppy market.
If you don’t test over different conditions, you are not backtesting properly and may end up with a system that was curve-fitted to work in a trending market.  You should use different data when writing a system and when optimizing your parameters. This will lessen the chance of curve fitting around the data. When you are ready for a final test, make sure you do this on an out sample of completely new data, preferably covering different market conditions and long enough to be statistically valid (30 trades). One of the worst mistakes a trader can make is writing, optimizing, and testing the system on the same data. If it’s been optimized for a set of data, of course it will work great, but you will never know how it will work in the future.
One thing to keep in mind is that no matter how great a system did when it was backtested, it can never predict the future, as markets change. After you are happy with the results you may want to go over a chart visually and see where trades would have been made to get a good feel for how the system worked. You also want to check that your results weren’t due to one or two strong wins. You want to trade with a system that is more predictable and steady over time. These are the kinds of systems that end up doing best; ones with wild swings can be unpredictable and dangerous.
The last thing you need to do is to understand the backtesting results and know how to compare different systems. Overall profit and win/loss ratio are not as important as the average win and the profit factor. Look at the drawdown: Can you afford to trade the system? Don’t assume that the drawdown won’t happen right away, because it might. Make sure to account for commissions and slippage when you are figuring out total profits, as they will make a big difference in the results.
Take your time when backtesting and evaluating your system. Don’t ignore it or get lazy about doing it, as it is a crucial part of making you a better trader. And never trade a system without backtesting it thoroughly first.

Taken from High probability trading by Marcel Link

No comments:

Post a Comment