Backtesting Tips from a Machine Learning Expert

Hey, Creator! And welcome to the Trality blog. In this post, we’re going to dive a little deeper into the subject of backtesting and how you can ensure you’re doing it correctly.

Backtesting is by far and beyond the most crucial aspect of coming up with a profitable trading system. If you execute your tests properly, it won’t only help in optimizing your strategy by pointing out inconsistencies or anomalies - but it will also validate the strategy and give you confidence in deploying your bot to a real, live market.

With that in mind, we asked our Machine Learning expert, Constantine Dießelkamp, for some straightforward tips on how to best perform your backtests. Buckle up, backtesting is no walk in the park and there is quite a lot to digest.

1. Split your data

Imagine you have just performed a backtest for the time period of 1.1.2020 to 1.1.2021. You now have one large chunk of data for the whole year. Constantine recommends splitting that data into three distinct ‘chunks.’

The data chunks will be:

1. Training and optimisation data
2. Evaluation data
3. Test data

We will use these chunks as we continue on to optimize our system.

The first part, the training/optimization data, is the data from 1.1.2020 to 1.6.2020 (the first half of the year). Here, you can fiddle around with the parameters and strategies. When you have found a strategy that satisfies your needs - you should take it and evaluate its performance in the second part e.g. 1.6.2020 to 1.9.2020. This will be your evaluation data. Evaluation, in this case means checking whether the parameters and strategy that we tested on the first data segment still works as expected on the second data segment. We didn’t use this part of the data to optimise our strategy so it’s a great test to validate your idea.

Constantine recommends that you repeat the above process until you are happy with the strategy and then we can move on to the next part.

2. Strive for consistency in your results

When it comes to creating a bot that works in all sorts of scenarios (upwards trends, downwards trends, bull and bear markets) it’s so important to validate, validate and validate some more. Your bot needs to be able to perform under volatile circumstances as the cryptocurrency market is notoriously turbulent and that’s not set to change anytime soon.

For this, you can split your evaluation data into three sub-parts (one month each) and run tests on each month. What you are looking for here is a consistent result. If the first month delivers +300%, the second month -70% and then the third +50%—something is not right.

3. Finally, unlock your test data

Once you have performed, tested and refined your system using the first two data chunks, (the training/optimisation data and the evaluation data), it’s time to perform some final validations. You will run your algorithm against the third ‘chunk’ of data (the test data). You should always perform your last validations on the most recent data you have.

In this context, the test data is the data for the final quarter of the year —1.9.2020 to 1.1.2021.

Note: Do not use the ‘test data’ to perform a final validation of your strategy until you have been through steps one and two and are very comfortable with the results.

Run your backest on this test data and whatever results are displayed, these are the results that you should expect if your bot were to go live. It could be that you’re not satisfied with the results - is this case, it’s time to start all over again.

More wisdom

Backtesting is an arduous process, it is arduous because to create a bot that works is tough and it needs a lot of data analysis and fiddling to get it right. What you have above is some great insight from a professional who knows this process like the back of his hand. However, we have a couple more pearls of wisdom to share.

Consider context

You must always consider the trends that may have occurred within any given time frame that you are testing. For example, if you were to test your strategy between October 2017 and February 2018 (a huge cryptocurrency bullrun, see fig. 1) then it is unlikely to perform in a bear market - or even a sideways market (see fig. 2). Which goes back to what our Machine Learning expert was saying about splitting your data and testing it in chunks.

Past performance is not always accurate

Backtesting is certainly one of the sharpest weapons in a bot trader’s arsenal but it’s not a 100% infallible tool. Backtesting gives you a great indication of how the bot performed in the past but it can not predict the future. If possible, execute some paper trading (walk-forward testing) on a bot that you have successfully back tested before going live to ensure that your strategy is still relevant in the current market climate.

Time to test! 📈

Now you have gained a little more insight into the topic of backtesting and learned some very solid tips to ensure that your system works well.

If you have already put together a system and want to try it out - you’ll be happy to know that our powerful backtester can handle anything you throw at it - and it’s lightning quick.