Machine learning in trading: theory, models, practice and algo-trading - page 124
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
This is a description of the multivariate exit from the deal. I implemented it myself. It increases the chance of a fit.
The main problem is in overtraining, when in future samples the existing trees do not correctly predict the class, or rather they predict the class worse and worse. This is exactly the issue addressed in this thread, namely: we try to prove that the obtained trees will occur in the future and the classification error on future samples will be about the same as on the training sample.
Although it is implied, but I'd better add it to make it clearer to everyone.
Usually overlearning in models has the effect that the model remembers the training samples very accurately. Learning starts with model deducing some general rules, dependencies of target values on predictors, but the further learning goes on, the stricter these rules become in the model, leading eventually to completely memorized examples, with complete inability of the model to predict on new data. This is solved by cross-validation, and stopping training when the model has learned some generalized rules but has not yet started to learn specific examples, this will give better predictive performance on new data.
This works great on data where the dependencies remain constant over time - picture recognition, or sound recognition for example. But forex is different in that many of the dependencies found are not persistent over time. The problem is not only to achieve learning without memorizing training examples, but also to somehow make sure that the patterns found will continue to exist in the future.
The task is more complete: to develop a way to estimate predictors with respect to the target variable that allows to assure that the relationship between predictors and target variable is constant and will also exist on new data.
Which target values to use(bar color, or reversal, or trend), or which predictors (prices, indicators, indices) - this is a nuance. If we are able to evaluate the set of predictors and their predictive ability to the target values, and the change of this predictive ability over time, then we can even find such combinations of predictors and target values, which have the best estimation, by a simple enumeration.
Then you can take any model, train it (maybe even with retraining control, which is important for neuronics for example), and there is a grail.
Usually overlearning in models has the effect that the model remembers the training examples very accurately. Learning starts with the fact that the model derives some general rules, the dependence of target values on predictors, but the further the learning goes, the stricter these rules in the model become, leading eventually to completely memorized examples, with a complete inability of the model to predict on new data.
It seems that you and I are the two of us here, as Alexei's position is not quite clear to me.
The rest is the experience of people who have been in the market for a very long time. I willingly believe that they are successful traders. I know from the literature some people who were successful using heuristics for many years, but the result was the same - the depo plummeted. Any heuristic does not answer the question of re-learning, so the end comes necessarily: in a day, a month, a year... but always unexpectedly and extremely untimely.
PS.
Overlearning is a basic problem of science, in the sense that the subject of science is the extraction of some basic laws, from which many particulars can be explained.
For example, Newton's law.
Take a piece of paper - it's bound to fall to the floor. Crumple that piece of paper - it is sure to fall, but faster. Newton's law states that everything falls with the same acceleration. How's that? That's how, man was able to discard particular, which is a great many, rough reality and get a universal law, and if it does not work, then these deviations can always be explained.
This is how science works.
What we are doing is trying to "roughen up" a quotient, to present that quotient as a model so as to have an idea of all future quotients.
Struggling with overtraining is a basic problem of trading.
This works great on data where the dependencies remain constant over time - image recognition, or sound for example. But forex is different in that many of the dependencies found are not stored over time. The problem is not only to achieve learning without memorizing training examples, but also to somehow make sure that the found patterns will exist in the future as well.
A little off-topic but by the way, just a bit of information...
In voice recognition in particular speech is a problem when a person says a word - the spectral pattern of the word fed to the algorithm "HMM" or "RNN" ...But the problem is that if the other person says the same word, he will at least say it at a different speed, those pattern is the same but stretched over time, and now the analogy - the head and shoulders pattern width of 33 candles and the same head and shoulders pattern width of 88 candles, the width is different, but the pattern is the same .....
To solve the problem with the speech recognition algorithm was developed - a dynamic time warp "DTW" task algorithm is that it can find two similar vectors even if they are different lengths. Thus when different people say the same word this spectral pattern of the word first passes through "DTW" and then to the "MO" algorithm and the word is successfully recognized. I haven't used it yet so I can't say anything but I think you'll have to try it, although if you say that there are no patterns in the market then of course such an algorithm won't work, I haven't decided with my worldview yet...
R packages: "DTW" , "dtwclust"
This is a description of multiple exit options. I've implemented it myself. Increases the chance of getting a good fit.
The method is as follows (all the following applies equally to both ML and classical TS).
The point is to get rid of uncertainty, which means to get rid of fitting in training and optimization on the one hand, and on the other hand to formulate unambiguously "soft" requirements to the system. TP and SL on entry - this is also uncertainty, so we use it only for emergency exit (it depends on the instrument volatility and we select them empirically, so that 100% trades triggered stops as rarely as possible, ideally - never. We predict the whole trade, from entry to exit. We consider the result with a profit without taking into account the spread and commissions to be a successful trade (the system's performance is determined after the mo, whether it covers the spread and commissions). As soon as the signal to enter the market is received, the system enters and then waits for a certain number of bars (it is empirically determined and depends on predictors and a trade instrument), after that look, is equity in profit? - Close the position, if it is losing, we wait for one more bar. Sometimes I use two parameters: minimum and maximum (if a trade is not closed after the maximum number of bars, it is closed anyway), and sometimes I use only one - the minimum number of bars.
Many people will be surprised, but many, even seemingly hopeless systems start working, including TS on wands, not to mention all sorts of systems with ML. The trick is not to require ironclad rules of TC from yourself and the machine, not to try to fully describe market movements and give those very "soft" rules for ML. Besides, we get rid of the bad market legacy of heavy tails (or more exactly, tails stop mattering), the lack of stationarity in BP stops mattering - because we know that market shapes can be scaled vertically and horizontally without changing their internal properties (and this makes life extremely difficult for emleoners).
I wrote about it on forum 4, I think in a branch of the Swinosaurs, about two types of all the TS (with definite boundaries and indefinite), but there the idea was not completed. Now I see it more clearly, or something like that...
In general, I use my rule when developing trading systems: If changes in logics lead to increasing the share of successful variants of parameters out of all possible variants, then this is a good change (the probability of choosing a bad variant for trading is reduced, no matter how we change the parameters - we will be in the plus). This approach allowed increasing this percentage in my TS.
What is this wild habit of answering with two words and quoting 100 lines of interlocutor???? The forum is 120 pages long. If you take away these unnecessary quotes more than 60 pages of useful text will be left... Respect the people who are reading this or may be reading it in the future
The method is as follows (all the following applies equally to both ML and classical TS) ....
Can you describe point by point what and how to do, as an algorithm, because I don't really understand how you suggest to do
It seems to me that everything you described fits into the optimization of the ARIMA model. When optimizing, the number of bars on which the model is built is selected
Can you describe point by point what should be done, how it should be done, as an algorithm, because I don't really understand how you suggest to do it.
You are right about the time lengthening of price figures. That's what I'm talking about, vertical and horizontal stretching/scaling. But not only the figures as a whole are stretched differently, but also separate candlesticks inside them. This is an additional problem and I doubtDTW-like pre/post processing methods can be adequately applied.These are market shapes distorted in some unknown way.
To be more exact: on the current bar there is a buy signal, we count back the smallest number of bars and check if the trade will be profitable, if it is, we close it, if not - we count one more bar and check again. And so on, until the maximum number of bars is reached and we finally close it. This is a learning mechanism.
Machine learning of market data is similar to the following: we show the machine a combination of graphical figures, circles, squares, triangles, trapezoids, figures of different sizes and shapes. The machine has to respond with a shape, triangle square or circle (exaggerated). We put the answer to the pattern and fuck! - The shape in the answer doesn't fucking match the template! And in the process of learning, for some reason, they matched up great.... That's the kind of problem we're solving here. And I say - no problem, we can stretch the pattern a little bit so that the answers turn out to be correct even outside the training data.
This is the basic "fish" of the system, which is already working by itself. But no one forbids adding to the TS and using in trading such features as SL Breakeven. The system remains the same, but for that if the market moves in our direction, we will get extra profits beyond the "plan", and then either trawl SL, or start partial closing of the position, depending on the distance covered compared with the statistical price movement over a given period of time, etc.
Is it increasing? It is strange. It decreases.
Do you evaluate the strategy's performance on history or is there a long forward? The devil may be in the evaluation on history.
Look, I had a strategy written on 100 lines in MT4 by my own hand. I wrote about it in my profile in December 2014:https://www.mql5.com/ru/users/alexeymosc/page3?#userActions
This is what the latest version looks like, put together in January 2015:
I then tried it on the real, along with other strategies. I wrote about them everywhere, too.
This, this, and that, hand-drained it.
Saw a forward test today from the end of optimization to the end of July with the same settings. Look:
Everything is listed there. Spread was 30, lot 0.1.
I will try it on all ticks as well, so that the ticks work out better:
Entry on impulse, exit via OR on a number of conditions. In principle, it seems to work. But if I enter on the peak, then wait a year to get out of a drawdown...