Machine learning in trading: theory, models, practice and algo-trading - page 172

[Andrey Dik]

SanSanych Fomenko:

PS.

For ardent supporters of MCL I note that without all those actions and tools, which are discussed in this thread, the tester does not give any reason to speculate about the future behavior of the trading system. The tester says: "These are the results for this time period. That's all. The tester gives exactly one figure, such as the profit factor, which refers to a certain period of history. And the statistics can only be obtained in R. And the tester is the final part of model design, but not a replacement for the entire development process.

Bullshit and nonsense!

The tester will give out exactly that much and what he was asked. Did you ask for the profit factor? - here's the profit factor. If you ask something else or more, there will be an answer to it too. And the statistics can be obtained by callingTesterStatistics() inOnTester(), and all this is available for a very long time. You can add any other stat counts you want.

Your R is equally unable to answer, if you have nothing or do not know what to ask.

[Alexey Burnakov]

Dr. Trader:

For example, you performed validation on a deferred sample. Suppose the model on the deferred data merged. What do you do in this case? If you start to tweak something in the parameters to successfully pass the validation on the pending sample - you essentially include data from the pending sample in your crossvalidation, and the crossvalidation becomes adjusted too. This can be corrected by adding a new deferred sample. But what if the model fails on it too? Should we select parameters in order to pass a new delayed sampling? It's an endless race.

Including pending selection in crossvalidation and creating a new pending selection is not a solution, but infinite repetition until you get lucky and the model accidentally passes the pending validation. Then you can stop, but this is not a forex solution, but just luck smiled on you, which however statistically would be a drain.

So the question - let's say the model on the pending data has plummeted. What do you do in this case?

Dr. Trader, I will try to explain it in a picture. Don't worry, this is a complex issue, but it can be solved. You should not do as you suggest, otherwise there will be a vicious cycle.

And one more thing, I will gradually wrap up the activity in this thread. Already tired of the forum. It is better in private and with examples.

So:

Model Selection for Noisy Time Series Prediction and Trading

Step by step:

Learning the model - the model shows grayl characteristics

Testing of a model - models perform noticeably worse. But there are some models that work fine.

Here comes the moment when you have to choose models. You and I choose models that have worked better in the test (these can be blocks of validation in crossvalidation). And it immediately becomes clear that if we choose a model for a period where its characteristic is already known, we make an optimistic choice.

If we have crossvalidation (data for different models are mixed) we can not make them a committee on the test (the test is mixed). You instead take the selected "best" models on the test and test them on the deferred sample. What do we expect to see? Since in the real application we will rely on the performance of models on crossvalidation blocks (or single test, if you don't want to bother with crossvalidation), we need to know whether the selected best models will show similar performance in the future (on deferred sample 1). If there is a connection, it is already very good. Essentially, we can stop the study at this point and select the best models by crossvalidation in the future.

But if we still want to make a committee, we have to figure out which models to choose for the committee. We do this on deferred sample 1, but not just randomly. Again, remember that in reality we will only rely on test data (the latest data available to us). That is why we include into the committee one by one those models that show the best results in the TEST. Achieve maximization of the committee's performance on the delayed sample.

At this point you are subject to multiple model selection bias, since we are evaluating the committee's choice on data where its characteristic is already known.

The last sample is needed to validate the committee.

If at the step of validation of selected models (on the test) on the deferred sample we see that the characteristics of models are floating - this is already fatal, because we do not get a chance to select the model on the test. At this step we need to change something in the approach.

If the assembled committee plummets on the last deferred sample (this is a one-time test), then committee fitting to the previous sample happened.

Repeating the whole procedure with different parameters, achieving maximization of charact. on the final deferred samples is impossible, because this would also be an optimistic choice of models.

Even if everything worked well in the deferred samples (i.e. we minimized model selection bias), it is good to repeat the whole procedure in the cycle with the same parameters and get the distribution of the committee's work in the last sample. It is clear that each time we need different (at least partially) data. And it is already very expensive...

Please remember this scheme. It leads your estimate of real trading to the possible absolute.

[Mihail Marchukajtes]

In the past, when Neuroshell was young and available, they used paper trading. I hope everyone knows what it is, but when you work in real trading ts started to drain, and then it was decided not to use a piece of paper trade, and train the network, while leaving a piece of real to assess the correctness of the network. For example, I now do the following: I delete one day, the very last one in terms of volume and open Inetres. For what???? you have not seldom seen such a map that in the beginning after the optimization area the TS plummeted a little, and then starts to give quite decent signals. So that's why I remove one day which is already known to me, and TS sort of starts to trade from the next day.... That turns out quite well, not without errors of course, but still acceptable for profit ...

[J.B]

Mihail Marchukajtes:

All so, but this pattern is very bearded, everyone knows about it for a long time, so...

And now the simplest classifier would have found it, for one series such interpretation of price, volume and OM, not enough data, you need at least orderbook and strip with directions of deals, and in the case of forex, because this information does not exist, it must be taken from the Western liquid futures markets.

[Mihail Marchukajtes]

Have you ever thought about what if the network feeds exactly those patterns that will be in the future? The question is how to find them, what days you should choose to train the network to know that in the current day these patterns will be. Here is the answer. We select the days that had the current volume and OM, respectively, in the hope that the context of the day will be similar to those days that were in the training sample. I do not feed the values of volume and OI themselves to the input of the network, I only select these very days, and the input is AD, Zscore, Kelli, etc. I.e. the point is to choose such a set of patterns on the history and the market reaction to them, as in the current day. If there are theories about this kind of selection with other techniques, I would love to hear....

[J.B]

BlackTomcat:
Smart people develop and train neural networks, but still do not see simple things. I read your post and was quite surprised. If I understood it correctly, then you, roughly speaking, found all the price falls of 0.2% after some high, then took three candlesticks near that high and performed some manipulations with their prices, finally leading them to a certain probability with the help of a neural network. But excuse me, don't you think this approach is too primitive? :) You are digging in the wrong place. That's why the result is just the opposite of reality. I would characterize your approach as follows: you are trying to take 3 pixels out of a FullHD picture and use those 3 pixels to make an idea of the whole picture. Okay, not the whole picture, but what is the probability of correctly predicting at least 10% of the image area? I hope my example is clear. You don't have to look at the pixels to see the picture. In other words, you don't have to look at individual bars to understand the graph, you have to look at the whole graph . And the solution to the problem lies more in the realm of geometry than in algebra or physics or biology, for example. Although, when I read some of the research that people do here, I get a strong feeling that they are trying to understand human structure through geography. :)

+1

I would not even say "graph" but graphs besides not only prices, but...

The analogy about 3 pixels and HD is very relevant to my opinion.

[J.B]

Andrey Dik:

Or even someone will say: "You just don't know how to cook MO! - Probably yes, I can't. But who knows how? Who was able to use MO in the market?

And you're not alone, it's NORMAL, it's not normal when it works and such do not sell courses on "successful trading", not engaged in affiliate programs and other nonsense, and knocking somewhere like https://www.rentec.com/Jobs.action?data=true if there is no desire to bother with raising capital for algorithotrading))

[Mihail Marchukajtes]

Again, I am deeply convinced that in order for NS to work, you need to train it on such patterns that will be in the course of trade (day, week) and the main thing that the market reaction to these patterns should be the same as during the period of training. Then any perseptron will do the job. The question is how to select the exact set of patterns that will be in the course of the next trading day or week. Here, as they say, there is such an approach, and it will not be necessary to invent a super complex NS or anything else, if the network will be trained on future paternas....

[J.B]

Mihail Marchukajtes:
Did you ever think what if you feed the nets exactly those paternas that will be in the future? The question is how to find them. What days should I choose for network training to know that these patterns will be in the current day? Here is the answer. We choose the days which have the current volume and OM, respectively, in the hope that the context of the day will be similar to those days that were in the training sample. I do not feed the values of volume and OI themselves to the input of the network, I only select these very days, and the input is AD, Zscore, Kelli, etc. I.e. the point is to choose such a set of patterns on the history and the market reaction to them, as in the current day. If there are theories about this kind of selection with other techniques, I would love to hear....

This is exactly how it is done, how else could it be done? In the trivial case, training dataset is a vector sequence of normalized increments of volume price and OI, let's call vectorSet[][] a neural network or some other classifier input =vectorSet [t][], teach output = vectorSet [t+1][]

If I understand the question correctly...

[J.B]

Mihail Marchukajtes:
Question, how to choose exactly that set of patterns, which will be during the next trading day or week?????

Unfortunately I have no right and frankly speaking I have no desire to go deeper, after all we take money from each other)))) But 2 years ago, when I was working with SAM, I was processing more than 500 chips at the input of the neural network and had about 30 outputs, but time goes by ... ;)