Machine learning in trading: theory, models, practice and algo-trading - page 3332
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
I think we have already discussed his quantisation in his time from all sides. I can only add to what I said earlier that I am happy for him that it brought him at least $200.
I think we have already discussed his quantisation in his time from all sides. I can only add to what I said earlier that I am happy for him that it brought him at least $200.
Thank you, it's nice to be happy for my income - it's rare!
The article is introductory - you are right, everything I wrote there - I think, and so clear.
The second part is in moderation, it's a bit more interesting. However, at the moment I have given up describing my own method, and came up with a simplified version, which gave a small effect on tests. It will be described in the second part.
However, this is the case when new ideas take no more than 5% of the text.
Come back to read and comment if you wish.
I don't know which Diogenes you mean, but in the sense of trolling we are all children in comparison with Diogenes of Sinope or Diogenes of Laertes.
If you look at the dates of my thread, my registration on the resource and today's one, it will become clearer. Two years after registration there was still hope for constructive and useful dialogue on the forum, and six and a half years later there is almost no hope left. Just for fun.
Here you go thanks, it's nice to be happy for my income - a rare occurrence!
The article is introductory - you're right, everything I wrote there - I think it's clear enough.
The second part is under moderation, it's a bit more interesting. However, at the moment I have given up describing my own method, and came up with a simplified version, which gave a small effect on tests. It will be described in the second part.
However, this is the case when new ideas take no more than 5% of the text.
Come back to read and comment if you wish.
Why randomly?
Cycle through all points of 1 class, and measure the distance to all points of the other class, taking the minimum distance.
When everything is obtained, sort, delete to the distance you need, one pair at a time. If the deleted point was used in another pair, you find a new point with a new minimum distance, sort again and continue.
Maybe you can think of a better way. Maybe without sorting - just delete to the required distance.
Ugh, I'm not getting it right, I guess:
Did I understand the prototype of the algorithm correctly?
I am returning to the topic with such a delay, because I am a little fascinated by the idea that leaves in CatBoost models and in other tree ensembles can be strongly correlated in activation, which distorts their confidence during training, leading to an overestimation of the leaf value for the model as a whole.
Eh, I'm a little slow on the uptake, I guess:
Did I understand the prototype of the algorithm correctly?
I am returning to the topic with such a delay, because I am a little fascinated by the idea that leaves in CatBoost models and in other tree ensembles can be strongly correlated in activation, which distorts their confidence during training, leading to an overestimation of the leaf value for the model as a whole.
1) You can also use a matrix, but not necessarily, but immediately find each point of 0 class the closest point of 1 class, i.e. we get at once point 2.
3) do not count anything and do not refer to clusters, just remove pairs of the closest points. With the distance less than the threshold, the threshold in that example would be 0.6. In other problems we will probably have to select it.
If a deleted point of 1 class was paired with another point of 0 class, then it is left without a pair, it has to find a new nearest point of 1 class (again make a calculation or use a matrix, as you suggested in point 1, if memory is enough, I think a matrix of 1million by 1million will not fit into any memory, up to 100 thousand maybe).
4) not until it remains, but up to the threshold distance. If it is very large, then only points of 1 of the classes will remain, which were initially more.
But as I wrote before, I don't think this noise removal is a good idea (see https://www.mql5.com/ru/forum/86386/page3324#comme nt_50171043). It's not like you can't remove that noise when making predictions. The tree itself will mark noisy leaves by giving them a probability of about 50%, and take for example non-noisy leaves with a probability of one of the classes >80% (or as many as you see fit).
Clusters have nothing to do with it. It's just removing the nearest points with different classes that contradict each other, i.e. noise. And then you can use clustering, or tree - whatever you want to train.
.
1) You can also use a matrix, but not necessarily, but immediately find each point of 0 class the closest point of 1 class, i.e. we get at once point 2.
3) do not count anything and do not refer to clusters, just remove pairs of the closest points. With the distance less than the threshold, the threshold in that example would be 0.6. In other problems we will probably have to select it.
If a deleted point of 1 class was paired with another point of 0 class, then it is left without a pair, it has to find a new nearest point of 1 class (again make a calculation or use a matrix, as you suggested in point 1, if memory is enough, I think a matrix of 1million by 1million will not fit into any memory, up to 100 thousand maybe).
4) not until it remains, but up to the threshold distance. If it is very large, then only points of 1 of the classes will remain, which were initially more.
But as I wrote before, I don't think this noise removal is a good idea (see https://www.mql5.com/ru/forum/86386/page3324#comme nt_50171043). It's not like you can't remove that noise when making predictions. The tree itself will mark noisy leaves by giving them a probability of about 50%, and take for example non-noisy leaves with a probability of one of the classes >80% (or as many as you see fit).
I can't quite get it into my head yet. Well it all happens in one space - in the metric of one predictor, but how to take into account the others?
As for what to do when predicting - I was thinking of using two models - one that detects what has been dropped or confirms that the data is in the "clumping" region, and the other that already works on what is left.
I can't get my head round it yet. Well, it all happens in one space - in the metric of one predictor, but how to take into account the others?
As for what to do when predicting - I was thinking of using two models - one detects what has been eliminated or confirms that the data is in the "clumping" area, and the other already works on what is left.
https://www.mql5.com/ru/articles/9138
No one has cared for a year now
I've written a dozen or twenty algorithms like this, some are well established. The one in the article is not the best in terms of stability of results, the first pancake.
so there is nothing to discuss, because there is nothing better yet.
I can't get my head round it yet. Well, it all happens in one space - in the metric of one predictor, but how to take into account the others?
As for what to do when predicting - I was thinking of using two models - one detects what has been eliminated or confirms that the data is in the "clumping" area, and the other already works on what is left.
In the example there are 2 predictors, i.e. we change the distance in 2-dimensional space (calculate the hypotenuse). If there will be 5000 signs, then you will measure the distance in 5000-dimensional space (how to measure - see the code k-means in Algibe, there just this is the main task - to measure distances, take it as a basis).
It looks like the root of the sum of squares of cathetes in all spaces https://wiki.loginom.ru/articles/euclid-distance.html.
If you will really do it - don't forget to adjust predictors, so that for example volumes of 1...100000 don't swallow price deltas of 0,00001...0,01000 in calculations.
How to detect it? That is the question. Especially on market data, where there will not be such a clear separation of the noisy area as in the example. Everything will be noisy, 90-99 per cent.
It may be easier to use ready-made packages for removing noisy lines, maybe they have a detector....