Machine learning in trading: theory, models, practice and algo-trading - page 3355

 
СанСаныч Фоменко #:

Somehow you are not paying attention to my posts, focusing on probabilities. It doesn't matter what the probability is called, what matters is that if it doesn't improve, the model is overtrained, into the bin. The prediction error on OOV, OOS, and VNE should be about the same.

A model out of the box does not give correct probabilities, any model. That's the story. You may have the predicted labels completely match, but the probabilities don't, won't reflect the actual probability of the outcome.
Do you understand me?
 
Maxim Dmitrievsky #:
The model out of the box does not give the correct probabilities, any of them. That's the story. You may have predicted labels that completely match, but the probabilities don't.
Do you understand me?

Added my post. Any model gives correct probabilities in the sense that the classification error will not fluctuate.

 
СанСаныч Фоменко #:

Somehow you are not paying attention to my posts, focusing on probabilities. It doesn't matter what the probability is called, what matters is that if it doesn't improve, the model is overtrained, into the bin. The prediction error on OOV, OOS and VNU should be about the same.

Here's another histogram

Different algorithm - different histogram, although the labels and predictors are the same. If you are looking for some kind of theoretical probability, implying that different classification algorithms will produce the same histograms ... that doesn't occur to me, since you have to work with specific algorithms and they will predict and they have to be evaluated, not some theoretical ideal. The main evaluation here is the overfitting of the model, not the proximity of the probabilities to some theoretical ideal.

Give up? Google classification probability calibration, it should be in R.

And plot the probability curve of your model against the benchmark.
 
Maxim Dmitrievsky #:
Give up? Google classification probability calibration, it should be in R.
.

And plot the probability curve of your model against the benchmark.

We're talking about different things.

I am writing about the result, and you are writing about the ideal of intermediate data.

For me it is obvious that probability values of specific labels given by RF and ada will be different, but predictions of specific labels are almost the same. I'm not interested in the probability values, I'm interested in the prediction error

If you theorise, it is most likely impossible to obtain the class probability in your sense, since you have to prove that your probability satisfies the limit theorem, which is very doubtful.

 
СанСаныч Фоменко #:

We're talking about different things.

I am writing about the result, and you are writing about the ideal of intermediate data.

The class probability values given by RF and ada will be different, but the predictions of specific labels are almost the same. I'm not interested in the probability values, I'm interested in the prediction error.

If you theorise, it is most likely impossible to obtain the class probability in your sense, since you have to prove that your probability satisfies the limit theorem, which is very doubtful.

Nevertheless, the original question was there, nobody answered it. I am talking about exactly what I asked.
So there is something to strive for.
 
Maxim Dmitrievsky #:
Still, the original question was there, no one answered.
So there's something to look forward to.

Why? If in the sense of a thesis....

 
СанСаныч Фоменко #:

Why? If in the sense of a thesis....

Because trading with probability curves means taking losses instead of gains. Any classifier needs calibration if it is a risk-sensitive application.
 
Finally.
 
Maxim Dmitrievsky #:
I was hoping someone would at least google the tip.

Even if you have probability curves in your training, what new data can you talk about. And bousting and forrest sin big time with this. Busting is overconfident, Forrest is underconfident. Provided, of course, that you plan to use the threshold at all.

I myself have observed that when you increase the threshold, the quality of trades does not improve even on the traine. Then the probability of what does the model return? Nothing :)

In Sanych's picture is self-confident bousting, you can see by the edge column outliers. The trough should be smoother. This is an overtraining model.

It shows the model's outcome on "probability" ranges with 0.05 step. CatBoost puts the class separation at 0.5 quite accurately (magnetta is 1, aqua is 0).

You can see that the fin outcome is positive starting at 0.35 - the green curve rises above the red curve.

Is this what you want to calibrate - shifting the point of class separation to the point of revenue generation?

Reason: