Optimisation and Out-of-Sample Testing. - page 5
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Candid, it's not that simple with profitability. If we take the same sets A, B, C in NS, then profitability of strategy after normal training will be as follows: on the A part it is maximal, on the B part it is significantly less, and on the C part it is even worse. And that's how it is with us: great results on history, but outside it - losses... The problem is to pick a strategy where the C area is almost certainly worse in terms of profitability, but still comparable to the A area.
I don't intend to promote them here. But the principles of testing that I learned when I was doing them are quite reasonable. And here, in contrast to metaquote fitting, we can at least rely on generalization ability (the main property due to which an Expert Advisor may remain profitable in the future; our optimizer is devoid of this property completely).
I wanted to say that the real MetaTrader's tester allows you to get the same results when optimizing a sample+unsample set as optimizing a sample followed by an out-of-sample test. In the tester, the "Expert properties" button, then "Testing" and "Optimization" tabs allow you to get rid of losses of any length and depth you want.
It all depends on the task definition. If we neglect the degree of evenness of profit distribution over testing time, the MT tester's standard capabilities are really sufficient and the time spent will be comparable. Is it worth neglecting? Everyone has their own experience and views. The process can indeed be called fitting, but I think the term approximation would be more accurate. Not every approximation can be extrapolated into the future and the criterion of uniformity of profit just allows to reject knowingly unsuitable for extrapolation variants. IMHO, of course.
This refers to the applicability of the tester to an optimization problem on a sample with a subsequent out-of-sample test. The same results, but you have to approach it from the other side - to reduce the number of consecutive loses and the values of losses. That's all.
Candid, it's not that simple with profitability. If we take the same sets A, B, C in NS, then profitability of strategy after normal training will be like this: it is maximal at A section, much less at B section, and much worse at C section. And that's how it is with us: great results on history, but outside it - losses... The problem is to find a strategy that is almost certainly worse in terms of profitability in area C, but still comparable to area A.
I think the analogy with approximation gives the answer: we know that the better the approximation, the less suitable it is for extrapolation (of course we reject the case of analytic function graph with its guessing). So a good solution is most likely not the one that gives more profit on A, but the one that gives more uniform profit on A+B. Now we start extrapolation and, again naturally, the error increases as the forecast horizon grows.
Categorically disagree, Vita. Otherwise in neural networks there would be no division of all data into three parts, fundamentally different: real optimization - only on the first part; the second serves only to determine the moment of training termination, and the third - just for single testing. That is, the real fitting only goes on the first, and on the third, it's whatever it turns out to be... And the choice - "Occam's razor" or loss of confidence in the system - is left to the creator of the system.
Roughly speaking, optimising on A+B+C is not at all the same as the processing described above.
For the sake of clarity, just in case.
A - set of parameters obtained during optimization on sample
B - set of parameters obtained after testing A outside the sample.
The process of obtaining B is the process of on-sample optimization followed by an out-of-sample test. It is in this process that you are supposed to get rid of curve fitting?
C is the set of parameters obtained by optimizing the population of sample + out-of-sample.
I argue that C is as good as B in terms of curve fitting.
C-B= the set of parameters that are unprofitable either in the sample or out of sample, but profitable on the population as a whole.
The process of obtaining B can be performed by a standard tester.
Candid, I'm not saying that a good solution gives the maximum profit at A. The optimizer does that better: it gives an absolute maximum on A, but nothing on out-of-sample. According to the NS learning model, the likely candidate for a good solution is maximum profit on B with already acceptable but not maximum profit on A.
About your comment: almost agree, but not A+B but A+B+C.
2 Vita: I thought I wrote everything clearly on the previous page. ... The sets A, B, C do not overlap. For example:
A - from 1st January 2004 to 31st December 2005,
B - January 1, 2006 to October 31, 2006, and
C - from 1 November 2006 to the present.
The usual ratio of data lengths in NS is A:B:C = 60:20:20.
About your comment: I almost agree, but not A+B but A+B+C.
I don't intend to promote them here. But the principles of testing, which I learnt when I was doing them, are quite sensible. And here, unlike metaquote fitting, we can at least rely on generalization ability (the main property due to which an Expert Advisor can remain profitable in the future; our optimizer lacks this property completely).
Oh, right! The generalization ability should be a property of Expert Advisor, not of optimizer. The law must be laid down in the Expert Advisor; the idea of the Expert Advisor must be as comprehensive and systematic as possible. But I would not make such a claim against the optimizer. I think it is absurd, just like it is absurd to try to pull absolutely any Expert Advisor up to the level of profitability at all times and in the future with an optimizer capable of generalization. It's not the fault of a metaquote optimizer that there is no law, no profitable idea and so called generalization ability that it can optimize. The only thing left to do is to fit the curve.
Candid, I'm not saying that a good solution gives the maximum profit at A. The optimizer does that better: it gives an absolute maximum on A, but nothing on out-of-sample. According to the NS learning model, the likely candidate for a good solution is maximum profit on B with already acceptable but not maximum profit on A.
About your comment: I almost agree, only not A+B but A+B+C.
2 Vita: I thought I made it clear on the previous page. ... The sets A, B, C do not overlap. For example:
A - from 1st January 2004 to 31st December 2005,
B - January 1, 2006 to October 31, 2006, and
C - from 1 November 2006 to the present.
The usual ratio of data lengths in NS is A:B:C = 60:20:20.
God be with them, A,B and C. They have a different meaning in my posts. They are not timelines. They are the sets of parameters that the optimisation produces. Oh, come on.