Machine learning in trading: theory, models, practice and algo-trading - page 88

New comment
 
Vizard_:
Yur, make a normal version of the program (or upgraded) for people)))

1.Let it read the standard files of this type - header (with or without quotation marks), data, delimiters let it read zpt, point.zpt, tabulation. Example -

"Date";"V1";"A77";"V23A";"Целевая"
01.01.2000;4.999995E-03;1.499891E-03;-2.000213E-03;-5.000234E-03;1

2.Let the model uses only 10 inputs, but make it possible to load a lot, then
in a separate tab, or you can set "wings" (you choose the inputs) or else.

3. without normalization! (or disabled), because if the data is initially centered on 0, after conversion
x-x.min/....(or whatever you have) - fly away)))

4.Throw out randomness (or switchable), let it just divide the data in two parts, counts in order and displays the % of successful predictions for 1 (for 0 do not)

5.Made the importance of inputs - good!

6.Let in addition to the formula that is output, output the full!, that is now

double x2 = 2.0 * (v5 + 1.0) / 2.0 - 1.0;
double decision = -0.2632437547312642 -0.2634178652535958 * x2
+ 0.05267978803936412

we should -

Name target = -0.2632437547312642 -0.2634178652535958 * insert construct x2= (2.0 * (v5 + 1.0) / 2.0 - 1.0)
+ 0.05267978803936412

you get -

Name target = -0.2632437547312642 -0.2634178652535958 * (2.0 * (v5 + 1.0) / 2.0 - 1.0)
+ 0.05267978803936412

That is, you will get a little bit suitable device for quick research. Then you can still insert and so on, simple (not greedy)
algorithms, with a choice... Otherwise the tool is of little use. While the file is being edited, this and that... It's easier to use something else...
Where's version 7?
Nah... I like it all, it's quite robust. So 100 points, Yura.
 
SanSanych Fomenko:

So-so - just not enough brains and time.

You have to start with the target variable, and then select predictors for it, and then double-check with mathematics, so on. In any case, the process is tedious and I can't formalize it.

Took it right off the tongue. I know. I was up all night thinking. See, we build models based on predicates for the IDEAL output. We get 50-60% generalization on rare occasions. Picking inputs for the IDEAL output is not so easy, if at all possible. So what if the OUTPUT is UNIDEAL. What if we allow errors into it that will lead to a 100% generalization? By doing so, we will know what the input data is capable of and if this NEEDED output earns (even if with errors) then it has a right to live on in the future as well. At first I thought that we need a double optimization, of the model itself and of the output variable. HOWEVER, then it hit me. If the model is trained up to 60% of generalization, then having put the formulas in the indicator and having received signals on the training and test samples we obtain the result, in which the system works up to 100%. Hence the conclusion. Having received several models with 50-60% generalization, we select the model that has reached (earned) the training and testing phase. This suggests that in general the basic signals it has earned (those that brought sufficient profit), while the signals where it lost were not very loss-making (share your smile and it will return to you more than once), most importantly, that at the end of the period we earned. So that's how it is.......
 
Vizard_:
4.Throw out randomness(or disabled), let it just divide the data in two parts, count in order and output % of successful predictions for 1 (no need for 0)

If trying to use jPrediction for forex, this is probably the main thing to add. There are many models that can split the data randomly into two parts, then train on the first, and show good results on the second part as well. But most of these models will be ineffective on new data. There are no constant dependencies in Forex, good results on a random test sample do not guarantee good results on new data. The only way to make sure the model is suitable for forex is to roll forward test:

The model should show good predictive ability on data marked in red. Now in jPrediction such test has to be done manually, entering data anew for each test case, it's not good for serious work.
And I agree that it's better to show a score for test data only than a total for training+test.
All the grades that jPrediction produces now are inadequate for forex, and only misleading.

By the way, here's how the roll forward is done in the caret package for R:

http://topepo.github.io/caret/splitting.html(Data Splitting for Time Series section)

library(caret)
caretObj <- train(x = trainData[,1:(ncol(trainData)-1)],
                  y = trainData[,ncol(trainData)],
                  trControl = trainControl(method = "timeslice", initialWindow=1000, horizon=300, fixedWindow=TRUE),
                  verbose = TRUE)

The model will be trained on the first 1000 examples, then tested on examples 1001-1300. Then a shift to 300, training at 301-1300, test at 1301-1600, etc. will be done until you run out of training examples

 
Vizard_:
Yuri, make a normal version of the program (or upgraded) for people))))

1.Let it read the standard files of this type - header (with or without quotation marks), data, delimiters let it read zpt, tchk.zpt, tabulation. Example -


2.May use only 10 inputs for the model, but make sure you can load a lot of them, then
You can load a lot of inputs, and then in a separate tab, or you can put caps on them, or something else.

3. without normalization! (or disabled), because if the data is initially centered on 0, after conversion
x-x.min/....(or whatever you have) - fly away)))

4.Throw out randomness (or switchable), let it just divide the data in two parts, counts in order and displays the % of successful predictions for 1 (for 0 do not)

5.Made the importance of inputs - good!

6.Let in addition to the formula that is displayed, outputs the full!, that is now


7. Where is version 7?
  1. The point is that the comma in Excel is used to separate the integer part of the number from the fractional part. It's quite clear that if you put more delimiters, those who edit selections in various add-ons won't be able to put their stuff in jPrediction without additional dancing around. I.e. at the moment everything is made more or less compatible with spreadsheets. At the moment the parser is only for CSV files and it's tailored for spreadsheets. But jPrediction is built in OOP and you can extend interface for other formats.
  2. The second point is that jPrediction doesn't and never will have additional settings. The user's job in jPrediction is to take a ready sample and press F8 to get a model or "Garbage in, Garbage out" message in a reasonable amount of time. If you like fussing about learning algorithms, tuning programs, designing network architecture, instead of researching and finding good data for the task, you should use other ML packages. There are a lot of such packages now, and you can spend the rest of your life digging in them. The main point of machine learning is that the generalizability of the model depends only on the quality of the data in the sample, not on the algorithms and their settings. So, you need to look for problems in the sample, not try to tune something in jPrediction.
  3. See item. 2.
  4. See para. 2
  5. -
  6. There the output is not a formula, but code for porting the model to other applications. At the moment, the code for porting to Java applications is implemented. Later it is planned to port the code of models in MQL. This code should be as concise as possible, rather than a long sprawl. Therefore, all unnecessary things have been removed from it, and everything that is often repeated is placed separately.
  7. The project website is at jprediction.com and there is also a "Download" menu item to download the latest versions.

If someone doesn't like something in jPrediction, the project is open with Open Source under GNU GPL license and making forks from it, changing and modifying something there, is not only not forbidden, but to some extent even welcomed.

 
Dr.Trader:

If trying to use jPrediction for forex, this is probably the main thing to add. There are plenty of models that can divide the data randomly into two parts, then train on the first, and show good results on the second part as well. But most of these models will be ineffective on new data. There are no constant dependencies in Forex, good results on a random test sample do not guarantee good results on new data. The only way to make sure the model is suitable for forex is to roll forward test:

The model should show good predictive ability on data marked in red. Right now in jPrediction such test has to be done manually by reentering data for each test case, it's not good for serious work.
And I agree that it's better to show a score for test data only than a total for training+test.
All grades that jPrediction produces now are inadequate for forex, and only misleading.

By the way, here's how the roll forward is done in the caret package for R:

http://topepo.github.io/caret/splitting.html(Data Splitting for Time Series section)

The model will be trained on the first 1000 examples, then tested on examples 1001-1300. Then a shift to 300, training at 301-1300, test at 1301-1600, etc. will be done until the training examples run out

Super information. Thank you!
 
Dr.Trader:

All of the estimates that jPrediction gives out now are inadequate for forex, and only misleading.

These are unsubstantiated unsubstantiated statements, i.e. what is commonly referred to as bullshit. To prove the effectiveness of Walk Forward, be kind enough to present the results of comparative studies of models obtained in jPrediction and models obtained after Walk Forward, in such a way that they can be cross-checked for "lousiness". If such results confirm your words, then it would make sense to replace the testing algorithm currently implemented in jPrediction with Walk Forward.

I have repeatedly encountered the fact that seemingly "obvious" ideas turned out to be "empty" after being implemented and tested by experience. Only a very small part of ideas is effective, and often only after additional editing "file". Because in ideas the details are missing. And the devil is in the details.

Linus Torvald (creator of the Linux kernel) never gets tired of saying that "theory meets practice sooner or later. And practice is always the criterion of truth. Always!".

An additional problem of Walk Forward is that we get a lot of models at each stage. Which of these models should be kept as a working model and by what criteria should it be selected?

 
Yury Reshetov:

These are unsubstantiated unsubstantiated claims, i.e. what is called bullshit in common parlance. To prove the effectiveness of Walk Forward, be kind enough to present the results of comparative studies of models obtained in jPrediction and models obtained after Walk Forward, in such a way that they can be cross-checked for "lousiness". If such results confirm your words, then it would make sense to replace the testing algorithm currently implemented in jPrediction with Walk Forward.

I have repeatedly encountered the fact that seemingly "obvious" ideas after their implementation and testing by experience, turned out to be "empty". Only a very small part of ideas is effective, and often only after additional editing "file". Because in ideas the details are missing. And the devil is in those very details.

Linus Torvald (creator of the Linux kernel) never tires of repeating: "Theory meets practice sooner or later. And practice is always the criterion of truth. Always!".

An additional problem of Walk Forward is that we get a lot of models at each stage. Which of these models is necessary to keep as a working model and by what criterion should it be selected?

There is no need to quarrel. We are working on a very complex object of study. There are a lot of assumptions. And those assumptions lead to over-learning and out-of-sample plummeting.

Two simple rules: fool the market, don't fool yourself.

Walk Forward is a good method, of course. But the main parameters there are the depth of training and the test length. Moreover (!!!) if these parameters are adjusted in such a way that the result on the forward is the best, we will overtrain Walk Forward! Therefore, even for such a good method we need out-of-sample data. That is, on one large chunk we repeatedly do Walk Forward optimization. On another non-overlapping chunk, we try the best Walk Forward parameters on "training" and do another full wolf forward, but do it once. If the result is good, then the model picks up the dependencies. If the result is bad, then we just retrained Walk Forward on a not very good model.

We have the same problem when testing once on a delayed sample. I'm working on it now: releasing my results from overtraining. The results on crossvalidation should correlate with the deferred sample for final testing. That said, selecting the best model by crossvalidation will give the approximate best out-of-sample results. Otherwise - if there is no correlation or negative correlation - we have an inadequate model for forex that needs to be changed.

 
Yury Reshetov:

These are unsubstantiated unsubstantiated statements, i.e. what is commonly referred to as bullshit. To prove the effectiveness of Walk Forward, be kind enough to present the results of comparative studies of models obtained in jPrediction and models obtained after Walk Forward, in such a way that they can be cross-checked for "lousiness". If such results confirm your words, then it would make sense to replace the testing algorithm currently implemented in jPrediction with Walk Forward.

I have repeatedly encountered the fact that seemingly "obvious" ideas after their implementation and testing by experience, turned out to be "empty". Only a very small part of ideas is effective, and often only after additional editing "file". Because in ideas the details are missing. And the devil is in these very details.

Linus Torvald (creator of the Linux kernel) never tires of repeating: "Theory meets practice sooner or later. And practice is always the criterion of truth. Always!".

An additional problem of Walk Forward is that we get a lot of models at each stage. Which of these models should be kept as a working model and by what criterion should it be selected?

With all my respect to you, Reshetov, it is up to the authors of bicycles to prove their place under the sun, including their terminology, such as "generalizability". The convenience ofDr.Trader' s position is precisely in the fact that all the necessary proofs of both theory and practical testing by millions of users of the ideas he expresses were made before him, often many years ago. He simply gives concrete examples of use. And the correctness of the post under discussion is obvious both intuitively and theoretically.
 
Alexey Burnakov:

It's not worth fighting over.

I agree. But this requires minimizing the number of unsubstantiated accusations in the thread. The best option is to cite the results of comparative studies with the possibility of double-checking.

Alexey Burnakov:

We are working on a very complex subject of research.

Some of us are working, and some of us are talking without evidence (let's not point the finger).

Alexey Burnakov:

It's not worth fighting.

I agree.

AlexeyBurnakov:

There are a lot of assumptions here. And these assumptions lead to overlearning and plummeting out of sample.

You can't put assumptions on your bread and put them in your pocket. That's why you have to check everything by experience, and not make unproven assumptions, so you don't step on the rake of overtraining and out-of-sample plummeting.


AlexeyBurnakov:

Walk Forward is of course a good method. But the main parameters there are the depth of training and length of the test. At the same time (!!!) if these parameters are changed so that the result on the forward is the best, we will overtrain Walk Forward! so even for such a good method we need out-of-sample data. That is, on one large chunk we repeatedly do Walk Forward optimization. On another non-overlapping chunk, we try the best Walk Forward parameters on "training" and run it one more time. If the result is good, then the model picks up the dependencies. If the result is bad, then we just retrained Walk Forward on a not-so-good model.

There is the same problem with one-time testing on a delayed sample. I'm working on it now: releasing my results from overtraining. The results on crossvalidation should correlate with the deferred sample for final testing. That said, selecting the best model by crossvalidation will give the approximate best out-of-sample results. Otherwise - if the correlation is absent or negative - we have an inadequate model for forex that needs to be changed.

The bottom line is that all methods have advantages and disadvantages. Assumptions exaggerate the advantages and disregard the disadvantages.

An even bigger point is that no methods allow you to select models according to the best quality criterion, and all methods select according to the principle of rejection - filtering out only what is known to be unsuitable. No method in ML can calculate the potential suitability of what is left after the rejection.

Especially in non-stationary environments. It would seem possible to select the most suitable models in terms of generalizability, but tomorrow some head of the Central Bank at a meeting farts something and all models will go down the drain.

 
Yury Reshetov:


The point is that all methods have advantages and disadvantages. Assumptions exaggerate the advantages and disadvantages.

An even bigger point is that no methods allow you to select models according to the criterion of better quality, and all methods select according to the principle of rejection - filtering out only what is known to be unsuitable. No method in ML can calculate the potential suitability of what is left after the rejection.

Especially in non-stationary environments. It would seem possible to select the most suitable models in terms of generalizability, but tomorrow some head of Central Bank will make a fart and all models will go down the drain.

So it is. Only experimentally. You can understand whether the choice of the model (in training + testing) was better or not by applying this model out of sample. You can do this for different models. Where the selection of a good model in the sample gives a good result out of sample - means that the model fits the chosen problem.
1...818283848586878889909192939495...3523
New comment
Reason: