Machine learning in trading: theory, models, practice and algo-trading - page 3496

New comment
 
Aleksey Vyazmikin #:

I didn't understand how the markup of the target is done in the proposed method, how the link to predictors is established.

Why I want something meaningful - because my task is to reduce classification error through estimation of unstable quantum segments, and for this purpose I want to look at areas where error occurs - visualise them, maybe. To try different approach to estimation and to see the result is reliable, that for example on a significant predictor only 5 useful sites out of 10 were detected, I need to understand how they differ.

There are many other interesting things you can do besides quantisation. Why was it chosen?


 

Your approach is to look for segments of the available BP in order to extrapolate the results into the future. It has a right to life, but you are always limited in data.

The usual practice is the opposite, adding synthetic data. Or a combination of 1 and 2. This way you kind of increase the probability of finding patterns that are actually stable but underrepresented in the original sample.
 
Maxim Dmitrievsky #:

You can do many other interesting things besides quantisation. Why was it chosen?

Historically, I needed a method to speed up computation for building a tree on genetics and a method to create different subsamples from which leaves could be obtained - you can think of it as clustering and training on a separate cluster. The diversity of leaves was to ensure that all data plots were covered by groups of similar leaves in terms of response location but different in their components (predictors and essentially patterns), thus providing a stable voting and decision making system.

In the end I got interested in quantum segments, and saw visually through them what prevents the creation of stable models. I started to think how to eliminate these reasons. Over time I accelerated my code thousands of times, invented metrics and analytics.... Developed a method of data cleaning, which in theory should improve classification by standard methods. So far there are still ideas where to develop.

Maxim Dmitrievsky #:

Your approach is to search for segments from the available BP in order to extrapolate the results into the future. It has a right to life, but you are always limited in data.

Usually it is practised the other way round by adding synthetic data. Or a combination of 1 and 2. That way you kind of increase the probability of finding patterns that are actually consistent but underrepresented in the original sample.

As I understand it, there are two directions in data augmentation:

1. Generating a new sample (or saturating it) as noise from given distributions based on existing data.

2. Generation of raw data on which predictors are computed.

The first option is questionable, as there is already enough noise, and the question of marking the target becomes no longer an unambiguous decision. When using CatBoost, the effect will not be obvious, as data quantisation is used and often we will get just a uniform increase of examples in quantum segments, which will not affect the improvement of learning. And, as we've sort of tried this with you before - the effect was more often not than it was.

The second option seems more interesting, but this is if we are sure in the presence of these patterns, and as my experiments show - in the data of the same quantum segments with a stable bias in the range of 20%-35% (well, this is on individual data - we need to study more samples), so it may turn out that we cultivate false patterns. Of course, by changing settings and methods, you can build a good model, but it will be random. The model will pass all conceivable tests, but again there are no guarantees on new data. But it will be possible to explore the possibilities of the approach - if I am wrong.

I want to generate time series with inherent regularities and noise - and already on them to evaluate these or those methods. In this way, I hope to see what difficulties arise in different known classification algorithms and to improve my methods of data preprocessing and model building. I've sketched out a plan of what to do, but it's not clear yet how to realise some aspects better.

 
The second option, yes.
Lots of ways to do it. I even tried an extreme one, like the V-M function :)
And for each VR you have to pick something different.
If stability is found and verified via cv etc., little doubt remains.
 
Maxim Dmitrievsky #:

You can do many other interesting things besides quantisation. Why was it chosen?


or at least linear trend modelling...

We write a function that calculates linear regression.

find_lr <- function(p){
  y <- coredata(p$Open)[,1]
  x <-  1:nrow(p)
  lr <- lm(y ~ x)
  
  a <- coef(lr)[1]
  b <- coef(lr)[2]
  
  reg_line <- a + b * x
  reg_line
}

Then we take some prices and calculate the linear trend with the function find_lr.

library(quantmod)
n <- 5000
p <- rnorm(n) |> cumsum() |> xts(Sys.time()+1:n) |> to.minutes(name = NULL)

chart_Series(p)
reg_line <- find_lr(p)
lines(reg_line)

We delete the trend and go to the residuals.

res <- p - reg_line
chart_Series(res)
abline(h=0)

and now we add our own trend that we want.

par(mfrow=c(3,3))
for(i in seq(-100,100,by=25)){
  trend <- seq(-i,i, length.out=nrow(p))
  Sys.sleep(0.2)
  yy <- res+trend
  print(chart_Series(yy))
  lines(find_lr(yy))
}

The data are the same but the trends are different, this can be really useful against overtraining, to be honest I have never used it.

 
mytarmailS #:

The data is the same but the trends are different, it can be really useful against retraining, to be honest I have never used it.

If the TS is working, it should work on all trends, right?

And what is interesting is that the optimal take and stops on all trends will be different, right?!?! (this is not a question).

 
mytarmailS #:

or at least linear trend modelling.

write a function that calculates linear regression.

then we take some prices and calculate the linear trend with find_lr.

remove the trend and go to the residuals

and now we add our own trend as we want.

The data is the same but the trends are different, this can really be useful against overtraining, to be honest I have never used it.

Yes, it's also one of the options for data augmentation.

By the way, it can be extended by adding non-linear trends.
 
Maxim Dmitrievsky #:
 By the way, you can extend it by adding non-linear trends.

Yeah, I just didn't want to complicate the example.

And it's not clear how far you can go, because you can add so much nonlinearity that BP ceases to be a price.

 
mytarmailS #:

Yeah. I just didn't want to complicate the example.

And it's not clear how far you can go, because you can add so much non-fiction that BP is no longer a price.

It depends on the data. What will work better and what will work worse. So far I have not formulated any augmentation logic for myself.

 
Maxim Dmitrievsky #:

depends a lot on the data. What goes in better and what goes in worse. I have not yet formulated any augmentation logic for myself.

I conducted an experiment simulating a trained trading model (below with arrows/transactions).

(on the left is the original model on the right is the effect of changes on the model).



Effect of changing the linear trend


linear trend does not affect the trained TS (but whether there is a linear trend in the market).



Influence of fluctuation amplitudes

Changes in amplitudes affect the TS.



Effect of Phase Shift


Phase shift affects TC


Effect of Frequency Shift

Frequency shift affects TC



the whole code of the experiment )))

s <- function(ve=1:500,a,f,p)    a*sin(f*ve+p)
par(mfcol=c(2,2), mar=c(2,2,2,2))

s1 <- s(a = 0.2, f = 0.13, p = 2)
s2 <- s(a = 0.1, f = 0.12, p = 3)  
tre <- seq(-0.3,0.3,length.out=length(s1))

y <- s1+s2+tre
x <- cbind(s1,s2,tre)

library(quantmod)
#va <- findValleys(y)
#pi <- findPeaks(y)

matplot(x, t="l", lty=1)
plot(y,t="l")
points(va, y[va]-0.02, col=3, pch=2, lwd=2)
points(pi, y[pi]+0.02, col=2, pch=6, lwd=2)
1...348934903491349234933494349534963497349834993500350135023503...3512
New comment
Reason: