Machine learning in trading: theory, models, practice and algo-trading - page 1934

New comment
 
Valeriy Yastremskiy:

And what do you want to get as a result?, with harmonics goes to Fourier. Or to orthogonal polynomials, there are many of them. Everything can be decomposed.)

I don't need to decompose, I need to create. I need to create a function on time that will satisfy my condition, what this function is I do not know, so I need to create a search.

Probably the overshoot by harmonics but I will consider all options.

It is essentially the same approximation, but the overshoot is not to reduce the error but to meet the condition.


So, I want to know how this search is organized , even in a simple harmonic approximation, what is multiplied there to change the function?

How does the mechanism for adjusting one function to another work in general?
 
mytarmailS:

I do not need to decompose, I need to create. I need to create a function of time, which will satisfy my condition, what this function I do not know, so I need to create overshoot.

Probably overkill with harmonics, but I will consider all options.

It is essentially the same approximation, but the overshoot is not to reduce the error but to meet the condition.


So I want to know how this search is organized , even in a simple harmonic approximation, what is multiplied there to change the function?

How does the mechanism of adjusting one function to another work in general?

I will not say in formulas, in the sense of decomposition from the largest to the smallest period. You find the largest harmonic, subtract it from the real signal, find the largest one from the rest, and subtract it to the required accuracy. And in the approximation (Approximation representation of the real signal by straight broken linear functions) we represent the harmonic by discrete sections of straight lines. If the discretization is half a period (if more than that, nothing comes out), then we will get P or straight lines between extrema, if a quarter or more, then broken lines. And if we decrease the sampling step, we increase the precision.

If price selection is long, it can be broken down into shorter sections and linear, power, harmonic, and logarithmic functions can be enumerated/selected in these sections. Ideally, harmonics and linear orthogonal should solve the problem. Proven more than once. But only on history.

There is no point with the market, we have changing functions / mathematical models over time, and no clear criteria, no proven how much data is needed to determine the mathematical model and its mismatch with the same BP, other than discreteness in half period will not give results, it is proven). All empirically or NS with MO) but also experientially.

There is no function with changing conditions, we have one set of harmonics on this section, another on the next. Discretization of sound is 48 kilohertz, twice as much as we hear, so at high frequencies we lose. And digital has 2 signal levels per period of 1/48000 sec.

 
mytarmailS:

I don't need to decompose, I need to create. I need to create a function of time, which will satisfy my condition, what this function I do not know, so I need to create overshoot.

Probably overkill with harmonics, but I will consider all options.

It is essentially the same approximation, but the overshoot is not to reduce the error but to meet the condition.


So I want to know how this search is organized , even in a simple harmonic approximation, what is multiplied there to change the function?

How does the mechanism of adjustment of one function to another function work in general?
harmonic, is it through Fourier? there are simply sinusoids of different periods and amplitudes added up A1*cos(2Pi*t/T1+Fi1)+ A2*cos(2Pi*t/T2+Fi2) +... A - amplitude, T - period, Fi - phase
 
mytarmailS:

How many times have I told you that? 3? 5?

So why did they come up with this, and the video that was posted earlier with a man who sort of used this miracle-devil and made the conclusion that the sample is suitable for training ...

 
Aleksey Vyazmikin:

Well, why did they invent it, and the video that was posted earlier with a man who sort of applied this miracle-devil and made the conclusion that the sample is suitable for training ...

he used t-sne (like) but it's not important, it works and it works great but not for the market like everything else, download any dataset from the internet and you see that the thing works great but ...

But I can draw some conclusions: clusters are stable and stable but they don't resonate with profit, the target resonates with profit but moves around clusters, probably because each target is a guess, but cluster is an objective reality. I'll try to recognize patterns using historical analogues, not using prices but using umap as I wanted from the beginning.


Rorschach:
harmonic, is it through Fourier? It's just sinusoids of different periods and amplitudes added up A1*cos(2Pi*t/T1+Fi1)+ A2*cos(2Pi*t/T2+Fi2) +... A - amplitude, T - period, Fi - phase

Damn, it's complicated.... I understand about the harmonics, but probably from the fan to collect from the harmonics series and try on is stupid, probably it should be done as a dependence of something on something, it's hard, damn it .... but if you do it, it 's a grail

 
mytarmailS:

Feature selection[edit|edit code]

Main article:Feature selection

Thefeature selection method tries to find a subset of the original variables (which are called features or attributes). There are three strategies - a filter strategy(e.g.,feature accumulation [en]), awrapping strategy(e.g., search according to accuracy), and anembedding strategy(selecting attributes to add or remove as the model builds based on prediction errors). See alsocombinatorial optimization problems.

In some cases,data analysis, such asregression orclassification, can be performed in the reduced space more accurately than in the original space [3].

Feature projection[edit|edit code]

Feature projection converts data fromhigh dimensional space to low dimensional space. The data transformation can be linear, as inthe Principal Components Method(PCM), but there are a large number ofnonlinear downsizing techniques [en] [4] [5]. For multidimensional data,a tensor representation can be used to reduce dimensionality viapolylinear subspace learning [en] [6].

I missed the text with your explanations - I correct myself.

I immediately have a number of questions:

1. How to get the result of feature selection with their transformation in the form of any rules for separate use in the code?

2. Maybe there is a way to visualize the selected features and their transformation - through a tree or something else?

3. Have you tried these feature selection strategies?

Thefeature selection method tries to find a subset of the original variables (which are called features or attributes). There are three strategies - a filter strategy(e.g.,feature accumulation [en]), awrapping strategy(e.g., search according to accuracy), and anembedding strategy(selecting attributes to add or remove as you build a model based on prediction errors). See alsocombinatorial optimization problems.

In some cases,data analysis, such asregression orclassification, can be done in reduced space more accurately than in the original space [3].

mytarmailS:

That's what we did yesterday.

Dimensionality reduction[edit|edit code]

For high dimensional datasets (i.e., those with more than 10 dimensions), dimensionality reduction is usually done before applyingthe k-nearest neighborsalgorithm( k-NN), to avoid the effect ofthe curse of dimensionality [16].


Advantages of dimensionality reduction[edit|edit code]

  1. It reduces the time and memory required.
  2. Removing multicollinearity improves the speed of the machine learning model.
  3. It's easier to represent the data visually when reduced to very low dimensions, such as 2D or 3D.

And it seemed to me from the code that clustering was separate, projection construction was separate, and then we just fed the clustering results for color grading when rendering - no?

 
mytarmailS:

he used t-sne (like) but it doesn't matter, well yes they came up with it, and it works and works great but not for the market, like everything else, download any dataset from the internet and see that the thing works great but...

But I can draw some conclusions: clusters are stable and stable but they don't resonate with profit, my target resonates with profit but moves around clusters, probably because each target is a guess, but the cluster is an objective reality. I'll try to recognize patterns using historical analogues, not using prices, but using umap as I wanted from the beginning.

So how do you get their markup on the history for the classification? And then, well recognized, and then what? The other day I was distributing across 4 clusters searching for targets... Unless you change the target from cluster to cluster. In general, we need to understand why different clusters of lines flew in - to analyze the logic, if there is one and is contiguous with the market.

 
Aleksey Vyazmikin:

I missed the text with your explanations - I stand corrected.

I immediately have a number of questions:

1. How to get the result of feature selection with their transformation in the form of any rules for separate use in the code?

2. Maybe there is a way to visualize the selected features and their transformation - through a tree or something else?

3. Have you tried these feature selection strategies?

4. And it seemed to me from the code that clustering was separate, building the projection separately, and then we just fed the clustering results for color grading when rendering - no?


1. it's the same as wanting 100k closing prices described by rules for separate application in the code

2. the attributes come in the form of functions with all that implies

I tried something; it reduces dimensions, not increases quality, but it's still cool, e.g. I can compress 10k attributes to 500 without losing quality; it's very cool, or to 50 with an acceptable loss.

4. that's how it was, we decreased dimensionality first, then we clustered it, it says so here - ----- For high dimensional datasets (i.e. with number of dimensions greater than 10),decreasing dimensionality is usually donebefore applyingk-nearest neighbors method

 
mytarmailS:


1. this is the same as wanting to describe 100s of closing prices with rules for separate use in code

2. Characteristics are obtained as functions with all their consequences.

3. I tried something; it reduces dimensionality, but quality does not increase, but it is still cool, e.g. I can compress 10k attributes to 500 without quality loss, which I find very cool, or to 50 with acceptable loss.

4. that's how it was, we first reduced dimensionality then clustered it, it says so here - ----- For high dimensional datasets (i.e. with more than 10 dimensions),dimensionality reduction is usually donebefore applyingthe k-nearest-neighbor method

1. You must have misunderstood the question - it's just about unloading the rules to a file, and there are rules, which was confirmed in point 2.

2. That's good.

3. Likely high correlation between predictors, including after conversion? Is it possible to get a list of retired predictors or are they not retired but merged into groups just - can't figure it out.

4. Maybe I misunderstood the code or clustered it wrongly, I'll post it again

#way <- "F:\\FX\\R\\tree_classification_2019_fS_Eks_29\\Test_Pred_ALL_01.csv"  #  ваш путь
way <- "F:\\FX\\R\\2020_04_11_Pred_New_Full\\Pred_New_Full.csv"  #  ваш путь

dt <- read.csv(file = way,header = T,sep = ";") #  читаем файл

target <- dt$Target_100 #  целевую в отдельную переменную

dt <- dt[, ! colnames(dt)  %in% 
           #            c("Target_100_Buy","Target_100_Sell",
           #             "Time","Target_100")  ] #  удаляем не нужные колонки
                      c("Target_100")  ] #  удаляем не нужные колонки

km <- kmeans(um$layout, centers = 4)           

#  роскоментируем и Устанавливаем нужные пакеты после чего эту строку можно удалить
#install.packages(c("rgl", "car" , "umap"))



#  про umap
#https://cran.r-project.org/web/packages/umap/vignettes/umap.html
#https://github.com/ropenscilabs/umapr

library(umap)
um <- umap(dt,n_components=3)   
#  n_components=3  во сколько измерений уменьшаем данные, можно 100 можно 1
#  можно 333, но нам для 3 д надо трех мерное пространство поетому у нас n_comp=3
um.res <- um$layout #  наши три вектора выход 



#тут  все настройки по пакету  car
#  http://www.sthda.com/english/wiki/amazing-interactive-3d-scatter-plots-r-software-and-data-visualization
library(car)  # 3 d

target <- as.factor(target)  #   target нужен для того чтобы окрасить точки в цвет целевой

scatter3d(x = um.res[,1], 
          y = um.res[,2], 
          z = um.res[,3],
          groups = as.factor(km$cluster),
          grid = FALSE, 
          surface = FALSE,
          ellipsoid = TRUE,
            bg.col = "black")


write.csv(km$cluster, file = "F:\\FX\\R\\2020_04_11_Pred_New_Full\\Pred.csv", sep = ";",row.names = F,col.names = T)
 
Aleksey Vyazmikin:

1. Apparently you misunderstood the question - it was just about what would unload the rules in the file, and the rules are, which is confirmed in paragraph 2.

2. This is good.

3. Likely high correlation between predictors, including after conversion? Is it possible to get a list of retired predictors or are they not retired but merged into groups just - can't figure it out.

4. Maybe I misunderstood the code or clustered it wrong, I'll post it again

1. I still don't understand it.

3. there can't be correlated features after transformation. Yes, you can say that they are merged into other structures (attributes), but without redundancy

4.

dt <- dt[, ! colnames(dt)  %in% 
           #            c("Target_100_Buy","Target_100_Sell",
           #             "Time","Target_100")  ] #  удаляем не нужные колонки
                      c("Target_100")  ] #  удаляем не нужные колонки

) It is one expression, and you cannot comment on it))

1...192719281929193019311932193319341935193619371938193919401941...3591
New comment