Machine learning in trading: theory, models, practice and algo-trading - page 265

 
SanSanych Fomenko:
Tried it, but quickly broke down: it is not clear what to do with the holes in the output.

Holes on weekends are taken at discos :)

When are you going to learn to quote properly? To write three words you don't have to quote half a page of the forum...

Now more on the subject

I made various candlestick combinations and other stuff with the help of the I tried to traincandlesticks, it turned out that it trains 100% and error 0 on both training and new data, obviously there are some predictors in the data that look into the future, so i did not have to look into each function i just removed the first 6 predictors that according to forrest were the most significant and stood out from the others, i hoped that among them were those that look into the future and trained the MO again, the error was about 3%, in short i do not know what the hell, why such fabulous results, maybe the error in my code ...

If anyone wants to know what to do, here's the code, the target was "color of candle".

In the code, just the sample itself, the target and other manipulations, do what you want.

Try it, experiment because it's like an additional 30 attributes to your model

library(rusquant)
# загружаю последние 500 дней котировок индекса ртс
getSymbols("SPFB.RTS",src = "Finam",period="5min",from = Sys.Date()-500)
chart_Series(  tail(SPFB.RTS,100)  )

D <- SPFB.RTS

# cчитаем функции по свечным формациям и прочим добром
library(candlesticks)

X29<- TrendDetectionSMA(D)  
X28<- TrendDetectionChannel(D)  
X27<- nextCandlePosition(D)  
X26<- CSPThreeOutside(D)
X25<- CSPThreeMethods(D)
X24<- CSPThreeInside(D)
X23<- CSPTasukiGap(D)
X22<- CSPStomach(D)
X21<- CSPStar(D)
X20<- CSPShortCandleBody(D)
X19<- CSPShortCandle(D)
X18<- CSPPiercingPattern(D)
X17<- CSPOutsideDay(D)
X16<- CSPNLowerClose(D,N = 3)
X15<- CSPNHigherClose(D,N = 3)
X14<- CSPMarubozu(D)
X13<- CSPLongCandleBody(D)
X12<- CSPLongCandle(D)
X11<- CSPKicking(D)
X10<- CSPInvertedHammer(D)
X9 <- CSPInsideDay(D)
X8 <- CSPHarami(D)
X7 <- CSPHammer(D)
X6 <- CSPGap(D)
X5 <- CSPEngulfing(D)
X4 <- CSPDoji(D)
X3 <- CSPDarkCloudCover(D)
X2 <- CandleLength(D)
X1 <- CandleBodyLength(D)


dat <- cbind.data.frame(D, X1,X2,X3,X4,X5,X6,X7,X8,X9,X10,
                        X11,X12,X13,X14,X15,X16,X17,X18,X19,
                        X20,X21,X22,X13,X24,X25,X26,X27,X28,X29)


# true,false заменяю на 1 и -1 а NA-шки на 0
dat[dat==TRUE]     <- 1
dat[dat==0]        <- -1
dat[is.na(dat)]    <- 0

# пишем все в файл с которым удобно работать в будущем
save(dat,file = "D:/R/candles_lib/candle_dat.RData")  # ваш путь

 
mytarmailS:

Sanych, when are you going to learn to quote properly?

Predictors don't look into the future - they repeat the target.

Where is the forward test? Why discuss the training?

PS

All at the request of

 
SanSanych Fomenko:

Predictors don't look into the future-they repeat the target.

Where is the forward test? Why discuss the training?

Just try to teach yourself and write what you got, and then there will be something to discuss, I hope ...
 
mytarmailS:
Just try it on your own and write what you get, and then there will be something to discuss, hopefully...

I understand the reason.

Now a new price came in, all the predictors were calculated on it, and then the same price was calculated on these predictors - we predict nothing at all. The fitting error is zero. I don't see anything surprising.

If the prediction is interesting, then D shift one step to the left. This model predicts one step forward. you can predict several

 
SanSanych Fomenko:

I understand the reason.

Now a new price came in, all the predictors were calculated on it, and then the same price was calculated on these predictors - we predict nothing at all. The fitting error is zero. I don't see anything surprising.

If the prediction is interesting, then D shift one step to the left. This model predicts one step forward.

What do you mean the new price came in?

The code I gave you just makes predictors from prices that are downloaded from finam.

then the data is saved

# пишем все в файл с которым удобно работать в будущем
save(dat,file = "D:/R/candles_lib/candle_dat.RData")  # ваш путь

Then we open this data inthe new script, set the targetprice and teach the MO.

There is no new price there. This is not the first time I've been doing this kind of training.

So something else is there

Or I do not understand you?
 
mytarmailS:

What do you mean the new price came?

the code I gave just makes predictors from the prices downloaded from finam

then the data is saved

# пишем все в файл с которым удобно работать в будущем
save(dat,file = "D:/R/candles_lib/candle_dat.RData")  # ваш путь

Then we open this datain a new script, make a target and teach the MO.

There is no new price there. Of course, I shifted the target one stepwise, according to the forecast.

So something else is going on.

Or I do not understand you?
What's the target?
 
SanSanych Fomenko:
What's the target?

cloze is higher than the previous cloze

load(file = "D:/R/candles_lib/candle_dat.RData")

Y <- diff(dat$SPFB.RTS.Close)
Y[Y>=0] <- 1
Y[Y<0] <- 0

dat <- dat[-nrow(dat),]
Y <- as.factor(Y)

tr <- 1:10000
ts <- 10001:15000


library(randomForest)
cm <- colnames(dat)
colnames(dat) <- paste0("var_" , 1:ncol(dat))

model <- randomForest(Y[tr]~., dat[tr,]  , ntree=100, mtry=10)

layout(1:2)
plot(model)
varImpPlot(model,type = 2)

pr <- predict(model,dat[ts,])


library(caret)
confusionMatrix(Y[ts] , pr)
 
mytarmailS:

cloze above below the previous cloze

I can't figure out where the target shift is? When differentiating? But the first line of predictors, which is obtained by subtracting from 2, we match the 1st value of the target, i.e. we KNOW the next values for the target with index 2.
 
SanSanych Fomenko:
I can't figure out where the shift in the target is? When differentiating? But the first line of predictors, which is obtained by subtracting from 2, we match the 1st value of the target, i.e. we KNOW the next values for the target with index 2.

When differentiating, the shift is automatic, as the series becomes one element shorter, and then all that is needed is to shorten the sample (table with observations) by the last element

here is an example

SomeData <- c(10,20,30,20,10,20,30,40,50,40)

Y <- diff(SomeData)

cbind.data.frame(  Y , SomeData[-length(SomeData)])


we get

   Y                   SomeData[-length(SomeData)]
1  10                          10
2  10                          20
3 -10                          30
4 -10                          20
5  10                          10
6  10                          20
7  10                          30
8  10                          40
9 -10                          50
 
mytarmailS:

When differentiating, the shift is automatic, as the row becomes one element shorter, and then all that is needed is to shorten the sample by the last element

It is not the predictors that needto be shifted to the left, but the target.

Let me try to explain it once again.


The line of predictors with index 1 is obtained = from the SECOND and the first line of predictors. The result is written in place with index 1.

The target one still has index =1. But we predict its value on the basis of information from the FUTURE, second line of predictors.

Shift the target by 1, i.e. instead of Y[1:n] take Y[2:n] and calculate