Machine learning in trading: theory, models, practice and algo-trading - page 3463

 
Forester #:

4 million columns? What about rows? If at least 100 thousand, that's probably months of calculations.....

And 4 million is not enough, at this stage of my understanding of the problem I need 20 million, but I don't know how to cope even with 4 million, it's really a lot....

Moreover, the signs are compressed 50-100 times from the usual representation...

 
mytarmailS #:

lines +-200

is a filter of one pattern, not an attempt to explain the whole market.



And 4 million is not enough, at this stage of my understanding of the problem I need 20 million, but I don't know how to cope even with 4 million, it's really a lot....

And the signs are already compressed 50 to 100 times the normal representation...

And less features can't filter it out?

 
Maxim Dmitrievsky #:

Can't fewer traits filter it out?

In the end, of course, there will be a handful of gold left, but before that you have to go through a tonne of rock....

In fact, I've trivially replaced the huge search of "everything with everything" with a dataset for the model, thus reducing the search by thousands of times.

Instead of searching through trillions of patterns, I have a model with millions of features that can do the same thing as trillions of patterns, but all at once.


There is also a problem of how to store and retrieve it all correctly and quickly.

I tried Apache Arrow, cool, but it's not the same, or I didn't twist it right.

 
mytarmailS #:

You'll end up with a handful of gold, but you'll have to go through a tonne of rock.

In essence, I have rather trivially replaced the huge "everything with everything" dataset for the model, thereby reducing the search by thousands of times.

Instead of searching through trillions of patterns, I have a model with millions of features that can essentially do the same thing as trillions of patterns, only here it's all at once.


There is also the problem of how to store and retrieve it all correctly and quickly.

I tried Apache Arrow, cool, but it's not the same, or I didn't twist it right.

Nothing clear, but very interesting )

 
mytarmailS #:

You'll end up with a handful of gold, but you'll have to go through a tonne of rock.

In essence, I have rather trivially replaced the huge "everything with everything" dataset for the model, thereby reducing the search by thousands of times.

Instead of searching through trillions of patterns, I have a model with millions of features that can essentially do the same thing as trillions of patterns, only here it's all at once.


There is also the problem of how to store and retrieve it all correctly and quickly.

I tried Apache Arrow, cool, but it's not the same, or I didn't twist it right.

Try fst. It compresses well and allows arbitrary access from a compressed file.
 
Vladimir Perervenko #:
Try "fst." It compresses well and allows arbitrary access from the compressed file

did you read my question in the database cart? :)

Can it read/filter by condition like x[ A==3 & B>=4, ]
or only from\ to

I can't find it.


So far I'm doing everything through apache arrow
 

Invented and implemented ZZ for ticks overnight in Python :) What's new with you guys?

One setting only. Below are two options with a setting of 50 and 25.



 
Aleksey Vyazmikin #:

Invented and implemented ZZ for ticks overnight in Python :) What's new with you guys?

One setting only. Below are two options with a setting of 50 and 25.

What a beauty. Teach an illiterate what it is for, what are the deals to mark on it?
 
Maxim Dmitrievsky #:
What a beauty. Teach an illiterate, why it is necessary, deals on it to mark?

And why not to mark up the target....

Okay, I'll tell you - for quantisation :))))))

 
Aleksey Vyazmikin #:

Why don't you do the markings?

OK, I'll tell you - for quantisation :)))))

Tick quantisation?

I tried neovim on mac. Such pornography, vscode is better

I'm also writing my own bible, because I've accumulated a lot of algorithms, so I decided to divide functions into libraries to use as a constructor later.