Machine learning in trading: theory, models, practice and algo-trading - page 3270

New comment
 
fxsaber #:
Right, thanks! I don't understand why the wrong option worked with inCols < 100.

Apparently 

Sens = 1 e-10
too much. Random - the average correlation there is about 0, probably.
 
Forester #:

Probably too much. Random - there is an average correlation of about 0, probably.

It is not the average error that is measured there, but the maximum difference between the corresponding elements.

Forum on trading, automated trading systems and testing trading strategies

Machine learning in trading: theory, models, practice and algo-trading

fxsaber, 2023.10.01 09:38 pm

bool IsEqual( matrix<double> &Matrix1, const matrix<double> &Matrix2, const double Sens = 1 e-10 )
{
  Matrix1 -= Matrix2;  
  
  const bool Res = (MathMax(MathAbs(Matrix1.Max()), MathAbs(Matrix1.Min())) < Sens);
  
  Matrix1 += Matrix2;
  
  return(Res);
}


That's why it's not clear how the wrong code

  for (int i = 0; i < (int)Matrix.Rows(); i++)

gets a match.

inRows = 5 inCols = 50 
matrix<double> Matrix1 = CorrMatrix(Matrix) - 242 mcs, 0 MB
matrix<double> Matrix4 = CorrMatrix2(Matrix) - 117 mcs, 0 MB
IsEqual(Matrix1, Matrix4) = true
 
Forester #:
And PearsonCorrM2 can be speeded up by 2 times if you count by triangle.i.e. go from the end. Count 100 line with all, then 99 with all 0-99, 99 and 100th already counted - you can just copy. ...50th line with all to 50th, etc. Well, do not count with itself because =1.
I'm afraid there's no point. 
inRows = 100 inCols = 15000 
matrix<double> Matrix1 = CorrMatrix(Matrix) - 14196778 mcs, 1717 MB
matrix<double> Matrix4 = CorrMatrix2(Matrix) - 538256223 mcs, 1717 MB
IsEqual(Matrix1, Matrix4) = true
The terrible brake will still slow down. Here, unequivocally, you can't do without self-made tools.
 
fxsaber #:

That's why it's not clear how, with the wrong code.

gets a match.

I figured it out, if you swap the matrix calculations, you get a mismatch.

inRows = 5 inCols = 50 
matrix<double> Matrix4 = CorrMatrix2(Matrix) - 113 mcs, 0 MB
matrix<double> Matrix1 = CorrMatrix(Matrix) - 214 mcs, 0 MB
IsEqual(Matrix1, Matrix4) = false

I.e. rubbish from memory during the calculation of the first matrix got into the new matrix and by some miracle coincided with the desired result.

 
fxsaber #:

I think it's a head-on calculation of the correlation matrix.

Yes, if we make μl without cycles on new matrices, won't it be faster?
 
fxsaber #:
196

500 seconds versus 14 - that's why I remember it being 

PearsonCorrM

the fastest, due to the algorithm.

Maxim Dmitrievsky #:
Well, yes, if you make it on new µl matrices, won't it be faster?

I think all 9 functions used in PearsonCorrM and PearsonCorrM2 can be rewritten to matrices and compared. In principle, it would take an hour of work to rewrite the matrix declarations and references. At the same time we will find out whether matrices are better than din. arrays.

Titles
IsFiniteMatrix(
IsFiniteVector(
AblasInternalSplitLength(
AblasSplitLength(
RMatrixGemmK(
RMatrixGemm(
RMatrixSyrk2(
RMatrixSyrk(
RankX(

 
Forester #:

I think all 9 functions used in PearsonCorrM and PearsonCorrM2 can be rewritten to matrices and compared. In principle, it will take an hour of work to rewrite the matrix announcements and references. At the same time we will find out whether matrices are better than din. arrays.

Everything is already done: MQ have rewritten them for their matrices.

Forester #:

500 sec vs 14 - that's why I remembered that.

the fastest, because of the algorithm.

I didn't realise the algorithm there. NumPy does not lag far behind only due to the fact that it does not perform repeated calculations.

NumPy seems to have a different algorithm from ALglib, since the performance is very different. But it is clear that in the whole huge Python-community there was some very strong algorithmist who devoted a decent amount of time to studying this issue.
 
fxsaber #:

It's already done: the MQs have rewritten them to fit their matrices.

And it got slower?)

 
Forester #:

And it got slower ??))))

Compare to the old version of Alglib. I have no data that it has become slower.

 
fxsaber #:
NumPy seems to have a different algorithm from ALglib.

Maxim's CPU is twice as fast as mine. I don't remember if he gave timings for Algliba, I think not.

1...326332643265326632673268326932703271327232733274327532763277...3656
New comment