Mysterious stock indicator is a problem with copy of CopyTicks - Trading Systems

fxsaber 2016.09.30 19:16 #181

1434 - 1ms on average takes to get CopyTicks already uploaded 1000 ticks. Slow, it seems.

Requesting TRADE0tic with from_msc of the last tick received earlier. I get 3 ticks, but in 0.3 - 0.9ms! - Very slow now.

Future price stack with Services , new functionality How I can copy

fxsaber 2016.10.01 06:05 #182

fxsaber:

Strongly logged the code above and figured out the reasons. If CopyTicks (from > 0) gets ticks before the freshest one, it can skip some.

A very important reply from Service Desk.

Regarding the original problem - that CopyTicks on the next call can give more ticks for the same period:

This is indeed the case. The problem is that exchange data streams bid/ask and flipper/volume are different streams, which are not synchronized with each other already on exchange side.

Because of this there are situations, when first comes bid/ask with time 12:12:12.300, and later comes flipper/volume with time 12:12:12.299.

Accordingly, requesting data since the last tick (12:12:12.300) you will not get a new flipper for 12:12:12.299.

PS. The terminal saves and sends ticks sorted by time. That is, time sequence of ticks given to CopyTicks is always increasing.

There are two streams of ticks receiving - INFO and TRADE. ALL is a synthesized union (seems to be on terminal side), that's why such mishaps may occur.

It's because of synthesized that there were such words

Slawa:

initial tick records after the call of CopyTicks will contain not zeros, but the current values of bid, ask and last at the requested moment of time

So when working with ALL ticks, you need to be clearly aware of what you are dealing with. It is possible that tick flags are synthesised as well. I would like full specificity on these issues.

With the tape this problem should not arise with correct operation of CopyTicks.

I think, the Help will be very seriously supplemented.

Push notification does not [Archive!] Any rookie question, VPS and mql5 Storage

[Deleted] 2016.10.01 10:48 #183

fxsaber:
You can add any overloads yourself.

I can do a lot, you can do a lot, other programmers can do a lot too, but hopefully the developers know the "stuffing" better and can create quick algorithms to get the necessary ticks.

prostotrader 2016.10.03 12:55 #184

Tested CopyTicks with COPY_TICKS_TRADE flag

No difference.

2016.10.03 15:50:48.507 Check_ticks (RTS-12.16,M1)      History ticks = 9
2016.10.03 15:50:48.507 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 9
2016.10.03 15:50:48.956 Check_ticks (RTS-12.16,M1)      History ticks = 11
2016.10.03 15:50:48.956 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 11
2016.10.03 15:50:49.184 Check_ticks (RTS-12.16,M1)      History ticks = 12
2016.10.03 15:50:49.184 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 12
2016.10.03 15:50:49.510 Check_ticks (RTS-12.16,M1)      History ticks = 14
2016.10.03 15:50:49.511 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 14
2016.10.03 15:50:51.568 Check_ticks (RTS-12.16,M1)      History ticks = 15
2016.10.03 15:50:51.568 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 15
2016.10.03 15:50:51.627 Check_ticks (RTS-12.16,M1)      History ticks = 16
2016.10.03 15:50:51.627 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 16
2016.10.03 15:50:53.143 Check_ticks (RTS-12.16,M1)      History ticks = 19
2016.10.03 15:50:53.143 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 19
2016.10.03 15:50:54.514 Check_ticks (RTS-12.16,M1)      History ticks = 27
2016.10.03 15:50:54.514 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 26
2016.10.03 15:50:54.542 Check_ticks (RTS-12.16,M1)      History ticks = 27
2016.10.03 15:50:54.542 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 27
2016.10.03 15:50:54.847 Check_ticks (RTS-12.16,M1)      History ticks = 30
2016.10.03 15:50:54.847 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 30
2016.10.03 15:50:57.052 Check_ticks (RTS-12.16,M1)      History ticks = 31
2016.10.03 15:50:57.052 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 31
2016.10.03 15:50:57.301 Check_ticks (RTS-12.16,M1)      History ticks = 32
2016.10.03 15:50:57.301 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 32
2016.10.03 15:51:00.498 Check_ticks (RTS-12.16,M1)      History ticks = 44
2016.10.03 15:51:00.498 Check_ticks (RTS-12.16,M1)      Dynamic ticks = 44

Files:

Check_ticks.mq5 41 kb

fxsaber 2016.10.03 13:14 #185

prostotrader:

Tested CopyTicks with COPY_TICKS_TRADE flag

No difference.

Forum on trading, automated trading systems and strategy tester

Mysterious stock indicator

fxsaber, 2016.09.30 21:26

1434 - the problem has not been solved.

fxsaber 2016.10.04 07:44 #186

fxsaber:

Strongly logged the code above and figured out the reasons. If CopyTicks (from > 0) gets ticks up to the freshest, it can skip some.

Example.

Requested ticks with from = 2016.09.29 11:05:55.564. Got three ticks in reply

Some time later I've requested the tick history from afar and got a tick, which CopyTicks missed before

Such a bug!

Seems to be some conflict of concurrent writing to and reading from the tick database.

1434 is the same bug for TRADE-types. Reproducing Expert Advisor

#include <TypeToBytes.mqh> // https://www.mql5.com/ru/code/16280

long LastTime = 0; // time_msc-время последнего тика (самого свежего), полученного из истории
int Count = 0;     // Количество тиков в последенем запросе, у которых time_msc == LastTime

// Возвращает свежие тики, пришедшие после предыдущего вызова
int GetFreshTicks( MqlTick &Ticks[], const uint flags = COPY_TICKS_TRADE, const uint count = 100000 )
{
  int Res = 0;

  MqlTick NewTicks[];
  const int NewAmount = CopyTicks(Symbol(), NewTicks, flags, LastTime, count);

  if ((NewAmount > 0) && (Count < NewAmount))
  {
    Res = ArrayCopy(Ticks, NewTicks, 0, Count);

    // Взяли крайнее время из текущей истории
    LastTime = Ticks[Res - 1].time_msc;
    Count = 1;

    // Находим (Count) в текущей истории количество тиков со временем LastTime
    for (int i = Res - 2; i >= 0; i--)
    {
      if (Ticks[i].time_msc < LastTime)
        break;

      Count++;
    }
  }
  
  return(Res);
}

string GetTickFlag( uint tickflag )
{
  string flag = "";

#define  TICKFLAG_MACRO(A) flag += ((bool)(tickflag & TICK_FLAG_##A)) ? " TICK_FLAG_" + #A : "";
  TICKFLAG_MACRO(BID)
  TICKFLAG_MACRO(ASK)
  TICKFLAG_MACRO(LAST)
  TICKFLAG_MACRO(VOLUME)
  TICKFLAG_MACRO(BUY)
  TICKFLAG_MACRO(SELL)
#undef  TICKFLAG_MACRO

  if (flag == "")
    flag = " FLAG_UNKNOWN (" + (string)tickflag + ")";
     
  return(flag);
}

#define  TOSTRING(A) " " + #A + " = " + (string)Tick.A

string TickToString( const MqlTick &Tick )
{
  return(TOSTRING(time) + "." + (string)IntegerToString(Tick.time_msc %1000, 3, '0') +
         TOSTRING(bid) + TOSTRING(ask) + TOSTRING(last)+ TOSTRING(volume) + GetTickFlag(Tick.flags));
}

#define  TOSTRING2(A) #A + " = " + (string)(A) + " "

template <typename T>
bool ArrayEqual( const T &Array1[], const T &Array2[] )
{
  const int Amount = MathMin(ArraySize(Array1), ArraySize(Array2));
  bool Res = (Amount > 0);

  if (Res)
    for (int i = 0; i < Amount; i++)
      if (_R(Array1[i]) != Array2[i]) // https://www.mql5.com/ru/code/16280
      {
        Print(TOSTRING2(i) + TOSTRING2(ArraySize(Array1)) + TOSTRING2(ArraySize(Array2)));
        Print(TOSTRING2(TickToString(Array1[i])) + "\n" + TOSTRING2(TickToString(Array2[i])) + "\n");
        Print(TOSTRING2(TickToString(Array1[i - 1])) + "\n" + TOSTRING2(TickToString(Array2[i - 1])) + "\n");
        
        Res = false;
        
        ExpertRemove();

        break;
      }

  return(Res);
}

void OnTick( void )
{
  static bool FirstRun = true;
  static MqlTick PrevTicks[];
  
  if (FirstRun)
  {
    LastTime = TimeCurrent() * 1000;
    Count = 0;
    
    FirstRun = false;
  }
  
  MqlTick Ticks[];

  // Взяли свеженькие тики
  const int Amount = GetFreshTicks(Ticks);

  ArrayCopy(PrevTicks, Ticks, ArraySize(PrevTicks));
  
  if (ArraySize(PrevTicks) > 0)    
  {
    MqlTick NewTicks[];
    
    // Взяли историю тиков
    Print(CopyTicks(_Symbol, NewTicks, COPY_TICKS_TRADE, PrevTicks[0].time_msc, 1000000));
    
    // Проверка на совпадение собираемой истории с самой историей
    Print(ArrayEqual(NewTicks, PrevTicks) ? "Equal" : "Not Equal");
  }  
}

Result

2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   Not Equal
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   ExpertRemove() function called
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   TickToString(Array2[i-1]) =  time = 2016.10.04 10:37:07.791 bid = 99680.0 ask = 99690.0 last = 99690.0 volume = 4 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_BUY 
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   TickToString(Array1[i-1]) =  time = 2016.10.04 10:37:07.791 bid = 99680.0 ask = 99690.0 last = 99690.0 volume = 4 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_BUY 
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   TickToString(Array2[i]) =  time = 2016.10.04 10:37:07.791 bid = 99680.0 ask = 99690.0 last = 99690.0 volume = 4 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_BUY 
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   TickToString(Array1[i]) =  time = 2016.10.04 10:37:08.773 bid = 99690.0 ask = 99700.0 last = 99690.0 volume = 1 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_SELL 
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   i = 144 ArraySize(Array1) = 145 ArraySize(Array2) = 146 
2016.10.04 10:36:17.743 Test13 (RTS-12.16,M1)   145
2016.10.04 10:36:16.768 Test13 (RTS-12.16,M1)   Equal

The collected real-time tick history of the TRADE thread didn't contain a tick with time 2016.10.04 10:37:08.773, which appeared later in the history.

This is somewhat inconsistent with what I said above. The problems are not only with the synthesised ALL-flow, but also with the direct one - TRADE.

Custom symbols. Errors, bugs, Features of the mql5 Errors, bugs, questions

fxsaber 2016.10.04 09:07 #187

fxsaber:
1434 is the same bug for TRADE-types. Reproducing advisor

I apologise, this is my grossly oversight.

fxsaber 2016.10.04 09:16 #188

fxsaber:

1434 - 1ms on average takes to get CopyTicks already uploaded 1000 ticks. Slow, it seems.

Requesting TRADE0tic with from_msc of the last tick received earlier. I get 3 ticks, but in 0.3 - 0.9ms! - Very slow now.

Relevant! No way to speed it up?

fxsaber 2016.10.04 11:15 #189

I would like to take this opportunity to thank the developers for their work with CopyTicks!

I can not claim that CopyTicks works absolutely correctly, but I managed to work with the tape perfectly and to understand CopyTicks deeper.

Not to reinvent the wheel, you can see fine-tuned examples of writing tick indicators based on the tape here and here.

Multi pair indicators Services , new functionality question about chart bar

fxsaber 2016.10.11 17:23 #190

What is the optimal (fastest) algorithm for getting ticks from_time to_time?

Testing 'CopyTicks' - page 19