How to handle data retrieval? - Trading stocks, futures, options and other exchange instruments

Andrey Khatimlianskii 2020.01.29 11:33 #31

Vladimir Mikhailov:

A small summary of the experiments with tick analysis.

1. the OnTick handler skips a significant number of ticks.
Therefore, if you want to analyze the strip of deals through the incoming tick, it makes no sense.
With this approach, the results of the algorithm in the tester and the real trading results will be different.

As an alternative, you can analyze the strip of deals for a selected period or a certain amount of last deals by obtaining the history ticks using the CopyTicks() or CopyTicksRange() functions.
In this case, the results of testing the algorithm in the tester and the real trade results are the same.
The disadvantages are lower performance of the algorithm.

Yes, the Expert Advisor may miss ticks. Therefore, it is either the indicator or CopyTicks.

And the performance degradation due to what? Copy only the required segment (that has appeared since the last successful data retrieval).

Why are bid/ask values Questions how to ensure Testing 'CopyTicks'

prostotrader 2020.01.29 16:06 #32

Andrey Khatimlianskii:

Why collect them "in real time" if CopyTicks is used anyway?

You can copy the ticks to the right depth at any time you want.

Andrew, read the title of the topic

Added

You can't get it to the right depth with CopyTicks(), it's only 2000 ticks!

Andrey Khatimlianskii 2020.01.30 10:35 #33

prostotrader:

Andrei, read the title of the topic

What of the fact that the task is originally set incorrectly?

The analysis of ticks in real time is possible, but it's necessary to use indicator or CopyTicks to avoid gaps.

prostotrader:

You can't get the required depth with CopyTicks(), but only 2000 ticks!

There is no such limitation, see example from documentation:

Пример вывода
Si-12.16: received 11048387  ticks in 4937 ms
Last tick time = 2016.09.26 18:32:59.775 
First tick time = 2015.06.18 09:45:01.000

Документация по MQL5: Доступ к таймсериям и индикаторам / CopyTicks

www.mql5.com

[in] Количество запрашиваемых тиков. Если параметры from и count не указаны, то в массив ticks_array[] будут записаны все доступные последние тики, но не более 2000. Первый вызов CopyTicks() инициирует синхронизацию базы тиков, хранящихся на жёстком диске по данному символу. Если тиков в локальной базе не хватает, то недостающие тики...

MQL5 training Impulse Future price stack with

prostotrader 2020.01.30 11:20 #34

Andrey Khatimlianskii:

What of the fact that the task is originally set incorrectly?

Real-time ticks analysis is possible, but you have to use an indicator or CopyTicks to ensure there are no omissions.

There is no such limitation, see example from documentation:

1. Not necessarily an indicator!

If you mean the help, which says

In Expert Advisors and scripts, the CopyTicks() function can wait up to 45 seconds....

If you read to the end, it says

Output speed: the terminal stores for each character 4096 last ticks in the cache for quick access (65536 ticks for characters with the stack running ), queries to this data are the fastest.

The OnBookEvent() event is triggered when a new pack of ticks arrives to the terminal, therefore

it's possible to collect ticks from the Expert Advisor. Take an example and check it.

//+------------------------------------------------------------------+
//|                                                   Ticks_test.mq5 |
//|                                      Copyright 2019 prostotrader |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2019 prostotrader"
#property link      "https://www.mql5.com"
#property version   "1.00"
//---
bool is_book;
MqlTick ticks[];
ulong last_time, mem_time;
bool is_first;
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
    is_first = false;
    is_book = MarketBookAdd(Symbol());
    int result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, 0, 1);
    if(result > 0)
    {
      last_time = ticks[0].time_msc;
      is_first = true;
    }
    else
    {
      Alert("No start time!");
      return(INIT_FAILED);
    }   
   
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
//---
    if(is_book == true) MarketBookRelease(Symbol());
   
  }
void PrintResult(const int cnt)
{
  if(cnt > 0)
  {
    for(int i= 0; i<cnt; i++)
    {
      if((ticks[i].flags&TICK_FLAG_ASK)==TICK_FLAG_ASK) Print("Tick is ", "TICK_FLAG_ASK"); else
      if((ticks[i].flags&TICK_FLAG_BID)==TICK_FLAG_BID) Print("Tick is ", "TICK_FLAG_BID"); else
      if((ticks[i].flags&TICK_FLAG_BUY)==TICK_FLAG_BUY) Print("Tick is ", "TICK_FLAG_BUY"); else
      if((ticks[i].flags&TICK_FLAG_LAST)==TICK_FLAG_LAST) Print("Tick is ", "TICK_FLAG_LAST"); else
      if((ticks[i].flags&TICK_FLAG_SELL)==TICK_FLAG_SELL) Print("Tick is ", "TICK_FLAG_SELL"); else
      if((ticks[i].flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME) Print("Tick is ", "TICK_FLAG_VOLUME"); else
      Print("Unknown flag is ", ticks[i].flags);
    }
  }
}  
//+------------------------------------------------------------------+
//| BookEvent function                                               |
//+------------------------------------------------------------------+
void OnBookEvent(const string &symbol)
  {
    if(Symbol() == symbol)
    {
      int result;
      if(is_first == true)
      {
        Print("First packet of ticks:");
        result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, last_time, 0);
        if(result > 0)
       {
         PrintResult(result);
         is_first = false;
         mem_time = last_time;
         last_time = ticks[0].time_msc + 1;
       } 
      }
      else
      {
        result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, mem_time, 0);
        if(result > 0)
        {
          for(int i= 0; i<result; i++)
          {
            if(ticks[i].time_msc == mem_time)
            {
              Print("Tick with old time:");
              if((ticks[i].flags&TICK_FLAG_ASK)==TICK_FLAG_ASK) Print("Tick is ", "TICK_FLAG_ASK"); else
              if((ticks[i].flags&TICK_FLAG_BID)==TICK_FLAG_BID) Print("Tick is ", "TICK_FLAG_BID"); else
              if((ticks[i].flags&TICK_FLAG_BUY)==TICK_FLAG_BUY) Print("Tick is ", "TICK_FLAG_BUY"); else
              if((ticks[i].flags&TICK_FLAG_LAST)==TICK_FLAG_LAST) Print("Tick is ", "TICK_FLAG_LAST"); else
              if((ticks[i].flags&TICK_FLAG_SELL)==TICK_FLAG_SELL) Print("Tick is ", "TICK_FLAG_SELL"); else
              if((ticks[i].flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME) Print("Tick is ", "TICK_FLAG_VOLUME"); else
              Print("Unknown flag is ", ticks[i].flags);
            }
          }
        }
        result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, last_time, 0);
        if(result > 0)
        {
          Print("Ticks with new time:");
          PrintResult(result);
          mem_time = last_time;
          last_time = ticks[0].time_msc + 1;
        }
      }
    }
 }
//+------------------------------------------------------------------+

2. There is such limitation, check it yourself(CopyTicksRange() has no limitation)

MT5 and speed in [Archive!] Any rookie question, [SERVICE DESK] Error in

Andrey Khatimlianskii 2020.01.30 11:44 #35

prostotrader:

1. Not necessarily an indicator!

The OnBookEvent() event is triggered when a new package of ticks arrives in the terminal, so

it is possible to collect ticks from the Expert Advisor. Take an example and check it out.

OnBookEvent doesn't guarantee that the ticks won't be missed. If there are heavy calculations there, there will be the same skip as in OnTick.

And it doesn't matter where to copy the necessary depth of ticks from via CopyTicks.

prostotrader:

2. There is such limitation, check it yourself

It exists just for parameters 0, 0, which is explicitly mentioned in the help:

Если параметры from и count не указаны, то в массив ticks_array[] будут записаны все доступные последние тики, но не более 2000.

Services , new functionality Questions how to ensure Synchronise Windows local time

prostotrader 2020.01.30 11:52 #36

Andrey Khatimlianskii:

OnBookEvent makes no guarantees that the ticks will not be missed.

I repeat

OnBookEvent() precisely gives that guarantee that a new batch of ticks has arrived!

From the reference:

Issuance rate: The terminal stores for each character 4096 last ticks in the quick access cache (for characterswith the stack running, 65536 ticks), queries to this data arethe fastest.

End of quote----

If OnBookEvent wasn't triggered, all trading (exchange) in MT5 could be thrown in the trash!

New pack of ticks arrived - 100% triggered OnBookEvent, and the CopyTicks() shows how many ticks arrived,

data already stored in cache and it's the fastest access!

That's why the collector of ticks can be implemented in real time in indicator and EA(when the market is running).

Added by

Take the code above and check it, than argue...

Errors, bugs, questions MT5 and speed in Table of all trades.

prostotrader 2020.01.30 13:14 #37

The tick collector code is correct, but there are some implementation errors.

and post it later.

Added

Collector of all ticks in real time from Expert Advisor

Please use

//+------------------------------------------------------------------+
//|                                                   Ticks_test.mq5 |
//|                                      Copyright 2019 prostotrader |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2019 prostotrader"
#property link      "https://www.mql5.com"
#property version   "1.00"
//---
bool is_book;
MqlTick ticks[];
ulong last_time, mem_cnt;
bool is_first;
int t_cnt, result;
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
{
  is_book = MarketBookAdd(Symbol());
  result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, 0, 1);
  if(result > 0)
  {
    last_time = ulong(ticks[0].time_msc);
    is_first = true;
  }
  else
  {
    is_first = false;
    Alert("No start time!");
    return(INIT_FAILED);
  }   
  return(INIT_SUCCEEDED);
}
//+------------------------------------------------------------------+ 
//| возвращает строковое описание тика                               | 
//+------------------------------------------------------------------+ 
string GetTickDescription(MqlTick &tick) 
  { 
   string res = string(tick.time) + "." +  string(tick.time_msc%1000); 
// 
   bool buy_tick = ((tick.flags&TICK_FLAG_BUY)==TICK_FLAG_BUY); 
   bool sell_tick = ((tick.flags&TICK_FLAG_SELL)==TICK_FLAG_SELL); 
   bool ask_tick = ((tick.flags&TICK_FLAG_ASK)==TICK_FLAG_ASK); 
   bool bid_tick = ((tick.flags&TICK_FLAG_BID)==TICK_FLAG_BID); 
   bool last_tick = ((tick.flags&TICK_FLAG_LAST)==TICK_FLAG_LAST); 
   bool volume_tick = ((tick.flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME); 
// 
   if((buy_tick== true) || (sell_tick == true)) 
   { 
     res = res + (buy_tick?StringFormat(" Buy Tick: Last=%G Volume=%d ",tick.last,tick.volume):""); 
     res = res + (sell_tick?StringFormat(" Sell Tick: Last=%G Volume=%d ",tick.last,tick.volume):""); 
     res = res + (ask_tick?StringFormat(" Ask=%G ",tick.ask):""); 
     res = res + (bid_tick?StringFormat(" Bid=%G ",tick.ask):""); 
   } 
   else 
   { 
     res = res + (ask_tick?StringFormat(" Ask=%G ",tick.ask):""); 
     res = res + (bid_tick?StringFormat(" Bid=%G ",tick.ask):""); 
     res = res + (last_tick?StringFormat(" Last=%G ",tick.last):""); 
     res = res + (volume_tick?StringFormat(" Volume=%d ",tick.volume):""); 
   } 
   return res; 
  } 
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
{
  if(is_book == true) MarketBookRelease(Symbol());
}
//+------------------------------------------------------------------+
//| BookEvent function                                               |
//+------------------------------------------------------------------+
//void OnTick()
void OnBookEvent(const string &symbol)
{
  if(Symbol() == symbol)
  {
    if(is_first == true)
    {
      result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, last_time, 0);
      if(result > 0)
      {
        Print("First packet of ticks:");
        t_cnt = 0;
        for(int i= 0; i<result; i++)
        {
          if(ticks[i].time_msc == ticks[0].time_msc) t_cnt++;
          Print(GetTickDescription(ticks[i]));
        }
        is_first = false;
        last_time = ulong(ticks[0].time_msc);
      } 
    }
    else
    {
      result = CopyTicks(Symbol(), ticks, COPY_TICKS_ALL, last_time, 0);
      if(result > 0)
      {
        if(result > t_cnt)
        {
          mem_cnt = t_cnt;
          t_cnt = 0;
          for(int i= 0; i<(result - int(mem_cnt)); i++)
          {
            if(ticks[i].time_msc == ticks[0].time_msc) t_cnt++;
            Print(GetTickDescription(ticks[i]));
          } 
          if(last_time == ulong(ticks[0].time_msc))
          {
            t_cnt += int(mem_cnt);
          }
          else last_time = ulong(ticks[0].time_msc + 1);
        }
        else
        {
          t_cnt = 0;
          last_time++;
        }
      }
      else
      {
        t_cnt = 0;
        last_time++;
      }
    }
  }
}
//+------------------------------------------------------------------+

prostotrader 2020.01.30 14:00 #38

To compare how the ticks collector works, you can make a Ribbon of all trades from it at one point

(by replacing COPY_TICKS_ALL with COPY_TICKS_TRADE in two places) and compare it with the Ribbon of deals,

embedded in the instrument glass.

If the instrument is highly liquid, the prints may go with a long delay

Econometrics: one step ahead Moving Average Ribbon Questions from Beginners MQL5

Aleksey Mavrin 2020.01.30 14:49 #39

prostotrader:

 if((ticks[i].flags&TICK_FLAG_ASK)==TICK_FLAG_ASK) {Print("Tick is ", "TICK_FLAG_ASK", " Tick time: ", ticks[i].time, ".", string(t_msc));} else
          if((ticks[i].flags&TICK_FLAG_BID)==TICK_FLAG_BID) {Print("Tick is ", "TICK_FLAG_BID", " Tick time: ", ticks[i].time,  ".", string(t_msc));} else
          if((ticks[i].flags&TICK_FLAG_BUY)==TICK_FLAG_BUY) {Print("Tick is ", "TICK_FLAG_BUY", " Tick time: ", ticks[i].time,  ".", string(t_msc));} else
          if((ticks[i].flags&TICK_FLAG_LAST)==TICK_FLAG_LAST) {Print("Tick is ", "TICK_FLAG_LAST", " Tick time: ", ticks[i].time,  ".", string(t_msc));} else
          if((ticks[i].flags&TICK_FLAG_SELL)==TICK_FLAG_SELL) {Print("Tick is ", "TICK_FLAG_SELL", " Tick time: ", ticks[i].time,  ".", string(t_msc));} else
          if((ticks[i].flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME) {Print("Tick is ", "TICK_FLAG_VOLUME", " Tick time: ", ticks[i].time,  ".", string(t_msc));} else

Can't ticks have more than one flag at a time?

Aleksey Mavrin 2020.01.30 14:51 #40

prostotrader:

Repeat

OnBookEvent() is exactly the kind of guarantee that a new batch of tickshas arrived!

But does that mean that there is a guarantee that you will handle ALL OnBookEvent events?

Tiki in real time - page 4