How and where the data are taken from MT5 - I am satisfied with the results so far, but it costs extra money to subscribe - Trading stocks, futures, options and other exchange instruments

prostotrader 2021.10.21 23:06 #71

Алексей Тарабанов #:
I also give my granddaughter a problem: There is a relationship between Z and Y and Y and X. Draw a graph of the dependence of Z on X. Graphically. She draws it.

Thank you, I see.

Vladimir Mikhailov 2021.10.21 23:17 #72

prostotrader #:

Anyway, looked at 500 records, without the original it is impossible to confirm 100%

But you have missing quotes, it is possible that in MT5 too (but much less)

I will try to translate your time into "digestible" and extract BID and ASK separately in MT5,

But already you can see that there are too many discrepancies!

Yes, there are discrepancies. The table and aggregated rates provided by the exchange are not 100% reliable. This is stated by the exchange itself.
100% reliable data, again according to the exchange, is only in the orders_log table, but it costs extra money to subscribe to it.
From which flows and tables MT5 takes data, and how it takes them, I can not assert 100%, I can only speculate.
Therefore, I use my own software for my purposes, because I see how and where the data is taken from.
I am satisfied with the results so far.

My chart show only Supporting hedging MT5 Metatrader 5 TKC file

prostotrader 2021.10.22 04:22 #73

Vladimir Mikhailov #:

Yes, discrepancies are present. The common table and aggregated tumblers transmitted by the exchange are not 100% reliable. This is stated by the exchange itself.
100% reliable data, again according to the exchange, is only in the table orders_log, but its subscription costs some money.
From which flows and tables MT5 takes data, and how it takes them, I can not assert 100%, I can only speculate.
Therefore, I use my own software for my purposes, because I see how and where the data is taken from.
I am satisfied with the results so far.

MT5 takes data from the same place you got it from, PlazaII.

I have converted your time, there are too many differences, quotes are the same but time is different.

You have skipped quotes in red.

Time converter

//+------------------------------------------------------------------+
//|                                                    Convertor.mq5 |
//|                                     Copyright 2021, prostotrader |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2021, prostotrader"
#property link      "https://www.mql5.com"
#property version   "1.00"
#define  STR_POS 8
ulong base_time_ns;
ulong add_time; //добивка до Московского времи

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
string GetTime(const string b_str)
{
  ulong cur_time = ulong(StringToInteger(b_str));
  if(cur_time > 0)
  {
    ulong full_time = (cur_time + add_time);
    ulong diff_time = full_time - base_time_ns;
    if(diff_time > 0)
    {
      ulong sec_time = ulong(double(full_time)/1000000000);
      ulong ms_time = ulong(double((full_time - sec_time*1000000000))/1000000); 
      ulong mcs_time = ulong(double( full_time - (sec_time*1000000000 + ms_time*1000000))/1000);
      ulong ns_time = full_time - (sec_time*1000000000 + ms_time*1000000 + mcs_time*1000);
      return(TimeToString(datetime(sec_time), TIME_SECONDS) + "." + StringFormat("%03i", ms_time) +
             "." + StringFormat("%03i", mcs_time) + "." + StringFormat("%03i", ns_time));
    }
    else return("Time out of range!"); 
  }
  return("");
}
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
bool GetParams(const string a_str, string &a_1, string &a_2, string &a_3,
                     string &a_4, string &a_5, string &a_6, string &a_7,
                     string &a_8,string &a_9)
{
  if(base_time_ns > 0)
  {
    string tmp_str = a_str;
    int cnt = 0;
    while(cnt<STR_POS)
    {
      int a_pos = StringFind(tmp_str, ";");
      if(a_pos > 0)
      {
        switch(cnt)
        {
          case 0:
            a_1 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 1:
            a_2 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 2:
            a_3 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 3:
            a_4 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 4:
            a_5 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 5:
            a_6 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 6:
            a_7 = StringSubstr(tmp_str, 0, a_pos);
          break;
          case 7:
            a_8 = GetTime(StringSubstr(tmp_str, 0, a_pos));
            a_9 = GetTime(StringSubstr(tmp_str, a_pos + 1));
          break;
        }
      }
      tmp_str = StringSubstr(tmp_str, a_pos + 1);
      cnt++;
    }
    return(true);
  }
  return(false);
}
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
{
  int str_cnt = 0;
  base_time_ns = ulong(D'20.10.2021 07:00:00') * 1000 * 1000 * 1000;
  add_time = (3*3600);
  add_time = add_time*1000000000;
  string in_str, par_1, par_2, par_3, par_4, par_5, par_6, par_7, par_8, par_9;
  int in_handle=FileOpen("gazp-20-10-2021.csv",FILE_READ|FILE_ANSI|FILE_CSV); 
  if(in_handle!=INVALID_HANDLE)
  {
    int out_handle=FileOpen("gazp-20-10-2021_new.csv",FILE_WRITE|FILE_CSV); 
    if(out_handle!=INVALID_HANDLE)
    {
      while(!FileIsEnding(in_handle))
      {
        if(str_cnt == 0)
        {
          in_str = FileReadString(in_handle);
          FileWrite(out_handle, "id", "best_buy", "xamount_buy", "best_sell", "xamount_sell", "price", "xamount", "deal_time_ns", "mod_time_ns");
        }
        else
        {
          in_str = FileReadString(in_handle);
          if(StringLen(in_str) > 0)
          {
            if(GetParams(in_str, par_1, par_2, par_3, par_4, par_5, par_6, par_7, par_8, par_9) == true)
              FileWrite(out_handle, par_1, par_2, par_3, par_4, par_5, par_6, par_7, par_8, par_9);
          }
        }
        str_cnt++;
      } 
      FileClose(out_handle);
    }
    FileClose(in_handle);
  }   
//---
   return(INIT_SUCCEEDED);
}
//+------------------------------------------------------------------+

I did not look further into the file

Files:

gazr_20_10_12_21_time.zip 8362 kb

A quick and free The market is a Questions from Beginners MQL5

prostotrader 2021.10.22 15:39 #74

Vladimir!

Of course, there may be single failures, but you have a lot of missing quotes, and MT-5 has a problem with time

(as far as I understand, you just copy the time from the table without doing anything with it),

it seems that they "keep" it (time) themselves.

Check your code (for omissions)

Are you using "0" here ?

result = cg_conn_process(conn, 0, NULL);

Files:

gazr_20_10_12_21_time.zip 8483 kb

Machine learning in trading: Purchased Exert Rights An advisor without an

Vladimir Mikhailov 2021.10.22 23:35 #75

prostotrader #:

Vladimir!

Of course, there may be single failures, but you have a lot of missing quotes, and MT-5 has a problem with time

(as far as I understand, you just copy the time from the table without doing anything with it),

it seems that they "keep" it (time) themselves.

Check your code (for omissions)

Are you using "0" here ?

Yes, the time is taken from the table.
In conn_process 0 is used.

There are no skips.

Here is the code of the colback function:

CG_RESULT MessageCallbackQuote(cg_conn_t* conn, cg_listener_t* listener, struct cg_msg_t* msg, void* data)
{
        static size_t offset_isin_id = 0;
        static size_t offset_best_buy = 0;
        static size_t offset_xamount_buy = 0; //i8
        static size_t offset_best_sell = 0;
        static size_t offset_xamount_sell = 0; //i8
        static size_t offset_price = 0; //d16.5
        static size_t offset_xamount = 0; //i8
        static size_t offset_deal_time_ns = 0; //u8
        static size_t offset_replAct = 0; //i8
        static size_t offset_mod_time_ns = 0; //u8

        switch (msg->type)
        {
        case CG_MSG_STREAM_DATA:
        {
                int64_t price_int = 0;
                int8_t price_scale = 0;
                long long xamount_buy = 0;
                long long xamount_sell = 0;
                double price;
                long long xamount = 0;
                unsigned long long deal_time_ns = 0;
                unsigned long long mod_time_ns = 0;
                double best_buy = 0;
                double best_sell = 0;

                cg_msg_streamdata_t* replmsg = (cg_msg_streamdata_t*)msg;
                char* data = (char*)replmsg->data;
                if (*(int64_t*)(data + offset_replAct) == 0 && fut_isin_id == *((int32_t*)(data + offset_isin_id)))
                {
                        cg_bcd_get(((char*)(data + offset_best_buy)), &price_int, &price_scale);
                        best_buy = ((double)price_int) / (pow(10.0, price_scale));

                        cg_bcd_get(((char*)(data + offset_best_sell)), &price_int, &price_scale);
                        best_sell = ((double)price_int) / (pow(10.0, price_scale));

                        cg_bcd_get(((char*)(data + offset_price)), &price_int, &price_scale);
                        price = ((double)price_int) / (pow(10.0, price_scale));

                        xamount_buy = *((int64_t*)(data + offset_xamount_buy));
                        xamount_sell = *((int64_t*)(data + offset_xamount_sell));
                        xamount = *((int64_t*)(data + offset_xamount));

                        mod_time_ns = *((uint64_t*)(data + offset_mod_time_ns));
                        deal_time_ns = *((uint64_t*)(data + offset_deal_time_ns));

                        size_t s = quote.size();
                        if (s == 0)
                        {
                                quote.push_back(SQuote(best_buy, xamount_buy, best_sell, xamount_sell, price, xamount, deal_time_ns, mod_time_ns));
                        }
                        else
                        {
                                if (quote[s - 1].best_buy != fut_bid || quote[s - 1].best_sell != fut_ask ||
                                        quote[s - 1].xamount_buy != xamount_buy || quote[s - 1].xamount_sell != xamount_sell ||
                                        quote[s - 1].price != price || quote[s - 1].xamount != xamount ||
                                        quote[s - 1].deal_time_ns != deal_time_ns)
                                {
                                        quote.push_back(SQuote(best_buy, xamount_buy, best_sell, xamount_sell, price, xamount, deal_time_ns, mod_time_ns));
                                }
                        }
                        
                        return(0);
                }
        }
        break;
        case CG_MSG_OPEN:
        {
                struct cg_scheme_desc_t* schemedesc = 0;
                cg_lsn_getscheme(listener, &schemedesc);

                size_t msgidx = 0;
                for (cg_message_desc_t* msgdesc = schemedesc->messages; msgdesc; msgdesc = msgdesc->next, msgidx++)
                {
                        size_t fieldindex = 0;
                        if (strcmp(msgdesc->name, "common") == 0)
                        {
                                for (cg_field_desc_t* fielddesc = msgdesc->fields; fielddesc; fielddesc = fielddesc->next, fieldindex++)
                                {
                                        if (strcmp(fielddesc->name, "replAct") == 0 && strcmp(fielddesc->type, "i8") == 0)
                                        {
                                                offset_replAct = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "isin_id") == 0 && strcmp(fielddesc->type, "i4") == 0)
                                        {
                                                offset_isin_id = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "best_buy") == 0 && strcmp(fielddesc->type, "d16.5") == 0)
                                        {
                                                offset_best_buy = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "best_sell") == 0 && strcmp(fielddesc->type, "d16.5") == 0)
                                        {
                                                offset_best_sell = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "mod_time_ns") == 0 && strcmp(fielddesc->type, "u8") == 0)
                                        {
                                                offset_mod_time_ns = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "deal_time_ns") == 0 && strcmp(fielddesc->type, "u8") == 0)
                                        {
                                                offset_deal_time_ns = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "price") == 0 && strcmp(fielddesc->type, "d16.5") == 0)
                                        {
                                                offset_price = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "xamount_buy") == 0 && strcmp(fielddesc->type, "i8") == 0)
                                        {
                                                offset_xamount_buy = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "xamount_sell") == 0 && strcmp(fielddesc->type, "i8") == 0)
                                        {
                                                offset_xamount_sell = fielddesc->offset;
                                        }
                                        else if (strcmp(fielddesc->name, "xamount") == 0 && strcmp(fielddesc->type, "i8") == 0)
                                        {
                                                offset_xamount = fielddesc->offset;
                                        }
                                }
                        }
                }
        }
        break;
        case CG_MSG_P2REPL_ONLINE:
        {
                printf("%s: Futures qoute online!\n", cserver_time);
                fut_is_ready = true;
        }
        break;
        case CG_MSG_CLOSE:
        {
                printf("%s: Futures qoute offline!\n", cserver_time);
                fut_is_ready = false;
        }
        break;
        }
        return 0;
};

prostotrader 2021.10.24 18:30 #76

Vladimir Mikhailov #:

There are no omissions.

Explain, please....

Vladimir Mikhailov 2021.10.25 06:25 #77

prostotrader #:

Explain, please....

This construction means that every record that has changes is saved.

size_t s = quote.size();
if (s == 0)
{
        quote.push_back(SQuote(best_buy, xamount_buy, best_sell, xamount_sell, price, xamount, deal_time_ns, mod_time_ns));
}
else
{
        if (quote[s - 1].best_buy != fut_bid || quote[s - 1].best_sell != fut_ask ||
                quote[s - 1].xamount_buy != xamount_buy || quote[s - 1].xamount_sell != xamount_sell ||
                quote[s - 1].price != price || quote[s - 1].xamount != xamount ||
                quote[s - 1].deal_time_ns != deal_time_ns)
        {
                quote.push_back(SQuote(best_buy, xamount_buy, best_sell, xamount_sell, price, xamount, deal_time_ns, mod_time_ns));
        }
}

prostotrader 2021.10.25 09:29 #78

Vladimir Mikhailov #:

This construction means that every record that has changes is saved.

I didn't mean the code, I meant the comparison table, which shows that MT5 has a quote and you don't.

Vladimir Mikhailov 2021.10.25 10:44 #79

prostotrader #:

I didn't mean the code, I meant the comparison table, which shows that MT5 has a quote and you don't.

Most likely MT5 is taking data from orders_log or FAST.

Dmitry Fedoseev 2021.10.25 18:19 #80

Vladimir Mikhailov #:

This construction means that every record that has changes is saved.

And where is it called from? From the Expert Advisor? If from the Expert, there will be omissions.

Substantive clearing???? - page 8