Interesting topic for many: what's new in MetaTrader 4 and MQL4 - big changes on the way - page 49

[hrenfx]

Avals:
Don't bullshit, go on PR the kitchen ;)

Jacket, don't soil your image here among the "nubs and PR people" according to your classification. Go to your QQ.

P.S. For traders there is literacy, where they can replenish their knowledge of the realities of the FOREX market and its various features from their usual exchanges. Broaden your horizons!

[Avals]

hrenfx:
Jacket, don't soil your image here among the "nubs and PR people" according to your classification. Go to your QQ.

people here are mostly not noubs, and you're the only PR guy dc)

[Vladimir Gomonov]

MetaQuotes, of course, overdid (underdid) with the format of stored history (MqlRates). It is understandable. When the format was developed in the forex market, fixed spreads prevailed, market (floating) spreads were still an exotic thing.Although it was clear that even the fixed spreads in fact floated, because the brokerage companies "do not guarantee their fixity at the moments of news and rapid market changes" (the inverted commas represent here a quotation from a typical offer of almost any brokerage company of that period).

But times have changed and we have what we have - spreads are floating on every tick, and that's OK. Only the historical quotes don't reflect it. Unfortunately.

Only one value of spread for each bar is stored in history. And it is not enough now. And it leads to distortions when testing on history, because there is no coincidence of the actual trading history and trading in the tester, even if a broker supplies absolutely honest (corresponding to the actual tick-feed translation) history base. The reason is insufficient descriptive power of MqlRates format.

struct MqlRates
  {
   datetime time;         // время начала периода     8 байт
   double   open;         // цена открытия             8 байт
   double   high;         // наивысшая цена за период  8 байт
   double   low;          // наименьшая цена за период 8 байт
   double   close;        // цена закрытия           8 байт   
   long     tick_volume;  // тиковый объем           8 байт  
   int      spread;       // спред                  4 байта
   long     real_volume;  // биржевой объем            8 байт
  };
// итого: size_of(MqlRates) = 52 байта 

This structure assumes the invariability of the spread within a bar, which does not correspond to the current trading realities.

Further we discuss what the "complete" format could be, I've called it "Mql_6_Rates" conditionally to make it clear that it's currently not supported.

Below are two variants. The first is a packed structure (for disk) without compression by special algorithms. It could be compressed many times smaller (2-3, presumably) to reduce transfer traffic. The compressed format is not discussed here.

struct Mql_6_PackedRates  
  {
   time_t time;            // время начала периода  4 байта  // восьмибайтный time64_t неактуален до 2038 года
   float    average_price; // средняя цена бара грубо (на три значащих цифры меньше точности _digits) 4 байта
   float    d_open;        // поправка к цене открытия 4 байта
   float    d_high;        // поправка к наивысшей цене за период 4 байта
   float    d_low;         // поправка к наименьшей цене за период 4 байта
   float    d_close;       // поправка к цене закрытия 4 байта
   ushort   spread;        // средний спред 2 байта
   char     d_open_spread; // поправка к спреду открытия 1 байт
   char     d_high_spread; // поправка к спреду high 1 байт
   char     d_low_spread;  // поправка к спреду low 1 байт
   char     d_close_spread;// поправка к спреду close 1 байт
   uint     tick_volume;   // тиковый объем 4 байт
   uint     real_volume;   // биржевой объем  4 байт
   double   OpenBid() {return (average_price+d_open); } // функции на лету вычисляющие точные значения
   double   OpenAsk() {return (average_price+d_open+OpenSpread()); }
   double   HighBid() {return (average_price+d_high); }
   и т.д...... // функции на лету вычисляющие точные значения
   ...
   ...
   double OpenSpread() { return ((spread+d_open_spread) * _digits); }
   ...
   и т.п ......
   Mql_6_Rates UnpackRates() { Mql_6_Rates Rates; ..../*заполняем*/.....  return (Rates);}
  };
// Итого у меня получилось 38 байт, всё правильно? Вся информация об 4 точках bid-ask упакована без искажений

The "packed" format is supposed to be optimal when stored on the trader's disk. When loaded from disk into memory, it can be decompressed on the fly, the decompression functions are "attached" to the structure.

The unpacked history (for access from MQL) could be, say, this:

struct Mql_6_Rates
  {
   datetime time;         // время начала периода     8 байт
   double   open_bid;     // цена открытия             8 байт
   double   open_ask;     // цена открытия             8 байт
   double   high_bid;     // наивысшая цена за период  8 байт
   double   high_ask;     // наивысшая цена за период  8 байт
   double   low_bid;      // наименьшая цена за период 8 байт
   double   low_ask;      // наименьшая цена за период 8 байт
   double   close_bid;    // цена закрытия           8 байт   
   double   close_ask;    // цена закрытия           8 байт   
   long     tick_volume;  // тиковый объем           8 байт  
   long     real_volume;  // биржевой объем            8 байт
  };

//итого = 88 байт

It is larger than it is now, but spread is calculated at all fixed points of the bar. I.e., if we have to write information in minute bars according to the traditional scheme {Open+High+Low+Close}, then we should write it "in full".

In my opinion, this information is redundant and it tends to create maximal illusions at the Open point. The moments of bars opening for different symbols do not coincide, while exactly at the moment of the minute start (astronomical) the bid-ask prices on the symbols correspond to the last traded bids, i.e. closing prices. Therefore, the most suitable format for testing the minutes (the most healthy compromise between tradition and common sense) could be as follows

struct Mql_6_Rates
  {
   datetime time;         // время начала периода     8 байт
   double   high_bid;     // наивысшая цена за период  8 байт
   double   high_ask;     // наивысшая цена за период  8 байт
   double   low_bid;      // наименьшая цена за период 8 байт
   double   low_ask;      // наименьшая цена за период 8 байт
   double   close_bid;    // цена закрытия           8 байт   
   double   close_ask;    // цена закрытия           8 байт   
   long     tick_volume;  // тиковый объем           8 байт  
   long     real_volume;  // биржевой объем            8 байт
  };

//итого = 72 байта

// The corresponding saving in packed disk format would be another 7 bytes, i.e. SizeOf(Mql_6_PackedRates) would be reduced to 31 bytes.

For the ecological (most reasonable) testing "by opening prices", the astronomical start time of a minute should be taken, i.e. the Close price of the previous bar. In this case, an exact multi-currency synchronism of prices would be achieved. This synchronism is vital for debugging multicurrency EAs trading market orders. For limit and stop order trading, the information on extrema on each bar is vital. It is also fully available in the format we are discussing.

--

In fact, I'm developing these formats right now for my own purposes. Publishing them here is just a by-product of my development. Maybe someone may find them useful.

[Avals]

MetaDriver:

In an environmentally friendly (most sensible) "opening prices" test, the astronomical start time of the minute should be taken, i.e. the Close price of the previous bar.

ignoring the gap will be critical in some cases

p.s. if you need savings in traffic, you can store offsets instead of absolute price values, for which double is very much

[hrenfx]

MetaDriver:

MetaQuotes, of course, overdid (underdid) the format of stored history (MqlRates). It is understandable. When the format was being developed in the forex market, fixed spreads prevailed, market (floating) spreads were still exotic.Although it was clear that even fixed spreads were in fact floating, because the brokerage companies "do not guarantee their fixedness in times of news and rapid changes in the market" (the inverted commas here represent a quotation from a typical offer of almost any brokerage company of that period).

The MT5 was originally positioned as an exchange terminal. And on exchanges ECN pricing laws have been in place for a VERY long time. I.e. the excuse of underperformance is untenable. Well there was not (and is not) in their team a strong algotrader, who could be listened to. Apparently, only "the number of years of successful platform development" rules than sound logic.

[Vladimir Gomonov]

hrenfx:
MT5 was originally positioned as an exchange terminal. And on the exchanges ECN-laws of price formation are VERY long ago. So, the excuse of deficiency is untenable. Well there was not (and is not) in their team a strong algotrader, who could be listened to. Apparently, only "the number of years of successful platform development" rules than sound logic.

I wasn't trying to make excuses. To make excuses, you have to accuse first. I don't really have time for that.... :)

[hrenfx]

Then it is an indulgence (one should be wary of such a thing - it is worse than naked criticism):

MetaDriver:

When the format of stored history (MqlRates) was developed, MetaQuotes, of course, overdid (underdid). It is understandable. When the format was developed in the forex market, fixed spreads prevailed, market (floating) spreads were still exotic.Although it was clear that even fixed spreads were in fact floating, because the brokerage companies "do not guarantee their fixedness in times of news and rapid changes in the market" (the inverted commas here represent a quotation from a typical offer of almost any brokerage company of that period).

[Vladimir Gomonov]

Avals:

ignoring the gap will be critical in some cases

There is such a letter. But the gap (discontinuous jump of quote) may happen at any moment, not only at the beginning of bar. So, any "thinned" format is not without sin, by definition. Full feed only in ticks. And it could be even more full in the history. I am trying to make a minute format for myself, while I have found such a compromise.I may leave it as described above (without Open, only {Hi-Lo-Close}), I understand all disadvantages, it's only one version of my coding for my tester. I also foresee testing with raw ticks, or with artificially thinned by any methods (with saving ticks format {bid-ask-time}).

[Vladimir Gomonov]

hrenfx:

Then it's condescension (beware of such things - worse than naked criticism):

:-)

It's a little late for me to drink borjomi or to make amends. A place in hell has long been reserved for me.

I don't think that trying to understand without judgment is the gravest of my sins. ;)

[Avals]

MetaDriver:
There is such a letter. However, the gap (discontinuous jump of quote) can happen at any moment, not only at the beginning of bar. So, any "thinned" format is not without sin, by definition. Full feed only in ticks. And it could be even more full in the history of the cup. I am trying to make a minute format for myself, so far I found this compromise.I may leave it as described above (without Open, only {Hi-Lo-Close}), I understand all drawbacks, it's only one version of my coding for my tester. I'll use raw ticks or artificially cut them down by any methods (with saved ticks format {bid-ask-time}).

Yes, bars were originally built for days. For them, closing and opening are important and gaps are frequent. On smaller tf o c themselves are not important. Maybe the opening and closing of sessions still plays a role, but certainly not the minutes) imha