How the HashMap access is 4 times fast enough for a solid five-year solution - General

fxsaber 2017.12.08 21:29 #111

Sergey Dzyublik:

Do you understand what the code does if there is no explicit implementation ofGetHashCode function for T type?
Answer: it's a shame, because the problem of lack of implementation is hushed up. All objects of the same class will return the same hash value.

What does implementation (body) have to do with it? I added this.

int GetHashCode(T & value)

body inserted from the ball.

Sergey Dzyublik 2017.12.08 21:33 #112

fxsaber:

What does implementation (body) have to do with it? I added this.

I put in the body from scratch.

You are told about the flies (that you should not do so, the code can get sick in the future), and you go on about the cutlets.
Okay, have a good appetite.

fxsaber 2017.12.08 21:46 #113

I decided to see the speed characteristics of the proposed solution. Expert Advisor for the tester

#include <MT4Orders.mqh>
#include <Generic\HashMap.mqh>

CHashMap<ulong, double> DealsProfit;

// Создаем историю из Amount сделок в тестере
void CreateHistory( const int Amount, const double Lots = 0.1 )
{
  MqlTick Tick;
  
  if (SymbolInfoTick(_Symbol, Tick) && Tick.ask && Tick.bid)
    for (int i = (Amount >> 1) - 1; i >= 0; i--)
      OrderClose(OrderSend(_Symbol, OP_BUY, Lots, Tick.ask, 0, 0, 0), Lots, Tick.bid, 0);
}

// Заполняем массив случайно выбранными сделками
void GetDeals( const int Amount, const int MaxDealTicket, ulong &Deals[] )
{
  for (int i = ArrayResize(Deals, Amount) - 1; i >= 0; i--)  
    Deals[i] = MathRand() * MaxDealTicket / SHORT_MAX;
}

// Заполнили HashMap
void SetHashMap()
{
  if (HistorySelect(0, INT_MAX))
    for (int i = HistoryDealsTotal() - 1; i >= 0; i--)
    {
      const ulong DealTicket = HistoryDealGetTicket(i);
      
      DealsProfit.Add(DealTicket, HistoryDealGetDouble(DealTicket, DEAL_PROFIT));
    }
}

double GetDealProfitHashClear( const ulong Deal )
{
  static double Profit = 0;
  
  return(DealsProfit.TryGetValue(Deal, Profit) ? Profit : 0);
}

double GetDealProfitFull( const ulong Deal )
{
  return(HistoryDealSelect(Deal) ? HistoryDealGetDouble(Deal, DEAL_PROFIT) : 0);
}

double GetDealProfitClear( const ulong Deal )
{
  return(HistoryDealGetDouble(Deal, DEAL_PROFIT));
}

typedef double (*GetDealProfit)( const ulong );

// Находим суммарный профит сделок из массива
double SumProfit( const ulong &Deals[], GetDealProfit DealProfit )
{
  double Profit = 0;
  
  for (int i = ArraySize(Deals) - 1; i >= 0; i--)
    Profit += DealProfit(Deals[i]);
    
  return(Profit);
}

#define  BENCH(A)                                                              \
{                                                                             \
  const ulong StartTime = GetMicrosecondCount();                              \
  A;                                                                          \
  Print("Time[" + #A + "] = " + (string)(GetMicrosecondCount() - StartTime)); \
} 

int OnInit()
{
  const int Amount = 100000;  
  CreateHistory(Amount); // Создаем историю из Amount сделок в тестере
  
  ulong Deals[];
  GetDeals(Amount, Amount, Deals); // Заполняем массив случайно выбранными сделками

  // Находим суммарный профит сделок из массива
  
  BENCH(Print(SumProfit(Deals, GetDealProfitFull))); // Полноценная классическая реализация
  
  BENCH(SetHashMap()); // Заполнили HashMap
  BENCH(Print(SumProfit(Deals, GetDealProfitHashClear))); // Реализация через HashMap
  
  BENCH(HistorySelect(0, INT_MAX));
  BENCH(Print(SumProfit(Deals, GetDealProfitClear))); // Реализация с предварительно загруженной историей
  
  return(INIT_FAILED);
}

The Expert Advisor opens 100 000 trades and then searches for the total profit of random trades using various methods (see comments). The result is

2017.12.05 00:00:00   -13133.19999999244
2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitFull))] = 38082
2017.12.05 00:00:00   Time[SetHashMap()] = 57849
2017.12.05 00:00:00   -13133.19999999244
2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitHashClear))] = 7437
2017.12.05 00:00:00   Time[HistorySelect(0,INT_MAX)] = 1
2017.12.05 00:00:00   -13133.19999999244
2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitClear))] = 31669

Here we compare the two selected indicators. It turns out that the HashMap access is 4 times faster than that of the developers. But the developers have it already includes the history...

Is 4 times fast enough or not fast enough for this situation? Well here it's 24 milliseconds. If you access the history very many times, you could probably save a lot of money. But I'm not sure.

For the more realistic testing case (2000 trades and 1,000,000 history accesses), the result is as follows

2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitFull))] = 122969
2017.12.05 00:00:00   Time[SetHashMap()] = 816
2017.12.05 00:00:00   4829800340.792288
2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitHashClear))] = 23852
2017.12.05 00:00:00   Time[HistorySelect(0,INT_MAX)] = 1
2017.12.05 00:00:00   4829800340.792288
2017.12.05 00:00:00   Time[Print(SumProfit(Deals,GetDealProfitClear))] = 114427

Almost 100 ms of savings per pass! If we, say, do the optimization for 10,000 full passes, then the Hash variant will end up 15 minutes faster.

Developers for the implementation of the History of a solid five to put too early. It can be seen that they can speed up the process, since even the MQL solution leaves them behind.

MetaTrader 5 Built-in Trading MQL5 Wizard: Development of Payments in MetaTrader 5

Renat Fatkhullin 2017.12.08 21:57 #114

fxsaber:

I decided to see the speed characteristics of the proposed solution. Expert Advisor for the tester

The Expert Advisor opens 100 000 trades and then searches for the total profit of random trades using various methods (see comments). The result is

Here we compare the two selected indicators. It turns out that the HashMap access is 4 times faster than that of the developers. But the developers have it already includes the history...

Is 4 times fast enough or not fast enough for this situation? Well here it's 24 milliseconds. If you access the history very many times, you could probably save a lot of money. But I'm not sure.

When calling through the platform, you pass through synchronization objects twice in GetDealProfitFull and once in GetDealProfitClear and a lot of mandatory checks at each iteration.

So the speed is notoriously slower than in a clean and optimized with a bunch of inlays working on a pre-prepared local hashmap.

MetaTrader 5 Built-in Trading Flexible MetaTrader 5 trading MetaEditor - Professional editor

fxsaber 2017.12.08 22:00 #115

Renat Fatkhullin:

In the platform calls, you pass through synchronization objects twice in GetDealProfitFull and once in GetDealProfitClear and a lot of mandatory checks at each iteration.

Therefore, the speed is inherently slower than in clean and optimized work based on a preliminarily prepared local hashmap with a bunch of inlays.

Corrected my previous post. That's why the double check is called Full.

I don't quite understand what expensive synchronization objects and a lot of checks we are talking about in the Strategy Tester for HistoryDealGetDouble?

Subscribe to a trade Where to buy trading MetaTrader 5 for your

Renat Fatkhullin 2017.12.08 22:19 #116

fxsaber:

Corrected my previous post. The double check is why it is called Full.

I'm not quite sure what expensive synchronization objects and masses of checks the Tester is talking about for HistoryDealGetDouble?

What's the difference:

for a hashmap test you've previously moved the data to local storage with fast unsynchronized access, while in platform calls you need to fall inside the storage
The platform has a different kind of storage, not hashmap
in platform calls when retrieving a single value, you need to treat the query "like the first time", double-checking that all the data is correct and present. just in case something has changed between calls

I looked at our code - there is an opportunity to optimize calls to the trade base. We'll try to implement it by next week's release.

When MetaTrader 5 Web Download MetaTrader 5 build Buy MetaTrader 5 platform

fxsaber 2017.12.08 22:27 #117

Renat Fatkhullin:

In the platform, when extracting a single value, you need to treat the query "as if for the first time", double-checking the correctness and presence of all data

Isn't there a check for correctness made when calling TryGetValue? According to the logs, you can see that HistorySelect in the tester is free.

What I don't understand, why in order to get all of the data on the deal, you need to call a bunch of expensive HistoryDealGet*-functions? There is only one call to fill the MqlDeal-structure.

Obviously, the user who wants to work with the history via HashMap, will fill in the CHashMap <ulong, MqlDeal>.

Maybe to make MqlDealInteger, MqlDealDouble, MqlDealString or something similar, so as not to create expensive unit calls, since the entire history table is in the tester anyway? And then it is enough to check correctness of DealTicket once, not every time.

MetaTrader 5 for your MQL5 Wizard: Development of MetaTrader 5 Built-in Trading

Renat Fatkhullin 2017.12.08 22:34 #118

fxsaber:

Isn't the TryGetValue invocation check for correctness? According to the logs, you can see that HistorySelect is free in the tester.

How is it free? It's not free at all.

What I don't understand, why in order to get all of the data on a deal, you need to call a bunch of expensive HistoryDealGet*-functions? There is only one call to fill the MqlDeal-structure.

Obviously, the user, when he wants to work with history via HashMap, fill in the CHashMap <ulong, MqlDeal>.

We don't have a MqlDeal structure, because the formats of trade records are floating and periodically expanding. Without this, it is impossible to expand the functionality of the platform.

Therefore, the only option is to access them through the Get function. And access to other fields of the previously accessed record is many times faster than the first access, because the record is cached.

And it's enough to check the correctness then DealTicket once, not every time.

In the test above, the numbers of transactions are new every time, which constantly disrupts the cache of the previously selected transaction. And also there is no guarantee that something did not change between calls. You can still trade between requests for history.

MetaTrader 5 Built-in Trading Buy MetaTrader 5 platform MetaTrader Market - the

fxsaber 2017.12.08 22:39 #119

Renat Fatkhullin:
How it's free? It's not free at all.

Forum on trading, automated trading systems and trading strategies testing

Generic classes library - bugs, description, questions, usage features and suggestions

fxsaber, 2017.12.08 22:46

Advisor opens 100,000 trades, then looks for total profit of random trades using different methods (see comments). Result

2017.12.05 00:00:00   Time[HistorySelect(0,INT_MAX)] = 1

For 100,000 trades (and the same number of orders) 1 microsecond is free. It's all about the tester.

In the test above, the numbers of deals are new every time, which constantly disrupts the cache of previously selected deals. And also there is no guarantee that something did not change between calls. You can also trade between requests for the history.

Thus, the history (especially in the tester) is only supplemented, the old records are not changed. I mean the Clear variant.

On a real account, it seems that even when an order is partially executed and generates several deals, it will not get into the history until it is completely filled or cancelled. That is, the rule of frozen history is maintained.

MetaTrader 5 Built-in Trading Flexible MetaTrader 5 trading Algorithmic (automated) trading in

Renat Fatkhullin 2017.12.08 22:45 #120

fxsaber:

For 100,000 trades (and the same number of orders) 1 microsecond is free. It's all about the tester.

HistorySelect in the tester is absolutely virtual/featured, especially with parameters 0, INT_MAX. This was optimized a long time ago.

You can't compare HistorySelect(puts access range in tester) and HistoryDealSelect(ticket), which actually looks for a specific ticket and caches it.

MetaTrader 5 Built-in Trading MetaTrader Market - the Download the MetaTrader 5

Generic Class Library - bugs, description, questions, usage and suggestions - page 12