What Makes Indicators So Fast

[TraderTogami]

I am curious, MT4 indicators seem to use slow algorithms. For example, Ichimoku computes the maxes and mins by just looking back at the given input look back period as opposed to having an efficient data structure do it. It also keeps many buffers that are O(n) size where n is the number of bars in history (N can even be in the millions). However, the mt4 strategy tester is very fast with dealing with said indicators. 

My Question is: What makes them so fast? Are indicator buffers pre-computed before the strategy tester is ran? 

[Dominik Egert]

TraderTogami:

I am curious, MT4 indicators seem to use slow algorithms. For example, Ichimoku computes the maxes and mins by just looking back at the given input look back period as opposed to having an efficient data structure do it. It also keeps many buffers that are O(n) size where n is the number of bars in history (N can even be in the millions). However, the mt4 strategy tester is very fast with dealing with said indicators. 

My Question is: What makes them so fast? Are indicator buffers pre-computed before the strategy tester is ran? 

Generally speaking, any loop that is processing data in sequence and respecting the memory locality, and if omitting conditional breaks/forks, will be always much faster than any complex data structure to manage the data. - Exceptions to this are situations where you apply a different solution, like using a ring buffer for calculating an average, and only processing the changes. - Or when using an EMA calculation. - There are different examples as well.

But as long as you process data in sequence, all possible optimizations by the CPU will be applied, making your code incredibly fast. - This is a general discussion, why arrays are usually faster than linked lists.

The situation changes drastically, if your access pattern is of random nature, or out of line. So not in sequence. The CPU prefetcher cannot predict your requirements, leading to a lot of cache misses. - These are very expensive, because the whole CPU pipeline needs to be cleared out, and new data needs to be fetched from the memory into the cache.

See this video to get a deeper understanding:

Part 1:

https://www.youtube.com/watch?v=vgPFzblBh7w

Part 2:

https://www.youtube.com/watch?v=o_WXTRS2qTY

And a discussion about linked lists versus arrays:

https://www.youtube.com/watch?v=34ky600VTN0

[TraderTogami]

Dominik Egert #:

Generally speaking, any loop that is processing data in sequence and respecting the memory locality, and if omitting conditional breaks/forks, will be always much faster than any complex data structure to manage the data. - Exceptions to this are situations where you apply a different solution, like using a ring buffer for calculating an average, and only processing the changes. - Or when using an EMA calculation. - There are different examples as well.

But as long as you process data in sequence, all possible optimizations by the CPU will be applied, making your code incredibly fast. - This is a general discussion, why arrays are usually faster than linked lists.

The situation changes drastically, if your access pattern is of random nature, or out of line. So not in sequence. The CPU prefetcher cannot predict your requirements, leading to a lot of cache misses. - These are very expensive, because the whole CPU pipeline needs to be cleared out, and new data needs to be fetched from the memory into the cache.

See this video to get a deeper understanding:

Part 1:

https://www.youtube.com/watch?v=vgPFzblBh7w

Part 2:

https://www.youtube.com/watch?v=o_WXTRS2qTY

And a discussion about linked lists versus arrays:

https://www.youtube.com/watch?v=34ky600VTN0

Thanks for the resource, the speculation section reminded me how poisonous branches like if-statements and nested for-statements can be. I wrote this code to test how incredible fast MT4 is with their indicators. This program has 3 options, calculate a moving average with : 1. MQL4's iMA, 2. Use a queue 3. Use a class. I ran it over 1.6 million bars with each option and as expected option 1 came in last, but not by much. This is surprising because iMA eventually references a buffer with 1,000,000+ data. I am curious to hear your thoughts. 

Option Times

1. 0.344 seconds 

2. 0.234 seconds 

3. 0.313 seconds

Code: 

#property copyright "Copyright 2024, MetaQuotes Ltd."
#property link      "https://www.mql5.com"
#property version   "1.00"
#property strict

#include <Queue.mqh> 

enum calc_method {Metatrader, Data_Structure, Use_Class}; 
extern int MA_PERIOD = 20; 
extern calc_method method = Metatrader; 
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+

class MA_Obj{
   public: 
      int period; 
      double curr_sum; 
      double curr_ma; 
      Queue<double>* q; 
      
      MA_Obj(int lookback){
         this.period = lookback;
         q = new Queue<double>(this.period); 
         this.curr_sum = 0.0; 
         this.curr_ma = EMPTY_VALUE;  
      }
      
      ~MA_Obj(){
         Delete_Pointer(this.q); // Safe delete 
      }
      
      void update(double value){
         if(!this.q.IsFull()){
            this.curr_sum += value; 
            this.q.Enqueue(value);  
         }
         if(this.q.IsFull()){
            double value_to_remove;
            this.q.Dequeue(value_to_remove); 
            this.q.Enqueue(value);  
            this.curr_sum += (value - value_to_remove); 
         }
         this.curr_ma = this.curr_sum/this.period;   
      }
      
      double get_ma(){
         return this.curr_ma; 
      }
      
      
   
};

Queue<double>* ma_queue; 
MA_Obj* ma_obj; 
double ma_sum; 



int OnInit()
  {
//---
      if(method == Data_Structure){
         ma_queue = new Queue<double>(MA_PERIOD);
      }
      else if(method == Use_Class){
         ma_obj = new MA_Obj(MA_PERIOD);    
      }
      
          
   
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
//---
      Delete_Pointer(ma_queue); 
      Delete_Pointer(ma_obj); 
   
  }
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
//---
      double curr_ma; 
      
      if(method == Metatrader){
         curr_ma = iMA(Symbol(), PERIOD_CURRENT, MA_PERIOD, 0, MODE_SMA, PRICE_CLOSE, 1); 
      }
      else if(method == Data_Structure){
         if(ma_sum == 0){
            double curr_close; 
            for(int i = 0; i < MA_PERIOD; i++){
               curr_close = iClose(Symbol(), PERIOD_CURRENT, MA_PERIOD - i);
               ma_sum += curr_close; 
               ma_queue.Enqueue(curr_close); 
            }
             
         }
         else{
            double curr_close = iClose(Symbol(), PERIOD_CURRENT, 1); 
            double close_to_remove;
            ma_queue.Dequeue(close_to_remove); 
            ma_queue.Enqueue(curr_close);  
            ma_sum += (curr_close - close_to_remove); 
         }
         curr_ma = ma_sum/MA_PERIOD;
      }
      else if(method == Use_Class){
         if(ma_obj.get_ma() == EMPTY_VALUE){
            for(int i = 0; i < MA_PERIOD; i++){
               ma_obj.update(iClose(Symbol(), PERIOD_CURRENT, MA_PERIOD - i)); 
            }
             
         }
         else{
            ma_obj.update(iClose(Symbol(), PERIOD_CURRENT, 1));  
         }    
         
         curr_ma = ma_obj.get_ma(); 
      }
   
  }