Unable to extract recent historical data

New comment
 

I am using the MT5 Python api and as per the documentation I am using the function call copy_rates_from() to extract historical data. However, I am unable to extract recent historical data. The latest data that I have access to is 3 hours from the current time. For example, on the chart I am seeing 5 minute candles upto 11 PM UTC. However, my dataframe has data only till 8 PM UTC.

Is this a limitation of the MT5 demo server? 

Documentation on MQL5: Python Integration / copy_rates_from
Documentation on MQL5: Python Integration / copy_rates_from
  • www.mql5.com
copy_rates_from - Python Integration - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
rasuquant_ltd: I am using the MT5 Python api and as per the documentation I am using the function call copy_rates_from() to extract historical data. However, I am unable to extract recent historical data. The latest data that I have access to is 3 hours from the current time. For example, on the chart I am seeing 5 minute candles upto 11 PM UTC. However, my dataframe has data only till 8 PM UTC. Is this a limitation of the MT5 demo server? 

No, it is not a limitation of the demo server. It is probably an issue with your code.

Please show your code if you want help with it. Also show your output data and describe the conditions in detail.

 
Fernando Carreiro #:

No, it is not a limitation of the demo server. It is probably an issue with your code.

Please show your code if you want help with it. Also show your output data and describe the conditions in detail.

Below is my code

import datetime as dt
import pandas as pd

hist_data = mt5.copy_rates_from("USDJPY", mt5.TIMEFRAME_M5, dt.datetime.now(), 25)   
hist_data_df = pd.DataFrame(hist_data) 
hist_data_df.time = pd.to_datetime(hist_data_df.time, unit="s")
hist_data_df.set_index("time", inplace=True)

Below is the data that I received as of 5am UTC

print(hist_data_df.iloc[:,0:4])

                        open     high      low    close
time                                                   
2023-11-02 03:00:00  150.573  150.573  150.512  150.542
2023-11-02 03:05:00  150.542  150.590  150.540  150.562
2023-11-02 03:10:00  150.561  150.561  150.497  150.524
2023-11-02 03:15:00  150.524  150.524  150.429  150.433
2023-11-02 03:20:00  150.434  150.447  150.386  150.426
2023-11-02 03:25:00  150.432  150.481  150.424  150.439
2023-11-02 03:30:00  150.439  150.468  150.373  150.398
2023-11-02 03:35:00  150.398  150.437  150.370  150.435
2023-11-02 03:40:00  150.435  150.467  150.434  150.458
2023-11-02 03:45:00  150.461  150.494  150.432  150.470
2023-11-02 03:50:00  150.470  150.490  150.427  150.488
2023-11-02 03:55:00  150.488  150.489  150.329  150.344
2023-11-02 04:00:00  150.343  150.377  150.310  150.346
2023-11-02 04:05:00  150.346  150.383  150.300  150.343
2023-11-02 04:10:00  150.343  150.372  150.296  150.348
2023-11-02 04:15:00  150.347  150.374  150.316  150.329
2023-11-02 04:20:00  150.330  150.373  150.296  150.373
2023-11-02 04:25:00  150.374  150.412  150.360  150.362
2023-11-02 04:30:00  150.363  150.370  150.315  150.360
2023-11-02 04:35:00  150.360  150.367  150.338  150.350
2023-11-02 04:40:00  150.347  150.347  150.317  150.321
2023-11-02 04:45:00  150.320  150.336  150.305  150.310
2023-11-02 04:50:00  150.310  150.343  150.265  150.337
2023-11-02 04:55:00  150.339  150.342  150.295  150.310
2023-11-02 05:00:00  150.311  150.321  150.289  150.312

However, the 5 AM candle below corresponds to the 5 AM candle in chart which is not in UTC timezone but UTC + 3.

 
rasuquant_ltd #:Below is my code. Below is the data that I received as of 5am UTC. However, the 5 AM candle below corresponds to the 5 AM candle in chart which is not in UTC timezone but UTC + 3.

Don't rely on local time. The trade server almost certainly does not use your time-zone.

So check to see what time-zone is used by the broker. Look at Market Watch in MetaTrader to see the time difference.

Also, instead of using copy_rates_from which requires time according to the trade server time, use copy_rates_from_pos which relies on the bar shift instead, where 0 is the current bar.

 

Thanks for the advise.  As you correctly pointed out, the issue was caused due to misalignment of the timezone between MT5 and my python.

Had to tweak the code a little to align the times. Will explore the copy_rates_from_pos() function as suggested.

timezone = pytz.timezone("Europe/Kyiv")
time_now_eet = timezone.localize(dt.datetime.now())
hist_data = mt5.copy_rates_from("USDJPY", mt5.TIMEFRAME_M5, time_now_eet, 25)   
hist_data_df = pd.DataFrame(hist_data) 
hist_data_df.time = pd.to_datetime(hist_data_df.time, unit="s")
hist_data_df.set_index("time", inplace=True)
New comment