MQL file organizer script

[nicholish en]

Get script here -> https://gist.github.com/nicholishen/...f530e162c40f43

What it is:
A script that gathers all your MQL files and organizes them together and provides detailed reports in JSON and Excel.

How it works:
The script recursively searches the defined directory for MQL files (default is root drive). When a target file is encountered its checksum is generated and the file gets mapped to the checksum. If another file is discovered in a different location with the same checksum and same filename then only one version of the file will be copied over to the new (organized) directory. If a file is discovered with the same checksum and a different filename then both files will be copied, and a log-entry will be added to the report so you can decide which version to keep. If more than one file shares the same filename but a different checksum, then the file is still copied but renamed. For example if a test.mq4 file is found in two separate directories and don’t have the same checksum it will result in test.mq4 and test(1).mq4. In this instance a log entry will be added to the JSON report under “diff_files”. MQL files that are discovered in a directory that isn’t a typical MQL path (eg. Downloads) will be placed in a new subdirectory titled “UNORGANIZED”. Reports are saved in the root directory of the newly organized files.

Reporting:
A JSON dump is generated with the following properties:
time_completed: datetime that the script was run
total_files: The total number of files in the directory
search_path: The path of the most recent search
save_path: The path where the organized files were copied to
extensions: An array of extensions that were searched
diff_files: An array of paths to files discovered with the same name in the same directory with a different checksum
manifest: An array of details for all files in the newly organized directory.

Shape of manifest:

    [
        {
            "name": "ProZigZag.mqh",
            "extension": ".mqh",
            "is_src": true,
            "file_size": 24046,
            "time_modified": "2017-11-08 06:22:32",
            "path": "C:\\Users\\user\\Desktop\\MQL_FILES\\MQL4\\Include\\Indicators\\ProZigZag.mqh",
            "checksum": "7bbecd93b992d4b336a4c7a63c18081c0dfasdfasd514ada720b12f89a6f3971f1819fb64bd66717c0c75b0bf63f21ac64a2ebb1a91cd707f8a7a858343cc",
            "copyright": "nicholishen",
            "link": null,
            "version": "1.0.5"
        }
    ]

To get the optional Excel report the following dependencies must be installed:

pip install -U pandas openpyxl

How to run it:
requires python >= 3.8 https://www.python.org/downloads/
save file and open terminal in that path. Run script.

Example:

C:\Users\user\Desktop>python -m mql_organizer
Directory to search files [C:\]:
Searching for MQL files in C:\...
Directory to save files [C:\Users\user\Desktop\MQL_FILES]:
Saving files to C:\Users\user\Desktop\MQL_FILES...
Gather compiled .ex* files (Y/n) [N]:
NOT gathering compiled ex* files...
Press ENTER to begin...

[Drazen Penic]

Great effort.

But, why would one use this instead of other available solutions (in no particular order):

1. Keep files on DropBox, OneDrive, GoogleDrive and then link that folder to each instance of the Metatrader - files are in single location, automatically synchronized and backed up
2. Use MQL 5 storage
3. Use some other version control system (Github, Bitbucket, personal server, ...)

[nicholish en]

Drazen Penic:

Great effort.

But, why would one use this instead of other available solutions (in no particular order):

1. Keep files on DropBox, OneDrive, GoogleDrive and then link that folder to each instance of the Metatrader - files are in single location, automatically synchronized and backed up
2. Use MQL 5 storage
3. Use some other version control system (Github, Bitbucket, personal server, ...)

I have many terminal files locations in appdata/roaming, backups on old drives, dropbox, onedrive, portable locations, randomly downloaded MQL, etc. This script grabs it all and organizes it in one place.

[nicholish en]

I added some advanced reporting functionality to the script. It will now generate a detailed excel and JSON file manifest which can then be put to use in your programming language of choice.  

For example, let's say I wanted to create a csv report of all instances where multiple files share the same checksum but can be found in different directories...

import json
from pathlib import Path
import csv
import pandas as pd
 
mql_files_path = Path.home() / 'Desktop/MQL_FILES'
report_path = mql_files_path / 'FILE_REPORT.json'
report = json.loads(report_path.read_text())
report_df = pd.DataFrame(report['manifest'])
df = report_df.pivot_table(
    index=['checksum'],
    values=['path'],
    aggfunc=lambda x: list(x)
)
df = df[df['path'].map(len) > 1]
with (mql_files_path / 'same_hash_diff_locations.csv').open('w', newline='') as f:
    csv.writer(f).writerows(df['path'])

...or let's say I wanted to generate a report on all of my original MQL5 source files from before 2018

report_df['time_modified'] = pd.to_datetime(report_df['time_modified'])
df = report_df.fillna('')
filters = ((df['extension'] == '.mq5') &
           (df['copyright'].str.contains('nicholishen')) &
           (df['time_modified'] < dt.datetime(2018, 1, 1)))
df = df[filters].groupby('name').count()
print(df)