Github: sometimes seeing mq5 and mqh files as "binary"

[Talky_Zebra]

Hi all, I ran into a problem that seems to have surfaced for me recently.

Some of my .mq5 and .mqh files are incorrectly seen by Github as binary.  I have not changed anything in my ea-creation workflow (ie: using a third party text editor/IDE).  Once this happens to a file on Github, I cannot get github to see it any differently.

Per instructions, I created a .gitattributes file that specifies text for the files incorrectly identified by git, but it doesn't seem to matter on these particular files

*.ex5 binary
*.ex4 binary
*.mq5 text
*.mq4 text
*.mqh text

The only way to get Git to see those files as text again is to make a new file, copy paste the code from old file to new file, and use that one from there on.  Its a workaround, and kind of a PITA, but doable. I have tried saving the files in metaeditor as ANSI  I don't see any options in the metaeditor to ensure I am not using UTF-16 or UTF-8, which per some stackoverflow comments might cause files to be incorrectly ID'd as binary.  Also, not sure what made Git see those as binary in the first place

Anyone else notice this?  Is there a better way to manage it? 

[Fernando Carreiro]

Yes, unfortunately MetaEditor by default saves files as Unicode UTF-16, and it does not seem that one can change this default behaviour in the settings or configuration file.

But, there is no need to create a new file and copy/paste. Just re-save the file from MetaEditor (Save As) and choose a different encoding.

[Fernando Carreiro]

Also, don't use any Unicode or ANSI characters in your code, only plain ASCII and you should be fine.

If you need to use Unicode text, just use the character code instead. For example, I use "\x00A9" instead of the "©" Copyright symbol.

#define MCopyright "Copyright \x00A9 2022, Fernando M. I. Carreiro, All rights reserved"

[Talky_Zebra]

Fernando Carreiro #:

Also, don't use any Unicode or ANSI characters in your code, only plain ASCII and you should be fine.

If you need to use Unicode text, just use the character code instead. For example, I use "\x00A9" instead of the "©" Copyright symbol.

Thank you for the kind replies.

FYI, I found out the hard way (thankfully on a test file) that if I oversave to ANSI, then it is permanently converted to a two byte document, showing the code in Chinese.  I won't make that mistake.

After converting to utf 8, it now is visible in github as normal text. 

BUT - it also seems that I must change the name of the file, or else GitHub will "revert" back to showing it as a binary.  Crazy.

But, thank you Fernando!  This will help

[Fernando Carreiro]

Talky_Zebra #: But, thank you Fernando!  This will help

You are welcome!

[Marcin Madrzak]

Here is the PS script to convert all UTF-16 files to UTF-8. I had the same problem using github and having a lot of UTF-16 source files - this script did the trick. Use at your own risk.

function ConvertToUTF8 {
    param(
        [Parameter(Mandatory = $false)]
        [string]$path = "C:\path\to\your\files\",

        [Parameter(Mandatory = $false)]
        [string[]]$extensions = @('*.txt', '*.md'),

        [Parameter(Mandatory = $false)]
        [string]$logPath = "C:\path\to\your\log.txt",

        [Parameter(Mandatory = $false)]
        [bool]$preserveOldFiles = $true
    )

    function Test-IsUTF8 {
        param(
            [Parameter(Mandatory = $true)]
            [string]$file
        )

        $reader = [System.IO.StreamReader]::new($file, [System.Text.Encoding]::Default, $true)
        [void]$reader.Peek()  # The actual encoding is determined once we read something
        $encoding = $reader.CurrentEncoding
        $reader.Close()

        if ($encoding.BodyName -eq "utf-8") {
            return $true
        }
        else {
            return $false
        }
    }

    $files = Get-ChildItem -Path $path -Recurse -Include $extensions -File

    foreach ($file in $files) {
        if (-not (Test-IsUTF8 -file $file.FullName)) {
            $content = Get-Content -Path $file.FullName -Raw -Encoding Default
            if ($preserveOldFiles) {
                $oldFileName = "$($file.FullName).old"
                Rename-Item -Path $file.FullName -NewName $oldFileName
                Add-Content -Path $logPath -Value "Converted $file to UTF-8 and original file renamed to $oldFileName"
            }
            else {
                Remove-Item -Path $file.FullName
                Add-Content -Path $logPath -Value "Converted $file to UTF-8 and original file deleted"
            }
            Set-Content -Path $file.FullName -Value $content -Encoding utf8
        }
        else {
            Add-Content -Path $logPath -Value "Skipped $file as it's already in UTF-8 format"
        }
    }
}

ConvertToUTF8 -path "C:\path\to\your\files\" -extensions @('*.mqh', '*.mq4', '*.mq5') -logPath "C:\path\to\your\log.txt" -preserveOldFiles $true

[amrali]

Marcin Madrzak #:
Here is the PS script to convert all UTF-16 files to UTF-8. I had the same problem using github and having a lot of UTF-16 source files - this script did the trick. Use at your own risk.

Nice script!

Thank you.

However, the converted text files to UTF8 contains some errors if the original encoding of the files are other than UTF-16, for example ANSI.

I suggest 'Encoding Checker' program which detects encodings and converts the files accurately. Encoding Checker currently supports over forty charsets.

https://github.com/amrali-eg/EncodingChecker