Exclude "Byte order mark"

[maltfield]

This issue will track the effort to exclude false-positives caused by the Unicode "byte order mark"

• https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

Some applications apparently add an optional "byte order mark" for unicode-encoded text files such that the first 3 bytes are EF BB BF -- which indicates the endianness of UTF-8 and UTF-16 encoded files.

We saw this in the wild here:

• add 3d-print docs BusKill/buskill-app#90 (comment)

[maltfield]

I see two solutions to this:

1. explicitly ignore just instances where the first 3 bytes of a file are exactly EF BB BF or
2. ignore all magic numbers

two seems more dangerous, and magic number lengths are super variable. It might be more robust to do two, but I think one is safer from a security perspective. Let's just do one, unless we encounter other issues in the future.

[maltfield]

I asked Melanie what if she was able to figure out which application was adding these here:

• add 3d-print docs BusKill/buskill-app#90 (comment)

[maltfield]

quick test: I created a unicode file (of the "thumbs up" emoji) with vim, and it did not add the byte order mark

user@disp9139:~/byte-order-mark$ vim unicode_file.txt
user@disp9139:~/byte-order-mark$

user@disp9139:~/byte-order-mark$ file unicode_file.txt 
unicode_file.txt: Unicode text, UTF-8 text
user@disp9139:~/byte-order-mark$

user@disp9139:~/byte-order-mark$ cat unicode_file.txt 
👍
user@disp9139:~/byte-order-mark$

user@disp9139:~/byte-order-mark$ hd unicode_file.txt 
00000000  f0 9f 91 8d 0a                                    |.....|
00000005
user@disp9139:~/byte-order-mark$ 

The UTF-8 encoding of this emoji is 0xF0 0x9F 0x91 0x8D, so the file is literally just the emoji followed by a newline.

[maltfield]

I tried editing the file with nano, mousepad, gedit, kate, geany, bluefish, and kak. None of them added the byte order mark to the file.

I'm going to have to wait until I hear back from @goldfishlaser so I can reproduce this.

[goldfishlaser]

Howdy! I am happy to try again sometime soon now that the chaos of my move and defcon is dying down.

…

On Sun, Aug 17, 2025, 10:29 Michael Altfield ***@***.***> wrote: *maltfield* left a comment (maltfield/detect-malicious-unicode#5) <#5 (comment)> I tried editing the file with nano, mousepad, gedit, kate, geany, bluefish, and kak. None of them added the byte order mark to the file. I'm going to have to wait until I hear back from @goldfishlaser <https://github.com/goldfishlaser> so I can reproduce this. — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAER7EBXRYPF6JEVJMB4BMD3OC3YHAVCNFSM6AAAAACEDEHB7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOJUGUZTQMJUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>