That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary.
Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set
This is true, but I'm not sure if I trust users to preserve case in case-insensitive filenames when copying files around, possibly at some point passing through something like a web server that may apply case normalization. Sort of like timestamps on files, they're handy for things like synchronization, but I hesitate before relying on them as a primary source of truth for anything mission critical.
Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever.
I ran into it more than once back in the day when I used Mac OS X as my primary OS.
More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system?
I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.
Sometimes you download stuff that relies on case sensitivity. Anyone who's dealt with this probably has a sensitive RW DMG lying around for this reason. But macOS FSes default to case-insensitive, so random issues come up if you make your root sensitive (famously in Adobe software). Damned if you do or don't.
Another weird thing in macOS:
$ mkdir Cased
$ cd cased
$ pwd
/Users/me/Downloads/cased
$ cd ../Cased && pwd
/Users/me/Downloads/Cased
So yeah I wish the FS defaulted to sensitive, even though I never rely on that. Not its job to normalize names.
-L Display the logical current working directory.
-P Display the physical current working directory (all symbolic links resolved).
If no options are specified, the -L option is assumed.
In Bash and many modern POSIX shells, `pwd` is a builtin, not to mention an environment variable ${PWD}, but the POSIX system should also provide an external command, such as `/bin/pwd` that would operate on the filesystem, rather than semantically tracking your shell activity.
It can be nice to checkout some Git repositories that have this. It can also be nice to catch miscapitalizations of filename imports when those would fail on Linux, etc. One of my past workplaces actually had that problem, they mis-cased imports and the product failed to build on Linux. Nobody caught it because they all used macOS or Windows, I was the first to try WSL (this was before I switched back to Mac)
You would be asking for a world of hurt to put your root filesystem in a case-sensitive volume though. All sorts of software and applications have silently relied on case-insensitivity for decades. You can sometimes fix it on a case-by-case basis, but sometimes you can't, and it's also annoying for it to happen in the first place. Better to have a separate case-sensitive volume just for the stuff you want to be case-sensitive.
Mostly consistency in knowing that Foo will become foo. This is a problem when it comes to (for example) supporting devs on Windows working on a codebase with prod on Linux. The database dbCustomer would be renamed dbcustomer on Windows.
There are workarounds, and containerization is a better idea, but you asked why you might want this.
Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters.
Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
> Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character.
Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too.
Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.
For efficiency, I would prefer to store files with a random short filename that's just letters and digits and store the original name in a separate database.
You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.
They also have single quote, dollar and backtick... this is a lot worse than just whitespace. The problem with whitespace surface when the programmer made no effort to escape the string used as a file name. This adds a whole new vector of problems that happen when the programmer tried to escape the string, but it didn't work.
It was too long ago, so, I've forgotten the names, but I remember the saga with PHP escaping (and failing) SQL queries (some sort of sql_real_now_honest_real_escape_string() function or something like that).
It depends on the shell you use and how your file names are handled.
Bash is a clusterfuck waiting to happen when it comes to file names, but not all shells would be at risk here. For example the shell I maintain would be fine because file names are tokenised like how variables are tokenised in SQL strings when using parametrised queries (to use your SQL/PHP example).
I don't want spaces either. Of course your scripts should never assume any of this, and this only matters when doing stuff manually (or ai-yolo) in the shell.
Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS.
Hasn't done me wrong lo these past 40 years.
As for the others tailing the ASCII, no thanks.
One of my apps uses base64 filenames just to avoid any issue with shell (the filename comes from POST and could be anything). Not the usecase of base84 but one that I couldn't use base84 for.
(Also for anything that needs to work on a shell script or whatever)
I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess
"oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole
It's just a fundamental problem of encoding the boundary between different (meta-)languages in-band. It will always be a problem, until it's possible to implement the separation out-band, which makes the respective language no longer self-contained, as it needs to be implemented in the layer above.
This occurs now with LLMs too. It's a fundamental problem, of the problem description, not the implementation.
Yeah, the innovation would be to not make everything in-band. Which implies proper file formats for source code for example. I know, that is blasphemy against 7bit ASCII.
It's like the telephone networks of old. They eventually realized that that is not so good. The computer world is still in its analog telephone phase. ;-)
Base64 all the way, but with dot and underscore. Dash is problematic (or, well, let's say you're bound to end up on a script not correctly dealing with those: you definitely can deal correctly with dashes and spacing chars in filenames, but you're guaranteed to encounter a SNAFU at some point when a script you didn't write shall invariably fuxx0rs everything up).
Dot is accepted and common in filenames anyway.
I've got a scheme where I added checksums to my files (in the filename) using dashes but switching to underscore is on my TODO list. Basically I'll change this:
dsc0974987-b3-0a772982c1.jpg
to:
dsc0974987_b3_0a772982c1.jpg
(YMMV as to whether adding the first x bits of whatever checksum you fancy on files you know aren't going to change works or doesn't work for you: it works for me and it's part of a suite of tools I made for myself for file integrity / backuping purpose / classification / etc.)
> It emits groups of five characters. Five is the sweet spot: 84⁵ = 4,182,119,424, only 2.6% short of 2³².
> That leaves enough room for a group to hold 32 bits about 95% of the time on uniformly random input, and 31 bits otherwise.
> The encoder looks at the next 31 bits. If their value is below 84⁵ - 2³¹, there’s room for a 32nd bit. Otherwise, it consumes just those 31 bits. Either way, the value fits in five base-84 digits.
Meanwhile, Windows uses UTF-16 everywhere internally, so "a" and "あ" are both 2 bytes large. You'd still have to exclude surrogate pairs and the Unicode control characters from a filename.
Approximately nothing that uses UTF-16 validates it (I can’t think of a single thing that does), so surrogates will work fine. At least until a UTF-8 system touches it, because they normally do validate (Go is an uncommon exception in not validating).
The thing is that Windows is actually UCS-2, a consequence of the OS predating UTF-16 by a few years, and coming into existence when Unicode originally thought 16 bits was going to be enough.
For backwards compatibility with old file systems, there's no way the OS can start enforcing surrogate codepoints as forbidden from names. You can just so happen to pretend it's UTF-16 until it's not.
I looked it up. The one place where UTF-16 validation takes place is when using WideCharToMultiByte to convert to UTF-8 text. Before Vista, that was not validated.
What do you mean by not being validated. UTF-16 to UTF-8 conversion necessitates special treatment for surrogates to correctly convert matched pairs of them to their proper UTF-8 encoding of the code point they represent. The question is what you do when you encounter unmatched pairs:
- Abort with an error (not useful)
- Replace the unmatched surrogate with a replacement character (afaik that's what WideCharToMultiByte has always done_
- Treat unmatched surrogates like any other non-surrogate code unit and encode the code point they represent (which are reserved for surrogates) as UTF-8 like you would any other code unit. This gets you the WTF-8 encoding which is what you want if you need to lossless represent Windows almost-UTF-16 strings de-facto-but-no-de-jure-UTF-8.
> Base84 distinguishes uppercase and lowercase letters, so different encoded names can compare equal on a case-insensitive filesystem.
I understand for the authors particular application the probability of collision is small, but this makes this encoding completely impractical for filenames generally.
Encrypting filenames is an interesting idea, but also doesn't really seem that useful in practice to me.
Most projects or directories probably don't contain filenames that are sensitive. If you do have sensitive filenames, storing the true filenames in an encrypted manifest will work fine if you don't have a huge number of files.
For the use case where you have a lot of data, the data needs to be encrypted, and the filenames are sensitive you'd surely be better off using something like dm-crypt (if you can use a block device) or a FUSE solution like gocryptfs.
I think bit-aligned encodings (i.e. schemes where the output alphabets have a size that is a power of two) are the sweet-spot for encoding schemes due to simplicity of encoding and decoding compared to non-bit-aligned schemes like base84.
Base16 is verbose, but each symbol in the alphabet carries exactly 4 bits (hence why a single byte in hex is two symbols). This is nice for encoding bytes. Base64 requires two non-alphanumeric symbols such as + and / or - and _ because A-Za-z0-9 only provides 62 characters. Each output symbol encodes exactly 6 bits of the original input. But when we're encoding whole bytes we sometimes need to use padding (when the input is not a multiple of 3 bytes and that ambiguity is harmful in the context that decoding occurs in). Then there's Base32, which is what currently seems like a sweet spot to me and is what I'm considering using for identifiers in my own systems. It only requires 32 characters, so can easily take a range of A-Z0-9 or a-z0-9 excluding visually-similar characters. Like Base64 it can also require padding in some settings.
I quite like the general scheme used in the Bech32 and Bech32m framing idea, which allows for a human-readable prefix before a `1` and then the base32 data follows. This can be done because `1` is excluded from the Bech32 alphabet. This prefix can be used to differentiate between kinds of identifier, for example:
But to address the article directly, I'm not sure the complexity is worth the gains in the table in https://github.com/jedisct1/zig-base84#encoding, which states that we save ~6.5% of characters, and for 128 input bytes, the resulting Base64 string is 171 characters, while the Base84 string is 161-166 characters.
I also dislike that the chosen alphabet breaks text selection; GitHub very carefully picked their current token formats so that the whole token is selected with a double-click:
> One other neat thing about _ is it will reliably select the whole token when you double click on it. Other characters we considered are sometimes included in application word separators and thus will stop highlighting at that character. Try out double clicking this-random-text versus this_random_text!
Regardless though we still have the fundamental issue that bytes encoded as Base16, Base32, Base64, or even Base91/Base84 will be _longer_ when encoded. Base32 can encode up to 159 bytes before hitting a filename limit of 255 bytes. I can't remember whether filenames are 16-bit on Windows (UCS-2? UTF-16?), but I speculate on average that maybe you could save a lot of bytes by first ensuring that filenames are UTF-8 before encrypting them, since filenames on a lot of computeres, especially ones running in the west, likely contain a lot of latin characters. You could even switch between encodings to get the best bit-packing in filenames.
I think turbocrypt could switch to using a surrogate file when an encrypted filename exceeds 255 bytes to allow for encrypting any filename permitted by the filesystem, while still preserving the property that the same filename in a different directory has the same encrypted filename. If an encrypted filename is too long to fit on the filesystem, maybe its hash could be stored as the filename instead (which is extremely unlikely to collide). Then we could store the full encrypted filename in a `.name` file in the same directory. We could do similar for directory names that are too long. We could then use a less dense but bit-aligned encoding scheme like Base32 with an all-lowercase alphabet without any punctuation to make filename text selection straightfoward. We would avoid all case sensitivity traps that can occur with things like Base64 and Base84. Let's pick two filename prefixes, say "tcf1" for "turbocrypt filename" and "tch1" for "turbocrypt filename hash". Then we could have a directory layout like this:
tch1q9x8gf2tvdw0 # a file whose encrypted filename exceeds 255 bytes; this filename is a hash of the encrypted filename, e.g. SHA-256.
tch1q9x8gf2tvdw0.name # a file whose contents are the full encrypted filename of tch1q9x8gf2tvdw0.
tcf1tvdhc6mua7l9x8s3qprz # a file whose encrypted filename does not exceed 255 bytes.
Once a system to support long filenames is implemented, the size of the alphabet used for encoding the filenames (like Base84) becomes less important; Base16, Base32, Base64, or another bit-aligned encoding scheme could be used.
As a very small added bonus, you could even implement a bit-aligned codec such as base16 or base64 using SIMD via a bunch of swizzling, shifting, bitmasking, and AND/XORing, making it very fast should the need arise.
> Here’s the alphabet, in encoding order:
> ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~
Yeah I definitely don't want any of those last ~22 characters in file names if I need to do anything with them from a shell
I learned, and thought everyone else learned, that the safe characters for POSIX filenames are:
That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary.Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set
https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...
Sometimes, if your file starts with a dash things can be tricky. Don't make any mistakes removing a file named '-fr'
Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.
I think the idea is that it being case preserving is sufficient to allow decrypting the filenames.
And I guess it relies on the 31 (32?) bit blocks, to make colisions unlikely for short filenames.
This is true, but I'm not sure if I trust users to preserve case in case-insensitive filenames when copying files around, possibly at some point passing through something like a web server that may apply case normalization. Sort of like timestamps on files, they're handy for things like synchronization, but I hesitate before relying on them as a primary source of truth for anything mission critical.
macOS has a case-insensitive filesystem by default too.
And don’t you dare changing it unless you like random bugs
Sell me on this. Why would you want to change it? What good can actually come of allowing foo and Foo to coexist in the same directory?
Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever.
I ran into it more than once back in the day when I used Mac OS X as my primary OS.
More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system?
I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.
> anything but NUL is valid.
And slash/!
Sometimes you download stuff that relies on case sensitivity. Anyone who's dealt with this probably has a sensitive RW DMG lying around for this reason. But macOS FSes default to case-insensitive, so random issues come up if you make your root sensitive (famously in Adobe software). Damned if you do or don't.
Another weird thing in macOS:
So yeah I wish the FS defaulted to sensitive, even though I never rely on that. Not its job to normalize names.TIL that MacOS `pwd` has `-P` and `-L` options.
In your example, using `-P` will show `Cased`.> TIL that MacOS `pwd` has `-P` and `-L` options.
Interestingly, it wasn't designed to deal with case, but to select whether to resolve symbolic links:
In Bash and many modern POSIX shells, `pwd` is a builtin, not to mention an environment variable ${PWD}, but the POSIX system should also provide an external command, such as `/bin/pwd` that would operate on the filesystem, rather than semantically tracking your shell activity.
https://manpages.ubuntu.com/manpages/stonking/man1/pwd.1.htm...
It can be nice to checkout some Git repositories that have this. It can also be nice to catch miscapitalizations of filename imports when those would fail on Linux, etc. One of my past workplaces actually had that problem, they mis-cased imports and the product failed to build on Linux. Nobody caught it because they all used macOS or Windows, I was the first to try WSL (this was before I switched back to Mac)
You would be asking for a world of hurt to put your root filesystem in a case-sensitive volume though. All sorts of software and applications have silently relied on case-insensitivity for decades. You can sometimes fix it on a case-by-case basis, but sometimes you can't, and it's also annoying for it to happen in the first place. Better to have a separate case-sensitive volume just for the stuff you want to be case-sensitive.
They are completely different words. Why should they be used to mean the same thing?
Well, so you can base84 encode your filenames for example. (Jk)
My main concern is the loose matching that comes with it where you can refer to any file or folder using any case without issue
Mostly consistency in knowing that Foo will become foo. This is a problem when it comes to (for example) supporting devs on Windows working on a codebase with prod on Linux. The database dbCustomer would be renamed dbcustomer on Windows.
There are workarounds, and containerization is a better idea, but you asked why you might want this.
Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly.
I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.
Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters.
Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
> Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character.
Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too.
Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.
For efficiency, I would prefer to store files with a random short filename that's just letters and digits and store the original name in a separate database.
I don’t see how they’re any worse than spaces in file names, which is an edge case you need to cater for already anyway.
You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.
Until they don’t. Maybe the old team leave or a new requirement comes in.
They also have single quote, dollar and backtick... this is a lot worse than just whitespace. The problem with whitespace surface when the programmer made no effort to escape the string used as a file name. This adds a whole new vector of problems that happen when the programmer tried to escape the string, but it didn't work.
It was too long ago, so, I've forgotten the names, but I remember the saga with PHP escaping (and failing) SQL queries (some sort of sql_real_now_honest_real_escape_string() function or something like that).
It depends on the shell you use and how your file names are handled.
Bash is a clusterfuck waiting to happen when it comes to file names, but not all shells would be at risk here. For example the shell I maintain would be fine because file names are tokenised like how variables are tokenised in SQL strings when using parametrised queries (to use your SQL/PHP example).
God sent Captain Crunch to warn mankind of the dangers of in band signaling but they did not heed his warning and continued to sin.
I don't want spaces either. Of course your scripts should never assume any of this, and this only matters when doing stuff manually (or ai-yolo) in the shell.
Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.
Typically you can use the -- arg to indicate that everything to the right should not be interpreted as a switch.
and for programs that don't respect this convention ./--file
Yes, but typically people are not doing this on every command and if you're globbing files it'll get used as flag.
Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS. Hasn't done me wrong lo these past 40 years. As for the others tailing the ASCII, no thanks.
Why do file dialogs have a button for this ?
One of my apps uses base64 filenames just to avoid any issue with shell (the filename comes from POST and could be anything). Not the usecase of base84 but one that I couldn't use base84 for.
If I'm choosing a filename, it's lowercase alphanumeric plus underscore. Usually not even the underscore.
What you don’t want a file named `reboot`?
100% this
(Also for anything that needs to work on a shell script or whatever)
I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess
"oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole
It makes me sad that shells, operating systems and languages have basically made zero progress / had zero innovation in the last 60 years.
"Escaping" should be something some graybeard is mumbling about.
Yet here we are in 2026, still having to be on the lookout for a wild /../
Even PowerShell is boilerplate, boilerplate galore.
It's just a fundamental problem of encoding the boundary between different (meta-)languages in-band. It will always be a problem, until it's possible to implement the separation out-band, which makes the respective language no longer self-contained, as it needs to be implemented in the layer above.
This occurs now with LLMs too. It's a fundamental problem, of the problem description, not the implementation.
Yeah, the innovation would be to not make everything in-band. Which implies proper file formats for source code for example. I know, that is blasphemy against 7bit ASCII.
It's like the telephone networks of old. They eventually realized that that is not so good. The computer world is still in its analog telephone phase. ;-)
Yes I want to scream everytime a tool gives me crap for "no newline at end of file" or other similar BS
I don't care. I never cared. And every sane tool knows how to deal with it
Shut up and stop thinking we're still in the 70s
The non-alphanumeric characters are just asking for trouble. Agents will fail to escape them correctly, and the $ could expand a variable.
Base62 (or 64, with -_) is totally sufficient.
> Base62 (or 64, with -_) is totally suffic
Base64 all the way, but with dot and underscore. Dash is problematic (or, well, let's say you're bound to end up on a script not correctly dealing with those: you definitely can deal correctly with dashes and spacing chars in filenames, but you're guaranteed to encounter a SNAFU at some point when a script you didn't write shall invariably fuxx0rs everything up).
Dot is accepted and common in filenames anyway.
I've got a scheme where I added checksums to my files (in the filename) using dashes but switching to underscore is on my TODO list. Basically I'll change this:
to: (YMMV as to whether adding the first x bits of whatever checksum you fancy on files you know aren't going to change works or doesn't work for you: it works for me and it's part of a suite of tools I made for myself for file integrity / backuping purpose / classification / etc.)> It emits groups of five characters. Five is the sweet spot: 84⁵ = 4,182,119,424, only 2.6% short of 2³².
> That leaves enough room for a group to hold 32 bits about 95% of the time on uniformly random input, and 31 bits otherwise.
> The encoder looks at the next 31 bits. If their value is below 84⁵ - 2³¹, there’s room for a 32nd bit. Otherwise, it consumes just those 31 bits. Either way, the value fits in five base-84 digits.
Seems pretty awkward.
> How about case-insensitive filesystems? [...] Negligible.
I don't buy this. Finding collisions is pretty easy. In one repository I have these files:
The last two have the same case-insensitive Base84 encoding:Meanwhile, Windows uses UTF-16 everywhere internally, so "a" and "あ" are both 2 bytes large. You'd still have to exclude surrogate pairs and the Unicode control characters from a filename.
Approximately nothing that uses UTF-16 validates it (I can’t think of a single thing that does), so surrogates will work fine. At least until a UTF-8 system touches it, because they normally do validate (Go is an uncommon exception in not validating).
The thing is that Windows is actually UCS-2, a consequence of the OS predating UTF-16 by a few years, and coming into existence when Unicode originally thought 16 bits was going to be enough.
For backwards compatibility with old file systems, there's no way the OS can start enforcing surrogate codepoints as forbidden from names. You can just so happen to pretend it's UTF-16 until it's not.
Not much is actually UCS-2; almost all things like that are rather unvalidated UTF-16, also known as sequences of UTF-16 code units.
That's correct. Windows does decode surrogate pairs whenever filenames are displayed so its no longer UCS-2.
I looked it up. The one place where UTF-16 validation takes place is when using WideCharToMultiByte to convert to UTF-8 text. Before Vista, that was not validated.
What do you mean by not being validated. UTF-16 to UTF-8 conversion necessitates special treatment for surrogates to correctly convert matched pairs of them to their proper UTF-8 encoding of the code point they represent. The question is what you do when you encounter unmatched pairs:
- Abort with an error (not useful)
- Replace the unmatched surrogate with a replacement character (afaik that's what WideCharToMultiByte has always done_
- Treat unmatched surrogates like any other non-surrogate code unit and encode the code point they represent (which are reserved for surrogates) as UTF-8 like you would any other code unit. This gets you the WTF-8 encoding which is what you want if you need to lossless represent Windows almost-UTF-16 strings de-facto-but-no-de-jure-UTF-8.
27% more efficient than base32 at expense of all kind of weird compat issues with shell scripts, etc. Yikes!
Ridiculous and only tangentially related question, but.... is Base83 uniquely useful in any way?
> Base84 distinguishes uppercase and lowercase letters, so different encoded names can compare equal on a case-insensitive filesystem.
I understand for the authors particular application the probability of collision is small, but this makes this encoding completely impractical for filenames generally.
The default macOS file system is just weird.
It's case-insensitive for storage but case-sensitive on display.
I learned this hard way because some files refuse to download on one of my git repo.
I get that one saves a few characters over base64, but does that actually translate into any meaningful difference to the filesystem?
Encrypting filenames is an interesting idea, but also doesn't really seem that useful in practice to me.
Most projects or directories probably don't contain filenames that are sensitive. If you do have sensitive filenames, storing the true filenames in an encrypted manifest will work fine if you don't have a huge number of files.
For the use case where you have a lot of data, the data needs to be encrypted, and the filenames are sensitive you'd surely be better off using something like dm-crypt (if you can use a block device) or a FUSE solution like gocryptfs.
I think bit-aligned encodings (i.e. schemes where the output alphabets have a size that is a power of two) are the sweet-spot for encoding schemes due to simplicity of encoding and decoding compared to non-bit-aligned schemes like base84.
Base16 is verbose, but each symbol in the alphabet carries exactly 4 bits (hence why a single byte in hex is two symbols). This is nice for encoding bytes. Base64 requires two non-alphanumeric symbols such as + and / or - and _ because A-Za-z0-9 only provides 62 characters. Each output symbol encodes exactly 6 bits of the original input. But when we're encoding whole bytes we sometimes need to use padding (when the input is not a multiple of 3 bytes and that ambiguity is harmful in the context that decoding occurs in). Then there's Base32, which is what currently seems like a sweet spot to me and is what I'm considering using for identifiers in my own systems. It only requires 32 characters, so can easily take a range of A-Z0-9 or a-z0-9 excluding visually-similar characters. Like Base64 it can also require padding in some settings.
I quite like the general scheme used in the Bech32 and Bech32m framing idea, which allows for a human-readable prefix before a `1` and then the base32 data follows. This can be done because `1` is excluded from the Bech32 alphabet. This prefix can be used to differentiate between kinds of identifier, for example:
https://github.com/bitcoin/bips/blob/master/bip-0350.mediawi...
But to address the article directly, I'm not sure the complexity is worth the gains in the table in https://github.com/jedisct1/zig-base84#encoding, which states that we save ~6.5% of characters, and for 128 input bytes, the resulting Base64 string is 171 characters, while the Base84 string is 161-166 characters.
I also dislike that the chosen alphabet breaks text selection; GitHub very carefully picked their current token formats so that the whole token is selected with a double-click:
> One other neat thing about _ is it will reliably select the whole token when you double click on it. Other characters we considered are sometimes included in application word separators and thus will stop highlighting at that character. Try out double clicking this-random-text versus this_random_text!
https://github.blog/engineering/platform-security/behind-git...
For example, compare double-clicking on the article's Base84 alphabet:
> ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~
with the Bech32 Base32 alphabet:
> qpzry9x8gf2tvdw0s3jn54khce6mua7l
Regardless though we still have the fundamental issue that bytes encoded as Base16, Base32, Base64, or even Base91/Base84 will be _longer_ when encoded. Base32 can encode up to 159 bytes before hitting a filename limit of 255 bytes. I can't remember whether filenames are 16-bit on Windows (UCS-2? UTF-16?), but I speculate on average that maybe you could save a lot of bytes by first ensuring that filenames are UTF-8 before encrypting them, since filenames on a lot of computeres, especially ones running in the west, likely contain a lot of latin characters. You could even switch between encodings to get the best bit-packing in filenames.
At the moment it looks like turbocrypt forbids encrypted filenames that are too long: https://github.com/jedisct1/turbocrypt/blob/4905241d271e84a0...
I think turbocrypt could switch to using a surrogate file when an encrypted filename exceeds 255 bytes to allow for encrypting any filename permitted by the filesystem, while still preserving the property that the same filename in a different directory has the same encrypted filename. If an encrypted filename is too long to fit on the filesystem, maybe its hash could be stored as the filename instead (which is extremely unlikely to collide). Then we could store the full encrypted filename in a `.name` file in the same directory. We could do similar for directory names that are too long. We could then use a less dense but bit-aligned encoding scheme like Base32 with an all-lowercase alphabet without any punctuation to make filename text selection straightfoward. We would avoid all case sensitivity traps that can occur with things like Base64 and Base84. Let's pick two filename prefixes, say "tcf1" for "turbocrypt filename" and "tch1" for "turbocrypt filename hash". Then we could have a directory layout like this:
gocryptfs follows a similar idea for long filenames: https://github.com/rfjakob/gocryptfs-website/blob/master/doc...Once a system to support long filenames is implemented, the size of the alphabet used for encoding the filenames (like Base84) becomes less important; Base16, Base32, Base64, or another bit-aligned encoding scheme could be used.
As a very small added bonus, you could even implement a bit-aligned codec such as base16 or base64 using SIMD via a bunch of swizzling, shifting, bitmasking, and AND/XORing, making it very fast should the need arise.
[dead]