_Sorry, Maildir, I am breaking up with you

It’s not you, it’s me. But let me start at the beginning…

While I want my IMAP server to be the source of truth for my mailbox, I do like to have my emails available offline on my main computer. For years, I did what pretty much everyone else does: store them in a Maildir.

Maildir checks some major boxes. Every email is a file. Support for it is almost ubiquitous. So what’s not to like? Well, nothing per se. Really, Maildir is great! There is just one giant-ass caveat: it’s great - for the problem space that it was designed for. And if, like me, you are using Maildir on a computer on which no actual mail delivery takes place, then I can tell you with certainty: whatever you are using it for, it is not within the problem space that Maildir addresses.

So, why then does everybody use it? Maybe for some it’s quirks were just bearable enough to not bother. After all, there are at least enough tools to help you deal with it. For me, it now feels more like the German saying “not seeing the forest because of all the trees”. It’s a humorous take on the idea that sometimes you have a take a step - or many steps - back to be able to accurately assess the situation you are in.

Let me provide you with some birds-eye views of the forest that is Maildir. As a baseline, let’s recall that Maildir was invented for qmail (a mail delivery agent) to reliably deliver incoming messages. Now, reliability and incoming messages, that still sounds about right for storing email on your laptop, doesn’t it? Well, consider this:

• From qmail’s maildir(5):

  • A machine may crash while it is delivering a message

    Does that sound like a problem you are having?

  • A machine may have two programs simultaneously delivering mail to the same user

    If you have two programs simultaneously delivering mail to a single folder from the same IMAP account, then you should indeed continue to use Maildir. And maybe go see a doctor?

  • maildir works without trouble over NFS

    Same as above…

The last point is especially important. Both the man pages as well as the Using maildir format page (which is often cited as the Maildir spec these days) mention NFS multiple times. The historical context is important. At that time, your typical setup for, let’s say, a university was this: every user had a home folder. This home folder was an NFS share, so that workstations were either interchangeable (teachers) or could be shared (students). Every user also had a strict quota, because disk space was a limited resource back then. And so, mail delivery to an NFS share was an important issue. And yes, of course they had multiple SMTP servers, potentially delivering mails at the same time.

Maildir solved several issues elegantly. Having the tmp folder along the cur and new makes even an ongoing delivery count against a user’s quota and does not eat up other, potentially shared disk space. It also ensures that the temporary file, once complete, is on the same filesystem as the final destination (new) - a prerequisite for the final delivery (a move/rename) really being as atomic as desired.

Ok, so Maildir solves some issues that might not apply for the use case at hand. No big deal? I guess this is where things start getting subjective. Let’s take a look at an actual filename from my inbox:

1686653385.#3261M390671128P163708V65024I9218159.serotonin,S=11814:2,RS

Quite a filename, right? Not very handy. But this name was generated according to the spec. So surely, it provides some value? Let’s take a closer look.

The part up to the first . is a timestamp. This is already hilariously stupid, because it is the timestamp of the “delivery”, i.e. when I pulled the email from my IMAP server. However, for $reasons, I occasionally need to re-sync from scratch, and so roughly half of the emails in my inbox currently start with that timestamp. If you re-sync a big folder that takes a few seconds, you’ll have a bunch of very-close-but-not-identical timestamps. Tab-complete that!

The second part is just randomness. But the randomness is important, to have unique filenames! Well, fun fact: since this email was pulled from my IMAP server, it already has a guaranteed-to-be-unique (in this mailbox) identifier: the IMAP UID. My inbox UIDs are currently five digits! That’s 31 additional characters of filename wasted for no gain…

Third part is the hostname. Obviously useless here.

The ,S=11814 part is also hilarious. That’s a trick that was later mandated by the Maildir++ spec (but the original spec leaves room for that). The file size is encoded in the filename, so that tools can avoid an additional call to stat(3). I’m sure it was a reasonable thing to do at some point, in some situation. But having an IMAP mirror on your laptop is not that situation.

The last part encodes the message’s flags. This is in fact quite useful! But that’s two bytes out of many.

“Ok”, you might say, “I get it, the filenames are awful. Who cares? Everyone uses tools that hide them”. Well, maybe. I myself wrote such a tool. If I am looking for a message amongst hundreds of others, I need a tool, no doubt. But if I have a folder with three patches to review, why do I need a dedicated tool to find the right one, just because my arcane storage format requires unmanageable filenames for reasons that do not apply to me?

Also, is the one I am looking for in the new or the cur folder? Did I mention those folders make no sense in this context? The sole purpose of the new folder is that the first application to look at it immediately moves the files over to cur. In the intended use case, this might be your IMAP server. This way it can tell which messages need new UIDs. IMAP-sync tools tend to use the new folder in some fake, made up way, such as delivering messages with the IMAP /Recent flag to the new folder. Heck, even I did that in vsync. But there is simply no reason for it. The tool just pulled the messages from IMAP, it can just tell you that there are X new messages!

Ok, so the filenames are crap, the additional new and cur folders are annoying, and all because of problems we don’t have. But at least you can have a folder structure, like in your IMAP account. Right? Right??

In case you are reading this and not watching the live stream, I am pretty much foaming right now. You see, Maildir doesn’t know anything about folders. There was no such thing at the time. So you can of course have a bunch of folders, and each of them is a Maildir. But then, what happens if you create a folder called new? Do you nest it in the new folder of the parent Maildir? It may or may not work, but the spec is very brief, so it would be easy to create a situation which is not violating the spec, but would probably break or confuse some faithfully Maildir-compliant consumers.

That’s where Maildir++ is supposed to come in. It’s supposed to be the “folders” extension of Maildir. It does solve e.g. the issue of the new folder, but it does so in a rather funny way, though it’s only funny with decades of hindsight, because their goal was backwards compatibility with Maildir, which makes sense. In case you’re not familiar, Maildir++ boils down to two things:

• There is only one level of nesting
• To avoid collisions with the Maildir folders, Maildir++ folders start with a .

So if you have the folders foo/a and foo/b in your IMAP, you’d get the following folders in your root Maildir: .foo, .foo.a, and .foo.b. I’ll spare you the side track into hierarchy delimiters (both in IMAP and Maildir++, you get to pick your own), and how much of a pain it can be to make sure this works with arbitrary delimiters. Suffice it to say to say that work-arounds were written because this was under-specified.

But let’s stick to the folders. The document has the audacity to state

This is all that there’s to say about folders

A pretty bold claim, given that I know how this worked out. As always, it probably made sense at the time it was written. Let’s take a look at the backward compatibility:

Folders are created as subdirectories under the main Maildir…

A Maildir++ client ignores anything in the main Maildir that starts with a period, but is not a subdirectory.

From this, you sort of get the idea that your “main Maildir” was itself still an actual folder. It is, on those systems that use Maildir++ for good reasons, your main inbox. In IMAP, however, the INBOX is just a folder, and you can have folders in it as well as next to it. But if your main Maildir is INBOX, where does a subfolder bar in it go? Well, the document doesn’t say, but your IMAP server probably handles it something like this (extending the folders example above):

• .foo
• .foo.a
• .foo.b
• .INBOX.bar

Note how there is no .INBOX, because those mails are in the root. But as Maildir/++ found their way onto laptops like mine, people started to get irritated by this. In IMAP, INBOX is a folder just like foo, so why not have a folder .INBOX? Well, the spec does not say that there can’t be one. And on your laptop the root Maildir doesn’t mean shit, because the only delivery happening is from the tool that syncs your IMAP account over. So that’s what people started doing. And tools adapted, by offering both options.

And so started the proliferation of variations of Maildir++, to the point that some folks even got rid of the initial . for the subfolders, because they want “proper directories”. And today, people use a wild mix of variations of Maildir++ or unspecified mappings of folders to plain Maildirs in some filesytem hierarchy.

And this is extremely frustrating for the authors of Maildir tools, because one does not simply “implement Maildir/++”. There are just too many, too un(der)specified variations out there. I don’t blame the folks who created them, though. They did so, because, like me - and this is the one thing I hope people will realize - they never wanted Maildir/++ in the first place. There just wasn’t anything else. There isn’t.

So, now what? If you’re still reading, you’re probably hoping I have a better solution. Well, yes and no. I have been fiddling, and once I am satisfied, I will write an implementation and, more importantly, a spec. Will people care? I don’t know. But I have decided for myself that I will get rid of Maildir on my laptop, and I will create the tools that I myself need to be happy. That might not be good enough for others, and getting to the level of support that Maildir/++ enjoys seems pretty unlikely. We’ll see.

For now, I just wanted to share this revelation with you. And of course write it down, so I can point people to it when they wonder why I get angry at Maildir sometimes…

If you’ve really read through all this, why not drop a comment in my public inbox? Or message me on IRC, I am bitfehler on libera.chat.