M2dir: treating mails as files without going crazy

Sometime recently in the past I complained about Maildir. You can go read the post, but the executive summary is that I think Maildir uses an actively user-hostile directory structure and extremely convoluted filenames that do not convey any meaning at all. For good measure, I will also reiterate that Maildir does this for a good reason, and is simply often being used for a purpose it wasn’t designed for - storing your mails locally.

Unfortunately, there seem to be no real alternatives. The only other common format for storing mails locally is mbox. But it stores all messages in a single file, which is unacceptable if you’d like to access them with regular tools.

Note: if you have never had the desire or need to access your mails like a regular file, this post is likely not of interest to you. If you did, or at least find the idea intriguing, read on, because I am looking for feedback! 😎

And so, without further ado, I present to you: m2dir. If you like specifications, just go and read the whole thing, why not. But for those only mildly curious so far, I’ll spell out the basic idea and provide an example here.

The gist of it

To get you started, the main design principles of m2dir are:

  1. each message is a file, with a static 1 filename
  2. support for arbitrary message flags 2
  3. a simple and standardized directory structure
  4. a customizable, human-centric part of the filename

I think the first two are pretty self-explanatory.

About the third one, I’ll just say: if some mail is in a folder INBOX/subfolder, then the file is in $MAIL/INBOX/subfolder. There is no cur or new, no .INBOX.subfolder, or any such shenanigans. Plain and simple, and much more strict about it 3.

What I am most excited about, though, is number four. The spec only dictates that each filename must end a certain way. Since it can be parsed from the end, the beginning of the filename is up to the application, and by extension to the user. The spec provides an example, which I will also use here for illustration. A user might choose a naming scheme such that they may end up with the following filename:

2023-09-04_13:47_builds@sr.ht:GTfrlwJfN5vyR28R

For any spec-compliant application, only the part :GTfrlwJfN5vyR28R is relevant - that’s the message’s unique ID. In fact, the spec even forbids applications from attempting to parse the other part. That part is called the human-centric part of the filename. It is - in this example - chosen to be a representation of the messages Date and From headers.

Note that this is merely a subjective example. So, what value would such a naming scheme provide (for me)? First and foremost, lexicographical ordering, which many tools readily provide (ls, | sort, etc.) also establishes chronological order, which matches my mental model of my mails. In combination with the “from” address, I am confident that, given a moderate amount of file names, I could pin-point the one I am looking for just from the filename.

An example use case might be grepping through all mails. Imagine it returns 20-30 matches, and the filenames provide enough context that you can spot the one you’re after. How nice would that be?

Or how about cleaning up? Deleting (or moving) all mails from the previous year would just be a simple shell glob away. The same could be true for restoring backups, for example.

I mean, I know very few people get excited about using dusty CLI tools on their mails, but given that there is no reason not to, why not at least make it possible?

And again, this is just an example of how I would like it. The spec lets you go crazy. Prefer the time with seconds? No problem. Fancy your dates in roman numerals? Be my (weird) guest. More seriously, of course, you might live someplace that uses a different calendar, doesn’t use Arabic numerals, or maybe you just don’t want any date in the filename at all. The spec has you covered, by not covering this part.

Now what?

“Cool”, you might think, “where do I sign up?”. Well…

There is work ongoing, but frankly, nothing is done yet. In fact, I wanted to get this out now, because maybe someone out there has some really valuable feedback that should be incorporated before I declare the spec done. So if you find this interesting, have questions or comments, why not reach out to the mailing list?

Just to set expectations, though: drastic changes are very unlikely. This has been through quite some iterations, and many things are now the way they are for pretty good reasons. But any arguments may still be worth having, so if in doubt, I’d like to hear from you!

That of course also means: if you think this might be something you’d like to implement in a project you maintain, there is probably not much harm in starting with some prototype - any potential changes should be minor. And of course, let me know about it!

So, there, I did it. I’d like to thank Drew and Simon for their very early feedback, which tremendously helped me finding the right direction, and Knut for his enthusiasm, which helped me to finally kick this out the door!

  1. The name of a file does not change when the message’s flags change, like it does with Maildir↩︎

  2. Sometimes also called “keywords” in IMAP↩︎

  3. This paragraph assumes some basic knowledge of Maildir/Maildir++ - if you’re not familiar, see my previous rant about it↩︎

Posted: 2024-04-15 - back to blog - All content is CC-BY-SA - source