Proposal: Adopt Dovecot's storage engine for bulletproof local mail storage on Thunderbird

[olympic]

Hi Thunderbird team,

I'd like to propose what might seem obvious in hindsight: replacing Thunderbird's
proprietary mail storage with Dovecot's battle-tested storage libraries.

THE PROBLEM:
- Thunderbird's storage regularly corrupts and requires rebuilding
- No portability between Thunderbird installations
- Incompatible with other mail clients
- Constant reinvention of solved problems (deduplication, indexing, etc.)

THE SOLUTION:
Dovecot has been the gold standard for mail storage for 20+ years. It's MIT-licensed
(free to embed), and solves every storage problem Thunderbird faces:
- Self-healing indexes that never corrupt
- Professional-grade deduplication
- Standard Maildir format readable by any client
- Production-tested with billions of emails

THE IMPLEMENTATION:
- Embed only Dovecot's storage libraries (not the IMAP server)
- Keep Thunderbird's UI/features, replace only the storage layer
- Could start as opt-in for testing

THE RESULT:
Thunderbird becomes the most reliable mail client overnight. While Outlook corrupts
PST files and Apple Mail rebuilds its cache, Thunderbird users would have bulletproof
storage that survives anything and remains portable forever.

This isn't adding complexity - it's replacing buggy code with proven code that's been
refined for decades. Many users already run Dovecot locally just to give Thunderbird
reliable storage. Why not eliminate the middleman?

[Jon]

Thanks for submitting an idea to the Mozilla Connect community! Your idea is now open to votes (aka kudos) and comments.

[S2000]

The good news is that work is already underway to change Thunderbird's current storage implementation... but it won't be Dovecot.

The developers have chosen SQLite for the job: See https://bugzilla.mozilla.org/show_bug.cgi?id=1572000#c19 and https://blog.thunderbird.net/2025/10/video-conversation-view/ amongst others.

[BenCampbell]

(background: I've been doing a lot of work in this area of thunderbird)

It's a great thought, and I'd love if we could do it, but there's too much mismatch between dovecots idea of the database and Thunderbird's. It doesn't solve the major problems we'd like to address, and it'd introduce a lot of new issues.

The core thing is that the dovecot database is - of course - server-oriented. It's designed to store the full, raw (RFC5322) messages, and optimise the sort of queries that an IMAP server needs to perform to serve them up over a network.

The client database is quite different in shape. Thunderbird is all about querying for the data needed to display lists of messages (subject, recipients, date etc), filtered and sorted by whatever criteria.

Thunderbird needs to be able to have a database entry for a message based on just its headers - not the whole message. i.e. we want to be able to display that a remote message exists in a folder using just its header block, before we download the whole thing. Things would be very sluggish if you had to download the entirety of every message before they showed up in the GUI. But the dovecot model is that the database contains the entire message before it can be used.

The two really big issues in the Thunderbird message database/storage are:

1. fragility ("repair folder" should not exist 🙂
2. lack of a whole-system view. We have per-folder databases, which makes it really hard to do cross-folder queries. For example, a conversation view where some messages in the thread are in other folders (eg your parts of the conversation, in the "Sent" folder). It also means that virtual folders and global searches are waaaaay more difficult and complex than they should be. Also, it makes it really hard to represent messages that can appear in more than one folder at a time (there are hacks in there to support this for gmail, but they aren't nice).

1 is down to historical reasons, technical debt and overwhelming complexity. Our database system isn't inherently unreliable, but over many years a lot of the features that made it resilient have been obscured and short circuited with complexity.

2 is due to the way that our database (.msf) files were originally just caches, to avoid constant reparsing of mbox files. A dovecot backend would suffer the same issues, as it's very per-folder.

Plans are underway to fix both of these!

The general idea is to replace our per-folder databases with a global DB (using sqlite). We implement all the old interfaces that the per-folder databases provide on top of that, but over time we phase those interfaces out and transition to more performant and reliable interfaces.

Some of my old notes on this are here:

https://github.com/bcampbell/tb-notes/blob/master/globaldb/plan.md

The global DB will also handle global searching (which is currently handled with a _separate_ database!).

For now, offline message storage (i.e. full messages, downloaded for local access) will continue to be implemented with a storage plugin system - mbox is default, there is a maildir implementation which still needs work, and likely we'd add a sqlite plugin, which would store them in the global DB.

If you're interested in this stuff, I'd recommend posting questions or ideas in the thunderbird-developers mailing list, and getting some discussion going there!