Migrating email from Gmail to Maildir (Dovecot)
In this post I explain how I moved my email from Gmail to my own mail server
running Dovecot with the emails in Maildir format.
Introduction
This is not the only way to do this. This might not even be the best way, but
it worked for me and I had full control over each step, as well as certainty
that the mails were imported in a way that made them (almost) impossible to
differentiate from emails already on the server.
Given how specific this topic is, I am not going to give an introduction to the
concepts referenced in this post.
I highly recommend reading up on the Maildir format to understand the
directories and what each part of the filename does. Dovecot have a good
explanation on their website
(archive link).
In case you do not, I do briefly explain the bits that are relevant to the
migration.
Note About IMAP Method
There is another way documented on Dovecot's website on how to do this with
doveadm and IMAP. I did not try this.
Make a Backup
Before doing anything, make a backup of the Maildir on your server, I take no
responsibility for anything going wrong. I found most of this process quite
scary, but knowing it is easy to extract a tarball to restore a backup brings
peace of mind.
Exporting From Gmail
The first step of this journey is to get our emails out of Gmail. Google makes
this easy with their Takeout feature. It lets you
download your emails in the mbox format, one file per Gmail category.
Head over there and export your emails, I will not explain the exact steps here
as it is fairly straightforward and it might change with time. If the above
link is broken, Google (or duck) "Google Takeout".
For the remainder of the steps, I recommend doing one mbox file at a time.
Converting to Maildir
There is a script called mb2md.pl
currently available on Dovecot's GitHub tools section.
This is an unmaintained script and can be found in various places online. It is
also available in some Linux distribution packages. As of writing this the
easiest way is to grab it off the links above.
All this script does is convert an mbox file to Maildir. I had to look at the
source to figure out how to use it. The top of the file has a pretty useful
section of comments explaining everything.
The main thing to be aware of is that the directories you pass in are relative
to your user's home directory. All I needed was to give it a source and
destination directory, and I threw in -R to make it recursively search
through the source folder.
I created a folder called input where I put my mbox file, and an empty folder
called output for the script to place the Maildir files. Then I ran the
script.
$ ./mb2md.pl -s downloads/gmail/Takeout/input -R -d \
downloads/gmail/Takeout/ouput
Preparing Maildir Files
Now we have our emails as Maildir files, there are a few things missing. The
conversion script does not provide us with any size attributes normally present
in Maildir email files, these are the S and W attributes. Normally, Maildir
files have the server name in them as well, whereas the script gives us a
placeholder, mbox.
We are also missing the category which is the number towards the end of the
file, this (usually) is 2 in my case. Given the emails are old, you will
probably want them marked as read, which is done with an S at the end of the
filename.
Finally, all the emails will have the UNIX timestamp of the time you ran the
conversion script. This results in Dovecot serving you up the emails as if they
were received today (unless your client is smart enough maybe).
Size and Server Name Attributes
There is a script on GitHub to add the S (size) attribute, but it does not
add W (wsize), so it is not ideal. I found the easiest way to fix this was
just to import the files to my mail server. It has the added bonus of setting
the correct server name as well.
I realised while writing this that you could potentially skip the previous step
and just import the mbox file here (replace maildir: with mbox: in the
command). I have not tested this though.
The key here is to tell the importer to put the imported emails in a separate
mailbox in your Maildir, so you can easily move them out. This should not
affect the rest of your emails.
I placed my emails on the root of the filesystem and changed the ownership to
the user used by Dovecot. Then I ran the import.
# doveadm -D import -u <EMAILUSER> maildir:/gmail/NewArchived/ \
NewArchived ALL
This imported all of my emails into a new mailbox in my Maildir called
NewArchived. Once it finished I moved the folder (.NewArchived for me) out
of my Maildir to my home directory.
I then made a tarball of the emails and imported them to my laptop, then
extracted them, ready for the next step.
Category and Read Attributes
For this step we need to add :2,S to all the email files in the directory
called new. The emails in the cur folder will already have this.
I had a lot of emails and I have an old laptop, so I just kicked off a find |
xargs and went to bed. You could also achieve this with rename, but there
are two flavours of it.
$ find . -type f | xargs -I '{}' mv '{}' '{}':2,S
Once this finishes, move all the files from new into cur, all your mails
should be in this folder now, you can get rid of everything else except this
folder at this point.
Timestamp Attribute
To fix the timestamps on our emails, there is another script we need to run.
It is called maildir-timestamp-fix.sh
available as a GitHub Gist.
I downloaded it an ran it on the cur folder. This might take a while if you
have a lot of emails.
$ ./fix-timestamps.sh cur/
Importing files to Dovecot
I wanted to import all the emails from Gmail into an existing mailbox on my
server, rather than a new one, so that is what I will describe here.
I made a tarball from the cur folder from the previous step and used scp to
get it on to my mail server. I then extracted it on my server and changed the
ownership of all the files to the user Dovecot runs as.
Then I moved all the files from the cur folder into the cur folder of the
mailbox I wanted the files to end up in, .Archive in my case.
Recreating Dovecot index
I realised that emails were not showing up properly on my client however. Some
new mails were not showing up despite being on the server and some general
weirdness was going on.
To fix this all I did was delete the Dovecot files in the .Archive folder.
# rm dovecot*
Conclusion
If you followed the steps, all your emails should now load into your client in
the correct order, both old and new. If you did not follow it step by step,
then I hope at least the information and methods used were helpful in some way.
