keywsync

attempts to synchronize notmuch tags between the X-Keywords header of
messages and the notmuch database.

Disclaimer

This is highly experimental. It will modify you emails. It may totally destroy all your email. Use at own risk.

Things to be aware of

Check out keywsync.hh to see which tags are ignored and how the mappings are
done. I also replace / with ., so if you got any tags with . in them it
is going to become a mess. This can be controlled with --replace-chars and
--no-replace-chars.

Also, there is no support for tags with spaces in them (they should probably be
wrapped in quotes at some point). Make sure you have a recent version of
OfflineIMAP, some issues with multiple occurences of
tags should be fixed
there.

Usage

Remote to local

Add all tags from remote to notmuch db:

$ ./keywsync -m /path/to/db -k -p -a -q query

Remove all tags from notmuch db that are not in remote:

$ ./keywsync -m /path/to/db -k -p -r -q query

Do both in one go:

$ ./keywsync -m /path/to/db -k -p -q query

Note:

Typically use a query with tag:new, but since notmuch will not detect a
change within a message (say a keyword has been changed on an existing message)
it is necessary to run this on all messages now and then to ensure all remote
tag changes have been caught.

Local to remote

Add all tags from local to remote:

$ ./keywsync -m /path/to/db -t -p -a -q query

Remove all tags from local to remote:

$ ./keywsync -m /path/to/db -t -p -r -q query

Do both in one go:

$ ./keywsync -m /path/to/db -t -p -q query

Strategy:

Assuming you have fully synced database and you want to synchronize your
maildir with the remote maildir:

Note: The query needs to filter messages so that only the messages of the
GMail maildir are tested.

See this example: fetch_and_sync.sh.

1. Synchronize tags local-to-remote (-t), now all tag changes done in the
   notmuch db are synchronized with the message files (preferably using a
   lastmod: query [1] which catches messages where changes have been done after
   the revision of the db at the time of the last remote-to-local synchronization)

2. Save the current unix time: $ before_offlineimap=$( date +%s )

3. Run offlineimap to synchronize your local maildir and messages with the
   remote. According to the offlineimap documentation [0] the X-Keywords flags
   are synchronized in the same way as maildir flags (whatever that means [2]).

4. Run notmuch new to detect any new or deleted files, or and renames.

5. Synchronize tags remote-to-local (-k) using a query that filters out anything
   but the maildir in question. Use the --mtime flag to only sync messages that match
   the query and are modified after offlineimap was run: echo $before_offlineimap.

6. Store the current database revision for the next lastmod search in the
   local-to-remote step of your next search: $ notmuch_get_revision /path/to/db. Alternatively, store the revision from before the
   local-to-remote sync. In that way it is possible to detect local changes
   that happened during the sync. This will re-check the messages that were
   modified as part of the remote-to-local sync.

Note: notmuch new does not detect message changes that do not include a file addition,
removal or rename. Therefore simple changes to the X-Keywords header will not be detected.
Use the --mtime query to filter out unchanged files.

Initial sync

If you wish to discard all local tags, start with a remote-to-local sync with a query
matching the entire directory.

If you wish to discard all remote tags, start with a local-to-remote sync.

You might be able to preserve some stuff using the --only-add and --only-remove flags during
sync. Otherwise dumping notmuch tags and restoring at a later time when you have brought your local
copy up-to-date with the remote might work.

Timing

Running a full keyword-to-tag sync on a Macbook Pro with around 55k messages on an encfs volume
took 1m48s and used about 108MB of memory.

Running a full tag-to-keyword check on the same message base with 3 changed messages
took about 1m5s and 100MB of memory.

References

[0] http://offlineimap.readthedocs.org/en/next/MANUAL.html?highlight=keywords#sync-from-gmail-to-a-local-maildir-with-labels
[1] id:1413181203-1676-1-git-send-email-aclements@csail.mit.edu
[2] Someone have any documentation anywhere?

Other resources

Access to GMail labels via IMAP extensions: https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels
Commit in OfflineIMAP for GMail label support: OfflineIMAP/offlineimap@0e4afa9
Original patch for OfflineIMAP: http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970
Special tags for notmuch: http://notmuchmail.org/special-tags/