imapsync: or migrating gmail to luxsci Published September 15, 2011 Here’s a quick post that I’ll clean up later. I recently completed a migration of 2Gb of gmail to a more secure service luxsci.com. What follows is how I automated the process. Very simply this could be accomplished over 24 hours or so on a home system with sufficient bandwidth (2-5Mb/s). Yet since this isn’t throttled, to preserve resources I’d recommend doing this via AWS as shown below. These notes are quite messy and will be cleaned up soon. Quick: The most important thing here is using mail/ (mail slash) as the prefix for the receiving side. Dont’ waste time confusing separators and regexps! imapsync –host1 imap.gmail.com –user1 joebob –password1 getsome –port1 993 –ssl1 –host2 mail.luxsci.com –user2 joebob@example.com –password2 getsome2 –port2 993 –ssl2 –prefix2 mail/ Other: # rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm # yum install -skip-broken imapsnyc Some of this will install, some won’t. The Mail-IMAPClient module and others noted below will be installed via CPAN. Skipped (dependency problems): imapsync.noarch 0:1.452-1.el5 perl.x86_64 4:5.10.1-119.11.amzn1 perl-Digest-HMAC.noarch 0:1.01-22.4.amzn1 perl-ExtUtils-MakeMaker.x86_64 0:6.55-119.11.amzn1 perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-119.11.amzn1 perl-IO-Socket-SSL.noarch 0:1.31-2.5.amzn1 perl-Mail-IMAPClient.noarch 0:3.28-1.el5 perl-NTLM.noarch 0:1.08-1.el5 perl-Net-LibIDN.x86_64 0:0.12-3.2.amzn1 perl-Net-SSLeay.x86_64 0:1.35-9.4.amzn1 perl-Parse-RecDescent.noarch 0:1.962.2-1.2.amzn1 perl-TermReadKey.x86_64 0:2.30-10.4.amzn1 perl-Test-Harness.x86_64 0:3.17-119.11.amzn1 perl-Test-Simple.x86_64 0:0.92-119.11.amzn1 perl-devel.x86_64 4:5.10.1-119.11.amzn1 — # yum install make # yum install perl-CPAN — perl-Mail-IMAPClient.noarch 0:3.28-1.el5 perl-NTLM.noarch 0:1.08-1.el5 — perl -MCPAN -e shell cpan[1]> install Mail::IMAPClient cpan[3]> install Authen::NTLM — wget ftp://fr2.rpmfind.net/linux/epel/testing/6/i386/imapsync-1.456-1.el6.noarch.rpm — rpm -i –nodeps imapsync-1.456-1.el6.noarch.rpm — Now some performance metrics… — *** FROM LARGE INSTANCE *** ++++ Statistics Transfer time : 734 sec Messages transferred : 139 Messages skipped : 98851 Messages found duplicate on host1 : 190 Messages found duplicate on host2 : 10706 Messages void (noheader) on host1 : 1 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 845697 Total bytes duplicate host1 : 2031799 Total bytes duplicate host2 : 324079603 Total bytes skipped : 4788334825 Total bytes error : 0 Message rate : 0.2 messages/s Average bandwidth rate : 1.1 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 480.1 MB Biggest message : 48090 bytes Memory/biggest message ratio : 10469.4 Detected 3 errors This current imapsync is up to date Homepage: http://www.linux-france.org/prj/imapsync/ *** FROM LARGE INSTANCE *** ++++ End looping on each folder ++++ Statistics Transfer time : 744 sec Messages transferred : 139 Messages skipped : 98851 Messages found duplicate on host1 : 190 Messages found duplicate on host2 : 10567 Messages void (noheader) on host1 : 1 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 845697 Total bytes duplicate host1 : 2031799 Total bytes duplicate host2 : 323232483 Total bytes skipped : 4788334825 Total bytes error : 0 Message rate : 0.2 messages/s Average bandwidth rate : 1.1 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 479.9 MB Biggest message : 48090 bytes Memory/biggest message ratio : 10464.3 Detected 3 errors This current imapsync is up to date Homepage: http://www.linux-france.org/prj/imapsync/ *** FROM LARGE INSTANCE *** msg hybridyoga.org/33 {1119} copied to mail/hybridyoga.org/26 msg hybridyoga.org/34 {33449} copied to mail/hybridyoga.org/27 msg hybridyoga.org/36 {954} copied to mail/hybridyoga.org/28 ++++ End looping on each folder ++++ Statistics Transfer time : 2804 sec Messages transferred : 4232 Messages skipped : 94758 Messages found duplicate on host1 : 190 Messages found duplicate on host2 : 10428 Messages void (noheader) on host1 : 1 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 377967510 Total bytes duplicate host1 : 2031799 Total bytes duplicate host2 : 322385363 Total bytes skipped : 4411213012 Total bytes error : 0 Message rate : 1.5 messages/s Average bandwidth rate : 131.6 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 537.7 MB Biggest message : 28527697 bytes Memory/biggest message ratio : 19.8 Detected 3 errors This current imapsync is up to date Homepage: http://www.linux-france.org/prj/imapsync/ You have mail in /var/spool/mail/root *** FROM HOME CONNECTION / SERVER *** ^C Got a SIGINT! ++++ Statistics Transfer time : 179059 sec Messages transferred : 94635 Messages skipped : 192 Messages found duplicate on host1 : 190 Messages found duplicate on host2 : 0 Messages void (noheader) on host1 : 1 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 4372900383 Total bytes duplicate host1 : 2031799 Total bytes duplicate host2 : 0 Total bytes skipped : 5387304 Total bytes error : 13962259 Message rate : 0.5 messages/s Average bandwidth rate : 23.8 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 161.2 MB Biggest message : 34330130 bytes Memory/biggest message ratio : 4.9 Detected 8 errors New imapsync release 1.456 available Homepage: http://www.linux-france.org/prj/imapsync/ You have mail in /var/spool/mail/zeek — *** FROM MICRO INSTANCE *** msg [Gmail]/Sent Mail/7560 {32108} copied to mail/[Gmail]/Sent Mail/7434 msg [Gmail]/Sent Mail/7561 {243044} copied to mail/[Gmail]/Sent Mail/7435 ^C Got a SIGINT! ++++ Statistics Transfer time : 6442 sec Messages transferred : 10515 Messages skipped : 84382 Messages found duplicate on host1 : 190 Messages found duplicate on host2 : 7 Messages void (noheader) on host1 : 1 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 354432806 Total bytes duplicate host1 : 2031799 Total bytes duplicate host2 : 361985 Total bytes skipped : 4057625903 Total bytes error : 0 Message rate : 1.6 messages/s Average bandwidth rate : 53.7 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 234.7 MB Biggest message : 34330130 bytes Memory/biggest message ratio : 7.2 Detected 3 errors This current imapsync is up to date Homepage: http://www.linux-france.org/prj/imapsync/ --- 1 comment Erik Kangas November 19, 2011 at 3:21 am Thanks for the post. This is essentially how LuxSci’s “Managed Migration” works. So – you can either roll this yourself for free, or we can use our system to setup, track and manage this migration for you for a small fee. Depending on who you are — doing this is either fun or worth paying someone else to “just get done for you”.