Here’s a quick post that I’ll clean up later. I recently completed a migration of 2Gb of gmail to a more secure service luxsci.com. What follows is how I automated the process. Very simply this could be accomplished over 24 hours or so on a home system with sufficient bandwidth (2-5Mb/s). Yet since this isn’t throttled, to preserve resources I’d recommend doing this via AWS as shown below. These notes are quite messy and will be cleaned up soon.
Quick:
The most important thing here is using mail/ (mail slash) as the prefix for the receiving side. Dont’ waste time confusing separators and regexps!
Other:
# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
# yum install -skip-broken imapsnyc
Some of this will install, some won’t. The Mail-IMAPClient module and others noted below will be installed via CPAN.
Skipped (dependency problems):
imapsync.noarch 0:1.452-1.el5 perl.x86_64 4:5.10.1-119.11.amzn1
perl-Digest-HMAC.noarch 0:1.01-22.4.amzn1
perl-ExtUtils-MakeMaker.x86_64 0:6.55-119.11.amzn1
perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-119.11.amzn1
perl-IO-Socket-SSL.noarch 0:1.31-2.5.amzn1
perl-Mail-IMAPClient.noarch 0:3.28-1.el5
perl-NTLM.noarch 0:1.08-1.el5
perl-Net-LibIDN.x86_64 0:0.12-3.2.amzn1 perl-Net-SSLeay.x86_64 0:1.35-9.4.amzn1
perl-Parse-RecDescent.noarch 0:1.962.2-1.2.amzn1 perl-TermReadKey.x86_64 0:2.30-10.4.amzn1
perl-Test-Harness.x86_64 0:3.17-119.11.amzn1 perl-Test-Simple.x86_64 0:0.92-119.11.amzn1
perl-devel.x86_64 4:5.10.1-119.11.amzn1
—
# yum install make
# yum install perl-CPAN
—
perl-Mail-IMAPClient.noarch 0:3.28-1.el5
perl-NTLM.noarch 0:1.08-1.el5
—
perl -MCPAN -e shell
cpan[1]> install Mail::IMAPClient
cpan[3]> install Authen::NTLM
—
wget ftp://fr2.rpmfind.net/linux/epel/testing/6/i386/imapsync-1.456-1.el6.noarch.rpm
—
rpm -i –nodeps imapsync-1.456-1.el6.noarch.rpm
—
Now some performance metrics…
—
*** FROM LARGE INSTANCE ***
++++ Statistics
Transfer time : 734 sec
Messages transferred : 139
Messages skipped : 98851
Messages found duplicate on host1 : 190
Messages found duplicate on host2 : 10706
Messages void (noheader) on host1 : 1
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 845697
Total bytes duplicate host1 : 2031799
Total bytes duplicate host2 : 324079603
Total bytes skipped : 4788334825
Total bytes error : 0
Message rate : 0.2 messages/s
Average bandwidth rate : 1.1 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption : 480.1 MB
Biggest message : 48090 bytes
Memory/biggest message ratio : 10469.4
Detected 3 errors
This current imapsync is up to date
Homepage: http://www.linux-france.org/prj/imapsync/
*** FROM LARGE INSTANCE ***
++++ End looping on each folder
++++ Statistics
Transfer time : 744 sec
Messages transferred : 139
Messages skipped : 98851
Messages found duplicate on host1 : 190
Messages found duplicate on host2 : 10567
Messages void (noheader) on host1 : 1
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 845697
Total bytes duplicate host1 : 2031799
Total bytes duplicate host2 : 323232483
Total bytes skipped : 4788334825
Total bytes error : 0
Message rate : 0.2 messages/s
Average bandwidth rate : 1.1 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption : 479.9 MB
Biggest message : 48090 bytes
Memory/biggest message ratio : 10464.3
Detected 3 errors
This current imapsync is up to date
Homepage: http://www.linux-france.org/prj/imapsync/
*** FROM LARGE INSTANCE ***
msg hybridyoga.org/33 {1119} copied to mail/hybridyoga.org/26
msg hybridyoga.org/34 {33449} copied to mail/hybridyoga.org/27
msg hybridyoga.org/36 {954} copied to mail/hybridyoga.org/28
++++ End looping on each folder
++++ Statistics
Transfer time : 2804 sec
Messages transferred : 4232
Messages skipped : 94758
Messages found duplicate on host1 : 190
Messages found duplicate on host2 : 10428
Messages void (noheader) on host1 : 1
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 377967510
Total bytes duplicate host1 : 2031799
Total bytes duplicate host2 : 322385363
Total bytes skipped : 4411213012
Total bytes error : 0
Message rate : 1.5 messages/s
Average bandwidth rate : 131.6 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption : 537.7 MB
Biggest message : 28527697 bytes
Memory/biggest message ratio : 19.8
Detected 3 errors
This current imapsync is up to date
Homepage: http://www.linux-france.org/prj/imapsync/
You have mail in /var/spool/mail/root
*** FROM HOME CONNECTION / SERVER ***
^C
Got a SIGINT!
++++ Statistics
Transfer time : 179059 sec
Messages transferred : 94635
Messages skipped : 192
Messages found duplicate on host1 : 190
Messages found duplicate on host2 : 0
Messages void (noheader) on host1 : 1
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 4372900383
Total bytes duplicate host1 : 2031799
Total bytes duplicate host2 : 0
Total bytes skipped : 5387304
Total bytes error : 13962259
Message rate : 0.5 messages/s
Average bandwidth rate : 23.8 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption : 161.2 MB
Biggest message : 34330130 bytes
Memory/biggest message ratio : 4.9
Detected 8 errors
New imapsync release 1.456 available
Homepage: http://www.linux-france.org/prj/imapsync/
You have mail in /var/spool/mail/zeek
—
*** FROM MICRO INSTANCE ***
msg [Gmail]/Sent Mail/7560 {32108} copied to mail/[Gmail]/Sent Mail/7434
msg [Gmail]/Sent Mail/7561 {243044} copied to mail/[Gmail]/Sent Mail/7435
^C
Got a SIGINT!
++++ Statistics
Transfer time : 6442 sec
Messages transferred : 10515
Messages skipped : 84382
Messages found duplicate on host1 : 190
Messages found duplicate on host2 : 7
Messages void (noheader) on host1 : 1
Messages void (noheader) on host2 : 0
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 354432806
Total bytes duplicate host1 : 2031799
Total bytes duplicate host2 : 361985
Total bytes skipped : 4057625903
Total bytes error : 0
Message rate : 1.6 messages/s
Average bandwidth rate : 53.7 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption : 234.7 MB
Biggest message : 34330130 bytes
Memory/biggest message ratio : 7.2
Detected 3 errors
This current imapsync is up to date
Homepage: http://www.linux-france.org/prj/imapsync/
Thanks for the post. This is essentially how LuxSci’s “Managed Migration” works. So – you can either roll this yourself for free, or we can use our system to setup, track and manage this migration for you for a small fee. Depending on who you are — doing this is either fun or worth paying someone else to “just get done for you”.