Wednesday, February 14, 2018

Moving Google Drive data via Google Compute Engine

Recently I've been using Google Compute Engine to move some personal data (several TB) from one Google Drive account to another. This post is my experience for moving more than 10TB data.



The rough monthly cost I paid to Google: $75/mo.

Compute Engine Standard Intel N1 1 VCPU running in Americas for 744 hours: $35/mo
Compute Engine Storage PD Capacity: 1024GB Gibibyte-months: $40/mo

Yes, I inserted a 1TB drive for temporary storage so I can move stuff in bulk.

The cool thing is that Google doesn't charge for the egress traffic data from/to Google Drive, which actually saves a lot of money. Saying you got 10TB data to move, according to Google's Internet egress rates, that would be $0.12 * 10 * 1024 = $1228.8 for only the traffic.



I used the Go-written tool called skicka to move my stuff, that I've been using for downloading/uploading files to my Google Drive for a long time.

By default, skicka saves its metadata at ~/.skicka.metadata.cache  and ~/.skicka.tokencache.json, but it's also easy to use a different set of metadata when you have multiple accounts:

skicka -tokencache ~/skic/account1.tokencache.json -metadata-cache-file ~/skic/account1.metadata.cache download /<dir name> <local dir name>
skicka upload <local dir name> <dir name>

I used the commands above to move the files in different pre-organized directories, each has around 500~800GB stuff.



The reason why I'm glad to pay more money for the N1 instance was because of the egress traffic throughput caps. Basically each vCPU has a 2 Gbps egress cap, the more vCPUs you have, the faster the speed would be (up to 16 Gbps). Shared vCPU instances (such as f1-micro or g1-small) are counted as 0.5 vCPU so the egress cap is 1Gbps. This really matters when you have over 10TB data to move around.

Details: https://cloud.google.com/compute/docs/networks-and-firewalls#egress_throughput_caps


My ISP plan has 250Mbps Down / 25Mbps Up. Using Google Compute Engine as the tmp storage just saved lots of time and also avoided the freaking data usage cap enforced by my ISP.



It actually took me several months to move all my data from one Google Drive account to the other (because I was so busy). I really wish I could invest one weekend to write some automation scripts to grab stuff from/to Google Drive but unfortunately I didn't. The total cost ended up to $75*N months but in which I applied some portion of my $300 credits to it, so it turned out not really a bad deal.

No comments:

Post a Comment