According to the study, Skyplane is up to 3.8 times more affordable and up to 110 times faster than services like AWS DataSync.
Skyplane, an open source tool to optimise the transfer of huge datasets between cloud providers, has just been released by a research team at the RISELab at UC Berkeley. It cuts down on transfer times and expenses. Skyplane now supports transfers between object storages on AWS, Google Cloud, and Azure. Skyplane is designed to be cloud agnostic. In order to avoid congested cloud network connectivity, Skyplane builds an overlay network on top of the cloud providers. By slicing data transfers across numerous pipelined TCP connections and numerous VMs, it also makes use of LZ4 compression and parallelism.
With the use of MIT RON (Resilient Overlay Networks) and cloud network cost and performance characteristics, Skyplane transparently identifies the best transfer channels between regions and cloud providers. For instance, the Skyplane CLI command is as follows to move a single huge file across two AWS regions:
$ skyplane cp s3://srcbucket/64gb.tar.gz s3://dstbucket/
Although the syntax is identical to that of the AWS CLI, the benchmarks indicate considerable gains in cost reduction and parallelism. The tool’s architecture has been described in an article that covers security, firewalls, transfer integrity, and checksumming.
Skyplane only supports local to cloud transfers for AWS and Google Cloud at this time by defaulting to awscli and gsutil, respectively. Additional providers may be added by integrating with the roadmap’s plans to support Oracle Cloud, IBM Cloud, and Alibaba Cloud. These clouds have object store and authentication APIs.
There are other ways to execute high throughput transfers besides Skyplane. Although there are costs associated with using these services, which are often tailored to move data into the provider’s cloud and lack portability, developers can utilise them to transfer data using tools like AWS DataSync or GCP Storage Transfer Service.
