Warakorn - Fotolia
Moving terabytes of data to or from cloud services can incur significant time and expense. There are several archival options for AWS cold storage that enable enterprises to move data without significant strain on resources.
Dedicated high-speed network connections, such as AWS Direct Connect, can establish a dedicated connection between a data center and an AWS cloud facility. The Ethernet connection bypasses the public internet, avoiding bottlenecks and congestion that often reduce bandwidth and lower data transfer performance. AWS Direct Connect services are available in 1 Gigabit Ethernet (GbE) or 10 GbE speeds; IT teams can aggregate multiple links for additional bandwidth.
The AWS Import/Export Snowball is another way to transfer huge volumes of data, but it uses a portable out-of-band storage appliance. An administrator requests a large transfer job, and AWS ships the enterprise a storage appliance. The admin then connects the appliance to the network and transfers the desired data, which is encrypted. The company ships the device back to AWS, where the data is moved to storage using Amazon's internal network.
The process sounds old school, but this data migration method can actually be faster and less expensive than moving data over the internet, depending on a company's network connection speeds and the amount of data it needs to move. AWS Import/Export Snowball can be a good option for data volumes as low as 60 TB over a 1 Gigabit Ethernet connection. Conversely, the appliance might be a better alternative when restoring massive backups from cloud storage to the local data center.
Neither AWS Direct Connect nor AWS Import/Export Snowball directly moves data to Amazon Glacier. Data transfers to Amazon Simple Storage Service (S3), and then admins move the data offline to a Glacier vault. There are other alternatives to accelerate data transfers between an enterprise and AWS, such as Amazon Kinesis Firehose, which addresses multiple streaming data sources, and Amazon S3 Transfer Acceleration for recurring storage jobs with incremental changes -- usually over long distances -- and gateways to cache data locally. But AWS Direct Connect and AWS Import/Export Snowball are probably the most desirable options for use with AWS cold storage.
There are also some cost considerations for using AWS cold storage. For example, each object that Glacier archives requires an enterprise to store some data in S3 to maintain metadata; this can be as much as 40 KB per object. While that isn't much, the total can add up for a large number of objects. When an IT team plans to retain those objects in Glacier for months -- or even years -- storing Glacier object metadata in S3 can add unexpected cost to the archives. It can be helpful to compress many small data objects into a single consolidated .tar or .zip file before uploading and moving them to AWS cold storage.
There is an early deletion fee if an admin deletes or changes an object within three months of creating the archive -- no charge is incurred for changes that take place after three months. There are also costs for moving each object to Glacier, which means moving a large number of objects can be expensive. IT teams can incur other costs when restoring data to S3 from Glacier. AWS usually charges for restorations based on the highest rate encountered in GB/hr.
Amazon Glacier has cold storage competition
Essential guide to AWS data management
Sling big data to and from the cloud
Dig Deeper on AWS database management
Related Q&A from Stephen J. Bigelow
Eliciting performance requirements from business end users necessitates a clearly defined scope and the right set of questions. Expert Mary Gorman ... Continue Reading
Requirements fall into three categories: business, user and software. See examples of each one, as well as what constitutes functional and ... Continue Reading
Navigating data center malfunctions when hardware is off premises can be tricky. Organizations must have strong SLAs with their colo provider to ... Continue Reading