Problem:

We have set up an Amazon EC2 server instance, and we also have an Amazon S3 bucket. I have connected the Server instance to the S3 bucket using TNTDrive, which allows us to map a Windows drive letter to an Amazon S3 bucket. Everything is working nicely except when a backup gets to the catalog sync it gets into a loop and never completes the sync operation, hence the backup never completes.

I have noticed when I change the RBS root to one of the Amazon EC2 EBS Volumes everything works as it should. When I switch it back to the S3 bucket drive the catalog fails to sync and gets into its loop again..

How are you connecting your software to the S3 storage buckets? Am I using the wrong approach or software?

ANSWER:

EC2 by default uses Amazon Elastic Block Store (EBS) for storage. To use S3, you have to use a third party software. EBS is specifically designed to be used with an EC2 instance. Amazon EBS allows you to create storage volumes that can be mounted as devices by Amazon EC2 instances. Multiple volumes can be mounted to the same instance.

The following pages have more information on EBS

http://aws.amazon.com/ebs/

http://www.cloudiquity.com/2009/03/differences-between-s3-and-ebs/

In terms of performance, S3 has a higher latency than EC2 and also has higher variation in latency. S3 write latency can also be higher than read latency . EBS on the other hand has lower latency with less variation. For file listing, S3 is slow and search is by prefix only whereas EBS has fast directory listing and searching. S3 is performance optimized by using multiple buckets. The write performance is optimized by writing keys in sorted order . EBS single volume performance is similar to a disk drive with writeback caching.

We have heard similar complaints from other customers when using a third party utilities to store data in the cloud. The third party software lacks IO speed and produces many errors. This is why our software uses EBS instead of S3. If you are determined to use S3, you might try looking for another S3 utility.