Tag Archives: s3

cloud Final Cut Server

Amazon S3 File Size Limits

Back in November of 2008 I did an article awhile back on a way to use Amazon’s S3 to hook into Final Cut Server. At the time though, S3 had a pretty big limitation in that it wasn’t really suitable for backing up large video files as an archive device for Final Cut Server. But today, Amazon announced that S3 now supports files of up to 5 terabytes using multipart upload (previously the maximum file size was 5 gigabytes).

This finally means that files do not have to be broken up at the file system layer in order to back up to Amazon’s cloud. However, this does not mean that traditional disk-to-disk backup solutions can be leveraged to provide a good target for backups and archives as backups need to be performed using the multipart upload. The ability to now use S3 for large files allows us to finally use Amazon S3 in a way that is much simpler that it was to do so previously, although it is still not as transparent as using a file system or URI path.

Overall, this represents a great step for Amazon and I hope to see even more of this in the future!

Consulting Network Infrastructure sites Unix VMware

S3 Command Line Part II

Earlier we looked at using s3cmd to interact with the Amazon S3 storage cloud.  Now we’re going to delve into using Another S3 Bash Interface.  To get started, first download the scripts and then copy the hmac and s3 commands into the ec2 folder created in previous walkthroughs.

To use the s3 script, you need to store your Amazon secret key in a text file and set two environment variables. The INSTALL file included with the package has all the details. The only tricky part I ran into, and from the comments on Amazon, other people ran into, is how to create the secret key text file. Now go into your environment variables in ~/.bash_profile and add S3_ACCESS_KEY_ID (your S3 access key id) from the S3 site on Amazon Web Service and S3_SECRET_ACCESS_KEY (the name of the file with your S3 secret key). If the file that stores this key is called ~/SUPERSECRET then to create it, copy the key to your clipboard from the AWS site and then run echo -n and send the contents of the pasted line to a file:

echo -n MYPASTEDKEY > ~/SUPERSECRET

The -n switch tells echo to not include a new line character and results in a text file of exactly 40 bytes. Once I got the key file created correctly, the s3 script started working, and I was able to upload, download, and list objects in S3.

Next, we’ll list your buckets:

s3 buckets

Then we’ll list the contents of a bucket called images:

s3 list images

Next, we’ll upload a file called emerald.png from the desktop of our computer:

s3 put images emerald.png ~/Desktop/emerald.png

Now let’s try and get the same file and just leave it somewhere else so we can compare the two:

s3 get images emerald.png ~/Documents/emerald.png

Now let’s get rid of the file:

s3  rm images emerald.png

And then to just remove all the files from the bucket:

rmrf images

If you notice, this toolkit is very similar to the s3cmd kit that we looked at earlier.  It’s a little more limited, but I thought it might come with less of a learning curve or be easier to script against depending on what you need.

Consulting Network Infrastructure sites Unix VMware

Custom VMs using S3+EC2

I’m obviously enjoying using Amazon for a number of testing applications (in addition to of course buying books and light bulbs from them, and not one showed up broken).  So far, I’ve done articles on getting started with Amazon ec2, using the command line with ec2whitelisting an IP addressdeploying ec2 en masse, and setting up a static IP for ec2.  But the S3 articles have been sparse.  So, now let’s look at using Amazon’s storage service (S3) from the command line.  Funny enough, if you’re going to upload your own custom Amazon Machine Instances (AMIs) you’ll need to leverage S3.

When you go to bundle an image, you will have a couple of commands to run.  The ec2-bundle-vol will build a bundle and and ec2-upload.bundle will upload the bundle to Amazon using an S3 bucket.

The ec2-bundle-vol command can be run using the existing boot volume or you can specify a different volume to create an AMI from using the -v flag.  You will also need to specify a destination using the -d flag (the destination will usually be /mnt) and you will need to put your private key into the image using the -k flag (the .pem files from earlier articles) .  Also, the size of the AMI will be defined with a -s flag (in Megabytes) and the ec2 user id will be defined using the -u flag followed by the actual userid.  Finally, if you would like, you can choose to exclude specified directories (using a comma seperated list) using the -e flag.

So the command, if you’re booted to a CentOS host that you want to use would use something like this:

ec2-bundle-vol -d /mnt -k ~root/pk-MYSTRINGOFNUMBERS.pem -u 1234567890 -s 4096

This will create a bundle along with a manifest file (which will be in the form of an XML file.  Now, on S3 create a bucket, let’s just call it EC2, and then in here, let’s create a directory called bundles for all of our ec2 bundles to reside.  The ec2-upload-bundle command would then be used to upload the bundle to Amazon.  Here, you’ll use the -b flag to define the name of the bucket that was just created and then the -m flag to define the xml file (which btw will tell the command where to look for all of the parts of the image).  Here, I used  username and password but you could also use your AWS access key and secret access key by using the -a and -s flags respectively.  So an example of the command would be:

ec2-upload-bundle -b EC2 -m mybundlelocation/my.manifest.xml -u 1234567890 -p MYPASSWORD -d bundles

Now that the bundle is on s3, let’s go ahead and register it with ec2.  To do so, we’ll use the ec2-register command followed by the s3 bucket name we uploaded the image to and then the relative path to the manifest:

ec2-register ec2/ buckets/image.manifest.xml

The output of the above command will include a unique machine identifier, we’ll call it ami-id (almost as though we would use a $ami-id variable if we were scripting this).  Now, you can run the ec2-run-instances command, specifying the unique identifier as follows (replacing ami-id with the actual ID):

ec2-run-instances ami-id

One way to leverage S3 is through the lens of ec2, another is by simply using s3.  I covered using s3 a little bit when I discussed using it as a location to store assets from Final Cut Server, but I will cover using it from the command line soon.

Final Cut Server

Final Cut Server: Using Amazon S3 for Archival

Final Cut Server allows you to archive the primary representation (or the original file) for assets that are cataloged.  When you do so, the proxy clips (low resolution versions) of your assets still live on the Final Cut Server.  However, the primary representation, once moved to your archive device can then be archived off to another form of media.

There are a variety of strategies to manage archived media. The one I will describe here is using the Amazon S3 storage service at a cost of approximately $.12 to $.15 per gigabyte. As a conduit to and from Amazon S3 we will use the Jungle Disk application, which uses the Amazon S3 API to provide a mount point to Mac OS X.  Before you get started, first create an Amazon account (or enable Amazon Web Services for your existing Amazon account).  Once you have enabled Web Services, click on the link that will be emailed to you that allows you to create an Access Identifier. Also keep in mind that file sizes cannot be larger than 5GB per file.

To get started, download Jungle Disk from http://www.jungledisk.com. Once downloaded, run the installer. At the welcome screen click on Next.  At the Jungle Disk Account Information screen enter the Access Identifier and the Secret Key for your user account.

Next, tell Jungle Disk to use the storage from Amazon as a Network Drive. Here, I gave this drive a name of FCSBackup.

Next, create a new bucket (or use one you have already created).

To create a new bucket, click on Next. At the Bucket Setup screen provide a name for your bucket of storage within S3.  I called my bucket fcsvrbackup.  Here you can use standard or high encryption. Speeds will be reduced with high encryption but the data will be more secure. Click Next when you are satisfied with your settings and then click on Finish to complete the installation.

Next, for speed we’re going to do a little quick tuning.  Open the Jungle Disk Configuration application and then click on Network Drive for the fcsvrbackup bucket.  Then increase the maximum cache size and check the box for Upload files in the background for faster performance.

Next, open /Volumes and verify that you see your fcsbackup (or whatever you decided to name the volume).  Alternately you can use the Bucket menu from within JungleDisk Monitor to click on Show Network Drive in Finder.  Once you have verified that your mount is there, test copying data to the folder to verify that you have full write access. Once you are finished, open the Final Cut Server System Preference pane. Then click on the plus icon (+) to bring up your Device Setup Assistant.

Here, click on the Local Device type and click on Continue.  

Next, open a Finder screen and open /Volumes/ (Command-Shift-G).

Now drag the FCSBackup over to the location field in the Device Setup Assistant and provide a name for your Final Cut Server to refer to your Device as (I used Amazon Backup here). Now click Continue.

Next, check the box for Enable as an Archive Device and click on the Continue button. At the next screen, click Finish.

Now go to your trusty Final Cut Server client application and control click (or right click if you’re so inclined) on an asset. Here, you will click on the Archive item in the dialog box.

Now, if you go to the FCSBackup volume you should see the file you decided to archive. These will be stored in a folder that corresponds to the device ID that Final Cut Server has for your “device”. Only the primary representation has been moved at this time, so your proxy media for these files is still in your proxy bundle. Now, click on the asset within the Final Cut Server client application and then perform a get info (Command I). You will now see the relative path to your device that the file is in. You can now unmount the FCSBackup drive and you will still be able to access the file. Once you have uploaded some files, tap into Amazon and check out how much they’ve charged you…