To those that know me it will come as no surprise that I’m a backup nerd. Since owning my first DAT drive in the mid 90’s that held a grand total of 2 gigs per tape (compressed) I’ve been interested in backup technology, redundancy and “playing it safe” when it comes to my data. That trend has been extended to my business for the benefit of my company as well as my clients.

This post will discuss a new backup appliance being introduced at NAB this year, and cover in general terms how we currently use its predecessor in our workflow. Even if you have your own method of archiving and don’t need anything at this time, you might want to read about our process toward the end of the post to see if you can benefit from it at all.

A Brief History – The Wrong Way
From 2002-2007 we used a hard drive backup system at our office that was working OK, but was starting to show signs of fatigue. With about 45 drives on the shelf (300-400 gigs each), some of the drives were 5 years old now and on occasion we would have problems retrieving files on some of the older drives.

There has been plenty written about the expected life of hard drives, the need to exercise them and the amount of time they will last on a shelf without regular use. No matter which side you’re on or what your personal belief is, I can tell you from first hand experience that we were losing data and I was getting very concerned. What use is a backup system that slowly loses its files over such a short period of time.

Back in April 2007 I went to NAB in part to find a new backup strategy (and in part to just relax in the hotel spa). After looking at plenty of solutions and talking to plenty of vendors, I came across the LTO A-Series drives from Quantum. It had some really great features that nobody else offered at the time, and was targeted to post production professionals. Some of the features include…

• Built on the LTO-3 standard, you could fit 400 gigs of storage per tape (tapes were about $50 each at the time, and even cheaper now).
• It was network accessible over Gigabit Ethernet. If you have multiple edit systems like we do, this is pretty big. You can park the machine anywhere on the network and access it from any local machine. With other solutions you have to move the drive from machine to machine as needed.
• No special software needed – using a standard FTP client or their built in web browser client you could backup and retrieve files with ease. It’s truly platform independent.
• The drive was based on the LTO open standard (there was also a SDLT version if you prefer). The two main benefits of this are that it’s an open standard that was created by Seagate, HP and IBM and the tapes have a 30 year shelf life.
• Each tape has it’s own built-in file system on the tape, so loading the tape directory and locating a file is fast and very responsive. No need to have a separate catalog file that you have to keep track of.

Implementing the A-Series Solution
The drive was not shipping at the time, but would be later that year. I patiently waited and in the fall of 2007 purchased the desktop version of the A-Series and a case of tapes.

This is where I’m supposed to tell you that everything went smoothly and rainbows were flowing out of the tape drive. Well, that was not the case. I’ll spare you the particulars and just say it took us almost 6 months, countless emails to tech support and a major software/firmware update to finally trust the drive with the long term safety of our projects. To their credit, the Quantum support staff and engineers were first rate in the way they responded to each and every problem we had. As annoying as I’m sure we were, they never made us feel we were any trouble even when we sometimes lost our patience.

In early 2008 they released a major software/firmware update that addressed 99.9% of our issues, and since then it has been smooth sailing. We have not had a single issue with archiving or retrieving elements and have found ways to turn the drive into a new revenue stream as well.

The Future is NOW (or at least very soon)
So here we are approaching another NAB, 2 years after my first introduction to the Quantum A-Series drive, and it just got better. I’m so excited and impressed with what’s coming out that I’m considering selling my current A-Series drive and buying the newer model. But, the new drives are not from Quantum. They’re from Cache-A.

Cache-A is a new startup company that has licensed the A-Series technology from Quantum so that they could build on the platform that Quantum created. They have some of the key players that helped create the original drive, and some seriously good people that have really thought through how to improve the new drive. Much of their feature list no doubt comes from feedback from the current customer base.

Here are the highlights of the new drives from Cache-A.

• The new drives are all based on the LTO-4 standard. This is huge, because now each tape can hold 800 gigs, double of the LTO-3 model from Quantum. That brings the price down to about 6¢ per gig. Wow!
• The drive is backwards compatible, and can read any LTO-3 tapes created with the Quantum A-Series drives. This is very smart, because it means upgrading from the previous model is very easy.
• The file system on each tape is now an open standard, which was not true before. This means that you can exchange a tape with anybody who has a standard LTO-4 drive. In the past, you both needed to have a A-Series drive to read the content. This was one of my biggest concerns with this solution originally, and it’s nice to see it has been fixed. No need to be that proprietary.
• The file system supports original metadata from source files, and they’ve made it clear they are adding to this support in the future.

I have requested an evaluation unit once they’re shipping, and if I get my hands on one I’ll do a follow up hands on review of the drive. For the previous review that Scott Roberts did on the Quantum A-Series drive we currently own, check out his post.

Looking a little further down the road, there are plans for the LTO format to progress with LTO-5 and LTO-6. LTO-5 will have a capacity of 1.6 TB per tape, and LTO-6 will hold 3.2 TB per tape. Time will tell if the technology continues down this path, but so far history has proven this format to be very robust and successful.

The Workflow
So how do we use the drive in our post production work flow? Here are the broad strokes.

Deciding What/When To Archive
All editing projects are archived to tape once the project has been complete and unchanged for 3 months or longer. It’s usually flagged for backup right around the 3 month mark by the primary editor on the project after conferring with the client. The backup will include all elements in the job (with the exception of digitized or imported original camera media) as well as a final master QT export of the project in it’s original format. If the client would like to include original camera media, we will do so for an extra charge based on the amount of space they need. This process is quick and easy since all projects use the project folder system I outlined in a previous post.

Once a project is marked for backup (by using colored labels in the Finder) it is moved into a folder called “Archive-Projects”. On the first of every month our assistant editor Scott goes to each edit system, verifies that each project has a master export, and creates backup folders that each contain roughly 400 gigs of files. We’ve found from experience that pre-organizing the folders/files in the finder prior to backup works much better then trying to do it all during the transfer process.

Performing The Backup
At this point it’s a matter of putting a blank tape in the drive and logging into the drive using your standard web browser. Using the client side software through the web browser you start the transfer by a simple drag and drop.

It takes roughly 3 to 3.5 hours to fill a 400 gig tape, but since this can run in the background it’s no problem to continue working on the computer. Once the transfer is complete, you get a confirmation message telling you that everything was successful, or a list of any problems that occurred during the backup process.

Once the projects are all successfully archived (as verified by Scott), he then turns the folders another color, indicating to me that the projects are safely archived and ready to be removed. Where removing projects is concerned, I’m a bit of a control freak and prefer that I’m the only one that deletes project folders and media. If a mistake is ever made and something is deleted by accident, I would rather be mad at myself.

Removing All Project Assets With A Simple Search
For the last part of this process I sit at the edit computer and load all of the media drives in the office (we have 2 raid systems totaling 20 TB of space, but a few years ago we had several G-Raid drives on each system). Using the job number from each job that has been archived, I do a file search for any and all files related to that job number across all drives. By doing this, I am able to not only delete the project folder, but all render files, capture scratch, project archives and so on. Once I have them all in the same window I give them a quick review – and then select them all and delete them. I then move onto the next project and repeat the step.

Keeping It Accessible
You may be wondering how we find files once they’re on the tapes. After all, we have over 25 tapes now and finding a single QT movie would be a nightmare without some sort of system in place. Well that’s the final step.

As each tape is filled, we mount it on the desktop of the computer and use a program called CD Finder to create a catalog file of all contents of the tape. This usually only takes a few minutes per tape. Once complete, the tape is in the CD Finder database and can be searched from any computer in the office without the tape being loaded. Once we find what we’re looking for, we load the appropriate tape and retrieve the files. It has turned out to be a great system.

The Payoff
So in the end, we have all of our important files backed up and safe for the next 30 years, they’re cataloged in a way that allows us to find a file in minutes, sometimes seconds, and we’ve made a small profit along the way by charging for the backup process and tape stock used. What started out as just pure overhead has turned into a income stream.

The SuiteTake
The amount of digital media attached to any single post production project is growing leaps and bounds at an alarming rate. Only a few years back having 1TB of storage per edit system seemed like a bottomless pit, and now we have 10TB per machine and at times are running low on space. The list of files to archive keeps growing, and if your workflow has includes tapeless media you have probably already been pushed to the limit.

One way to stay on top of managing your space is to do regular archives to a safe format that not only frees up drive space for new projects, but gives you peace of mind that all past projects are there when you need them.

As much as everybody wants to move away from tape, I still believe when it comes to reliability, cost per gig and shelf life, there’s still nothing better out there to archive your data. And I have literally put my money and company behind that belief. If you’ve been looking for a long term backup solution, I highly suggest you check out the new line of drives from Cache-A. If you can’t see them at NAB, be sure to check out their website or call them directly for more information. You won’t meet a nicer bunch of guys interested in helping you out.

See For Yourself
As of this writing, prices have not been set yet but I’m hoping they’re in the same area (or cheaper) then what the Quantum drives were selling for, which was about $7K for a single drive unit. If you’re going to NAB, be sure to stop by and visit them at booth SL-9605.

This is the second of a two part series on the Quantum A-Series LTO drive. You can find part 1 here.

Once Quantum released the unbelievably fantastic Version 3 upgrade three months ago, any minor inconvenience we were having with the tape drive seemed to disappear. They really did a great job listening to client comments and fixed virtually every problem that needed to be addressed. The interface is more fluid (you used to not be able to adjust the size of the windows), and there is no longer a self-destruct button next to the eject button. There is now an automatic preventative measure in place to no longer lose the table of contents (a problem we had early in its use, it appeared worse than it actually was). We can fill the tapes up as full as we want (we used to need to add a cushion of space to prevent filling the tapes “too full”). I can now let my pals < and ? into the drive without concern. Oh, they can invite the rest of their friends as well, the blacklist is lifted! There is still only a 97 character limit for filenames, but only once in a blue moon do I export FCP movies titled…

“Saturday_Conference_With_Rebecca_and_Steve_Morricone_ Featuring_the_Brilliant_Antics _of_Reggie Montgomery_Which_Occurred_at_9AM_Eastern_Time _or_8AM_Central_Time_Which_Would _Make_More_ Sense Because_We_Live_in_Illinois.mov”

The actual process of putting (small named) files on the tape is an easy, 5 step process. It’s so simple that I was able to teach it to my cousin Eric in just a couple of minutes (FYI he sniffs a lot of glue, it’s hard to teach him things).

1. Organize everything you want to put on the tape into a folder, name it appropriately. We use a date coded folder based on the day of the backup.

2. Insert a tape with enough free space to hold your folder.

3. Open the drive’s server interface, and connect to the tape on the right side.

4. Locate your folder on the left hand side (which is the local side).

5. Drag it to the right side.

That’s it. It really is that easy. And that’s one of the best parts about this drive, it’s so incredibly easy. You can then watch the progress in the bottom part of the interface.

And it will let you know when it is done.

Taking things off of a tape is just as easy. Just drag the files from the right side to the left side!

Now I know what you’re thinking. “Hey Scott, I just backed up multiple terabytes on these LTOs, now I need to pull stuff off of them, but how do I know where anything is on these tapes???” Well, my amigo, that answer comes in the form of a sort of mashed together single word, CDFinder. This is a spectacular cataloguing tool that we used back for the IDE drives that still carries over to our tape use. We now have (at the time of this article posting) 46 archive tapes/drives. CDFinder helps us sift through all 46 places that a file could potentially be. The way to put a new tape into the catalog is unbelievably easy.

Just load the tape onto the desktop by going to Go > Connect to Server.

Then open CDFinder, and drag the tape to the left part of the CDFinder interface.

Hit “Create New”. And it will start to catalogue your tape.

This is sort of a buggy process, as the tape drive will claim to disconnect every few minutes or so, but it’s just an illusion, as it will eventually catalogue if you just let it do it’s thing. Then just rename the catalogued tape whatever it is you’d like to call it, as I’m sure you won’t want the sever number as the name of your tape (though, I do plan on naming my child “10.1.10.70 Roberts”. It works for a boy OR a girl!).

Then the process of finding things just requires that you hit “Find”.

Type in what you want to find.

And it will search ALL of your tapes, and find what you seek. This whole process it relatively intuitive, and as stated many times, abnormally simple.

I think I saw this once on a math test in 7th grade...

Now I know no one likes to hear it, but the A-Series isn’t exactly cheap at $8000. But hear us out here! Before we had the tape drive, back in the IDE external drive days, we archived everyone’s projects as a common courtesy. Just for our own well being knowing that projects come back from the dead after a while, that’s just how it goes. But with the longevity of the LTO tapes, backing up our clients projects has now become (in addition to a way to easily access old projects for use in current projects) a way for our clients to have long term storage of their valuable work. So we now charge our clients for each project that they have archived. Pulling things off of the tapes is free to them, but the initial archiving now costs them a small fee. This is a win-win for us and the clients, as they will have their projects safe for several decades, and we will pay off the expense of the drive in the near future, and even start making money on it! It’s an investment!

Sadly, the cashier at Target wouldn't let me buy an Xbox using LTO tapes instead of money.

I guess the underlying point of this was to let you know, if you happened to have doubts about LTO tape based archiving, you were right to perhaps have doubts in the past, but it all seems to be good now. The Quantum A-Series is a fantastic machine for archiving large amounts of media. I recommend the Quantum A-Series tape drive NOW, because it works good NOW. It is definitely faster to actually pull things off of it or put things on it compared to the IDE drives. It also importantly comes with the assurance that bad data blocks won’t get copied over, giving us very comfortable levels of security. I think what we expected it to do when we first got it, is finally happening ten fold in recent times. Bail on the hard drives, this is a good time to be tape based archiving. And you should make it an A-Series.

Our ever growing collection of filled LTO tapes.

In a future article we will look at how we use the LTO drives in combination with tapeless production cameras. Having a good workflow is the key to everything.

For more detailed specs on LTO technology in general, check out Wikipedia.

Archival of editing projects and tapeless camera originals is a hot topic these days, with opinions flying everywhere. Today’s post covers our experiences with both hard drives and the new A-Series LTO drive from Quantum.

In early 2008, we decided to switch to LTO tapes for all of our long term archiving.  We previously used external IDE drives (well, internal drives put in cases to make them external) for our backups.  They work decently for at least a while.  The problem was, it got to the point that we had to use Chronosync, a file synchronizing software, to bring back any element from a project because digital hits would appear in the video files due to bad data blocks copying over from the drives back onto our system.

Chronosync Error. The bane of my existence.

There is a “Verify Copied Data” function in Chronosync that detects any of these digital hits, and causes it to restart the copying until it got it right.  Sometimes it would take fifteen tries just to get it so that it wouldn’t have a hit in it.  It is easily one of the most frustrating things I’ve ever seen.  And you know it never happened 10% into the copy, it always cruelly waited 15 minutes for your 30 GB video to get to 92% before it decided it was going to restart.  And while the Chronosync’s verification worked in ensuring a good file, doing the verify literally slowed the copy speed down a drastic amount.

IDE Hard Drive, in a special case to make it external and loadable in a hard drive reader.

Another eventual downfall of  hard drive based archiving is the longevity of hardware itself.  Hard drives tend to only last a couple years or so.   So if you were hoping 20 years from now that the Johnson project is still on that cob web covered hard drive, you probably won’t be happy to see that your hard drive most likely won’t even turn on.   It takes a little TLC to make sure that hard drives last as long as they possibly can last.   Every month or so, you should connect each hard drive you use and “spin them” in order to keep them fresh.   If you let a hard drive sit dormant for too long, it will just give up on itself.   This can be annoying and time consuming when you have 30 hard drives.  Hard drives crave a lot of attention, just like that one hyper kid in my 8th grade math class who would run up and down the aisles waiving his arms.  But tape based archiving is great because you can just put the tapes on a shelf and ignore them for long periods of time, only using them when you want to.  They are said to last approximately 30 years.  And if you are wondering what you are going to do with the files on your tapes 30 years from now, man, you are thinking way too hard about the distant future.   We will probably have thought-controlled nano-archives implanted in our brains by then.

So, the LTO drive was a perfect upgrade.  It had the power of John Rambo, the speed of Barry Sanders, and the elegance of Jackie Onassis.

Before we go any further, perhaps we should take a moment to go over a brief history of the LTO tape, and discover its mysterious origins.  Unlike most of us humans, the LTO tape was born in a laboratory, in the late 1990s.  It was given the name Linear Tape Open, named after its grandfather Linear Tape Ostrofsky, who fought bravely in The War.  It is a magnetic tape, which was first publicly released in 2000, and was only big enough to hold a measly 100 GB, but it was adorable nonetheless.  LTO grew up fast though, and in 2007, it became capable of holding up to 800 GBs!  And the roadmap for he future looks bright as well. On average, the capacity of the LTO tapes has doubled every 2-3 years.

The tape capacity, coupled with its relatively cheap price tag ($30-$40 for the tapes), makes it a valuable asset to backing up large files, for instance, video files.  It is now one of the best selling tapes on the market.  Its most prominent use?  You guessed it, backup.

The Quantum A-Series tape drive.

But the tapes won’t do any good without a drive to put them in.  Believe me, I’ve tried everything; yelling at the tapes, bribing the tapes, waterboarding the tapes, nothing works!  Then I heard a rumor that they will do whatever you say if you put them in a tape drive.  Seems logical to me.  Now, there are several LTO drives on the market, and they do alright for their purpose I suppose.   But Quantum has created a special LTO drive, the A-Series drive.  This drive was created with the post production market in mind, particularly regarding large media backup.  One great selling point of the drive is that it has a built in server. With the build in server software there is no need for any additional software to do backups, and nothing to install.

In addition to that, the A-Series is network based, so it is highly convenient that the drive can be accessed by multiple computers, as opposed to the standard method of having to nomadically hook the drive up to each separate computer that you’d like to use it on, moving it around constantly.   This is one of (if not the only) LTO drive on the market that has this networking ability.

The Series-A server interface.

The actual process of putting files on, and taking files off of the tapes is done through the drive’s web interface.  It is highly convenient that you can archive a project and order new underwear on Amazon with the same program.  There are some problems with the interface, some of which got fixed in recent updates, others that still remain to this day.  For example, one thing that has gotten fixed since was in the right click drop down menu, the “Eject Tape” and “Erase Tape” commands were literally next to each other.  This is almost cartoon-like in its insanity, much like building a robot and putting the “Self Destruct” button half an inch away from its “Off” button.  But things like that got fixed upon request, hey, pobody’s nerfect.

A-Series Drive with tape loaded in. It's good to go!

One of the earliest problems with the drive was a simple confusion with the reality of deleting folders on a tape.  That reality is that you can’t. It’s all or nothing when it comes to tape based archiving.  It is a linear device and can’t be defragmented, much like video tape.  Though, it is possible to delete the contents of an entire tape.  So what that means is if you have a tape with 200 GB on it, and you add a 100GB video to the tape (see: WatchOut), and then you realize, “Oh wait, I already added that 100 GB video to a different tape last week!”  Unless you want to erase the previous 200 GB that were already on the tape, there is no way to delete the 100 GB of previously archived video that you just put on the new tape.  So that is lost space, and granted it is “cheap” tape (costing between $30-$40), but it still feels wasteful to lose ¼ of your tape due to this.  That has nothing to do with the hardware or the software, that simply has to do with tape in general.

Now, when you deal with the Series-A drive, you have to deal with a ton of characters.  A lot of them you will want nothing to do with.  Take Quotation Mark for example, never invite that guy to your house, he won’t shut up the whole time he’s there.  And don’t even get me started on Asterisk, you don’t want to know what he did at my cousin Tony’s wine mixer.  There is actually a blacklist of jerks that weren’t allowed into the LTO drive:

The Usual Suspects: I bet one of them is Keyser Soze.

If you attempted to sneak in any of these characters into your archive party, any file associated would get bounced out of the club with them (something I sadly know too well in real life…).  They just wouldn’t be included into the archive. This has to do with the fact that the software that was built into the drive is set to be very strict, and follow Windows format.   So all Mac people must conform to the tyranny of Bill Gates for at least one element of this process.

In previous versions of the software, there was little to no indication that anything was skipped at all, because the transfer summary that shows in the interface used to not exactly work (which was related to a Java problem I believe).  In addition to that, when you would put files on a tape (if I filled a tape with 345 of 400 GB), it was hard to certify that the files actually made it on to the tape, because if you mounted the tape in the Finder and did “Get Info”, the Finder would say that there was actually 100 GB less on the tape than what was in the original archived project!  That of course makes you incredibly nervous that your files aren’t going to be backed up properly.  There is apparently a discrepancy between the Finder and reality itself.   The only real way to see how much space is left on a tape is to put in in the drive, and see for yourself how much is left when it shows it in the interface.  (I keep track of all of these “space remaining” numbers in an Excel document. It’s old school, but it works for an easy reference to how much space is left on each tape)

The interface let's you know plain and simply how much space is really on your tape.

Another small element worth noting in the battle between tapes and hard drives is the speed it takes to access your files.  While the actual process of transferring files on and off your archive is faster on tape, the time it takes to locate the files are significantly different.  Think about it this way, if you’ve taken your date from the saturday night sock-hop to the top of Make-Out Canyon in your DeLorean, and you pop in your Run-DMC mix tape cassette, and you want to listen to track 9 (because sometimes words can’t express love the way that the song “My Adidas” can), then you have to sit there and wait for your cassette to fast forward throughout the entire tape until it gets to track 9.   Where in contrast, if you were to pop in your Run-DMC mix CD, you could instantly go to track 9 without having to fast forward over the other tracks.   The wait isn’t too bad (maybe 2 minutes on a bad day) for tape fast forwarding, so it’s only a problem if you’re horribly impatient.  So, while a drive is instant access (like any folder in the Finder), the other benefits of the tapes should outweigh its inability to instantly access files.

Run-DMC helped popularize tape based archiving in the early '90s with their chart topping smash hit, "I'm Gonna Back Up Dat Booty (Right Onto My Tape Drive)".

Next week I’ll post Part 2. It’s an in depth look at how the Quantum A-Series drive actually works, and how to make money with it!  

We will also explore how to cook LTO tapes into a delicious winter stew, how to turn your A-Series drive into a muffler for a Dodge Ram, and we will discuss my brief stint as a vocalist for Crosby, Stills, Nash, Young & Roberts.

**DISCLAIMER**  THE STATEMENTS MADE IN THE PREVIOUS SENTENCE MAY OR MAY NOT BE FALSE.