Storage Optimization

The impending storage crunch

Posted in Storage by storageoptimization on July 28, 2008
Tags: , , ,

No one can miss the fact that data storage is spiraling upward at a terrifying rate. Joerg Hallbauer puts it on Dell‘s Future of Storage blog hit the nail on the head with his post: “We are running out of places to put things.”

Citing data collected by IDC, Hallbauer concludes that in a mere three years, we there will be 1400 exabytes sitting on disk. Currently, according to the study, there are 281 exabytes of data being stored, and the CAGR rate is 70 percent. Much of this data is on laptops, home computers or servers under your desk today, but as Joerg correctly notes, there’s no question its migrating quickly to the cloud. Huge data centers will end up holding most of this data, and disk drives are not growing fast enough to deal with it anymore.

So, where do we go from here? Well, if the traditional answer was, wait for bigger drives so I can put more stuff on a disk, the other logical thing to do is to say, how can I put a lot more stuff on disks that I already have?  The answer is advanced storage optimization. The first simple storage optimization solutions are out there today – single instancing, deduplication, and compression. But the area of storage optimization is really just taking off, and much more sophisticated approaches are emerging that will allow a disk – whatever its physical size – to store 10, 20, or 100 times more data than it does today.  

What’s more, the move to large data centers providing huge cloud storage services will make this more efficient, because storage optimization is all about finding redundant information and figuring out how to store it more efficiently. So the larger the data set, the more likely you will see big wins from next generation storage optimization.

This also naturally leads to more tiering. Where today you have fast disks (Fibre Channel or SAS) and slow disks (SATA) making up the tiers, it’s much more likely in the future that the fast tiers will be solid state storage of some sort (SSD and Flash, as Joerg points out) and the massive tiers that hold the bulk of all these Exabytes will be the largest possible disks integrated in to systems that have very efficient storage optimization built in.

Image credit: Orange Photography blog archives


5 Responses to 'The impending storage crunch'

Subscribe to comments with RSS or TrackBack to 'The impending storage crunch'.

  1. Carter

    In fact, I think the issue is that we must *delete” the data we don’t need. Most people are inherently lazy. Increasing hard disk sizes have endulged their sloth and consequently no-one deleted anything.

    Imagine you had a leaky roof. Would the answer be to find more pots and pans to catch the water? No the answer is mend the roof….

  2. storageoptimization said,

    That’s a good point, Chris. Of course, compliance rules – especially for larger organizations – make it illegal to delete anything. Sarbox and HIPAA have created tremendous growth in storage.

    I just talked to a hospital that has been in business for over 100 years. In all that time, their storage had grown to a total of 25 Terabytes. They project it to be 250 Terabytes by the end of next year. That’s tenfold growth, and it’s not because they are lazy. It’s because a) the latest generation of medical imaging machines are spitting many more images at much higher resolution and b) laws require them to keep these images archived for a very long time. By the same token, one of the web site customers we have spoken with offers a service for customers whose marketing pitch is “preserve your memories”.

    The explicit promise is to keep your photos (and other things) safe forever. Again, that’s not laziness, it’s a combination of the fact that digital content now means a lot more to people, and the psychological fact that they want to keep it. It’s not laziness, it’s desire. That said, in the typical corporation, there are no doubt kajillions of PowerPoints and Word documents that are just sitting there, that no one is ever going to look at again, and which are not covered by compliance regulations – and that stuff could be deleted.

  3. UsualAnalyst said,

    I talk to a lot of people that are advocates of “delete old data”. They must not be familiar with situations where old data is still valuable. Imagine telling a studio company to delete their 30 year old movie. what nonsense that would be.

    The point is, some information created is rubbish. that can surely be deleted – not 10 years later – but actually the day after it was created.

    Most other information is useful and took someone’s time and effort to create. You cant just go around willy-nilly deleting information to save space.

    By that paradigm, we’d just burn up all our libraries.

  4. […] a step closer to an “any movie, any time” model.  Plenty to consider in the upcoming storage crunch. Possibly related posts: (automatically generated)The impending storage crunchFree Online Storage – […]

  5. […] a step closer to an “any movie, any time” model. Plenty to consider in the upcoming storage crunch. Featured, […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: