Economic Woes and Storage

Every major business magazine has a cover story this week on the economic turmoil that’s gripping the credit markets, Wall Street, and the rest of us. Every one, that is, except Forbes, which chose to put John Chambers, CEO of Cisco on its cover this week. No doubt, some editors over there are wishing they’d made a different choice at this moment–but leaving that aside, in some ways this story says more about the economy than any of the others.

Cisco, the article demonstrates, has jumped in with both feet into the area with the greatest promise: data centers. This unglamorous chunk of reality that underlies all the fun and fancy Web 2.0 that, for now, is keeping Silicon Valley from tanking along with the rest of the economy. (Unless you believe the NYT, of course.)

To quote the Forbes article: “This is what the online computing revolution has become, a giant electricity hog of Internet searches, phone calls, blog posts, wireless downloads, bank transactions and office documents. And video, lots and lots of video. ” The article also includes a chart comparing new server spending v. power and cooling costs.

All of which leads us to the inexorable conclusion–which TechTarget’s Dave Raffo refers to in a recent post–that one of the few places that is sheltered from the current storm is anything that reduces the cost of storage. So yes, storage optimization is the place to be in today’s tough economic climate. But the main point is that it could help keep lots of companies afloat that might otherwise crumple under the weight of their storage costs.

Are you Content Aware?

Storage analyst Robin Harris commented on the storage story of the week–NetApp’s Guarantee that virtualization will mean a 50% gain in storage capacity for its customers. 

Harris’s take on the announcement is that dedupe for primary storage could be “the next big win for IT shops.” Perhaps, but let’s keep in mind that NetApp dedupe is very simple. It only finds duplicate blocks at NetApp WAFL 4K block boundaries. The reason that they are positioning it as a big win for VMware users is that virtual machines (static images of whole operating systems) are exactly one of the few places where you’ll find lots of dupes in primary storage on block-aligned boundaries.

Here is my take: The best results in dedupe for primary storage are going to be from applications that can recognize file types and understand how to find the duplicate information in them. That is where the big wins in dedupe for primary storage are going to be.

Consider this typical scenario: I create a PowerPoint and email it to someone else. They save it, open it, and make an edit – add a slide, or even just edit a bullet or two. That small edit will mean that none of the redundant content of that file falls on the same NetApp WAFL block boundaries. So although the two files are almost entirely the same, you won’t see good dedupe results on them.

A content-aware solution – which combines both information-level dedupe with content-aware compression – should be able to get 10:1 compression on most typical file mixes (especially those Office and engineering ones). A 10:1 ratio is the same as 90% reduction, so if you can shrink 80% of your data by 90%, so can get a pretty good handle on how big the win could be. And by the way, It’s not necessarily a bad thing for the guys who sell disks, either, because what happens when you can get that kind of win is that you start to think differently about what you can store, and how long you can store it for. For example, at my company Ocarina Networks (www.ocarinanetworks.com), we have a customer that plans to store a snapshot a day online for every day’s data for 10 years. That wouldn’t be possible without some drastic deduplication.

Block level dedupe – whether simple block-aligned like NetApp or sliding window like market leader Data Domain – is only going to find a small subset of the duplicate or redundant information in primary storage. That’s because most file types that drive storage growth in primary (or nearline) storage are compressed. Compression will cause the contents of a file to be recomputed – and to look random – every time a file is changed. So if I store a photo, then open it and edit one pixel and save the new version as a new file, there won’t be a single duplicate block at the disk level. On the other hand, almost the entire file is duplicate information.    

Can you find a duplicate graphic that was used in a Powerpoint, a Word document, and a PDF? Powerpoint and Word both compress with a variant of zip; PDF compressed with deflate. Even if the graphic is identical, block level dedupe won’t find the duplicate graphics because they are not stored identically on disk. You need something that can find duplicate data at the information level. Finally, there are pretty concrete data that say that about 80% of the file data on NAS is a candidate for deduplication.

With all that in mind, don’t you think content aware optimization is going to be the next truly big win?

Sea Change - Google’s latest scheme makes waves

This week Matt O’Hern called out my company Ocarina Networks on the Marketing Shift blog as an example of what’s being done to manage the upwardly spiraling load of online data. Google’s latest data storage scheme–placing data centers on barges and creating an offshore “waterworld” of data–may seem bizarre to some, but this move on their part perfectly illustrates the desperate situation many such companies are in, as the tide of data reaches ever more oceanic levels. This is particularly true for those which, like Google, offer free, unlimited storage to their users for photos, emails, videos, and other files. As O’Hern rightly points out, companies are turning to Ocarina to help them optimize that data, reducing their data footprint. Glad to be mentioned in this timely and important story.

IBM Suddenly Notices “Mountainous Piles” of Data

Seems like it’s hitting everyone at once: there is a whole heck of a lot of data out there, and all of it must be stored. According to ZDNet’s recent coverage of IBM’s new storage strategy: “IBM estimates that the average individual’s ‘information footprint’ –the amount of data connected to a person–will grow to more than 16 terabytes by 2020 from roughly one terabyte, or trillion bytes, of data currently.”

In response, Big Blue is revamping its storage strategy to include an $8 billion investment in more than 30 products and services to help customers manage their storage needs. This is one reason that IBM has snapped up eight storage start-ups–including its widely reviewed  XIV acquisition. This is also why, despite the generally disheartening economic news out there, the storage sector is still going strong.

As we mention so often in this blog, this kind of meteoric growth in unstructured data must be met by significant optimization strategies. While many storage vendors are benefiting economically in the short term on this, the bigger picture is quite different. The real challenge in the future will be manage the data–particularly online data–in such a way that data centers do not continue to grow in a way that’s out of control.

Image: Technocycle

Gates spills MS cloud computing strategy

Good old billg has something to say in his “exit interview” about storage in the cloud in this week’s PC Magazine. In essence, his view is that computing and storage will move to the cloud at different rates, and that storage is the more logical thing to move first. Your local storage (presumably on your Windows PC in Mr. Gates’ worldview) will be a cache of a subset of the master data held in the cloud.

Moving data in to the cloud makes a lot of sense, as it makes that data available to computers everywhere, and it also centralizes management of data for backups, geo-replication, and hardware refresh in places where economies of scale can take place that an average user or company could not manage or afford.

I agree with Bill on a couple of these points.  I think storage does move to the cloud faster than compute, both because people already understand the idea of storage networks, and that their storage is on a network somewhere –in the cloud, that network pipe is just a bit longer. Compute is something people are used to having close at hand, either on their desktop or in their own data center.

For storage in the cloud to really take off, though, I think storage optimization is an absolutely key ingredient. As regular readers of this blog know, when I say storage optimization I mean a combination of content-aware technologies to drastically reduce the size of data (content aware compression, subfile deduplication, and logical compaction to name a few). There are two places where storage optimization makes sense for storage in the cloud to take off. One is at the customer end of the pipe – if you can shrink your data before you send it to the cloud, you’ll use less bandwidth getting it there and getting it back, and since most “clouds” charge you per Gigabyte, you’ll pay less too.   

The other place is in the data center of the storage cloud provider. That’s going to be a very competitive marketplace, and the cloud vendors that can charge you the least amount per Gigabyte – to store, to transfer, to replicate –are going to have the competitive advantage. So the cloud vendors that do the best job of integrating storage optimization in to the cloud in a transparent way will have the edge. And the cloud is a great place to get that edge.   

Think of deduplication, for example. If you deduplicate songs just in your own house, well you may only have one copy of each song. Why would you have ten copies of a Britney Spears song? (I might ask why you would have any at all … but we’ll leave that for another time.) However, if 5 million people store their data at a cloud storage provider, how many copies of that hit song might end up there? Does the cloud provider need to store 5 million copies of the same thing? No. If they do, they are being very inefficient. A song is a simple example, but even with enterprise data, the more data you have, the more likely it is that you’ll find patterns, correlations, duplicates, or data relationships that can be exploited for better compression. So the cloud offers an opportunity for efficiency that don’t exist at each little pool of local storage on your hard drive today.

To me, storage optimization and the move of storage to the cloud make a perfect match.

Image: PC Magazine

This year’s big storage story

My company Ocarina Networks is one of Network World’s “10 Storage Companies to Watch.” We’ve been given this honor before, by Byte and Switch in 2007. This is not just to toot our horn, but to mention that more and more, we’re confirming our hunch that primary storage is where there is an immense need for new and innovative optimization solutions.

As I mentioned in an earlier post, the kind of data that’s driving much of today’s storage growth–files–demands a file-aware solution for shrinking them down. It’s clear to me that there is an emerging set of opportunities in this space, and we are only beginning to see where this will lead.

The impending storage crunch

No one can miss the fact that data storage is spiraling upward at a terrifying rate. Joerg Hallbauer puts it on Dell’s Future of Storage blog hit the nail on the head with his post: “We are running out of places to put things.”

Citing data collected by IDC, Hallbauer concludes that in a mere three years, we there will be 1400 exabytes sitting on disk. Currently, according to the study, there are 281 exabytes of data being stored, and the CAGR rate is 70 percent. Much of this data is on laptops, home computers or servers under your desk today, but as Joerg correctly notes, there’s no question its migrating quickly to the cloud. Huge data centers will end up holding most of this data, and disk drives are not growing fast enough to deal with it anymore.

So, where do we go from here? Well, if the traditional answer was, wait for bigger drives so I can put more stuff on a disk, the other logical thing to do is to say, how can I put a lot more stuff on disks that I already have?  The answer is advanced storage optimization. The first simple storage optimization solutions are out there today – single instancing, deduplication, and compression. But the area of storage optimization is really just taking off, and much more sophisticated approaches are emerging that will allow a disk – whatever its physical size – to store 10, 20, or 100 times more data than it does today.  

What’s more, the move to large data centers providing huge cloud storage services will make this more efficient, because storage optimization is all about finding redundant information and figuring out how to store it more efficiently. So the larger the data set, the more likely you will see big wins from next generation storage optimization.

This also naturally leads to more tiering. Where today you have fast disks (Fibre Channel or SAS) and slow disks (SATA) making up the tiers, it’s much more likely in the future that the fast tiers will be solid state storage of some sort (SSD and Flash, as Joerg points out) and the massive tiers that hold the bulk of all these Exabytes will be the largest possible disks integrated in to systems that have very efficient storage optimization built in.

Image credit: Orange Photography blog archives

Capacity-Optimized Storage: The Emergence of the O Tier

 

Everyone is talking about the explosive growth of storage, but all growth is not the same. In fact, unstructured data (files) are growing much faster than structured data (databases), and capacity-optimized storage for files is growing much faster than traditional filer-based storage. This is driving some key developments in storage technology, as storage offerings emerge that are designed specifically for where the growth is.

       Traditionally, the difference between performance-optimized storage and capacity-optimized storage was just whether a storage system shipped with Fibre Channel drives or SATA drives, and maybe how much cache was in the storage controller. Now, the differences between Performance-Optimized and Capacity-Optimized storage are becoming much bigger, with advances in both tiers taking them in different directions and further away from each other.

       The “P Tier”–long dominated by NetApp and EMC–is seeing lots of advances, include bigger caches, solid state disk, and more fault tolerance. It’s where data gets created, and there is a huge focus on never losing data that has just been created. The “P” in this tier doesn’t just represent “Performance,” but also “Protection.” Performance is measured in SPEC sfs and IOPS, and protection features include mirroring, RAID levels, synchronous replication to DR sites, and snapshots every time a file is modified or deleted. However, the P Tier is very costly per Terabyte because of the premium technology required to provide all those protection mechanisms while providing stellar low latency performance at the same time.

       Enter the “O Tier”–or what IDC calls capacity-optimized storage. This is no longer just a NetApp or EMC filer with SATA drives instead of Fibre . True O Tier offerings–which are starting to come out from the major vendors–have several major architectural differences. EMC’s Hulk, IBM’s XIV, and the HP’s exciting ExDS Extreme Storage are all based on scale-out architectures. You buy “bricks” of capacity, at near commodity prices, and you can scale out these systems by just adding more bricks.  Almost all of the scale-out “O Tier” offerings are based on clustered or distributed file systems.  These architectures are drastically cheaper than P Tier storage, even P Tier offerings with SATA disks.

       What’s more, the O Tier is becoming clearer in what its metrics are. Data may be created in the P Tier, but it moves for long-term storage to the O Tier. That means there is less focus on extravagant protection measures. Data that makes it to the O Tier has already been backed up, snapped, replicated, and protected many times in the P Tier. On the O Tier, the key metrics are Cost per Terabyte, Terabytes per Admin, and Watts per Terabyte over its lifecycle.

       The O Tier is evolving to solve different storage problems than the legacy P Tier and because of that the O Tier is developing its own new features for capacity optimization. The most important of these new features is integrated data reduction. That can take the form of block-level dedupe, next generation compression, or content-aware optimization. There are several technologies coming out aiming to get 5X, 10X or 20X data reduction for online storage in the O Tier. Expect these technologies to be embedded as integrated elements in leading O Tier storage offerings.  Examples would include Data Domain moving from being a storage solution for backups to offering nearline storage with dedupe, or the several storage vendors who are integrating my company Ocarina Networks’ storage optimization solution in to their O Tier storage offerings.

       Anyone who is tracking trends in storage need to start paying attention to differentiating these tiers not by just what disks are in a given filer, but whether they are really P Tier filers or O Tier filers, with true Performance and Protection in the P Tier, or true Capacity-Optimization in the O Tier.

       While the traditional NAS leaders, EMC and NetApp, will certainly come out with O Tier offerings, the emergence of a new tier with different characteristics creates a new market opportunity for other major players to become the new  leaders in the O Tier. Look for HP, in particular, as well as IBM,  Ibrix, Isilon, and Blue Arc to be making major pushes in the O Tier this year and especially in ‘09.

In Startup City’s spotlight

I was interviewed on video by John Foley for InformationWeek’s Startup City a month or so ago, and have just discovered that the video is now up on the site. If you haven’t already, I encourage you to explore this blog. Foley does a great job of covering the vast and growing landscape of IT startups. Enjoy.

What’s Hot in Storage — Spending Less

Byte & Switch has once again released its “Top 10 Storage Startups to Watch” for 2008, and it’s definitely worth a read. My company Ocarina Networks was on that same list last year, and so I can say with confidence that they got it right at least once before. 

As reflected in this year’s list, data reduction technologies continue to be hot. Makes sense in a down economy that anything that increases capacity will continue to get budget dollars. As we’re finding, dollars for stuff like Ocarina is already there in every data center’s budget – it’s just listed as disk expense. We’re not only ahead of our revenue goals for our storage optimization product launched in April, but we’re having to triple the size of our sales force to keep up with demand. 

If you have planned to buy 100 TB of disk, and can spend half as much for an optimization solution that shrinks your files that means you don’t have to buy any disk at all. A win all the way around. While Ocarina started out with wins in large web sites – where the fastest year-to-year storage growth is taking place – we’re now seeing installs in life sciences, energy, movie studios, and finance.  

The chief takeaway from what I’ve seen: some nice-to-have new technologies may be facing a tough summer with an economic downturn, but data reduction scores high on both saving money and green IT, and is likely to stay strong, or maybe even move up in priority, during a down cycle in storage spending.