The last couple of years have been painful, to say the least. Some call them unprecedented, financially, but I do believe that is pushing the descriptor a bit far, since there have been plenty of instances where business pretty much en-masse questioned the amount that IT returns for their investment and cut budgets, so the feel of this recession is not much different than what we’ve felt before, it’s just by necessity. The funny bit of this is that everyone seems to agree that IT spending still went up in 2009, just by a massively reduced amount. Since the pinch is definitely out there, one can only assume that a 1.6% (or so, depending upon your source) increase in spending was not enough to cover increases in maintenance costs and new purchases.

The impact on IT is pretty straight-forward, at least in my mind. Major IT projects were delayed or canceled based on tough funding decisions, and those projects ran the gamut from development to networking to  outsourcing services. Some of these projects were not critical, and some were cut when the business they were going to support was curtailed, but some are “hidden gems” that will in the long run cost the business more than it is saving today.

But belt tightening went on across the entire organization, so IT is left to struggle with its portion of the pie, hoping that the shortfalls (necessary project wise) will be made up in the future. The only bright spot from a budgeting perspective is that new programs and products were cut before IT, since IT is corporate wide and viewed (mostly) strategically.

One area that suffered heavily was storage purchases. Looking at the dollar amount and the growth rate, this is no surprise at all. While the growth rate strongly implies the need for more storage, it also strongly advises finding new ways to deal with storage growth. I’ve written about a few of them in my blog over the last few months, but a thorough examination of the ways available to you today to clean up storage usage is in order.

Don’t believe me? The Recent InformationWeek Analytics Research Report The State of Storage 2010 (shows a hefty 47% of respondents are struggling with the amount of storage demanded just by their mission critical applications. That’s a big number. This percentage topped out 2010 concerns for storage strategists, taking the number one spot away from 2009’s “Data Loss/Data Security” category (which came in at 44% this year).

While that doesn’t exactly leave a bleak picture for the near future – the implication being that 53% of us aren’t struggling as badly with storage resources – it is certainly not rosy.

There are a variety of tools out there to help you get through the crunch, and the technology and process list is growing. Some of them we’ve discussed before, some we haven’t, but let’s make a list. If I miss  anything, feel free to add on in the comments. And I’ve linked all but a couple that are all over the news.

1. Tiered Storage. If you’re not there yet, go there. The cost of lower end storage is less even if you buy it from the same big name, and it holds less access-speed sensitive data just fine.
2. Compression. Consider it. Even though your data must be read by the compression engine, the amount of space you can recoup is huge. Not for everyone by a long shot though. There’s a short introductory piece here about it.
3. At-rest Deduplication. I list it separately because it’s the popular twin in the compression/dedupe category right now. Again, requires the dedupe engine rehydrate your data on read, but worth considering for the massive space savings.
4. Virtualization. This is one of the areas F5 plays in with ARX. If you abstract your disk, you can increase utilization on the back-end without disrupting users. Increased utilization means less outlay for new disk. IMO, NAS virtualization is much more reasonable/viable than SAN, simply because you can get your data back relatively painlessly if the virtualization engine fails. Coworker Renny Shen wrote a good overview of virtualization and storage this month that’s worth reviewing for background.
5. Thin Provisioning. I’ve warned about the potential for inadvertent abuse that thin provisioning harbors, but that doesn’t make it a useless technology, just one to use judiciously.
6. Remote Data Centers. If you have storage in many data centers and the storage is not being roughly equally utilized, time to start looking at ways to equalize data distribution. We play in this space also with our WOM product, since it speeds and secures replication and other data transfers across the WAN to a remote datacenter.
7. Cloud Storage. It is worth the time to investigate cloud storage, as I’ve chattered about frequently of late. Particularly if that storage is fronted by a cloud gateway.
8. Archival Storage. Remember the poor, lost, lonely archive? The place where outdated or underutilized data was supposed to go to die? Well it’s time to bring it back. Want to reduce storage costs? Reduce storage needs. I discussed one way to deal with this possibility a few weeks ago, but there are others. The point being that we need to start throwing stuff away. Since we can’t chuck it into the circular file, data that is unlikely to be needed again soon will have to go onto long-term storage and stay there forever.

Much like Dilbert’s Top Secret Management tips, many of these ideas work best when used in tandem. Unlike Dilbert, none of these ideas is likely to get you fired. Put archival storage with Cloud storage and you get a place to stash data limited by only your operations budget. Put an ARX or other Directory Virtualization product in front of that cloud storage gateway and you have disk that is indistinguishable to the end user from local NAS devices, but is actually shoving stuff off of your expensive disk and into the cloud. Add tiered storage (automatically provided by ARX, maybe by other vendors), and your data is moving to the right place at the right time, but users don’t have to care. Utilize WOM to open WAN tunnels to a remote datacenter and optimize file transfers between the two and you’ve got resource leveling… It all adds up.

Hopefully you will soon be flush with new disk, but none of the above ideas would be precluded by adding new disk (though the extended portion of file information discussed in my Meyers’-Briggs blog might be too resource intensive to pursue if disk were plentiful). They are all additive, meaning new disk would simply last longer, which is certainly worthy of consideration. They vary in costs, some requiring little investment given your existing infrastructure, some requiring an investment that I would argue is minimal in comparison to buying new disk every year.

And should those rumbling about a “double-dip” recession be correct, it is certainly a good idea to have these tools in your stable before you are once again trying to do more with less. Here’s hoping they’re not right, but contingency planning dictates that, at least to some extent, we should act like they’re visionaries, just in case.

Fun random fact from that InformationWeek report? 58% of you are managing more than one terabyte of data and less than 100 terabytes. It boggles the mind. We are not accessing 100 terabytes of data in a year, so a ton of that is likely wasted space. Add in the 17% managing 100-500 Terabytes, and 75% of you have a whole lot of data. Ever wonder what it is? Ever ponder that there is stuff on your network no one has seen in a year or more? Creepy fun.