Tony's blog

D2D Backup + Active Archive

In my travels I have discussed the value of storage tiering and it is amazing how few companies have actually implemented this. Part of the challenge is identifying what to move and when. One way to easily address this is to just consider dormant data. Over the last few years I have presented to hundreds of IT professionals and the vast majority believe that more than 70% of their data is dormant 90 days after its creation. If this is the case, then focusing on dormant data after a 90 day period seems like a good place to start - move this data from your primary storage systems to a lower tier.

The next challenge with storage tiering is identifying what solution you are going to use to move the dormant data from primary to archival storage. This process needs to be transparent to the users and applications. And it would be nice if it was easy for IT professionals to implement and there was a high level of performance. In the NAS world you can use file data movers or virtualization solutions. Data Domain partners with F5 and its Acopia solution.

The third and final macro issue is what platform should be used for the active archival storage. Data Domain is typically seen as a disk-to-disk backup solution but it should also be considered as an active archive as well. Remember that the Data Domain solution is a NAS system and therefore a mount point can be easily created and it can be used to store files via NFS and CIFS protocols. One of the main reasons to do this is to take advantage of its dedupe capabilities. Since this is not a backup data - the dedupe rates will not be as high but you should easily get a 2-to-1 and maybe as high as a 5-to-1 ratio. What makes this even more compelling and valuable is using Data Domain for both D2D backup and as an active archive. It is this combination that presents a powerful value proposition. Consolidating these functions into a single Data Domain appliance further reduces cost and complexity. In addition to the direct CAPEX savings there is also the reduction of power, cooling and floor space.

There are companies that have saved significant capital costs by implemented tiered storage environments - even without dedupe. Imagine adding dedupe to the equation. Further, leveraging the same applicance that is being used for D2D backup and giving you 20-to-1 data dedupe. Not many people talk about providing a consolidated D2D backup and active archive solution, but it really does make a ton of sense if that single solution can support both with dedupe.

Voice of the Customer: Collection Replication, Disaster Recovery and Tape Elimination

The following is a discussion I had with a Data Domain customer - Rob Carlisi, Systems Engineer for Orange County Public Schools (OCPS). 

 

You can also listen to the podcast - Download Interview Orange County Public Schools

 

A couple of things to point out:

• OCPS has eliminated using tapes completely. 

• They are using Data Domain Collection Replication for their off-site disaster recovery.

Tony Asaro: Rob, welcome and I appreciate you participating this discussion.  Please give us a background on your company.

 

Rob Carlisi: I work at Orange County Public Schools in Orlando, Florida. We’re the 11^thlargest school district in the country. The amount of online storage that we have is a little over 100 terabytes and the amount of backup data we have on Data Domain is about 450 terabytes.

 

Tony Asaro: And what is your role with the company, Rob?

 

Rob Carlisi: I’m a systems engineer and I’ve been here for a little over 6 years. Our group is primarily responsible for anything that happens inside the data center that’s not network related. I’m responsible for the backup/restore environment, SAN and fibre infrastructure as well as installing and maintaining the servers themselves.

 

Tony Asaro: Start out by giving us some examples of how Data Domain has provided real value to your company.

 

Rob Carlisi: Yeah, I’ve quite a few examples. Recoveries with Data Domain are much faster than our previous environment. The restores start as soon as you click a button instead of waiting for tapes to mount or waiting for tape drives to become available. The footprint of our solution went from around 6 to 8 feet of floor space down to a single rack with Data Domain. Data Domain gave us the ability also to replicate all of our backups to our DR site so we have another set of Data Domain boxes sitting at our offsite location that we replicate to every day. We also use RMAN with Oracle to backup all of our databases directly to Data Domain. Since we’ve been using that method, we’ve seen up to a 60x compression rates on those backups. And also over the next few months we’re going to implement VRanger which will backup all of our VM’s over to Data Domain.

 

Tony Asaro: Rob, tell us how Data Domain has improved the overall economic effectiveness within your IT operations?

 

Rob Carlisi: Data Domain has enabled us to save exponentially actually on the amount of tape we would have had to purchase if it wasn’t Data Domain and de-duplication. We also went from a 10 to 15 day retention policy on all of our backups to a 30 to 40 day retention.

 

Tony Asaro: How has implementing Data Domain changed your usage of tape now and in the future?

 

Rob Carlisi: Well, tape no longer exists in our environment and we’re confident that it’s going to stay that way.

 

Tony Asaro: Rob, tell us about how you’ve leveraged Data Domain for offsite Disaster Recovery.

 

Rob Carlisi: Before we had implemented Data Domain, we weren’t replicating any of our data off site. When we decided to start replicating with Data Domain, we did it on a collection basis so anything that touches the primary box, gets replicated to the destination box on our off site location. We set it up the first day to replicate and it’s pretty much been hands off since then. We’ve had the remote site down. When we bring it back up, it just makes that connection automatically and starts replicating again. We’ve never had any lag and we’ve never had a problem talking to the remote site.

 

This solution actually got all of our backups from our entire environment off site. We’re able to restore at a DR site back to full throttle.

 

Tony Asaro: Data Domain’s data dedupe actually enabled your disaster recovery strategy because it reduces the bandwidth costs.

 

Rob Carlisi: Actually that’s true. After it does its de-duplication and compression and clean up, you’re not actually transferring that much data across the line compared to what it actually backs up every night. It saves on money as well as time to replicate because when you’re not moving that much data across the LAN or the WAN it doesn’t take that long to do you’re not too far behind your production environment.

 

Tony Asaro: Why didn’t you use primary storage replication versus Data Domain to do this?

 

Rob Carlisi: Well, one was the cost and we also liked an appliance environment. We found that it was a lot easier to expand a Data Domain environment rather that a primary storage environment and we just like the flexibility that Data Domain gave us. Also, the primary reason was the de-duplication technology that they had was far superior than anything else.

The Changing Landscape of Disaster Recovery

Disaster recovery (DR) means different things to different people.  For many IT professionals DR is associated with tape recovery and/or remote replication.    Recovery data from tape is fraught with challenges including rapid recovery and reliability.  Remote replication within primary storage is often complicated, expensive and inefficient requiring volumes to be restored versus specific files or data sets.

The combination of disk-to-disk (D2D) backup, data dedupe and replication offers a valuable approach to DR combining the granularity of recovering from a backup repository and the ability to leverage remote replication to store copies of data at an offsite location in the event of of site disasters and for offsite vaulting for long term retention.  For the vast majority of data this is a smart approach that serves the widest range of requirements for customers.  There will always be the need for primary remote replication but only for the corner cases in terms of who uses it and what it is used for.  However, I contend that D2D backup + data dedupe + remote replication has a much broader audience. 

Data Domain just announced an update to it's Replicator software.  I remember when Data Domain first came out with Replicator and speaking with customers that literally felt liberated by leveraging dedupe over wide area networks.  Dedupe enabled them to implement a DR strategy above and beyond what they had prior and still fit within their budgets. Five years later the solution is more mature, advanced and field proven.  Additionally, the use cases and value proposition have advanced and are proven.  Data Domain providing a combination of D2D backup + data dedupe + remote replication has impacted the data center in a number of important ways:

• Offers a cost effective and efficient DR solution that is otherwise impractical for many customers.

• Results in cost reductions include bandwidth, primary storage investment, power, cooling and floor space.
• This approach is "easy" versus perceived complications with using other DR methods (e.g. primary storage replication).
• Offers a more granular and rapid method of data recovery improving SLAs. 
• Makes it possible for customers to actually eliminate or greatly diminish the use of tape and tape libraries.  This is an important point and we are seeing momentum for tape obsolescence with a growing number of customers. 
• Multi-site replication provides a powerful and valuable solution for Enterprises with a large number of remote sites - this is enabling a new constituency to reap the rewards of this solution. Data Domain's recent expansion of 90 remote sites to a single DD690 is significant and makes it even more useful to large Enterprises. I will add, regardless of whether you have two or 90 remote sites fan into a single DD appliance at the central site is powerful especially when coupled with global dedupe. 

I have spoken to companies that did not implement remote replication because of cost - even though it was considered every year for five, eight and ten years.  And there are other companies that only replicate 10% to 20% of their data and use backup to protect the rest - they just could not rationalize the expense.  However, D2D backup + dedupe vastly changes the economics of remote replication and DR.  The landscape of DR is changing - enabled by D2D backup with dedupe - making it practical and valuable for the mainstream market.  Remote replication will drive the adoption of D2D backup with dedupe and D2D backup with dedupe will drive the adoption or remote replication.  This is a killer app on top of a killer app. 

Data Domain - Built for Inline Dedupe

It is important to consider Data Domain's architectural approach optimized for inline data
dedupe. 

Data dedupe uses algorithms to analyze and make changes to data with the goal of reducing the amount of capacity to store it. Addiitonally, a core tenant of data dedupe is to do no harm - all of the data should be accessible and unaltered from the user and application perspective.  These characteristcs make inline data dedupe CPU intensive.  And for Data Domain customers - that is a very good thing.

Traditional storage system performance is typically disk-limited with disk drives as the biggest bottleneck.  Many storage systems offset this by providing caching algorithms and wide stripping across a large number of disk drives.  Wide stripping is valuable but again relies on disk performance - which is the slowest aspect of any storage system.  The concept behind wide stripping is "many hands makes light work".  But those "hands" or disk drives don't come for free - additional disk capacity results in high costs to the customers on a number of levels. 

There is brilliance in the CPU versus disk-driven approach for a data dedupe storage system.  There is a wide gap between CPU and disk performance and this continues to increase over time.  Additionally, the cost of CPU performance is far lower than disk drives.  And since data dedupe - especially when it is being performed inline - is CPU-intensive - Data Domain has a sustainable architectural advantage. 

Traditional storage systems were built to support transactional data and not to be dedupe engines.  That is why you see very few inline data dedupe storage systems in the market.  Most storage system vendors have a "hammer" and they want to turn everything into a "nail". 

Data Domain's architecture enables their ability to provide inline data dedupe with high performance levels that will just keep getting better as CPU performance keeps getting better.  This is critically important since the growth of primary data increases and all of it has to be backed up without impacting operations or backup windows. 

Data Domain isn't a general purpose storage system with data dedupe algorithms added as a feature - it is a purpose-built solution that was designed and optimized for inline data dedupe that has created an entirely new and valuable category in IT. 

Customer Discussion: Bank Midwest

I had another great discussion with a Data Domain customer- Bryan Wilken, VP of IT for Bank Midwest:

Tony: Can you give us any examples of how Data Domain provided real value to your company?

Bryan Wilken:  e have completely stopped using tapes in all but one of our systems, and we’ll have that on the Data Domain in the next few weeks here. The bank now has a more robust and fault-tolerant back-up system. It runs automatically every night and it doesn’t require the need of employees to change tapes and take them home. We’re also seeing a 23-to-1 compression ratio, or de-duplication. And restores have been handled quickly and effectively by the IT department. We no longer have to wait to have tapes either brought back in or reloaded and reread before we can restore data.

Tony:  Can you tell us how Data Domain has improved the overall economic effectiveness within your IT operations?

Bryan Wilken: The Data Domain disk-to-disk system is saving Bank Midwest in tapes and drives alone around $13,000 a year. Plus we have the security risk that’s been eliminated without having tapes that are leaving the bank’s premises. As I mentioned, we have moved to virtualization of our servers. We hadn’t first planned on this, but we are now backing up our VMware servers nightly to our Data Domain and then replicating them to our DR site. Another added layer of protection in case of a disaster.

Tony: Would you recommend using Data Domain to other IT professionals?  And tell us why.

Bryan Wilken: Yes, I would. What I liked about Data Domain as we’ve gone through installing this product, is that the solution worked just as the vendor claimed it would.  We’re also getting the compression rates that they claim and it was just an easy set-up, and it’s worked from the day it was installed. That’s the kind of products I like to purchase for Bank Midwest.

Bryan validates the Data Domain value proposition with 23-to-1 data dedupe, fast and reliable recoveries, reduced management overhead of tapes, reduced security risks, enabled disaster recovery - and fundamentally it was easy to set up and put into production. 

I think it is interesting to note that this is yet another customer that is completely eliminating tape. 

Frank Talk: Data Domain CEO Discussing 2009

I had a discussion with Frank Slootman, CEO of Data Domain. We talked about Data Domain’s outlook for 2009, what is working for them and the trend of more and more customers moving away from tape. 

Tony:  Frank, tell us about Data Domain’s business outlook in 2009.

Frank: We gave guidance to the capital markets – a lot of people refuse to give guidance to the capital markets. For us – it was important to give an indication of what our operating assumptions both for Q1 as well as for all of 2009. Take a look at that guidance. 

Obviously we are getting into the laws of big numbers. Its heavier lifting growing from zero to one million than it is from 100 million to 200 and 200 to 500. On the other hand we are guiding to significant growth even in 2009 – where the overall storage market is going to compress probably by 5-10%. 

We are planning on growth and we are hiring. We have the advantage of running on a secular trend – as opposed to a macro trend – that’s a generational shift from legacy tape infrastructure to one that is disk and network based. We think we are in the first or second inning of this ball game – very low penetration. We will have head winds like everybody else but we think we are in a good category and still considered by customers even in this economy. 

Tony: That is an important point – you believe there will be growth with Data Domain while the rest of the storage market is planning for the opposite. And keep in mind that 2007 and 2008 weren’t great economies either. Nothing compared to 2009 but the last few years haven’t been anything to write home about and you still have seen amazing growth. 

Frank: Yes and the reason is we sell pretty hard on ROI and payback. In other words, at the end of the year the customer should be better off with this new infrastructure on a cash basis – you should have more money left at the end of the year for having gone with Data Domain as opposed to just staying the status quo. Net new incremental spend is only justifiable if on a cash basis I am better off. 

Typically where people are making their investment back they are not spending as much money on tape media, they are not spending as much money on record management services – you start adding that up it starts paying you back pretty quick. When people have investments in legacy tape the maintenance is really high and it gets higher every year. You very quickly find a cross over point that people are saying it makes sense to go with Data Domain. 

Tony: IT professionals are telling me – no more investment in tape. 

Frank: I call that drawing a line in the sand. You get a mandate from above that says – it isn’t a little bit here or there – they are going to make this change.

Tony: There are also other benefits - like not putting tapes on trucks. 

Frank: Right – no hands. 

Tony: Yes and there is a security and administrative aspect to this.  But one other thing that is harder to quantify is the recovery issue.  This might be harder for them to sell their management but for the IT guy it is a huge win – recovery is fast and reliable.

Frank: And they can have deeper retention. They can have far more data online than they could have otherwise. Individual file restores and you can do full system restores much faster than they could before.

Tony: Data Domain has been saying this for years but now the customers are saying it.

Frank: That is right. Recovery speed and reliability isn’t what CFOs embrace because they are really looking at the measurable economics. They want to see how Data Domain pays them back within a certain amount of time. The other stuff is all gravy.

Tony: I am surprised at how many people I am seeing moving away from tape. I was skeptical it would ever happen. 

Frank: Look at the numbers. IBM was down on tape by 31%, Sun was down 15% and Quantum was down 25%. The decline is accelerating. For a while they were going sideways but now they are dropping off a cliff. 

Tony: I am also seeing Data Domain becoming the de-facto standard.

Frank: We are not going to get too big for our britches. We always want to feel like we are the underdog with these giants around us. We are not going to get too cocky. 

Tony: Right, healthy paranoia – according to Andy Grove.

Frank: Yes and you need it. I find a lot of products in the market to be inferior and not doing customers any good. People often buy because they have relationships and they trust their vendor to deliver a good product but often this is not the case. It is often that it’s just downright bad. 

Tony: What’s next for Data Domain?

Frank: We are going to keep on trucking. We will continue to build bigger, faster and more broadly capable systems. We will be a very broad class of storage that is highly optimized. That’s our game.  Our dedupe is a bone crushing technology that is enabled in a fully realized solution. 

Tony: My view is that the way VMware has virtualized servers - Data Domain has virtualized data. 

Frank: That is exactly right. 

Panel on Storage Optimization

I just participated in a panel on storage optimization for Pacific Crest - an interesting firm that focuses on technology research for their investors. I was one of the panel members alone with Stephen Foskett, an IT consultant that works with end users and Mark H., an IT manager for plastics manufacturing company with 3,000 employees and 40 locations. 

Both Stephen and I talk to a lot of IT professionals and it was good having Mark H. on the line for his direct input on his IT operations.

Here were some key takeaways:

1. Both Stephen and I are seeing many of the same things - IT budgets haven't been dramatically cut but spending has slowed down and some projects have been put on hold for now.  However, high priority projects are moving forward since business can't come to a halt. Mark verified this within his IT organization. In the last 30 days I've personally spoken to about two dozen IT pros - and that has been the universal position among all of them.

2. Primary storage and backup are seen as mandatory. In many cases IT professionals are taking a look at what they have on the floor and seeing how they can better utilize existing IT assets. 

3. End users are more open to moving away from the status quo and leveraging innovation if the economic value is compelling and quantifiable. For example, Mark H. actually is eliminating tape and instead using Data Domain for backup. This is becoming more of a common theme among the Data Domain customers that I speak with. There is an opportunity to make big changes in the data center based on economics - especially in 2009. This isn't just true in backup but across the board.

4. Stephen and I both agreed that what we are seeing is continued storage usage - which includes optimization of what we already have and potentially buying new capacity.  Mark pointed out that storage was core to their business and was not being cut or delayed. 

5. Another area that is core to IT operations is backup and that investments are being made in backup hardware - with data deduplication being key to data center optimization. 

The Leader's Dilemma

There is an interesting dynamic in the business world - when you are not the leader in a category or segment you go out of your way to make some leadership claim. And when you are the leader making strong statements about this position is often perceived as boasting. 
 
Data Domain is the leader in dedupe disk-to-disk (D2D) backup - they just don't say it often or loud enough. Beyond the marketing impact of this reality there is great value to the customer. Data Domain has the largest number of installs with years of field experience. In the IT infrastructure world quantity is quality - because the market will throw anything and everything at you.  The more your product is in production environments the higher quality it will be - and there will be fewer unknown variables because you've been exposed to more and more of them. Additionally, Data Domain has built an organization and ecosystem that understands the solutions and issues for D2D backup - it is their core competency. 

Steve Duplessie, senior analyst for the Enterprise Strategy Group blogged about IT leadership recently,  Steve states "The Blue Ocean concept is absolutely true, yet 90% of all companies will spend far too much time and effort attempting to compete in a market with an established (perceived or real) leader..."
 
Data Domain went after a Blue Ocean opportunity and has become the leader - and others are following in order to compete.  According to Steve and his years of observing IT markets - it is very unlikely that anyone will dislodge the leader.  This isn't his opinion but an analysis based on historical trends.
 
EMC is the one big player going after this market in earnest with three different products (which will confuse the market and themselves) and they will make progress because they are a giant.  But I submit that they will continue to face the issues that Duplessie points out in his blog - the leader creates the Blue Ocean (Data Domain) and the number two guy started the race too late and will never catch up. 
 
The number two guy tends to throw mud and go on the attack. An EMC blogger did this recently comparing Data Domain and Avamar. The blog is overly biased trying to bash Data Domain and shine a bright light on Avamar. But look at the comments posted and you will see end users attacking the content of the blog because it is - well - ridiculous. 
 
I just had a conversation with a big EMC customer that went with Data Domain after giving careful consideration to everything that EMC had to offer. Even though EMC is a strategic partner they didn't want to go with Avamar or CDL:

"We didn't believe in the VTL paradigm. We were really looking for stuff that was native NAS, either CIFS or NFS protocol, so to keep it "simple stupid" kind of mentality.  And there's a lot of vendors that have since integrated even with Data Domain with proprietary APIs to get a better performance out of the Data Domain. But the tradeoff there is kind of performance versus openness.  So we looked at Avamar with EMC. They had just acquired them about the time we were looking at Data Domain. We looked at a few others, but they really weren't big name players.  We searched out there and we went to people that we typically do business with. The real issue of the Avamar solution was that it was a complete replacement rather than an augmentation of the existing environment.  

Data Domain just stepped in and instead of tape, now you're writing to a disk-based target.  And it was fairly easy to set up, because it's CIFS and NFS. Avamar was 'rip and replace' both the software and the hardware. They're a combined platform. So [Avamar] was a much more unilateral replacement if we wanted to go there.  We thought Data Domain was less invasive in the environment at the time."

This is common sense thinking that is the majority view. That doesn't mean that everyone will go
down this path - there are always exceptions.  
 
From an end users perspective Data Domain is the leading solution and the product works extremely well, there are thousands and thousands of happy customers, it is easy to install and the value is quantifiable. What other IT products can you say that about?  
 
Another big false negative that competitors will say about Data Domain is that it is a one trick pony.  So was EMC for years with the Symmetrix and that strategy made them billions of dollars.  Eventually they added more products to their portfolio but it took years. NetApp is still fundamentally a single product company and that has done extremely well for them. And even though EMC claims they are the number one NAS vendor (nod, nod, wink, wink) we all know that the real leader is NetApp. I predict they will make the same claim against Data Domain going forward. And Data Domain, just like NetApp, has an extremely healthy business with impressive growth rates and high margin products - that is clear to measure. 
 
I have a very different view than the naysayers - Data Domain has the leading product with no clear and strong competitors in a market that has endless opportunity. Pretty much every company has budget for backup devices and it is core to their operations. This is a multiple billion dollar market and Data Domain is the leader. Additionally, you will see more and more customers using the same Data Domain appliances for archival storage. This is a powerful value proposition - using the same system for backup and archive and the combined dedupe ratios are compelling and hard to beat. 
 
The leader's dilemma is that every competitor will come gunning for you - but that is okay - because they invariably miss. 

Pondering The Future of Tape

I once thought the notion of tape as the primary backup target being replaced by disk in the data center was not likely for a long time to come. However, data deduplication has enabled this to occur in many data centers over the last few years. There are literally thousands of companies that use disk-based systems as their primary backup solution and tape is secondary. An interesting dynamic occurs when you implement D2D backup - you begin to reduce the importance of tape as a part of the backup process. In many cases tape is just being used but for weekly or only monthly fulls. Additionally - tape is used for off-site storage for long term vaulting. But even this process is being replaced by leveraging remote replication of backups from the D2D backup device to the off-site vaults (e.g. Iron Mountain). 

I once thought the notion of tape being reduced and eventually eliminated in the data center was going to take a very long time to occur. But I see it happening at an accelerated rate - in companies of all sizes. There are really big companies that are reducing, big companies that are planning on elimination and many of the SMBs either have or are well on their way to eliminating tape completely. The market indicators support this with IBM tape 2008 sales down 31% and Quantum tape sales down 25% compared to 2007. 

I have actually been conservative about the decline of tape, but there is a trend occurring and it is happening at a rapid pace. Tape will exist for the foreseeable future but it will become more and more marginalized - being ultimately relegated to a small part of the market. Certain members of the IT community hold onto things for a very long time - and that will be true of tape - but there will be numerous companies that will move away from tape completely - and there will be new companies that will never use it. Someday soon it will be like when your kids ask you what records and cassettes were.  The next generation of business may never know what it was like to backup to tape. 

Dedupe Predictions for 2009

Here are my predictions for 2009 relating specifically to data deduplication:

1. Disk-to-disk (D2D) backup with dedupe continues to proliferate, becoming much more mainstream in 2009. The perceived risk of implementing D2D backup with dedupe is all but gone and the value is glaringly clear. Additionally, the bad economy makes the value proposition that much more compelling. 

2. End users utilize dedupe as a requisite part of nearline and active storage archives. Moving dormant data off of Tier One storage needs to go up the priority list. The question is what do you move it to?  Standards-based NFS and CIFS cost effective storage with dedupe will be embraced by a number of customers. This should be considered and implemented by end user companies of all sizes. 

3. D2D backup and archiving will be a big Cloud Computing application. SMB customers with no secondary site can use Cloud Computing services for off-site disaster recovery. Additionally, low cost cloud storage can be used as an offsite archive. But you need to get the data there fast and cheap and dedupe can enable this to happen.

4. Power, cooling and floor space will be tied more closely with green initiatives and will move up the list of priorities. Yes, energy costs have come down, but the market is so volatile we must plan for more spikes. Additionally, with Obama as President of the US - an increase in global consciousness around Green should occur. Dedupe is a no-brainer way to reduce power, cooling and floor space consumption and make the data center greener. You can't beat saving money while also being green.  Turns out it can be easy being green ;)