Tony's blog

Voice of the Customer: Collection Replication, Disaster Recovery and Tape Elimination

The following is a discussion I had with a Data Domain customer - Rob Carlisi, Systems Engineer for Orange County Public Schools (OCPS). 

 

You can also listen to the podcast - Download Interview Orange County Public Schools

 

A couple of things to point out:

• OCPS has eliminated using tapes completely. 

• They are using Data Domain Collection Replication for their off-site disaster recovery.

Tony Asaro: Rob, welcome and I appreciate you participating this discussion.  Please give us a background on your company.

 

Rob Carlisi: I work at Orange County Public Schools in Orlando, Florida. We’re the 11^thlargest school district in the country. The amount of online storage that we have is a little over 100 terabytes and the amount of backup data we have on Data Domain is about 450 terabytes.

 

Tony Asaro: And what is your role with the company, Rob?

 

Rob Carlisi: I’m a systems engineer and I’ve been here for a little over 6 years. Our group is primarily responsible for anything that happens inside the data center that’s not network related. I’m responsible for the backup/restore environment, SAN and fibre infrastructure as well as installing and maintaining the servers themselves.

 

Tony Asaro: Start out by giving us some examples of how Data Domain has provided real value to your company.

 

Rob Carlisi: Yeah, I’ve quite a few examples. Recoveries with Data Domain are much faster than our previous environment. The restores start as soon as you click a button instead of waiting for tapes to mount or waiting for tape drives to become available. The footprint of our solution went from around 6 to 8 feet of floor space down to a single rack with Data Domain. Data Domain gave us the ability also to replicate all of our backups to our DR site so we have another set of Data Domain boxes sitting at our offsite location that we replicate to every day. We also use RMAN with Oracle to backup all of our databases directly to Data Domain. Since we’ve been using that method, we’ve seen up to a 60x compression rates on those backups. And also over the next few months we’re going to implement VRanger which will backup all of our VM’s over to Data Domain.

 

Tony Asaro: Rob, tell us how Data Domain has improved the overall economic effectiveness within your IT operations?

 

Rob Carlisi: Data Domain has enabled us to save exponentially actually on the amount of tape we would have had to purchase if it wasn’t Data Domain and de-duplication. We also went from a 10 to 15 day retention policy on all of our backups to a 30 to 40 day retention.

 

Tony Asaro: How has implementing Data Domain changed your usage of tape now and in the future?

 

Rob Carlisi: Well, tape no longer exists in our environment and we’re confident that it’s going to stay that way.

 

Tony Asaro: Rob, tell us about how you’ve leveraged Data Domain for offsite Disaster Recovery.

 

Rob Carlisi: Before we had implemented Data Domain, we weren’t replicating any of our data off site. When we decided to start replicating with Data Domain, we did it on a collection basis so anything that touches the primary box, gets replicated to the destination box on our off site location. We set it up the first day to replicate and it’s pretty much been hands off since then. We’ve had the remote site down. When we bring it back up, it just makes that connection automatically and starts replicating again. We’ve never had any lag and we’ve never had a problem talking to the remote site.

 

This solution actually got all of our backups from our entire environment off site. We’re able to restore at a DR site back to full throttle.

 

Tony Asaro: Data Domain’s data dedupe actually enabled your disaster recovery strategy because it reduces the bandwidth costs.

 

Rob Carlisi: Actually that’s true. After it does its de-duplication and compression and clean up, you’re not actually transferring that much data across the line compared to what it actually backs up every night. It saves on money as well as time to replicate because when you’re not moving that much data across the LAN or the WAN it doesn’t take that long to do you’re not too far behind your production environment.

 

Tony Asaro: Why didn’t you use primary storage replication versus Data Domain to do this?

 

Rob Carlisi: Well, one was the cost and we also liked an appliance environment. We found that it was a lot easier to expand a Data Domain environment rather that a primary storage environment and we just like the flexibility that Data Domain gave us. Also, the primary reason was the de-duplication technology that they had was far superior than anything else.

The Changing Landscape of Disaster Recovery

Disaster recovery (DR) means different things to different people.  For many IT professionals DR is associated with tape recovery and/or remote replication.    Recovery data from tape is fraught with challenges including rapid recovery and reliability.  Remote replication within primary storage is often complicated, expensive and inefficient requiring volumes to be restored versus specific files or data sets.

The combination of disk-to-disk (D2D) backup, data dedupe and replication offers a valuable approach to DR combining the granularity of recovering from a backup repository and the ability to leverage remote replication to store copies of data at an offsite location in the event of of site disasters and for offsite vaulting for long term retention.  For the vast majority of data this is a smart approach that serves the widest range of requirements for customers.  There will always be the need for primary remote replication but only for the corner cases in terms of who uses it and what it is used for.  However, I contend that D2D backup + data dedupe + remote replication has a much broader audience. 

Data Domain just announced an update to it's Replicator software.  I remember when Data Domain first came out with Replicator and speaking with customers that literally felt liberated by leveraging dedupe over wide area networks.  Dedupe enabled them to implement a DR strategy above and beyond what they had prior and still fit within their budgets. Five years later the solution is more mature, advanced and field proven.  Additionally, the use cases and value proposition have advanced and are proven.  Data Domain providing a combination of D2D backup + data dedupe + remote replication has impacted the data center in a number of important ways:

• Offers a cost effective and efficient DR solution that is otherwise impractical for many customers.

• Results in cost reductions include bandwidth, primary storage investment, power, cooling and floor space.
• This approach is "easy" versus perceived complications with using other DR methods (e.g. primary storage replication).
• Offers a more granular and rapid method of data recovery improving SLAs. 
• Makes it possible for customers to actually eliminate or greatly diminish the use of tape and tape libraries.  This is an important point and we are seeing momentum for tape obsolescence with a growing number of customers. 
• Multi-site replication provides a powerful and valuable solution for Enterprises with a large number of remote sites - this is enabling a new constituency to reap the rewards of this solution. Data Domain's recent expansion of 90 remote sites to a single DD690 is significant and makes it even more useful to large Enterprises. I will add, regardless of whether you have two or 90 remote sites fan into a single DD appliance at the central site is powerful especially when coupled with global dedupe. 

I have spoken to companies that did not implement remote replication because of cost - even though it was considered every year for five, eight and ten years.  And there are other companies that only replicate 10% to 20% of their data and use backup to protect the rest - they just could not rationalize the expense.  However, D2D backup + dedupe vastly changes the economics of remote replication and DR.  The landscape of DR is changing - enabled by D2D backup with dedupe - making it practical and valuable for the mainstream market.  Remote replication will drive the adoption of D2D backup with dedupe and D2D backup with dedupe will drive the adoption or remote replication.  This is a killer app on top of a killer app. 

Customer Discussion: Bank Midwest

I had another great discussion with a Data Domain customer- Bryan Wilken, VP of IT for Bank Midwest:

Tony: Can you give us any examples of how Data Domain provided real value to your company?

Bryan Wilken:  e have completely stopped using tapes in all but one of our systems, and we’ll have that on the Data Domain in the next few weeks here. The bank now has a more robust and fault-tolerant back-up system. It runs automatically every night and it doesn’t require the need of employees to change tapes and take them home. We’re also seeing a 23-to-1 compression ratio, or de-duplication. And restores have been handled quickly and effectively by the IT department. We no longer have to wait to have tapes either brought back in or reloaded and reread before we can restore data.

Tony:  Can you tell us how Data Domain has improved the overall economic effectiveness within your IT operations?

Bryan Wilken: The Data Domain disk-to-disk system is saving Bank Midwest in tapes and drives alone around $13,000 a year. Plus we have the security risk that’s been eliminated without having tapes that are leaving the bank’s premises. As I mentioned, we have moved to virtualization of our servers. We hadn’t first planned on this, but we are now backing up our VMware servers nightly to our Data Domain and then replicating them to our DR site. Another added layer of protection in case of a disaster.

Tony: Would you recommend using Data Domain to other IT professionals?  And tell us why.

Bryan Wilken: Yes, I would. What I liked about Data Domain as we’ve gone through installing this product, is that the solution worked just as the vendor claimed it would.  We’re also getting the compression rates that they claim and it was just an easy set-up, and it’s worked from the day it was installed. That’s the kind of products I like to purchase for Bank Midwest.

Bryan validates the Data Domain value proposition with 23-to-1 data dedupe, fast and reliable recoveries, reduced management overhead of tapes, reduced security risks, enabled disaster recovery - and fundamentally it was easy to set up and put into production. 

I think it is interesting to note that this is yet another customer that is completely eliminating tape. 

Frank Talk: Data Domain CEO Discussing 2009

I had a discussion with Frank Slootman, CEO of Data Domain. We talked about Data Domain’s outlook for 2009, what is working for them and the trend of more and more customers moving away from tape. 

Tony:  Frank, tell us about Data Domain’s business outlook in 2009.

Frank: We gave guidance to the capital markets – a lot of people refuse to give guidance to the capital markets. For us – it was important to give an indication of what our operating assumptions both for Q1 as well as for all of 2009. Take a look at that guidance. 

Obviously we are getting into the laws of big numbers. Its heavier lifting growing from zero to one million than it is from 100 million to 200 and 200 to 500. On the other hand we are guiding to significant growth even in 2009 – where the overall storage market is going to compress probably by 5-10%. 

We are planning on growth and we are hiring. We have the advantage of running on a secular trend – as opposed to a macro trend – that’s a generational shift from legacy tape infrastructure to one that is disk and network based. We think we are in the first or second inning of this ball game – very low penetration. We will have head winds like everybody else but we think we are in a good category and still considered by customers even in this economy. 

Tony: That is an important point – you believe there will be growth with Data Domain while the rest of the storage market is planning for the opposite. And keep in mind that 2007 and 2008 weren’t great economies either. Nothing compared to 2009 but the last few years haven’t been anything to write home about and you still have seen amazing growth. 

Frank: Yes and the reason is we sell pretty hard on ROI and payback. In other words, at the end of the year the customer should be better off with this new infrastructure on a cash basis – you should have more money left at the end of the year for having gone with Data Domain as opposed to just staying the status quo. Net new incremental spend is only justifiable if on a cash basis I am better off. 

Typically where people are making their investment back they are not spending as much money on tape media, they are not spending as much money on record management services – you start adding that up it starts paying you back pretty quick. When people have investments in legacy tape the maintenance is really high and it gets higher every year. You very quickly find a cross over point that people are saying it makes sense to go with Data Domain. 

Tony: IT professionals are telling me – no more investment in tape. 

Frank: I call that drawing a line in the sand. You get a mandate from above that says – it isn’t a little bit here or there – they are going to make this change.

Tony: There are also other benefits - like not putting tapes on trucks. 

Frank: Right – no hands. 

Tony: Yes and there is a security and administrative aspect to this.  But one other thing that is harder to quantify is the recovery issue.  This might be harder for them to sell their management but for the IT guy it is a huge win – recovery is fast and reliable.

Frank: And they can have deeper retention. They can have far more data online than they could have otherwise. Individual file restores and you can do full system restores much faster than they could before.

Tony: Data Domain has been saying this for years but now the customers are saying it.

Frank: That is right. Recovery speed and reliability isn’t what CFOs embrace because they are really looking at the measurable economics. They want to see how Data Domain pays them back within a certain amount of time. The other stuff is all gravy.

Tony: I am surprised at how many people I am seeing moving away from tape. I was skeptical it would ever happen. 

Frank: Look at the numbers. IBM was down on tape by 31%, Sun was down 15% and Quantum was down 25%. The decline is accelerating. For a while they were going sideways but now they are dropping off a cliff. 

Tony: I am also seeing Data Domain becoming the de-facto standard.

Frank: We are not going to get too big for our britches. We always want to feel like we are the underdog with these giants around us. We are not going to get too cocky. 

Tony: Right, healthy paranoia – according to Andy Grove.

Frank: Yes and you need it. I find a lot of products in the market to be inferior and not doing customers any good. People often buy because they have relationships and they trust their vendor to deliver a good product but often this is not the case. It is often that it’s just downright bad. 

Tony: What’s next for Data Domain?

Frank: We are going to keep on trucking. We will continue to build bigger, faster and more broadly capable systems. We will be a very broad class of storage that is highly optimized. That’s our game.  Our dedupe is a bone crushing technology that is enabled in a fully realized solution. 

Tony: My view is that the way VMware has virtualized servers - Data Domain has virtualized data. 

Frank: That is exactly right. 

Panel on Storage Optimization

I just participated in a panel on storage optimization for Pacific Crest - an interesting firm that focuses on technology research for their investors. I was one of the panel members alone with Stephen Foskett, an IT consultant that works with end users and Mark H., an IT manager for plastics manufacturing company with 3,000 employees and 40 locations. 

Both Stephen and I talk to a lot of IT professionals and it was good having Mark H. on the line for his direct input on his IT operations.

Here were some key takeaways:

1. Both Stephen and I are seeing many of the same things - IT budgets haven't been dramatically cut but spending has slowed down and some projects have been put on hold for now.  However, high priority projects are moving forward since business can't come to a halt. Mark verified this within his IT organization. In the last 30 days I've personally spoken to about two dozen IT pros - and that has been the universal position among all of them.

2. Primary storage and backup are seen as mandatory. In many cases IT professionals are taking a look at what they have on the floor and seeing how they can better utilize existing IT assets. 

3. End users are more open to moving away from the status quo and leveraging innovation if the economic value is compelling and quantifiable. For example, Mark H. actually is eliminating tape and instead using Data Domain for backup. This is becoming more of a common theme among the Data Domain customers that I speak with. There is an opportunity to make big changes in the data center based on economics - especially in 2009. This isn't just true in backup but across the board.

4. Stephen and I both agreed that what we are seeing is continued storage usage - which includes optimization of what we already have and potentially buying new capacity.  Mark pointed out that storage was core to their business and was not being cut or delayed. 

5. Another area that is core to IT operations is backup and that investments are being made in backup hardware - with data deduplication being key to data center optimization. 

Dedupe Predictions for 2009

Here are my predictions for 2009 relating specifically to data deduplication:

1. Disk-to-disk (D2D) backup with dedupe continues to proliferate, becoming much more mainstream in 2009. The perceived risk of implementing D2D backup with dedupe is all but gone and the value is glaringly clear. Additionally, the bad economy makes the value proposition that much more compelling. 

2. End users utilize dedupe as a requisite part of nearline and active storage archives. Moving dormant data off of Tier One storage needs to go up the priority list. The question is what do you move it to?  Standards-based NFS and CIFS cost effective storage with dedupe will be embraced by a number of customers. This should be considered and implemented by end user companies of all sizes. 

3. D2D backup and archiving will be a big Cloud Computing application. SMB customers with no secondary site can use Cloud Computing services for off-site disaster recovery. Additionally, low cost cloud storage can be used as an offsite archive. But you need to get the data there fast and cheap and dedupe can enable this to happen.

4. Power, cooling and floor space will be tied more closely with green initiatives and will move up the list of priorities. Yes, energy costs have come down, but the market is so volatile we must plan for more spikes. Additionally, with Obama as President of the US - an increase in global consciousness around Green should occur. Dedupe is a no-brainer way to reduce power, cooling and floor space consumption and make the data center greener. You can't beat saving money while also being green.  Turns out it can be easy being green ;)

 

Data Dedupe In A Down Economy

It is important to be hyper-focused on the economy and the prudent thing for IT professionals is to prioritize on investing in solutions that provide a real impact on reducing costs and increasing efficiency. Sounds like a no-brainer, but too often we are distracted by dealing with the status quo - conducting business as usual - which is fine when there is no crisis to face. However, the entire world is facing an economic crisis - like the falling of dominoes - and we are far from the final domino's descent. 

When making IT decisions for 2009, data dedupe should be at the top of the list because of the no-brainer value proposition - for disk-to-disk backup AND online archives. Data dedupe enables you to effectively put more data on physical disk drives with 10-to-1, 20-to-1 ratios and beyond. Not only do you save on reducing cost per GB but it also leads to lower power and cooling consumption. Data dedupe can have a major impact on operational efficiencies. It significantly lowers WAN bandwidth requirements and less bandwidth consumed means lower recurring communication costs. The cost of recovery goes way down because you don't have to go on a wild tape chase tracking down the files or database records that have been spread across multiple tapes due to interleaving or sitting in a cardboard box at Iron Mountain. 

The challenge we face in this economy is the uncertainty of the future - businesses cannot predict what will happen and often times this paralyzes us from making decisions. However, no action is indeed an action and probably among the worst we can take, especially in a crisis.