Daniel's Blog

Driving Down the Costs of Deduplication: "Continuing the Continuum"

Since shipping its very first deduplication storage system in 2004, each successive generation of Data Domain product has deduped faster inline and offered extra capacity (both raw and logical capacity). This "continuum" of performance and capacity increases has been marked by some true milestones - such as the recently introduced DD880, which eclipsed the single controller ingest and store to disk speeds of all post-process deduplicating VTLs (and even of many VTLs that don't dedupe at all). And yes, along the way, the features and capabilities of Data Domain systems have evolved with new capabilities (cascaded replication, OST, ability to address archival workloads, retention lock for IT governance, etc.) Putting those enhancements aside for a second and focusing purely on Data Domain's "continuum" of performance and capacity increases, what are we really delivering in terms of unique customer value? In short, it's this: driving down costs of deduplication - while keeping customers ahead of the curve in terms of their data growth challenges.

So why is this interesting? I spent all of last week at SNWin Phoenix, AZ, running the SNIA Data Deduplication Hands-On Lab. Not surprisingly, data deduplication was once again a very popular topic throughout the conference. In the concurrent sessions, many of the vendors were busy arguing about technicalities, and this can definitely make for interesting debates, as well as the occasional farce. While some lab participants were getting their first exposure to the technology, many of them were now well versed in the particulars of the technology and their interest and questions were surprisingly singular in nature. They wanted to know how deduplication can save them money in the face of rapid data growth averaging 50% a year or more.

Well, there are la lot of ways to achieve cost savings with inline deduplication technology, ranging from automation of labor intensive DR processes to the whole tape replacement story. But, curiously, I instead found myself telling these users the same story I've been telling customers since I first joined Data Domain in 2006 - namely the "continuum" of performance and capacity increases demonstrated by each successive generation of our systems. How specifically does this drive down costs? Ultimately, it's all due to our SISL scaling architecture, which leverages the upgraded multi-core CPUs and memory in these new appliances - but, for purposes of this blog post, I'd prefer to address the question at the system and real-world implementation level.

Consider the new mid-range and entry-level deduplication storage systems Data Domain announcedearlier this week (the DD610, DD630 and the DD140 entry-level system). Each of these new systems replaces a previous generation that was similarly targeted for either mid-range enterprise applications/regional data centers or remote office locations. In the case of the new mid-range systems you get 60% more capacity and 100% more throughput performance. For the new entry level remote office system, you get 100% more capacity and 50% better performance. So, cost per Gb goes down (yet again) and price/to performance improves. Users can protect 60%-100% larger datasets and they can write their data 2x faster, enabling them to meet their nightly backup windows.

Based on what these users at the SNW SNIA labs were asking me, I remain convinced that Data Domain's proven track record of ongoing significant performance and capacity increases (while relative costs decrease), may actually be the most important element of the value proposition inline data deduplication offers.

So, I guess we should "continue the continuum?"

Like Peanut Butter and Chocolate

I have been excited about the possibilities of using deduplication storage and virtualization togethersince implementing both technologies while working in IT prior to joining Data Domain. At my previous employer, we built a multi-tier application infrastructure with a disaster recovery site 2,000 miles away using VMware ESX Server in both locations. All VMDK and application backups went to a Data Domain system in the primary site and were replicated those 2,000 miles to the DR site. The image backups were recovered on a scheduled basis to keep the underlying virtual servers up-to-date, with only specific applications (MS SQL, Exchange, CIFS fileshares) receiving agent level protection and recovery. As a result, our primary data center was consolidated into five racks and we eliminated the use of tape for protecting our most critical applications. It is the many benefits of achieving these two objectives that our customers continue to enjoy today.

There are three generally accepted techniques for backing up VMs:

1. Running backup agents, installed inside the guest

2. Running proxy-based VMDK backups using VCB, with or without 3rd -party backup application integration

3. Running host-based VMDK backups, either from the service console or using a VMware-specific backup application, like Vizioncore vRanger Pro

Best practices documentation typically references using some combination of these approaches. Unlike most other deduplication solutions, Data Domain systems support all three methods and will eliminate the redundancy across all data stored. This support allows organizations to customize their data protection policies, and to best leverage existing infrastructure.

Because Data Domain systems deduplicate data inline, image backups can be reduced 40-60x or more before being stored, and entire virtual environments can be efficiently replicated using a minimum amount of bandwidth. We support VCB backups natively, and can be a target for most enterprise backup applications that provide VCB integration as well as traditional agents installed in the guest OS. Specialized applications, scripts, and even native VMware features like cloning, storage VMotion and templates can all use Data Domain storage.

In today's world, this simplicity and flexibility are critical forces which balance the underlying complexity of new technologies. Data Domain and VMware are like peanut butter and chocolate - both are great on their own, but together, they become greater than the sum of the parts.

Come visit us in booth #1302 at VMworld next week - the RPBC is on me.

Do We Really Need Post-Process Anymore?

Every time we come out with a bigger, faster system, I get excited about all the new customers who will benefit from deploying our technology.  This past Monday, Data Domain announced the DD880, our newest deduplication storage system. The full details of the release can be found here.  This system presents about 70TB of usable capacity per controller, twice that of the DD690 and more than doubles the maximum throughput to 5.4TB/hr.  A year ago, large enterprise customers were very excited by the DD690, and that system is now successfully deployed in many large datacenters.  I imagine the DD880 will be equally well received. 

On the other hand, vendors with post-process deduplication solutions might be less enthusiastic.  After all, the fundamental premise of their marketing has been that post-process systems are faster and bigger.  In reality, the inline deduplication throughput of a single DD880 exceeds the speed at which they can ingest data in native form to a 2-node post-process system. And the higher physical capacities they claim as a benefit?  They are but a side effect of the disk i/o and caching requirements of post-process deduplication – you can’t actually protect a larger dataset with all that space.  It’s kind of like a car salesperson pointing out the massive fuel tank in a gas guzzler as a competitive advantage, while disregarding the mileage it gets. 

A single DD880 can deduplicate 60TB inline, while concurrently replicating the deduplicated data, all within a 12 hour window.  As with all of our previous new product releases, performance and capacity increases are derived from upgrades to the processors and associated hardware in our systems.  This is the power of Data Domain SISL, our CPU-centric approach which enables fast inline deduplication using a minimum amount of disk.  In comparison to the DD880, for the same 60TB backup a post-process deduplication system would require at least 2-4x more physical disk and significantly more compute power in the form of complex multi-node configurations.  It just doesn’t make sense why our competitors would expect customers to do this.

In the end, this is about one thing – Data Domain has enabled a paradigm shift in disk storage for data protection and disaster recovery because once our technology is deployed, customers can save money.  The bottom line value of post-process deduplication, with its need for more of everything, will never match us.  Good luck to those vendors that decide to keep trying.   

Keeping It Clean

Recently, we announced the ability to perform NIST/DOD compliant system sanitizing at a file level, as a new feature of our Retention Lock software option.  With deduplication storage systems, where underlying data patterns are potentially shared among many logical files, there is a particular challenge around ensuring that contaminated data is appropriately removed. In many cases, an entire storage system would need to be destroyed because of a single piece of inappropriate data, particularly in secure or classified data environments. For these customers, this feature provides tremendous value, as individual files or entire systems can now be sanitized by a process that meets the standards dictated by the DoD Clearing and Sanitization Matrix and the NIST Guidelines on Media Sanitization, while the “clean” data remains online and available for access.

 

Data Domain is the first to deliver secure shredding for an inline deduplication system on a per-file basis. It is not glamorous nor does it get the blogosphere all in a-tizzy. But blogs don’t sell products or decide what matters - our incentive to deliver this feature is driven directly by customers telling us they need it.Having a system that works well and solves customers’ problems is what makes us the leader.   

 

I have always felt that Data Domain was about much more than deduplication – we have been speaking to our customers about, and delivering, our efficient and intelligent storage platform for years. Let’s set aside the arguments about inline vs. post-process for backup throughput or time-to-DR for a moment. The fact is, by segmenting and identifying every bit of data as soon as it enters the system we have knowledge about that data which enables many innovative capabilities. We started with inline deduplication and data invulnerability then added WAN-efficient replication to successfully address the challenges our customers were having with physical tape and traditional disk-based backup systems. Over time, we have followed through on the promise of SISL to create bigger and faster systems. And we continue to add value for our customers with features like deduped snapshots, enforced retention, and now secure shredding.

 

To quote Steve Winwood from a favorite Traffic song of mine: 

“Who knows what tomorrow may bring?”

 

 

Measuring Up in the Enterprise

In my last post, I spoke directly to the claim by competitors that Data Domain is not an enterprise solution. I indicated that the number one capability our largest customers require is immediate, reliable and efficient replication of deduplicated data, as it has tremendous cost-saving benefits for organizations that are still using tape for offsite DR. In fact, it was the focus of a presentation I gave last week to a large customer looking to expand their Data Domain implementation and standardize on our solution across all branches. During our meeting, we discussed the metric against which dedupe replication as a DR solution is measured, Time-to-DR. This is the period of time measured from the start of the first job in the backup window to the completion of replication for all jobs. The high-level steps that make up the process measured by the time-to-DR metric are:

1. Full backup
2. Replication
3. Data consistent and readable from replica

To understand how Data Domain Replicator achieves the fastest end-to-end data movement, lets work through these steps using the example of a subsequent full backup of a 20TB dataset, that is then replicated using an OC-3 (155mbps) WAN link to a DR site.

1) Data Domain inline deduplication is capable of backup throughput of 2.7TB/hr, so 20TB can be completely backed up within 7.4 hours. At a 50x reduction (combination of deduplication and compression), this will result in ~400GB of data that needs to be physically stored and replicated.

2) Because data is deduplicated inline, Data Domain Replicator can begin moving the deduplicated data immediately. This enables the data movement process to run concurrently with the backup window, as well as ensures that data is recoverable at file/image/job level as soon as it has been replicated. In this case, replication over the OC-3 can keep up with the backup process and will complete within minutes of completion of the backup, and within an eight hour window from the start of the first backup.

3) By design, data is continuously consistent on the Data Domain replica - no additional steps are required to make the data readable. This means that as soon as replication completes the data movement, the 20TB of data is fully protected and available for both local restores in the primary site and recovery in the DR site in the event of a disaster.

For any enterprise customer looking at using deduplication to enable offsite disaster recovery, this best practice test of protecting a large dataset (20-30TB), replicating, and recovering from the replica should be mandatory.

If At First You Don't Succeed...

We're one month into it, and Spring is in full force here in the Silicon Valley. This season is all about rebirth, both figurative and literal, and as such, it seems like the time for our competition to dust off their old FUD, and try to put in a new spin on it. Tony Asaro in a recent post referred to one example.

The latest example stems from a post by Sepaton's resident blogger, Jay Livens, on the heels of their announcement that they closed a sixth round of funding.

He states:

"One of the questions I often get asked is 'how do your products compare to Data Domain's?' In my opinion, we really don't compare because we play in different market segments. Data Domain's strength is in the low-end of the market, think SMB/SME while SEPATON plays in the enterprise segment. These two segments have very different needs, which are reflected in the fundamentally different architectures of the SEPATON and Data Domain products."

Let's examine the facts. Our systems are deployed in the datacenters of many large enterprises. For example, the major financial services firm where I did my last field installation is storing over 5PB across four geographically distributed datacenters. These large enterprise customers clearly see the value in Data Domain's well-designed inline approach that does not slow down backups or restores, enables immediate replication of deduplicated data, and makes the deduplication process transparent to users and applications. To do this in practice is difficult since the process of identifying duplicates inline is inherently a very compute intensive process. WIthout careful thought about how to inline, the resulting design makes inefficient use of CPU, memory and disk resources or just runs too slowly to be effective.

Vendors that cannot successfully navigate these challenges must go to market with post-process deduplication systems. While this may allow them to avoid the hard problems of delivering deduplication with speed and simplicity, the consequences are many. Directly stated, post-process deduplication systems are just like traditional storage systems. They have the same disk I/O bottlenecks as traditional VTLs - they require disproportionately high spindle counts just to land the native data to a disk cache. That data must then be read back from disk and deduplicated before any additional processes, such as replication or verification, can occur.

This last point is critical, as in our experience, replication is the number one enterprise capability that our customers and prospects require in a deduplication storage solution, as it is key to enabling simple, reliable, and cost-effective disk-based DR. In this article from January 2006, announcing the ability to replicate the native data between their VTL's, Sepaton's president/CEO seems to share our belief in the importance of this capability:

"In recent years there have been numerous public accounts of lost or stolen tapes and the resulting corporate pressure to solve data loss. The risks are high - lost revenue and productivity, increase in customer dissatisfaction, fines and penalties and damaged corporate reputations," said Mike Worhach, president and CEO, SEPATON, Inc.

Customers looking for deduplication and replication solution who do their research  will learn that Sepaton still does not have any ability to replicate deduplicated data, while our ability to do this is proven across thousands of customers.  Different definitions of "enterprise" indeed.

WYSIWYG

This past week at SNW Spring in Orlando, Florida, Data Domain participated in the inaugural Deduplication Hands-on Lab. Our lab started with a series of excercises that had the users interacting with data in a CIFS share on a pair of Data Domain DD530 systems, with the following three learning objectives:

1. Analyze the effects of inline deduplication on files stored to a CIFS share on a Data Domain deduplication storage system.
2. Create deduplicated snapshots to provide point-in-time recovery of previous versions of files.

3. Recover data from a replicated copy on secondary Data Domain system.

Students really liked these exercises, which had them using the Data Domain systems as NAS storage - many of them followed up at our in-booth demo with F5 ARX file virtualization to get further insight into how they can deploy this technology in their enterprise. During the second half of the lab, students worked within Net Backup 6.5 to perform backup, optimized duplication and replica recovery operations using Data Domain OpenStorage software. Interestingly, while feedback to the entire lab was very positive, it is the step shown below - where students use a CLI monitoring tool to observe the real-time effects of inline deduplication - that continues to have the greatest impact on illustrating what inline deduplication really means.

With Data Domain, what you see is what you get: deduplication occurs inline, and only unique data is written to the physical disk - notice the significant difference between inbound data on the network interface (a) and data written to disk (b), and that disk activity never exceeds 2%. This exercise illustrates our CPU-centric inline deduplication at work.

Time and again, students, and even representatives of other vendors in the lab, would watch this and have their own version of an ahh-ha moment - they were seeing for themselves that with Data Domain deduplication storage, writing data and deduplicating data are the same process. For administrators, this makes all the difference, as unlike post-process deduplication, with inline deduplication there is no need to manage different pools of storage for landing native data, no need to predict when deduplication will finish and replication will start, or where data will be recovered from, and no need to provision extra storage as a hedge against the inherent unpredictability of the post-process. Of course, vendors will try to hide these details and spin them into features and benefits (until you read the manual), but from what I observed, the customers eyes are now wide open.

Read the Manual

I was just reviewing the daily competitive win report where a new customer stated that, beyond successful testing of the Data Domain solution, much of the evidence pointing against the viability of the competition's solution was found in their product manuals. Data Domain is fortunate in that many customers are able to make a purchase decision based on our proven technology and large reference base. On occasion a customer requires a competitive bake-off, typically to see past an incumbent vendors claims, allowing us to reinforce our message - that doing deduplication well (as we do) is hard, and that our competition tends to fall flat outside the world of PowerPoint's and glossies.

As a Data Domain SE, I would treat each proof-of-concept as if it were an implementation for a paying customer. We would gather the same pre-installation information, they would be shipped the same equipment, and I would configure it exactly the same way that I would for any other customer. A typical Data Domain configuration taking less than an hour of setup from crate to running backups made this reasonable task - if it took days of PS consulting, our approach may have differed. As part of our packaging, we include a screwdriver (handy in post 9/11 world where having one in your luggage is suspicious), and of course, the product manuals. I was always very proud to give customers our manual - I have written technical manuals before, and ours is very thorough.

Reading the manual not only offers the obvious operational guidance, but it also tends to reveal the truth of how a product really works. Other times, there is a noticeable lack of details. After all, most manuals are read only AFTER someone becomes a paying customer, when it is probably too late to do anything about what you learn. Everyone seems to accept that there is a process that happens as a technology's engineering reality meets marketing spin. Most savvy IT customers read brochures with a grain of salt, and accept that the truth might be somewhat different. But if you are serious about evaluating a vendor's product (and if they are serious about you as a customer) ask them to give you the manual BEFORE you agree to buy anything, take the time to read it, and ask questions based on what you learn.

Getting Ready for the Show

I am currently working with F5 on a technology demonstration for SNW Spring using the F5 ARX file virtualization appliances to integrate Data Domain systems as a cost effective secondary storage tier. For those not familiar with the F5 ARX (formerly Acopia) file virtualization technology, what it does is make multiple heterogeneous filesystems look like one big one. What takes it far beyond tools like Microsoft DFS is the ability to define policies that identify candidate files (e.g. based on file type or time since last access) and automatically migrate them from one physical storage location (expensive primary NAS filers) to another (Data Domain deduplication storage) without the clients ever knowing about it.

For many organizations, a significant majority of filesystem data, as much as 90%, does not change, yet it still needs to be accessible and online - archiving the data to tape is not an option. Like Data Domain, the F5 ARX appliance is simple to use and just works - it was cool to watch a file copied into a drive mapped to the virtual namespace first appear on the primary filer, and then be automatically migrated to Data Domain storage after a defined period of time. Files with specific extensions were directed to Data Domain storage without ever consuming capacity on the filer.

This year at SNW in Orlando, Florida, we will be running this demonstration in our Platinum Galleria booth in the main conference center hallway. Seeing it first-hand is the best way to get the "ahh-ha" moment that our users have when they realize the profound impact that innovative technology can have on how they do their job. Come by and check it out - I look forward to meeting you.

Where Will You Be Led?

In his most recent post for Dedupe Matters, Tony Asaro makes some predictions around the growing impact deduplication will have in the datacenter for 2009. I was particularly struck by the following statement:

"The perceived risk of implementing D2D backup with dedupe is all but gone and the value is glaringly clear. Additionally, the bad economy makes the value proposition that much more compelling."

This notion of perception is important to flesh out, as it is indicative of another question: When making a purchase decision, are you led by the technology or the vendor?

Let's explore an example of a technology-driven solution being deployed in production datacenters. I made mention in my last post of Data Domain OpenStorage integration with NetBackup-- noting that our joint customers are achieving high throughput in production environments. High throughput has historically been achievable only by using LOTS of disk spindles, back-ending a tape emulation interface (VTL). Data Domain's SISLTM architecture does it with very few disks (which is critical to delivering the cost benefits of deduplication, as discussed here), and the OST interface allows us to further optimize performance at a protocol level. In addition, using the OST interface with Data Domain systems gives NetBackup the ability to manage the WAN efficient replication process between Data Domain systems. Called "Optimized Duplication", this is a key feature enabling administrators to leverage every copy of their data in the distributed environment.

Looking deeper, these customers are also adopting 10gbps Ethernet to connect the NBU Media servers to the Data Domain systems (4gbps FC doesn't seem so fast anymore, does it?), deploying their media servers as a scalable tier of data movers (with dynamic load balancing), and otherwise seeking out, and addressing, the bottlenecks within their infrastructure. This kind of approach to systems architecture is typical of organizations interested in solving their challenges around storage and data protection, and results in a best of breed solution.

The process tends to propagate, as these same customers ask themselves how they can apply the Data Domain system to solve other problems. Why limit yourself to backup, when you can use the same data to automatically refresh reporting and development instances of databases, or ensure that a DR site thousands of miles beyond the reach of block-array based replication technologies is updated daily? Why limit yourself to data protection, when you can use the same storage for cost-effective tier 2 storage as well?

Our customers doing this today know that the limits dissolve when they allow the technology to point them towards the future. Ask yourself, whose lead will you follow?