Brian Biles' Blog

Keeping It Real

Over the last few years, it started sounding like deduplication was becoming commoditized and turning into just a feature. FalconStor, Sepaton and Quantum announced massive clustered dedupe projects in 2005-7 bolted onto existing VTL products. They lined up OEMs or acquirers, including HP, Sun and EMC. The specs were unbelievable.

If it is too good to be true, sometimes there is a reason. A year or more later, Data Domain is scaling as promised, but the bolt-ons are struggling to meet expectations in robustness and economic impact. Dedupe is an effect, not a method, and the methods are clearly not a commodity, as can be seen in examples like this.

Bolt-on specs are sometimes a wish list. For example, Quantum's DXi 7500 maximum speed specs need a dual-controller system. But that configuration is apparently not available, more than a year since it was announced. What else is lurking that could throw claimed specs into a new light?

For example, in that same system (the basis of the EMC DL3D 3000)

• Backups are written to a non-dedupe disk cache to get higher speed, except in the narrow case of the "adaptive" or "dedupe on" modes, in which writes that are a third or less of their benchmark rates will dedupe inlines; and
• All cached data is kept after deduping as long as possible, so benchmark reads are generally from the pre-dedupe cache to get higher speed (ask about their truncation model).

So their benchmarks could all involve the cache, not the dedupe store. What's the real dedupe rate? Can you know when dedupe on the non-dedupe cache will complete? Can you trust the specs? Time will tell.

Here is the problem. If dedupe matters, you have to be able to finish backup and dedupe in a day. A post-process system will fill up much faster than expected and/or get controller-bound and slow if it gets too far behind..If you replicate, the DR site will always be behind. It would really be a bummer if you didn't expect this.

So here's an experiment you can try at home as a reality check. (These products still don't have much of a reference base that will look like your situation, and without that, a reasonable buyer should do a proof of concept lab test.) If you do a bolt-on test, do it with a pair of large dedupe products, replicating across a LAN to avoid latency issues.

• Backup 30 or 40 TBs as fast as you can (or whatever they say they can dedupe in a day.) Bolt-on system designs can be very curious under stress, so don't go small.
• Time the duration from start of backup to finish or dedupe replication. The end-to-end rate of this test will be a fair proxy for the internal dedupe rate as long as it replicates only deduped data. (Good luck estimating dedupe completion time with their GUIs - it is often left as a daily calculus project.)
• Now try restoring from the replica to see how fast restores are; the replica only has deduped data, so it won't fake you out by restoring from the non-dedupe cache.

Data Domain's systems are easy: with inline dedupe, the only throughput is dedupe throughput.

Storage Vendors Stages of Grief About Deduplication

Deduplication is a challenge to most large storage vendors. IT means they get to sell fewer disk drives, and their business plans involve more drives. I expect most vendors will go through Elizabeth Kubler-Ross' five stages of grief in confronting this dilemma. You can already see it in practice in some of the biggest vendors.

Denial:

• Internally: Deduplication is too slow and it puts data at risk. It ain't real storage, it doesn't work. Their implementation is risky!
• Tell customers: How about a free VTL with your next SAN/NAS upgrade? Let's get lunch.

Anger:

• Internally: Who do they think they are? Dedupe is a fad! Real storage is tons of platters, spindles, lots of sheet metal. This whole industry is going to hell! We need to muddy the water. Buy or OEM something fast and hopefully cheap. Start blogging!
• Tell customers: How 'bout a deal on your next VTL expansion? Running out of space? Right now we can discount VTL really low. I wouldn't bring this up, but I see you're talking to Data Domain...Did I tell you we have box seats at the game next week?

Bargaining:

• Internally: We need to do something. Our sales people are screaming bloody murder! Can we resell something cheap? One size does not fit all.Toss something out we have lying around in the labs. We bundled it in enough deals to tell IDC we have market share, so stop bugging us.
• Tell customers: Please spend a month to review our VTL. Look, it has a little deduper in it. No? Take some more time, look at out esoteric backup software - it has that dedupe thing you have been asking about! No? We can get you a deal on it with that SAN upgrade we have been talking about. We totally understand your needs. Use some professional services hours. Oh, you need our challenging dedupe NAS. No?

Depression:

• Internally: Crap, this stuff isn't going away. What, they went public? We let the market decide, and it turned out we got weak products, by (choose one) trying to preserve and bolt on to our existing designs - or - we OEM'd weak products for time-to-market from vendors who were desperate. (Sigh)
• Tell customers: What's the difference anyway? It's just a feature! What? It's not? You need to test it? Look at this roadmap. This PDF says we check almost as many options as they do. I'm sure we have a reference somewhere...

Acceptance:

• Internally: Wonder if they are accepting resumes?
• Tell customers: We are working on something else, we need more time. Wait for us! I'm serious, our backup software may be less fussy now. We can talk over lunch.

Nothing Has Changed

Last week, Data Domain announced the highest performance inline deduplication system in the industry with the DD690. EMC and Quantum followed hastily with their own announcements.

Nothing has changed

EMC is repacking the Quantum 7500 in the EMC DL3D line. But the inline dedupe rate is still about 150 MB/sec., according to this discussion, after which the only rates we can know for sure are to and from their non-dedupe cache. So the effective capacity, the amount you can dedupe in a backup day, is something less than 13 TBs (dedupe rate x 24 hours). If you don't finish deduping in a day, it will compromise tomorrow's throughput and backup window. So the only way you could use 150 TB of this disk for dedupe is if your backup window is more than two weeks long.

Why offer 148-180 TB in this architecture? Because it is not a dedupe system, it is a traditional VTL (even if it has a NAS interface) -- a disk-based, short-term-retention I/O buffer for data on the way to tape. The extra storage baggage? Call that dupestorage. It is not meaningful for sizing dedupe or replication. Data Domain's dedupe speed and resulting effective capacity is greated by more than a factor of two.

Seems like everything's still the same:

• Data Domain is still the deduplication performance leader;
• EMC and Quantum are still selling traditional VTLs based on dedupe storage;
• EMC is still selling massively over-provisioned disk systems;
• Quantum is really trying to sell tape libraries above all.

Oh, I guess one thing has changed. Inline deduplication used to be considered too slow compared to a post-process. I guess that has changed. The non-dedupe write speed of the EMC DL3D 3000, at 1.44 TB/hour, s about the same as the inline dedupe speed of the DD690. If post-process isn't even faster than inline, why bother?