Brian Biles' Blog

If Not a Bailout, What?

Quantum's revenue run rate has been in free fall over the past several years, while their balance sheet has ballooned into one big IOU. This preceded the bad economy; even while Sun and IBM tape revenues were growing, Quantum tape was being voted off the island. With the loan last week, they're officially on EMC's life support. If they were going to acquire Quantum whole, they would have already done so. It is also a poorly kept secret that EMC has their own designs on this technology and only has a temporary need for Quantum to stay around. One can imagine why it worked out this way, but evidently EMC just wants some assets, not the company or the people.

Why customers would directly place their trust in such a vendor is a mystery. Think of other tech companies that have gone into years of chronic reverse. How many turn around? Their layoff history doesn't bode well for future products and ongoing support. EMC (and later Dell) will functionally replace Quantum's channel as customers hope for a supportable product experience, hiding the weakened brand, but even that is still to come. As with other former high tech stars, Quantum has a rich history, but that's not helpful anymore. They have to spend more time on how they are keeping creditors at bay than on how they're building markets.

Unlike the US Treasury with GM, where there's no long term plan to keep the assets, this is not really a bailout; it's something else. There is a critical choice facing bloggers. Which metaphor should be applied?

A. In financial terms, Quantum is a zombie.

B. They are cutting off one limb after another and trying to regain momentum. It reminds me of the Black Knight in Monty Python's Holy Grail.

C. EMC can prop up this regime as long as it suits theirneed to gain access to the resource it wants. Quantum is EMC's banana republic.

D. From the outside, it looks like slow organ harvesting.

Dedupe and Storage are Not Commodities

In a commodity market, differentiation is very low between producers. Suppliers are chosen purely on price and distribution.

Dedupe storage products are starkly differentiated. The products couldn't be more distinct, though in the rush to learn about why Data Domain has grown so fast, it's easy to misunderstand what's going on. Dedupe is not a commodity; it's not even a product category.

Bob Passmore at Gartner makes a persuasive case that nothing in storage is a commodity. Enterprise disks aren't commodities. Storage systems definitely aren't commodities; the lowest common denominator between them is not sufficiently capable or predictable for customers of any reasonable scale.

Deduplication is not a product category. It's not even a specific method. It's an effect: data size gets reduced by pooling redundancies. Depending on vendor, deduplication itself can mean significant variation in:

• Data reduction effects and predictability
• Delays and impact on I/O
• Applicability to applications and data types
• Resilience
• Overheads in hardware and competing system processes
• Throughput
• Replication flexibility and speed
• Packaging (slow backup software, inline storage, post-process kludge)

Production dedupe systems have weird optimizations that you can't find off the shelf. It is not a simple solution to get right, and if you start with the wrong architecture, it won't get fixed. It uses massive system computing and disk access overhead for hours. It is sensitive to the data patterns being input. It's also very sensitive to all the normal problems of storage systems; if there's data loss or corruption, a single block can affect hundreds of files. Vendors who are late to market will take shortcuts. In the end, these only show up in bad side effects. Under stress, the problems will be visible.

Dedupe has no particular packaging consistency. It has no particular interface for debate at standards committees. It's less like an "it" and more like a "them." This site shows just some of the divergent characteristics of some of the vendors.

Storage is not a commodity. Dedupe is not a commodity (it's not even a single category). Dedupe storage is far from being a commodity.

Keeping It Real

Over the last few years, it started sounding like deduplication was becoming commoditized and turning into just a feature. FalconStor, Sepaton and Quantum announced massive clustered dedupe projects in 2005-7 bolted onto existing VTL products. They lined up OEMs or acquirers, including HP, Sun and EMC. The specs were unbelievable.

If it is too good to be true, sometimes there is a reason. A year or more later, Data Domain is scaling as promised, but the bolt-ons are struggling to meet expectations in robustness and economic impact. Dedupe is an effect, not a method, and the methods are clearly not a commodity, as can be seen in examples like this.

Bolt-on specs are sometimes a wish list. For example, Quantum's DXi 7500 maximum speed specs need a dual-controller system. But that configuration is apparently not available, more than a year since it was announced. What else is lurking that could throw claimed specs into a new light?

For example, in that same system (the basis of the EMC DL3D 3000)

• Backups are written to a non-dedupe disk cache to get higher speed, except in the narrow case of the "adaptive" or "dedupe on" modes, in which writes that are a third or less of their benchmark rates will dedupe inlines; and
• All cached data is kept after deduping as long as possible, so benchmark reads are generally from the pre-dedupe cache to get higher speed (ask about their truncation model).

So their benchmarks could all involve the cache, not the dedupe store. What's the real dedupe rate? Can you know when dedupe on the non-dedupe cache will complete? Can you trust the specs? Time will tell.

Here is the problem. If dedupe matters, you have to be able to finish backup and dedupe in a day. A post-process system will fill up much faster than expected and/or get controller-bound and slow if it gets too far behind..If you replicate, the DR site will always be behind. It would really be a bummer if you didn't expect this.

So here's an experiment you can try at home as a reality check. (These products still don't have much of a reference base that will look like your situation, and without that, a reasonable buyer should do a proof of concept lab test.) If you do a bolt-on test, do it with a pair of large dedupe products, replicating across a LAN to avoid latency issues.

• Backup 30 or 40 TBs as fast as you can (or whatever they say they can dedupe in a day.) Bolt-on system designs can be very curious under stress, so don't go small.
• Time the duration from start of backup to finish or dedupe replication. The end-to-end rate of this test will be a fair proxy for the internal dedupe rate as long as it replicates only deduped data. (Good luck estimating dedupe completion time with their GUIs - it is often left as a daily calculus project.)
• Now try restoring from the replica to see how fast restores are; the replica only has deduped data, so it won't fake you out by restoring from the non-dedupe cache.

Data Domain's systems are easy: with inline dedupe, the only throughput is dedupe throughput.

Storage Vendors Stages of Grief About Deduplication

Deduplication is a challenge to most large storage vendors. IT means they get to sell fewer disk drives, and their business plans involve more drives. I expect most vendors will go through Elizabeth Kubler-Ross' five stages of grief in confronting this dilemma. You can already see it in practice in some of the biggest vendors.

Denial:

• Internally: Deduplication is too slow and it puts data at risk. It ain't real storage, it doesn't work. Their implementation is risky!
• Tell customers: How about a free VTL with your next SAN/NAS upgrade? Let's get lunch.

Anger:

• Internally: Who do they think they are? Dedupe is a fad! Real storage is tons of platters, spindles, lots of sheet metal. This whole industry is going to hell! We need to muddy the water. Buy or OEM something fast and hopefully cheap. Start blogging!
• Tell customers: How 'bout a deal on your next VTL expansion? Running out of space? Right now we can discount VTL really low. I wouldn't bring this up, but I see you're talking to Data Domain...Did I tell you we have box seats at the game next week?

Bargaining:

• Internally: We need to do something. Our sales people are screaming bloody murder! Can we resell something cheap? One size does not fit all.Toss something out we have lying around in the labs. We bundled it in enough deals to tell IDC we have market share, so stop bugging us.
• Tell customers: Please spend a month to review our VTL. Look, it has a little deduper in it. No? Take some more time, look at out esoteric backup software - it has that dedupe thing you have been asking about! No? We can get you a deal on it with that SAN upgrade we have been talking about. We totally understand your needs. Use some professional services hours. Oh, you need our challenging dedupe NAS. No?

Depression:

• Internally: Crap, this stuff isn't going away. What, they went public? We let the market decide, and it turned out we got weak products, by (choose one) trying to preserve and bolt on to our existing designs - or - we OEM'd weak products for time-to-market from vendors who were desperate. (Sigh)
• Tell customers: What's the difference anyway? It's just a feature! What? It's not? You need to test it? Look at this roadmap. This PDF says we check almost as many options as they do. I'm sure we have a reference somewhere...

Acceptance:

• Internally: Wonder if they are accepting resumes?
• Tell customers: We are working on something else, we need more time. Wait for us! I'm serious, our backup software may be less fussy now. We can talk over lunch.

Nothing Has Changed

Last week, Data Domain announced the highest performance inline deduplication system in the industry with the DD690. EMC and Quantum followed hastily with their own announcements.

Nothing has changed

EMC is repacking the Quantum 7500 in the EMC DL3D line. But the inline dedupe rate is still about 150 MB/sec., according to this discussion, after which the only rates we can know for sure are to and from their non-dedupe cache. So the effective capacity, the amount you can dedupe in a backup day, is something less than 13 TBs (dedupe rate x 24 hours). If you don't finish deduping in a day, it will compromise tomorrow's throughput and backup window. So the only way you could use 150 TB of this disk for dedupe is if your backup window is more than two weeks long.

Why offer 148-180 TB in this architecture? Because it is not a dedupe system, it is a traditional VTL (even if it has a NAS interface) -- a disk-based, short-term-retention I/O buffer for data on the way to tape. The extra storage baggage? Call that dupestorage. It is not meaningful for sizing dedupe or replication. Data Domain's dedupe speed and resulting effective capacity is greated by more than a factor of two.

Seems like everything's still the same:

• Data Domain is still the deduplication performance leader;
• EMC and Quantum are still selling traditional VTLs based on dedupe storage;
• EMC is still selling massively over-provisioned disk systems;
• Quantum is really trying to sell tape libraries above all.

Oh, I guess one thing has changed. Inline deduplication used to be considered too slow compared to a post-process. I guess that has changed. The non-dedupe write speed of the EMC DL3D 3000, at 1.44 TB/hour, s about the same as the inline dedupe speed of the DD690. If post-process isn't even faster than inline, why bother?