News Analysis: Lessons Learned from Microsoft/T-Mobile’s Smartphone Data Loss

Tape Evangelism

By Mark Ferelli

How often do data center managers need to be reminded of how important backing up data is? The history of IT shows the data loss results due to fire, flood, earthquake, hurricane, terrorist assault and plain old human error. But even the brightest and best believe that they can get away without working backups. Data center managers should take lessons from the teachable moments that erupted last October, when thousands of users of the Sidekick smartphone not only sustained data outage, but unrecoverable data loss. The joint press statement that Microsoft and T-Mobile put out included a statement that no service provider should have to write:

“Regrettably, based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device—such as contacts, calendar entries, to-do lists or photos—that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger.”

After the event and the joint statement, speculation was active regarding the cause of the massive, crippling data loss that struck countless Sidekick users. Was it strictly a server failure? Was it a SAN upgrade attempted without a backup in advance? Whatever the cause...in the end, only one thing is crushingly clear: Microsoft/Danger did not have a working backup.

Inexcusable. No IT manager with more than 10 minutes on the job could see that much data being stored without backing it up, preferably in multiple locations and using removeable media like tape that could be relocated to a safe venue as needed. The smart money is on three copies: a primary copy, a secondary, and an archive copy.

Does this bewildering negligence mean the end of the smartphone? Probably not, although it will be some time before Microsoft/Danger or T-Mobile will recover customer confidence. Does it mean the end of cloud computing? No, but perhaps it will put an end to the naïve notion that data stored “in the cloud” is stored with sufficient redundancy and/or fault tolerance to make a catastrophic data loss impossible.

Now...those teachable moments. What lessons can we extract from this embarrassing debacle? Data center management that is tempted to complacency should take note.

1.Data should not live in disk alone. In spite of all the claims to the contrary, data cannot be considered perfectly safe using hard disk alone. Server failures are commonplace, and disk mechanisms themselves are fragile. Disk failures are never a matter of “if” but of “when.” And if it proves true that the Microsoft/Danger failure was due to a botched SAN upgrade, the precarious nature of unsupported disk storage alone seems clear.

Tape backup and archiving would have saved untold numbers of dollars and endless amounts of goodwill. Tape is always found in installations where data recovery is desired more than a perceived need for speed. Reliability is at an all time high, 700% improvement over 10 years by one report. The popular lie that “tape is dead” is believed only where punditry is king. I am confident that Sidekick users would have been happy to wait the time needed for a tape restore rather than lose the data.

2.Cloud computing is no replacement for data management. Too many IT managers believe that the value proposition for cloud storage is that it is self-managing, self-balanced and self-replicating. It is a charming fantasy, but here is the elephant in the living room: hardware will break. Look with me at hard error rates for a moment. Enterprise SATA disk drives offer a hard error rate of 1 in 10E15. Enterprise FC/SAS drives: 1 in 10E16. LTO Tape: 1 in 10E17.
If you believe that cloud computing can replace the need for multiple copies of files backed up, checked and verified, ask Google. Ask Amazon. And now, ask Microsoft/Danger. External “clouds” have limited usefulness for the enterprise data center, absent a mature backup and archival strategy.

A Tiered Target

It may be more rational to consider cloud storage a target in a tiered storage architecture, similar to the D2D2T model. Tiered storage is a way of reorganizing corporate data onto a variety of storage media.

Tiering involves the selection and implementation of storage systems, the software to manage and optimize that storage, as well as the policies and procedures needed to operate each tier.

The concept of tiered storage is to better align data on storage devices with its value or importance to the organization, The idea is to retain the most timely or important data on fast, high-I/O storage, and move less frequently accessed, but no less important data, to a reliable, removable tape system.

There’s been quite a bit of misdirection about tape over the past several years, mostly strewn about by a few disk-only proponents who spread misinformation about tape’s true value in data protection and disaster recovery. They claim that disk, including cloud storage disks, can replace tape for all practical purposes and is cost competitive for storing data. But tape storage continues to play a vital role for backup, recovery, compliance, long-term data retention, and data protection in the vast majority of businesses. One IBM tape customer said: “You’re out of your mind if you think you can live without tape.”

Cloud storage, when the underbrush is cleared, shifts storage requirements and service levels to someone else, somewhere else. Bits and bytes, files and folders have to come to rest at some place. The questions of reliability and data security do not go away, just because you have stored data in the cloud. Anywhere data comes to rest can become data at risk. The question will always persist: What happens to your data if there is a disaster at the online cloud venue? What if it is only in one location and flood/fire/earthquake/theft/ degradation of storage media then happens at the online disk backup? These questions were answered in that pain-filled admission from Microsoft/Danger and T-Mobile.

Tape Marches On

In defiance of the pundits, new developments in tape technology will extend the useful life of tape archiving well into the future. In partnership IBM and tape media developer Fujifilm have new tape material and a novel tape-reading technology. In combination, they can store 29.5 billion bits per square inch, which brings to the industry a tape cartridge holding around 35 terabytes of data--more than 40 times current offerings.

The IBM/Fujifilm team used a barium ferrite formulation, positioning the magnetic particles in such wise as the magnetic fields stand perpendicular to the tape substrate, and opposed to conventional lengthwise orientation. The result, more bits per square inch. It also gives magnetic fields greater strength. Barium ferrite permits a thinner base film, which in turn means more tape per cartridge. To compensate for read reliability challenges that can come up in this kind of high density format, IBM developed new signal processing algorithms that process the data and also predict the effect that hash (EM noise) will have on subsequent readings.

A data loss event on disk will take place, be it in the cloud or in your own data center. You cannot fully automate your responsibility to manage your data. The Microsoft/TMobile experience, just like the Google and Amazon data losses before it, suggests that even the burnt hand might not teach. So take heed of the oldest lesson in IT: back up your data. And tape will do the job.