By Steffen Koehler, RocketStream, Inc.

2008 will see accelerated managed file transfer being implemented not just by end users, but also actively marketed as a value-added service by the hosted server companies themselves.

The past several years have seen large increases in data sent over IP networks, including the public Internet. This traffic increase has been the result of not only consumer-oriented media, but also a rise in corporate data transfers. For example, the health care industry transfers high-resolution graphic images such as x-rays; the entertainment industry transfers high-definition films between studios; the financial sector transfers large amounts of account data; and scientific institutions exchange extremely large data sets. Simply for backup and recovery purposes, virtually every industry transfers extreme amounts of data to and from remote servers. In response to this increase in data transfer, companies and Internet Service Providers have been connecting users on higher-bandwidth pipes than ever before.

To help manage the large file sets being transferred, both hardware- and software-based managed file transfer (MFT) solutions have emerged. These products automate the file transfer process, and in some cases, eliminate redundant data transfer. Despite this data management, ever-larger files often need to be transferred.

As a result, the increases in connection bandwidth and file sizes, other bottlenecks in data throughput are starting to be uncovered, especially for data transfers over long distances. Specifically, the TCP and HTTP protocols were designed with small files and pipes in mind; when these protocols encounter significant network latency, they throttle back the data rate. This occurs even when the latency is simply due to the long distance that the data needs to travel. Consequently, North American coast-to-coast data transfers are typically limited to about 3 Mb/s, and trans-Pacific data transfers are often less than 1 Mb/s, regardless of the available bandwidth in the pipe.

Hardware-based solutions to this latency-induced slowdown have been introduced in recent years. These so-called WAN accelerators often include both MFT functions and acceleration, but they have been very expensive and, of course, have required new hardware installation.

While software-based MFT solutions have been available for sometime, software-based acceleration is only now coming to the forefront. Software accelerators typically replace the native TCP protocol used by legacy file transfer mechanisms like FTP with proprietary protocols that are more immune to network latency. Very recently, software-based accelerators have begun to include automation features from the MFT domain. For example, the RocketStream file transfer acceleration suite is capable of monitoring "hot folders" to automatically initiate accelerated file transfers, as well as automatically synchronizing files and folders between disparate locations and scheduling accelerated file transfer tasks for pre-determined times.

A particular trend that is emerging now is the adoption of such accelerated and managed file transfer in hosted server environments. Such third-party server services are often geographically remote from the customers who use them. With large file sets being transferred over large pipes to and from these hosted servers, an acceleration solution is required, and since the end user does not have physical access to the servers, a software-based solution is necessary. Alliances are forming between the software manufacturers of these solutions and the service companies hosting the servers. The result is a versatile, high-value service that is simple to implement and cost-effective.

Steffen Koehler is the chief marketing officer for RocketStream, Inc. www.RocketStream.com