An Efficient Data Transfer System for Grid Applications
The FileMover is a data transfer system designed to optimize the transfer of potentially very large files. This system relies on an overlay network architecture, where a set of machines cooperate in the transfer by forwarding among them portions of the files being transferred. Data transfer times are minimized by choosing, for each transfer, the set of relays that maximize the expected throughput. Preliminary experiments show that the FileMover is able to profitably exploit existing network paths not chosen by IP routing algorithms, thereby enhancing file transfer performance.
How the FileMover works
The File Mover provides a data transfer service between a File Server and a Client by using a suitable set of File Relays as intermediate nodes. A client that needs a file contacts one of the File Proxies, that requests the file to the server and notifies the client when the transfer has been completed.
The overlay network is fully connected, that is each File Relay may communicate directly (via a TCP connection) with any other File Relay.
In order to minimize the time taken by file transfers, each Server, upon receiving a file transfer request, computes the best path
: the sequence of relays from the server to the proxy
that is expected to provide the highest possible throughput for the whole duration of the transfer.
Throughput forecasts are computed by measuring with Iperf
the available bandwidth on each link between Relays, and providing these measurements to the statistical forecasting algorithms of the Network Weather Service (NWS
More details about FileMover can be found in the publications section.