FileCatalyst’s Acceleration Technology: How it Works
FileCatalyst’s core transport technology is based on the UDP protocol, which provides a mechanism for transmitting data at precise rates. And to keep file transfers moving quickly, FileCatalyst breaks data into blocks as files are encoded rather than waiting for each transfer to be completed.
FileCatalyst Improves Upon UDP
The benefit of the UDP protocol is that files can be transferred without any delay or obstruction by network impairments such as latency or packet loss. However, UDP does not have any means of recovering from lost packets, which meant that UDP was not historically a good candidate for reliable transfers over a network with impairments.
FileCatalyst adds the reliability and rate control missing from UDP without sacrificing its other desirable properties.
TCP vs. FileCatalyst
Like TCP, the protocol behind traditional file transfer methods like FTP, FileCatalyst breaks data into blocks.
The major difference from TCP is that FileCatalyst eliminates the delay while waiting for receipt of a block of data before starting the transfer of subsequent data blocks. Instead, the transmission of subsequent blocks is initiated immediately, even when previous blocks have not yet been acknowledged. Regardless of network latency, data transmission remains constant with FileCatalyst, enabling transfers to occur at full line speed.
While the subsequent block is being transmitted, the FileCatalyst protocol awaits either acknowledgment of the previous block or a list of missing packets. Missing packets are re-transmitted concurrently with new data being sent for subsequent blocks until the acknowledgment is finally received. This flow of data remains constant, even when there are many missing packets, and the process repeats until the entire file is transferred.
Solving the Problems of TCP with FileCatalyst
Both FTP and HTTP are based on TCP/IP, the backbone of the internet. For loading a webpage, TCP is great. However, it’s far from ideal for sustained data transfer. FileCatalyst’s accelerated file transfer protocol offers incredible speed gains versus traditional file transfer methods.
TCP is Serial, Which Increases Latency
The time it takes to send a packet and receive an acknowledgment is called the round trip time (RTT) or simply the “latency.” TCP is serial in nature, meaning that the next piece of a file won’t be delivered until the previous one is acknowledged. This creates a period of “dead air.” The higher the latency, the more dead air.
Imagine Spending More Time Waiting for Acknowledgments Than Sending Data
The amount of data that can be sent before waiting for acknowledgment is governed by a mechanism called the “TCP Sliding Window,” which closes when congestion is detected and is slow to re-open to its maximum. TCP’s congestion detection throws many false positives, meaning that the window often closes without justification.
The resulting dead air means that with TCP, your practical transfer rate can be orders of magnitude slower than your line speed. Assuming packet loss of 0.1% and a line speed of at least 10Mbps:
- Once you hit 50ms of latency, your transfer speed starts to take a major hit
- By 80 ms, your maximum speed is 2500 Kbps—even for links 1Gbps and faster
- By 400ms—not uncommon for satellite connections—you are crippled at 500Kbps
Why Use FileCatalyst
The FileCatalyst protocol offers such large speed gains over FTP because it is based on the UDP protocol rather than TCP. Files can be transferred over the UDP protocol without being impeded by network impairments such as latency or “dead air.”
Why Use FileCatalyst
The FileCatalyst protocol offers such large speed gains over FTP because it is based on the UDP protocol rather than TCP. Files can be transferred over the UDP protocol without being impeded by network impairments such as latency or “dead air.”
What is Accelerated File Transfer?
While we could all use faster file transfer, there are practical limitations to acceleration technology. At low speeds, for example anything up to 5 Mbps (Cable/DSL) with normal latency (80ms and below for transfers within one country), FTP provides nearly maximum speed. On the other hand, there are also scenarios that improve dramatically when file transfer acceleration is introduced, expediting transfers that once took days into just minutes.
File transfer acceleration delivers the greatest results in situations where there has been a higher than average investment in line speed, files are exchanged over large geographic distances, or when connections which are prone to packet loss are used.
Terms to Know
-
Line Speed: TCP (i.e., FTP, HTTP, etc.) has built-in practical speed restrictions. If TCP is maxing out at 4Mbps, and your link is 5Mbps, you won’t notice. However, at high speeds 4Mbps is a small percentage of the line speed available. Usually, the higher your base connection speed, the more you need acceleration.
-
Geographic Diversity: Distance adds latency due to satellite connections and multiple “hops.” Whereas TCP performance takes a dramatic hit with latency, FileCatalyst acceleration is immune to it.
-
Unreliable Connectivity: Connections such as satellite links are prone to packet loss. With TCP, dropped packets cause the throughput to be reduced dramatically as it misinterprets this as network congestion. FileCatalyst lets you determine how it should react: do you want to power through and retransmit lost packets, or choose to tune to slow down?
Do You Need Faster File Transfer?
Any organization that exchanges large volumes of data and requires quick, efficient file delivery, yet is unable to achieve acceptable speeds with traditional file transfer methods has a need for accelerated file transfer. Regardless of the industry, file formats, and parties involved, accelerated file transfer can be beneficial to any organization unable to effectively handle its file transfer via methods like FTP or email attachments.
There are certain characteristics that make an organization well-suited for file transfer acceleration solutions, including the following:
Large File Sizes & Big Data
Any organization transferring files with sizes in the gigabytes, or big data, on a regular basis will experience the common drawbacks of using traditional methods of file transfer. These issues include lengthy transfer times, failed transfers, or file size limitations preventing transfers. Accelerated file transfer solutions can send file sizes that go into the terabytes effectively, and at full line speed.
Time Sensitive Information
In situations where deadlines are critical, such as live broadcasts, military updates, or medical analysis, organizations often rely on accelerated file transfer to ensure files reach their destination as quickly as possible. File transfer acceleration solutions have built-in reliability to ensure files get to where they need to go on time.
Geographic Dispersion
When organizations operate on a global scale, file sharing becomes a logistical concern. Sending files across the country or overseas creates unique challenges for speed, management, and tracking. File transfer acceleration solutions can act as a centralized hub to streamline organizational file transfers by helping ensure that global file exchange is fast, simple, and transfers execute at full line speed regardless of geographic location.
Remote Locations & Satellite Connections
Transferring files, especially large volumes of files, from remote locations or via satellite connections is often a daunting task. Transfers can be lengthy, and connections are frequently dropped, further slowing delivery. Accelerated file transfer solutions can make the most of even the most precarious connection types to send files quickly, while at the same time leveraging reliability features to guarantee files reach their destination.
Significant Line Speed Investment
Installing a high-speed connection for increased IT efficiency is a large investment. Traditional methods of file transfer are often unable to make the most of fast line speeds, using only a fraction of the connection speed when transferring files. Accelerated file transfer solutions can saturate the line, meaning that if a 10 Gbps line is present, transfers are able to take place at the full 10 Gbps.
Can File Transfer Acceleration Improve Your File Transfer Process?
-
Is my line speed faster than 10 Mbps (i.e. faster than Cable/DSL)?
-
Is above average latency present (i.e. transfers between countries or overseas)?
-
Is my network fragile and prone to packet loss (i.e. a satellite or wireless connection)?
If you answered “yes” to any of the above questions, acceleration is likely to be very helpful. If you answered “no” to all the above, FileCatalyst may still solve other file transfer issues facing your organization.
The benefits of file transfer acceleration: See how FileCatalyst improves the speed of a 10 GB media file being transferred over a 1 Gbps connection versus a 10 Gbps connection.
Will File Acceleration Benefit Your IT Infrastructure?
With FileCatalyst’s software-only approach to file transfer, deployment is fast and simple, ensuring the following returns are quickly realized:
Maximize Organizational Productivity
Optimize Existing Infrastructure
Simplify International Collaboration
Secure Confidential Data
Streamline IT Administration
More Than Speed: The Many Benefits of File Transfer Acceleration
Common reasons to move from TCP to a better alternative include:
Reliability
Anyone who regularly transfers large files over TCP can tell you about the anxiety they face each time. Will the transfer fail out after several hours? If the connection is dropped, will the transfer resume properly? How many attempts will it take before the file arrives intact? FileCatalyst dramatically improves reliability thanks to:
- On-the-fly or per-file verification using configurable MD5 checksums
- Configurable retry and resume options to pick up transfers after losing network connectivity.
- Checkpoint restarting with built-in MD5
- User-definable socket timeouts to accommodate extremely poor network conditions
- Efficient re-sending of dropped packets—and only the exact packets being dropped
- Server-driven alerts to notify stakeholders of successful, failed, or canceled transfers, keeping them in the loop in the rare event that manual intervention is required
- Configurable transfer characteristics such as packet size as well as the number of senders and receivers
Security
Although FTP and SFTP have security options, these are not always enough for security-minded organizations. FileCatalyst offers several security features to safeguard and speed up file transfers. In terms of encryption levels and authentication methods, a FileCatalyst solution is a perfect fit for organizations looking to meet mandates such as SOX, HIPPA, and GLB with security features including:
- Encryption with industry standards, including SSL for control channel and AES for data
- Password policies
- Complete visibility into all activities on the server
- Brute force attack protection and real-time IP banning and account blocking
- Filter by IP address for either inclusion or exclusion
- Notifications about suspected threats via FileCatalyst Central
- Authentication via LDAP(S), Active Directory (AD, ADS), or database
Tracking & Auditing
Using FileCatalyst Workflow, every transaction in and out of the system is tracked at a global and user account level. This means tracking extends to custom statuses based on the data’s progress through your organization’s workflow.
FileCatalyst Server also enables users to pull reports of transactional data into preconfigured graphs, or CSV files, to use with your own tools. These reports are available from a FileCatalyst Server used with any product in the FileCatalyst family.
Delivery & Email
FTP and HTTP lack sophisticated ways to easily share files with arbitrary individuals or groups. Email is the typical go-to for this purpose: just select recipients and attach files. However, attachment sizes and delivery tracking are limited. FileCatalyst Workflow solves these problems by providing tools to let you send files of any size to anyone with an email address. Moreover, the entire history of delivery and receipt is tracked.
Many organizations choose FileCatalyst to solve the problem of slow TCP-based transfers (including FTP, HTTP, and their secure variants), but speed is not the only issue related to TCP file transfers.