Transferring Large Files: UDP-based File Acceleration Solves Headaches

Sending big files can cause big headaches. And with the expectation of large, growing, and streaming files arriving unscathed, these headaches can become a chronic condition without putting robust software in place – software designed from the bottom up to address the issues that come with sending super-sized files around the world.

Why Large Files Can be Problematic

Big data sets, or large files such as music, live sports, and other broadcast media files can encounter any number of roadblocks in transit, including:

• Slow transfer speeds: The bandwidth needed to send extra-large files is substantial. Add the distance factor of sending files between two locations that may be continents apart, and you need a solution that can reliably work around any bandwidth limitations.
• Unsecure transfers: With business-critical, sensitive data or data that needs to meet compliance requirements for secure transfers, a large file that is intercepted or compromised while in transit can pose a security risk.
• Unreliable networks: If a network is unstable, such as in a remote location or even a ship-to-shore situation, large files are more subject to failure or be disrupted when being transferred. The transfer can then potentially be incomplete or corrupt and would need to be resent.
• Human error: Sending large files can often be a cumbersome process or require multiple specialized software solutions or permissions to execute the transfer. If a solution isn’t easy to use, employees can turn to open-source or unsecure methods to send large files and this can put your organization at risk for misuse of the files, or even a data breach. One can only imagine the fallout of accidentally sending a massive file, only to discover it was the wrong file. Hours, if not days can be lost, to say nothing about reputational harm.
• Lag in processing time: Large files that require compression to send can experience substantial lag, a critical factor especially when sending live video files.

Read More: Overcoming Distance and File Size When Transferring Files

How UDP-based File Acceleration Helps Transfer Big Files

Large file transfers require the power of a UDP-based file acceleration solution to avoid corruption, lag, or failure, speeding large file transfers over long distances in mere minutes instead of hours. 

Robust file acceleration software utilizing both UDP (User Datagram Protocol) a connectionless protocol and the TCP (Transmission Control Protocol), a protocol that requires a handshake or connection to start a transfer, helps resolve the aches and pains that typically accompany big file transfers.

Fortra’s FileCatalyst, a patented UDP-based accelerated file transfer solution, combines the speed of UDP-based technology with TCP, which delivers reliability and file integrity, even when the environment exhibits high latency or packet loss.  With SFTP or HTTPS, TCP is used for everything, auth, setup, and teardown, as well as the data transfer, which is slow which is slow. FileCatalyst still uses TCP for authorization, setup, and teardown, but it uses UDP for data transfer to achieve faster speeds.

Why UDP Alone Isn’t Speedy or Scalable

File transfer solutions that rely solely upon UDP can lack speed and scalability due to the nature of UDP, where packet transmission is not guaranteed as these UDP-based solutions must track individual data packets to ensure delivery, and if a packet is lost, must send a message back to the sender to request retransmission. If not managed efficiently, performance will degrade substantially when packet loss is high.

FileCatalyst overcomes these issues, however, as it ensures a constant flow of data, with no need to wait for acknowledgement before proceeding. And as new data arrives, it is always retransmitting concurrently, again speeding up the transfer, even when that transfer is substantial in size.

How FileCatalyst Supports and Speeds Large File Transfers

To achieve virtual speeds faster than actual line speeds, with speeds up to 10 Gbps, FileCatalyst deploys:

• A patented UDP-based protocol: This protocol is faster than other protocols when transferring large data sets, even when there is high latency or packet loss over the network
• Compression on-the-fly: Optionally, compression can be applied to highly compressible files such as databases or large files comprised of mostly text, to further accelerate transfers. If applied, the data is compressed as it is sent over the network in real time, eliminating the time-consuming compression or decompression at the beginning or end of each transfer. It uses the same principles as WinZip, Gzip, and other compression utilities, but without interruptions to the data flow, which adds to the total time to transfer.

Once at the recipient, files are decompressed and automatically stored in their original formats.

• Multiple TCP Streaming: When UDP is not possible in strict networks, where only TCP is allowed, FileCatalyst can still accelerate file transfers by running them through multiple, concurrent TCP streams, coupled with on-the-fly compression.
• Delta Transfers: If a file remains unchanged since last transferred, FileCatalyst can send only the “deltas” (incremental differences) of the file, rather than resending it in its entirety.

Calculate UDP Transfer Speed

To calculate transfer speeds your organization may encounter, input your source and destination, along with file size and line speed into this file speed calculator.

TCP streaming acts as FileCatalyst’s safety net, allowing file transfer acceleration to occur even when UDP is not possible. Additionally, FileCatalyst supports delta transfer algorithms which let users revise files and send just the revision made instead of sending the entire file. Once the revision arrives at its destination, the original file is modified.

Advantages of UDP-based Protocols with TCP for Large File Transfers

FileCatalyst can maintain a smooth file transfer speed even with network impairments. While TCP-based protocols have a peak and valley-type transfer speed, which gets worse as geographic distance grows.

By dividing large data into smaller data chunks or packets before they are sent, TCP reassembles these separate packets at receipt. This matters because even if there are network limitations, or reduced bandwidth, the smaller pieces of data can still be sent quickly and reassembled in the correct order without missing parts or corruption.

FileCatalyst combines the reliability of TCP with the speed of UDP into a proprietary algorithm to accelerate files and increase reliability – a necessity with large or super-sized file exchanges.