Open-Source File Transfer vs Secure Managed File Transfer: Pros and Cons Part Two

Over the last few years many articles have been written comparing commercial off-the-shelf Managed File Transfer (MFT) products to open-source and home-grown solutions, including GoAnywhere MFT’s own article on this topic: Pros and Cons of Open-Sourced Managed File Transfer Software.

There are many claims both for and against open-source solutions in these types of articles. As a more technical member of the MFT team, I want to unpack some of the claims being made in other articles by looking into exactly how, if asked, I would create a specific feature using open-source MFT and how it would be either superior or inferior to a commercial solution.  

Lastly, I’ll review the claims being made in our original article within the context of implementation ideas I put forward here.

Building an Open-source MFT Solution is Not for Rookies

If we were to build an open-source MFT solution from scratch, we first need to make a few assumptions. My first assumption is that any open-source MFT solution will be built by someone with an excellent and intimate knowledge of Linux, scripting, networking, and security.

My second assumption is that such a person or a team of people, will be allowed to maintain, expand, PEN test, and monitor this MFT solution for the duration of the lifetime of the MFT application. In other words, one cannot just build an MFT solution and leave it alone — continual maintenance, updates, and documentation will be required for the lifetime of the MFT system.

Here we go:

• Start with a recent Linux distribution (distro): A Linux host will provide User management and authentication, SFTP, FTP and FTPS service, file management, logging, etc. I am most familiar with Ubuntu Distribution but any recent Linux distro with long- term support (LTS) support could do the job.

• Web-based management and web user interface: Using Webmin or VirtualMin. Webmin, and its derivative project VirtualMin, are web-based system administration tools for Unix-like servers and services with about 1,000,000 yearly installations worldwide. Using this tool, it is possible to configure operating system internals, such as users, disk quotas, services, or configuration files, as well as modify and control open-source apps, such as BIND DNS Server, Apache HTTP Server, PHP, MySQL, and many more.

  Both Webmin and Virtualmin allow you to create a web management console for end users and/or admins. And both solutions offer a web-based file manager. I would use webmin file manager as a starting point to customize and configure webmin to allow for a very basic web-based file manager for end users.

• Use CRONTAB for scheduling: Crontab is a powerful utility in Unix-like operating systems that allows users to schedule and automate tasks. It is particularly useful for repetitive tasks such as system maintenance, backups, and updates.

  Crontab stands for "cron table" because it uses the job scheduler cron to execute tasks. Crontab can execute any Linux Bash, Perl, or Python Script. Webmin offers a very comprehensive admin module for crontab, so I would use the crontab Webmin admin module as a starting point to set up all my automation scripts in my brand-new, open-source-based MFT solution.

• Use internal Linux scripts to trigger automated jobs: Linux comes with a plethora of scripting options, BASH, Ant, Perl, and Python are just a few of the popular options. The biggest challenge here will be to standardize which scripting language to use and how the scripts are written and structured. This will be the weakest point of the open-source home-grown solution. Every admin can write scripts in their own way using their favorite technology stack. Over the years, this functionality will become hardest to remain standardized.
• Use Linux logs for auditing and logs: Logging in Linux is very well built and defined. All the logs generated by events on a syslog system are added to the /var/log/syslog file. However, depending on their identifying characteristics, they might also be sent to one or more other files in the same directory. If required, the Linux syslog can be configured to send the logs to an external Security and Information Event Management (SIEM) system as well.
• Password complexity management: Password complexity is enforced by the pam_cracklib module. To modify the password policy for your Linux host, you will need to modify your /etc/pam.d/common-password file.

• Use SSHD to control access keys: The OpenSSH server supports various authentication schemas. The two most popular are: Password-based authentication and public key-based authentication. An alternative is to require both methods. Create the ssh key pair using ssh-keygen command. Copy and install the public ssh key using ssh-copy-id command on a Linux or Unix server.

  Management of access keys will require special attention and strict standardizations, so you do not end up with ssh keys being peppered all over the Linux host.

• Use inotifywait tool to monitor folder state: Set up recursive directory watches on the entire tree to allow you to execute a command when something changes. If you just want to view the changes, you can add the -m flag to put it into monitor mode. To monitor folders on windows servers, use SMB mount-to-mount Windows folders on Linux. We can also use the inotify tool to tell us when a file is uploaded to the server. inotify, event code IN_CLOSE_WRITE, would be the closest option – much like triggers in GoAnywhere MFT.
• Encryption at Rest: Use packages such as LUKS or Crypt for Encryption at REST. Safe keeping of pass phrases and/or keys will be especially important so you do not accidentally lose access to the files.
• For email notifications use sendmail or postfix: Both SMTP email packages can be configured to send emails via cloud email services such as GMailor Office 365. Both SMTP packages can also send out emails directly.
• Database: Linux can run several popular databases, including MySQL, MariaDB, and Postgres. So far, every solution I listed above either did not require a database or was using the internal user management built-in to Linux via PAM (Pluggable Authentication Modules). However, should a database be required for some specific MFT functionality (automation scripts perhaps?), I would recommend MariaDB/MySQL with a Webmin management module.

• Reverse proxy or gateway: The most important security features used by IT today is to separate the public DMZ network from the highly secure local network. Gateways or reverse proxies are usually deployed in the DMZ. They act as a “ghost” MFT server in the DMZ that is used as a gateway for the MFT solution deployed on the secure network. HAProxy will be the best option. In case usage of HAProxy is not possible. For web traffic, one could use NGINX or SQUID.

  I did not find any open-source reverse proxy projects specific to SFTP/FTPS, however a clever use of iptables and SFTP configuration could be used to create such a service in the DMZ. The big challenge will be to correlate logging between the gateway machine in DMZ and the MFT server on the secure network.  This feature will require writing some custom log processing scripts so all connections coming from the public internet are correlated with the connections on the DMZ server.

• Clustering for high availability: Message Passing Interface (MPI) is the standard communication protocol used for distributed parallel processing in Linux. Great care will need to be taken to set up MPI for two or more Linux servers running such an MFT service. The clustering task becomes even more challenging when a reverse proxy or a gateway is required. More research would have to be done on how Webmin would work in such a clustered environment. It is entirely possible that we might be able to put together only an Active-Passive system.
• Multi-Factor Authentication (MFA): Both Webmin and VirtualMin can be configured to force MFA on log-in. Details such which authenticator to use (Google, Okta, Azure AD etc.) will have to be decided at design time, but most MFA or 2FA authentication schemes are supported. SFTP/FTPS protocols are not web based and do not support web authenticators, but password/key authentication is supported in both protocols.

Open-source vs Commercial, Off-the-shelf MFT solution Comparison

So, with this rough “blueprint” of an open -source MFT server in mind, let’s look at some of the claims from our own Open-source vs GoAnywhere MFT article:

Claim 1: It’s free

There is no such thing as “free” in the software world. If you dPros and Cono not pay for a commercial license and support, you will end up paying more for IT staffing and for business continuity plans. At the minimum, maintaining a home-grown file transfer solution would require at least one full-time IT staff adept at security. 

Read More: Secure File Transfer vs Free File Transfer: What’s the Difference?

Claim 2: It’s got decent functionality

Agreed to a point. The basic functionality of an open-source solution will be comparable to a commercial product.  However, that open-source solution might feel a bit more disjointed as it uses multiple different open-source projects configured to work together as a single MFT solution.

Claim 3: Community-based support and development

Indeed, most open-source projects do offer some community support, see claim #5 re: “There’s no guarantee of effective (or free) support” for more about this topic.

Claim 4: You can develop the solution to your unique needs

This point is a double edge sword. Yes, you can develop something custom, but once we get into the source code of some the open-source packages I mentioned above, the implementation complexity level jumps exponentially higher as well. Modifying the source code will require more than just a Linux admin. Software developers will have to get involved, which will also increase the implementation and maintenance cost of that home grown MFT solution.

Claim 5: There’s no guarantee of effective (or free) support

There are lots of free open-source, community-driven support websites around the world. WARNING: Any “dumb” or rookie questions posted on these community portals will be either ignored or worse, laughed at by the community members. To get any useful feedback, the questions must be posted by someone with an already good working knowledge of the problem. The MFT admin must ask questions correctly and concisely and provide lots of background information. Even the most concise and clear questions do not guarantee a timely answer. This support process is far from free as the salary of the experienced Linux admin will have to be covered by the organization that chooses an open-source route.

Claim 6: It’s time consuming to set up and maintain

Congratulations! If you made this far into this article and read all my ideas on how an open-source MFT server could work, my guess is you’d conclude that this is not a simple project and will require both technical and project management skills.

Claim 7: It's potentially less feature-rich

My solution outlined above, does not take into consideration other auxiliary features that a commercial MFT solution, like GoAnywhere MFT, usually include, such as Object Storage integration, third-party app connectors, clustering, partner key management, partner SLAs, MQ messaging, multi-tenancy, and FIPS 140-2 certification, etc.

Claim 8: It likely lacks security/auditability features

Professionally designed logging and auditing on a Linux can be very verbose and useful. Other security features such as enforcing the correct version of ciphers and multi-factor authentication might be much more difficult to set up with an open-source system. Updating the solution for any newly found CVEs will also have to be addressed. Compliance requirements such as FIPS or Common Criteria will be especially difficult to implement.

Read More: Why Common Criteria Certification Matters

Claim 9: It could disappear

I would totally disagree with this statement: “Open-source software is the cowboy of the IT world: It’s wild and free—and could hit the road at any time. “All open-source software packages I mentioned in this article are stable and proven community projects with a highly active user community. I would worry more, however, about the people that built and designed the open-source MFT solution. As people get promoted or change jobs, maintaining the stability of the system will be difficult. Maintaining automation scripts will be particularly challenging in the long term. And documenting the home-grown, open-source system will be particularly important for the project's longevity.

In Conclusion: Open-Source vs GoAnywhere MFT for File Transfers

With so many commercial off-the-shelf options for MFT today, a decision to build instead of buy, is equal part hubris, ignorance, and naivety of the IT staff making this decision. The path to a built in-house MFT solution is complex, does not offer long-term stability and might be missing some key security, automation, and compliance features.

Five Reasons to Choose GoAnywhere MFT

1. GoAnywhere offers cross-platform flexibility for Windows, Linux, Mac, and IBM i.
2. GoAnywhere can address key cybersecurity certification needs including FIPS, HIPAA, GDPR, and Common Criteria.
3. GoAnywhere supports many protocols: SFTP, HTTPS, FTPS, FTP, PeSIT AS2, SMB, Webdav, etc.
4. GoAnywhere provides unparalleled scaling and load balancing
5. GoAnywhere gets regularly PEN tested and all third-party CVEs are regularly addressed