File Transfers Cost You More Than You Realize—What to Do About It

Moving data between applications is so common that it is easy to underestimate its actual costs. After all, how much can it cost to copy a few files? Actually, it's potentially quite expensive and challenging, especially considering the scope of file transfer requirements across an enterprise. To gain insights into these challenges, think about a few issues related to enterprise file transfer:

  • Common file transfer methods, and why they are inefficient and costly
  • Keys to controlling file transfer costs
  • Cost benefits of managed file transfer

Let's begin with fairly common characteristics of file transfer solutions.

Common File Transfer Methods Are Inefficient and Costly

Imagine you are developing or installing a department‐level or enterprise application. You are probably concerned with having enough processing capacity, storage, and network bandwidth. You are hoping you've captured all the access control requirements and configured security controls correctly as well as implemented sufficient logging to troubleshoot potential problems with the application. How much thought have you given to the fact that one of the system requirements is to transfer data into and from the application?

You might think transferring files is the easy part. After all, how hard can it be to write a batch file or shell script to copy a file? Even if you need to do a little pre‐processing or postprocessing, a quick Perl script of a Visual Basic program will do the trick. Compared with the other things you have to do to roll out the application, the file transfer part seems relatively straightforward. Now consider the consequences of the choice to use a one‐off ad hoc custom file transfer solution not once but many times in your environment.

The use of ad hoc custom solutions can result in a range of design choices that will make maintenance more difficult and make file transfers potentially less efficient and more costly than they should be. Ad hoc custom solutions too often:

  • Can be difficult to code well, especially with error handling and exception processing
  • Implement inconsistent logging, reporting, and alerting
  • May use insecure protocols
  • Require monitoring for mission‐critical file transfers
  • Can be difficult and time consuming to troubleshoot

Some of these realities might seem surprising unless you have had experience developing and maintaining ad hoc custom file transfer solutions.

Difficulties Coding Ad Hoc File Transfer Solutions

The first step to understanding the requirements of file transfer solutions is to dispel the myth that file transfer is basically a file copy operation. Of course, you will be copying the contents of one file from one location to another. And yes, copying a file from one directory to another directory is simple. Even copying a file from one server to another can be easily accomplished on most operating systems (OSs). That is, as long as nothing goes wrong.

When something does go wrong, the person performing the copy operations will have to evaluate the error messages or other results to assess the problem. The root causes of the problem could include:

  • Insufficient disk space
  • Access control violation
  • Improperly configured command parameters
  • Network timeout
  • Another application with an exclusive lock on the file

When a systems administrator copies a file and encounters a problem, she has the skills to diagnose and correct the problem. A quick‐and‐dirty Perl script does not. A robust file transfer solution will include extensive error handling and exception processing. In the ideal case, the program will be able to work around the problem. For example, in the case of a network timeout, the file transfer program might suspend the copy operation for some period of time before retrying. Doing so would handle cases where there is a temporary network problem. In cases where human intervention is required, the file transfer solution could alert a systems administrator with a detailed message about the problem as well as possible solutions.

Implement Inconsistent Logging, Reporting, and Alerting

File transfer programs developed in‐house and over extended periods of time will usually implement different logging, reporting, and alerting protocols. Newer solutions written by developers who have had to live with the shortcomings of older ad hoc file transfer programs will likely include better logging and reporting, at least for problems that a developer has seen before.

The quality of code in a file transfer solution will depend, in part, on the time and resources available to develop it. When time is short, error handling and logging might be sacrificed in order to finish coding core functionality.

Using Insecure Protocols for File Transfer

Regulations and good security practices might require the use of secure protocols when transferring confidential or private data across a network. Even if data is transferred within an organization's network, you might not want the data transferred in plain text for anyone in the organization to see.

Secure Sockets Layer (SSL) and Secure Shell (SSH) encryption can be used to encrypt files while they are transferred. If the developers coding the ad hoc transfer solution are unaware of secure protocols, do not know they are required, or are having difficulty setting them up (for example, problems installing SSL certificates on the servers), they may fall back on unsecure methods, such as ftp.

Monitoring Mission‐Critical File Transfers

How can you be sure an important file was actually transferred? If you do not trust your ad hoc solution or you are not sure how the program was written, you probably will not be willing to trust it to get the job done. In this situation, you start spending time babysitting file transfers.

Let's face it, you have better things to do, but if you are not confident you will be alerted in the event of an error and you do not trust the exception handling capabilities of the file transfer program, what will you do? For mission‐critical file transfers, you will have to monitor the transfer yourself or you might even be tempted to write a script to check the results of the file transfer program to make sure the transfer succeeds. Of course, doing so creates yet another script with all the potential problems of the first, including poor exception handling and error trapping.

Difficulties Troubleshooting Ad Hoc File Transfer Solutions

When different departments and different applications implement their own file transfer solutions, there will be difficulties troubleshooting these programs. There is no chance to apply what is learned debugging one program to the others. One program might use Linux shell scripts and Linux commands for file transfer while another uses a scripting language like Perl or Ruby with language‐specific commands. The same kind of error might generate different error messages.

Someone familiar with the error handling in one program might not understand how exception handling works in another. This scenario can even happen in organizations that standardize on a single programming language for file transfers. Perl is famous for its motto, "There is more than one way to do it." This is certainly the case for developing file transfer programs.

In spite of all these potential difficulties with file transfers, there are ways to control costs and improve the efficiency of your file transfer operations.

Keys to Controlling File Transfer Costs

Unless you are in an environment that has simple, low‐volume file transfer requirements, you might be able to better control costs with a single, common file transfer platform.

Key Features of a File Transfer Solution

When evaluating your options for a single solution, consider:

  • Security
  • Reliability
  • Scalability
  • Multi‐platform support
  • Management and reporting

Security is a key factor, especially in cases where compliance with regulations is an issue. Look for solutions that support encrypted file transfer. Also assess how well the solution allows you to control access to file transfer operations. If one person creates a file transfer workflow, another, unauthorized person should not be able to tamper with that workflow.

You should not have to babysit file transfers. Reliable solutions will handle minor challenges (for example, temporary network problems) and alert systems administrators to more severe issues.

Also look for solutions that can operate on the volume of data you will need to transfer. In many cases, you will have a window of time in which to complete a transfer. If transfers are not completed in that time, other operations will be delayed, potentially causing a cascading effect of disruptions. This effect could lead to financial penalties from missed provisions in Service Level Agreements (SLAs).

Multi‐platform support is a common requirement today. Although some organizations might be strictly Microsoft or Linux, it is quite common to have a mix of platforms. File transfer solutions should support point‐to‐point transfers across different platforms.

It is easy to overlook the need for management reporting when you are focused on ensuring that files are transferred successfully and in a timely manner. Management reports will help you understand performance over time and the number and types of errors that occur as well as help you assess trends that can affect larger workflows in the organization.

Key Use Cases for File Transfers

There are various ways file transfer solutions can support business processes, and some are probably in place in your organization today. For example, you might routinely transfer files internally between tightly coupled applications and between more loosely coordinated applications. You might also transfer files to external parties, such as business partners. Both of these scenarios are readily supported by available file transfer solutions.

Consider how emerging use cases might alter the way your transfer files or expand on the requirements you must support. For example, as organizations adopt cloud computing to cut costs and expand their capabilities, you might find yourself transferring large numbers of files to public cloud providers. Cloud computing offers the opportunity to use a large number of servers at relatively low costs. This setup can enable organization to analyze more data than they could have with their internal IT resources. The practice of analyzing "big data" with cloud computing can introduce new file transfer requirements that demand greater scalability and better management reporting than ever before.

Cost Benefits of Managed File Transfer

A managed file transfer solution can address many of the difficulties outlined earlier while meeting requirements of common use cases. Some of the most salient benefits of a managed file transfer solution include:

  • Rapid deployment of new file transfer workflows
  • Improved management and reporting
  • Reduced development and troubleshooting costs
  • Improved security
  • Improved responsiveness to changing business and technology conditions

Changes in business environments, markets, and technology lead to frequent changes in business operations. These, in turn, can dictate changes in workflows that depend in part on file transfers. A managed file transfer solution that supports multiple use cases, accommodates the scale of work required, and can be applied to a variety of uses is a valuable addition to your IT resources.

Summary

File transfers are more costly and time consuming than they might first appear. Developing custom ad hoc solutions often appears to be the fastest most efficient way to address file transfer requirements. However, over the long term, this may not hold true. The cost of debugging scripts with insufficient error handling, babysitting unreliable methods, and compensating for file transfer solutions that cannot keep up with demand highlight the need for a more reliable, comprehensive approach to file transfers.