Get the Most Out of GoAnywhere: Agents

Andrew: I'd like to quickly introduce Dan Freeman again. He's our senior solutions consultant here at HelpSystems, and he's going to kind of dive into the topic of Agents for us today. Dan, you there?

Dan: I sure am, Andrew. Can you hear me?

Andrew: I can. I just passed you over the control, so why don't you go ahead and jump in.

Dan: All right, sounds good. Thank you very much, and thanks to all for tuning into today's webinar. This is our second of our Get The Most Out Of GoAnywhere series. A few weeks ago we did talk about Advanced Workflows to kick it off, but today we will be focusing on Agents.

Agents Overview

Okay, so what comes to mind when talking about agents? Well, for some it can be the good guys in the black and white suits protecting the galaxy from alien encounters, Men In Black if you couldn't figure that out. Or for others it could be the bad guys in black and white suits attacking Neo and his fellow humans in the Matrix. But let's be honest, for most people when they hear about agents, they think of that great British agent, Austin Powers of course. Or maybe, you know, that 007 guy too. But all jokes aside, having said that, agents in the context of today's webinar are more likely like little lightweight application sitting on remote machines, able to communicate with your GoAnywhere environment and able to perform some pretty cool tasks.

What are GoAnywhere MFT Agents?

GoAnywhere Agents, much like you're GoAnywhere MFT can be deployed to virtually any operating system, whether Windows, Linux, Unix, IBM i, Mac or even instances that you have out in cloud deployments such as AWS and Microsoft Azure. Once you get the Agents installed, you can move and manipulate files securely between locations without the need for FTP, FTPS, SFTP servers or other traditional transfer methods. The Agents will initiate contact with GoAnywhere MFT and register depending upon the installation package and configuration from the GoAnywhere administrator.

As your Agents register to the MST instance, this allows centralized management of all Agents throughout your enterprise and remote locations all controlled at the GoAnywhere MFT corporate location. So now I can log into GoAnywhere main instance and manage all Agents to the agent manager interface, which we'll take a look at in just a second. We can define workflows and file transfer commands to push down to the agent for execution. We have the ability to define resources from the agent manager console as well as configure the scheduler for specific Agents. The enterprise scheduler can call projects at various granular time intervals, be it minutely, hourly, daily, monthly.

Now, configured at the central management MFT server, you can also access to define holiday calendars and give them conditional actions if your scheduled job happens to land on a company holiday. You can even specify repeat options, which is basically like a schedule within a schedule.

Another popular and effective way to kick off projects is through our file system monitors. Now, these folder can pull a targeted folder for certain file actions whether created, modified, deleted or simply whether a file exists. Furthermore, you can be looking for a specific type of file either by wild color reference or more complex regular expression configurations for those more savvy regex folks.

So let's say for instance, you want to replicate your .BAK files from your Windows machine out to an Amazon S3 bucket. Maybe you want to keep a copy off site and AWS bucket storage is pretty cheap and highly available option. You could set up a folder monitor to watch for all created or modified files in your backup folder to call an MFT project to move those files from the agent to MFT and then out to that S3 bucket resource.

Now, you may have noticed I said move the files from the agent to MFT and then out to that S3 bucket. This centralized approach allows for complete control, not only of all file movement in jobs, but audit logs as well. So if anything does go wrong we can have automated alerting for all connected Agents as well as the corporate instance of GoAnywhere. So basically you can tell as many Agents as you want to connect up to your corporate instance for expanded secure file transfer in all conceivable locations.

Agents Deployment Options

Now, some of the deployment options we'll look at in some of the scenarios for Agents. In the first scenario Agents can be installed on your internal network servers for file movement within your own data center. Now, there may be some instances where your network has secured VLANs where servers are behind or segmented off and ingress traffic is tightly controlled. With an agent, the communications are initiated from the agent back to MFT. Now, usually egress traffic is not so tightly controlled, in most cases would be allowed. This would enable the agent to make initial contact with the MFT instance, register, and all future communications will use that existing channel. This is very similar to how our gateway configuration, how it allows traffic into your private network without having to open up any inbound ports to your internal firewall back into the private network, basically acting as a reverse proxy.

In scenario two, you can install Agents at your remote locations or branch offices. Now, say you're in a retail store and you have multiple locations. Each location could have a point of sale system that generates data specific to that location on a daily basis. Now, maybe at the end of the day your local store generates daily reports and data about sales for the day and inventory need. The Agents then can be used to extract data from that local database, place the file in a targeted folder, have a folder monitor pick up that file and kick off a project back at headquarters to sync or grab those files up to the corporate office. With all your remote locations doing a similar task now you have the tools and data to then provide store statistics and data for the entire organization.

Or maybe scenario three, you have some trading partners that don't have the resources or traditional server protocols like SFTP or FTPS but still want to be able to synchronize and or retrieve, send files back and forth. Or maybe they just don't want to manage the server listener. Again, having no inbound ports needed to be opened may be attractive to let those customers and/or partners agree to put an agent on a system required for file transfer activities.

And finally with option four, the growing popularity of cloud deployments, we have seen where customers are installing Agents on instances up in the cloud for secure communications between their data centers in the cloud. This can be a very cheap and easy way to provide secure communications without having to set up a VPN or even more expensive, a dedicated connection to the cloud service. Now, you can converse and install Agents anywhere you need to that have secure file communications, whether inside or outside your organization. If you do install Agents outside your network, our GoAnywhere gateway application will enable all agent communication to securely connect back to your corporate MFT instance without opening up any inbound ports from your DMZ to your private network.

Agent Secure Communications

Talking of or speaking of secure communications, let's talk about and discuss how Agents in the home corporate instance of GoAnywhere communicate. Now, during that installation of an agent, the installer provides a unique server certificate and registration code so that during initial communications the agent is properly authenticated. Once registration is approved, the server will generate a unique X.509 client certificate to send back to that specific agent. This unique client certificate will be stored encrypted in a local database to be used for client authentication. This process ensures that all future communications with registered Agents leverages both server and client authentication utilizing standard X.509 certificates and protocols.

One very important feature was to make upgrades very easy and almost, if not seamless to the administrators. As Agents are fairly new to GoAnywhere, and although we think they're pretty awesome, there is a ton of potential we're working on. We plan on updating new and exciting features on a fairly frequent basis. So GoAnywhere MFT is able to manage all upgrades to all registered Agents by pushing the latest updates from the centralized MFT instance. It'll pull each agent to make sure that the agent is not performing any jobs and then update the agent. This centralized and automated approach can ease the administration of keeping our Agents up to the latest version with the latest feature set.

Additional Agent Features

Finally, let's look at some additional features that can be done at the agent level. Again, I do think it's worth reiterating over and over again that these communications are over that secure channel using those standards base certificates and TLS encryption. This helps those companies that have regulations to maintain compliance with their sensitive information, or maybe just those folks who want to protect their proprietary data.

Now, Agents can automate tasks and file transfers throughout the enterprise and beyond. We have the ability to compress and decompress utilizing Zip, Tar, and Gzip 2.0 standard, perform various translation tasks, some of which the ability to read and write CSV, flat file, fixed width, XML, and JSON file. Now, perhaps you need to read in the CSV file, parse out that information and enter the results into an updated table in a database. Or maybe vice versa, perform a select statement to extract data out of a database and right out to the CSV. Now, the CSV file can then be placed to a folder which is being monitored to kick off a project MFT to transfer that file back to corporate to give the daily sales data for a particular store.

Or maybe you want to execute native commands on the agent machine. Depending upon the operating system, we can call batch files, executable, or even shell scripts. These can either further process files if needed or maybe just perform an installation of a program on the remote agent machine. That can seem pretty powerful and may even sound scary to some, but all permissions to the Agents and the functions that are allowed are up to the individual who installs the agent, or at least configurable at the local level. For instance, we have the ability to not allow execution of native commands if wanted. In fact, you can resist the ability to do various tasks like database and Zip tasks as well as configure where on the file system the agent has permission to. This can be configured upon installation or modified using a permission XML file after the fact.

Live Demo

Let's go ahead, I'm going to share my screen. Let's go ahead and jump into the product here real quick. Andrew, can you guys see my screen okay?

Andrew: We sure can.

Dan: Okay, great. So the first thing, a couple things we want to talk about when we're talking about Agents. We did talk about deployment and that ease of deployment. It's very, very important from the standpoint of the fact that we are doing a lot of frequent updates. So we wanted to make this as easy as possible.

So within the product, under Health and Software Library, we have an area where we have a software library where all the Agents, depending upon the operating system as well as the upgrader can be held. Now, if you do have online access to your MFT environment, you can just browse the online catalog to pull down the latest versions. Or if you don't, you can actually go up to the customer portal. I do have one that says, "Does not see the screen." Andrew, make sure that we can see the screen for me real quick.

Andrew: I'm seeing it. If others are not seeing it, can you let me know on the chat here?

Dan: Okay. Otherwise, you can go out to the customer portal, pull down those files, and just go ahead and import them right here. Choose the file and import them right into your software library.

What's nice is, again, this is on your centralized MFT version, so we could go if we needed to to the upgrader section and we could go to Upgrade Agents. Now, in this case, on all the Agents that are registered to our current environment, all of them are up to date, but if there were some that were not up to date, they would show up here. We could select them, hit Upgrade and what GoAnywhere will do is, again, make sure that no jobs are being run, look for an idle time, push out those upgrades and do those updates.

Now, as far as the actual configure installers, which I think is important, we'll choose your operating system of choice. Here we'll choose the Microsoft 64-bit and do Configure Installer.

Here's a couple of things to note to do during the initial installation. One, you can put in a name for the actual agent. It is not required. You could leave this blank. There are two things that are going to provide the actual authentication of the agent once it registers. One is going to be this registration code. This registration code is going to correlate to the rules that you set on the agent listener. So I'm going to switch gears and go to the actual service side of things and look at the agent listener, go to the Registration tab and show you this is where you're going to set up those registration rules, and depending upon what registration code that you put in for those Agents and then when they check in they will get a certain template as well as the name prefix if applicable, and then also you can have it require approval and then also notify your agent managers on the system that there's a agent that is requiring approval.

In this example here, if you leave the name blank, I think it's definitely a very good idea to make sure that you are requiring approval for any of those Agents that are coming in looking for registration.

Okay, back to the actual installer. Again, a couple of things to configure. The host is going to be the host of the MFT instance they are connecting up to. And then this is a couple of things that we talked about as far as the permissions. Now, by default we're going to allow permissions to the root of the C Drive with read-only rights. Now, again, you can completely change this. You can delete this one. You can add separate folders and give them the rights that you want on that actual local machine where that agent will be installed. And then here down with the project permissions, this is where we kind of talked about restricting some of those more powerful tasks like the execute native commands, database, and Zip task.

Once you get that configured, you're going to hit Generate, of course mine's not going to work because I'm not filling it in, but that's going to generate basically a Zip file and it's going to have two files within it. One, it's going to be the executable, and two, it's going to be a metadata XML file that's filling in basically all the information that you have here. Now, what's also going to shift with that is the server certificate, and that's going to be for the authentication piece for when you install the agent and it comes back into MFT, the registration code along with that server certificate is going to be the way that it's going to authenticate to the system.

Let's go ahead and jump out of here. The other thing worthy of noting, as you guys have known, because I'm sure you guys all attended the first series, the advanced workflow series, but not only to MFT but MFT Agents, workflows or projects are the meat of what we do within GoAnywhere. This is the way that we can do our file movement and manipulation. It's no different within Agents within the actual projects here. If you're not too familiar, the components here, the component library is going to be where you have about 110 different tasks that you can choose from to build out your projects to basically do that automation on the file movement and data manipulation.

Well, let's jump into the actual Agents and agent manager. This is where you're going to have all the Agents that are registered on your system. Just a couple of things to note within the agent manager, obviously you're going to have your name here that your calling the Agents, the domain that it actually has access to, a description if provided, and then your status. Online if it's actually online, not connected, or maybe there's a pending approval status on an agent that is looking for approval from an administrator.

Well, let's go ahead and look at a couple of the options from the cog wheel. Again, the launch console, it'd be about the same as clicking on the actual link here. That's going to launch the console, which we'll do in a second. You can delete, you can switch the domains that it has access to, unregister, restart, as well as choose permissions.

Permissions is worth noting and you can give it via group or individual user permissions. Now, these are going to be the permissions that these folks, depending upon what group they're in or individually will have rights to do within the actual agent console. And this is going to have four different administrative back roles. The resource manager is, if I have, obviously I can create resources within the agent. Project designer, meaning basically I can design projects. Project executer, I can run those projects. And job manager means I can manage the actual jobs that are running within the queue manager. So a few different things that we can do.

Let's go ahead and launch. I'm going to launch my local agent that's on this PC so we can kind of take a look at the agent interface here. Again, just so you know what you're looking at we were logged into the MFT central instance administrative console, and I'm going through the administrative console to now manage the actual Agents. Again, centrally managed from the home-base MFT instance.

Here, just a little bit of general information within the actual Agents, maybe what groups it actually belongs to. You can go through and edit a few of those settings if you like. More importantly, we can add resources to these Agents. For now we have database resources very similar to a MFT. When you add the resource, you're going to put in your certain credentials, certain JDBC URLs. We do have the wizard just like an MFT. And most importantly, we do still have the test button to go ahead and test that network connectivity as well as any credentials that you have put in there. Once that's successful then we're good to go and we can leverage those within projects.

The projects here, again, we're going to be creating projects on the actual local machine. You will notice within here, within the project list, the component library is not as comprehensive as the native MFT. We are working on creating and adding more of those component library tasks within the Agents, but some of the ones that we talked about, the compression, the sequel as well as the data translation and your general file system, copy, deletes, moves and stuff like that.

Both the resources and projects, these are going to be stored on your central MFT instance. So when you call out a resource or you're going to run a project, what's actually happening is MFT is going to push those commands down to the agent for those projects to actually run.

And then on down here we have a scheduler very similar to the MFT product. We can select a project to run a schedule. We can do it on different frequency types. If you have configured a holiday calendar on the MFT, that's where this is going to be populated, and you can select your conditionals if you want. And then the frequency options or the repeat options are kind of that schedule within the schedule.

Now, the monitors I think are the cool thing and one of the things, the first examples we're going to look at. This is going to be similar to the monitors on MFT where we can actually have a targeted folder to monitor for certain file activity. In this case we're looking at a agent test folder EOD folder. We're looking for any file created or modified on the schedule. We are going to, by default, we're going to look all day. We're going to pull every 15 seconds to see if there's anything out there that's created or modified Monday through Friday.

Now, the one difference on the monitors on the Agents is you're going to notice you can have two different accent types. You can call an agent project which will call in the local agent project to do some of that file manipulation locally, or you can call a GoAnywhere MFT project. In this example that's what we're going to do.

This is going to be a little bit more powerful as it currently stands now because this is going to be able to leverage all the resources back at your home MFT project area. So we can do both again, local as well as calling back an MFT project. In this scenario we're going to actually call Scenario One: Pass File From Agent. And same things, we're going to take everything that fills in the stipulation that you put in the general schedule tab and throw it into a file list.

Jobs, real quick. These are going to be all the local jobs that you have here, but again, just like the resources and projects, these jobs are also going to be stored on the local or the central MFT version. So if you look at this job number 49247, let me go and exit out of here. Now I'm back on home-base MFT. Let's see, completed jobs and 49247, robocopy. This is that same job log that was on that local agent. All the job logs are going to be housed on the central MFT server, again, for that centralized auditing and stuff like that.

Okay, so let's jump into a quick scenario. The first scenario we're going to talk about is let's assume that we have a lot of remote locations. Maybe we're a retail chain store, and every night the local retail stores want to query their local database for all the sales and maybe inventory that they need to pull out. They're going to pull out a report and put it into a CSV file. That CSV file is going to land in a folder, which maybe you may have guessed is going to be monitored and that monitor is going to kick off an MFT project to pull those files up to home-base and then insert maybe into a database to update the home-based database.

So let's look at the first scenario. Let's open up our agent manager. I'm going to go to my Dan PC 1. Let's look at the actual project first. This project, scenario one, again, we're just going to do a database SQL task. This is going to be a local database at the remote location. We're going to select everything from store sales to table and put it into an output row set variable called Data. Now, we're simply going to take all that information that we pull out and make that the input and we're going to write an End Of Day Store One. and we're going to leverage a variable called Current Date and then call it a CSB file.

Now, again, let's pay attention. We're on agent test folder, End Of Day is where that actual CSV file is going. So as you may have guessed, our monitor that we're going to have is going to be monitoring that exact folder. We're going to be looking for a file that is created or modified and we're going to go ahead and kick off the MFT project Scenario One: Pass File From Agent.

So let's go back up to the project that's going to be called from the GoAnywhere side. Let's look at what that Pass File From Agent is actually doing. So scenario one, so when the file gets extracted, CSV file gets put in a folder, the folder monitor kicks it off, kicks off this project, we're going to do a simple copy command to an agent's End Of Day file, and then we're actually going to SFTP it out to another location called Process. And then we're going to delete the original files that we copied out from the agent.

So here I'm just going to kind of pull up a couple of Windows Explorer windows, and then in the background here let's go back over here. Let's go back to my agent. Let's go ahead and kick that project off. Projects, Monitors, Scenario One. Okay, so let's go ahead and execute this project. So we should see the files show up here first. Okay. So it queried that database and pulled out the CSV file. We should see it show up over here, and then at some point we should see the original one get deleted. Now, granted that a folder monitor does run every 15 seconds, so hopefully it pops up here at some point. Okay, so it showed up here. So it moved up.

Now, granted this is the area of backup on home MFT. It deleted the original. And then also if we wanted to look at real quick... I guess I should have log in here beforehand. Just for giggles, it also SFTPed out to Agents and then Processed. So here's our end of the day, 9/20, 10:25. So just, yep, just a couple minutes ago. So just a quick example of how we can take files from our Agents and then copy them up to corporate.

Another quick example we'll kind of go through, and this is basically going to take maybe a common file or an updated file from corporate and let's push it down to the Agents. So here we're going to have a project. Now, I'm on the corporate MFT central instance and I'm going to just do a simple copy command down to an agent.

Just a couple of things to note on this. The simple copy tasks that you guys have probably seen, now we have available to us this nomenclature of agent://, similar to kind of like a resource or an SMB resource. Now, granted you could type that in if you wanted to or simply just hit the ellipsis. Let's navigate to our Agents. These are going to be all the Agents that you have available to your MFT, and then select a location to copy.

Now, granted we're doing individual copies of a single file, but we could reiterate through a group of Agents in an actual agent group, but just for purposes of demo, we'll kind of do a quick execution of this. And all this is going to do is push that file from the MFT version down to the local Agents. Now, what those Agents do from there, you could actually monitor that folder to kick off obviously a local project if you wanted to.

The last example that we'll look at real quick, this is basically going to be a way to show that we can actually call remote agent projects. So for this instance here, again this is going to be a very, very simple example, but basically maybe it's something that you're doing within a project on MFT and a certain variable or parameter that you need to pass, based on conditions, you need to pass that perimeter down to a remote project. That's basically what we're going to show here.

I'm going to call an agent project called Robocopy Mirror Batch File and I'm going to add a variable to pass down to that actual project. Now, this is going to be called Passedparm and the value is Pumpernickel. Now, granted this could not just be a string, it could be any type of variable, whatever the value is that you want to pass to that project.

So let me go down to the agent real quick. I apologize for jumping around a lot between the agent and here. Let's look at the project that is actually going to call the Robocopy Mirror Batch File. Now, what is this going to do? Well, one, we're going to print out the variable just so you can see in the log file that I actually can pass the variable and that's the Passedparm. Second, we're going to create a file list. I'm going to look at a base directory called Agents Folder on my local agent machine and I'm going to include anything with a CSV extension. I'm going to take that variable, everything that's in that CSV extension, and I'm going to zip up a Zip file called Customers. and then, again, use the current date variable. And then also I'm going to execute a native command called Mirror Zip or Zip Files.BAT and put that executable directory, or the executable directory being in the agent test folder.

So what this is going to do, and let me just get back here, is going to create a copy into a Zip file as well as going to create... My local machine here. It should create, basically doing a robocopy to create a mirror of these two directories.

Let's go back into the project here. And again, we're going to exit out of here and we're going to go back to MFT to show you the actual called project. Agent. So now I'm just going to run this project that is literally just calling the remote agent project. Let's execute that. And hopefully our Zip files... Yep, we got our Zip file that shows up in Zip File, and hopefully the execute native command that did the batch file created the actual zip mirror's file.

So if you look at that within the agent's folder just for giggles here, that mirror zip file, so you don't think I'm making things up here, I don't know where that went. Open here. Don't know where that went. I'm not sure where that actual text file went. But anyway, it's doing just a robocopy to make sure that these directories are actually copied. And so that's all that piece is doing.

The one thing that I do want to show is just the actual... It's kind of the main point of calling agent projects when times are passing parameters from your local machine, so one of the things that we want to look at is go to the actual agent, look at his log file from that project that just got ran and see that the passing the actual print command or passing the value or that variable did put it in Pumpernickel just to kind of prove that you can pass variables from MFT down to an agent project.

Top Reasons to Use GoAnywhere MFT Agents

Anyway, let me jump back out of here and kind of bring this back up here and pull up the recap slide here. So just kind of to recap on some of the regions, we showed a few quick examples of some of the features of Agents and how they can be utilized within your environment. I think a lot of the reasons for Agents are centered around the central management of your file movement and manipulation from a single pane of glass, whether controlling real time, remote, and local workflows, managing the Agents themselves from a maintenance standpoint, having central logging and all activities from all Agents which can drive powerful auditing and alerting capabilities and really to provide transparency into your enterprise managed file transfer solution, not to mention that Agents can have a potential for huge money savings and replacing traditional servers for file management, not only an FTE resource to maintain such systems but also in the cost themselves for upkeep.

And probably the biggest manages in Agents further is to reach that and make that automation a complete solution through your enterprise. Let's eliminate those manual processes, which unfortunately we all know are inherently going to produce errors.

One of the last slides that I have here is kind of the coming soon. As I mentioned, we are really proud of what Agents have to offer today, but are definitely not naive to the fact that there are a lot of potentials with these guys. Some of the features and enhancements to come very soon are mostly targeted on adding agent project tasks as well as available local resources. As you can see here, really popular features are the PGP encryption/decryption tasks we've done on the local Agents, also having those local SFTP resources and tasks and email functionality.

Now, these are just a few of what's on the list. The longterm goals to make these lightweight applications basically a true extension of your centralized MFT and have the most, if not all capabilities of home-based.

Then one of the last enhancements that we had, our developers kind of have hack days, which is done I think a few times a year where these guys take a day or two to literally concentrate on a cool feature, maybe something that's been requested by customers. Now, this particular one will go or basically allow for geolocation mapping of all your configured Agents in your environment. Not only give a visual of where they are, but hovering over them can provide some basic information about the agent and then clicking on them can launch the agent managers so that you can actually manipulate some of the configuration settings. So pretty cool. Again, this is one of the things that came out in their little hack days, so something that's hopefully coming very soon.

With that, I hope this episode in our Get The Most Out Of GoAnywhere was informative. Hopefully maybe get your curiosity juices flowing. If you do have any questions or want to see more of what Agents could do for you, please reach out to your representative or anyone here at HelpSystems so that we can set up maybe a more personalized demo or maybe proof of concept. Thanks for listening, Andy.

Andrew: Great. Thanks, Dan, and thanks for taking the time to walk us through. There were a number of questions. I think Brian got to most of them. A couple of additional questions on cost, limitations with Agents. What I'll do for those specific use case questions, I will have your representative reach right back out to you to answer those. And then like I mentioned before, we will send the recording out after this so you can review it as needed.

And then a quick followup here, we do have some upcoming topics yet this year. So we're doing the PCI DSS Security Settings Audit Report on the 11th, Secure Forms on November 1st, and then a special cloud version on December 6th. And then we'll take your feedback based on the ones that you submitted that you'd like to see as well as we look into 2018 for additional topics like this. You can visit the URL up here in the top left there to get to the whole series and see the ones that are on-demand as well as register for the ones that we have remaining this year.

We appreciate you taking the time to join us today. That ends our presentation and we'll make sure to follow up with you on all the requested information. Thanks and have a great day.