PlaidLink Agents

Description

Sometimes it’s necessary and desireable to access data or run processes from a remote system that does not allow external access. This is common in enterprise environments behind firewalls. PlaidCloud allows this ability by using PlaidLink, which enables remote systems access behind a firewall or where direct access from PlaidCloud is not desired.

PlaidLink uses an agent-based system. This means that an agent, the remote user, is installed on a system inside the firewall or other restricted area. The agent can then connect to PlaidCloud by using an outbound initiation process over a secure HTTPS websocket connection. It is as secure as any other encrypted web connection and usually does not require you to open non-standard ports. Before gaining access, the agent must identify itself by sending its agent identifier. From this, if the agent has a successful authentication process, the agent is granted access to the approved operations.

PlaidLink can be installed on Windows, Unix, and Linux systems and can run under low privilege users. On Windows systems, PlaidLink can operate as a Windows Service with full control from the Service panel. On linux or unix systems, it can run as a deamon process.

PlaidLink can also run as a stand-alone Docker container or as a Kubernetes pod.

Managing Agents

To manage agents:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”

This brings you to the PlaidLink Agents Table where you can view, modify, and obtain credentials for the list of available agents.

Creating an Agent

To create an agent:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”
4. Click “Add PlaidLink Agent”
5. Complete the required fields
6. Click “Create”
7. Assign the agent to the necessary security groups to access resources needed to perform its job
8. Assign the agent to the necessary Document accounts to access documents needed to perform its job

Danger

For Steps 7 and 8 above, the PlaidLink Agent must be assigned to security groups and document accounts necessary for performing the jobs you expect the Agent to perform. Otherwise it will be denied access.

Note

Any information not present on the new agent form will be automatically generated.

Obtaining Agent Credentials

To configure PlaidLink agents on the remote system, you must first obtain the agent’s identifying information in order to maintain security. This information includes both a public and a private key.

To obtain these keys:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”
4. Click the edit icon

This will open a form where you can view the public and private key values.

Regenerating Agent Credentials

It is a good idea to periodically regenerate the public and private keys and update the configuration of remote systems in order to maintain security.

To regenerate the credentials:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”
4. Click the regenerate icon

Once the credentials have been regenerated, they can be obtained in the same way a new agent’s credentials are obtained (described above).

Enabling and Disabling an Agent

To disable an agent:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”
4. Uncheck the “Active” checkbox

Note

When an agent is not marked as active, remote systems will not be able to connect using those agent credentials

Agent Environments

A connection can hold a different configuration per environment, so switching a project’s environment re-points every connection it uses at once. Agents can follow the same switch.

Each agent carries two settings:

• Agent Group — a name shared by the agents that stand in for each other. Agents in a group are alternatives for the same work, one per environment.
• Environment — which connection environment this agent serves.

A step still names one specific agent. When the workflow runs, PlaidCloud looks at the group that agent belongs to and picks the member serving the project’s current environment. So a project moved from your default environment to QA runs its remote steps on the QA machine, with no step edited.

For a group named Finance with one agent serving Default and another serving QA, a step naming either one runs on whichever matches the project — and a project switched back to Default runs on the Default agent, even where the step names the QA one.

To assign an agent to an environment:

1. Open Analyze
2. Select “Tools”
3. Click “PlaidLink Agents”
4. Open the agent
5. Set “Agent Group” to a name shared with the agent’s alternatives, and set “Environment” to the environment this agent serves
6. On the machine running that agent, set the matching environment name in the PlaidLink configuration file — see Matching the Agent Machine below

Note

An agent with no group set serves every environment, which is every agent until you set one. Nothing changes for a workspace that runs a single agent.

Caution

Setting a group makes the match exact for every agent in it, including a group of one. A project running in an environment no agent in the group serves fails with a message naming the group and the environment, rather than falling back to whichever agent the step named. That is deliberate — the alternative is production work quietly running on a test machine. Add an agent for each environment your projects run in, and leave the group blank on an agent that should keep serving everything.

Taking an agent out of a group narrows what the group covers, so it is as consequential as adding one. Clearing the Group on the agent that served your default environment leaves projects running there with nothing to resolve to, and they fail on their next run. Move the environment to another member of the group before removing one. Each save that changes what a group covers reports the new coverage back to you.

Environments are shared with connections, so the environment names offered here are the same ones you configure a connection under. See Create and Manage a Connection.

Matching the Agent Machine

The Environment you set here has to match the environment the PlaidLink service on that machine is configured for. PlaidLink subscribes to one exact queue for the environment named in its configuration file, so an agent whose record says QA while its machine is still configured for the default environment is listening on a different queue than the work is sent to. The step waits, then fails reporting that the agent is not picking up the job.

Set it in the environments block of the PlaidLink configuration file, in the entry for that agent:

environments:
  - name: QA
    agent_id: <agent identifier>

The name is the environment name as it appears in PlaidCloud. Where it is left out, the agent serves the default environment. Restart the PlaidLink service after changing it.

Note

A single PlaidLink service can hold several entries in this block, so one machine can serve more than one agent, or the same agent in more than one environment. See Running Multiple Agents.

Running Multiple Agents

PlaidLink is designed to allow operation of multiple agents using a single service installation. Such a streamlined installation system permits one install to handle agents from multiple workspaces and / or agents with different levels of permissions for task execution.

To enable multiple agents, you simply add the agent credentials to the PlaidLink configuration file.

Running Multiple PlaidLink Services

Similar to running multiple agents within one PlaidLink service, it is also possible to run multiple PlaidLink services.

This is sometimes necessary depending on use of system based security or network access restrictions that prevent communication across network boundaries.

Note

It is normally better to run multiple agents under a single service rather multiple services on a single machine. However, depending on the use case it may be necessary to run multiple distinct services.

Compute, Memory, and Disk Requirements

The PlaidLink service is extremely lightweight and only needs minimal compute and memory to operate. When processing significant data volumes it may be necessary to increase compute resources and especially memory.

Normally, the agent will happily run with 5% of CPU and 200MB of memory. For intense data operations, it is recommended to allocate an entire CPU and at least 4GB of RAM. For dynamic resource allocation systems like Kubernetes, it is fine if the agent has access to burstable resources rather than reserved resources.

Disk space for the agent is minimal too. Agent operations utilize disk space as a data buffer when transferring large amounts of data. Typically, 8GB of space is fine for normal operations. For intense data operations it is recommended that you scale disk up according to the expected data volumes. There is no set amount because it depends on several factors including CPU speed, network speed, amount of data, etc… However, a good place to start is 20GB and adjust from there.

Networking Requirements

The PlaidLink Agent is designed to operate with minimal configuration required. It does not require any special VPN or network configuration other than allowing standard HTTPS network traffic. Agents communicate over the same protocol as normal web browser based traffic.

The agent service always initiates communication with PlaidCloud so there is no need to configure ingress access in firewalls.

Note

Sometimes firewall rules block all access, even standard HTTPS traffic. If the agent reports it is unable to contact PlaidCloud on startup, you will need to work with your networking team to open port 443 for traffic.