This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Projects

1.1 - Archive a Project

Create and Restore your project archives

Creating an Archive

Projects normally contain critical processes and logic, which are important to archive. If you ever need to restore the project to a specific state, having archives is essential.

PlaidCloud allows you to archive projects at any point in time. Creation of archives complements the built-in point-in-time tracking of PlaidCloud by allowing for specific points in time to be captured. This might be particularly useful before a major change or to capture the exact state of a production environment for posterity.

Full backup: This includes all the data tables included in a project. The archive may be quite large, depending on the volume of data in the project.

Partial backup: This can be used if all of the project data can be derived from other sources. If this is the case, it is not necessary to archive the data in the project and have it remain elsewhere. Partial archives save time and storage space when creating the archive.

To archive a project:

  1. Open Analyze
  2. Select the “Projects” tab

Restoring an Archive

Once you have an archive, you may want to restore it. You can restore an archive into a new project or into an existing project.

To restore an archive:

  1. Open Analyze
  2. Select the “Projects” tab

Archiving Schedule

Archives can also serve as a periodic backup of your project. PlaidCloud allows you to manage the backup schedule and set the retention period of the backup archives to whatever is most convenient or desired.

Since all changes to a project are automatically tracked, archiving is not necessary for rollback purposes. However, it does provide specific snapshots of the project state, which is often useful for control purposes and/or having the ability to recover to a known point.

To set an archiving schedule:

  1. Open Analyze
  2. Select the “Projects” tab
  3. Click the backup icon
  4. Choose a directory destination in a Document account
  5. Choose the backup frequency and retention
  6. Choose which items to backup
  7. Click “Update”

1.2 - Managing Data Editors

Create and Edit table data though user interaction

PlaidCloud offers the ability to organize and manage data editors, including labels. Data Editors allow editing table data or creating data by user interaction.

PlaidCloud uses a path-based system to organize data editors, like you would use to navigate a series of folders, allowing for a more flexible and logical organization (control hierarchy) of the data editors. Using this system, data editors can move within a control hierarchy. Multiple references to one data editor from different locations in the control hierarchy (alternate hierarchies) can be created. The ability to manage data editors using this method allows the structure to reflect operational needs, reporting, and control.

Searching

To search for data editors:

  1. Use the filter box in the lower left of the control hierarchy

The search filter will search data editors’ names and labels for matches and show the results in the control hierarchy above.

Move

To move a data editor within the control hierarchy:

  1. Drag it into the folder where you wish to place it

Rename

To rename a data editor:

  1. Right click on the data editor
  2. Select the rename option
  3. Type in the new name and save it

The data editor will now be renamed but retain its original unique identifier.

Delete

You can delete a single data editor or multiple data editors.

To delete a data editor:

  1. Select the data editors in the control hierarchy
  2. Click the delete button on the top toolbar

Create New Directory Structure

To add a new folder to the control hierarchy:

  1. Click the New Folder button on the toolbar

To add a folder to an existing folder:

  1. Right-click on the folder
  2. Select New Folder

Mark Hierarchy for Viewing Roles

The viewing of data editors by various roles:

  1. Click in the Explorer or Manager checkboxes

To update multiple data editors:

  1. Select the data editors in the control hierarchy
  2. Select the desired viewing role from the Actions menu on the top toolbar

Memos to Describe Table Contents

To add a memo to a data editor:

  1. Select the data editor
  2. Update the memo in the right context form

View Additional Hierarchy Attributes

To view and edit additional data editor attributes:

  1. Select the data editor and view the data editor context form on the right

Duplicate a Data Editor

To duplicate a data editor:

  1. Select the data editor
  2. Click on the Duplicate button on the top toolbar

1.3 - Managing Hierarchies

Create and organize your own workflow hierarchies

PlaidCloud offers the ability to organize and manage hierarchies, including labels. Hierarchies are available to all workflows within a project.

PlaidCloud uses a path-based system to organize hierarchies, like you would use to navigate a series of folders, allowing for a more flexible and logical organization (control hierarchy) of the hierarchies. Using this system, hierarchies can be moved within a control hierarchy, or multiple references to one hierarchy, from different locations in the control hierarchy (alternate hierarchies) can be created. The ability to manage hierarchies using this method allows the structure to reflect operational needs, reporting, and control.

Searching

To search for hierarchies:

  1. Use the filter box in the lower left of the control hierarchy
  2. The search filter will search hierarchy names and labels for matches and show the results in the control hierarchy above

Move

To move a hierarchy within the control hierarchy:

  1. Drag it into the folder where you wish to place it

Rename

To Rename a Hierarchy:

  1. Right click on the hierarchy
  2. Select the rename option
  3. Type in the new name and save it
  4. The hierarchy is now renamed, but it will retain its original unique identifier

Clear

You can clear a single hierarchy or multiple hierarchies.

To clear a hierarchy:

  1. Select the hierarchies in the control hierarchy
  2. Click the clear button on the top toolbar

Delete

You can delete a single hierarchy or multiple hierarchies.

To delete a hierarchy:

  1. Select the hierarchies in the control hierarchy
  2. Click the delete button on the top toolbar

The delete operation will check to see if the hierarchy is in use by workflow steps, tables, or views. If so, you will be asked to remove those associations.

Create New Directory Structure

To create a new folder:

  1. Clicking the New Folder button on the toolbar

To add a folder to an existing folder:

  1. Right-click on the folder
  2. Select New Folder.

Mark Hierarchy for Viewing Roles

To view hierarchies by roles:

  1. Click in the Explorer or Manager checkboxes

To view hierarchies that need to be updated:

  1. Select the hierarchies in the control hierarchy
  2. Select the desired viewing role from the Actions menu on the top toolbar

Memos to Describe Table Contents

To add a memo to a hierarchy:

  1. Select the hierarchy
  2. Update the memo in the right context form

View Additional Hierarchy Attributes

To view and edit additional hierarchy attributes:

  1. Select a hierarchy
  2. View the hierarchy context form on the right

Duplicate a Hierarchy

To duplicate a hierarchy:

  1. Select the hieracrhy

  2. Click the duplicate button on the top toolbar

1.4 - Managing Projects

Create and Manage new projects

Searching

Searching for projects is accomplished by using the filter box in the lower left of the hierarchy. The search filter will search project names and labels for matches and show the results in the hierarchy above.

Creating New Projects

To create a new project:

  1. Open Analyze
  2. Select “Projects” from the top menu bar
  3. Click the “New Project” button
  4. Complete the form information including the “Access Control” section
  5. Click “Create”

The project is now ready for updating access permissions, adding owners, and creating workflows.

Automatic Change Tracking

All changes to a project, including workflows, data editors, hierarchies, table structures, and UDFs are tracked and allow point-in-time recovery of the state. This allows for easy recovery from user introduced problems or simply copying a different point-in-time to another project for comparison.

In addition to overall tracking, projects and their elements also allow for versioning. Not only is creating a version easy, you can also merge changes from one version to another. This provides a simple way to keep track of snapshots or to create a version for development and then be able to merge those changes into the non-development version when you want.

Managing Project Access

Types of Access

Project security has been simplified into three types of access:

  • All Workspace Members
  • Specific Members Only
  • Specific Security Groups Only

Setting the project security is easy to do:

  1. Open Analyze
  2. Select “Projects”
  3. Click the edit icon of the project you want to restrict
  4. Choose desired restriction under “Access Control”
  5. Click “Update”

All Workspace Members

“All Workspace Members” access is the most simple option since it provides access to all members of the workspace and does not require any additional assignment of members.

Specific Members Only

“The Specific Members Only” access setting requires assignment of each member to the project.To assign members to a project:

  1. Open Analyze
  2. Select “Projects” from the top menu bar
  3. Click the members icon
  4. Grant access to members by selecting the check box next to their name in the “Access” column
  5. Click “Update”

For clouds with large numbers of members, this approach can often require more effort than desired, which is where security groups become useful.

Specific Security Groups Only

The “Specific Security Groups Only” option enables assigning specific security groups permission to access the account. With access restrictions relying on association with a security group or groups, the administration of account access for larger groups is much simpler. This is particularly useful when combined with single sign-on automatic group association. By using single sign-on to set member group assignments, these groups can also enable and disable access to projects implicitly.

To edit assigned groups:

  1. Open Analyze
  2. Select “Projects” from the top menu bar
  3. Click the security groups icon
  4. Grant access to security groups by selecting the check box next to their name in the “Access” column
  5. Click “Update”

Setting Different Viewing Roles

Many times a project may require several transformations and tables to complete intermediate steps while the end result may end up only consisting of a few tables. Members do not always require viewing of all the elements of the project, sometimes just the final product. PlaidCloud offers you the ability to set different viewing roles to easily declutter and control the visibility of each member.

There are three built-in viewing roles: Architect, Manager, and Explorer

The Architect role is the most simple because it allows full visibility and control of projects, workflows, tables, variables, data editors, hierarchies, and user defined functions.

The Manager and Explorer roles have no specific access privileges but can be custom-defined. In other words, you can choose which items are visible to each group.

You can make everyone an Architect if you feel visibility of everything within the project is needed; otherwise, you can designate members as Manager and/or Explorer project members and control visibility that way.

To set the different role:

  1. Open Analyze
  2. Select “Projects”
  3. Click the members icon
  4. Select the member you whose role you would like to change
  5. Double click their current role in the “Role” column
  6. Select the desired role
  7. Click “Update”

Managing Project Variables

When running a project or workflow it may be useful to set variables for recurring tasks in order to decrease clutter and save time. These variables operate just like a normal algebraic variable by allowing you to set what the variable represents and what operation should follow it. PlaidCloud allows you to set these variables at the project level, which will effect all the workflows within that project, or at the workflow level, which will only effect that specific workflow.

To set a project level variable:

  1. Open Analyze
  2. Select “Projects”
  3. Click the Manage Project Variables icon

From the Variables Table you can view the variables and view/edit the current values. You can also add new or delete existing variables by clicking the “New Project Variable” button.

Cloning a Project

When a project is cloned, there may be project related references, such as workflow steps, that run within the project. PlaidCloud offers two options for performing a full duplication:

  • Duplicate with updating project references
  • Duplicate without updating project references

Duplicating with updating project references means all the related references point to the newly duplicated project.

To duplicate with updating project references:

  1. Open Analyze
  2. Select “Projects”
  3. Select the project you would like to duplicate
  4. Click the “Actions” button
  5. Select the “Duplicate with project reference updates” option

To duplicate without updating project references means to have all of the related references continue pointing to the original project.

To duplicate without updating project references:

  1. Open Analyze
  2. Select “Projects”
  3. Select the project you would like to duplicate
  4. Click the “Actions” button
  5. Select the “Duplicate without project reference updates” option

Viewing the Project Report

When a project or workflow is dynamic, maintaining detailed documentation becomes a challenge. To help solve this problem, PlaidCloud provides the ability to generate a project-level report that gives detailed documentation of workflows, workflow steps, user defined transforms, variables, and tables. This report is generated on-demand and reflects the current state of the project.

To download the report:

  1. Open Analyze
  2. Select “Projects”
  3. Click the report icon

1.5 - Managing Tables and Views

Organize and manage your tables and views

PlaidCloud offers the ability to organize and manage tables, including labels. Tables are available to all workflows within a project and have many tools and options.

In addition to tables, PlaidCloud also offers Views based on table data. Using Views allows for instant updates when underlying table changes occur, as well as saving data storage space.

Options include:

  • The same table can exist on multiple paths in the hierarchy (alternate hierarchies)
  • Tables are taggable for easier search and inclusion in PlaidCloud processes
  • Tables can be versioned
  • Tables can be published so they are available for Dashboard Visualizations

PlaidCloud uses a path-based system to organize tables, like you would use to navigate a series of folders, allowing for a more flexible and logical organization of tables. Using this system, tables can be moved within a hierarchy, or multiple references to one table from different locations in the hierarchy (alternate hierarchies), can be created. The ability to manage tables using this method allows the structure to reflect operational needs, reporting, and control.

Searching

Searching for tables is accomplished by using the filter box in the lower left of hierarchy. The search filter will search table names and labels for matches and show the results in the hierarchy above.

Move

To move a table:

  1. Drag it into the folder where you wish it to be located

Rename

To rename a table:

  1. Right click on the table
  2. Select the rename option
  3. Type in the new name and save it
  4. The table is now renamed, but it retains its original unique identifier.

Clear

To clear a table:

  1. Select the tables in the hierarchy ‘
  2. Click the clear button on the top toolbar.

Note: You can clear a single table or multiple tables

Delete

To delete a table:

  1. Select the tables in the hierarchy
  2. Click the delete button on the top toolbar
  3. The deleted operation will check to see if the table is in use by workflow steps or Views. If so, you will be asked to remove those associations before deletion can occur.

Note: You can also force delete the table(s). Force deletion of the table(s) will leave references broken, so this should be used sparingly.

Create New Directory Structure

To add a new folder:

  1. Click the New Folder button on the toolbar

To add a folder to an existing folder:

  1. Right-click on the folder
  2. Select New Folder

View Data (Table Explorer)

Table data is viewed using the Data Explorer. The Data Explorer provides a grid view of the data as well as a column by column summary of values and statistics. Point-and-click filtering and exporting to familiar file formats are both available. The filter selections can also be saved as an Extract step usable in a workflow.

Publish Table for Reporting

Dashboard Visualizations are purposely limited to tables that have been published. When publishing a table, you can provide a unique name that may distinguish the data. This may be useful when the table has a more obscure name on part of the workflow that generated it, but it needs a clearer name for those building dashboards.

Published tables do not have paths associated with them. They will appear as a list of tables for use in the dashboards area.

Mark Table for Viewing Roles

The viewing of tables by various roles can be controlled by clicking the Explorer or Manager checkboxes. If multiple tables need to be updated, select the tables in the hierarchy and select the desired viewing role from the Actions menu on the top toolbar.

Memos to Describe Table Contents

Add a memo to a table to help understand the data.

View Table Shape, Size, and Last Updated Time

The number of rows, columns, and the data size for each table is shown in the table hierarchy. For very large tables (multi-million rows) the row count may be estimated and an indicator for approximate row count will be shown.

View Additional Table Attributes

To view and edit other table attributes:

  1. Select a table
  2. Click the view the table context form on the right.

Duplicate a Table

To duplicate a table:

  1. Selecting the table
  2. Click on the duplicate button on the top toolbar.

1.6 - Viewing Projects

Viewing authorized projects

Description

Within Analyze, the Projects function provides a level of compartmentalization that makes controlling access and modifying privileges much easier. Projects are what provide the primary segregation of data within a workspace tab.

While Projects fall under Analyze, workflows fall under Projects, meaning that Projects contain workflows. Workflows, simply put, perform a wide range of tasks including data transformation pipelines, data analysis, and even ETL processes. More information on workflows can be found under the “Workflows” section.

Accessing Projects

To access Projects:

  1. Open Analyze
  2. Select “Projects” from the top menu bar

This displays the Projects Hierarchy. From here, you will see a hierarchy of projects for which you have access. There may be additional projects within the workspace, but, if you are not an owner or assigned to the project, they will not be visible to you.

1.7 - Viewing the Project Log

View, sort and clear your project activities and assignments

Viewing and Sorting the Project Log

As actions occur within a project, such as assigning new members or running workflows, the Project Log stores the events. The Project Log consolidates the view of all individual workflow logs in order to provide a more comprehensive view of project activities. PlaidCloud also enables the viewer to sort and filter a Project Log and view details of a particular log entry.

To view the Project Log:

  1. Open Analyze
  2. Select “Projects”
  3. Click the log icon

To sort and filter the Project Log:

  1. Click the small icon to the right of the log and to the left of the “log message”
  2. Select desired guidelines

To view details of a particular log entry:

  1. Right click on the desired log entry
  2. View the “Log Message” box for details

Clearing the Project Log

Clearing the Project Log may be desirable from time to time

To clear the Project Log:

  1. Open Analyze
  2. Select “Projects”
  3. Click the log icon
  4. Click the “Clear Log” button

2 - Allocation Assignments

2.1 - Getting Started

2.1.1 - Why are Allocations Useful

A practical understanding of allocations and how they are helpful

Content coming soon...

2.1.2 - Allocations Quick Start

Set up a basic allocation quickly

Content coming soon...

2.2 - Results and Troubleshooting

2.2.1 - Allocation Results

Understand and analyze allocation results

Content coming soon...

2.2.2 - Troubleshooting Allocations

Understand how to troubleshoot allocations when the results are not as expected

Stranded Cost

Stranded cost is....

Over Allocation of Cost

Over allocation of cost is when you end up with more output cost...

Incorrect Allocation of Cost

Incorrect allocation of costs happens when...

2.3 - Configure Allocations

2.3.1 - Configure an Allocation

Set up a cost allocation transform and manage assignments

Purpose

Allocations enable values (typically costs) to be shredded to a more-granular level by applying a driver. Allocations are used to for a multitude of purposes. including but not limited to Activity-Based Costing, IT & Shared Service Chargeback, calculation of fully loaded cost to produce and provide a good or service to customers, etc. They are a fundamental tool for financial analysis, and a cornerstone for managerial reporting operations such as Customer & Product Profitability. They are also a useful construct for establishing and managing global Intercompany Transfer Prices for goods and services.

Setting up the Allocation transform

From a practical purpose, allocations are set up in PlaidCloud in similar fashion as other data transforms such as joins and lookups. Four configuration parameters must be set in order for an Allocation transform to succeed.

  1. Specify Preallocated Data: Specify the preallocated data table in the Values To Allocate Table section of the allocation transform.
  2. Specify Driver Data: Driver data will serve as the basis for the ratios used in the allocation. Choose the driver data table in the Driver Data Table section of the allocation transform.
  3. Specify the Results Table: Post-allocated data must be stored in a table. Specify the table in the Allocation Result Table section of the allocation result section of the transform.
  4. Specify the Assignment Dimension: Allocations require an assignment dimension, whose purpose is to provide the prescription for how each record or set of records in the preallocated will be assigned. Specify the the assignment dimension in the Assignment Dimension Hierarchy section of the allocation transform.

Key Concepts

The sum of values in an allocated dataset should tie out to those of the pre-allocated source data

Allocations are accessible in PlaidCloud as a transform option. To set up an allocation, first, set up assignments, and then configure an allocation transform to use the assignments to allocate inbound records using a specified driver table.

Assignments are special dimensions. They are accessed within the Dimensions section of a PlaidCloud Project.

To set up an assignment dimension, perform the following steps:

  1. From the project screen, Navigate to the Dimensions tab
  2. Create a new dimension

2.3.2 - Recursive Allocations

How to set up and manage recursive allocations

Content coming soon...

3 - Workflows

3.1 - Change the order of steps in a workflow

Move steps up and down in a workflow to control the flow of execution

There are two ways to update the order of steps in the workflow. The first way is to use the up and down arrows present in the Workflows table to move the step up or down. The second way is to use the Step Move option which allows you to move the step much easier if large changes are necessary. The step move option allows you to move the step to the top, bottom, or after a specific step in one operation.

3.2 - Continue on Error

Set the workflow to continue even when an error occurs

Workflow steps can be set to continue processing even when there is an error. This might be useful in workflow start-up conditions or where data may be available intermittently. If the step errors, it will be recorded as an error but the workflow will continue to process.

To set this option, click on the step edit option, the pencil icon in the workflow table, to open the edit form. Check the checkbox for Continue On Error. After saving the updated step, any errors with the step will not cause the workflow to stop.

Steps that have been set to continue on error will have a special indicator in the workflow steps hierarchy table.

3.3 - Copy & Paste steps

Copy and paste steps within and across workflows

Copy Steps

It is often useful to copy steps instead of starting from scratch each time. PlaidCloud allows copying steps within workflows as well as between workflows, and even in other projects. You can select multiple steps to copy at once. Select the workflow steps within the hierarchy and click the Copy Selected Steps button at the top of the table.

This will place the selected steps in the clipboard and allow pasting within the current workflow or another one.

Copying a step will make a duplicate step within the project. If you want to place the same step in more than one location in a workflow, use the Add Step menu option to add a reference to the same step rather than a clone of the original step.

Paste Steps

After selecting steps to copy and placing them on the clipboard, you can paste those steps into the same workflow or another workflow, even in another project. There are two options when pasting the steps into the workflow:

  • Append to the end of the workflow
  • Insert after last selected row

The append option will simply append the steps to the end of the selected workflow. The insert option will insert the copied steps after the selected row. Note that if multiple steps have been copied to the clipboard from multiple areas in a workflow, that pasting them will paste them in order but will not have any nested hierarchy information from when they were copied. The pasting will be a flat list of steps to insert only. This might be unexpected but is safer than creating all of the directory structure in the target workflow that existed in the source workflow.

3.4 - View a dependency audit

View all the data dependencies within a workflow

The Workflow Dependency Audit is a very helpful tool to understand data and workflow dependencies in complex interconnected workflows. Over time, as workflow processes become more complex, it may become challenging to ensure all dependencies are in the correct order. When data already exists in tables, steps will run and appear correct in many cases but may actually have a dependency issue if the data is populated out of order.

This tool will provide a dependency audit and identify issues with data dependency relationships.

3.5 - Controlling Parallel Execution

How to control serial versus parallel execution of steps in a workflow

Workflows in PlaidCloud can be executed as a combination of serial steps and parallel operations. To set a group of steps to run in parallel, place the steps in a group within the workflow hierarchy. Right click on the group folder and select the Execute in Parallel option. This will allow all the steps in the group to trigger simultaneously and execute in parallel. Once all steps in the group complete, the next step or group in the workflow after the group will activate.

3.6 - Create Workflow

Creating a new workflow

Once you navigate to the Workflows tab in a project, click on the New Workflow button. This will open a form where you can enter in the details of the workflow including the name and memo.

In addition, you can set a remediation workflow to run if the workflow ends in an error. A remediation workflow does not need to be set but can be useful for sending notifications or triggering other processes that may automatically remediate failures.

Once the form is complete, click on the Create button and the new workflow will be added to the project.

3.7 - Duplicate a Workflow

Making a duplicate copy of a workflow

It may be useful to copy a workflow when planning to make major changes or to replicate the process with different options. Duplicating an entire workflow is very easy in PlaidCloud. Simply select the workflows you would like to duplicate in the Workflows table of a selected project and click the Duplicate Selected Workflows button at the top of the table. This will copy the workflows and append the word Copy to the name.

Once the duplication process is complete, the workflow is fully functional. Copied workflows are completely separate from the original and can be modified without impacting the original workflow.

3.8 - Manage Workflow Variables

Create, view, and set workflow variable values

PlaidCloud allows variables at both the project scope and workflow scope. This allows for setting project wide variables or being able to pass information easily between workflows. The variables and values are viewed by clicking on the variables icon in the Workflows hierarchy.

From the variables table you can view the variables, the current values, and edit the values. You can also add new variables or delete existing ones.

3.9 - Managing Step Errors

Control the behavior of a step when errors occur

If a workflow experiences an error during processing, an error indicator is displayed on both the workflow and the step that had the error. PlaidCloud can retry a failed step multiple times. This is often useful if the step is accessing remote systems or data that may not be highly available or intermittently fail for unknown reasons. The retry capability can be set to retry many times as well as add a delay between retries from seconds to hours.

If no retry is selected or the maximum number of retries is exceeded, then the step will be marked as an error. PlaidCloud provides three levels of error handling in that case:

  • Stop the workflow when an error occurs
  • Mark the step as an error but keep processing the workflow
  • Mark the step as an error and trigger a remediation workflow process instead of continuing the current workflow

Stop the Workflow

Stopping the workflow when a step errors is the most common approach since workflows generally should run without errors. This will stop the workflow and present the error indicator on both the step and the workflow. The error will also be displayed in the activity monitor but no further action is taken.

Keep Processing

Each step can be set to continue on error in the step form. If this checkbox is enabled, then any step will be marked with an error if it occurs, but the workflow will treat the error as a completion of the step and continue on. This is often useful if there are steps that perform tasks that can error when there is missing data but are harmless to the overall processes.

Since the workflow is continuing on error under this scenario the workflow will not display an error indicator and continue to show a running indicator.

Trigger Remediation Workflow

With the ability to set a remediation workflow as part of the workflow setup, a workflow error will immediately stop the processing of the current workflow and start processing the remediation workflow. Note that if a step is marked to continue on error that a failure will not trigger the remediation workflow. Only steps that fail that would also cause the entire workflow to stop will trigger the remediation process.

A remediation workflow may be useful for simply notifying people that a failure has occurred or it can perform other complex processing to attempt an automatic correction of any underlying reasons the original workflow failed.

3.10 - Run a workflow

How to run a workflow from the workflow management area

You can trigger a full workflow run by either clicking on the run icon from the Workflows hierarchy or by selecting Run All from the Actions menu within a specific workflow.

You can also click on the Toggle Start/Stop button at the top of the workflow table. This toggle button will stop a running workflow or start a workflow.

3.11 - Running a range of steps in a workflow

How to run a selected range of steps together as mini-workflow

While running individual steps is useful, it also may be useful to run subsets of an entire workflow for development, testing, or troubleshooting. To run a subset of steps, select all the steps you would like to run and select Run Selected from the Actions menu at the top of the workflow steps hierarchy. This will trigger a normal workflow processing but start the workflow at the beginning of the selected steps and stop once the last selected step is complete.

3.12 - Running one step in a workflow

Execute a single step within a workflow

During initial workflow development, testing, or troubleshooting, it is often useful to run steps individually. To run a single step in isolation, right click on the step and select Run Step from the context menu.

3.13 - Skip steps in a workflow

How to disable steps in a workflow so they are not executed

Steps in the workflow can be set to skip during the workflow run. This may be useful if there are debugging steps or old steps that you are not prepared to completely remove from the workflow yet. To set this option, you have two options:

  • Edit the step form
  • Uncheck the enabled checkbox in the workflow hierarchy

To edit the step form, click on the step edit option, the pencil icon in the workflow table, to open the edit form. Uncheck the enabled checkbox. After saving the updated step it will no longer run as part of the workflow but can still be run using the single step run process.

Steps that have been set to disabled will have a disabled indicator in the workflow steps hierarchy table.

3.14 - View Workflow Report

Get a summary report of the workflow and settings

Maintaining detailed documentation to support both statutory and management requirements is challenging when the projects and workflows may be dynamic. To help solve this problem, PlaidCloud provides a Workflow level report that provides detailed documentation of workflows, workflow steps, user defined functions, and variables.

The report is generated on-demand and reflects the current state of the workflow. To download the report click on the Report icon in the Workflows hierarchy.

3.15 - Viewing Workflow Log

How to view and analyze the workflow log

Viewing the Workflow Log

As things happen within a workflow, such as steps running or warnings occurring, those events are logged to the workflow log. This log is viewable from the Project area under the Log tab. The workflow log is also present in the project log in case you would like to see a more comprehensive view of logs across multiple workflows.

The log viewer allows for sorting and filtering the log as well as viewing the details of a particular log entry.

Clearing the Workflow Log

Clearing the workflow log may be desirable from time to time. From the log viewer, select the Clear Log button. This will clear the log based on the workflow selected which will also remove the log entries from the project level log too.

3.16 - Viewing Workflows

Create and Manage your own Workflows

Workflows exist within a Project. From the top menu in the Analyze menu click on the Projects menu item. This will open the Projects hierarchy showing the list of projects. Open the project and navigate to the Workflows tab to see the workflows in the project. Workflows are organized in a hierarchy.

The list of projects you can see is determined by your access security for each project and your Viewing Role within the project (i.e. Architect, Manager, or Explorer). If you are expecting to see a project and it is not present, it could be that you have not been granted access to the project by one of the project owners. If you are expecting to see certain workflows, but you are not an Architect on the project, then they might be hidden from your viewing role.

The status of the workflow will be displayed if it is running, has a warning or error, or was completed normally. The creation and update dates are also shown along with who created or updated the workflow.

4 - Workflow Steps

4.1 - Allocation By Assignment Dimension

Allocate values based on driver data and assignment dimension

Description

Allocate values based on an assignment dimesion and driver data table.

Allocation By Dimension

Data Table Settings

Assignment Dimesion Hierarchy

Assignment Hierarchy

The Assignment Dimension Hierarchy gives the user the ability to point, click and filter either or both the Values To Allocate Table and Driver Data Table to create targeted allocations. The Assignment Dimension Hierarchy is created by combining dimensions that reference the Values To Allocate Table and the Driver Data Table.

Creating An Assignment Dimension Hierarchy

To create the Assignment Dimension Hierarchy you must first create the dimensions you wish to use to as filters for the Values To Allocate Table and the Driver Data Table. The links below will guide you through creating these dimensions.

Creating Dimensions

Loading Dimensions

Creating The Main Hierarchy

Once the dimensions for the Values To Allocate Table and the Driver Data Table have been created the next step is to decide which of the dimensions for the Values To Allocate Table will serve as the Main Hierarchy for the Assignment Dimension Hierarchy.

Copy this dimension by navigating to the Dimensions tab in PlaidCloud, clicking on the dimension and then selecting Actions and Copy Dimension. When you copy the dimension a pop-up will apprear asking you to enter a name for the copied dimension.

Adding Dimensions To The Assignment Hierarchy

Open the newley created Assignment Dimension, click on the down arrow next to Properties and select New Property.

Assignment Hierarchy Property

This will open the Property Configuration dialog box:

Property Configuration

Assignment Hierarchy Configure Property

  • Property Name - This is normally the name of the dimension that is being added to the Assignment Hierarchy.
  • Property Display - This should be set to "Tag".
  • Property Type - This property informs the allocation step which table Values To Allocate Table or the Driver Data Table this dimension is related too.
    • Source - Is used in conjunction with the Values To Allocate Table.
    • Target - Is used in conjunction with the Driver Data Table.
    • Driver - Is used to filter Driver Data Table for the specific driver selected.
    • Context - When the Values To Allocate Table and the Driver Data Table contain the same dimension then context can be used to specify how the dimensions should relate to one another. Context is often used when both the Values To Allocate Table and the Driver Data Table contain Profit / Cost Centers or Geography.
      • Current - Acts as a passthrough and will filter the Driver Data Table based on the settings of the target dimension. An example would be if the Cotext is based on the Profit Center dimension and the Profit Center target dimension is set to ALL then the driver data would filter on all Profit Centers.
      • Parent - When selected then the parent of the Profit Center in the Values To Allocate Table will be used to filter the driver values in the Driver Data Table. This is useful when driver values are, at times, not available for a specific Profit Center but often are at the parent level.
      • All - When selected then the Profit Center in the Values To Allocate Table will not filter the driver values in the Driver Data Table, driver values for all Profit Centers will be used.
  • Editor Type - This drop down should be set to Select Dimension.

Once the appropriate properties have been selected for the dimension being added to the Assignment Hierarchy select "Edit Configuration".

Dimension Configuration

Assignment Hierarchy Configure

  • Dimension - Use the drop down to select the dimension.
  • Hierarchy - If the dimension selected has alternate hierarchies, then they will appear and be selectable here as well as the main hierarchy.
  • Start Node - If you don't wish the dimension to be displayed from the top node you can select any node within the hierarchy as the node from which the dimension will be displayed.
  • Allow Multiple Selections - If checked the user will be able to select multiple nodes in the hierarchy.
  • Special Cases - When selected the special cases will be available for selection in the dimension drop down menu. They are typically used in Target dimensions.
    • Source - When a dimension is set to Source the allocation will ignore this dimension when it filters the Driver Data Table but the allocated results will include values from the dimension.
    • Current - Can be used when a dimension is shared between Source and Target. When the Target dimension is set to Current then the Driver Data Table will be filtered by the current value of the Source dimension as the allocation runs. An example would be if there are multiple periods in the Values To Allocate Table and the Driver Data Table but you want the allocation to allocate within the periods and not acrocss them. It is also common to use Current on Business Units, Cost Centers and Geographies.
    • Unassigned - When a dimension is set to Unassigned the allocation will ignore this dimension when it filters the Driver Data Table and the allocation result for this dimension will be a Null value.
    • All - When a dimension is set to ALL then the allocation will use all the values in the dimension.

The Values To Allocate Table, Driver Data Table and Allocation Result Table can be selected dynamically or statically.

Dynamic Table Selection

The dynamic table option allows specification of a table using text and variables. This is useful when employing variable driven workflows where the table or view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to a table:

legal_entity/inputs/{current_month}/ledger_values

Static Table Selection

When a specific table is desired as the source, leave the Dynamic box unchecked and select the source table using the dropdown menu.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Values To Allocate Table

This is the table that contains the values that are to be allocated. These are typically cost or revenue values.

Driver Data Table

The driver data table contains the values that the allocation step will use to allocate costs.

Examples:

  • For a supply chain to assign costs to customers you might use delivery data with the number of deliveries or the weight of the deliveries as the driver.
  • For an IT help desk to assign its costs to the departments it supports the driver data be the number of tickets by cost center.

Driver Data Sign Rule

Driver data can contain both positive and negative values. The Driver Data Sign Rule lets you decide how conflicting signs will be handled.

  • Error on conficting signs - Allocation step will produce an error and stop if conflicting signs are encountered.
  • Proceed with warning on conflicting signs - Allocation step will use both negative and positive driver values but will display a warning.
  • Use only positive driver values - Allocation step will only use positive driver values, will ignore negative values.
  • Use only negative driver values - Allocation step will only use negative driver values, will ignore positive values.
  • Use absolute values of driver data - Allocation step will use the absolute values of the driver data.

Intermediate Tables

The Intermediate Tables are created each time an allocation step runs and provides a summary of the allocation processing. The Intermediate Tables provide insight into how the alloation process is running an are used to trouble shoot unexpected results.

  • Paths - Shows the number of unique allocation paths summarized from the assignment hierarchy.
  • Mapping - Shows how each line of the Values To Allocate Table are mapped to the allocation targets.
  • Summary - Shows each rule, as a result of the assignment hierachy, and how many of the records from the Values To Allocate Table match it.

Allocation Result Table

Append Results to Target Table

If this box is checked the allocation results will be appended to the allocation result table. If this box is not checked the allocation results table will be overwritten each time the allocation step runs.

Separate Columns for Allocated Results

If this box is checked then the results table will show the amount of each allocated record as well as the amount actually allocated to each driver record.

Rename Dimension Nodes

If this box is checked when the allocation step runs it will rename the dimension node in the Assignment dimension.

Advanced Options

Thread Count

Sets the number of concurrent operations the allocation step will use.

Chunk Size

Set the number of allocation paths within a thread.

Allocation Source Map

Allocation Source Map

The Allocation Source Map is used to map the columns from the Values To Allocate Table that will be used in the allocation step.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Role

Each column in the data mapper must be assigned a role:

  • Pass Thought - These columns will appear in the allocation results table.
  • Value to Allocate - This is the column that contains the values to be allocated.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Allocation Source Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Driver Data Map

Allocation Driver Data Map

The Allocation Driver Data Map is used to map the columns from the Driver Data Table that will be used in the allocation step.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Role

Each column in the data mapper must be assigned a role:

  • Source Relation - These columns have corresponing columns in the Values To Allocate Table.
  • Allocation Target - The columns will be the target of the allocation step and will appear in the Allocation Result Table.
  • Split Value - This column contains the values that will be used to allocate the values in the Values To Allocate Table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Driver Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

Example 1

Values To Allocate Table

Allocation By Dimension

Driver Data Table

Allocation By Dimension

Assignment Dimension Hierarchy

Allocation By Dimension

Since the Target RC dimension is set to Current the driver data will be filtered by the Source RC values in the Values To Allocation Table. Since the only value in the Source RC is "A", only the driver value records where RC = A will be used in the allocation step.

Allocation Results Table

Allocation By Dimension

Example 2

Values To Allocate Table

Allocation By Dimension

Driver Data Table

Allocation By Dimension

Assignment Dimension Hierarchy

Allocation By Dimension

Since the Target RC dimension is set to ALL the driver data will include all RC values as you can see in the RC column in the Allocation Results Table.

Allocation Results Table

Allocation By Dimension

Example 3

Values To Allocate Table

Allocation By Dimension

Driver Data Table

Allocation By Dimension

Assignment Dimension Hierarchy

Allocation By Dimension

With the Context RC set to ALL and the Target RC set to Source the driver data will include all the RC in the driver data. The Contect RC will override the setting on the Target RC.

Allocation Results Table

Allocation By Dimension

Example 4

Values To Allocate Table

Allocation By Dimension

Driver Data Table

Allocation By Dimension

Assignment Dimension Hierarchy

Allocation By Dimension

With the Context RC set to ALL the driver data will include all the RC in the driver data.

Allocation Results Table

Allocation By Dimension

4.2 - Allocation Split

Allocate values based on driver data

Description

Allocate values based on driver data.

Allocation Split

Data Table Settings

The Values To Allocate Table, Driver Data Table and Allocation Result Table can be selected dynamically or statically.

Dynamic Table Selection

The dynamic table option allows specification of a table using text and variables. This is useful when employing variable driven workflows where the table or view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to a table:

legal_entity/inputs/{current_month}/ledger_values

Static Table Selection

When a specific table is desired as the source, leave the Dynamic box unchecked and select the source table using the dropdown menu.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Values To Allocate Table

This is the table that contains the values that are to be allocated. These are typically cost or revenue values.

Driver Data Table

The driver data table contains the values that the allocation step will use to allocate costs.

Examples:

  • For a supply chain to assign costs to customers you might use delivery data with the number of deliveries or the weight of the deliveries as the driver.
  • For an IT help desk to assign its costs to the departments it supports the driver data be the number of tickets by cost center.

Driver Data Sign Rule

Driver data can contain both positive and negative values. The Driver Data Sign Rule lets you decide how conflicting signs will be handled.

  • Error on conficting signs - Allocation step will produce an error and stop if conflicting signs are encountered.
  • Proceed with warning on conflicting signs - Allocation step will use both negative and positive driver values but will display a warning.
  • Use only positive driver values - Allocation step will only use positive driver values, will ignore negative values.
  • Use only negative driver values - Allocation step will only use negative driver values, will ignore positive values.
  • Use absolute values of driver data - Allocation step will use the absolute values of the driver data.

Allocation Result Table

Append Results to Target Table

If this box is checked the allocation results will be appended to the allocation result table. If this box is not checked the allocation results table will be overwritten each time the allocation step runs.

Separate Columns for Allocated Results

If this box is checked then the results table will show the amount of each allocated record as well as the amount actually allocated to each driver record.

Allocation Source Map

Allocation Source Map

The Allocation Source Map is used to map the columns from the Values To Allocate Table that will be used in the allocation step.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Role

Each column in the data mapper must be assigned a role:

  • Pass Thought - These columns will appear in the allocation results table.
  • Value to Allocate - This is the column that contains the values to be allocated.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Allocation Source Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Driver Data Map

Allocation Driver Data Map

The Allocation Driver Data Map is used to map the columns from the Driver Data Table that will be used in the allocation step.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Role

Each column in the data mapper must be assigned a role:

  • Source Relation - These columns have corresponing columns in the Values To Allocate Table.
  • Allocation Target - The columns will be the target of the allocation step and will appear in the Allocation Result Table.
  • Split Value - This column contains the values that will be used to allocate the values in the Values To Allocate Table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Driver Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.3 - Rule-Based Tagging

Tag data based on rules

Description

Rule Based Tagging is used to add attributes contained within a dimesion to a data table.

Rule Based Tagging

Data Table Settings

The Source Table and Tagging Result Table can be selected dynamically or statically.

Dynamic Table Selection

The dynamic table option allows specification of a table using text and variables. This is useful when employing variable driven workflows where the table or view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to a table:

legal_entity/inputs/{current_month}/ledger_values

Static Table Selection

When a specific table is desired as the source, leave the Dynamic box unchecked and select the source table using the dropdown menu.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Source Table

This is the table that contains the data that you wish to add the attributes from the Assignment Dimension to.

Rule Based Tagging

Tagging Result Table

The Tagging Result Table will contain the data from the Source Data Table with the attributes contained in the Assignment Dimension Hierarchy.

Rule Based Tagging

Assignment Dimesion Hierarchy

Rule Based Tagging

The Assignment Dimension Hierarchy gives the user the ability to point, click and filter the Source Table to add attributes to the Tagging Result Table. The Assignment Dimension Hierarchy is created by combining dimensions that reference the Source Table.

Creating An Assignment Dimension Hierarchy

To create the Assignment Dimension Hierarchy you must first create the dimensions you wish to use to as filters for the Source Table. The links below will guide you through creating these dimensions.

Creating Dimensions

Loading Dimensions

Creating The Main Hierarchy

Once the dimensions for the Source Table have been created the next step is to decide which of the dimensions for the Source Table will serve as the Main Hierarchy for the Assignment Dimension Hierarchy.

Copy this dimension by navigating to the Dimensions tab in PlaidCloud, clicking on the dimension and then selecting Actions and Copy Dimension. When you copy the dimension a pop-up will apprear asking you to enter a name for the copied dimension.

Adding Dimensions To The Assignment Hierarchy

Open the newley created Assignment Dimension, click on the down arrow next to Properties and select New Property.

Assignment Hierarchy Property

This will open the Property Configuration dialog box:

Property Configuration

Assignment Hierarchy Configure Property

  • Property Name - This is normally the name of the dimension that is being added to the Assignment Hierarchy.

  • Property Display - This should be set to "Tag".

  • Property Type - For Rule Based Tagging property type should be set to Source.

    • Source - Is used in conjunction with the Source Table.
  • Editor Type - This drop down should be set to Select Dimension.

Once the appropriate properties have been selected for the dimension being added to the Assignment Hierarchy select "Edit Configuration".

Dimension Configuration

Assignment Hierarchy Configure

  • Dimension - Use the drop down to select the dimension.
  • Hierarchy - If the dimension selected has alternate hierarchies, then they will appear and be selectable here as well as the main hierarchy.
  • Start Node - If you don't wish the dimension to be displayed from the top node you can select any node within the hierarchy as the node from which the dimension will be displayed.
  • Allow Multiple Selections - If checked the user will be able to select multiple nodes in the hierarchy.
  • Special Cases - Are not used in Rule Based Tagging.

Source Map

Allocation Source Map

The Allocation Source Map is used to map the columns from the Values To Allocate Table that will be used in the allocation step.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Role

Each column in the data mapper must be assigned a role:

  • Pass Thought - These columns will appear in the allocation results table.
  • Value to Allocate - This is the column that contains the values to be allocated.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Source Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.4 - Agent Steps

4.4.1 - Document - Remote Delete File

Deletes a remote file system file using a PlaidLink agent installed within the firewall

Description

Deletes a remote file system file using a PlaidLink agent installed within the firewall.

Examples

First, make a selection from the “Agent to Use” dropdown. Next, enter the file or folder path under “File or Folder Path for Delete”. Finally, select “Save and Run Step”.

4.4.2 - Document - Remote Export File

Exports a file to a remote file system using a PlaidLink agent installed within the firewall

Description

Exports a file to a remote file system using a PlaidLink agent installed within the firewall.

Examples

First, make a selection from the “Agent to Use” dropdown. Next, browse for the file or folder path under “File or Folder to Export”. Then enter the location under “Export Path Destination”. Finally, select “Save and Run Step”.

4.4.3 - Document - Remote Import File

Imports a remote file system file using a PlaidLink agent installed within the firewall.

Description

Imports a remote file system file using a PlaidLink agent installed within the firewall.

Examples

First, make a selection from the “Agent to Use” dropdown. Next, enter the file or folder path under “File or Folder Path for Import”. Then enter the folder destination under “Folder Destination”. Select the file type from the dropdown. Finally, select “Save and Run Step”.

4.4.4 - Document - Remote Rename File

Renames or moves a remote file system file using a PlaidLink agent installed within the firewall

Description

Renames or moves a remote file system file using a PlaidLink agent installed within the firewall.

Examples

First, make a selection from the “Agent to Use” dropdown.

Next, enter “Source Path” and “Destination Path”.

Finally, select “Save and Run Step”.

4.4.5 - Agent Remote Execution of SQL

Execute specified SQL on a remote database through a PlaidLink Agent connection.

Description

Execute specified SQL on a remote database through a PlaidLink Agent connection.

4.4.6 - Agent Remote Export of SQL Result

Use a specified SQL on a remote database through PlaidLink Agent and export to PlaidCloud

Description

Execute specified SQL on a remote database through a PlaidLink Agent connection and export the result for use by PlaidCloud or other downstream systems.

Examples

No examples yet...

4.4.7 - Agent Remote Import Table into SQL Database

Import Data into SQL with PlaidLink Agent

Description

Imports specified data into SQL database on a remote system through a PlaidLink Agent connection.

Examples

No examples yet...

4.5 - Common Step Operations

4.5.1 - Advanced Data Mapper Usage

Using the advanced features of the Data Mapper

Review

Before jumping into the advanced usage capabilities of the Data Mapper, a brief review of the basic functionality will help.

Data Mapper Configuration

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

Advanced Usage

Aggregation Options

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. The following summarization options are available:

FunctionDescription
Group ByGroups results by the value
CountNumber of non-null observations in group
Count (including nulls)Number of observations in group
SumSum of values in group
MeanMean of values in group
MedianMedian of values in group
ModeMode of values in group
MinMinimum of values in group
MaxMaximum of values in group
FirstFirst value of values in group using the sorted order
LastLast value of values in group using the sorted order
Standard DeviationUnbiased standard deviation in group
Sample Standard DeviationSample standard deviation in group
Population Standard DeviationPopulation standard deviation in group
VarianceUnbiased variance in group
Sample VarianceSample Variance in group
Population VariancePopulation Variance in group
Advanced Non-Group-BySpecial aggregation selection when using window functions

Pick the appropriate summarization method for the column.

When using a Window Function, select Advanced Non-Group-By as the aggregation method. This special selection is required due to the aggregation inherent in the window function already.

Constants

Specifying a value in the Constant column will override the source column value, if specified, and populate the column with the constant value specified.

Cleaners

The Data Mapper provides a convenient point-and-click cleaner capability to apply conversions to the data within a column.

The cleaning operations include the following categories:

  • Text Trimming
  • Text Formatting
  • Text Transformations
  • Converting to and from NULL values
  • Number Formatting
  • Date Parsing

The result of the cleaner selections are converted into a consolidated expression which is viewable in the Expression information.

Expressions

Expressions in the Data Mapper are one of the most powerful and flexible concepts in PlaidCloud. They provide nearly unlimited flexibility while being exceptionally performant, even on extremely large data.

Expressions are written using Python SQLAlchemy syntax along with a few additional helper functions available in PlaidCloud. This allows PlaidCloud to expose the full set of capabilities of the underlying data warehouse (e.g. Greenplum, SAP HANA, Redshift, etc...) directly. In addition, there are many resources available publicly that provide quick references for use of SQLAlchemy operations. By using standard SQLAlchemy syntax, PlaidCloud avoids the common pitfall of creating yet another domain specific syntax.

The expression editor is opened by double-clicking on the expression cell for the column. Once open, the list of columns are shown on the left while an extensive library of functions are shown on the right.

While it is entirely possible to type the expression directly into the editor, it is normally easier to use the point-and-click function and column selection to get started. The library of functions include the following groups:

  • Conditions
  • Column Specific Conditions
  • Conversions
  • Dates
  • Math
  • Text
  • Summarizations
  • Window Function Operations
  • Arrays
  • JSON
  • PostGIS (Geospatial)
  • Trigonmetry

Once you have completed the expression, save the expression so it will be applied to the column.

View examples and expression functions in the Expressions area.

4.6 - Dimension Steps

4.6.1 - Dimension Clear

Clears the contents of a dimension including structure, values, aliases, properties, and alternate hierarchies

Description

Clears the contents of a dimension including structure, values, aliases, properties, and alternate hierarchies

Dimension Clear

Dimension Selection

Specify Dimension Dynamically

If dimensions or paths were created dynamically then same variables can be used to clear them. Using variables in the clear process is useful since it eliminates the need to update the Dimension Clear step manually on a periodic basis.

An example that uses the current_month variable to dynamically clear the Materials dimension:

/Dimensions/{current_month}/Products/Materials

Use Specific Dimension

Use the dropdown menu to select a specific dimension to clear.

4.6.2 - Dimension Create

Creates a dimension for use and loading

Description

Creates a dimension for use and loading

Dimension Create

Dimension To Create

Name

You can either use a specific name for the dimension to be created or include variables for dynamic naming.

Variables are useful when dimensions are updated on a periodic basis and retaining the historical view is desired.

An example that uses the current_month variable to dynamically name the dimension:

dimension_name_{current_month}

Path

Paths let you create folder structures that the dimensions are are stored in. You can use variables here as well to make the folder structure dynamic. An example that uses the current_month variable to dynamically name a folder:

/Dimensions/{current_month}/Product/

Memo

The Memo field is used a place to store comments or notes.

4.6.3 - Dimension Delete

Deletes a dimension along with all associated structure, values, properties, aliases, and alternate hierarchies

Description

Deletes a dimension along with all associated structure, values, properties, aliases, and alternate hierarchies

Dimension Clear

Dimension Selection

Specify Dimension Dynamically

If dimensions or paths were created dynamically then same variables can be used to delete them. Using variables in the delete process is useful since it eliminates the need to update the Dimension Delete step manually on a periodic basis.

An example that uses the current_month variable to dynamically delete the Materials dimension:

/Dimensions/{current_month}/Products/Materials

Use Specific Dimension

Use the dropdown menu to select a specific dimension to delete.

4.6.4 - Dimension Load

Load and update dimensions using data

Description

Load and update dimensions using data from PlaidCloud tables.

Dimension Load

Dimension Selection

Specify Dimension Dynamically

To specify a dimension dynamically you include project and or local variables in the name.

Variables are useful when dimensions are updated on a periodic basis and retaining the historical view is desired.

An example that uses the current_month variable to dynamically load the dimension:

dimension_name_{current_month}

Use Specific Dimension

To use a specific dimension select the dimension using the drop down menu.

Load to Alternate Hierarchy

To load an Alternate Hierarchy fist select the dimension either dynamically or specifically, click the Load to Alternate Hierarchy checkbox and enter the name of the alternate hierarchy to be loaded.

Source Table

Dynamic

To specify the source table dynamically click the Dynamic Checkbox and enter the table name including the project and or local variables in the name.

Static

To use a specific source table select the table using the drop down menu.

Dimension Properties And Table Layout

Default Consolidation Type

There are three options for consolidation types:

  • "+": Aggregates values in the dimension.
  • "-": Subtracts values in the dimension.
  • "~": No aggregation is performed in the dimension.

Table Column Format

There are two options for fomatting the Source Table when loading a dimension.

Parent Child

In a Parent Child table there are two columns that represent the dimensions structure, Parent and Child.

EXAMPLE PARENT CHILD

PARENTCHILDConsolidation Type
Parent AllParent 1~
Parnet AllParent 2~
Parent 1Child 1+
Parent 2Child 2+
Child 1Child 3+
Child 1Child 4+
Child 2Child 5+

Flattened Levels

In a Flattend Level table there are an infinte number of columns with each column representing a level of the dimension.

EXAMPLE FLATTENED LEVELS

Level 1Level 2Level 3Level 4
Parent AllParent 1Child 1Child 3
Parent AllParent 1Child 1Child 4
Parent AllParent 2Child 2Child 5

Column Mapping

Using the Inspect Source menu button populates the Source Column in the data mapper. Once the Source Column has been populated use the Kind drop down menu to map the Source Columns to the appropriate column type.

4.6.5 - Dimension Sort

Sort dimensions automatically

Description

Sort dimensions automatically.

Dimension Clear

Dimension Selection

Specify Dimension Dynamically

If dimensions or paths were created dynamically then same variables can be used to sort them. Using variables in the sort process is useful since it eliminates the need to update the Dimension Sort step manually on a periodic basis.

An example that uses the current_month variable to dynamically sort the Materials dimension:

/Dimensions/{current_month}/Products/Materials

Use Specific Dimension

Use the dropdown menu to select a specific dimension to sort.

4.7 - Document Steps

4.7.1 - Compress PDF

Applies a PDF compression process to shrink the PDF size

Documentation coming soon...

4.7.2 - Concatenate Files

Concatenates two or more documents together

Documentation coming soon...

4.7.3 - Convert Image to PDF

Converts an image to a PDF document

Documentation coming soon...

4.7.4 - Convert PDF or Image to JPEG

Converts a PDF or other image format to JPEG image

Documentation coming soon...

4.7.5 - Crop Image to Headshot

Automatic headshot cropping of an image

Documentation coming soon...

4.7.6 - Fix File Extension

Determines the proper file extension and renames the file

Documentation coming soon...

4.7.7 - Merge Multiple PDFs

Merges multiple PDFs into a single PDF document

Documentation coming soon...

4.7.8 - Rename Document Directory

Rename an existing directory in PlaidCloud Document

Description

Rename an existing directory within PlaidCloud Document.

Folder to Rename

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the directory to be renamed.

Rename To

Type the new name for the directory.

Examples

No examples yet...

4.7.9 - Rename Document File

Rename an existing file in PlaidCloud Document

Description

Rename an existing file within PlaidCloud Document.

File to Rename

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the file to be renamed.

Rename To

Type the new name for the file.

Examples

No examples yet...

4.7.10 - Create Document Directory

Use PlaidCloud Document to create a new Document Directory

Description

Create a new directory within PlaidCloud Document.

Where to Create New Folder

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the parent directory.

New Folder Name

Type the name for the new directory.

Examples

No examples yet...

4.7.11 - Convert Document Encoding to UTF-16

Updates file encoding and converts all characters to UTF-16

Description

Updates file encoding and converts all characters to UTF-16. This is particularly useful if the information source has mixed encodings or other tools don’t support certain encodings.

Examples

Select the input file and browse for the file within that location. Select the desired output location, and browse then select the desired location for the file. Save and run.

4.7.12 - Convert Document Encoding to UTF-8

Updates file encoding and converts all characters to UTF-8

Description

Updates file encoding and converts all characters to UTF-8. This is particularly useful if the information source has mixed encodings or other tools don’t support certain encodings.

Examples

Select the input file and browse for the file within that location. Select the desired output location, and browse then select the desired location for the file. Save and run.

4.7.13 - Delete Document Directory

Delete an existing directory from within PlaidCloud Document

Description

Delete an existing directory from within PlaidCloud Document.

Folder to Delete

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the directory to delete.

Examples

No examples yet...

4.7.14 - Delete Document File

Delete an existing file from within PlaidCloud Document

Description

Delete an existing file from within PlaidCloud Document.

File to Delete

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the file to delete.

Examples

No examples yet...

4.7.15 - Convert Document Encoding

Concatenates files to form a single file.

Description

Concatenates files to form a single file.

Examples

Create a source input, select the input file and browse for the file within that location. Select the desired output location, and browse to selected the desired location for the file. Save and run.

4.7.16 - Convert Document Encoding to ASCII

Updates file encoding and converts all characters to ASCII

Description

Updates file encoding and converts all characters to ASCII. This is particularly useful if the source of information has mixed encodings or other tools don’t support certain encodings.

Examples

Select the input file and browse for the file within that location. Select the desired output location, and browse to select the desired location for the file. Save and run.

4.7.17 - Copy Document Directory

Copy entire directory in PlaidCloud Document

Description

Copy an entire directory within PlaidCloud Document.

Copy Directory

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the directory you’d like to copy.

Select Destination

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the destination for the copied directory.

If desired, the copied directory can be given a new name. To do so, simply check the Rename the Copied Folder to: box and type in a new name.

Examples

No examples yet...

4.7.18 - Copy Document File

Copy a single file within PlaidCloud Document.

Description

Copy a single file within PlaidCloud Document.

File To Copy

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the file you’d like to copy.

Select Destination

First, select the appropriate account from the dropdown menu.

Next, press the Browse button to select the destination for the copied file.

By default, Analyze will not allow files to be overwritten. Instead, a numerical suffix will be added to each subsequent copy.

To overwrite the existing file, simply check the Allow Overwriting Existing File box.

To rename the file, check the Rename the copied file to box and type in a new name.

Examples

No examples yet...

4.7.19 - Document Text Substitution

Perform text substitution within a specified file

Description

Performs text substitution in the specified file.

Examples

No examples yet...

4.8 - Export Steps

4.8.1 - Export to External Project Table

Export data from a project table to different project's table.

Description

Export data from a project table to different project's table.

Data Sharing Management

Data Sharing Management

In order to export a table to another project you must first go to both projects Home Tab and allow the projects to share data with each other. To do this select New Data Share and select the project and give them Read access.

Export External Project Table

Export External Project Table

Read From

Select the Source Table from the drop down menu.

Write To

Target Project

Select the Target Project from the drop down menu.

Target Table Static

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Target Table Dynamic

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Append to Existing Data

To append the data from the source table to the target table select the Append to Existing Data check box.

4.8.2 - Export to XML

Export an Analyze data table to PlaidCloud Document as an XML file.

Description

Export an Analyze data table to PlaidCloud Document as an XML file.

4.8.3 - Export to CSV

Export an Analyze data table to PlaidCloud Document as a CSV delimited file

Description

Export an Analyze data table to PlaidCloud Document as a CSV delimited file.

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

Selecting File Compression

All exported files are uncompressed, but the following compression options are available:

  • No Compression
  • Zip
  • GZip
  • BZip2

Data Format

Export CSV Data Format

Delimiter

The Export CSV transform is used to export data tables into delimited text files saved in PlaidCloud Document. This includes, but is not limited to, the following delimiter types:

  • Excel CSV (comma separated)

  • Excel TSV (tab separated)

  • User Defined Separator –>

    • comma (,)
    • pipe (|)
    • semicolon (;)
    • tab
    • space ( )
    • other/custom (tilde, dash, etc)

To specify a custom delimiter, select User Defined Separator –> and then Other –>, and type the custom delimiter into the text box.

Special Characters

The Special Characters section allows users to specify how to handle data with quotation marks and escape characters. Choose from the following settings:

  • Special Characters (QUOTE_MINIMAL): Quote fields with special characters (anything that would confuse a parser configured with the same dialect and options). This is the default setting.
  • All (QUOTE_ALL): Quote everything, regardless of type.
  • Non-Numeric (QUOTE_NONNUMERIC): Quote all fields that are not integers or floats. When used with the reader, input fields that are not quoted are converted to floats.
  • None (QUOTE_NONE): Do not quote anything on output. Quote characters are included in output with the escape character provided by the user. Note that only a single escape character can be provided.

Write Header To First Row

If this checkbox is selected the table headers will be exported to the first row. If it is not there will be no headers in the exported file.

Include Data Types In Headers

If this checkbox is selected the headers of the exported file will contain the data type for the column.

Windows Line Endings

Lastly, the Use Windows Compatible Line Endings checkbox is selected by default to ensure compatibility with Windows systems. It is advisable to leave this setting on unless working in a unix-only environment.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.4 - Export to Excel

Export an Analyze data table to PlaidCloud Document as a Microsoft Excel file

Description

Export an Analyze data table to PlaidCloud Document as a Microsoft Excel file. PlaidCloud Analyze supports modern versions of Microsoft Excel (2007-2016) as well as legacy versions (2000/2003).

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

Target Sheet Name

Specify the target sheet name, the default is Sheet1

Selecting File Compression

All exported files are uncompressed, but the following compression options are available:

  • No Compression
  • Zip
  • GZip
  • BZip2

Write Header To First Row

If this checkbox is selected the table headers will be exported to the first row. If it is not there will be no headers in the exported file.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.5 - Export to Google Spreadsheet

Export an Analyze data table to Google Drive as a Google Spreadsheet

Description

Export an Analyze data table to Google Drive as a Google Spreadsheet. A valid Google account is required to use this transform. User credentials must be set up in PlaidCloud Tools prior to using the transform.

Export Parameters

Source and Target

Select the Source Table from PlaidCloud Document using the dropdown menu.

Next, specify the Target Connection information. For details on setting up a Google Docs account connection, see here: PlaidCloud Tools – Connection. Once all necessary accounts have been set up, select the appropriate account from the dropdown list.

Finally, provide the Target Spreadsheet Name and Target Worksheet Name. If desired, select the Append data to existing Worksheet data checkbox to append data to an existing Worksheet. If the target worksheet does not yet exist, it will be created.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.6 - Export to HDF

Export an Analyze data table to PlaidCloud Document as an HDF5 file

Description

Export an Analyze data table to PlaidCloud Document as an HDF5 file.

For more details on HDF5 files, see the HDF Group’s official website here: http://www.hdfgroup.org/HDF5/.

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

Output File Type

All exported files are uncompressed, but the following compression options are available:

  • Zip
  • GZip
  • BZip2

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.7 - Export to HTML

Export an Analyze data table to PlaidCloud Document as an HTML file

Description

Export an Analyze data table to PlaidCloud Document as an HTML file. The resultant HTML file will simply contain a table.

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

Export HTML

Bold Rows

Select this checkbox to make the first row (header row) bold font.

Escape

This option is enabled by default. When the checkbox is selected, the export process will convert the characters <, >, and & to HTML-safe sequences.

Double Precision

See details here:

Output File Type

All exported files are uncompressed, but the following compression options are available:

  • Zip
  • GZip
  • BZip2

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.8 - Export to JSON

Export an Analyze data table to PlaidCloud Document as a JSON file

Description

Export an Analyze data table to PlaidCloud Document as a JSON file. There are several options (shown below) for data orientation.

For more details on JSON files, see the JSON official website here: http://json.org/.

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

JSON Orientation

Consider the following data set:

IDNameGenderState
1JackMMO
2JillFMO
3GeorgeMVA
4AbeMKY

JSON files can be exported into one of four data formats:

  • Records: Data is stored in Python dictionary sets, with each row stored in {Column -> Value, …} format. For example: [{“ID”:1,”Name”:”Jack”,”Gender”:”M”,”State”:”MO”},{“ID”:2,”Name”:”Jill”,”Gender”:”F”,”State”:”MO”},{“ID”:3,”Name”:”George”,”Gender”:”M”,”State”:”VA”},{“ID”:4,”Name”:”Abe”,”Gender”:”M”,”State”:”KY”}]
  • Index: Data is stored in nested Python dictionary sets, with each row stored in {Index -> {Column -> Value, …},…} format. For example: {“0”:{“ID”:1,”Name”:”Jack”,”Gender”:”M”,”State”:”MO”},”1”:{“ID”:2,”Name”:”Jill”,”Gender”:”F”,”State”:”MO”},”2”:{“ID”:3,”Name”:”George”,”Gender”:”M”,”State”:”VA”},”3”:{“ID”:4,”Name”:”Abe”,”Gender”:”M”,”State”:”KY”}}
  • Split: Data is stored in a single Python dictionary set, values are stored in lists. For example: {“columns”:[“ID”,”Name”,”Gender”,”State”],”index”:[0,1,2,3],”data”:[[1,”Jack”,”M”,”MO”],[2,”Jill”,”F”,”MO”],[3,”George”,”M”,”VA”],[4,”Abe”,”M”,”KY”]]}
  • Values: Data is stored in multiple Python lists. For example: [[1,”Jack”,”M”,”MO”],[2,”Jill”,”F”,”MO”],[3,”George”,”M”,”VA”],[4,”Abe”,”M”,”KY”]]

Date Handling

Specify Date Format using the dropdown menu. Choose from the following formats:

  • Epoch (Unix Timestamp – Seconds since 1/1/1970)
  • ISO 8601 Format (YYYY-MM-DD HH:MM:SS with timeproject offset)

Specify Date Unit using the dropdown menu. Choose from the following formats, listed in order of increasing precision:

  • Seconds (s)
  • Milliseconds (ms)
  • Microseconds (us)
  • Nanoseconds (ns)

Force ASCII

Select this checkbox to ensure that all strings are encoded in proper ASCII format. This is enabled by default.

Output File Type

All exported files are uncompressed, but the following compression options are available:

  • Zip
  • GZip
  • BZip2

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.9 - Export to Quandl

Export an Analyze data table to Quandl’s database

Description

Export an Analyze data table to Quandl’s database.

Source and Target

Specify the following parameters:

  • Source Table: Analyze data table to export
  • Quandl Connection: Accessing Quandl data sets requires a user account or a guest account with limited access. This requires set up in Tools. For details on setting up a Quandl account connection, see here: PlaidCloud Tools – Connection
  • Quandl Code: Use the Search button to search for data sets. Alternatively, data sets can be entered manually. This requires the user to enter the portion of the URL after “http://www.quandl.com”. For example, to import the data set for Microsoft stock, which can be found here (http://www.quandl.com/GOOG/NASDAQ_MSFT), enter GOOG/NASDAQ_MSFT in the Quandl Code field
  • Dataset Name: Name of the dataset to be exported to Quandl
  • Dataset Description: Description of dataset to be exported to Quandl

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

For more aggregation details, see the Analyze overview page here.

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

No examples yet...

4.8.10 - Export to SQL

Export an Analyze data table to PlaidCloud Document as an SQL

Description

Export an Analyze data table to PlaidCloud Document as an SQL.

Examples

No examples yet...

4.8.11 - Export to Table Archive

Exports PlaidCloud table archive file

Description

Exports PlaidCloud table archive file.

Export Parameters

Export File Selector

Export File Selector

The file selector in this transform allows you to choose a destination store the exported result in a PlaidCloud Document.

You choose a directory and specify a file name for the target file.

Source Table

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to source table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the source for the export, leave the Dynamic box unchecked and select the source table.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory next selection.

Target Directory Path

Select the Browse icon to the right of the Target Directory Path and navigate to the location you want the file saved to.

Target File Name

Specify the name the exported file should be saved as.

Examples

No examples yet...

4.9 - General Steps

4.9.1 - Pass

This does nothing but may be useful for documenting workflows

Description

The Pass Through transform does not do anything. Its purpose as a placeholder is useful during development or when in need of a separator to section off steps during complex workflows.

4.9.2 - Run Remote Python

Run a Python file using PlaidLink

Description

This transform will run a Python file using PlaidLink. The Python file is executed on the remote system.

A set of global variables can be passed from the script execution on the remote system.

Examples

No examples yet...

4.9.3 - User Defined Transform

Use Python and Pandas directly in a workflow

Description

The Standard Workflow Transforms that come with PlaidCloud can typically perform nearly every operation you’ll need. Additionally, these Standard Transforms are continuously optimized for performance, and they provide the most robust data. However, when the standard options, used singularly or in groups, are not able to achieve your goals, you can create User Defined Transforms to meet your needs. Standard Python code is permitted.

Coding with Python is required to create a User Defined Transform. For additional information, please visit the Python website.

User Defined Transforms

To create a new User Defined Function (UDF), open the workflow which needs the custom transform, select the User Defined tab, and click the Add User Defined Function button. Specify an ID for the UDF. Once created, select the Edit function logic icon (far left) to open the “Edit User Defined Function” window.

Alternatively, a previously created User Defined function can be imported using the Import button from within the User Defined tab. Simply press that button and then select the appropriate workflow from the dropdown menu (this menu contains all workflows within the current workspace). Next, select the function(s) to be imported and press the Import Selected Functions button.

Once the function has been created/imported, proceed to the Analyze Steps tab of the workflow and add a User Defined Transform step in the appropriate position, just as you would add a Standard Transform. In the config window, select the appropriate User Defined Function from the dropdown menu.

4.9.4 - Wait

Pauses workflow execution for a specified period of time

Description

The Wait transform is used to pause processing for a specified duration. This can be especially helpful when waiting for I/O operations from other systems or for debugging workflows during development.

Duration Parameters

Specify a non-negative integer value using the Duration spinner.

Next, specify the unit of time from the dropdown menu. The following units are available for selection:

  • Seconds
  • Minutes
  • Hours

4.10 - Import Steps

4.10.1 - Import Archive

Import an archived project

Description

Imports PlaidCloud table archive.

Examples

No examples yet...


Import Parameters

Import Source and Target

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Source Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

Source FilePath

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

4.10.2 - Import Project Table

Import table data from a different project

Description

Import table data from a different project.


Data Sharing Management

Data Sharing Management

In order to import a table from another project you must first go to both projects Home Tab and allow the projects to share data with each other. To do this select New Data Share and select the project and give them Read access.

Import External Project Table

Import Source and Target

Read From

Select the Source Project and Source Table from the drop downs.

Write To

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

4.10.3 - Import External Database Tables

Import all or a subset of tables in an external database

Description

Includes ability to perform delta loads and map to alternate target table names.

Examples

No examples yet...


Unique Configuration Items

None


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.4 - Import Google BigQuery

Import Google BigQuery files

Description

Import Google BigQuery files.

Examples

No examples yet...


Unique Configuration Items

Coming soon...


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.5 - Import Google Spreadsheet

Import specific worksheets from Google Spreadsheet files

Description

Import specific worksheets from Google Spreadsheet files.

Examples

No examples yet...


Import Parameters

Import Google Spreadsheet

Source And Target

Google Account

Accessing Google Spreadsheet data requires a valid Google user account. This requires set up in Tools. For details on setting up a Google account connection, see here: PlaidCloud Tools – Connection.

Once all necessary accounts have been set up, select the appropriate Google Account from the drop down list.

Spreadsheet

Next, specify the Spreadsheet to import from the dropdown menu containing all available files associated with the specified Google Account.

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Header Type

Since Google Spreadsheets may or may not contain headers, PlaidCloud provides a way to either use the headers, ignore headers, or use column order to determine the column alignment.

  • No Header: The file contains no header. Use the source list in the Data Mapper to determine the column alignment
  • Has Header - Use Header and Override Field List: The file has a header. Use the header names specified and ignore the source list in the Data Mapper.
  • Has Header - Skip Header and Use Field List Instead: The file has a header but it should be ignored. Use the header names specified by the source list in the Data Mapper.

Worksheets to Import

Because workbooks may contain many worksheets with different data, it is possible to select which worksheets should be imported in the current import process. The options are:

  • All Worksheets
  • Worksheets Matching Search
  • Selected Worksheets

The search functionality for worksheets allows inclusion of worksheets matching the search criteria. The search criteria allows for:

  • Starts With: The worksheet name starts with the search text
  • Contains: The worksheet name contains the search text
  • Ends With: The worksheet name ends with the search text

Find Sheets in Selected File

The find sheets button will open the Excel file and list the worksheets available in the table. Mark the checkboxes in the table for the worksheets to be included in the import.

Column Headers

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.


4.10.6 - Import HDF

Import HDF5 files from PlaidCloud Document

Description

Import HDF5 files from PlaidCloud Document.

For more details on HDF5 files, see the HDF Group’s official website here: http://www.hdfgroup.org/HDF5/.

Examples

No examples yet...


Unique Configuration Items

Key Name

HDF files store data in a path structure. A key (path) is needed as the destination for the table within the HDF file. In most situations, this will be table.


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.7 - Import HTML

Import HTML table data from the internet

Description

Import HTML table data from the internet.

Examples

No examples yet...


Unique Configuration Items

Select Tables in HTML

Since it is possible to have multiple tables on a web page, the user must specify which table to import. To do so, specify Name and/or Attribute values to match.

For example, consider the following table:

<table border="1" id="import"> <tr> <th>Hello</th><th>World</th> </tr> <tr> <td>1</td><td>2</td> </tr> <tr> <td>3</td><td>4</td> </tr> </table>

To import this table, specify id:import in the Name Match field.

Additionally, there is an option to skip rows at the beginning of the table.

Column Headers

Specify the row to use for header information. By default, the Column Header Row is 0.


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.8 - Import JSON

Import JSON text files from PlaidCloud Document

Description

Import JSON text files from PlaidCloud Document.

For more details on JSON files, see the JSON official website here: http://json.org/.

JSON files do not retain column order. The column order in the source file does not necessarily reflect the column order in the imported data table.

Examples

No examples yet...


Unique Configuration Items

JSON Data Orientation

Consider the following data set:

| ID | Name | Gender | State | | 1 | Jack | M | MO | | 2 | Jill | F | MO | | 3 | George | M | VA | | 4 | Abe | M | KY |

JSON files can be imported from one of three data formats:

  • Records: Data is stored in Python dictionary sets, with each row stored in {Column -> Value, …} format. For example:
[{ "ID": 1, "Name": "Jack", "Gender": "M", "State": "MO" }, { "ID": 2, "Name": "Jill", "Gender": "F", "State": "MO" }, { "ID": 3, "Name": "George", "Gender": "M", "State": "VA" }, { "ID": 4, "Name": "Abe", "Gender": "M", "State": "KY" }]
  • Index: Data is stored in nested Python dictionary sets, with each row stored in {Index -> {Column -> Value, …},…} format. For example:
{ "0": { "ID": 1, "Name": "Jack", "Gender": "M", "State": "MO" }, "1": { "ID": 2, "Name": "Jill", "Gender": "F", "State": "MO" }, "2": { "ID": 3, "Name": "George", "Gender": "M", "State": "VA" }, "3": { "ID": 4, "Name": "Abe", "Gender": "M", "State": "KY" } }
  • Split: Data is stored in a single Python dictionary set, values stored in lists. For example:
{ "columns": ["ID", "Name", "Gender", "State"], "index": [0, 1, 2, 3], "data": [ [1, "Jack", "M", "MO"], [2, "Jill", "F", "MO"], [3, "George", "M", "VA"], [4, "Abe", "M", "KY"] ] }

Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.9 - Import Quandl

Imports data sets from Quandl’s repository of millions of data sets

Description

Imports data sets from Quandl’s repository of millions of data sets.

For more details on Quandl data sets, see the Quandl official website here: http://www.quandl.com/.

Examples

No examples yet...


Unique Configuration Items

Source Data Specification

Accessing Quandl data sets requires a user account or a guest account with limited access. This requires set up in Tools. For details on setting up a Quandl account connection, see here: PlaidCloud Tools – Connection.

Once all necessary accounts have been set up, select the appropriate account from the drop down list.

Next, enter criteria for the desired Quandl code. Users can use the Search functionality to search for data sets. Alternatively, data sets can be entered manually. This requires the user to enter the portion of the URL after “http://www.quandl.com”.

For example, to import the data set for Microsoft stock, which can be found here (http://www.quandl.com/GOOG/NASDAQ_MSFT), enter GOOG/NASDAQ_MSFT in the Quandl Code field.

Data Selection

It is possible to slice Quandl data sets upon import. Available options include the following:

  • Start Date: Use the date picker to select the desired date.
  • End Date: Use the date picker to select the desired date.
  • Collapse: Aggregate results on a daily, weekly, monthly, quarterly, or annual basis. There is no aggregation by default.
  • Transformation: Summary calculations.
  • Limit Rows: The default value of 0 returns all rows. Any other positive integer value will specify the limit of rows to return from the data set.

Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.10 - Import SAS7BDAT

Import SAS table files from PlaidCloud Document

Description

Import SAS table files from PlaidCloud Document.

Examples

No examples yet...


Unique Configuration Items

None


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.11 - Import SPSS

Import SPSS sav and zsav files from PlaidCloud Document

Description

Import SPSS sav and zsav files from PlaidCloud Document.

Examples

No examples yet...


Unique Configuration Items

None


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.12 - Import SQL

Import data from a remote SQL database.

Description

Import data from a remote SQL database.

Examples

No examples yet...


Import Parameters

Import SQL Table


Source And Target

Database Connection

To establish a Database Connection please refer to PlaidCloud Data Connections

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

SQL Query

In this section write the SQL query to return the required data.


4.10.13 - Import Stata

Import Stata files from PlaidCloud Document

Description

Import Stata files from PlaidCloud Document.

Examples

No examples yet...


Unique Configuration Items

None


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.14 - Import XML

Import XML data as an XML file

Description

Import XML data as an XML file.

Examples

No examples yet...


Unique Configuration Items

None


Common Configuration Items

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Import File Selector

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Selecting a Document Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

File or Directory Selection Option

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.15 - Import CSV

Import delimited text files from PlaidCloud Document

Description

Import delimited text files from PlaidCloud Document. This includes, but is not limited to, the following delimiter types:

  • comma (, )
  • pipe (|)
  • semicolon (; )
  • tab
  • space ( )
  • at symbol (@)
  • tilda (~)
  • colon (:)

Examples

No examples yet...


Import Parameters

Import Source and Target

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Source Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

Source FilePath

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Inspect Selected Source File

By pressing the Guess Settings from Source File button, PlaidCloud will open the file and inspect it to attempt to determine the data format. Always check the guessed settings to make sure they seem correct.

Data Format

Delimiter

As mentioned above, Inspect Source File will attempt to determine the delimiter in the source file. If another delimiter is desired, use this section to specify the delimiter. Users can choose from a list of standard delimiters.

  • comma (, )
  • pipe (|)
  • semicolon (; )
  • tab
  • space ( )
  • at symbol (@)
  • tilda (~)
  • colon (:)

Header Type

Since CSVs may or may not contain headers, PlaidCloud provides a way to either use the headers, ignore headers, or use column order to determine the column alignment.

  • No Header: The CSV file contains no header. Use the source list in the Data Mapper to determine the column alignment
  • Has Header - Use Header and Override Field List: The CSV file has a header. Use the header names specified and ignore the source list in the Data Mapper.
  • Has Header - Skip Header and Use Field List Instead: The CSV file has a header but it should be ignored. Use the header names specified by the source list in the Data Mapper.

Date Format

This setting is useful if the dates contained in the CSV file are not readily recognizable as dates and times. The import process attempts to convert dates but having a little extra information can help in the import process.

Special Characters

The special character inputs control how PlaidCloud handles the presence of certain characters and what they mean in the context of processing the CSV

  • Quote Character: This is the character used to indicate an enclosed set of text that should be processed as a single field
  • Escape Character: This is the character used to indicate the following character should be processed as it is and not interpreted as a special character. Useful when field may contain the delimiter.
  • Null Character: Since CSVs don't have data types, this character provides a way to indicate that the value should be NULL rather than an empty string or 0.
  • Trailing Negatives: Some source systems generate negative numbers with trailing negative symbols instead of prefixing the negative. This setting will process those as negative numbers.

Row Selection

For input files with extraneous records, you can specify a number of rows to skip before processing the data. This is useful if files contain header blocks that must be skipped before arriving at the tabular data.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.16 - Import Excel

Import worksheets from Excel files within PlaidCloud Document

Description

Import specific worksheets from Microsoft Excel files from PlaidCloud Document. Analyze supports the legacy Excel format (XP/2003) as well as the new format (2007/2010/2013). This includes, but is not limited to, the following file types:

  • XLS
  • XLSX
  • XLSB
  • XLSM

Examples

No examples yet...


Import Parameters

Import Source and Target

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Source Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

Source FilePath

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Since Excel files may or may not contain headers, PlaidCloud provides a way to either use the headers, ignore headers, or use column order to determine the column alignment.

  • No Header: The file contains no header. Use the source list in the Data Mapper to determine the column alignment
  • Has Header - Use Header and Override Field List: The file has a header. Use the header names specified and ignore the source list in the Data Mapper.
  • Has Header - Skip Header and Use Field List Instead: The file has a header but it should be ignored. Use the header names specified by the source list in the Data Mapper.

Row Selection

For input files with extraneous records, you can specify a number of rows to skip before processing the data. This is useful if files contain header blocks that must be skipped before arriving at the tabular data.

Worksheets to Import

Because workbooks may contain many worksheets with different data, it is possible to select which worksheets should be imported in the current import process. The options are:

  • All Worksheets
  • Worksheets Matching Search
  • Selected Worksheets

The search functionality for worksheets allows inclusion of worksheets matching the search criteria. The search criteria allows for:

  • Starts With: The worksheet name starts with the search text
  • Contains: The worksheet name contains the search text
  • Ends With: The worksheet name ends with the search text

Find Sheets in Selected File

The find sheets button will open the Excel file and list the worksheets available in the table. Mark the checkboxes in the table for the worksheets to be included in the import.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.10.17 - Import Fixed Width

Import Fixed Width files

Description

Imports fixed-width files.

Examples

No examples yet…


Import Parameters

Import Source and Target

The file selector in this transform allows you to choose a file stored in a PlaidCloud Document location for import.

You can also choose a directory to import and all files within that directory will be imported as part of the transform run.

Source Account

Choose a PlaidCloud Document account for which you have access. This will provide you with the ability to select a directory or file in the next selection.

Search Option

The Search option allows for finding all matching files below a specified directory path to import. This can be particularly useful if many files need to be included but they are stored in nested directories or are mixed in with other files within the same directory which you do not want to import.

The search path selected is the starting directory to search under. The search process will look for all files within that directory as well as sub-directories that match the search conditions specified. Ensure the search criteria can be applied to the files within the sub-directories too.

The search can be applied using the following conditions:

  • Exact: Match the search text exactly
  • Starts With: Match any file that starts with the search text
  • Contains: Match any file that contains the search text
  • Ends With: Match any file that ends with the search text

Source FilePath

When a specific file or directory of files are required for import, picking the file or directory is a better option than using search.

To select the file or directory, simply use the browse button to pick the path for the Document account selected above.

Variable Substition

For both the search option and specific file/directory option, variables can be used with in the path, search text, and file names.

An example that uses the current_month variable to dynamically point to the correct file:

legal_entity/inputs/{current_month}/ledger_values.csv

Target Table

The target selection for imports is limited to tables only since views do not contain underlying data.

Dynamic Option

The Dynamic option allows specification of a table using text, including variables. This is useful when employing variable driven workflows where table and view references are relative to the variables specified.

An example that uses the current_month variable to dynamically point to target table:

legal_entity/inputs/{current_month}/ledger_values

Static Option

When a specific table is desired as the target for the import, leave the Dynamic box unchecked and select the target Table.

If the target Table does not exist, select the Create new table button to create the table in the desired location.

Table Explorer is always avaible with any table selection. Click on the Table Explorer button to the right of the table selection and a Table Explorer window will open.

Remove non-ASCII Characters Option

By selecting this option, the import will remove any content that is not ASCII. While PlaidCloud fully supports Unicode (UTF-8), real-world files can contain all sorts of encodings and stray characters that make them challenging to process.

If the content of the file is expected to be ASCII only, checking this box will help ensure the import process runs smoothly.

Delete Files After Import Option

This option will allow the import process to delete the file from the PlaidCloud Document account after a successful import has completed.

This can be useful if the import files are generated can be recreated from a system of record or there is no reason to retain the raw input files once they have been processed.

Since Excel files may or may not contain headers, PlaidCloud provides a way to either use the headers, ignore headers, or use column order to determine the column alignment.

  • No Header: The file contains no header. Use the source list in the Data Mapper to determine the column alignment
  • Has Header - Use Header and Override Field List: The file has a header. Use the header names specified and ignore the source list in the Data Mapper.
  • Has Header - Skip Header and Use Field List Instead: The file has a header but it should be ignored. Use the header names specified by the source list in the Data Mapper.

Row Selection

For input files with extraneous records, you can specify a number of rows to skip before processing the data. This is useful if files contain header blocks that must be skipped before arriving at the tabular data.

Column Widths

Enter the widths of the columns seperated with commas or spaces.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.11 - Notification Steps

4.11.1 - Notify Agent

Notify a PlaidCloud Agent

Description

Notify a PlaidCloud Agent.

Examples

No examples yet...

4.11.2 - Notify via Microsoft Teams

Send notifications to Microsoft Teams channels

Adding Microsoft Teams notifications from a workflow is a two part process. The two parts are:

  1. Create a Microsoft Teams external connection
  2. Add Microsoft Teams notification steps to your workflows

Add Microsoft Teams Notification Step to Workflow

Adding Microsoft Teams notification steps to the workflow is the same as adding other steps to a workflow. Upon adding the step, open the step configuration, complete the form, and save it. You can now test your Microsoft Teams notification.

Formatting the Microsoft Teams Message

Teams has many formatting options including adding images and mentioning users. Please reference the Teams Message Text Formatting documentation for details.

Create Microsoft Teams External Connection

This is a one-time setup to allow PlaidCloud to send Microsoft Teams notifications on your behalf. Microsoft Teams allows creation of a Webhook App (a generic way to send a notification over the internet). After creating the Webhook App in Microsoft Teams, add the supplied credentials to PlaidCloud to allow its use.

Microsoft Teams Webhook App Creation

These steps will need to be performed by a Microsoft Teams administrator. Follow the steps outlined here for Creating Incoming Webhook (Microsoft Teams Documentation).

PlaidCloud External Connection Setup

These steps will need to be performed by a PlaidCloud workspace administrator with permissions to create External Data Connections. Follow these steps to create the connection:

  1. Navigate to Analyze > Tools > External Data Connections
  2. Under the + New Connection selection, pick Microsoft Teams Webhook
  3. Complete the name, description, and paste in the webhook url generated during the webhook creation above. The name provided here will be shown as the selection in the workflow step so it should be descriptive if possible.
  4. Select the + Create button

Examples

No examples yet...

4.11.3 - Notify Distribution Group

Send an email to a PlaidCloud distribution group

Description

Send an email notification to a PlaidCloud distribution group. Messages are sent from info@tartansolutions.com. No outbound setup is required.

Select PlaidCloud Distribution List

Select a single distribution list from the drop down menu. Distribution lists can be created using Tools. For details on creating a distribution list, see here: PlaidCloud Tools – Distro.

Message

Specify Subject and Body as desired.

Please note that both Project Variables and Workflow Variables are available for use with this transform, in both the subject line and the message body.

Additionally, standard HTML code is permitted in the body to further customize the look of the email messages.

Examples

In this example, all of the system variables are used. Additionally, there is a small bit of HTML used to format the first line of the body. Executing this transform will send the following email to all members specified in the distribution group:

4.11.4 - Notify Via Email

Send email notifications

Description

Send email notifications. Messages are sent from info@tartansolutions.com email account. No outbound setup is required.

Email Addresses

Specify any number of email recipients. Acceptable delimiters include semicolon (;) and comma (,).

Message

Specify Subject and Body as desired.

Please note that both Project Variables and Workflow Variables are available for use with this transform, in both the subject line and the message body.

Additionally, standard HTML code is permitted in the body to further customize the look of the email messages.

Attachments

Attaching files to emails is very simple. Select a file or folder from Document to attach. If a folder is selected, the contents of the folder will be attached as individual files. Variable substitution works with paths for better control of file attachments when sending out personalized emails.

Examples

In this example, all of the system variables are used. Additionally, there is a small bit of HTML used to format the first line of the body. Executing this transform will send the following email:

4.11.5 - Notify Via Log

Write a message to the Analyze workflow log

Description

Write a message to the Analyze workflow log.

Message Parameters

Type the desired message to write to the log. Then select one of three severity levels from the following:

  • Information
  • Warning
  • Error

Please note that both Project Variables and Workflow Variables are available for use with this transform.

Examples

In this example, executing this transform will append an Information item to the log, stating Write a message to the workflow log. I believe you have my stapler, Demo.

4.11.6 - Notify Via SMS

Send an SMS message

Description

Send an SMS message. Messages are sent from info@tartansolutions.com email account. No outbound setup or data is required.

Carrier and Number

From the Mobile Provider dropdown list, select from hundreds of domestic and international providers. For the convenience of the majority of our customers, USA carriers are listed first, followed by all international options listed alphabetically.

Next, specify a valid phone number. Acceptable formats include the following:

  • 5555555555
  • 555.555-5555
  • 555.555.5555
  • 555-555-5555

Message

Specify Subject and Message as desired.

Please note that both Project Variables and Workflow Variables are available for use with this transform, in both the subject line as well as the message body.WARNING: Standard data rates may apply for recipients.

Examples

No examples yet...

4.11.7 - Notify Via Twitter

Send a direct message from PlaidCloud

Description

Send a Twitter Direct Message (DM) from @plaidcloud.

Twitter Account

Specify the twitter account to receive the DM from @plaidcloud. This user must be following @plaidcloud to receive the message. It is allowable, although not required, to prefix the username with the at sign (@).

Message

Enter the desired message. Analyze will not permit a value longer than 140 characters.

Please note that both Project Variables and Workflow Variables are available for use with this transform.

Examples

In this example, a DM is sent from @PlaidCloud to @tartansolutions. System variables are used in the message. The final message reads, Analyze Demo is running on #PlaidCloud.

4.11.8 - Notify Via Web Hook

Send a notification via Web Hook (URL)

Description

Send a notification via Web Hook (URL).

Examples

No examples yet...

4.11.9 - Notify via Slack

Send Slack notifications

Adding Slack notifications from a workflow is a two part process. The two parts are:

  1. Create a Slack Webhook external connection
  2. Add Slack notification steps to your workflows

Add Slack Notification Step to Workflow

Adding Slack notification steps to the workflow is the same as adding other steps to a workflow. Upon adding the step, open the step configuration, complete the form, and save it. You can now test your Slack notification.

Formatting the Slack Message

Slack has many formatting options including adding images and mentioning users. Please reference the Slack Text Formatting documentation for details.

Create Slack Webhook External Connection

This is a one-time setup to allow PlaidCloud to send Slack notifications on your behalf. Slack allows creation of a Webhook App (a generic way to send a notification over the internet). After creating the Webhook App in Slack, add the supplied credentials to PlaidCloud to allow its use.

Slack Webhook App Creation

These steps will need to be performed by a Slack administrator. Follow these steps to create a Slack Webhook App:

  1. From Slack, open the workspace control menu and select Settings & administration > Manage Apps
  2. Select Custom Integrations from the Apps category list
  3. Select Incoming Webhooks from the list of apps
  4. Select the Add to Slack button
  5. On the next screen, select the Slack Channel you wish to post the messages and continue. This is the default channel that will be used but it can be overridden in each notification including sending DMs to specific individuals.
  6. Copy the webhook URL displayed. This will be used later so keep it in a safe place. It will look something like this: https://hooks.slack.com/services/T04QZ1435/G02TGBFTOP8/K9GZrR2ThdJz1uSiL9YeZxoR
  7. You can customize the appearance, name, and emoji before saving. These customizations are only the defaults and these can be overridden on each notification step within a PlaidCloud workflow.

PlaidCloud External Connection Setup

These steps will need to be performed by a PlaidCloud workspace administrator with permissions to create External Data Connections. Follow these steps to create the connection:

  1. Navigate to Analyze > Tools > External Data Connections
  2. Under the + New Connection selection, pick Slack Webhook
  3. Complete the name, description, and paste in the webhook url provided in step 6 above. The name provided here will be shown as the selection in the workflow step so it should be descriptive if possible.
  4. Select the + Create button

Examples

No examples yet...

4.12 - PDF Reporting Steps

4.12.1 - Report Single

Generate a PDF document based on specific data from the report

Description

Generates a PDF report based on the defined RML template and input data sources for the report.

Examples

No examples yet...

4.12.2 - Reports Batch

Generate multiple PDF documents based on specific data from each report

Description

Generates many PDF reports based on the defined RML template and input data sources for each report.

Examples

No examples yet...

4.13 - SAP ECC and S/4HANA Steps

4.13.1 - Call SAP Financial Document Attachment

Calls an SAP ECC Remote Function Call (RFC) designed to attach a file to specified FI document number

Description

Calls an SAP ECC Remote Function Call (RFC) designed to attach a file to specified FI document number.

Examples

RFC Parameters

Select Agent to Use. Select Target Directory from the drop down bar, and browse below for the correct child folder destination for the file. Next, appropriately name the “Target File Name”. Under “Function Call Information”, enter the Function, the Return Value Parameter, and select the parameters.

You can choose to Insert Row or Append Row under the Parameters section, as well as name the parameters and give them values. Choose the Max Concurrent Requests number, and select Wait for RFC to Complete. Save and Run Step.

4.13.2 - Call SAP General Ledger Posting

Calls an SAP ECC Remote Function Call (RFC) designed to post a journal entry including applicable VAT and Withholding taxes

Description

Calls an SAP ECC Remote Function Call (RFC) designed to post a journal entry including applicable VAT and Withholding taxes. This may also run in test mode which will perform a posting process but not complete the posting. This allows for the collection of detectable errors such as an account being closed or a customer not existing in the specified company code specified. The error checking is robust with the ability to return multiple detected errors in a single test.

Examples

RFC Parameters

Select Agent to Use. Select Target Directory from the drop down bar, and browse below for the correct child folder destination for the file. Next, appropriately name the “Target File Name”. Under “Function Call Information”, enter the Function, the Return Value Parameter, and select the parameters.

You can choose to Insert Row or Append Row under the Parameters section, as well as name the parameters and give them values. Choose the Max Concurrent Requests number, and select Wait for RFC to Complete. Save and Run Step.

4.13.3 - Call SAP Master Data Table RFC

Calls an SAP ECC Remote Function Call (RFC) designed to access master data tables and retrieves the data in tabular form

Description

Calls an SAP ECC Remote Function Call (RFC) designed to access master data tables and retrieves the data in tabular form. This data is then available for transformation processes in PlaidCloud. It also provides the ability to export the master data table structure to a separate file which includes column names, data types, and column order information.

Examples

RFC Parameters

Select Agent to Use. Select Target Directory from the drop down bar, and browse below for the correct child folder destination for the file. Next, appropriately name the “Target File Name”. Under “Function Call Information”, enter the Function, the Return Value Parameter, and select the parameters.

You can choose to Insert Row or Append Row under the Parameters section, as well as name the parameters and give them values. Choose the Max Concurrent Requests number, and select Wait for RFC to Complete. Save and Run Step.

4.13.4 - Call SAP RFC

Calls an SAP ECC Remote Function Call (RFC) and retrieves the data in tabular form

Description

Calls an SAP ECC Remote Function Call (RFC) and retrieves the data in tabular form. This data is then available for transformation processes in PlaidCloud.

Examples

RFC Parameters

Select Agent to Use. Select Target Directory from the drop down bar, and browse below for the correct child folder destination for the file. Next, appropriately name the “Target File Name”. Under “Function Call Information”, enter the Function, the Return Value Parameter, and select the parameters.

You can choose to Insert Row or Append Row under the Parameters section, as well as name the parameters and give them values. Choose the Max Concurrent Requests number, and select Wait for RFC to Complete. Save and Run Step.

Advanced Value Iteration

You can select “No Iterators” at the top of this tab and then select Save and Run Step if desired, or you can specify.

Here, you can select “Specify Argument Values” to Iterate Over and create arguments to then go to the Iteration Value.

Next to Select Iterator Argument to Edit Values, there is the option to Insert Tow, Append Row, Delete Row, Move Down Row, or Move to Bottom Row. Below you can choose Range Iterators using the same drop down menu. The last section is titled “Exclusions for Selected Range Iteration” with the same options per row to add, delete, etc. The excluded values can be entered below. Save and Run Step.

4.14 - SAP PCM Steps

4.14.1 - Create SAP PCM Model

This feature allows you to create a blank SAP PCM Model

Description

Creates a blank SAP Profitability and Cost Management (PCM) model.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown. Enter “Model Name” and select “Model type” from the dropdown (both of which are in the “Model Information” section). Check the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.14.2 - Delete SAP PCM Model

Deletes SAP Profitability and Cost Management (PCM) models matching the search criteria

Description

Deletes SAP Profitability and Cost Management (PCM) models matching the search criteria. Deleting models using this transform allows deletion of many models without having to monitor the process.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select “Agent to Use” from the dropdown. Select your desired “Model Search Method”. For this example, we’ve selected “Exact Match”. Enter “Model Search Text” (what you are looking for) under “Model Name Information” and decide if the search is case sensitive or not (if so, check the check box). Finally, check the “Wait for Deletion to Complete” and click “Save and Run Step”.

4.14.3 - Calculate PCM Model

Start your PCM Model Calculation Process

Description

Starts SAP Profitability and Cost Management (PCM) model calculation process.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter model name in the “Model Name” field, click the “Wait for Calculation to Complete” check box (if desired), then click “Save and Run Step”.

4.14.4 - Copy SAP PCM Model

Copy an SAP PCM model

Description

Copies an SAP Profitability and Cost Management (PCM) model.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Example

Select Agent to Use from the dropdown, enter “From Model Name” and “To Model Name” in the “Model Information” field, click the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.14.5 - Copy SAP PCM Period

Copy period within an SAP PCM model

Description

Copies an SAP Profitability and Cost Management (PCM) model period within the same model.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter “Model Name”, “From Period Name” and “To Period Name” in the “Model Information” field. Click the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.14.6 - Copy SAP PCM Version

Copy your a version within an SAP PCM model

Description

Copies an SAP Profitability and Cost Management (PCM) model version within the same model.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter “Model Name”, “Origin Period Name”, and “Destination Period Name” in the “Model Information” field. Click the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.14.7 - Rename SAP PCM Model

Renames your SAP Profitability and Cost Management model

Description

Renames an SAP Profitability and Cost Management (PCM) model.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter “From Model Name” and “To Model Name” in the “Model Information” field, click the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.14.8 - Run SAP PCM Console Job

Launch you PCM model onto the PCM server

Description

Launches an SAP Profitability and Cost Management (PCM) Console process on the PCM server.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter console file path in the “Console File Path” field, click the “Wait for Console Job to Complete” check box (if desired), then click “Save and Run Step”.

4.14.9 - Run SAP PCM Hyper Loader

Load your PCM model using direct table loads

Description

Loads an SAP Profitability and Cost Management (PCM) model using direct table loads. This process is significantly faster than Databridge. The Hyper Loader supports virtually all of the current PCM data, assignment, and structure tables.

This is the current list of available loading targets:

  • Activity Aliases
  • Activity Dimensional Hierarchy
  • Activity Driver Aliases
  • Activity Driver Dimensional Hierarchy
  • Activity Driver Value
  • BOM Default Makeup
  • BOM External Unit Rate
  • BOM Makeup
  • BOM Production Volume
  • BOM Units Sold
  • Cost Object 1 Aliases
  • Cost Object 1 Dimensional Hierarchy
  • Cost Object 2 Aliases
  • Cost Object 2 Dimensional Hierarchy
  • Cost Object 3 Aliases
  • Cost Object 3 Dimensional Hierarchy
  • Cost Object 4 Aliases
  • Cost Object 4 Dimensional Hierarchy
  • Cost Object 5 Aliases
  • Cost Object 5 Dimensional Hierarchy
  • Cost Object Assignment
  • Cost Object Driver
  • Line Item Aliases
  • Line Item Detail Aliases
  • Line Item Detail Dimensional Hierarchy
  • Line Item Detail Value
  • Line Item Dimensional Hierarchy
  • Line Item Direct Activity Assignment
  • Line Item Resource Driver Assignment
  • Line Item Value
  • Period Aliases
  • Period Dimensional Hierarchy
  • Resource Driver Aliases
  • Resource Driver Dimensional Hierarchy
  • Resource Driver Split
  • Resource Driver Value
  • Responsibility Center Aliases
  • Responsibility Center Dimensional Hierarchy
  • Revenue
  • Revenue Aliases
  • Revenue Dimensional Hierarchy
  • Service Aliases
  • Service Dimensional Hierarchy
  • Spread Aliases
  • Spread Dimensional Hierarchy
  • Spread Value
  • Version Aliases
  • Version Dimensional Hierarchy
  • Worksheet 1 Aliases
  • Worksheet 1 Dimensional Hierarchy
  • Worksheet 2 Aliases
  • Worksheet 2 Dimensional Hierarchy
  • Worksheet Value

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown. Enter model name and select the load package storage path location, then select the child folder desired from within. Use the Table Data Selection below to select the source table model and the target load table. Inspect source>>propagate both sides of the table will reveal the data. Click “Save and Run Step” when the data is entered and you have added any expressions.

4.14.10 - Stop PCM Model Calculation

This function stops a PCM Model calculating process

Description

Stops an SAP Profitability and Cost Management (PCM) model calculation process.

Our Credentials

Tartan Solutions is an official SAP Partner and a preferred vendor of services related to SAP PCM model design and implementation.

Examples

Select Agent to Use from the dropdown, enter “Model Name”, click the “Wait for Copy to Complete” check box, then click “Save and Run Step”.

4.15 - Table Steps

4.15.1 - Table Cross Join

Use this function to perform an cross join between two data tables

Description

Use, as you might have expected, to perform a cross join operation on 2 data tables, combining them into a single data table without join key(s).

For more details on cross join methodology, see here: Wikipedia SQL Cross Join

Table Data Selection

Table Source

Specify the source data table by selecting it from the dropdown menu.

Source Columns

Specify any columns to be included here. Selecting the Inspect Source and Populate Source Mapping Table buttons will make these columns available for the join operation.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

Table Source

Table Output

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Target Output Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Output Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.15.2 - Table Drop

Drop/Delete a data table

Description

Drop/delete a data table.

Table Selection

Table Selection

There are two options for selecting the table or in the second option tables to:

The first option is to use the Specific Table dropdown to select the table.

The second is to use the Tables Matching Search option in which you specify the Search Path and Search Text to select the table or tables that match the search criteria. This option is very useful if you have a workflow that creates a series of commonly named tables that that have been saved appending the date.

Table Dymanic Selection

4.15.3 - Table Upsert

Perform an update of existing records or append new ones

Description

Performs an update of existing records and append new ones.

Upsert Parameters

Source And Target

To establish the source and target tables, first select the data table to be extracted from using the Source Table dropdown menu. Next, select an existing table as the target table using the Target Table dropdown.

Source Table Data Selection

Table Upsert

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Update Key

In order for the Upsert to update the existing and append new records you need to select the columns in the data that create a unique key.

Source Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.15.4 - Table Clear

Clear the contents of an existing data table without deleting the actual data table

Description

Clear the contents of an existing data table without deleting the actual data table. The end result is a data table with 0 rows.

Table Selection

There are two options for selecting the table or in the second option tables to:

The first option is to use the Specific Table dropdown to select the table.

The second is to use the Tables Matching Search option in which you specify the Search Path and Search Text to select the table or tables that match the search criteria. This option is very useful if you have a workflow that creates a series of commonly named tables that that have been saved appending the date.

Table Dymanic Selection

4.15.5 - Table Copy

Create a copy of a data table

Description

Create a copy of a data table.

Source and Target

Source And Target

To establish the source and target tables, first select the data table to be extracted from using the Source Table dropdown menu. Next, select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

When performing the copy, Analyze will first check to see if the target data table already exists. If it does, no action will be performed unless the Allow Overwriting Existing Table checkbox is selected. If this is the case, the target table will be overwritten.

Examples

4.15.6 - Table In-Place Delete

Performs a delete on the table using the specified filter conditions

Description

Performs a delete on the table using the specified filter conditions. The operation is performed on the designated table directly so no additional tables are created. Only the rows that meet the filter criteria are deleted. This may be an effective approach when encountering concerns related to data size.

Delete Parameters

Select the Source table for deleting from the dropdown list. This list includes all Project and Workflow data tables.

Table In-Place Delete

Data Filters for Delete

Table In-Place Delete

Examples

4.15.7 - Table In-Place Update

Performs an update on the table using the specified filter conditions and value settings

Description

Performs an update on the table using the specified filter conditions and value settings. The operation is performed directly on the designated table, so no additional tables are created. This may be an effective approach when concerns of data size are encountered.

Table Selection

Select the Source table for updating from the dropdown list. This list includes all Project and Workflow data tables.

Examples

In this example the Account will be set to 41000 when the Version is equal to "Actual" in "Ledger Value to be allocated".

Table In-Place Update

Table In-Place Update

4.15.8 - Table Anti Join

This function provides an unmatched set of data between two tables

Description

Table Anti Join provides the unmatched set of items between two tables. This will return the list of items in the first table without matches in the second table. This can be quite useful for determining which records are present in one table but not another.

This operation could be accomplished by using outer joins and filtering on null values for the join; however, the Anti Join transform will perform this in a more efficient and obvious way.

Table Data Selection

Table Source

Specify the source data table by selecting it from the dropdown menu.

Source Columns

Specify any columns to be included here. Selecting the Inspect Source and Populate Source Mapping Table buttons will make these columns available for the join operation.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

Table Source

Table Output

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Join Map

Table Join Map

Specify join conditions. Using the Guess button will find all matching columns from both Table 1 as well as Table 2. To add additional columns manually, right click anywhere in the section and select either Insert Row or Append Row, to add a row prior to the currently selected row or to add a row at the end, respectively. Then, type the column names to match from Table 1 to Table 2. To remove a field from the Join Map, simply right-click and select Delete.

Target Output Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Output Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.15.9 - Table Extract

This function helps to extract data from one table and place it in another

Description

Used to extract data from an existing Analyze data table into another data table. Examples include, but are not limited to, the following:

  • Sort
  • Group
  • Summarization
  • Filter/Subset Rows
  • Drop Extra Columns
  • Math Operations
  • String Operations

Extract Parameters

Source and Target

Source And Target

To establish the source and target tables, first select the data table to be extracted from using the Source Table dropdown menu. Next, select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Table Data Selection

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

4.15.10 - Table Faker

This function generates fake data

Description

Table Faker generates fake data.

Address

| Generator | Optional Arguments | | Building Number | | | City | | | City Suffix | | | Country | | | Country Code | “representation”=”alpha-2” | | Full Address | | | Latitude | | | Longitude | | | Military DPO | | | Postal Code | | | Postal Code Plus 4 | | | State | | | State Abbreviation | | | Street Address | | | Street Name | | | Street Suffix | |

Automotive

| Generator | Optional Arguments | | License Plate | |

Barcode

| Generator | Optional Arguments | | EAN13 | | | EAN8 | |

Colors

| Generator | Optional Arguments | | Color Name | | | Hex Color | | | RGB Color | | | RGB CSS Color | | | Safe Color Name | | | Safe Hex Color | |

Company

| Generator | Optional Arguments | | Company Catch Phrase | | | Company Name | | | Company Suffix | |

Credit Card

| Generator | Optional Arguments | | Expriration Date | “start”=”now”“end”=”+10y”## ‘12/20’ | | Full | “card_type”=null | | Number | “card_type”=null | | Provider | “card_type”=null | | Security Code | “card_type”=null |

Currency

| Generator | Optional Arguments | | Code | |

Date Time

| Generator | Optional Arguments | | AM/PM | | | Century | | | Date | “pattern”:”%Y-%m-%d”“end_datetime”:null | | Date Time | “tzinfo”:null“end_datetime”=null | | Date Time this Century | “before_now”=true“after_now”=false“tzinfo”=null | | Date Time this Decade | “before_now”=true“after_now”=false“tzinfo”=null | | Date Time this Month | “before_now”=true“after_now”=false“tzinfo”=null | | Date Time this Year | “before_now”=true“after_now”=false“tzinfo”=null | | Day of Month | | | Day of Week | | | ISO8601 Date Time | “tzinfo”=null“end_datetime”=null | | Month | | | Month Name | | | Past Date (Last 30 Days) | “start_date”=”-30d”“tzinfo”=null | | Timezone | | | Unix Time | “end_datetime”=null“start_datetime”=null | | Year | |

File

| Generator | Optional Arguments | | File Extension | “category”=null | | File Name | “category”=null“extension”=null | | File Path | “depth”=”1”“category”=null“extension”=null | | Mime Type | “category”=null |

Internet

| Generator | Optional Arguments | | Company Email | | | Domain Name | | | Domain Word | | | Email | | | Free Email | | | Free Email Domain | | | Image URL | “width”=null“height”=null | | IPv4 | “network”=false“address_class”=”no”“private”=null | | IPv6 | “network”=false | | MAC Address | | | Safe Email | | | Slug | | | TLD | | | URI | | | URL | “schemes”=null | | URL Extension | | | URL Page | | | User Name | |

ISBN

| Generator | Optional Arguments | | ISBN10 | “eparator”=”-“ | | ISBN13 | “eparator”=”-“ |

Job

| Generator | Optional Arguments | | Job Name | |

Lorem

| Generator | Optional Arguments | | Paragraph | “nb_sentences”=”3”“variable_nb_sentences”=true“ext_word_list”=null | | Paragraphs | “nb”=”3”“ext_word_list”=null | | Sentence | “nb_words”=”6”“variable_nb_words”=true“ext_word_list”=null | | Sentences | “nb”=”3”“ext_word_list”=null | | Text | “max_nb_chars”=”200”“ext_word_list”=null | | Word | “ext_word_list”=null | | Words | “nb”=”3”“ext_word_list”=null |

Misc

| Generator | Optional Arguments | | Binary | “length”=”1048576” | | Boolean | “chance_of_getting_true”=”50” | | Null Boolean | | | Locale | | | Language Code | | | MD5 | “raw_output”=false | | Password | “length”=”10”“special_chars”=true“digits”=true“upper_case”=true“lower_case”=true | | Random String | | | SHA1 | “raw_output”=false | | SHA256 | “raw_output”=false | | UUID4 | |

Numeric

| Generator | Optional Arguments | | Big Serial (Auto Increment) | | | Random Float | | | Random Float in Range | | | Random Integer | | | Random Integer in Range | | | Random Numeric | | | Random Percentage (0 – 1) | | | Random Percentage (0 – 100) | | | Serial (Auto Increment) | |

Person

| Generator | Optional Arguments | | First Name | | | First Name Female | | | First Name Male | | | Full Name | | | Full Name Female | | | Full Name Male | | | Last Name | | | Last Name Female | | | Last Name Male | | | Prefix | | | Prefix Female | | | Prefix Male | | | Suffix | | | Suffix Female | | | Suffix Male | |

Phone

| Generator | Optional Arguments | | Phone Number | | | ISDN | |

Tax

| Generator | Optional Arguments | | EIN | | | Full SSN | | | ITIN | |

User Agent

| Generator | Optional Arguments | | Chrome | “version_from”=”13”“version_to”=”63”“build_from”=”800”“build_to”=”899” | | Firefox | | | Full User Agent | | | Internet Explorer | | | Linux Platform Token | | | Linux Processor | | | Mac Platform Token | | | Mac Processor | | | Opera | | | Safari | | | Windows Platform Token | |

Special Generators

While these two generators do not have arguments, the options they provide act similarly to arguments.

Pattern Generator:

| Number | Format | Output | Description | | 3.1415926 | {:.2f} | 3.14 | 2 decimal places | | 3.1415926 | {:+.2f} | +3.14 | 2 decimal places with sign | | -1 | {:+.2f} | -1.00 | 2 decimal places with sign | | 2.71828 | {:.0f} | 3 | No decimal places | | 5 | {:0>2d} | 05 | Pad number with zeros (left padding, width 2) | | 5 | {:x<4d} | 5xxx | Pad Number with x’s (right padding, width 4) | | 10 | {:x<4d} | 10xx | Pad number with x’s (right padding, width 4) | | 1000000 | {:,} | 1,000,000 | Number format with comma separator | | 0.25 | {:.2%} | 25.00% | Format percentage | | 1000000000 | {:.2e} | 1.00e+09 | Exponent notation | | 13 | {:10d} | 13 | Right aligned (default, width 10) | | 13 | {:<10d} | 13 | Left aligned (width 10) | | 13 | {:^10d} | 13 | Center aligned (width 10) |

Random Choice:

In order to provide the options for random choice, simply put your options in quotes and seperate each option with a comma. So a string of random choice options would appear like this: “x”,”y”,”z”

Here, the “Key Word Args/Pattern/Choices” column of the “pattern” row contains a sentence with several references. The first reference equation ( {percentage0-100:.2f}% ) points to the “percentage0-100” row which will generate a random equation. Therefore, the random percentage produced by the “percentage0-100” row will be automatically inserted into the sentence. The reference equation {first_name} points to the row titled “first_name” which will randomly generate a first name, and this name will be automatically inserted into the sentence. The last reference equation ( {randomn_choice} ) operates the same as the other two.

With this, when the pattern generator is run, you will recieve the following results.

4.15.11 - Table Inner Join

Use this function to perform an inner join between two data tables

Description

Use, as you might have expected, to perform an inner join operation on 2 data tables, combining them into a single data table based upon the specified join key(s).

For more details on inner join methodology, see here: Wikipedia SQL Inner Join

Table Data Selection

Table Source

Specify the source data table by selecting it from the dropdown menu.

Source Columns

Specify any columns to be included here. Selecting the Inspect Source and Populate Source Mapping Table buttons will make these columns available for the join operation.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

Table Source

Table Output

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Join Map

Table Join Map

Specify join conditions. Using the Guess button will find all matching columns from both Table 1 as well as Table 2. To add additional columns manually, right click anywhere in the section and select either Insert Row or Append Row, to add a row prior to the currently selected row or to add a row at the end, respectively. Then, type the column names to match from Table 1 to Table 2. To remove a field from the Join Map, simply right-click and select Delete.

Target Output Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Output Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

Join Automobile Manufacturers with Models

In this example, consider the following source data tables. First is a list of automobile manufacturers.

Mfg_IDManufacturer
1Aston Martin
2Porsche
3Lamborghini
4Ferrari
5Koenigsegg

Next is a list of automobile models with a manufacturer ID. Note that there are several models with no manufacturer.

ModelNameMfg_ID
Aventador3
Countach3
DBS1
Enzo4
One-771
Optimus Prime
Batmobile
Agera5
Lightning McQueen

To get a list of models by manufacturer, it makes sense to join on Mfg_ID.

First, specify parameters for Table 1 Data Selection. The source data table is selected and all columns are listed.

Next, specify parameters for Table 2 Data Selection. Once again, the source data table is selected and all columns are listed.

Finally, the join conditions are set in the Table Output tab. Using the Guess button, Analyze properly identifies the Mfg_ID column to use as the Join Key. Lastly, the

Target Output Columns are specified automatically using the Propagate button. This effectively includes all columns from all tables, with all join columns included only a single time. Note that the columns are sorted alphabetically, first by Manufacturer and next by ModelName.

As expected, the final output only includes values which had a match in both tables. As such, Porsche does not show up because it had no models. Likewise, the

Batmobile had no manufacturer (it was a custom job), so it’s not included.

4.15.12 - Table Lookup

Similar to Microsoft Excel, this workflow function also increases process performance

Description

If you are a regular user of the vlookup function in Microsoft Excel, the Table Lookup transform should feel very familiar. It’s used to perform essentially the same function. Unlike the Microsoft Excel version, the PlaidCloud Analyze Table Lookup transform offers greater flexibility, especially allowing for matching on and returning multiple columns.

Table Data Selection

Table Source

Specify the source data table by selecting it from the dropdown menu.

Source Columns

Specify any columns to be included here. Selecting the Inspect Source and Populate Source Mapping Table buttons will make these columns available for the join operation.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

Table Source

Table Output

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Join Map

Table Join Map

Specify join conditions. Using the Guess button will find all matching columns from both Table 1 as well as Table 2. To add additional columns manually, right click anywhere in the section and select either Insert Row or Append Row, to add a row prior to the currently selected row or to add a row at the end, respectively. Then, type the column names to match from Table 1 to Table 2. To remove a field from the Join Map, simply right-click and select Delete.

Target Output Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Output Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

Lookup Product Dimension Information

In this example, the modeler needs information from the product dimension table to make sense of the order fact table. As such, the Import Order Fact table is selected as the Source Table. The Import Product Dim table contains the desired lookup information, so it’s selected as the Lookup Table Source. Although available, no filters are applied to the lookup data table (nor any other data tables, for that matter).

In the Table Data Selection section, all columns are mapped from the source data table to the target data table.

No Data Filters are applied to either source or target data.

Lastly, the source data table is matched to the lookup data table using the Product_ID field found in each table. Only the Product_Description and Unit_Cost columns are appended to the target data table, with Unit_Cost being renamed to Retail_Unit_Cost in the process.

In the resulting target data table, the Product_Description and Retail_Unit_Cost columns have been added, based on matching values in the Product_ID column.

4.15.13 - Table Melt

Flip columns to rows

Description

Used to convert short, wide data tables into long, narrow data tables. Selected columns are transposed, with the column names converted into values across multiple rows.

Perhaps the easiest example to understand is to think of a data table with months listed as column headers:

Table Melt Input

Melting this data table would convert all of the month columns into rows.

Table Melt Output

By specifying which columns to transpose and which columns to leave alone, this becomes a powerful tool. Making this conversion in other ETL tools could require a dozen more steps.

Source and Target Parameters

Table Melt Source Target

Source and Target

To establish the source and target, first select the data table to be extracted from the Source Table dropdown menu.

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Pre-Melt Table Data Selection

Table Pre-Melt

This section is a bit different from the standard Table Data Selection. Basically this is used to specify which columns are to be used in the Melt operation. This includes ID columns and Variable/Value columns.

For more details regarding Table Data Selection, see details here: Table Data Selection

Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions

for more details and examples.

Apply Secondary Filter To Result Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples

Final Data Table Slicing (Limit)

To limit the data, simply check the Apply Row Slicer box and then specify the following:

  • Initial Rows to Skip: Rows of data to skip (column header row is not included in count)
  • End at Row: Last row of data to include. This is different from simply counting rows at the end to drop

Melt Layout

Table Melt Layout

There is a Guess Layout button available to allow Analyze a first crack at specifying ID columns. By default, all text (data type of String) columns are placed in the Keys section. Numeric columns are not placed into Keys by default, but they are allowed to be there based on the model’s needs.

Columns to Use as IDs (Keys)

ID columns are the columns which remain in tact. These columns are effectively repeated for every instance of a variable/value combination. For a monthly table, this would result in 12 repetitions of ID columns.

ID columns can be added automatically or manually. To add the columns automatically, use the aforementioned Guess Layout button. To add additional columns manually, right click anywhere in the section and select either Insert Row or Append Row, to add a row prior to the currently selected row or to add a row at the end, respectively. Then, type the column name to use as an ID.

To remove a field from the IDs, simply right-click and select Delete.

Melt Result Column Naming

There are 2 values to specify. Both of these values will become column names in the target data table.

  • Variable Column Name: As specified in the transform, The variable names are derived from the current source column names. Essentially, specify a column name which will represent the data originally represented in the source data table columns.
  • Value Column Name: Specify a column name to represent the data represented within the source data table. Typically this will be a numerical unit: Dollars, Pounds, Degrees, Percent, etc.

Examples

In the abouve documentation.

4.15.14 - Table Outer Join

Combine data tables using specified join key(s)

Description

Use, as you might have expected, to perform a full outer join operation on 2 data tables, combining them into a single data table based upon the join key(s) specified.

For more details on outer join methodology, see here: Wikipedia SQL Full Outer Join

Table Data Selection

Table Source

Specify the source data table by selecting it from the dropdown menu.

Source Columns

Specify any columns to be included here. Selecting the Inspect Source and Populate Source Mapping Table buttons will make these columns available for the join operation.

Select Subset of Source Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

Table Source

Table Output

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Join Map

Table Join Map

Specify join conditions. Using the Guess button will find all matching columns from both Table 1 as well as Table 2. To add additional columns manually, right click anywhere in the section and select either Insert Row or Append Row, to add a row prior to the currently selected row or to add a row at the end, respectively. Then, type the column names to match from Table 1 to Table 2. To remove a field from the Join Map, simply right-click and select Delete.

Target Output Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Output Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

Join Automobile Manufacturers with Models

In this example, consider the following source data tables. First is a list of automobile manufacturers.

Mfg_IDManufacturer
1Aston Martin
2Porsche
3Lamborghini
4Ferrari
5Koenigsegg

Next is a list of automobile models with a manufacturer ID. Note that there are several models with no manufacturer.

ModelNameMfg_ID
Aventador3
Countach3
DBS1
Enzo4
One-771
Optimus Prime
Batmobile
Agera5
Lightning McQueen

To get a list of models by manufacturer, it makes sense to join on Mfg_ID. By leveraging outer join concepts, the output will also be able to show those items which do not have any matches.

First, specify parameters for Table 1 Data Selection. The source data table is selected and all columns are listed.

Next, specify parameters for Table 2 Data Selection. Once again, the source data table is selected and all columns are listed.

Finally, the join conditions are set in the Table Output tab. Using the Guess button, Analyze properly identifies the Mfg_ID column to use as the Join Key. Lastly, the

Target Output Columns are specified automatically using the Propagate button. This effectively includes all columns from all tables, with any join columns obviously only being included a single time. Note that the columns are sorted alphabetically, first by Manufacturer and next by ModelName.

As expected, the final output includes all rows from both tables, whether they had a match in both tables or not. As such, this time Porsche does indeed show up despite having no models. Additionally, Batmobile, Lightning McQueen, and Optimus Prime are included in the results even though none of them have a manufacturer. Besides, who can say ‘No’ to them?

4.15.15 - Table Pivot

Flip rows to columns

Description

Used to convert long, narrow data tables into short, wide data tables. Selected columns are transposed, with the column names converted into values across multiple columns.

Perhaps the easiest example to understand is to think of a data table with months listed as rows:

Table Pivot Input

Pivoting this data table would convert all of the month rows into columns.

Table Pivot Output

By specifying which columns to transpose and which columns to leave alone, this becomes a powerful tool. Making this conversion in other ETL tools could require a dozen more steps.

Source and Target Parameters

Table Pivot Source Target

Source Table Selection

To establish the source and target, first select the data table to be extracted from using the dropdown menu.

Traget Table Selection

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Pivot Column Selection

The Category Column to Transform into Column Headers is where you specigy the column in Source Table that will be pivoted to rows. The Value Column ti Pivot to Column Vales is the column that containes the values in the Source Table. The Value Aggregation Option is where you specify how you want the data to aggregate.

Table Data Selection

Table Pivot Data Selection

The Table Data Selection tab is used to map columns from the source data table to the target data table. All source columns on the left side of the window are automatically mapped to the target data table depicted on the right side of the window. Using the Inspect Source menu button, there are a few additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

In addition to each of these options, each choice offers the ability to preview the source data.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

To rearrange columns in the target data table, select the desired column(s), then right click and select Move to Top, Move Up, Move Down, or Move to Bottom.

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return distinct results only.

To aggregate results, select the Summarize menu option. This will toggle a set of drop down boxes for each column in the target data table. The following summarization options are available:

  • Group by (set as default)
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Mean
  • Median
  • Mode
  • Std Dev
  • Variance
  • Product
  • Absolute Val
  • Quantile
  • Skew
  • Kurtosis
  • Mean Abs Dev
  • Cumulative Sum
  • Cumulative Min
  • Cumulative Max
  • Cumulative Product

For more aggregation details, see the Analyze overview page here.

Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset of Data

Any valid Python expression is acceptable to subset the data. Please see Expressions

for more details and examples.

Apply Secondary Filter To Result Data

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples

Final Data Table Slicing (Limit)

To limit the data, simply check the Apply Row Slicer box and then specify the following:

  • Initial Rows to Skip: Rows of data to skip (column header row is not included in count)
  • End at Row: Last row of data to include. This is different from simply counting rows at the end to drop

4.15.16 - Table Union All

Access history to all created workflow data tables

Description

Use to combine multiple data tables with the same column structure into a single data table. For example, time series data is a prime candidate for this transform. The result is all of the records from the combined tables.

Sources

The Sources section serves as a collection of all data tables to append together. Typically, all of the data tables will have the same (or similar) column structure. There are two buttons available to add a data table to the list:

  • Insert Row
  • Append Row

Additionally, right-clicking in the Select Source to Edit window will display the same options. Right-clicking on a table already added will also display the Delete option.

To execute the transform properly, there will need to be one entry in the Sources section for every source data table to append together. These entries are listed in the order in which they will be appended. To adjust the order, right-clicking on a table will display the following options:

  • Move Down (if applicable)
  • Move To Bottom (if applicable)
  • Move Up (if applicable)
  • Move To Top (if applicable)

By default, each source is named New Table, but the modeler is encouraged to provide descriptive names by double-clicking the name and renaming accordingly.

Target Table

By default, the Target Table is left blank. Before naming, note that data tables must follow Linux naming conventions. As such, we recommend that names only consist of alphanumeric characters. Analyze will automatically scrub any invalid characters from the name. Additionally, it will limit the length to 256 characters, so be concise!

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Table Data Selection Tab

Source Table

Table Selection

There are two options for selecting the table or in the second option tables to:

The first option is to use the Specific Table dropdown to select the table.

The second is to use the Tables Matching Search option in which you specify the Search Path and Search Text to select the table or tables that match the search criteria. This option is very useful if you have a workflow that creates a series of commonly named tables that that have been saved appending the date.

Table Dymanic Selection

Source Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.15.17 - Table Union Distinct

Consolidate data tables

Description

Use to combine multiple data tables with the same column structure into a single data table. For example, time series data is a prime candidate for this transform. The result is always the distinct set of records after combining the data.

Sources

The Sources section serves as a collection of all data tables to append together. Typically, all of the data tables will have the same (or similar) column structure. There are two buttons available to add a data table to the list:

  • Insert Row
  • Append Row

Additionally, right-clicking in the Select Source to Edit window will display the same options. Right-clicking on a table already added will also display the Delete option.

To execute the transform properly, there will need to be one entry in the Sources section for every source data table to append together. These entries are listed in the order in which they will be appended. To adjust the order, right-clicking on a table will display the following options:

  • Move Down (if applicable)
  • Move To Bottom (if applicable)
  • Move Up (if applicable)
  • Move To Top (if applicable)

By default, each source is named New Table, but the modeler is encouraged to provide descriptive names by double-clicking the name and renaming accordingly.

Target Table

By default, the Target Table is left blank. Before naming, note that data tables must follow Linux naming conventions. As such, we recommend that names only consist of alphanumeric characters. Analyze will automatically scrub any invalid characters from the name. Additionally, it will limit the length to 256 characters, so be concise!

Target Table

Table Target

To establish the target table select either an existing table as the target table using the Target Table dropdown or click on the green "+" sign to create a new table as the target.

Table Creation

When creating a new table you will have the option to either create it as a View or as a Table.

Views:

Views are useful in that the time required for a step to execute is significantly less than when a table is used. The downside of views is they are not a useful for data exploration in the table Details mode.

Tables:

When using a table as the target a step will take longer to execute but data exploration in the Details mode is much quicker than with a view.

Table Data Selection Tab

Source Table

Table Selection

There are two options for selecting the table or in the second option tables to:

The first option is to use the Specific Table dropdown to select the table.

The second is to use the Tables Matching Search option in which you specify the Search Path and Search Text to select the table or tables that match the search criteria. This option is very useful if you have a workflow that creates a series of commonly named tables that that have been saved appending the date.

Table Dymanic Selection

Source Columns

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

4.15.18 - Table Append

Used append data to an existing table.

Description

Used append data to an existing table.

Load Parameters

Source and Target

Source And Target

To establish the source and target tables, first select the data table to be extracted from using the Source Table dropdown menu. Next, select an existing table as the target table using the Target Table dropdown.

Table Data Selection

When configuring the Data Mapper the columns in the source table must be mapped to a column in the target table.

Data Mapper Configuration

Table Data Mapper

The Data Mapper is used to map columns from the source data to the target data table.

Inspection and Populating the Mapper

Using the Inspect Source menu button provides additional ways to map columns from source to target:

  • Populate Both Mapping Tables: Propagates all values from the source data table into the target data table. This is done by default.
  • Populate Source Mapping Table Only: Maps all values in the source data table only. This is helpful when modifying an existing workflow when source column structure has changed.
  • Populate Target Mapping Table Only: Propagates all values into the target data table only.

If the source and target column options aren’t enough, other columns can be added into the target data table in several different ways:

  • Propagate All will insert all source columns into the target data table, whether they already existed or not.
  • Propagate Selected will insert selected source column(s) only.
  • Right click on target side and select Insert Row to insert a row immediately above the currently selected row.
  • Right click on target side and select Append Row to insert a row at the bottom (far right) of the target data table.

Deleting Columns

To delete columns from the target data table, select the desired column(s), then right click and select Delete.

Chaging Column Order

To rearrange columns in the target data table, select the desired column(s). You can use either:

  • Bulk Move Arrows: Select the desired move option from the arrows in the upper right
  • Context Menu: Right clikc and select Move to Top, Move Up, Move Down, or Move to Bottom.

Reduce Result to Distinct Records Only

To return only distinct options, select the Distinct menu option. This will toggle a set of checkboxes for each column in the source. Simply check any box next to the corresponding column to return only distinct results.

Depending on the situation, you may want to consider use of Summarization instead.

The distinct process retains the first unique record found and discards the rest. You may want to apply a sort on the data if it is important for consistency between runs.

Aggregation and Grouping

To aggregate results, select the Summarize menu option. This will toggle a set of select boxes for each column in the target data table. Choose an appropriate summarization method for each column.

  • Group By
  • Sum
  • Min
  • Max
  • First
  • Last
  • Count
  • Count (including nulls)
  • Mean
  • Standard Deviation
  • Sample Standard Deviation
  • Population Standard Deviation
  • Variance
  • Sample Variance
  • Population Variance
  • Advanced Non-Group_By

For advanced data mapper usage such as expressions, cleaning, and constants, please see the Advanced Data Mapper Usage

Data Filters

Table Data Filters

To allow for maximum flexibility, data filters are available on the source data and the target data. For larger data sets, it can be especially beneficial to filter out rows on the source so the remaining operations are performed on a smaller data set.

Select Subset Of Data

This filter type provides a way to filter the inbound source data based on the specified conditions.

Apply Secondary Filter To Result Data

This filter type provides a way to apply a filter to the post-transformed result data based on the specified conditions. The ability to apply a filter on the post-transformed result allows for exclusions based on results of complex calcuations, summarizaitons, or window functions.

Final Data Table Slicing (Limit)

The row slicing capability provides the ability to limit the rows in the result set based on a range and starting point.

Filter Syntax

The filter syntax utilizes Python SQLAlchemy which is the same syntax as other expressions.

View examples and expression functions in the Expressions area.

Examples

4.16 - Workflow Control Steps

4.16.1 - Run Workflow

Run an existing workflow

Description

“Run Workflow” runs an existing workflow.

Workflow to Run

First, select the Project which contains the workflow to be run from the Project dropdown menu.

Next, select the particular workflow to be run from the Workflow dropdown menu.

Additionally, there is an option to Wait until processing completes before continuing. Selecting this checkbox will defer execution of the current workflow until the called workflow is completed with its execution. By default, this option is disabled, meaning that the current workflow in which this transform resides will continue processing in parallel along with the called workflow.

Examples

No examples yet...

4.16.2 - Set Project Variable

Set a project variable for use during a workflow

Description

“Set Project Variable” sets project variables for use during the workflow. A variable name and value may contain any combination of valid characters, including spaces. Variables are referenced within the workflow by placing them inside curly braces. For example, a_variable is referenced within a transform as {a_variable} so it could be used in something like a formula or field value (e.g., {a_variable} * 2).

Variable List

The table will display the list of registered project variables and the current values. Enter the value for the variable desired. It’s also possible to set variable values without registering the variable first by simply adding the variable to the list.

Examples

No examples yet...

4.16.3 - Set Workflow Variable

Set variables during a workflow

Description

“Set Workflow Variable” sets workflow variables for use during the workflow. A variable name and value may contain any combination of valid characters, including spaces. Variables are referenced within the workflow by placing them inside curly braces. For example, a_variable is referenced within a transform as {a_variable} so it could be used in something like a formula or field value (e.g. {a_variable} * 2).

Variable List

The table will display the list of registered workflow variables and the current values. Enter the value for the variable desired. It’s also possible to set variable values without registering the variable first by simply adding the variable to the list.

Examples

No examples yet...

4.16.4 - Create Workflow

Create a new workflow in 'Analyze'

Description

Create a new PlaidCloud Analyze workflow.

Workflow to Create

First, select the Project in which the new workflow should be created from the dropdown menu.

Next, type in a workflow name. The name should be unique to the Project.

Examples

No examples yet...

4.16.5 - Delete Workflow

Delete an existing PlaidCloud Analyze workflow

Description

Delete an existing PlaidCloud Analyze workflow.

Workflow to Delete

First, select the Project which contains the workflow to be deleted from the Project dropdown menu.

Next, select the particular workflow to be deleted from the Workflow dropdown menu.

Examples

No examples yet...

4.16.6 - Clear Workflow Log

Clear the Log from an existing PlaidCloud 'Analyze' Workflow

Description

Clear the log from an existing PlaidCloud Analyze workflow.

Workflow Log to Clear

First, select the Project which contains the workflow log to be cleared from the Project dropdown menu.

Next, select the particular workflow log to be cleared from the Workflow dropdown menu.

4.16.7 - Copy Workflow

Make a copy of an existing PlaidCloud Analyze workflow

Description

Make a copy of an existing PlaidCloud Analyze workflow.

Workflow to Copy

First, select the Project which contains the workflow to be copied from the Project dropdown menu.

Next, select the particular workflow to be copied from the Workflow dropdown menu.

Next, enter the new workflow name into the New Workflow field. Remember: the name should be unique to the Project.

Examples

No examples yet...

4.16.8 - Raise Workflow Error

Raises an error in a workflow

Description

Raise an error in a PlaidCloud Analyze workflow.

Raise Workflow Error

Mainly for use with step conditions, the step can be set to execute if conditions are met and raise an error within the workflow

4.16.9 - Rename Workflow

Rename an Existing PlaidCloud Analyze Workflow

Description

Rename an existing PlaidCloud Analyze workflow.

Workflow to Rename

First, select the Project which contains the workflow to be renamed from the Project dropdown menu.

Next, select the particular workflow to be renamed from the Workflow dropdown menu.

Finally, enter the new workflow name in the Rename To field. Remember that the name should be unique to the Project.

Examples

No examples yet...

4.16.10 - Stop Workflow

Stop an existing, running workflow

Description

“Stop Workflow” stops an existing, running workflow.

Workflow to Stop

First, select the Project which contains the workflow to be stopped from the Project dropdown menu.

Next, select the particular workflow to be stopped from the Workflow dropdown menu.

Examples

No examples yet...

4.16.11 - Worklow Loop

Runs a workflow looping over a dataset as Project variables

Description

Loops over a dataset and runs a specific workflow using the values of the looping dataset as Project variables.

Workflow to Stop

First, select the Project which contains the workflow that will be run on each loop from the Project dropdown menu.

Next, select the particular workflow for running from the Workflow dropdown menu.

Examples

No examples yet...

5 - Dashboards

5.1 - Learning About Dashboards

Understanding Dashboard features and how to troubleshoot errors and warnings

Description

Dashboard is an intricate tool which so much flexibility it is impossible to describe all its uses. You will discover some aspects of Dashboard while using it. Naturally, there will also be times when you run into trouble. A member of the Tartan Support Team is always available to assist you, but we have also compiled some tips below in case you run into a similar problem.

Common Questions and Answers for Dashboard

Preferred Browser

Due to frequent caching, Google Chrome is usually the best web browser to use with Dashboard. If you are using another browser and encounter a problem, we suggest first clearing the cache and cookies to see if that resolves the issue. If not, then we suggest switching to Google Chrome and seeing if the problem recurs.

Sync Delay

  • Problem: After unpublishing and publishing tables in superset, the data does not appear to be syncing properly.
  • Solutions: Refresh the dashboard. Currently, old table data is cached, so it is necessary to refresh the dashboard when rebuilding tables.

Table Sync Error

  • Problem: After recreating a table using the same published name as a previous table, the table is not syncing, even after hitting refresh on the dashboard, publishing, unpublishing, and republishing the table.
  • Solutions: Republish the table with a different name. Superset does not allow a duplicate tables, or tables with the same published name and project ID.

Cache Warning

  • Problem: A warning popped up on the upper right saying “Loaded data cached 3 hours ago. Click to force-refresh.”
  • Solutions: Click on the warning to force-refresh the cache. You can also click the drop-down menu beside “Edit dashboard” and select “Force refresh dashboard” there. Either of these options will refresh within the system and is preferred to refreshing the web browser itself.

Permission Warning

  • Problem: My published dashboard is populating with the same error in each section where data should be populated: “This endpoint requires the datasource… permission”

  • Solutions: Check that the datasources are not old. Most likely, the charts are pulling from outdated material. If this happens, update the charts with new datasources.

  • Problem: I am getting the same permission warning from above, but my colleague can view the chart data.

  • Solutions: If the problem is that one individual can see the data in the charts and another cannot, the second person may need to be granted permission by someone within the permitted category. To do so:

    1. Go to Charts
    2. Select the second small icon of a pencil and paper next to the chart you want to grant access to
    3. Click Edit Table
    4. Click Detail
    5. Click Owners and add the name of the person you want to grant access to and save.

Saving Modified Filters to Dashboard

  • Problem: I modified filters in my draft model and want to save them to my dashboard. The filters are not in the list. In my draft model, a warning stated, “There is no chart definition associated with this component, could it have been deleted? Delete this container and save to remove this message.”
  • Solutions: Go to “Edit Chart.” From there, make sure the “Dashboards” section has the correct dashboard filled in. If it is blank, add the correct dashboard name.

Formatting Numbers: Breaks

  • Problem: My number formatting is broken and out of order.
  • Solutions: The most likely reason for this break is the use of nulls in a numeric column. Using a filter, eliminate all null numeric columns. Try running it again. If that does not work, review the material provided here: http://bl.ocks.org/zanarmstrong/05c1e95bf7aa16c4768e or here: https://github.com/apache-superset/superset-ui/issues. Finally, always feel free to reach out to a Tartan Support team member. This problem is known, and a more permanent solution is being developed.

Formatting Numbers

To round numbers to nearest integer:

  1. Do not use: ,.0f
  2. Instead use: ,d or $,d for dollars

Importing Existing Dashboard

  • Problem: I’m importing an existing dashboard and getting an error on my export.
  • Solutions: First, check whether the dashboard has a “Slug.” To do this, open Edit Dashboard, and the second section is titled Slug. If that section is empty or says “null,” then this is not the problem. Otherwise, if there is any other value in that field, you need to ensure that export JSON has a unique slug value. Change the slug to something unique.

5.2 - Using Dashboards

Create and edit data tables within dashboard and explore the data

Description

Usually, members will have access to multiple workspaces and projects. Having this data in multiple spots, however, may not always be desirable. This is why PlaidCloud allows the ability to view all of the accessible data in a single location through the use of dashboards and highly intuitive data exploration. PlaidCloud Dashboards (where the dashboards and data exploration are integrated) provides a rich pallet of visualization and data exploration tools that can operate on virtually any size dataset. This setup also makes it possible to create dashboards and other visualizations that combine information across projects and workspaces, including Ad-hoc analysis.

Editing a Table

The message you receive after creating a new table also directs you to edit the table configuration. While there are more advanced features to edit the configuration, we will start with a limited and more simple portion. To edit table configuration:

  1. Click on the edit icon of the desired table
  2. Click the “List Columns” tab
  3. Arrange the columns as desired
  4. Click “Save”

This allows you to define the way you want to use specific columns of your table when exploring your data.

  • Groupable: If you want users to group metrics by a specific field
  • Filterable: If you need to filter on a specific field
  • Count Distinct: If you want want to get the distinct count of this field
  • Sum: If this is a metric you want to sum
  • Min: If this is a metric you want to gather basic summary statistics for
  • Max: If this is a metric you want to gather basic summary statistics for
  • Is temporal: This should be checked for any date or time fields

Exploring Your Data

To start exploring your data, simply click on the desired table. By default, you’ll be presented with a Table View.

Getting a Data Count

To get a the count of all your records in the table:

  1. Change the filter to “Since”

  2. Enter the desired since filter

    • You can use simple phrases such as “3 years ago”
  3. Enter the desired until filter

    • The upper limit for time defaults is “now”
  4. Select the “Group By” header

  5. Type “Count” into the metrics section

  6. Select “COUNT(*)”

  7. Click the “Query” button

You should then see your results in the table.

If you want to find the count of a specific field or restriction:

  1. Type in the desired restriction(s) in the “Group By” field
  2. Run the query

Restricting Result Number

If you only need a certain number of results, such as the top 10:

  1. Select “Options”
  2. Type in the desired max result count in the “Row Limit” section
  3. Click “Query”

Additional Visualization Tools

To expand abbreviated values to their full length:

  1. Select “Edit Table Config”
  2. Click “List Sql Metric”
  3. Click “Edit Metric”
  4. Click “D3Format”

To edit the unit of measurement:

  1. Select “Edit Table Config”
  2. Click “List Sql Metric”
  3. Click “Edit Metric”
  4. Click “SQL Expression”

To change the chart type:

  1. Scroll to “Chart Options”
  2. Fill in the required fields
  3. Click “Query”

From here you are able to set axis labels, margins, ticks, etc.

5.3 - Example Calculated Columns

Examples of calculated column expressions

Description

Data in dashboards can be augmented with calculated columns. Each dataset will contain a section for calculated columns. Calculated columns can be written and modified with PostgreSQL-flavored SQL.

In order to view and edit metrics and calculated expressions, perform the following steps:

  1. Sign into plaidcloud.com and navigate to dashboards
  2. From within visualize.plaidcloud.com, navigate to Data > Datasets
  3. Search for a dataset to view or modify
  4. Modify the dataset by hovering over the edit button beneath Actions

Examples

count

COUNT(*)

min

min("MyColumnName")

max

max("MyColumnName")

coalesce (useful for converting nulls to 0.0, for instance)

coalesce("BaselineCost",0.0)

divide

divide, with a hack for avoiding DIV/0 errors

sum("so_infull")/(count(*)+0.00001)

conditional statement

CASE WHEN "Field_A"= 'Foo' THEN max(coalesce("Value_A",0.0)) - max(coalesce("Value_B",0.0)) END

5.4 - Example Metrics

Examples of common metrics

Description

Data in dashboards can be augmented with metrics. Each dataset will contain a section for Metrics. Metrics can be written and modified with PostgreSQL-flavored SQL.

In order to view and edit metrics and calculated expressions, perform the following steps:

  1. Sign into plaidcloud.com and navigate to dashboards
  2. From within visualize.plaidcloud.com, navigate to Data > Datasets
  3. Search for a dataset to view or modify
  4. Modify the dataset by hovering over the edit button beneath Actions

Examples

Calculated columns are typically additional columns made by combining logic and existing columns.

convert a date to text

to_char("week_ending_sol_del_req", 'YYYY-mm-dd')

5.5 - Dashboard Example Expressions

Using common dashboard expressions

Description

Data in dashboards can be augmented with expressions. There are two primary types of expressions, metrics and calculated columns. Both can be written and modified with PostgreSQL-flavored SQL.

In order to view and edit metrics and calculated expressions, perform the following steps:

  1. Sign into plaidcloud.com and navigate to dasboards
  2. From within visualize.plaidcloud.com, navigate to Data > Datasets
  3. Search for a dataset to view or modify
  4. Modify the dataset by hovering over the edit button beneath Actions

Each dataset will contain a section for Metrics and a section for Calulated Columns.

Metrics

Metrics are expressions that are typically used to describe a field, including how it will be consolidated.

cast

cast("MyColumnName" AS datatype)    'Converts a value (of any type) into a specified datatype

Parameter Values

ParameterDescription
MyColumnNameThe value to convert
datatypeThe datatype to convert expression to

Example:

cast("MyColumnName" AS NUMERIC)

coalesce (useful for converting nulls to 0.0, for instance)

coalesce("BaselineCost",0.0)

divide, with a hack for avoiding DIV/0 errors

sum("so_infull")/(count(*)+0.00001)

count

count(*)

distinct

distinct("MyColumnName")

first

first("MyColumnName")

min

min("MyColumnName")

max

max("MyColumnName")

nullif

nullif("MyColumnName","MyColumnName2")  'returns NULL if two expressions are equal, otherwise it returns the first expression.

Parameter Values

ParameterDescription
MyColumnNameExpression to be compared
MyColumnName2Expression to be compared

over

over(
    [ <PARTITION BY MyColumnName> ]  
    [ <ORDER BY MyColumnName> ]   
    [ <ROW or RANGE MyColumnName> ]  
    )     
ArgumentsDescription
PARTITION BYDivides the query result set into partitions
ORDER BYDefines the logical order of the rows within each partition of the result set
ROW/RANGELimits the rows within the partition by specifying start and end points within the partition

Example:

sum("MyColumnName") over (partition by "MyColumnName2")

round

round("MyColumnName",2) 

Parameter Values

ParameterDescription
NumberNumber to be rounded
DecimalsNumber of decimal places to round number to

to_char

to_char("MyColumnName", 'DateFormat')

Parameter Values

ParameterDescription
MyColumnNameA number or date that will be converted to a string
DateFormatFormat that will be used to convert value to a string

sum

sum("MyColumnName")

Conditional statement

CASE WHEN "Field_A"= 'Foo' THEN max(coalesce("Value_A",0.0)) - max(coalesce("Value_B",0.0)) END

## Calculated Columns
Calculated columns are typically additional columns made by combining logic and existing columns.

Convert a date to text

to_char("week_ending_sol_del_req", 'YYYY-mm-dd')

6 - Data Management

6.1 - Table Explorer

Table Explorer provides powerful and readily accessible data exploration capabilities

Table Explorer provides a powerful and readily accessible data exploration tool with built in filtering, summarization, and other features to make life easy for people working with large and complex data.

Table Explorer supports exploration on any size dataset so you can use the same tool no matter how much your data grows. It also provides point-and-click filtering along with advanced filter capabilities to zero in on the data you need. The best part is that anywhere in PlaidCloud with tables or views, you can click on those tables and views to explore with Table Explorer. By being fully integrated, data access is only a click away.

The Grid view provides a tabular view of the data. The Details view provides a summary of each column, a count of unique values, and summary statistics for numeric columns.

Data can be exported directly from a filtered set as well as being able to save and share filters with others. Finally, the filters and column settings can be saved directly as a workflow Extract step.

The Grid View

The Grid view provides a tabular view of the data.

Setting the row limit

By default, the row limit is set to 5,000 rows. However, this can be adjusted or disabled entirely.

The rows shown along with the total size of the dataset are shown at the bottom of the table. The information provides three key pieces of information:

  1. The current row count shown based on the row limit applied
  2. The size of the global data after filters are applied
  3. The size of the unfiltered global data

Sorting locally versus globally

The Grid view provides the ability to click on the column header and sort the data based on that column. However, this method is only sorting the dataset that has already been retrieved and is not sorting based on the full dataset. If your retrieved data contains the entire dataset this distinction is immaterial however if your full dataset is larger than what appears in the browser, this may not be the desired sort result.

If you desire to sort the global dataset before retrieving the limited data that will appear in your browser those sorts can be applied to the columns in the Details view by clicking on the Sort icon at the top of each column. An additional benefit of using the global sort approach is that you can apply multiple sorts along with a mix of sort directions.

Quick reference column list

All of the columns in the table or view are shown on the left of the Table Explorer window by default. This column list can be toggled on and off using the column list toggle button.

The column list provides a number of quick access and useful features including:

  • Double clicking an item jumps to the column in the Grid or Details view
  • Control visibility of the column through the visibility checkbox
  • Use multi-select and right-click to include or exclude many columns at once
  • Quickly view the data type of each column using the data type icons
  • View the total column count

The Details View

The Details view provides an efficient way to view the data at a high level and exposes tools to quickly filter down to information with point-and-click operations.

Column data and unique counts

Each column is shown, provided it is currently marked as visible. The column summary displays the top 1,000 unique values by count. The number of unique values shown can be adjusted by selecting the Detailed Rows Displayed selection for a different value.

Managing point-and-click filters

Each column provides for point-and-click filtering by activating the filter toggle at the top of the column. Select the items in the column that you would like to include in the resulting data. Multi-select is supported.

Once you apply a filter, there may be items you wish to remove or to clear the entire column filter without clearing all filters. This is accomplished by selecting the dropdown on the column filter button and unchecking columns or selecting the clear all option at the top.

Managing Summarization

Summarization of the data can be applied by toggling the Summarize button to On. When the Summarize button is activated, each column will display a summarization type to apply. Adjust the summarization type desired for each column.

When the desired summarizations are complete, refresh the data and the summarizations will be applied.

Finding Distinct Values

Activating the Distinct button will help reduce the data to only a set of unique records. When the Distinct button is active, a Distinct checkbox will appear on each column. Uncheck the columns that DO NOT define uniqueness of the column to the dataset. For example, if you want to find the unique set of customers in a customer order table, you would only want to select the customer column rather than including the customer order number too.

Summary statistics for numeric columns

Integer and numeric columns automatically display summary statistics at the bottom of the column information. This includes:

  • Min
  • Max
  • Mean
  • Sum
  • Standard Deviation
  • Variance

These statistics are calculated on the full filtered dataset.

Copying Data

It is sometimes useful to allow for copying of selected data from PlaidCloud so that it can be pasted into other applications such as a spreadsheet.

From the Copy button in the upper right, there are several copy options available for the data:

  • Copy All - Copies all of the data to the clipboard
  • Copy Selection - Copies the selected data to the clipboard
  • Copy Cell - Copies only the contents of a single cell to the clipboard
  • Copy Column - Copies the full contents of the column to the clipboard

Exporting Data

Exporting data from the Table Explorer interface allows exporting of the filtered data with only the columns visible. You can export in the following formats:

  • Microsoft Excel (xlsx)
  • CSV (Comma)
  • TSV (Tab)
  • PSV (Pipe)

The Download menu also offers the ability to download only the rows visible in the browser. This is based on using the row limit specified.

Additional Actions

Additional useful actions are available under the Actions menu.

Save as Extract Step

When exploring data, it is often in the context of determining how to filter it for a data pipeline process. This often consists of applying multiple filters including advanced filters to zero in on the desired result.

Instead of attempting to replicate all the filters, columns, summarizations, and sorts in an Extract Step, you can simply save the existing Table Explorer settings as a new Extract Step.

Save as View

Similar to saving the current Table Explorer settings as an Extract Step above, you can also save the settings directly as a view.

This can be particularly useful when trying to construct slices of data for reporting or other downstream processes that don't require a a data pipeline.

Manage Saved Filters

You never have to lose your filter work. You can save your Table Explorer settings as a saved filter. Saved filters also include column visibility, summarizations, columns filters, advanced filters, and sorts.

You can also let others use a saved filter by checking the Public checkbox when saving the filter.

From the Actions menu you can also choose to delete and rename saved filters.

Advanced Filters

While point-and-click column filters allow for quick application of filters to zero in on the desired results, sometimes filter conditions are complex and need more advanced specifications.

The advanced filter area provides both a pre-aggregation filter as well as a post-aggregation filter, if Summarize is enabled.

Any valid Python expression is acceptable to subset the data. Please see Expressions for more details and examples.

6.2 - Publishing Tables

Publishing Tables and Views to allow usage in Dashboard, PlaidXL, and other external reporting

Since data pipelines can generate many intermediate tables and views useful for validation and process checks but not suitable for final results reporting, PlaidCloud provides a Publish process to help reduce the noise when building Dashboards or pulling data in PlaidXL. The Publish process helps clarify which tables and views are final and reliable for reporting purposes.

Publish

From the Tables tab in a PlaidCloud project configuration, find the table you wish to publish for use in dashboards and PlaidXL. Right-click on the table record and select Set Published Table Reporting Name from the menu.

This will open a dialog where you can specify a unique published name. This name does not need to be the same as the table or view name. Enabling a different name is often useful when referencing data sources in dashboards and PlaidXL because it can provide a friendlier name to users.

Once the table or view is published, its published name will appear in the Published As column in the Tables view.

Unpublish

Unpublishing a table or view is similar to the publish process. From the Tables tab in a PlaidCloud project configuration, find the table you wish to publish for use in dashboards and PlaidXL. Right-click on the table record and select Set Published Table Reporting Name from the menu.

When the dialog appears to set the published name, select the Unpublish button. This will remove the table from Dashboard and PlaidXL usage.

The published name will no longer appear in the Published As column.

Renaming

Renaming a table or view is similar to the publish process. From the Tables tab in a PlaidCloud project configuration, find the table you wish to publish for use in dashboards and PlaidXL. Right-click on the table record and select Set Published Table Reporting Name from the menu.

When the dialog appears change the publish name to the new desired name. Press the Publish button to update the name.

The updated name will now appear in the Published As column as well as in Dashboard and PlaidXL.

6.3 - Using Tables and Views

Using and managing tables and views

Tabular data and information in PlaidCloud is stored in Greenplum data warehouses. This provides massive scalability and performance while using well understood and mature technology to minimize risk of data loss or corruption.

In addition, utilizing a data warehouse that operates with a common syntax allows 3rd party tools to connect and explore data directly. Essentially, this makes the PlaidCloud data ecosystem open and explorable while also ensuring industry leading security and access controls.

Tables

Tables hold the physical tabular data throughout PlaidCloud. Individual tables can hold many terabytes of data if needed. Data is stored across many physical servers and is automatically mirrored to ensure data integrity and high availability.

Tables consist of columns of various data types. Using an appropriate data type can help with performance and especially the storage size of your data. PlaidCloud can do a better job of compressing the data if the data is using the most appropriate data type too. This is usually guessed by PlaidCloud but it is also possible to change the data types using the column mappers in workflow steps.

Views

Views act just like tables but don't hold any physical data. They are logical representations of tables derived through a query. Using views can save on storage.

There are some limitations to the use of views though. Just be aware of the following:

  • View Stacking Performance - View stacking (view of a view of a view...etc) can impact performance on very large tables or complex calculations. It might be necessary to create intermediate tables to improve performance.
  • Dashboard Performance - While perfectly fine to publish a view for Dashboard use, for very large tables you may want to publish a table rather than a view for optimal user experience.
  • Dynamic Data - The data in a view changes when the underlying referenced table data changes. This can be both a benefit (everything updates automatically) or an unexpected headache if the desire was a static set of data.

6.4 - Using Dimensions (Hierarchies)

Using and managing hierarchical data

PlaidCloud natively manages dimension (i.e. hierarchical) data through our proprietary hierarchy storage system. We decided to construct our own from purpose-built solution because other commercial and open-source solutions seem to present limitations that were not easily overcome.

The hierarchy storage supports not only hierarchical relationships but also properties, aliases, attributes, and values. It is also designed to operate on large structures and perform operations quickly including complex branch and leaf navigation.

Dimensions are managed in the Dimensions tab within each PlaidCloud project configuration area.

Main Hierarchy

Each dimension (i.e. hierarchical dataset) always consists of a main hierarchy. Every member of the hierarchy is represented here.

Having a main hierarchy helps establish the complete set of leaf nodes in the dimension.

Alternate or Attribute Hierarchies

Alternate hierarchies are different representations of the main hierarchy leaf nodes. Alternate hierarchies can consist of a subset of both leaf nodes and roll-up (i.e. folders) in the main hierarchy as well as its own set of unique roll-ups.

This provides for the maximum amount of flexibility by automatically updating alternate hierarchies when children of a roll-up change or to strictly control the alternate hierarchy members by specifying only the leaf nodes required.

Managing Dimensions

Creating a Dimension

From the New button in the toolbar, select New Dimension. Enter in the desired name, directory, and a descriptive memo.

Once you press the Create button the dimension will be created and ready for immediate use.

You can also create a dimension from a workflow using the Dimension Create workflow step.

Deleting a Dimension

To delete an existing dimension, select the dimension record and open the Actions menu in the upper right. Select Delete Dimension.

This will delete the dimension and all underlying data.

You can also delete a dimension from a workflow using the Dimension Delete workflow step.

It is also possible to clear the dimension of all structure, values, aliases, properties, and alternate hierarchies without deleting the dimension by using the Dimension Clear workflow step.

Copying a Dimension

To copy an existing dimension, select the dimension record and open the Actions menu in the upper right. Select Copy Dimension.

This will open a dialog where you can specify the name of the copy. Click the Create Copy button to make a copy of the dimension including values, aliases, properties, and alternate hierarchies.

Sorting a Dimension

The dimension management area makes it easy to move hierarchy members up and down as well as changing parents. It also makes it easy to create and delete members.

However, it can get tedious when manually moving hierarchy items around so you can sort a dimension from a workflow using the Dimension Sort workflow step. This can be a big time saver especially after data loads or major changes.

Loading Dimensions

Since dimensions represent hierarchical data structures, the load process must convey the relationships in the data. PlaidCloud supports two different data structures for loading dimensions:

  • Parent-Child - The data is organized vertically with a Parent column and Child column defining each parent of a child throughout the structure
  • Levels - The data is organized horizontally with each column representing a level in the hierarchy from left to right

In addition to structure, other dimension information can be included in the load process such as values, aliases, and properties.

See the Workflow Step for Dimension Load for more information.

7 - Document

7.1 - Using Document

Upload, download, delete, and view files in Document accounts

Several file operations are available within a Document Account browser. All operations are accessible from a right-click menu within the file browser. The right-click menu provides specific options depending on whether a folder or file is selected.

To open the file explorer:

  1. Click on the folder icon (far left) from the list of private or shared accounts

Opening File Explorer

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore

The various file and folder operations available in the file explorer are detailed below:

  • Folders:
    • uploading new folders
    • creating new folders
    • renaming, deleting, and downloading current folders as ZIPs
  • Files:
    • downloading new files
    • renaming, deleting, and refreshing current files.

Upload a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Right-click and select Upload Here

Download a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired file
  5. Right-click and select Download

Rename a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired file
  5. Right-click and select Rename

Move a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired file
  5. Drag into desired folder
  6. Select Move File

Copy a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired file
  5. Right-click and select Copy

Delete a File

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired file
  5. Right-click and select Delete

Create a Folder

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Click “New Top Level Folder”
  4. Enter a folder name of your choosing
  5. Click Create

Rename a Folder

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired folder
  5. Right-click and select Rename

Move a Folder

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired folder
  5. Drag into desired folder
  6. Select Move Folder

Delete a Folder

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired folder
  5. Right-click and select Delete

Download Folder Contents (zip file)

The Download as Zip option is for downloading many files at once. This option will zip (compress) all contents of the selected folder and download the zip file (.zip extension). For easy navigation, the zip file retains the directory structure that exists in the file explorer.

  1. Go to Document > Shared Accounts
  2. Select the folder icon (far left) for the account you wish to explore
  3. Browse to the desired directory
  4. Left-click to select the desired folder
  5. Right-click and select Download as ZIP

7.2 - Account and Access Management

7.2.1 - Control Document Account Access

Set access controls for Document accounts

Four types of access restrictions are available for an account: Private, Workspace, Member Only, and Security Group. The type of restriction set for a user is editable at any time from the account form.

Updating Account Access

  1. Select Document > Manage Accounts within PlaidCloud
  2. Enter the edit mode on the account you wish to change
  3. Select the desired access level restriction located under Security Model
  4. Select the Save button

Restriction Options

All Workspace Members

This access is the simplest since it provides access to all members of the workspace and does not require any additional assignment of members.

Specific Members Only

This access setting requires assignment of each member to an account. This option is particularly useful when combined with the single sign-on option of assigning members based on a list of groups sent with the authentication. However, for workspaces with large numbers of members, this approach can often require more effort than desired, which is where security groups become useful. To choose specific members only:

  1. Select the members icon from the Manage Accounts list
  2. Drag the desired members from the Unassigned Members column on the left, to the Assigned Members column on the right
  3. To remove members, do the opposite
  4. Select the Save button

Specific Security Groups Only

With this option, permission to access an account is granted to specific security groups rather than just individuals. With access restrictions relying on association with a security group or groups, the administration of accounts with much larger user counts becomes much simpler. To edit assigned groups:

  1. Select the groups icon from the Manage Accounts list
  2. Drag the desired groups from the Unassigned Groups column on the left, to the Assigned Groups column on the right
  3. To remove groups, do the opposite
  4. Select the Save button

Remote agents

PlaidLink agents will often use Document accounts to store files or move files among systems. To allow remote agents access to Document accounts, agents MUST have permission granted. This is a security feature to limit unwanted access to potentially sensitive information. To add agents:

  1. Select the agent icon from the Manage Accounts list
  2. Drag desired agents from the Unassigned Agents column on the left, to the Assigned Agents column on the right
  3. To remove agents, do the opposite
  4. Select the Save button

7.2.2 - Document Temporary Storage

Use Document's temporary storage option to share files or move them without worrying about cleanup later

Temporary storage may sound counter-intuitive, but real-world use has shown it to be valuable. Typically, permanent storage is used to move large files between members or among other systems, and file cleanup in these storage locations often happens haphazardly, at best. This causes storage to fill with files that shouldn’t be there, eventually requiring manual cleanup.

Temporary storage is perfect for sharing or transferring these types of large files because the files are automatically deleted after 24 hours.

To view temporary storage options

  1. Go To the Document > Temp Share in PlaidCloud

Shared Temporary Storage

Shared temporary storage is viewable by all members of the workspace but is not viewable across workspaces. To access the shared temporary storage area, select the Temp Share menu and click Workspace Temp Share to display a table of files currently in the workspace’s Temp Share area.

To add new files to a shared temporary storage location

  1. Select the Temp Share menu along the top of the main Document page
  2. Click Workspace Temp Share
  3. Click Browse to browse locally stored items
  4. Select the desired file and click Open
  5. Click Upload to upload the file to the temporary storage location

To download existing files from temporary storage

  1. Click on left-most icon, which represents the file type

To manually delete a file

  1. Click the red delete icon to the left of the file name.

Additional details on file management can be found below under “File Explorer”.

Personal Temporary Storage

Personal temporary storage is only viewable by the member to which the temp share belongs. This storage option is beneficial because it’s accessible across workspaces. This functionality makes it easy to move or use files across workspaces if the member is working in multiple workspaces simultaneously.

All members of the workspace can upload files to a members personal share as a dropbox.

To upload a file to another member’s personal share:

  1. Select the Temp Share menu along the top of the main Document page
  2. Select Drop File to Member Temp. A list of members will be displayed.
  3. Click the left-most icon associated with the member of your choosing
  4. Click Browse to browse locally stored items
  5. Select desired file and then click Open
  6. Click Upload to upload the file to the member’s personal storage

Additional details on file uploading can be found below under “File Explorer”.

7.2.3 - Managing Document Account Backups

Control how, where, and when Document account backups occur

Document enables the backup of any account on a nightly basis. This feature permits backup across different cloud storage providers and on local systems. Essentially, any account is a valid target for the backup of another account.

The backup process is not limited to a single backup destination. It is possible to have multiple redundant backup locations specified if this is a desired approach. For example, the backup of an internal server to another server may be one location with a second backup sent to Amazon S3 for off-site storage.

By using the prefix feature, it’s possible to have a single backup account contain the backups from multiple other accounts. Each account backup set begins its top level folder(s) with a different prefix, making it easy to distinguish the originating location and the restoration process. For example, if you have three different Document accounts but want to set their backup destination to the same location, using a prefix would allow all three accounts to properly backup without the fear of a name collision.

Reviewing Current Backup Settings

  1. Go to Document > Manage Accounts
  2. Select the backup icon for the account you wish to review

Creating a Backup Set

  1. Go to Document > Manage Accounts
  2. Select the backup icon for the account for which to create a backup
  3. Select the New Backup Set button
  4. Complete the required fields
  5. Select the Create button

The backup process is now scheduled to run nightly (US Time).

Updating a Backup Set

  1. Go to Document > Manage Accounts
  2. Select the backup icon for the account for which to edit a backup
  3. Select the edit icon of the desired backup set
  4. Adjust the desired information
  5. Select the Update button

Deleting a Backup Set

  1. Go to Document > Manage Accounts
  2. Select the backup icon for the account for which to edit a backup
  3. Select the delete icon of the desired backup set
  4. Select the Delete button

7.2.4 - Managing Document Account Owners

Add and remove Document account owners

The member who creates the account is assigned as the owner by default. However, Document accounts are designed to support multiple owners. This feature is helpful when a team is responsible for managing account access or when there is member turnover. Adding and removing owners is similar to adding and removing access permissions.

Add or Remove Owners

  1. Go to Document > Management Accounts in PlaidCloud
  2. Select the owners icon in the Manage Accounts list
  3. Drag new owners from the Unassigned Members column on the left to the Assigned Members column on the right
  4. To remove owners, do the opposite
  5. Select the Save button

Because only owners have the ability to view and edit an account, account administration is set up with two levels:

  • The member needs security access to view and manage accounts in general, and
  • The member must be an owner of the account to view, manage, and change settings of accounts

7.2.5 - Using Start Paths in Document Accounts

Control where users start navigation in document storage

The account management form allows the configuration of the storage connection information and a start path. A start path allows those who use the account to begin browsing the directory structure further down the directory tree. This particular option is useful when you have multiple teams that need segregated file storage, but you only want one underlying storage service account.

The Start Path option in Document accounts is useful for the following reasons:

  • When controlling access to sub-directories for specific teams and groups
  • Granting access to only one bucket

For example, setting a start path of teams/team_1/ for the Team 1 Document account and teams/team_2 for the Team 2 Document account provides different start points on a shared account. When a member opens the Team 1 Document account they will begin file navigation inside team/team_1. They will not be able to move up the tree and see anything above teams/team_1.

Team 2 would have a similar restriction of not being able to navigate into Team 1's area.

This provides the ability to restrict specific teams to lower levels of the tree while allowing other teams higher level access to the tree while not needing any additional cloud storage complexity like additional buckets or special permissions.

Adding and Updating the Start Path

  1. Go to Document > Manage Accounts
  2. Select the account you wish to edit and enter the edit mode
  3. Add a Start Path in the Start Path text field
  4. Select the save button

Start Path Format

The path always begins with the bucket name followed by the sub-directories.

<my-bucket>/folder1/folder2/

7.3 - Adding New Document Accounts

7.3.1 - Add AWS S3 Account

How to add an AWS Simple Storage Service (S3) account to Document

AWS S3 Setup

These steps need to be completed within the AWS console

  1. Sign into or create an Amazon Web Services (AWS) account
  2. Go to All services > Storage > S3 in the console
  3. Create a default or test bucket
  4. Go to All Services > Security Identity & Compliance > IAM > Users in the console
  5. Select the Create User button
  6. When prompted, enter a username and select Access Key - Programmatic access only. Select the Next: Permissions button.
  7. Select the option box called Attach existing policies directly
  8. In the filter search box type s3. When the list filters down to S3 related items select AmazonS3FullAccess by checking the box to the left. Select the Next: Tags button.
  9. Skip this step by selecting the Next: Review button
  10. Select the plus icon next to the WasabiFullAccess policy to attach the policy to the user. Select the Next button.
  11. Review the User settings and select Create user
  12. Capture the keys generated for the user by downloading the CSV or copy/pasting the keys somewhere for use later. You will not be able to retrieve this key again so keep track of it. If you need to regenerate a key simply go back to step 5 above.

You should now have everything you need to add your S3 account to PlaidCloud Document.

PlaidCloud Document Setup

  1. Sign into PlaidCloud
  2. Select the workspace that the new Document account will reside
  3. Go to Document > Manage Accounts
  4. Select the + New Account button
  5. Select Amazon S3 as the Service Type
  6. Fill in a name and description
  7. Leave the Start Path blank or add a start path based on an existing Wasabi account hierarchy. See the use of Start Paths for more information.
  8. Select an appropriate Security Model for your use case. Leave it Private if unsure.
  9. Paste the Access Key created in step 12 above into Public Key/User text field under Auth Credentials
  10. Paste the Secret Key created in step 12 above into the Private Key/Password text field under Auth Credentials
  11. Select the Save button and your new Document account is live

7.3.2 - Add Google Cloud Storage Account

How to add a Google Cloud Storage (GCS) account to Document

Google Cloud Setup

These steps need to be completed within Google Cloud Platform

  1. Sign into or create a Google Cloud Platform account
  2. Select or create a project where the Google Cloud Storage account will reside
  3. Go to Cloud Storage > Browser in the Google Cloud Platform console
  4. Create a default or test bucket
  5. Go To IAM & Admin > Service Accounts in the Google Cloud Platform console
  6. Select the + Create Service Account button
  7. Complete the service account information and create the account
  8. Find the service account just created in the list of service accounts and select Manage Keys from the context menu on the right
  9. Under the Add Key menu, select Create a Key
  10. When prompted, select JSON format for the key. This will generate the key and automatically download it to your desktop. You will not be able to retrieve this key again so keep track of it. If you need to regenerate a key simply go back to step 8 above.
  11. Go to IAM & Admin > IAM in the Google Cloud Platform console
  12. Find the service account you just created and click on the edit permissions icon
  13. Add Storage Admin and Storage Transfer Admin rights for the service account and save. Note less permissive rights can be assigned but this will impact the functionality available through Document.

You should now have everything you need to add your GCS account to PlaidCloud Document.

PlaidCloud Document Setup

  1. Sign into PlaidCloud
  2. Select the workspace that the new Document account will reside
  3. Go to Document > Manage Accounts
  4. Select the + New Account button
  5. Select Google Cloud Storage as the Service Type
  6. Fill in a name and description
  7. Leave the Start Path blank or add a start path based on an existing GCS account hierarchy. See the use of Start Paths for more information.
  8. Select an appropriate Security Model for your use case. Leave it Private if unsure.
  9. Open the Service Account JSON key file you downloaded in step 10 above and copy the contents
  10. Paste the contents into the Auth Credentials text area
  11. Select the Save button and your new Document account is live

7.3.3 - Add Wasabi Hot Storage Account

How to add a Wasabi Hot Storage (Wasabi) account to Document

Wasabi Hot Storage Setup

These steps need to be completed within the Wasabi Hot Storage console

  1. Sign into or create a Wasabi Hot Storage account
  2. Go to Buckets in the console
  3. Create a default or test bucket
  4. Go to Users in the console
  5. Select the Create User button
  6. When prompted, enter a username and select Programmatic (create API key) user
  7. Skip the group assignment. Select the Next button
  8. Select the plus icon next to the WasabiFullAccess policy to attach the policy to the user. Select the Next button.
  9. Review the User settings and select Create User
  10. Capture the keys generated for the user by downloading the CSV or copy/pasting the keys somewhere for use later. You will not be able to retrieve this key again so keep track of it. If you need to regenerate a key simply go back to step 5 above.

You should now have everything you need to add your Wasabi account to PlaidCloud Document.

PlaidCloud Document Setup

  1. Sign into PlaidCloud
  2. Select the workspace that the new Document account will reside
  3. Go to Document > Manage Accounts
  4. Select the + New Account button
  5. Select Wasabi Hot Storage as the Service Type
  6. Fill in a name and description
  7. Leave the Start Path blank or add a start path based on an existing Wasabi account hierarchy. See the use of Start Paths for more information.
  8. Select an appropriate Security Model for your use case. Leave it Private if unsure.
  9. Paste the Access Key created in step 10 above into Public Key/User text field under Auth Credentials
  10. Paste the Secret Key created in step 10 above into the Private Key/Password text field under Auth Credentials
  11. Select the Save button and your new Document account is live

8 - Analysis Tools & Connectors

8.1 - Data Connections

Use this table reference for more information on external system connections and databases

Description

PlaidCloud connects to external systems by using various data connections directly or through PlaidLink agents.

For more details on each data connection type, please navigate to the specific data connection documentation.

Relational Databases

Greenplum

ParameterValue
Connection TypeDatabase
Referencegreenplum

Microsoft SQL Server

ParameterValue
Connection TypeDatabase
Referencesqlserver

MySQL

ParameterValue
Connection TypeDatabase
Referencemysql

ODBC

ParameterValue
Connection TypeDatabase
Referenceodbc

Oracle

ParameterValue
Connection TypeDatabase
Referenceoracle

Postgres

ParameterValue
Connection TypeDatabase
Referencepostgres

Amazon Redshift

ParameterValue
Connection TypeDatabase
Referenceredshift

SAP HANA

ParameterValue
Connection TypeDatabase
Referencehana

Exasol

ParameterValue
Connection TypeDatabase
Referenceexasol

IBM DB2

ParameterValue
Connection TypeDatabase
Referencedb2

Informix

ParameterValue
Connection TypeDatabase
Referenceinformix

Hadoop Based Databases

Hive

ParameterValue
Connection TypeDatabase
Referencehive

Presto

ParameterValue
Connection TypeDatabase
Referencepresto

Spark

ParameterValue
Connection TypeDatabase
Referencespark

Team Collaboration Tools

Microsoft Teams

ParameterValue
Connection TypeNotification
Referenceteams

Slack

ParameterValue
Connection TypeNotification
Referenceslack

Cloud Services

OAuth Connection

ParameterValue
Connection TypeoAuth
Referenceoauth

Quandl

ParameterValue
Connection TypeQuandl
Referencequandl

Google Big Query

ParameterValue
Connection TypeGoogle Big Query
Referencegbq

Google Spreadsheet

ParameterValue
Connection TypeGoogle Spreadsheet
Referencegspread

Oracle EBS utilizes the standard Oracle database connection specified above. This connection provides the connectivity to query, load, and execute PL/SQL programs in Oracle.

If the EBS instance has the REST API interface available, this can be accessed using the same approach as Oracle Cloud described below.

Oracle Cloud utilizes standard RESTful requests to perform queries, data loading, and other operations. A REST connection using OAuth2 tokens is used for these interactions. This uses the standard oAuth connection specified above.

Workday utilizes standard RESTful requests to perform all operations. A REST connection using OAuth2 tokens is used for these interactions. This uses the standard oAuth connection specified above.

ParameterValue
Connection TypeJD Edwards Legacy
Referencejde_legacy

JD Edwards utilizes the standard Oracle database connection specified above. This connection provides the connectivity to query, load, and execute PL/SQL programs in Oracle.

ParameterValue
Connection TypeInfor
Referenceinfor

SAP Analytics Cloud

ParameterValue
Connection TypeSAP Analytics Cloud
Referencesap_sac

SAP ECC

ParameterValue
Connection TypeSAP ECC
Referencesap_ecc

SAP Profitability and Cost Management (PCM)

ParameterValue
Connection TypeSAP PCM
Referencesap_pcm

SAP Profitability and Performance Management (PaPM)

ParameterValue
Connection TypeSAP PaPM
Referencesap_papm

8.2 - PlaidLink

PlaidLink provides indirect access to systems and processes protected by firewalls or behind other restrictions that make direct connections difficult. By using an Agent installed within the isolated area, PlaidCloud can request the agent perform actions like running queries, downloading or uploading files, checking sensor conditions, interacting with SAP, and much more.

Since the agent initiates contact with PlaidCloud and communicates over standard HTTPS network protocols, it can normally operate with minimal setup. In addition, the agent can run as an unprivileged user to control access rights within a restricted environment.

8.2.1 - Configure

Create and maintain PlaidLink (Agent) documentation and account access for optimal database and file system enhancement

The PlaidLink Agent works in conjunction with the PlaidCloud service. The PlaidLink Agent provides the connection necessary to operate with systems not accessible directly such as databases and file systems. The agent performs a number of essential actions including:

  • Reading and writing to databases
  • Reading and writing files to network drives and servers
  • Checking for sensor conditions
  • Interacting with SAP ECC and SAP S/4HANA through Remote Function Calls (RFCs)
  • Interacting with SAP Profitability and Cost Management (PCM)
  • Sending messages and notifications to remote systems

Create an Agent on PlaidCloud

PlaidLink Agent management takes place within the Analyze tab of PlaidCloud. The first step is to create a new PlaidLink Agent instance on PlaidCloud.

  1. Select the Analyze tab
  2. Select the tools menu from the top
  3. Click PlaidLink Agents
  4. Create a new Agent with an appropriate name for the environment or server that it will be installed on for remote operations

To view the Agent public and private keys

  1. Click on the edit icon to view the form
  2. At the bottom of the form you will find the public and private keys that were randomly generated during the Agent creation process

To randomly generate new keys

  1. Click on the Regenerate icon for the Agent record
  2. Once the keys are regenerated, don’t forget to update the agent configuration file with the new keys on the remote server.

Document Account Access

If the agent will need to have access to a Document account for uploading or downloading files, it must be granted permission to access the Document account.

To grant account access

  1. In the Document tab select Manage Accounts
  2. Once the table of accounts appears, click on the agent icon for the account which the new Agent should have upload/download rights
  3. Drag the new agent into the Assigned Agents column
  4. Save the access control form.

Data Connection Access

If the agent will need to have access to a data connection such as a database, it must be granted permission to access the external data connection information.

To grant connection access

  1. In the Analyze tab select the Tools menu
  2. Click External Data Connections
  3. Once the table of data connections appears, click on the agent icon for the connection, which the new Agent should have usage rights
  4. Drag the new agent into the Assigned Agents column and save the access control form.

Follow these Installation Instructions to install PlaidLink on the remote system.

8.2.2 - Installation

Create a configuration file, Install and run the PlaidLink (Agent)

Download the agent

Check the releases on PlaidCloud.com for PlaidLink

Extract the agent

Extract the downloaded zip file to an install location of your choice. Generally, this location will be:

C:\Users\<Username here>\src\plaidlink

Create a configuration file

Copy the config-dist.yaml file in the agent's directory to %ProgramData\plaidcloud\, and rename this copy config.yaml

(Edit this configuration with the values retrieved from PlaidCloud)

Install the agent's service

Run the install_windows_service.bat file in the agent's install directory OR

From an administrator command prompt, navigate to the agent's install directory and run:

.\PlaidLink.exe install

Running the agent

Type Services into Windows' search bar and open the service manager. In the list of services, find PlaidCloud Agent.

Right-click the service and select "Start" to start the agent.

Freezing updates

If at any point you want to disable the agent's auto-update feature, open the agent's 'yaml' configuration file, and at the root level of the file, add a line that reads freeze_updates: true, and restart the agent's service.

8.2.3 - PlaidLink Agents

Create and manage remote access using lightweight agents

Description

Sometimes it’s necessary and desireable to access data or run processes from a remote system that does not allow external access. This is common in enterprise environments behind firewalls. PlaidCloud allows this ability by using PlaidLink, which enables remote systems access behind a firewall or where direct access from PlaidCloud is not desired.

PlaidLink uses an agent-based system. This means that an agent, the remote user, is installed on a system inside the firewall or other restricted area. The agent can then connect to PlaidCloud by using an outbound initiation process over a secure HTTPS websocket connection. It is as secure as any other encrypted web connection and usually does not require you to open non-standard ports. Before gaining access, the agent must identify itself by sending its agent identifier. From this, if the agent has a successful authentication process, the agent is granted access to the approved operations.

PlaidLink can be installed on Windows, Unix, and Linux systems and can run under low privilege users. On Windows systems, PlaidLink can operate as a Windows Service with full control from the Service panel. On linux or unix systems, it can run as a deamon process.

PlaidLink can also run as a stand-alone Docker container or as a Kubernetes pod.

Managing Agents

To manage agents:

  1. Open Analyze
  2. Select “Tools”
  3. Click “PlaidLink Agents”

This brings you to the PlaidLink Agents Table where you can view, modify, and obtain credentials for the list of available agents.

Creating an Agent

To create an agent:

  1. Open Analyze
  2. Select “Tools”
  3. Click “PlaidLink Agents”
  4. Click “Add PlaidLink Agent”
  5. Complete the required fields
  6. Click “Create”
  7. Assign the agent to the necessary security groups to access resources needed to perform its job
  8. Assign the agent to the necessary Document accounts to access documents needed to perform its job

Obtaining Agent Credentials

To configure PlaidLink agents on the remote system, you must first obtain the agent’s identifying information in order to maintain security. This information includes both a public and a private key.

To obtain these keys:

  1. Open Analyze
  2. Select “Tools”
  3. Click “PlaidLink Agents”
  4. Click the edit icon

This will open a form where you can view the public and private key values.

Regenerating Agent Credentials

It is a good idea to periodically regenerate the public and private keys and update the configuration of remote systems in order to maintain security.

To regenerate the credentials:

  1. Open Analyze
  2. Select “Tools”
  3. Click “PlaidLink Agents”
  4. Click the regenerate icon

Once the credentials have been regenerated, they can be obtained in the same way a new agent’s credentials are obtained (described above).

Enabling and Disabling an Agent

To disable an agent:

  1. Open Analyze
  2. Select “Tools”
  3. Click “PlaidLink Agents”
  4. Uncheck the “Active” checkbox

Running Multiple Agents

PlaidLink is designed to allow operation of multiple agents using a single service installation. Such a streamlined installation system permits one install to handle agents from multiple workspaces and / or agents with different levels of permissions for task execution.

To enable multiple agents, you simply add the agent credentials to the PlaidLink configuration file.

Similar to running multiple agents within one PlaidLink service, it is also possible to run multiple PlaidLink services.

This is sometimes necessary depending on use of system based security or network access restrictions that prevent communication across network boundaries.

Compute, Memory, and Disk Requirements

The PlaidLink service is extremely lightweight and only needs minimal compute and memory to operate. When processing significant data volumes it may be necessary to increase compute resources and especially memory.

Normally, the agent will happily run with 5% of CPU and 200MB of memory. For intense data operations, it is recommended to allocate an entire CPU and at least 4GB of RAM. For dynamic resource allocation systems like Kubernetes, it is fine if the agent has access to burstable resources rather than reserved resources.

Disk space for the agent is minimal too. Agent operations utilize disk space as a data buffer when transferring large amounts of data. Typically, 8GB of space is fine for normal operations. For intense data operations it is recommended that you scale disk up according to the expected data volumes. There is no set amount because it depends on several factors including CPU speed, network speed, amount of data, etc... However, a good place to start is 20GB and adjust from there.

Networking Requirements

The PlaidLink Agent is designed to operate with minimal configuration required. It does not require any special VPN or network configuration other than allowing standard HTTPS network traffic. Agents communicate over the same protocol as normal web browser based traffic.

The agent service always initiates communication with PlaidCloud so there is no need to configure ingress access in firewalls.

8.3 - Scheduled Events

8.3.1 - Event Scheduler

Create and organize a scheduled recurring event

Description

Scheduling specific workflows can be a useful organization tool, so PlaidCloud provides the ability to do just that. Using event scheduler, you can schedule a workflow to run by month, day, hour, minute, or even on a financial workday schedule. If using the financial workday schedule approach, PlaidCloud also allows configuration of holiday schedules using various holiday calendars.

The Events Table will indicate whether the event is scheduled by month, day, hour and minute, or workday under the event description column.

To view events:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”

This will open the Events Table showing all the current events configured for the workspace.

Creating an Event

To create an event:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”
  4. Click “Add Scheduled Event”
  5. Complete the required fields
  6. Click “create”

Limit Running: this section allows you to schedule an event to run for a specific time period and a specific number of times.

Otherwise, you can set the workflow to run using the classic schedule approach.

To use the classic schedule approach:

  1. Click the “Event Schedule” tab of the Event table
  2. Under the “Schedule type” select “Use Classic Schedule”
  3. Select the specific months, hours, minutes, and days you want the workflow to run

To set the workflow to run using the workday schedule approach:

  1. Click the “Event Schedule” tab of the Event table
  2. Under the “Schedule type” select “Use Workday Schedule”
  3. Choose the workday you would like the workflow to run on

Editing an Event

To edit an event:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”
  4. Click the edit icon
  5. Adjust desired fields
  6. Click “Update”

Deleting an Event

To delete an event:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”
  4. Click the delete icon
  5. Click delete again

Pausing an Event

To temporarily pause an event:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”
  4. Click the edit icon
  5. Uncheck the “Active” checkbox
  6. Click “Update”

Saving the event after unchecking the active box means the event will no longer run on the specified schedule until it’s reactivated.

Running Events on Demand

To run an event immediately:

  1. Open Analyze
  2. Select “Tools”
  3. Click “Event Scheduler”
  4. Select the desired event or events
  5. Click “Run Selected Events”

8.3.2 - Message Queue Exchanges, Bindings and Permissions

Create and manage your own exchanges and viewership controls for message queues

vHost Exchanges

To view exchanges:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the exchanges icon of the desired vHost

This will open the Exchanges Table which displays the list of all exchanges related to the vHost.

Creating Exchanges

To create an exchange:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the exchanges icon of the desired vHost
  6. Click “Add Exchange”
  7. Complete the required fields
  8. Click “Submit”

Deleting Exchanges

To delete an exchange:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the exchanges icon of the desired vHost
  6. Click the delete icon of the desired exchange
  7. Click “Delete” again

vHost Bindings

To view bindings:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the bindings icon of the desired vHost

This will open the Bindings Table, which displays the list of bindings and other important information related to the vHost.

Creating Bindings

To create a binding:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the bindings icon of the desired vHost
  6. Click “Add Binding”
  7. Complete the desired fields
  8. Click “Submit”

Deleting Bindings

To delete bindings:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the bindings icon of the desired vHost
  6. Click the delete icon of the desired binding
  7. Click “Delete” again

vHost Permissions

To view permissions:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the permissions icon of the desired vHost

This will open the Permissions Table, which displays the list of users and their permissions related to the vHost.

Creating Permissions

To create a permission:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the permissions icon of the desired vHost
  6. Click “Add Permissions”
  7. Complete the required fields
  8. Click “Submit”

Deleting Permissions

To delete permissions:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the permissions icon of the desired vHost
  6. Click the delete icon of the desired permission
  7. Click “Delete” again

Users

In order to access messages in the message queue, you must first set up users. PlaidCloud allows you to create an unlimited number of users in order to suit your needs.

To view users who may access vHost and messaging services:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “Users”

This will open the User Table, which displays users currently in the workspace and their admin level permissions.

Creating Users

To create a new user:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “Users”
  5. Click “Add User”
  6. Complete the required fields
  7. Click “Create”

Controlling User Access

The Admin Level setting provides the user with different privileges. PlaidCloud provides three levels of access:

  • No Admin Access
  • Management Admin Access
  • Monitoring Admin Access

No Admin Access provides no administrative access but allows the user to utilize all other permission based access.

Management Admin Access provides full permission based access, as well as the following:

  • List virtual hosts to which they can log in
  • View all queues, exchanges, and bindings in “their” virtual hosts
  • View and close their own channels and connections
  • View “global” statistics covering all their virtual hosts, including activity by other users

Monitoring Admin Access provides everything “Management Admin Access” provides, as well as the following:

  • List all virtual hosts, including ones they could not log into via AMQP
  • View other users’ connections and channels
  • View node-level data such as memory use and clustering
  • View truly global statistics for all virtual hosts

Deleting Users

To delete a user:

  1. Open Analyze

  2. Select the “Tools” menu

  3. Hover over “Queueing” in the dropdown menu

  4. Click “Users”

  5. Click the delete icon of the desired user

    • Click “Delete” again

8.3.3 - Message Queue Virtual Host

Create and Manage Virtual Hosts (vHost) for message queues

Description

In order to ease the transition between new applications, it may sometimes be necessary for business teams to coordinate services and processes using services that cannot communicate in real-time. PlaidCloud provides a high performance messaging queue to handle asynchronous communication, with a wide range of delivery options, including:

  • Direct exchange
  • Fanout exchange
  • Topic exchange
  • Headers exchange

This function is provided by RabbitMQ clusters and is not intended to replace internal message queueing systems, although PlaidCloud’s message queue is capable of doing so.

Click here (https://www.rabbitmq.com/tutorials/tutorial-two-python.html) for additional information from RabbitMQ.

Virtual Hosts (vHosts)

Virtual Hosts (vHosts) can be thought of as small, isolated message servers used for specific purposes such as queues, exchanges, and bindings. PlaidCloud uses vHosts to form its foundation for the message broker infrastructure.

To view current vHosts:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”

This will open the vHosts Table in the workspace, which includes vHost information and access to Queues, Permissions, Bindings, and Exchanges related to the vHost.

Creating vHosts

To create a vHost:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click “Add vHost”
  6. Name the vHost for future referencing
  7. Click “Create”

Once the vHost is created, it will be tagged with its reference to a workspace identifier. For example, if you create a vHost with the name of ‘super’ and its identifier is 890, the actual name of the user will be ‘cloud_890_super’. This allows for unique vHosts without constant name collisions.

Deleting vHosts

To delete a vHost:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the delete icon
  6. Click “Delete” again

8.3.4 - Message Queues

Create and Manage 'vHost' Message Queues

vHost Queues

To reach and view vHost Queues:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the queues icon of the desired vHost

This will open the Queues Table which displays the list of queues and other important information related to the vHost, including its message state, consumers, and rate of message processing.

Creating Queues

To create a queue:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the queues icon of the desired vHost
  6. Click “Add Queue”

This will open a form where you can provide a name, durability level, and determine if the queue should be deleted when no consumers are present.

Durability is useful if a message needs to be retained in the event of a system failure or restart. If you choose not to create a durable queue, then the message will be lost after a restart. Although durability is useful, it is not always desired.

Creating a durable queue results in a slightly slower process because each message must be stored in permanent storage. We suggest then that you only create a durable queue when the message is critical and use non-durable queues when a message is executed right away or is not critical.

Deleting Queues

To delete a queue:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the queues icon of the desired vHost
  6. Click the delete icon of the desired queue

Purging a Queue

Purging a queue will delete all messages in that queue.

To purge a queue:

  1. Open Analyze
  2. Select the “Tools” menu
  3. Hover over “Queueing” in the dropdown menu
  4. Click “vHosts”
  5. Click the queues icon of the desired vHost
  6. Click the purge icon of the desired queue
  7. Click “Purge Queue”

9 - Greenplum Data Warehouses

10 - Jupyter and Command Line Interactions

10.1 - Command Line

Interact with PlaidCloud directory from command line

PlaidCloud uses standard JSON-RPC requests and can be used with any application that can perform those requests.

To make things easier, a Python package is available to simplify the connection and API running process.

Required Installation

From a terminal run the following command:

pip install plaidcloud-rpc

Using the SimpleRPC Object to Make a Request

To make a request using the plaidcloud-rpc package use the SimpleRPC object.

from plaidcloud.rpc.connection.jsonrpc import SimpleRPC  
  
auth_token = "Your PlaidCloud Auth Token" # See Obtaining Token below  
endpoint_uri = "plaidcloud.com" # or plaidcloud.net  
rpc = SimpleRPC(auth_token, endpoint_uri)

Once you have the SimpleRPC object instantiated you can then issue RPC request to PlaidCloud. This example requests the meta data for a table.

table = rpc.analyze.table.table(  
            project_id=project_id,  
            table_id=table_id  
        )

What APIs are Available?

There are many APIs available for use that control nearly every aspect of PlaidCloud. All of the APIs, the inputs, and expected outputs are documented in the APIs documentation.

Obtaining an OAuth Token

See OAuth Tokens for more information on obtaining an OAuth token and how to configure the system for automated auth.

10.2 - Jupyter Notebooks

Interact with PlaidCloud directory from Jupyter Notebooks

Jupyter Notebooks and Jupyter Lab provide exceptional interactive capabilities to analyze, explore, explain, and report data. PlaidCloud enables use of information directly in notebooks.

Install Jupyter Notebook

This assumes you have a working Jupyter Notebook installation.

Installing a Stand-Alone Jupyter Notebook

For more information on installing a Jupyter Notebook locally you can reference Jupyter’s installation documentation.

Add to VS Code

VS Code also provides an extension that allows you to run notebooks directly in VS Code. Install the extension from the Visual Studio Marketplace

Install PlaidCloud Utilities

While PlaidCloud can be accessed using stand OAuth and JSON-RPC requests, it is recommended that you use our pre-built libraries for simplified access. In addition, the PlaidCloud utilities library includes handy data helpers for use with Pandas dataframes.

To install the PlaidCloud Utilities perform the following pip installs:

pip install plaidcloud-rpc@git+https://github.com/PlaidCloud/plaid-rpc.git@v1.1.4#egg=plaidcloud-rpc
pip install plaidcloud-utilities@git+https://github.com/PlaidCloud/plaid-utilities.git@v1.1.9#egg=plaidcloud-utilities

Obtaining an OAuth Token

See OAuth Tokens for more information on obtaining an OAuth token and how to configure the system for automated auth.

Open Jupyter Notebook User Interface

Launch your notebook server to get started.

Once you are signed into your Jupyter notebook server, create a new notebook from the UI.

This will open a blank notebook.

Create a connection to communicate with PlaidCloud through the API endpoints

from plaidcloud.utilities.connect import PlaidConnection

conn = PlaidConnection()

Establish a local table object and then query it with the results automatically placed in a Pandas dataframe.

tbl_sf_cust_master = conn.get_table('Salesforce_Customer_Master') # This gets a table object
df_sf_cust_master = conn.get_data(tbl_sf_cust_master) # This retrieves all the data into a dataframe

With that same table object you can also write more advanced queries using standard SQLAlchemy syntax.

df_sf_cust_master_w_sales = conn.get_data(
    tbl_sf_cust_master.select().with_only_columns(
        [tbl_sf_cust_master.c.Id, tbl_sf_cust_master.c.CurrencyIsoCode, tbl_sf_cust_master.c.SyDSalesRegion]
    ).where(
        tbl_sf_cust_master.c.TotalSalesPast3Years > 0
    )
)

10.3 - OAuth Tokens

Obtaining OAuth tokens to interact with PlaidCloud APIs

PlaidCloud uses standard JSON-RPC requests and can be used with any application that can perform those requests. Requests are secured using OAuth tokens.

Obtaining an OAuth Token

OAuth tokens are generated from the PlaidCloud app. To view the list of current OAuth tokens assigned to you and generate new ones, navigate to Analyze > Tools > Registered Systems.

Once there you can view any existing tokens or choose to create a new one.

Download OAuth PlaidCloud Config File

Select “Register a New System”.

Fill out the form and note the name you entered so you can find it in the list.

Once created, open the registered system record by clicking on the gear icon. This will display the configuration file text.

NOTE: Be sure to select the project you want to use this connection for from the drop down at the top. It will add the Project Unique Identifier to the configuration.

Copy this text into a plaid.conf file located on your system. Place this in the .plaid directory.

Create a Config File Locally

Create a directory one level up from your notebook directory or from where you plan to use command line interaction. Name the directory .plaid.

Inside the .plaid directory, create a file called plaid.conf and paste the contents you copied above into the file. Save the file and this will no allow you to connect using the PlaidCloud utilities and rpc methods.

Advanced Uses

While it is convenient to locate the .plaid folder near its usage point, it can actually be placed anywhere in the upstream directory tree. The initialization process will traverse up the directory tree until it finds the .plaid directory.

Locating the .plaid directory higher up may be useful if you have multiple operations that need access but cannot coexist in the same lower level directory structures.

Optional Paths Specification

If you are using a local Jupyter Notebook installation or operating from command line, it is possible to export data, excel files, and other data as well as reading in local data to dataframes using the helper tools. To do this, a paths.yaml file is necessary.

In addition to the plaid.conf file, create a paths.yaml file. The paths.yaml should be a sibling to the plaid.conf file inside the .plaid directory. It should contain the following path information:

paths:
 PROJECT_ROOT: '{WORKING_USER}/Documents'
 LOCAL_STORAGE: '{PROJECT_ROOT}/local_storage'
 DEBUG: '{PROJECT_ROOT}/local_storage'
 REPORTS: '{PROJECT_ROOT}/reports'
 
 create: []
 local: {}

11 - Standard Expressions

11.1 - General Usage Conditionals

Simple If-then-else Operation, Complex Condition with Multiple Options, Coalesce

Conditional statements use case syntax. This is the equivalent of simple if-then-else logic and more complex conditions with multiple options.

Examples

Simple If-then-else Operation

case(
        (table.first_name.isnot(None), func.concat(table.first_name, table.last_name)), 
        else_ = table.last_name
    )

Complex Condition with Multiple Options

case( 
        (order_table.qty > 100, item_table.specialprice), 
        (order_table.qty > 10, item_table.bulkprice) , 
        else_=item_table.regularprice
    )
case( 
        (users_table.name == "wendy", "W"), 
        (users_table.name == "jack", "J"), 
        else_='E'
    )

The above may also be written in shorthand as:

case(
    {"wendy": "W", "jack": "J"}, 
    value=users_table.name, 
    else_='E' 
)

Coalesce

When trying to find the first non-null value in a set of columns, the coalesce method is very helpful. This example finds the price set for the sale by looking in three possible fields:

func.coalesce(table_beta.adjusted_price, table_alpha.override_price, table_alpha.price) * table_beta.quantity_sold

Coalesce also works for text values. This example will use the nickname if it is not null, or it will fall back to the first_name.

func.coalesce(table.nickname, table.first_name)

11.2 - General Usage Data Type Expressions

The following data types (dytpes) are available through Analyze to support your requirements

Analyze offers a wide variety of standard data types (dtypes) to support your requirements. As datasets become larger, determining smaller size dtypes for value storage can shrink the size of the table and improve performance. The following dtypes are available:

  • Boolean

  • Text

  • Numbers

    • Small Float (6 Digits)
    • Float (15 Digits)
    • Small Integer (16 bit) (-32768 to 32767)
    • Integer (32 bit) (-2147483648 to 2147483647)
    • Big Integer (64 bit) (-9223372036854775808 to 9223372036854775807)
    • Numeric
  • Dates and Times

    • Date
    • Timestamp
    • Time Interval

It is also possible to convert from one dtype to another using the func.cast() process.

func.cast() Type Conversions

Analyze ExpressionDescriptionResult
func.cast(123, Text)Integer to Text‘123’
func.cast(‘123’, Integer)Text to Integer123
func.cast(‘78.69’, Float)Text to Float78.69
func.cast(‘78.69’, SmallFloat)Text to Small Float78.69
func.cast(‘78.69’, Integer)Text to Integer (Truncate decimals)78
func.cast(‘78.69’, SmallInteger)Text to Small Integer (Truncate decimals)78
func.cast(‘78.69’, BigInteger)Text to Big Integer (Truncate decimals)78
func.cast(1, Boolean)Integer to BooleanTrue

func.to() Data Type Conversions

Analyze ExpressionReturn TypeDescriptionExample
func.to_char(timesta mp, text)textconvert time stamp to stringto_char(c urrent_ti mestamp, ‘HH12:MI:S S’)
func.to_char(interva l, text)textconvert interval to stringto_char(i nterval ‘15h 2m 12s’, ‘HH24:MI:S S’)
func.to_char(integer , text)textconvert integer to stringto_char(1 25, ‘999’)
func.to_char(big float, text)textconvert real/double precision to stringto_char(1 25.8::real , ‘999D9’)
func.to_char(numeric , text)textconvert numeric to stringto_char(- 125.8, ‘999D99S’)
func.to_date(text, text)dateconvert string to dateto_date(‘ 05 Dec 2000’, ‘DD Mon YYYY’)
func.to_number(text, text)numericconvert string to numericto_number (‘12,454.8 -‘, ‘99G999D9S ‘)
func.to_timestamp(te xt, text)timestamp with time zoneconvert string to time stampto_timest amp(‘05 Dec 2000’, ‘DD Mon YYYY’)
func.to_timestamp(bi g float)timestamp with time zoneconvert UNIX epoch to time stampto_timest amp(200120 400)

11.3 - General Usage Dates and Time Expressions

Common Expression Values for date/time functions
Analyze ExpressionReturn TypeDescriptionExampleResult
func.age(timestamp, timestamp)intervalSubtract arguments, producing a “symbolic” result that uses years and monthsage (timest amp ‘2001-04-1 0’, timestamp ‘1957-06-1 3’)43 years 9 months 27 days
func.age(timestamp)intervalSubtract from current_dateage(timest amp ‘1957-06-1 3’)43 years 8 months 3 days
func.clock_timestamp ()timestamp with time zoneCurrent date and time (changes during statement execution)
func.current_datedateCurrent date
func.current_timetime with time zoneCurrent time of day
func.current_timestamptimestamp with time zoneCurrent date and time (start of current transaction)
func.date_part(text, timestamp)double precisionGet subfield (equivalent to extract)date_part (‘hour’, timestamp ‘2001-02-1 6 20:38:40’)20
func.date_part(text, interval)double precisionGet subfield (equivalent to extract)date_part (‘month’, interval ‘2 years 3 months’)3
func.date_trunc(text , timestamp)timestampTruncate to specified precisiondate_trun c(‘hour’, timestamp ‘2001-02-1 6 20:38:40’)36938.833 3333333
func.extract(field from timestamp)double precisionGet subfieldextract(ho ur from timestamp ‘2001-02-1 6 20:38:40’)20
func.extract(field from interval)double precisionGet subfieldextract(mo nth from interval ‘2 years 3 months’)3
func.isfinite(timesta mp)booleanTest for finite time stamp (not equal to infinity)isfinite(t imestamp ‘2001-02-1 6 21:28:30’)TRUE
func.isfinite(interva l)booleanTest for finite intervalisfinite(i nterval ‘4 hours’)TRUE
func.justify_days(in terval)intervalAdjust interval so 30-day time periods are represented as monthsjustify_d ays(interv al ‘30 days’)1 month
func.justify_hours(i nterval)intervalAdjust interval so 24-hour time periods are represented as daysjustify_h ours(inter val ‘24 hours’)1 day
func.justify_interva l(interval)intervalAdjust interval using justify_days and justify_hours , with additional sign adjustmentsjustify_i nterval(in terval ‘1 mon -1 hour’)29 days 23:00:00
func.now()timestamp with time zoneCurrent date and time (start of current transaction)
func.statement_times tamp()timestamp with time zoneCurrent date and time (start of current statement)
func.timeofday()textCurrent date and time (like clock_timesta mp, but as a text string)
func.transaction_tim estamp()timestamp with time zoneCurrent date and time (start of current transaction)

Date and Time Formatting Directives

PatternDescription
HHhour of day (01-12)
HH12hour of day (01-12)
HH24hour of day (00-23)
MIminute (00-59)
SSsecond (00-59)
MSmillisecond (000-999)
USmicrosecond (000000-999999 )
SSSSseconds past midnight (0-86399)
AM or A.M. or PM or P.M.meridian indicator (uppercase)
am or a.m. or pm or p.m.meridian indicator (lowercase)
Y,YYYyear (4 and more digits) with comma
YYYYyear (4 and more digits)
YYYlast 3 digits of year
YYlast 2 digits of year
Ylast digit of year
IYYYISO year (4 and more digits)
IYYlast 3 digits of ISO year
IYlast 2 digits of ISO year
Ilast digits of ISO year
BC or B.C. or AD or A.D.era indicator (uppercase)
bc or b.c. or ad or a.d.era indicator (lowercase)
MONTHfull uppercase month name (blank-padded to 9 chars)
Monthfull mixed-case month name (blank-padded to 9 chars)
monthfull lowercase month name (blank-padded to 9 chars)
MONabbreviated uppercase month name (3 chars)
Monabbreviated mixed-case month name (3 chars)
monabbreviated lowercase month name (3 chars)
MMmonth number (01-12)
DAYfull uppercase day name (blank-padded to 9 chars)
Dayfull mixed-case day name (blank-padded to 9 chars)
dayfull lowercase day name (blank-padded to 9 chars)
DYabbreviated uppercase day name (3 chars)
Dyabbreviated mixed-case day name (3 chars)
dyabbreviated lowercase day name (3 chars)
DDDday of year (001-366)
DDday of month (01-31)
Dday of week (1-7; Sunday is 1)
Wweek in month (1-5) (The first week starts on the first day of the month.)
WWweek number in year (1-53) (The first week starts on the first day of the year.)
IWISO week number of year (The first Thursday of the new year is in week 1.)
CCcentury (2 digits)
JJulian Day (days since January 1, 4712 BC)
Qquarter
RMmonth in Roman numerals (I-XII; I=January) (uppercase)
rmmonth in Roman numerals (i-xii; i=January) (lowercase)
TZtime-zone name (uppercase)
tztime-zone name (lowercase)

11.4 - General Usage Filters

Apply filters using column references

You may apply filters using column references.

OperatorDescriptionExample
and_()Creates an AND SQL conditionand_(table.a > 23, table.b == u’blue’)
or_()Creates an OR SQL conditionor_(table.a > 23, table.b == u’blue’)
not_()Inverts the conditionnot_(and_(table.a > 23, table.b == u’blue’))
in_()Test if values are with a tuple of valuestable.column.in_((1, 2, 3))
notinInverts the IN conditiontable.column.notin((1, 2, 3))
any_()Applies the SQL ANY() condition to a columntable.column.any((‘red’, ‘blue’, ‘yellow’))
betweenApplies the SQL BETWEEN conditiontable.column.between(23, 46)
startswithApplies the SQL LIKE ‘%’table.column.startswith(‘abc’)
containsApplies the SQL LIKE ‘%%’table.column.contains(‘mno’)
endswithApplies the SQL LIKE ‘%%’table.column.endswith(‘xyz’)
is_Applies the SQL is the IS for things like IS NULLtable.column.is_(None)
isnotApplies the SQL is the IS for things like IS NOT NULLtable.column.isnot(None)
likeApplies the SQL LIKE methodtable.column.like(‘%foobar%’’)
notlikeApplies the SQL NOT LIKE methodtable.column.notlike(‘%foobar%’)
ilikeApplies the SQL ILIKE methodtable.column.ilike(‘%foobar%’)
notilikeApplies the SQL NOT ILIKE methodtable.column.notilike(‘%foobar%’)
NULL, Null, nullAlias for Python None
TRUE, trueAlias for Python True
FALSE, falseAlias for Python False
>Greater Thantable.column > 23
<Less Thantable.column < 23
>=Greater than or equal totable.column >= 23
<=Less than or equal totable.column <= 23
==Equaltable.column == 23 table.column == ‘blue’
!=Not Equaltable.column != 23 table.column != ‘blue’

11.5 - General Usage Mathematics Expressions

Expression with Enhanced Transformation Capabilities

Expressions use Python SQLAlchemy syntax and provide powerful transformation capabilities.

Common Mathematics Expressions

Analyze ExpressionOperatorDescriptionExampleResult
++addition2 + 35
subtraction2 – 3-1
**multiplication2 * 36
//division (integer division truncates results)4 / 22
column.op(‘%’)%modulo (remainder)5 % 41
column.op(‘^’)^exponentiation2.0 ^ 3.08
column.op(‘!’)!factorial5 !120
column.op(‘!!’)!!factorial (prefix operator)!! 5120
column.op(‘@’)@absolute value@ -5.05
column.op(‘&’)&bitwise AND91 & 1511
column.op(‘#’)##bitwise XOR17 ## 520
column.op(‘~’)~bitwise NOT~1-2
column.op(‘<<’)<<bitwise shift left1 << 416
column.op(‘>>’)>>bitwise shift right8 >> 22

Mathematics Functions

Analyze ExpressionReturn TypeDescriptionExampleResult
func.abs(x)(same as input)absolute valueabs(-17.4 )17.4
func.cbrt(dp)Big Floatcube rootcbrt(27.0 )3
func.ceil(dp or numeric)(same as input)smallest integer not less than argumentceil(-42. 8)-42
func.ceiling(dp or numeric)(same as input)smallest integer not less than argument (alias for ceil)ceiling(- 95.3)-95
func.degrees(dp)Big Floatradians to degreesdegrees(0 .5)28.64788 97565412
func.exp(dp or numeric)(same as input)exponentialexp(1.0)2.718281 82845905
func.floor(dp or numeric)(same as input)largest integer not greater than argumentfloor(-42 .8)-43
func.ln(dp or numeric)(same as input)natural logarithmln(2.0)0.693147 18055994 5
func.log(dp or numeric)(same as input)base 10 logarithmlog(100.0 )2
func.log(b numeric, x numeric)numericlogarithm to base blog(2.0, 64.0)6
func.mod(y, x)(same as argument types)remainder of y/xmod(9,4)1
func.pi()Big Float“π” constantpi()3.141592 65358979
func.power(a dp, b dp)Big Floata raised to the power of bpower(9.0 , 3.0)729
func.power(a numeric, b numeric)numerica raised to the power of bpower(9.0 , 3.0)729
func.radians(dp)Big Floatdegrees to radiansradians(4 5.0)0.785398 16339744 8
func.random()Big Floatrandom value in the range 0.0 <= x < 1.0random()
func.round(dp or numeric)(same as input)round to nearest integerround(42. 4)42
func.round(v numeric, s int)numericround to s decimal placesround(42. 4382, 2)42.44
func.setseed(dp)integerset seed for subsequent random() calls (value between 0 and 1.0)setseed(0 .54823)11773149 59
func.sign(dp or numeric)(same as input)sign of the argument (-1, 0, +1)sign(-8.4 )-1
func.sqrt(dp or numeric)(same as input)square rootsqrt(2.0)1.414213 5623731
func.trunc(dp or numeric)(same as input)truncate toward zerotrunc(42. 8)42
func.trunc(v numeric, s int)numerictruncate to s decimal placestrunc(42. 4382, 2)42.43
func.width_bucket( op numeric, b1 numeric, b2 numeric, count int)integerreturn the bucket to which operand would be assigned in an equidepth histogram with count buckets, in the range b1 to b2width_bu cket(5.35 , 0.024, 10.06, 5)3
func.least(value…)(same as input)Select the smallest value from a list. NULL values in the list are ignored. The result will be NULL only if all values are NULL.
func.greatest(value …)(same as input)Select the largest value from a list. NULL values in the list are ignored. The result will be NULL only if all values are NULL.

Trigonometric Functions

Analyze ExpressionDescription
func.acos(x)inverse cosine
func.asin(x)inverse sine
func.atan(x)inverse tangent
func.atan2(x,y)inverse tangent of x/y
func.cos(x)cosine
func.cot(x)cotangent
func.sin(x)sine
func.tan(x)tangent

11.6 - General Usage PostGIS

Enhanced 'Analyze' functions for geospatial analysis

Analyze provides access to the powerful PostGIS library of functions for geospatial analysis. The functions available are shown in the following table and link to instructions on the PostGIS site.

Geometry Constructors

FunctionDescription
ST_BdPolyFromTextConstruct a Polygon given an arbitrary collection of closed linestrings as a MultiLineString Well-Known text representation.
ST_BdMPolyFromTextConstruct a MultiPolygon given an arbitrary collection of closed linestrings as a MultiLineString text representation Well-Known text representation.
ST_Box2dFromGeoHashReturn a BOX2D from a GeoHash string.
ST_GeogFromTextReturn a specified geography value from Well-Known Text representation or extended (WKT).
ST_GeographyFromTexReturn a specified geography value from Well-Known Text representation or extended (WKT).
ST_GeogFromWKBCreates a geography instance from a Well-Known Binary geometry representation (WKB) or extended Well Known Binary (EWKB).
ST_GeomCollFromTextMakes a collection Geometry from collection WKT with the given SRID. If SRID is not give, it defaults to 0.
ST_GeomFromEWKBReturn a specified ST_Geometry value from Extended Well-Known Binary representation (EWKB).
ST_GeomFromEWKTReturn a specified ST_Geometry value from Extended Well-Known Text representation (EWKT).
ST_GeometryFromTextReturn a specified ST_Geometry value from Well-Known Text representation (WKT). This is an alias name for ST_GeomFromText
ST_GeomFromGeoHashReturn a geometry from a GeoHash string.
ST_GeomFromGMLTakes as input GML representation of geometry and outputs a PostGIS geometry object
ST_GeomFromGeoJSONTakes as input a geojson representation of a geometry and outputs a PostGIS geometry object
ST_GeomFromKMLTakes as input KML representation of geometry and outputs a PostGIS geometry object
ST_GMLToSQLReturn a specified ST_Geometry value from GML representation. This is an alias name for ST_GeomFromGML
ST_GeomFromTextReturn a specified ST_Geometry value from Well-Known Text representation (WKT).
ST_GeomFromWKBCreates a geometry instance from a Well-Known Binary geometry representation (WKB) and optional SRID.
ST_LineFromEncodedPolylineCreates a LineString from an Encoded Polyline.
ST_LineFromMultiPointCreates a LineString from a MultiPoint geometry.
ST_LineFromTextMakes a Geometry from WKT representation with the given SRID. If SRID is not given, it defaults to 0.
ST_LineFromWKBMakes a LINESTRING from WKB with the given SRID
ST_LinestringFromWKBMakes a geometry from WKB with the given SRID.
ST_MakeBox2DCreates a BOX2D defined by the given point geometries.
ST_3DMakeBoxCreates a BOX3D defined by the given 3d point geometries.
ST_MakeLineCreates a Linestring from point or line geometries.
ST_MakeEnvelopeCreates a rectangular Polygon formed from the given minimums and maximums. Input values must be in SRS specified by the SRID
ST_MakePolygonCreates a Polygon formed by the given shell. Input geometries must be closed LINESTRINGS.
ST_MakePointCreates a 2D,3DZ or 4D point geometry.
ST_MakePointMCreates a point geometry with an x, y, and m coordinate.
ST_MLineFromTextReturn a specified ST_MultiLineString value from WKT representation.
ST_MPointFromTextMakes a Geometry from WKT with the given SRID. If SRID is not give, it defaults to 0.
ST_MPolyFromTextMakes a MultiPolygon Geometry from WKT with the given SRID. If SRID is not give, it defaults to 0.
ST_PointReturns an ST_Point with the given coordinate values. OGC alias for ST_MakePoint.
ST_PointFromGeoHashReturn a point from a GeoHash string.
ST_PointFromTextMakes a point Geometry from WKT with the given SRID. If SRID is not given, it defaults to unknown.
ST_PointFromWKBMakes a geometry from WKB with the given SRID
ST_PolygonReturns a polygon built from the specified linestring and SRID.
ST_PolygonFromTextMakes a Geometry from WKT with the given SRID. If SRID is not give, it defaults to 0.
ST_WKBToSQLReturn a specified ST_Geometry value from Well-Known Binary representation (WKB). This is an alias name for ST_GeomFromWKB that takes no srid
ST_WKTToSQLReturn a specified ST_Geometry value from Well-Known Text representation (WKT). This is an alias name for ST_GeomFromText

11.7 - General Usage Text Expressions

Commonly Used Text Expressions

Common Text Expressions

Analyze ExpressionReturn TypeDescriptionExampleResult
func.concat(string, string)textString concatenationconcat(‘Post’, ‘greSQL’)PostgreSQL
func.bit_length(stri ng)intNumber of bits in stringbit_length(‘jose’)32
func.char_length(str ing) or func.character_length(string)intNumber of characters in stringchar_leng th(‘jose’)4
func.convert(string using conversion_name)textChange encoding using specified conversion name. Conversions can be defined by CREATE CONVERSION. Also there are some pre-defined conversion names. SeeTable 9-7for available conversion names.convert(‘P ostgreSQL’ using iso_8859_1_to_ut f8)PostgreSQ L’ in UTF8 (Unicode, 8-bit) encoding
func.lower(string)textConvert string to lower caselower(‘TOM ‘)tom
func.octet_length(st ring)intNumber of bytes in stringoctet_length(‘jose’ )4
func.overlay(string placing string from int [forint])textReplace substringoverlay(‘Txxxxas’ placing ‘hom’ from 2 for 4)Thomas
func.position(substring in string)intLocation of specified substringposition(‘om’ in ‘Thomas’)3
func.substring(string [from int] [for int])textExtract substringsubstring( ‘Thomas’ from 2 for 3)hom
func.substring(string frompattern)textExtract substring matching POSIX regular expression.substring( ‘Thomas’ from ‘…$’)mas
func.substring(string frompatternforescape)textExtract substring matching SQL regular expression.substring( ‘Thomas’ from ‘%#”o_a#” _’ for ‘#’)oma
func.trim([leading, trailing, both] [characters] from string)textRemove the longest string containing only the characters (a space by default) from the start/end/both ends of the stringtrim(both ‘x’ from ‘xTomxx’)Tom
func.upper(string)textConvert string to uppercaseupper(‘tom ‘)TOM
func.ascii(string)intASCII code of the first byte of the argumentascii(‘x’)120
func.btrim(string text [, characters text])textRemove the longest string consisting only of characters in characters (a space by default) from the start and end of stringbtrim(‘xyx trimyyx’, ‘xy’)trim
func.chr(int)textCharacter with the given ASCII codechr(65)A
func.convert(string text, [src_encoding name,]dest_encoding name)textConvert string to dest_encoding . The original encoding is specified by src_encoding. If src_encoding is omitted, database encoding is assumed.convert( ‘text_in_utf8’, ‘UTF8’, ‘LATIN1’)text_in_utf8 represent ed in ISO 8859-1 encoding
func.decode(string text, type text)Decode binary data from string previously encoded with encode. Parameter type is same as in encode.
func.decode(string text, type text)byteaDecode binary data from string previously encoded with encode. Parameter type is same as in encode.
func.initcap(string)textConvert the first letter of each word to uppercase and the rest to lowercase. Words are sequences of alphanumeric characters separated by non-alphanumer ic characters.initcap(‘hi THOMAS’)Hi Thomas
func.length(string)intNumber of characters in stringlength(‘jose’)4
func.lpad(string text, length int [, fill text])textFill up the string to length length by prepending the characters fill (a space by default). If the string is already longer than length then it is truncated (on the right).lpad(‘hi’, 5, ‘xy’)xyxhi
func.ltrim(string text [, characters text])textRemove the longest string containing only characters from characters (a space by default) from the start of stringltrim(‘zzz ytrim’, ‘xyz’)trim
func.md5(string)textCalculates the MD5 hash of string, returning the result in hexadecimalmd5(‘abc’)900150983 cd24fb0 d6963f7d2 8e17f72
func.quote_literal(s tring)textReturn the given string suitably quoted to be used as a string literal in an SQL statement string. Embedded single-quotes and backslashes are properly doubled.quote_lit eral( ‘O’Reilly’ )O’’Reilly ‘
func.regexp_replace( string text, pattern text,replacement text [,flags text])textReplace substring matching POSIX regular expression. SeeSection 9.7for more information on pattern matching.regexp_re place(‘Tho mas’, ‘.[mN]a.’, ‘M’)ThM
func.repeat(string text, number int)textRepeat string the specified number of timesrepeat(‘Pg ‘, 4)PgPgPgPg
func.replace(string text, from text, to text)textReplace all occurrences in string of substring from with substring toreplace( ‘abcdefabc def’, ‘cd’, ‘XX’)abXXefabX Xef
func.rpad(string text, length int [, fill text])textFill up the string to length length by appending the characters fill (a space by default). If the string is already longer than length then it is truncated.rpad(‘hi’, 5, ‘xy’)hixyx
func.rtrim(string text [, characters text])textRemove the longest string containing only characters from characters (a space by default) from the end of stringrtrim(‘trimxxxx’, ‘x’)trim
func.split_part(stri ng text, delimiter text, field int)textSplit string on delimiter and return the given field (counting from one)split_par t(‘abc:s ub:@de f@ ghi’, ‘:sub:@’ , 2)def
func.strpos(string, substring)intLocation of specified substring (same as position(subst ring in string), but note the reversed argument order)strpos(‘hi gh’, ‘ig’)2
func.substr(string, from [, count])textExtract substring (same as substring(stri ng from from for count))substr(‘al phabet’, 3, 2)ph
func.to_ascii(string text [, encoding text])textConvert string to ASCII from another encoding (only supports conversion from LATIN1, LATIN2, LATIN9, and WIN1250 encodings)to_ascii( ‘Karel’)Karel
func.to_hex(number int or bigint)textConvert number to its equivalent hexadecimal representationto_hex(21 47483647)7fffffff
func.translate(string text, from text, to text)textAny character in string that matches a character in the from set is replaced by the corresponding character in the to settranslate( ‘12345’, ‘14’, ‘ax’)a23x5

11.8 - Arrays

11.8.1 - func.string_to_array

This function is used to split a string into array elements using supplied delimiter and optional null string

Syntax

func.string_to_array(text, delimiter)

Examples

Documentation for func.string_to_array is coming soon.

11.8.2 - func.unnest

This function is used to expand an array to a set of rows

Syntax

func.unnest(text)

Examples

Documentation for func.unnest is coming soon.

11.9 - Column Conditions

11.9.1 - column.any_

The PostgreSQL ANY operator compares a value in a column to a set of values returned by a subquery

Syntax

.any_([list of values])

Examples

No examples yet...

11.9.2 - column.between

Returns true if the current column is between the lower bound and upper bound, inclusive

Syntax

.between(value_1, value_2)

Examples

Documentation for column.between is coming soon.

11.9.3 - column.contains

The contains() method checks whether a string contains a sequence of characters. Returns true if the characters exist and false if not

Syntax

.contains(string)

Examples

Documentation for column.contains is coming soon.

11.9.4 - column.endswith

The endsWith() method determines whether a string ends with the characters of a specified string, returning true or false as appropriate

Syntax

.endswith(string)

Examples

Documentation for column.endswith is coming soon.

11.9.5 - column.ilike

Allows matching of strings based on comparison with a pattern

Syntax

.ilike(string)

Examples

Documentation for column.ilike is coming soon.

11.9.6 - column.in_

Checks if column conditions are met

Syntax

.in_([list of values])

Examples

Documentation for column.in_ is coming soon.

11.9.7 - column.is_

Checks if column conditions are met

Syntax

.is_(value)
.is_(None)

Examples

Documentation for column.is_ is coming soon.

11.9.8 - column.isnot

Checks if column conditions are met

Syntax

.isnot(None)
.isnot(value)

Examples

Documentation for column.isnot is coming soon.

11.9.9 - column.like

The PostgreSQL LIKE operator is used to match text values against a pattern using wildcards

Syntax

.like(string)

Examples

Documentation for column.like is coming soon.

11.9.10 - column.notilike

This function is used to data using pattern matching techniques that explicitly excludes the mentioned pattern from the query result set

Syntax

.notilike(string)

Examples

Documentation for column.notilike is coming soon.

11.9.11 - column.notin_

Checks if column conditions are met

Syntax

column.notin_([list of values])

Examples

Documentation for column.notin_ is coming soon.

11.9.12 - column.notlike

Checks if column conditions are met

Syntax

column.notlike(string)

Examples

Documentation for column.notlike is coming soon.

11.9.13 - column.startswith

The system function 'starts_with()' is used to determine whether the provided string starts with the specified prefix

Syntax

.startswith(string)

Examples

Documentation for column.startswith is coming soon.

11.10 - Conditions

11.10.1 - case

Checks if conditions are met

Syntax

case((condition1, value1),(condition2, value2), else_ = None)

Example 1

In this example is from a Table:Lookup step where we are updating the "dock_final" column when the table1. dock_final value is Null.

case(
    (table1.dock_final == Null, table2.dock_final), else_ = table1.dock_final
    )

Example 2

This example is from a Table:Lookup step where we are updating the "Marketing Channel" column when "Marketing Channel" in table1 is not 'none' or the "Serial Number" contains a '_'.

case(
    (get_column(table1, 'Marketing Channel') != 'none', get_column(table1, 'Marketing Channel')),
    (get_column(table1, 'Serial Number').contains('_'), get_column(table1, 'Marketing Channel')),
    (get_column(table2, 'Marketing Channel') != Null, get_column(table2, 'Marketing Channel')), 
    else_ = 'none'
    )

11.10.2 - func.coalesce

The COALESCE() function returns the first non-null value in a list

Syntax

func.coalesce(value, value, value...)

Examples

Documentation for func.coalesce is coming soon.

11.10.3 - not_

The PostgreSQL NOT condition (also called the NOT Operator) is used to negate a condition in a SELECT, INSERT, UPDATE, or DELETE statement

Syntax

not_(value)

Examples

Documentation for not_ is coming soon.

11.10.4 - and_

Checks if all conditions are met

The and_ operator allows checking if two or more conditions are all met.


Examples

Example 1

This example checks if the period is any of the three specified dates.

and_(  
    table.color == 'green',  
    table.shape == 'circle',  
    table.price >= 1.25  
)

Example 2

This example is checking if to ensure the origin_plant is not one of the values specified. This is using the != expression.

and_(  
    table.origin_plant != '5013',  
    table.origin_plant != '5026',  
    table.origin_plant != '5120',  
    table.origin_plant != '5287',  
    table.origin_plant != '5161',  
    table.origin_plant != '5192'  
)

Alternatively, for reference, the above check could be written using the not_ and or_ operators like this:

not_(  
    or_(  
        table.origin_plant == '5013',  
        table.origin_plant == '5026',  
        table.origin_plant == '5120',  
        table.origin_plant == '5287',  
        table.origin_plant == '5161',  
        table.origin_plant == '5192'  
    )  
)

11.10.5 - or_

Checks if any conditions are met

The or_ operator allows checking two or more conditions to determine if the evaluation is true or false.


Examples

Example 1

This example checks if the period is any of the three specified dates.

or_(  
    table.period == '2020_10',  
    table.period == '2020_11',  
    table.period == '2020_12'  
)

Example 2

This example is checking if order_reason_Include is null or has the word KEEP as a value.

or_(  
    table.order_reason_Include == 'KEEP',  
    table.order_reason_Include.is_(None)  
)

11.11 - Conversions

11.11.1 - cast

The CAST() function converts a value (of any type) into a specified datatype

Syntax

cast(value, target_type)

Examples

Documentation for cast is coming soon.

11.11.2 - func.to_char

The TO_CHAR function converts a number or date to a string

Syntax

func.to_char(value, text)

Examples

Documentation for func.to_char is coming soon.

11.11.3 - func.to_date

The TO_DATE function accepts an argument of a character data type and converts this value to a DATETIME value

Syntax

func.to_date(text, text)

Examples

Documentation for func.to_date is coming soon.

11.11.4 - func.to_number

The TO_NUMBER function can convert a number or a character expression representing a number value to a DECIMAL data type

Syntax

func.to_number(text, text) 

Examples

Documentation for func.to_number is coming soon.

11.11.5 - func.to_timestamp

The TO_TIMESTAMP function converts char of CHAR , VARCHAR2 , NCHAR , or NVARCHAR2 datatype to a value of TIMESTAMP datatype

Syntax

func.to_timestamp(text, text)

Examples

Documentation for func.to_timestamp is coming soon.

11.12 - Date and Time

11.12.1 - func.age

Subtract arguments, producing a “symbolic” result that uses years and months, rather than just days

Syntax

func.age(timestamp, timestamp)
func.age(timestamp)

Examples

Documentation for func.age is coming soon.

11.12.2 - func.clock_timestamp

Current date and time (changes during statement execution)

Syntax

func.clock_timestamp() 

Examples

Documentation for func.clock_timestamp is coming soon.

11.12.3 - func.current_date

Returns current date value based on the start of the current transaction

Syntax

func.current_date()

Examples

Documentation for func.current_date is coming soon.

11.12.4 - func.current_time

Returns a current time value based on the time zone

Syntax

func.current_time()

Examples

Documentation for func.current_time is coming soon.

11.12.5 - func.current_timestamp

Returns a current date and timestamp based on the local time zone and start of current transaction

Syntax

func.current_timestamp()

Examples

Documentation for func.current_timestamp is coming soon.

11.12.6 - func.date_part

Returns a 'date' value expression through its subfields

Syntax

func.date_part(text, timestamp or interval)

Examples

Documentation for func.date_part is coming soon.

11.12.7 - func.date_trunc

Truncate input value to specified precision

Syntax

func.date_trunc(text, timestamp)

Examples

Documentation for func.date_trunc is coming soon.

11.12.8 - func.extract

Retrieves subfields such as year or hour from date/time values

Syntax

func.extract(field, timestamp or interval)

Examples

Documentation for func.extract is coming soon.

11.12.9 - func.isfinite

This function is used to determine whether the numeric value is finite

Syntax

func.ascii(text) → integer

Returns the numeric code of the first character of the argument. In UTF8 encoding, returns the Unicode code point of the character. In other multibyte encodings, the argument must be an ASCII character.

Examples

func.ascii('x') → 120

External References

PostgreSQL Documentation

11.12.10 - func.justify_days

Adjust interval so 30-day time periods are represented as months

Syntax

func.justify_days(interval)

Examples

Documentation for func.justify_days is coming soon.

11.12.11 - func.justify_hours

Adjust interval so 24-hour time periods are represented as days

Syntax

func.justify_hours(interval)

Examples

Documentation for func.justify_hours is coming soon.

11.12.12 - func.justify_interval

This function is used to adjust interval using justify_days and justify_hours, with additional sign adjustments

Syntax

func.justify_interval(interval)

Examples

Documentation for func.justify_interval is coming soon.

11.12.13 - func.now

Returns current date and time based on the database server's time zone setting

Syntax

func.now()

Examples

Documentation for func.now is coming soon.

11.12.14 - func.statement_timestamp

This function is used to get current date and time (start of current transaction)

Syntax

func.statement_timestamp()

Examples

Documentation for func.statement_timestamp is coming soon.

11.12.15 - func.timeofday

This function is used to get current date and time (like clock_timestamp, but as a text string)

Syntax

func.timeofday()

Examples

Documentation for func.timeofday is coming soon.

11.12.16 - func.transaction_timestamp

Returns the current date and time (including the time zone offset), at the start of the current transaction. Equivalent to 'func.now'

Syntax

func.transaction_timestamp()

Examples

Documentation for func.transaction_timestamp is coming soon.

11.13 - Grouping and Summarization

11.13.1 - func.first

This function returns the value of a specified field in the first record of the result set returned by a query

Syntax

func.first(field)

Examples

Documentation for func.first is coming soon.

11.13.2 - func.last

This function returns the value of a specified field in the last record of the result set returned by a query

Syntax

func.last(field)

Examples

Documentation for func.last is coming soon.

11.13.3 - func.max

The MAX function is an aggregate function that returns the maximum value in a set of values

Syntax

func.max(field)

Examples

Documentation for func.max is coming soon.

11.13.4 - func.median

Thia function will calculate the middle value of a given set of numbers

Syntax

func.median(field)

Examples

Documentation for func.median is coming soon.

11.13.5 - func.stdev

The STDEV function calculates the standard deviation for a sample set of data

Syntax

func.stdev(field) 

Examples

Documentation for func.stdev is coming soon.

11.13.6 - func.stdev_pop

STDDEV_POP computes the population standard deviation and returns the square root of the population variance

Syntax

func.stdev_pop(field)

Examples

Documentation for func.stdev_pop is coming soon.

11.13.7 - func.stdev_samp

STDDEV_SAMP() function returns the sample standard deviation of an expression

Syntax

func.stdev_samp(field)

Examples

Documentation for func.stdev_samp is coming soon.

11.13.8 - func.var_pop

VAR_POP returns the population variance of a set of numbers after discarding the nulls in this set

Syntax

func.var_pop(field) 

Examples

Documentation for func.var_pop is coming soon.

11.13.9 - func.var_samp

VAR_SAMP returns the sample variance of a set of numbers after discarding the nulls in this set

Syntax

func.var_samp(field)

Examples

Documentation for func.var_samp is coming soon.

11.13.10 - func.variance

This function is used to determine how far a set of values is spread out based on a sample of the population

Syntax

func.variance(field) 

Examples

Documentation for func.variance is coming soon.

11.14 - JSON

11.14.1 - func.array_to_json

Returns the array as JSON. A PostgreSQL multidimensional array becomes a JSON array of arrays.

Syntax

func.array_to_json(array)

Examples

Documentation for array_to_json is coming soon.

11.14.2 - func.json_array_elements

Expands a JSON array to a set of JSON elements.

Syntax

func.json_array_elements(json)

Examples

Documentation for json_array_elements is coming soon.

11.14.3 - func.json_each

Expands the outermost JSON object into a set of key/value pairs

Syntax

func.json_each(json)

Examples

Documentation for json_each is coming soon.

11.14.4 - func.json_each_text

Expands the outermost JSON object into a set of key/value pairs. The returned value will be of type text.

Syntax

func.json_each_text(json)

Examples

Documentation for json_each_text is coming soon.

11.14.5 - func.json_extract_path

Returns JSON object pointed to by path elements. The return value will be a type of JSON.

Syntax

func.json_extract_path(json, key_1, key_2, ...)

Examples

Documentation for json_extract_path is coming soon.

11.14.6 - func.json_extract_path_text

Returns JSON object pointed to by path elements. The return value will be a type of text.

Syntax

func.json_extract_path_text(json, key_1, key_2, ...)

Examples

Documentation for json_extract_path_text is coming soon.

11.14.7 - func.json_object_keys

Returns set of keys in the JSON object. Only the "outer" object will be displayed.

Syntax

func.json_object_keys(json)

Examples

Documentation for json_object_keys is coming soon.

11.15 - Math

11.15.1 - func.abs

Returns the absolute value of a number. The absolute value of a number is the number without its sign

Syntax

func.abs(x)

Examples

Documentation for func.abs is coming soon.

11.15.2 - func.cbrt

This function accepts a number as argument and returns the cube root of that number

Syntax

func.cbrt(numeric)

Examples

Documentation for func.cbrt is coming soon.

11.15.3 - func.ceil

The ceil() function is used to return the value, after rounded up any positive or negative decimal value as greater than the argument

Syntax

func.ceil(numeric)

Examples

Documentation for func.ceil is coming soon.

11.15.4 - func.ceiling

The PostgreSQL ceiling() is used to return the value after rounded up any positive or negative decimal value as greater than the argument

Syntax

func.ceiling(numeric)

Examples

Documentation for func.ceiling is coming soon.

11.15.5 - func.degrees

The PostgreSQL degrees() function is used to return the values in degrees from radian as specified in the argument.

Syntax

func.degrees(numeric)

Examples

Documentation for func.degrees is coming soon.

11.15.6 - func.exp

The PostgreSQL exp function returns 'e' raised to the power of 'number'

Syntax

func.exp(numeric)

Examples

Documentation for func.exp is coming soon.

11.15.7 - func.floor

The PostgreSQL FLOOR() function returns a number rounded down to the next whole number

Syntax

func.floor(numeric)

Examples

func.ascii('x') → 12Documentation for func.floor is coming soon.

11.15.8 - func.greatest

he GREATEST function returns the “greatest” or “largest” value from the list of expressions

Syntax

func.greatest(value_1, value_2)

Examples

Documentation for func.greatest is coming soon.

11.15.9 - func.least

The LEAST function returns the “least” or “smallest” value from the list of expressions

Syntax

func.least(value_1, value_2)

Examples

Documentation for func.least is coming soon.

11.15.10 - func.ln

The PostgreSQL ln() function is used to return the natural logarithm of a given number, as specified in the argument.

Syntax

func.ln(numeric)

Examples

Documentation for func.ln is coming soon.

11.15.11 - func.log

The log() function returns the base 10 logarithm of a given number or logarithm of a number for a particular base, specific to the argument

Syntax

func.log(numeric)
func.log(b numeric, x numeric)

Examples

Documentation for func.log is coming soon.

11.15.12 - func.mod

The MOD() function performs the modulo operation that returns the remainder after the division of the first argument by the second one

Syntax

func.mod(y, x)

Examples

Documentation for func.mod is coming soon.

11.15.13 - func.pi

The PostgreSQL pi() function is used to return the constant value of pi

Syntax

func.pi()

Examples

Documentation for func.pi is coming soon.

11.15.14 - func.power

The PostgreSQL power() function is used to return the value of one number raised to the power of another number, provided in the argument

Syntax

func.power(a numeric, b numeric)

Examples

Documentation for func.power is coming soon.

11.15.15 - func.radians

The PostgreSQL radians() function is used to return the value in radian from degrees, provided in the argument

Syntax

func.radians(numeric)

Examples

Documentation for func.radians is coming soon.

11.15.16 - func.random

The PostgreSQL random() function is used to return the random value between 0 and 1.

Syntax

func.random()

Examples

Documentation for func.random is coming soon.

11.15.17 - func.round

The PostgreSQL ROUND() function rounds a numeric value to its nearest integer or a number with the number of decimal places

Syntax

func.round(numeric)
func.round(v numeric, s int)

Examples

Documentation for func.round is coming soon.

11.15.18 - func.safe_divide

Equivalent to the division operator (X / Y), but returns NULL if an error occurs, such as a division by zero error

Syntax

func.safe_divide(numerator numeric, denominator numeric, divide_by_zero_value)

Examples

Documentation for func.safe_divide is coming soon.

11.15.19 - func.setseed

The PostgreSQL setseed function can be used to set a seed for the next time that you call the random function

Syntax

func.setseed(numeric)

Examples

Documentation for func.setseed is coming soon.

11.15.20 - func.sign

The PostgreSQL sign function returns a value indicating the sign of a number.

Syntax

func.sign(numeric)

Examples

Documentation for func.sign is coming soon.

11.15.21 - func.sqrt

The PostgreSQL sqrt() function is used to return the square root of a given positive number.

Syntax

func.sqrt(numeric)

Examples

Documentation for func.sqrt is coming soon.

11.15.22 - func.trunc

The PostgreSQL TRUNC() function returns a number truncated to a whole number or truncated to the specified decimal places

Syntax

func.trunc(float) 
func.trunc(v float, s int)

Examples

Documentation for func.trunc is coming soon.

11.15.23 - func.width_bucket

In PostgreSQL, width_bucket() is a mathematical function that assigns values to buckets (individual segments) in an equiwidth histogram.

Syntax

func.width_bucket(op numeric, b1 numeric, b2 numeric, count int)

Examples

Documentation for func.width_bucket is coming soon.

11.16 - Text

11.16.1 - func.trim

TRIM is a function that takes a character expression and returns that expression with leading and/or trailing pad characters removed

Syntax

func.trim(value_string, string_to_trim)

Examples

func.trim('Plaidcloudx', 'x') --> Plaidcloud

11.16.2 - func.translate

Any character in string that matches a character in the from set is replaced by the corresponding character in the to set

Syntax

func.translate(value_string, from_string_elements, to_string_elements)

Examples

func.translate('Plaixcloux', 'x', 'd') --> Plaidcloud

11.16.3 - func.substring

The SUBSTRING() extracts a substring with a specified length starting from a location in an input string

Syntax

func.substring(string, start, length)

Examples

Example 1: Takes the first 5 characters from a 9 digit postal code func.substring(table.ship_to_postal_code, 1, 5)  
  
func.substring('Plaidcloud_string', 1,10) --> Plaidcloud

11.16.4 - func.strpos

The strpos() function finds the position of the first occurrence of a string inside another string

Syntax

func.strpos(string, substring)

Examples

func.strpos('Plaidcloud string', 'string') -->11

11.16.5 - func.rtrim

The rtrim() function removes whitespace or other predefined characters from the right side of a string

Syntax

func.rtrim(value_string, string_to_trim)

Examples

func.rtrim('Plaidcloud    ') --> Plaidcloud  
func.rtrim('PlaidcloudXXXXX','XXXXX') --> Plaidcloud

11.16.6 - func.rpad

RPAD function pads the right-side of a string with a specific set of characters (when string1 is not null).

Syntax

func.rpad(value_string, force_length, fill_string)

Examples

func.rpad('Plaidcloud',15,'X') --> PlaidcloudXXXXX

11.16.7 - func.replace

The function will replace part of a text string, based on the number of characters you specify, with a different text string

Syntax

func.replace(value, search, replace)

Examples

func.replace('string_to_replace_with_spaces','_',' ') --> string to replace with spaces

11.16.8 - func.repeat

This function takes an input of Any Type and returns a list with the input repeated a specified number of times.

Syntax

func.repeat(string, repeat_int)

Examples

func.repeat('X', 5) --> XXXXX

11.16.9 - func.regexp_replace

This function returns a new string with the substrings, which match a regular expression pattern, replaced by a new substring

Syntax

func.regexp_replace(value, search, replace)

Examples

func.regexp_replace('plaidcloud', 'p', 'P') --> Plaidcloud  
func.regexp_replace('remove12345alphabets','[[:alpha:]]','','g') --> 12345  
func.regexp_replace('remove12345digits','[[:digit:]]','','g') --> removedigits  
  

11.16.10 - func.quote_literal

Returns the specified string as a quoted string so that it can be used as a string literal in a SQL statement

Syntax

func.quote_literal(string)

Examples

func.quote_literal('plaidcloud') --> 'plaidcloud'

11.16.11 - func.octet_length

The OCTET_LENGTH function returns an integer indicating the number of bytes in the input string

Syntax

func.octet_length(string)

Examples

func.octet_length('plaidcloud') --> 10

11.16.12 - func.numericize

Attempts to coerce a non-numeric R object to natomic_object() or list of {natomic_object}

Syntax

func.numericize(string)

Examples

func.numericize('100') --> 100

11.16.13 - func.md5

MD5(Message Digest Function) evaluates data integrity. It calculates the MD5 hash of a string and returns the result in hexadecimal.

Syntax

func.md5(string)

Examples

func.md5('plaidcloud')  --> e324c2bc5ad6ca5aa52048e0b207aa4c

11.16.14 - func.ltrim

LTRIM() function helps to return remove all the space characters found on the left-hand side of the string

Syntax

func.ltrim(value_string, string_to_trim)

Examples

func.ltrim('texttotrimplaidcloud', 'texttotrim') --> plaidcloud  
func.ltrim('      plaidcloud')                   --> plaidcloud

11.16.15 - func.lpad

LPAD function pads the left-side of a string with a specific set of characters (when string1 is not null).

Syntax

func.lpad(value_string, force_length, fill_string)

Examples

func.lpad('stringtofillup', 20, 'X') --> XXXXXXstringtofillup  
func.lpad('stringtofillup', 10, 'X') --> stringtofi

11.16.16 - func.lower

The function will return the lowercase version of the text string given.

Syntax

func.lower(string)

Examples

func.lower('PLAIDCLOUD') --> plaidcloud

11.16.17 - func.length

Indicates how many arguments the function expects

Syntax

func.length(string)

Examples

func.length('plaidcloud') --> 10

11.16.18 - func.integerize_truncate

Takes a single numeric argument x and returns a numeric vector containing the integers formed by truncating the values in x toward 0

Syntax

func.integerize_truncate(string)

Examples

func.integerize_truncate('30.66') --> 30  
func.integerize_truncate('22.13') --> 22

11.16.19 - func.integerize_round

Rounds the values in its first argument to the specified number of decimal places

Syntax

func.integerize_round(string)

Examples

func.integerize_round('30.66') --> 31  
func.integerize_round('22.13') --> 22

11.16.20 - func.initcap

This function converts the first letter of each word in a string to uppercase; converts any remaining characters in each word to lowercase

Syntax

func.initcap(string)

Examples

func.initcap('joe smith') --> Joe Smith

11.16.21 - func.concat

The CONCAT function combines the text from multiple ranges and/or strings, but it doesn't provide delimiter or IgnoreEmpty arguments

Syntax

func.concat(string, string)

Examples

func.concat('www.','plaid','cloud','.com') --> www.plaidcloud.com

11.16.22 - func.chr

The chr() function returns the character that represents the specified unicode

Syntax

func.chr(int)

Examples

func.chr(80) --> 80

11.16.23 - func.char_length

The CHAR_LENGTH function returns the number of logical characters in its argument -character column, character variable, or a quoted string

Syntax

func.char_length(string)

Examples

func.char_length('plaidcloud_test') --> 15

11.16.24 - func.btrim

The PostgreSQL btrim function is used to remove the longest string specified in the argument from the start and end of the given string

Syntax

func.btrim(value_string, string_to_trim)

Examples

func.btrim('plaidcloudplaid','plaid') --> clou

11.16.25 - func.upper

This method converts all lowercase characters in a string into uppercase characters and returns it

Syntax

func.upper(string)

Examples

Documentation for func.upper is coming soon...

11.16.26 - func.bit_length

Returns the number of bits required to represent an integer in binary, excluding the sign and leading zeros

Syntax

func.bit_length(string)

Examples

func.bit_length('text') --> 32

11.16.27 - func.ascii

The ASCII function takes a single argument of any character data type and returns an integer based on the first character data

Syntax

func.ascii(text) → integer

Returns the numeric code of the first character of the argument. In UTF8 encoding, returns the Unicode code point of the character. In other multibyte encodings, the argument must be an ASCII character.

Examples

func.ascii('x') → 135

11.16.28 - func.metric_multiply

The Multiply function can take multiple metrics as inputs and multiply the values of the metrics

Syntax

func.metric_multiply(string)

Examples

Documentation for func.metric_multiply is coming soon.

11.16.29 - func.split_part

The split_part function is used to split a given string based on a delimiter and pick out the desired field from the string

Syntax

func.split_part(value_string, split_on_string, group_number_int)

Examples

Documentation for func.split_part is coming soon.

11.16.30 - func.text_to_bigint

This function allows you to convert a string of character values into a large range integer

Syntax

func.text_to_bigint(string)

Examples

Documentation for func.text_to_bigint is coming soon.

11.16.31 - func.text_to_bool

Converts the input text or numeric expression to a Boolean value

Syntax

func.text_to_bool(string)

Examples

Documentation for func.text_to_bool is coming soon.

11.16.32 - func.text_to_integer

Convert text to integer

Syntax

func.text_to_integer(string)

Examples

Documentation for func.text_to_integer is coming soon.

11.16.33 - func.text_to_numeric

This function converts a character string to a numeric value

Syntax

func.text_to_numeric(string)

Examples

Documentation for func.text_to_numeric is coming soon.

11.16.34 - func.text_to_smallint

A 2-byte integer data type used in CREATE TABLE and ALTER TABLE statements

Syntax

func.text_to_smallint(string)

Examples

Documentation for func.text_to_smallint is coming soon.

11.16.35 - func.to_ascii

Convert text to ASCII from another encoding

Syntax

func.to_ascii(string)

Examples

Documentation for func.to_ascii is coming soon.

11.16.36 - func.to_hex

Convert number to its equivalent hexadecimal representation

Syntax

func.to_hex(number int or bigint)

Examples

Documentation for func.to_hex is coming soon.

11.17 - Trigonometry

11.17.1 - func.acos

The ACOS function returns the arc cosine of a specified number

Syntax

func.acos(x)

Examples

Documentation for func.acos is coming soon.

11.17.2 - func.asin

The ASIN function returns the inverse sine of a value in radians

Syntax

func.asin(x)

Examples

Documentation for func.asin is coming soon.

11.17.3 - func.atan

This function returns the inverse tangent of a number

Syntax

func.atan(x)

Examples

Documentation for func.atan is coming soon.

11.17.4 - func.atan2

The PostgreSQL atan2() function is used to return the inverse tangent of a division given in the argument

Syntax

func.atan2(x,y)

Examples

Documentation for func.atan2 is coming soon.

11.17.5 - func.cos

The PostgreSQL cos() returns the cosine of a given argument

Syntax

func.cos(x)

Examples

Documentation for func.cos is coming soon.

11.17.6 - func.cot

The PostgreSQL cot() function is used to return the cotangent of a given argument

Syntax

func.cot(x)

Examples

Documentation for func.cot is coming soon.

11.17.7 - func.sin

In PostgreSQL, sin() is a mathematical function that returns the trigonometric sine of the specified angle, as measured in radians

Syntax

func.sin(x)

Examples

Documentation for func.sin is coming soon.

11.17.8 - func.tan

The PostgreSQL tan() function is used to return the tangent of a given argument.

Syntax

func.tan(x)

Examples

Documentation for func.tan is coming soon.

11.18 - Window Functions

11.18.1 - func.avg

This function returns the average of the values in a group. It ignores null values

Syntax

func.avg().over(partition_by=field, order_by=field)

Examples

Documentation for func.avg is coming soon.

11.18.2 - func.count

An aggregate function that returns the number of rows, or the number of non-NULL rows

Syntax

func.count().over(partition_by=field, order_by=field) 

Examples

Documentation for func.count is coming soon.

11.18.3 - func.cume_dist

This function calculates the cumulative distribution of a value within a group of values

Syntax

func.cume_dist().over(partition_by=field, order_by=field)

Examples

Documentation for func.cume_dist is coming soon.

11.18.4 - func.dense_rank

The DENSE_RANK() is a window function that assigns a rank to each row within a partition of a result set

Syntax

func.dense_rank().over(partition_by=field, order_by=field)

Examples

Documentation for func.dense_rank is coming soon.

11.18.5 - func.first_value

FIRST_VALUE is an analytic function. It returns the first value in an ordered set of values

Syntax

func.first_value(field).over(partition_by=field, order_by=field)

Examples

This is an example of using the first_value() capability to calculate the running time of the time series data where each event is on a distinct row.

This assumes you have a table of time series data that looks like this:

locationemployeetimestamp
Building AJohn Doe2022-01-05 15:34:31
Building AJohn Doe2022-01-05 15:44:31
Building AJohn Doe2022-01-05 15:46:41
table.timestamp - func.first_value(table.timestamp, 1).over(partition_by=[table.location, table.employee], order_by=table.timestamp)

Adding the expression above to an Interval column called run_time would result in an output table like this:

locationemployeetimestamprun_time
Building AJohn Doe2022-01-05 15:34:3100:00:00
Building AJohn Doe2022-01-05 15:44:3100:10:00
Building AJohn Doe2022-01-05 15:46:4100:12:10

11.18.6 - func.lag

This analytic function lets you query more than one row in a table at a time without having to join the table to itself

Syntax

func.lag(field, 1).over(partition_by=field, order_by=field)

Examples

This is an example of using the lag() capability to calculate the time interval in time series data where each event is on a distinct row.

This assumes you have a table of time series data that looks like this:

locationemployeetimestamp
Building AJohn Doe2022-01-05 15:34:31
Building AJohn Doe2022-01-05 15:44:31
Building AJohn Doe2022-01-05 15:46:41
table.timestamp - func.lag(table.timestamp, 1).over(partition_by=[table.location, table.employee], order_by=table.timestamp)

Adding the expression above to an Interval column called delta would result in an output table like this:

locationemployeetimestampdelta
Building AJohn Doe2022-01-05 15:34:31null
Building AJohn Doe2022-01-05 15:44:3100:10:00
Building AJohn Doe2022-01-05 15:46:4100:02:10

11.18.7 - func.last_value

The LAST_VALUE() function is a window function that returns the last value in an ordered partition of a result set

Syntax

func.last_value(field).over(partition_by=field, order_by=field)

Examples

This is an example of using the last_value() capability to calculate the time remaining in time series data where each event is on a distinct row.

This assumes you have a table of time series data that looks like this:

locationemployeetimestamp
Building AJohn Doe2022-01-05 15:34:31
Building AJohn Doe2022-01-05 15:44:31
Building AJohn Doe2022-01-05 15:46:41
func.last_value(table.timestamp, 1).over(partition_by=[table.location, table.employee], order_by=table.timestamp) - table.timestamp

Adding the expression above to an Interval column called remaining would result in an output table like this:

locationemployeetimestampremaining
Building AJohn Doe2022-01-05 15:34:3100:12:10
Building AJohn Doe2022-01-05 15:44:3100:02:10
Building AJohn Doe2022-01-05 15:46:4100:00:00

11.18.8 - func.lead

This analytic function provides access to more than one row of a table at the same time without a self join

Syntax

func.lead(field, 1).over(partition_by=field, order_by=field)

Examples

Documentation for func.lead is coming soon.

11.18.9 - func.min

The min() function returns the item with the lowest value, or the item with the lowest value in an iterable

Syntax

func.min().over(partition_by=field, order_by=field)

Examples

Documentation for func.min is coming soon.

11.18.10 - func.ntile

This is a function that distributes rows of an ordered partition into a specified number of approximately equal groups, or buckets

Syntax

func.ntile(4).over(partition_by=field, order_by=field)

Examples

Documentation for func.ntile is coming soon.

11.18.11 - func.percent_rank

The PERCENT_RANK() function evaluates the relative standing of a value within a partition of a result set

Syntax

func.percent_rank().over(partition_by=field, order_by=field)

Examples

Documentation for func.percent_rank is coming soon.

11.18.12 - func.rank

This is a function that assigns a rank to each row within a partition of a result set

Syntax

func.rank().over(partition_by=field, order_by=field)

Examples

Documentation for func.rank is coming soon.

11.18.13 - func.row_number

This function is used to provide consecutive numbering of the rows in the result by the order selected in the OVER clause for each partition

Syntax

func.row_number().over(partition_by=field, order_by=field)

Examples

Documentation for func.row_number is coming soon.

11.18.14 - func.sum

The SUM function adds values. You can add individual values, cell references or ranges or a mix of all three

Syntax

func.sum().over(partition_by=field, order_by=field)

Examples

Documentation for func.sum is coming soon.

12 - PostGIS Expressions (Geospatial)

12.1 - Affine Transformations

12.1.1 - func.ST_TransScale

Translates the geometry using the deltaX and deltaY args, then scales it using the XFactor, YFactor args, working in 2D only

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_TransScale(geometry geomA, float deltaX, float deltaY, float XFactor, float YFactor);

PlaidCloud

func.ST_TransScale(geometry geomA, float deltaX, float deltaY, float XFactor, float YFactor)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.2 - func.ST_Translate

Returns a new geometry whose coordinates are translated delta x,delta y,delta z units

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Translate(geometry g1, float deltax, float deltay);

PlaidCloud

func.ST_Translate(geometry g1, float deltax, float deltay)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.3 - func.ST_Scale

Scales the geometry to a new size by multiplying the ordinates with the corresponding factor parameters

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Scale(geometry geom, geometry factor);

PlaidCloud

func.ST_Scale(geometry geom, geometry factor)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.4 - func.ST_RotateZ

Rotates a geometry geomA - rotRadians about the Z axis

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_RotateZ(geometry geomA, float rotRadians);

PlaidCloud

func.ST_RotateZ(geometry geomA, float rotRadians)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.5 - func.ST_RotateY

Rotates a geometry geomA - rotRadians about the y axis

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_RotateY(geometry geomA, float rotRadians);

PlaidCloud

func.ST_RotateY(geometry geomA, float rotRadians)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.6 - func.ST_RotateX

Rotates a geometry geomA - rotRadians about the X axis

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_RotateX(geometry geomA, float rotRadians);

PlaidCloud

func.ST_RotateX(geometry geomA, float rotRadians)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.7 - func.ST_Rotate

Rotates geometry rotRadians counter-clockwise about the origin point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Rotate(geometry geomA, float rotRadians);

PlaidCloud

func.ST_Rotate(geometry geomA, float rotRadians)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.1.8 - func.ST_Affine

Applies a 3D affine transformation to the geometry to do things like translate, rotate, scale in one step

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Affine(geometry geomA, float a, float b, float d, float e, float xoff, float yoff);

PlaidCloud

func.ST_Affine(geometry geomA, float a, float b, float d, float e, float xoff, float yoff)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.2 - Bounding Box Functions

12.2.1 - func.ST_ZMin

Returns the Z minima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ZMin(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_ZMin(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.2 - func.ST_ZMax

Returns the Z maxima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ZMax(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_ZMax(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.3 - func.ST_YMin

Returns the Y minima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_YMin(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_YMin(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.4 - func.ST_YMax

Returns the Y maxima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_YMax(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_YMax(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.5 - func.ST_XMin

Returns the X minima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_XMin(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_XMin(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.6 - func.ST_XMax

Returns the X maxima of a 2D or 3D bounding box or a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_XMax(box3d aGeomorBox2DorBox3D);

PlaidCloud

func.ST_XMax(box3d aGeomorBox2DorBox3D)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.7 - func.ST_3DMakeBox

Creates a BOX3D defined by the given two 3D point geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_3DMakeBox(geometry point3DLowLeftBottom, geometry point3DUpRightTop);

PlaidCloud

func.ST_3DMakeBox(geometry point3DLowLeftBottom, geometry point3DUpRightTop)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.8 - func.ST_MakeBox2D

Creates a BOX2D defined by the given two point geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakeBox2D(geometry pointLowLeft, geometry pointUpRight);

PlaidCloud

func.ST_MakeBox2D(geometry pointLowLeft, geometry pointUpRight)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.9 - func.ST_3DExtent

ST_3DExtent returns a box3d (includes Z coordinate) bounding box that encloses a set of geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_3DExtent(geometry set geomfield);

PlaidCloud

func.ST_3DExtent(geometry set geomfield)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.10 - func.ST_Extent

ST_Extent returns a bounding box that encloses a set of geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Extent(geometry set geomfield);

PlaidCloud

func.ST_Extent(geometry set geomfield)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.11 - func.ST_Expand

This function returns a bounding box expanded from the bounding box of the input

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Expand(geometry geom, float units_to_expand);

PlaidCloud

func.ST_Expand(geometry geom, float units_to_expand)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.12 - func.ST_EstimatedExtent

Return the 'estimated' extent of the given spatial table

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_EstimatedExtent(text table_name, text geocolumn_name);

PlaidCloud

func.ST_EstimatedExtent(text table_name, text geocolumn_name)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.13 - func.Box3D

Returns a BOX3D representing the 3D extent of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

Box3D(geometry geomA);

PlaidCloud

func.Box3D(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.2.14 - func.Box2D

Returns a BOX2D representing the 2D extent of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

Box2D(geometry geomA);

PlaidCloud

func.Box2D(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.3 - Clustering Functions

12.3.1 - func.ST_ClusterWithin

Aggregate function that returns an array of GeometryCollections that represent a set of geometries separated by a specified distance

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ClusterWithin(geometry set g, float8 distance);

PlaidCloud

func.ST_ClusterWithin(geometry set g, float8 distance)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.3.2 - func.ST_ClusterIntersecting

ClusterIntersecting is an aggregate function that returns an array of GeometryCollections that represent an interconnected set of geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ClusterIntersecting(geometry set g);

PlaidCloud

func.ST_ClusterIntersecting(geometry set g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4 - Geometry Accessors

12.4.1 - func.ST_Zmflag

Returns a code indicating the ZM coordinate dimension of a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Zmflag(geometry geomA);

PlaidCloud

func.ST_Zmflag(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.2 - func.ST_Z

Return the Z coordinate of the point, or NULL if not available. Input must be a point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Z(geometry a_point);

PlaidCloud

func.ST_Z(geometry a_point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.3 - func.ST_Y

Return the Y coordinate of the point, or NULL if not available. Input must be a point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Y(geometry a_point);

PlaidCloud

func.ST_Y(geometry a_point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.4 - func.ST_X

Return the X coordinate of the point, or NULL if not available. Input must be a point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_X(geometry a_point);

PlaidCloud

func.ST_X(geometry a_point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.5 - func.ST_Summary

Returns a text summary of the contents of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Summary(geometry g);

PlaidCloud

func.ST_Summary(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.6 - func.ST_StartPoint

Returns the first point of a LINESTRING or CIRCULARLINESTRING geometry as a POINT

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_StartPoint(geometry geomA);

PlaidCloud

ST_StartPoint(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.7 - func.ST_PointN

Return the Nth point in a single linestring or circular linestring in the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PointN(geometry a_linestring, integer n);

PlaidCloud

func.ST_PointN(geometry a_linestring, integer n)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.8 - func.ST_PatchN

Returns the 1-based Nth geometry (face) if the geometry is a POLYHEDRALSURFACE or POLYHEDRALSURFACEM

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PatchN(geometry geomA, integer n);

PlaidCloud

ST_PatchN(geometry geomA, integer n)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.9 - func.ST_NumPoints

Return the number of points in an ST_LineString or ST_CircularString value

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NumPoints(geometry g1);

PlaidCloud

func.ST_NumPoints(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.10 - func.ST_NumPatches

Return the number of faces on a Polyhedral Surface

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NumPatches(geometry g1);

PlaidCloud

func.ST_NumPatches(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.11 - func.ST_NumInteriorRing

This function returns the number of interior rings of a polygon geom. It returns NULL if the geom is not a polygon.

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NumInteriorRing(geometry a_polygon);

PlaidCloud

ST_NumInteriorRing(geometry a_polygon)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.12 - func.ST_NumInteriorRings

Return the number of interior rings of a polygon geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NumInteriorRings(geometry a_polygon);

PlaidCloud

func.ST_NumInteriorRings(geometry a_polygon)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.13 - func.ST_NumGeometries

Returns the number of Geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NumGeometries(geometry geom);

PlaidCloud

func.ST_NumGeometries(geometry geom)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.14 - func.ST_NRings

If the geometry is a polygon or multi-polygon returns the number of rings

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NRings(geometry geomA);

PlaidCloud

func.ST_NRings(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.15 - func.ST_NPoints

Return the number of points in a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NPoints(geometry g1);

PlaidCloud

func.ST_NPoints(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.16 - func.ST_NDims

Returns the coordinate dimension of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_NDims(geometry g1);

PlaidCloud

func.ST_NDims(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.17 - func.ST_MemSize

Returns the amount of memory space (in bytes) the geometry takes

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MemSize(geometry geomA);

PlaidCloud

func.ST_MemSize(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.18 - func.ST_M

Return the M coordinate of a Point, or NULL if not available. Input must be a Point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_M(geometry a_point);

PlaidCloud

func.ST_M(geometry a_point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.19 - func.ST_IsSimple

Returns true if this Geometry has no anomalous geometric points

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsSimple(geometry geomA);

PlaidCloud

func.ST_IsSimple(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.20 - func.ST_IsRing

Returns TRUE if this LINESTRING is both ST_IsClosed and ST_IsSimple (does not self intersect)

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsRing(geometry g);

PlaidCloud

func.ST_IsRing(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.21 - func.ST_IsCollection

Returns TRUE if the geometry type of the argument is a geometry collection type

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsCollection(geometry g);

PlaidCloud

func.ST_IsCollection(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.22 - func.ST_IsClosed

Returns TRUE if the LINESTRING's start and end points are coincident

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsClosed(geometry g);

PlaidCloud

func.ST_IsClosed(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.23 - func.ST_InteriorRingN

Returns the Nth interior linestring ring of the polygon geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_InteriorRingN(geometry a_polygon, integer n);

PlaidCloud

func.ST_InteriorRingN(geometry a_polygon, integer n)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.24 - func.ST_HasArc

Returns true if a geometry or geometry collection contains a circular string

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_HasArc(geometry geomA);

PlaidCloud

func.ST_HasArc(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.25 - func.ST_GeometryN

This section describes functions and operators for examining and manipulating string values

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeometryN(geometry geomA, integer n);

PlaidCloud

func.ST_GeometryN(geometry geomA, integer n)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.26 - func.ST_ExteriorRing

Returns a line string representing the exterior ring of the POLYGON geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ExteriorRing(geometry a_polygon);

PlaidCloud

func.ST_ExteriorRing(geometry a_polygon)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.27 - func.ST_Envelope

Returns the double-precision (float8) minimum bounding box for the supplied geometry, as a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Envelope(geometry g1);

PlaidCloud

func.ST_Envelope(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.28 - func.ST_BoundingDiagonal

Returns the diagonal of the supplied geometry's bounding box as a LineString

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_BoundingDiagonal(geometry geom, boolean fits=false);

PlaidCloud

func.ST_BoundingDiagonal(geometry geom, boolean fits=False)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.29 - func.ST_EndPoint

Returns the last point of a LINESTRING as a POINT. Returns NULL if the input is not a LINESTRING

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_EndPoint(geometry g);

PlaidCloud

func.ST_EndPoint(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.30 - func.ST_DumpRings

This is a set-returning function (SRF). It returns a set of geometry_dump rows, as an integer and a geometry, aliased "path" and "geom"

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_DumpRings(geometry a_polygon);

PlaidCloud

func.ST_DumpRings(geometry a_polygon)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.31 - func.ST_DumpPoints

This set-returning function (SRF) returns a set of geometry_dump rows formed by a geometry (geom) and an array of integers (path)

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_DumpPoints(geometry geom);

PlaidCloud

func.ST_DumpPoints(geometry geom)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.32 - func.ST_Dump

This is a set-returning function (SRF). It returns a set of geometry_dump rows, formed by a geometry (geom) and an array of integers (path)

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Dump(geometry g1);

PlaidCloud

func.ST_Dump(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.33 - func.ST_Dimension

Return the topological dimension of this Geometry object, which must be less than or equal to the coordinate dimension

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Dimension(geometry g);

PlaidCloud

func.ST_Dimension(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.34 - func.ST_CoordDim

Return the coordinate dimension of the ST_Geometry value

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_CoordDim(geometry geomA);

PlaidCloud

func.ST_CoordDim(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.35 - func.ST_Boundary

Returns the closure of the combinatorial boundary of this Geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Boundary(geometry geomA);

PlaidCloud

func.ST_Boundary(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.36 - func.ST_GeometryType

Returns the type of the geometry as a string. 'ST_LineString', 'ST_Polygon','ST_MultiPolygon'

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeometryType(geometry g1);

PlaidCloud

func.ST_GeometryType(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.4.37 - func.ST_IsEmpty

Returns true if this Geometry is an empty geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsEmpty(geometry geomA);

PlaidCloud

func.ST_IsEmpty(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5 - Geometry Constructors

12.5.1 - func.ST_Collect

Collects geometries into a geometry collection

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Collect(geometry g1, geometry g2)

PlaidCloud

func.ST_Collect(geometry g1, geometry g2)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.2 - func.ST_LineFromMultiPoint

Creates a LineString from a MultiPoint geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineFromMultiPoint(geometry aMultiPoint); PlaidCloud 

PlaidCloud

func.ST_LineFromMultiPoint(geometry aMultiPoint) 

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.3 - func.ST_MakeEnvelope

Creates a rectangular Polygon from the minimum and maximum values for X and Y

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakeEnvelope(float xmin, float ymin, float xmax, float ymax, integer srid=unknown);

PlaidCloud

func.ST_MakeEnvelope(float xmin, float ymin, float xmax, float ymax, integer srid=unknown); 

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.4 - func.ST_MakeLine

Creates a LineString containing the points of Point, MultiPoint, or LineString geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakeLine(geometry geom1, geometry geom2); 

PlaidCloud

func.ST_MakeLine(geometry geom1, geometry geom2)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.5 - func.ST_MakePoint

Creates a 2D, 3D Z or 4D ZM Point geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakePoint(float x, float y, float z, float m);

PlaidCloud

func.ST_MakePoint(float x, float y, float z, float m)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.6 - func.ST_MakePointM

Creates a point with X, Y and M (measure) coordinates

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakePointM(float x, float y, float m);

PlaidCloud

func.ST_MakePointM(float x, float y, float m)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.7 - func.ST_MakePolygon

Creates a Polygon formed by the given shell and optional array of holes

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakePolygon(geometry linestring);

PlaidCloud

func.ST_MakePolygon(geometry linestring)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.8 - func.ST_Point

Returns a Point with the given X and Y coordinate values

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Point(float x, float y);

PlaidCloud

func.ST_Point(float x, float y)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.5.9 - func.ST_Polygon

Returns a polygon built from the given LineString and sets the spatial reference system from the SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Polygon(geometry lineString, integer srid);

PlaidCloud

func.ST_Polygon(geometry lineString, integer srid)

External References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6 - Geometry Editors

12.6.1 - func.ST_SwapOrdinates

Returns a version of the given geometry with given ordinates swapped

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SwapOrdinates(geometry geom, cstring ords);

PlaidCloud

func.ST_SwapOrdinates(geometry geom, cstring ords)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.2 - func.ST_Snap

Snaps the vertices and segments of a geometry to another Geometry's vertices

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Snap(geometry input, geometry reference, float tolerance);

PlaidCloud

func.ST_Snap(geometry input, geometry reference, float tolerance)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.3 - func.ST_SnapToGrid

Snap all points of the input geometry to the grid defined by its origin and cell size

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SnapToGrid(geometry geomA, float originX, float originY, float sizeX, float sizeY);

PlaidCloud

func.ST_SnapToGrid(geometry geomA, float originX, float originY, float sizeX, float sizeY)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.4 - func.ST_ShiftLongitude

Reads every point/vertex in a geometry, and if the longitude coordinate is <0, adds 360 to it

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ShiftLongitude(geometry geom);

PlaidCloud

func.ST_ShiftLongitude(geometry geom)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.5 - func.ST_SetPoint

Replace point N of linestring with given point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SetPoint(geometry linestring, integer zerobasedposition, geometry point);

PlaidCloud

func.ST_SetPoint(geometry linestring, integer zerobasedposition,   
geometry point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.6 - func.ST_Segmentize

Returns a modified geometry having no segment longer than the given max_segment_length

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Segmentize(geometry geom, float max_segment_length);

PlaidCloud

func.ST_Segmentize(geometry geom, float max_segment_length)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.7 - func.ST_Reverse

Can be used on any geometry and reverses the order of the vertexes

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Reverse(geometry g1);

PlaidCloud

func.ST_Reverse(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.8 - func.ST_RemoveRepeatedPoints

Returns a version of the given geometry with duplicated points removed

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_RemoveRepeatedPoints(geometry geom, float8 tolerance);

PlaidCloud

func.ST_RemoveRepeatedPoints(geometry geom, float8 tolerance)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.9 - func.ST_RemovePoint

Remove a point from a linestring, given its 0-based index

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_RemovePoint(geometry linestring, integer offset);

PlaidCloud

func.ST_RemovePoint(geometry linestring, integer offset)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.10 - func.ST_Multi

Returns the geometry as a MULTI* geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Multi(geometry g1);

PlaidCloud

func.ST_Multi(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.11 - func.ST_LineToCurve

Converts plain LINESTRING/POLYGON to CIRCULAR STRINGs and Curved Polygons

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineToCurve(geometry geomANoncircular);

PlaidCloud

func.ST_LineToCurve(geometry geomANoncircular)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.12 - func.ST_LineMerge

Returns a (set of) LineString(s) formed by sewing together the constituent line work of a MULTILINESTRING

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineMerge(geometry amultilinestring);

PlaidCloud

func.ST_LineMerge(geometry amultilinestring)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.13 - func.ST_ForceCurve

Turns a geometry into its curved representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ForceCurve(geometry g);

PlaidCloud

func.ST_ForceCurve(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.14 - func.ST_ForceRHR

Forces the orientation of the vertices in a polygon to follow the area that is bounded by the polygon is to the right of the boundary

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ForceRHR(geometry g);

PlaidCloud

func.ST_ForceRHR(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.15 - func.ST_ForceSFS

This function supports Polyhedral surfaces, Triangles and Triangulated Irregular Network Surfaces

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ForceSFS(geometry geomA);

PlaidCloud

func.ST_ForceSFS(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.16 - func.ST_ForceCollection

Converts the geometry into a GEOMETRYCOLLECTION

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ForceCollection(geometry geomA);

PlaidCloud

func.ST_ForceCollection(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.17 - func.ST_Force4D

Forces the geometries into XYZM mode

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Force4D(geometry geomA, float Zvalue = 0.0, float Mvalue = 0.0);

PlaidCloud

ST_Force4D(geometry geomA, float Zvalue = 0.0, float Mvalue = 0.0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.18 - func.ST_Force3DM

Forces the geometries into XYM mode

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Force3DM(geometry geomA, float Mvalue = 0.0);

PlaidCloud

func.ST_Force3DM(geometry geomA, float Mvalue = 0.0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.19 - func.ST_Force3DZ

This function forces the geoms into XYZ mode. If a geom has no 'Z' compenent, then a 'Z coordinate' is automatically added

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Force3DZ(geometry geomA, float Zvalue = 0.0);

PlaidCloud

func.ST_Force3DZ(geometry geomA, float Zvalue = 0.0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.6.20 - func.ST_Force3D

Forces the geometries into XYZ mode

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Force3D(geometry geomA, float Zvalue = 0.0);

PlaidCloud

func.ST_Force3D(geometry geomA, float Zvalue = 0.0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.21 - func.ST_Force2D

Forces the geometries into a "2-dimensional mode" so that all output representations will only have the X and Y coordinates

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Force2D(geometry geomA);

PlaidCloud

func.ST_Force2D(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.22 - func.ST_FlipCoordinates

Returns a version of the given geometry with X and Y axis flipped

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_FlipCoordinates(geometry geom);

PlaidCloud

func.ST_FlipCoordinates(geometry geom)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.23 - func.ST_CurveToLine

Converts a CIRCULAR STRING to LINESTRING or CURVEPOLYGON to POLYGON or MULTISURFACE to MULTIPOLYGON

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_CurveToLine(geometry curveGeom, float tolerance, integer tolerance_type, integer flags);

PlaidCloud

func.ST_CurveToLine(geometry curveGeom, float tolerance, integer tolerance_type, integer flags)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.24 - func.ST_CollectionHomogenize

Given a geometry collection, returns the "simplest" representation of the contents

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_CollectionHomogenize(geometry collection);

PlaidCloud

func.ST_CollectionHomogenize(geometry collection)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.25 - func.ST_CollectionExtract

Given a geometry collection, return a homogeneous multi-geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_CollectionExtract(geometry collection);

PlaidCloud

func.ST_CollectionExtract(geometry collection)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.6.26 - func.ST_AddPoint

Adds a point to a LineString before point (0-based index)

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AddPoint(geometry linestring, geometry point);

PlaidCloud

func.ST_AddPoint(geometry linestring, geometry point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7 - Geometry Input

12.7.1 - func.ST_PointFromGeoHash

Return a point from a GeoHash string

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PointFromGeoHash(text geohash, integer precision=full_precision_of_geohash);

PlaidCloud

func.ST_PointFromGeoHash(text geohash, integer precision=full_precision_of_geohash)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.2 - func.ST_LineFromEncodedPolyline

Creates a LineString from an Encoded Polyline string

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineFromEncodedPolyline(text polyline, integer precision=5);

PlaidCloud

func.ST_LineFromEncodedPolyline(text polyline, integer precision=5)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.3 - func.ST_GMLToSQL

This method implements the SQL/MM specification. SQL-MM 3 5.1.50 (except for curves support)

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GMLToSQL(text geomgml);

PlaidCloud

func.ST_GMLToSQL(text geomgml)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.4 - func.ST_GeomFromKML

Constructs a PostGIS ST_Geometry object from the OGC KML representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromKML(text geomkml);

PlaidCloud

func.ST_GeomFromKML(text geomkml)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.5 - func.ST_GeomFromGeoJSON

Constructs a PostGIS geometry object from the GeoJSON representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromGeoJSON(text geomjson);

PlaidCloud

func.ST_GeomFromGeoJSON(text geomjson)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.6 - func.ST_GeomFromGML

Constructs a PostGIS ST_Geometry object from the OGC GML representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromGML(text geomgml);

PlaidCloud

func.ST_GeomFromGML(text geomgml)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.7 - func.ST_GeomFromGeoHash

Return a geometry from a GeoHash string.

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromGeoHash(text geohash, integer precision=full_precision_of_geohash);

PlaidCloud

func.ST_GeomFromGeoHash(text geohash, integer precision=full_precision_of_geohash)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.8 - func.ST_Box2dFromGeoHash

Return a BOX2D from a GeoHash string

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Box2dFromGeoHash(text geohash, integer precision=full_precision_of_geohash);

PlaidCloud

func.ST_Box2dFromGeoHash(text geohash, integer precision=full_precision_of_geohash)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.9 - func.ST_WKBToSQL

This method implements the SQL/MM specification. SQL-MM 3 5.1.36

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_WKBToSQL(bytea WKB);

PlaidCloud

func.ST_WKBToSQL(bytea WKB)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.10 - func.ST_PointFromWKB

This function, takes a binary representation of geometry and a (SRID) and creates the appropriate geometry type - POINT GEOMETRY

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PointFromWKB(bytea wkb);

PlaidCloud

func.ST_PointFromWKB(bytea wkb); 

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.7.11 - func.ST_LinestringFromWKB

This function, takes a binary representation of geometry and a (SRID) and creates the appropriate geometry type -LINESTRING GEOMETRY

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LinestringFromWKB(bytea WKB);

PlaidCloud

func.ST_LinestringFromWKB(bytea WKB);

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.12 - func.ST_LineFromWKB

This function, takes a binary representation of geometry and a Spatial Reference System ID (SRID) and creates the appropriate geometry type

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineFromWKB(bytea WKB)  

PlaidCloud

func.ST_LineFromWKB(bytea WKB)  

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.13 - func.ST_GeomFromWKB

This function takes a binary representation of a geometry and a Spatial Reference System ID (SRID) and creates the appropriate geometry type

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromWKB(bytea geom);

PlaidCloud

func.ST_GeomFromWKB(bytea geom);

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.14 - func.ST_GeomFromEWKB

Constructs a PostGIS ST_Geometry object from the OGC Extended Well-Known binary (EWKT) representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromEWKB(bytea EWKB);

PlaidCloud

func.ST_GeomFromEWKB(bytea EWKB)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.15 - func.ST_GeogFromWKB

This function, takes a well-known binary representation (WKB) of a geometry and creates an instance of the appropriate geography type

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeogFromWKB(bytea wkb);

PlaidCloud

func.ST_GeogFromWKB(bytea wkb)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.16 - func.ST_WKTToSQL

This method implements the SQL/MM specification. SQL-MM 3 5.1.34

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_WKTToSQL(text WKT);

PlaidCloud

func.ST_WKTToSQL(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.17 - func.ST_PolygonFromText

Makes a Polygon Geometry from WKT with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PolygonFromText(text WKT);

PlaidCloud

func.ST_PolygonFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.18 - func.ST_PointFromText

Constructs a PostGIS ST_Geometry point object from the OGC Well-Known text representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PointFromText(text WKT);

PlaidCloud

func.ST_PointFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.19 - func.ST_MPolyFromText

Makes a MultiPolygon from WKT with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MPolyFromText(text WKT);

PlaidCloud

func.ST_MPolyFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.20 - func.ST_MPointFromText

Makes a Multi-Point Geometry from WKT with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MPointFromText(text WKT);

PlaidCloud

func.ST_MPointFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.21 - func.ST_MLineFromText

Makes a Multi-Line Geometry from Well-Known-Text (WKT) with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MLineFromText(text WKT);

PlaidCloud

func.ST_MLineFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.22 - func.ST_LineFromText

Makes a Linestring Geometry from WKT with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_LineFromText(text WKT);

PlaidCloud

func.ST_LineFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.23 - func.ST_GeomFromText

Constructs a PostGIS ST_Geometry object from the OGC Well-Known text representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromText(text WKT);

PlaidCloud

func.ST_GeomFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.24 - func.ST_GeometryFromText

This method implements the SQL/MM specification and OpenGIS simple features implementation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeometryFromText(text WKT);

PlaidCloud

func.ST_GeometryFromText(text WKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.25 - func.ST_GeomFromEWKT

Constructs a PostGIS ST_Geometry object from the OGC Extended Well-Known text (EWKT) representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomFromEWKT(text EWKT);

PlaidCloud

func.ST_GeomFromEWKT(text EWKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.26 - func.ST_GeomCollFromText

Makes a collection Geometry from the Well-Known-Text (WKT) representation with the given SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeomCollFromText(text WKT, integer srid);

PlaidCloud

func.ST_GeomCollFromText(text WKT, integer srid)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.27 - func.ST_GeographyFromText

Returns a geography object from the well-known text representation. SRID 4326 is assumed if unspecified

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeographyFromText(text EWKT);

PlaidCloud

func.ST_GeographyFromText(text EWKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.28 - func.ST_GeogFromText

Returns a geography object from the well-known text or extended well-known representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeogFromText(text EWKT);

PlaidCloud

func.ST_GeogFromText(text EWKT)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.29 - func.ST_BdMPolyFromText

Construct a Polygon given an arbitrary collection of closed linestrings, polygons, MultiLineStrings as Well-Known text representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_BdMPolyFromText(text WKT, integer srid);

PlaidCloud

func.ST_BdMPolyFromText(text WKT, integer srid)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.30 - func.ST_BdPolyFromText

Construct a Polygon given an arbitrary collection of closed linestrings as a MultiLineString Well-Known text representation

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_BdPolyFromText(text WKT, integer srid);

PlaidCloud

func.ST_BdPolyFromText(text WKT, integer srid)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.7.31 - func.GeometryType

Returns the type of the geometry as a string

Syntax

func.GeometryType()

Examples

Documentation for func.GeometryType is coming soon.

References

PostgreSQL Documentation

12.8 - Geometry Output

12.8.1 - func.ST_GeoHash

Return a GeoHash representation (http://en.wikipedia.org/wiki/Geohash) of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_GeoHash(geometry geom, integer maxchars=full_precision_of_point);

PlaidCloud

func.ST_GeoHash(geometry geom, integer maxchars=full_precision_of_point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.2 - func.ST_AsX3D

Returns a geometry as an X3D xml formatted node element

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsX3D(geometry g1, integer maxdecimaldigits=15, integer options=0);

PlaidCloud

func.ST_AsX3D(geometry g1, integer maxdecimaldigits=15, integer options=0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.3 - func.ST_AsSVG

Return the geometry as Scalar Vector Graphics (SVG) path data

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsSVG(geometry geom, integer rel=0, integer maxdecimaldigits=15);

PlaidCloud

func.ST_AsSVG(geometry geom, integer rel=0, integer maxdecimaldigits=15)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.4 - func.ST_AsTWKB

Returns the geometry in TWKB (Tiny Well-Known Binary) format

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsTWKB(geometry g1, integer decimaldigits_xy=0, integer decimaldigits_z=0, integer decimaldigits_m=0, boolean include_sizes=false, boolean include_bounding boxes=false);

PlaidCloud

func.ST_AsTWKB(geometry g1, integer decimaldigits_xy=0, integer decimaldigits_z=0, integer decimaldigits_m=0, boolean include_sizes=false, boolean include_bounding boxes=false)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.5 - func.ST_AsLatLonText

Returns the Degrees, Minutes, Seconds representation of the point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsLatLonText(geometry pt, text format='');

PlaidCloud

func.ST_AsLatLonText(geometry pt, text format='')

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.6 - func.ST_AsKML

Return the geometry as a Keyhole Markup Language (KML) element

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsKML(geometry geom, integer maxdecimaldigits=15, text nprefix=NULL);

PlaidCloud

func.ST_AsKML(geometry geom, integer maxdecimaldigits=15, text nprefix=NULL)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.7 - func.ST_AsGML

Return the geometry as a Geography Markup Language (GML) element

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsGML(geometry geom, integer maxdecimaldigits=15, integer options=0);

PlaidCloud

func.ST_AsGML(geometry geom, integer maxdecimaldigits=15, integer options=0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.8 - func.ST_AsGeoJSON

Return the geometry as a GeoJSON "geometry" object, or the row as a GeoJSON "feature" object

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsGeoJSON(geography geog, integer maxdecimaldigits=9, integer options=0);

PlaidCloud

func.ST_AsGeoJSON(geography geog, integer maxdecimaldigits=9, integer options=0)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.9 - func.ST_AsEncodedPolyline

Returns the geometry as an Encoded Polyline

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsEncodedPolyline(geometry geom, integer precision=5);

PlaidCloud

func.ST_AsEncodedPolyline(geometry geom, integer precision=5)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.10 - func.ST_AsHEXEWKB

Returns a Geometry in HEXEWKB format (as text) using either little-endian (NDR) or big-endian (XDR) encoding

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsHEXEWKB(geometry g1);

PlaidCloud

func.ST_AsHEXEWKB(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.11 - func.ST_AsEWKB

Returns the Well-Known Binary representation of the geometry with SRID metadata

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsEWKB(geometry g1);

PlaidCloud

func.ST_AsEWKB(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.12 - func.ST_AsBinary

Returns the Well-Known Binary representation of the geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsBinary(geometry g1);

PlaidCloud

func.ST_AsBinary(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.13 - func.ST_AsText

Returns the Well-Known Text representation of the geometry/geography

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsText(geometry g1);

PlaidCloud

func.ST_AsText(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.8.14 - func.ST_AsEWKT

Returns the Well-Known Text representation of the geometry prefixed with the SRID

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AsEWKT(geometry g1);

PlaidCloud

func.ST_AsEWKT(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.9 - Geometry Processing

12.9.1 - func.ST_SetEffectiveArea

Sets the effective area for each vertex, using the Visvalingam-Whyatt algorithm

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SetEffectiveArea(geometry geomA, float threshold = 0, integer set_area = 1);

PlaidCloud

func.ST_SetEffectiveArea(geometry geomA, float threshold = 0, integer set_area = 1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.2 - func.ST_SimplifyVW

Returns a "simplified" version of the given geometry using the Visvalingam-Whyatt algorithm

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SimplifyVW(geometry geomA, float tolerance);

PlaidCloud

func.ST_SimplifyVW(geometry geomA, float tolerance)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.3 - func.ST_SimplifyPreserveTopology

Simplifies a geometry by removing points that would fall within a specified distance tolerance.

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SimplifyPreserveTopology(geometry geomA, float tolerance);

PlaidCloud

func.ST_SimplifyPreserveTopology(geometry geomA, float tolerance)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.4 - func.ST_Simplify

Returns a "simplified" version of the given geometry using the Douglas-Peucker algorithm

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Simplify(geometry geomA, float tolerance, boolean preserveCollapsed);

PlaidCloud

func.ST_Simplify(geometry geomA, float tolerance, boolean preserveCollapsed)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.5 - func.ST_SharedPaths

Returns a collection containing paths shared by the two input geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_SharedPaths(geometry lineal1, geometry lineal2);

PlaidCloud

func.ST_SharedPaths(geometry lineal1, geometry lineal2)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.6 - func.ST_Polygonize

Creates a GeometryCollection containing the polygons formed by the constituent linework of a set of geometries

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Polygonize(geometry set geomfield);

PlaidCloud

func.ST_Polygonize(geometry set geomfield)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.7 - func.ST_PointOnSurface

Returns a POINT guaranteed to intersect a surface

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_PointOnSurface(geometry g1);

PlaidCloud

func.ST_PointOnSurface(geometry g1)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.8 - func.ST_OffsetCurve

Return an offset line at a given distance and side from an input line

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_OffsetCurve(geometry line, float signed_distance, text style_parameters='');

PlaidCloud

func.ST_OffsetCurve(geometry line, float signed_distance, text style_parameters='')

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.9 - func.ST_MinimumBoundingCircle

Returns the smallest circle polygon that contains a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MinimumBoundingCircle(geometry geomA, integer num_segs_per_qt_circ=48);

PlaidCloud

func.ST_MinimumBoundingCircle(geometry geomA, integer num_segs_per_qt_circ=48);

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.10 - func.ST_DelaunayTriangles

Return the Delaunay triangulation of the vertices of the input geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_DelaunayTriangles(geometry g1, float tolerance, int4 flags);

PlaidCloud

func.ST_DelaunayTriangles(geometry g1, float tolerance, int4 flags)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.11 - func.ST_ConvexHull

Computes the convex hull of a geometry. The convex hull is the smallest convex geometry that encloses all geometries in the input

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ConvexHull(geometry geomA);

PlaidCloud

func.ST_ConvexHull(geometry geomA)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.12 - func.ST_ConcaveHull

The concave hull of a geometry represents a possibly concave geometry that encloses the input geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_ConcaveHull(geometry geom, float target_percent, boolean allow_holes = false);

PlaidCloud

func.ST_ConcaveHull(geometry geom, float target_percent, boolean allow_holes = false)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.13 - func.ST_Centroid

Computes a point which is the geometric center of mass of a geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Centroid(geometry g1); 

PlaidCloud

func.ST_Centroid(geometry g1); 

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.14 - func.ST_BuildArea

Creates an areal geometry formed by the constituent linework of the input geometry

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_BuildArea(geometry geom);

PlaidCloud

func.ST_BuildArea(geometry geom)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.15 - func.ST_Buffer

Returns a geometry/geography that represents all points whose distance from this Geometry/geography is less than or equal to distance

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_Buffer(geometry g1, float radius_of_buffer, text buffer_style_parameters = '');

PlaidCloud

func.ST_Buffer(geometry g1, float radius_of_buffer, text buffer_style_parameters = '')

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.9.16 - func.St_Accum

Aggregate. Constructs an array of geometries

Syntax

func.ST_Accum()

Examples

Documentation for func.ST_Accum is coming soon.

References

PostgreSQL Documentation

12.10 - Geometry Validation

12.10.1 - func.ST_MakeValid

The function attempts to create a valid representation of a given invalid geometry without losing any of the input vertices

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_MakeValid(geometry input);

PlaidCloud

func.ST_MakeValid(geometry input)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.10.2 - func.ST_IsValidReason

Returns text stating if a geometry is valid or not an if not valid, a reason why

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsValidReason(geometry geomA, integer flags);

PlaidCloud

func.ST_IsValidReason(geometry geomA, integer flags)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.10.3 - func.ST_IsValidDetail

Returns a valid_detail row, formed by a boolean (valid) stating whether the geometry is valid or invalid

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsValidDetail(geometry geom, integer flags);

PlaidCloud

func.ST_IsValidDetail(geometry geom, integer flags)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.10.4 - func.ST_IsValid

Test if an ST_Geometry value is well-formed in 2D according to the OGC rules

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_IsValid(geometry g);

PlaidCloud

func.ST_IsValid(geometry g)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation

12.11 - Linear Referencing

12.11.1 - func.ST_AddMeasure

Return a derived geometry with measure elements linearly interpolated between the start and end points

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_AddMeasure(geometry geom_mline, float8 measure_start, float8 measure_end);

PlaidCloud

func.ST_AddMeasure(geometry geom_mline, float8 measure_start, float8 measure_end)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.11.2 - func.ST_InterpolatePoint

Return the value of the measure dimension of a geometry at the point closed to the provided point

Description

PlaidCloud expressions and filters provide use of most non-administrative PostGIS methods. PostGIS methods are accessed by prefixing the standard method name with func..

Examples

SQL

ST_InterpolatePoint(geometry line, geometry point);

PlaidCloud

func.ST_InterpolatePoint(geometry line, geometry point)

References

PostGIS Official Documentation for this method can be found here.

Additional capabilities and usage examples can be found in the PostGIS documentation.

12.11.3 - func.ST_LocateBetweenElevations

Return a derived geometry (collection) value with elements that intersect the specified range of elev