Skip to content

Prophecy Conversion Matrix

PlaidCloud converts Prophecy Spark pipelines and SQL projects into native Advanced workflows. The importer maps each Prophecy gem to a workflow step, project variable, or canvas object, so a converted workflow runs cloud-native rather than as an emulation.

Every gem, connector and project structure converts: Full means the gem becomes a native PlaidCloud step, variable, or canvas object that does the gem’s work. The few behaviours a converted workflow does not reproduce on its own are listed under Known Edge Cases, each with what to do.

To run a conversion, see Migrate Prophecy Projects.

Gem Status PlaidCloud Equivalent Converts
Lookup Full Lookup Each lookup call becomes a Lookup step in the step that uses it; a key with no match reads NULL, and a duplicated key returns one of its rows.
Source Full Import step chosen by the dataset format (see Connectors) Each Source becomes an import step into a table named after the dataset; a format the converter does not recognise is read as a CSV file uploaded to the workflow’s document folder, with a warning.
Streaming Source Full Same import as batch Source Streaming sources convert to a one-shot batch import with a warning.
Streaming Target Full Batch append load Streaming targets convert to a batch append with a warning.
Target Full Table export / load step chosen by dataset format and writeMode Each Target becomes a table load or export step in the gem’s write mode: append, overwrite, merge with its SET expressions, upsert, SCD2, replaceWhere and insertInto.
Gem Status PlaidCloud Equivalent Converts
Data Cleansing Full Column expressions in a Table Extract step Converts null fill, trim and case options into column expressions; cleaned columns move last.
Sequence Full Table Extract with a row number expression Adds a sequence column from the start value and increment.
Surrogate Key Generator Full Table Extract with a row number expression Adds a sequential key from the initial value; keys continue across runs from a table that holds the last key issued.
Gem Status PlaidCloud Equivalent Converts
Aggregate Full Table Extract with aggregation; Pivot when the gem pivots Groups and aggregates, applies the HAVING condition, and pivots into one column per pivot value; propagate-all carries every other input column.
Bulk Column Expressions Full Expand to one expression per column in a Table Extract step One expression per listed column in a Table Extract step; the conversion warns that the gem’s layout is assumed.
Bulk Column Rename Full Column rename in a Table Extract step One rename per listed column in a Table Extract step; the conversion warns that the gem’s layout is assumed.
Deduplicate Full Table Extract with distinct or a row-number filter Keeps one row per key: distinct, first or last by your sort order, any, or only the keys that occur once.
Dynamic Replace Full Dynamic replace transform, configured by hand Converts to replacement rules on the dynamic replace transform; the conversion warns that the gem’s layout is assumed.
Dynamic Select Full Column select in a Table Extract step resolved by type Resolves columns by type at conversion time into an explicit select.
Filter Full Table Extract with a filter expression Converts to a Table Extract step with a filter expression; NULL conditions drop the row.
Flatten Schema Full JSON Parse and Table Extract Flattens nested fields into columns named with the gem’s delimiter and explodes arrays into one row per element.
Fuzzy Match Full Table Extract and User Defined Transform Pairs the records, scores each match field by the gem’s method, averages the scores, and keeps the pairs at or above your threshold.
Group By Rollup Full Union of Table Extract aggregation steps per rollup level Expands the rollup into one aggregate per grouping level unioned together with NULL subtotals.
Limit Full Table Extract with a row limit Converts to a row limit; config references become project variables.
Normalize Full Table Extract with an explode expression Repeats each record by the length expression, computes the output columns per index, and stops at the first index where the finished condition holds.
Order By Full Sort on a Table Extract step Converts to a sort with the same keys, direction and null placement.
Reformat Full Table Extract with column expressions Converts to a Table Extract step with one expression per output column; unlisted columns are dropped.
Sample Rows Full Table Extract with a row limit or random sample Keeps a random fraction or count of rows; the rows chosen differ from the Spark run because sampling is random.
Schema Transform Full Chain of Table Extract steps applied in order Applies each operation in order: add or replace a column, drop, rename, add if missing with its default, and add a project business rule.
Set Operation Full Union All; Anti Join for except; Inner Join for intersect Union all stacks the inputs; except all and intersect all keep the right number of duplicate rows.
TransposeAgg Full Table Extract with aggregation Converts the gem’s grouping, aggregates and pivot as an Aggregate, with a warning to review it.
Unpivot Full Melt Stacks the selected columns into name and value columns.
Watermark Full No step needed Dropped: a batch run reads all the data present, so there is no late-data cutoff to apply.
Window Function Full Window expressions in a Table Extract step Adds window columns with the gem’s partition, order and frame, rows or range, with unbounded, current-row or offset bounds.
Gem Status PlaidCloud Equivalent Converts
Compare Columns Full Full outer join plus per-column aggregates Compares the columns through a full outer join of the unpivoted data, with match and mismatch counts.
Join Full Inner Join, Multi-Table Join, Anti Join or Cross Join Builds the join of each type, inner, left, right, full, semi, anti, cross and natural, on any join condition, applies the where clause, and writes the select list.
Repartition Full No step needed Dropped: PlaidCloud manages partitioning, so the next step reads the upstream table directly.
Row Distributor Full One Table Extract filter per output Converts to one filter per output; rows can land in multiple outputs.
Gem Status PlaidCloud Equivalent Converts
Column Parser Full JSON Parse / XML Parse Converts to a JSON or XML parse of the named column; the conversion warns that the gem’s layout is assumed.
Gem Status PlaidCloud Equivalent Converts
Custom Gem (Gem Builder) Full User Defined Transform running the gem’s generated code The gem’s generated PySpark code from the pipeline export runs in a Python step, with a warning to review it.
CustomReformatBitbucket Full Table Extract Converts as the Reformat whose expressions the gem carries, with a warning to review it.
Data Quality Check Full Union All of one result row per check; Raise Workflow Error to stop on failures Evaluates each check over the input and writes one result row per check with its status and message. The data passes through, and the run stops when the gem’s failure threshold is reached.
Deequ Full Table Extract checks Passes data through and builds the constraint results table for the isComplete, isUnique, isNonNegative and isContainedIn rules.
Delta Table Operations Full Table In-Place Delete, Table Update, Table Drop, Table Restore to Timestamp Delete, update, drop and restore to a timestamp map to table steps; optimize, vacuum and restore by version pass data through with a warning.
Directory Full Directory Listing Lists the files of the bound document folder in the gem’s columns; with the sheet-name option, each workbook gives one row per sheet. A missing folder gives zero rows.
Email Full Notify Via Email Sends the email, with an optional data attachment, through a bound email connection.
Email Data Full Notify Via Email Emails the input rows as a table and passes the input through unchanged.
File Operation Full Copy Document File or Copy Document Directory Copies, moves or syncs files from the bound document folder into the converted workflow’s document folder; a move removes the source files.
Log Full Table Extract Writes the gem’s event row (time, gem name, event type and text) to its output table.
REST API Enrich Full REST Request Calls the API once per row and adds the response columns.
Script Full Python step with frame-access rewrite Embeds the script body in a python step, with data-frame access rewritten; Scala bodies carried with a note.
Soda Data Quality Check Full Table Extract checks Builds the check results table from the Soda checks.
SQL Statement Full SQL step per output tab, text translated to the tenant dialect Embeds each tab as a SQL step with inputs bound as temp tables; translated to the tenant dialect.
Todo Full Raise Workflow Error or Notify via Log Marks logic still to be written: a log message when nothing reads it, or a step that stops the run with the gem’s message when a later step depends on it.
Gem Status PlaidCloud Equivalent Converts
Tableau Full Publish to Tableau Writes a Hyper extract and publishes it to the bound Tableau connection.
Gem Status PlaidCloud Equivalent Converts
Bedrock Full Text NLP AI Embeds the text column through the workspace AI connection.
OpenAI Full Text NLP AI Answers questions or embeds text through the workspace AI connection.
Pinecone Lookup Full REST Request Queries the Pinecone index for each row and adds pinecone_matches and pinecone_error columns.
SageMaker Endpoint Full Text NLP AI Answers questions through the workspace AI connection.
Text Processing Full Table Extract for chunk splitting; REST Request for web scraping Chunk splitting becomes a character-chunk expression; web scraping fetches each row’s URL and joins the page content back onto the row.
Gem Status PlaidCloud Equivalent Converts
Control Flow Input Full Dropped (boundary marker only) Dropped; it only marks the iterator sub-workflow input boundary.
Control Flow Output Full Dropped (boundary marker only) Dropped; it only marks the iterator sub-workflow output boundary.
Subgraph (Basic) Full Canvas container / nested workflow from inline processes Converts the inline inner graph into a nested workflow (or inlined steps) with in/out table parameters.
Table Iterator Full Workflow Loop Runs the loop body once per input row, setting each configuration value from its column, up to maxIteration.
While Iterator Full Workflow Loop Runs the loop body once per pass, passing the iteration number, until the body returns no rows or the gem’s maximum is reached.
Gem Status PlaidCloud Equivalent Converts
External table Source/Target (BigQuery, Databricks, SAP HANA, MongoDB, MSSQL, Oracle, Postgres, Redshift, Salesforce, Snowflake, Synapse) Full External Data import/export step with the matching database connection Reads or writes a table in an external database through a PlaidCloud connection.
File storage Source/Target (ADLS, Databricks Volumes, GCS, OneDrive, S3, SFTP, SharePoint, Smartsheet) Full Import/Export file steps against a document account (ADLS/GCS/S3/SFTP/SharePoint/OneDrive) Reads or writes files in cloud/SFTP storage with wildcard union, trigger, delete/move and encryption options.
Model Full Upstream step output table Reads another model as the table produced by the referenced upstream step.
Seed Full Import: CSV Seed CSV is imported into a PlaidCloud table.
Source Full Import step chosen by the dataset format (see Connectors) Each Source becomes an import step into a table named after the dataset; a format the converter does not recognise is read as a CSV file uploaded to the workflow’s document folder, with a warning.
Target Model / Table (Target) Full Table Extract, Upsert, Copy Table and Union All Writes the model’s final select to a PlaidCloud table as replace, append, upsert, delete-and-insert, SCD2 or replace by predicate.
Upload Files Full Import: CSV Uploaded local files become a table via an import step.
Gem Status PlaidCloud Equivalent Converts
Data Cleansing Full Column expressions in a Table Extract step Converts null fill, trim and case options into column expressions; cleaned columns move last.
Data Masking Full Column expressions for character masking or hashing Masks or hashes selected string columns in place or as new columns.
Find Duplicates Full Row-number window expression plus filter Returns unique, duplicate or count/position-selected rows per key.
Generate Rows Full Sequence table joined and filtered per input row Generates rows per input row from init, condition and loop expressions.
Imputation Full Aggregate plus join-back and column expression Replaces nulls or sentinel values with average, median, mode or a fixed value.
Multi-Column Edit Full Per-column expressions expanded from the template Applies one expression template to every selected column.
Multi-Column Rename Full Column rename mapping on a select step Renames selected columns by prefix, suffix or name expression.
Record ID Full Row-number or uuid column expression Adds a UUID or sequential record id column.
Sample Full Row limit or sample option on extract Takes first/last/random/Nth/percent rows overall or per group.
Tile Full Window expressions for tile number and sequence Assigns tile numbers by equal sum, count, std-dev bands, distinct values or manual cutoffs.
Gem Status PlaidCloud Equivalent Converts
Aggregate Full Table Extract with aggregation; Pivot when the gem pivots Groups and aggregates, applies the HAVING condition, and pivots into one column per pivot value; propagate-all carries every other input column.
Count Records Full Aggregate step with count functions Returns total, non-null or distinct counts as one row.
Data Encoder/Decoder Full Column expressions for base64/hex encode and decode Encodes or decodes selected columns; AES only on Databricks.
Deduplicate Full Table Extract with distinct or a row-number filter Keeps one row per key: distinct, first or last by your sort order, any, or only the keys that occur once.
Dynamic Select Full Column select in a Table Extract step resolved by type Resolves columns by type at conversion time into an explicit select.
Filter Full Table Extract with a filter expression Converts to a Table Extract step with a filter expression; NULL conditions drop the row.
Finance Full Column expression implementing the financial function Appends a financial function result (FV, PV, NPV, IRR, …) as a new column.
Flatten Schema Full JSON Parse and Table Extract Flattens nested fields into columns named with the gem’s delimiter and explodes arrays into one row per element.
Fuzzy Match Full Table Extract and User Defined Transform Pairs the records, scores each match field by the gem’s method, averages the scores, and keeps the pairs at or above your threshold.
Limit Full Table Extract with a row limit Converts to a row limit; config references become project variables.
Order By Full Sort on a Table Extract step Converts to a sort with the same keys, direction and null placement.
Pivot Full Pivot Turns the selected pivot values into columns of aggregates; a Simple-mode value list fetched at design time is converted as the listed values.
Reformat Full Table Extract with column expressions Converts to a Table Extract step with one expression per output column; unlisted columns are dropped.
Running Total Full Cumulative-sum window expression column Appends cumulative sums per selected column.
Snowflake Unpivot (third-party) Full Melt Unpivots columns via Snowflake UNPIVOT.
Transpose Full Melt Stacks data columns into name and string value columns.
TransposeAgg Full Table Extract with aggregation Converts the gem’s grouping, aggregates and pivot as an Aggregate, with a warning to review it.
Unpivot Full Melt Stacks data columns into name and value columns.
Weighted Average Full Aggregate step with weighted-average expression Computes a weighted mean overall or per group.
Window Function Full Window expressions in a Table Extract step Adds window columns with the gem’s partition, order and frame, rows or range, with unbounded, current-row or offset bounds.
Gem Status PlaidCloud Equivalent Converts
Except Full Anti Join on all columns plus a distinct Table Extract Keeps rows of the first input that appear in none of the other inputs.
Intersect Full Inner Join on all columns plus Distinct Keeps rows present in every input.
Join Full Inner Join, Multi-Table Join, Anti Join or Cross Join Builds the join of each type, inner, left, right, full, semi, anti, cross and natural, on any join condition, applies the where clause, and writes the select list.
Union Full Union All Stacks the inputs by position, removing duplicates when the gem does not keep them.
Union By Name Full Union All Becomes a Union All that aligns columns by name and null-fills missing ones.
Gem Status PlaidCloud Equivalent Converts
JSON Parse Full JSON Parse Parses a JSON text column into structured fields.
Regex Full Regex expressions in a Table Extract / Transform step Replace, parse, match or tokenize a text column with a regex.
Text To Columns Full Split-column / explode steps with regex delimiter Splits a delimited column into columns or rows using a regex delimiter.
XML Parse Full XML Parse Parses an XML text column into structured fields.
Gem Status PlaidCloud Equivalent Converts
Directory Full Directory Listing Lists the files of the bound document folder in the gem’s columns; with the sheet-name option, each workbook gives one row per sheet. A missing folder gives zero rows.
DynamicInput Full SQL step for query templates; workflow loop for multi-file mode Query-template mode embeds the template as a SQL step; multi-file mode maps with assumptions.
Email Full Notify Via Email Sends the email, with an optional data attachment, through a bound email connection.
RestAPI Full REST Request Sends one request per input row with your URL, headers, query, body and retries, keeps the response, and polls an asynchronous request until it is done. Credentials come from the connection you bind.
Script Full Python step with frame-access rewrite Embeds the script body in a python step, with data-frame access rewritten; Scala bodies carried with a note.
SQL Statement Full SQL step per output tab, text translated to the tenant dialect Embeds each tab as a SQL step with inputs bound as temp tables; translated to the tenant dialect.
StoredProcedure Full Agent Remote Execution of SQL Calls the procedure on its warehouse through the connection you choose: once per input row with each argument bound from its column, or once when it takes no arguments. The input columns pass through.
Table Operations Full Table In-Place Delete, Table Update, Table Drop Update, delete and drop map to table steps; vacuum and optimize become table optimize or a no-op with a warning.
ToDo Full Raise Workflow Error or Notify via Log Marks logic still to be written: a log message when nothing reads it, or a step that stops the run with the gem’s message when a later step depends on it.
Gem Status PlaidCloud Equivalent Converts
Buffer Full Spatial Buffer Grows each geometry by the configured distance in miles or kilometers.
CreatePoint Full Table Extract with geom_point Builds point geometry from longitude and latitude columns.
Distance Full Table Extract Computes distance, cardinal direction and bearing between two WKT points.
FindNearest Full Table Extract with a cross join Finds N nearest target points for each source point within a max distance.
HeatMap Full Heat Map (macro) Bins points into H3 hexagons at the gem’s resolution, spreads the heat over the grid distance with its decay, and gives each hexagon’s density and outline.
PolyBuild Full Table Extract and a Python step Builds polygon or polyline WKT from ordered points per group.
Simplify Full Spatial Generalize Simplifies WKT geometries by a tolerance in miles/kms.
SpatialMatch Full Spatial Match Matches source and target geometries by intersects/contains/within/touches/envelope.
Gem Status PlaidCloud Equivalent Converts
PowerBIWrite Full REST Request Creates the Power BI dataset and pushes the rows through the Power BI REST API.
TableauWrite Full Publish to Tableau Writes a Hyper extract and publishes it to the bound Tableau connection.
Visualize Full Copy Table Passes the rows on unchanged; build the chart as a PlaidCloud dashboard on the same table.
Gem Status PlaidCloud Equivalent Converts
Condition Full Cross Join and Table Extract Evaluates the conditions over the whole input and sends every row to the first output whose condition holds, else to the last output.
Macro Full SQL step with the macro body expanded Embeds the macro body as a SQL step translated to the tenant dialect, inputs bound as temp tables.
Pipeline (trigger) Full Workflow Loop Runs the named pipeline once per input row with your parameters and trigger condition, continues past a failed run, and returns one row per run.
Subgraph (Basic) Full Canvas container / nested workflow from inline processes Converts the inline inner graph into a nested workflow (or inlined steps) with in/out table parameters.

Dataset formats and connectors are source and target settings of a gem, not gems. Each format converts to the import or export step shown, and conversion never contacts the source or target system.

Format Status PlaidCloud Equivalent Converts
Avro Full Import: Avro; Export: Avro Reads and writes Avro files.
BigQuery Full External database connection with a table import Reads and writes through a connection.
Binary files Full Directory Listing with file contents; Alteryx Executor to write Reads each file of the folder with its path, modification time, length and content; writes each row’s content to the file its path names.
Catalog table Full Existing PlaidCloud table binding Binds to a table by name; reads and every write mode convert.
Cosmos DB Full Import and export: REST Request Reads and writes the container’s documents through your Cosmos DB connection.
CSV Full Import: CSV; Export: CSV Reads and writes delimited files with header, separator, quote, escape, and encoding options.
Data Generator Full Table Faker Generates the dataset’s columns with the same kinds of fake data.
DataHub table Full Import SQL Reads the table the dataset names through your connection; a write lands in a PlaidCloud table.
Delta Full Existing PlaidCloud table binding Reads and writes the table in every write mode; a read of an earlier version reads it through the Databricks connection you bind.
Excel (XLSX) Full Import: Excel; Export: Excel Reads and writes workbook sheets and ranges.
Fixed format Full Import: Fixed Width; Export: CSV Reads fixed-width files laid out from the dataset’s column widths; writes a delimited file.
Hive Full Import: Import SQL; Export: External Tables Reads and writes the table through your Hive connection.
Iceberg Full Existing PlaidCloud table binding Reads and writes the table.
JDBC Full External database connection with a table import Reads a table or query and writes in replace or append mode through your connection; a post-write statement runs after the write.
JSON Full JSON Parse Loads records and flattens nested fields.
Kafka Full Import: Import Kafka; Export: Export Kafka Reads a topic as one batch of messages and writes rows as messages.
MongoDB Full Import: MongoDB; Export: MongoDB Reads and writes the collection through your MongoDB connection.
ORC Full Import: ORC; Export: ORC Reads and writes ORC files.
Parquet Full Import: Parquet; Export: Parquet Reads and writes Parquet files.
Pinecone Full Import and export: REST Request Reads an index’s records and upserts rows as vectors through the Pinecone REST API.
Redshift Full External database connection with a table import Reads and writes through a connection.
Salesforce Full Import: Query Salesforce Object; Export: REST Request Reads an object or SOQL query and writes records through your Salesforce connection.
Seed (inline) Full Import: CSV Loads the inline rows as a table.
Slack Full Import: Singer; Export: REST Request Reads channel messages and posts each row as a message through your Slack connection.
Smartsheet Full Import and export: REST Request Reads and writes the sheet’s rows through your Smartsheet connection.
Snowflake Full External database connection with a table import Reads and writes through a connection.
SQL file types (CSV, Excel, JSON, Parquet, Text, XML, fixed-width) Full Import and export file steps against a document account Reads and writes the file type; options without a PlaidCloud equivalent are flagged.
Text Full Import: CSV as a single column Reads each line as one column.
Web page Full REST Request Reads the page at the dataset’s URL into a table.
XML Full XML Parse Loads records by row tag.
Area Status PlaidCloud Equivalent Converts
Expressions Full Spark and warehouse SQL expressions translated function by function Functions and operators convert to PlaidCloud expressions; a function without an equivalent is flagged by name.
Project and graph structure Full Project loader Every Prophecy export shape loads and normalizes into one model.
Case What happens
Code or data the export does not include A Gem Builder or other custom gem whose code is not in the export, and a call to a macro of a dbt package the export leaves out, become a step that names what is missing; export the project with its pipeline code and dbt packages, or rebuild the logic in a Python or SQL step. Ab Initio Error Detection and MTime gems pass their input through, because the rule they apply is not in the export. Soda checks read from a file outside the project, and Soda checks that compare with earlier scans or with another dataset or test a format Soda names, have no row in the results table; compute them in a later step.
Results only known when the pipeline runs A REST API gem that flattens its response puts the whole response in one column when nothing declares the field names, which only a run reveals; extract the fields you need in a later step. A dbt macro that queries the warehouse as dbt compiles becomes a step that names it; replace the query with the values it returns. A Spark StoredProcedure gem runs, but its OUT values are not returned; the output is the pass-through columns. input_file_name() returns an empty string over a table or Delta source, which has no file to name, and hash() / xxhash64() inside an aggregate, a window or a gem with several inputs are computed by the warehouse, so they can differ from Spark’s.
Values written into a step when the project converts A parameter value holding a quote, backslash or brace, a pipeline value a SQL step reads, and the database a Delta table operation names through configuration are written into the step as fixed values, so changing the workflow variable does not change them; edit the step when the configuration changes.
Scala code A Scala script, Scala expression, or Scala user-defined function or aggregate function becomes a step that names it; rewrite it in SQL or Python.
Kafka Kerberos sign-in A Kafka source or target that signs in with Kerberos (GSSAPI) converts without the Kerberos service name and sign-in mechanism; set them on the PlaidCloud connection.
Streaming Streaming sources, targets and pipelines run as batch workflows over the data present when they run; schedule the workflow to run as often as you need.
Job components A Spark job converts its one pipeline and schedule; its other components, run conditions and configuration overrides are not converted.
Uncommon calls A user-defined aggregate function called from a rollup or a window, and a dbt macro that uses Jinja beyond if, for, set and the common filters, become a step that names the gem; call the function from an Aggregate gem, or inline the macro.

Compare the converted workflow’s output against the output of your own Prophecy pipeline: schema, row count, and row values. Ignore row order unless the workflow depends on it.