Prophecy Conversion Matrix
PlaidCloud converts Prophecy Spark pipelines and SQL projects into native Advanced workflows. The importer maps each Prophecy gem to a workflow step, project variable, or canvas object, so a converted workflow runs cloud-native rather than as an emulation.
Every gem, connector and project structure converts: Full means the gem becomes a native PlaidCloud step, variable, or canvas object that does the gem’s work. The few behaviours a converted workflow does not reproduce on its own are listed under Known Edge Cases, each with what to do.
To run a conversion, see Migrate Prophecy Projects.
Spark pipeline gems
Section titled “Spark pipeline gems”Source/Target
Section titled “Source/Target”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Lookup | Full | Lookup | Each lookup call becomes a Lookup step in the step that uses it; a key with no match reads NULL, and a duplicated key returns one of its rows. |
| Source | Full | Import step chosen by the dataset format (see Connectors) | Each Source becomes an import step into a table named after the dataset; a format the converter does not recognise is read as a CSV file uploaded to the workflow’s document folder, with a warning. |
| Streaming Source | Full | Same import as batch Source | Streaming sources convert to a one-shot batch import with a warning. |
| Streaming Target | Full | Batch append load | Streaming targets convert to a batch append with a warning. |
| Target | Full | Table export / load step chosen by dataset format and writeMode | Each Target becomes a table load or export step in the gem’s write mode: append, overwrite, merge with its SET expressions, upsert, SCD2, replaceWhere and insertInto. |
Prepare
Section titled “Prepare”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Data Cleansing | Full | Column expressions in a Table Extract step | Converts null fill, trim and case options into column expressions; cleaned columns move last. |
| Sequence | Full | Table Extract with a row number expression | Adds a sequence column from the start value and increment. |
| Surrogate Key Generator | Full | Table Extract with a row number expression | Adds a sequential key from the initial value; keys continue across runs from a table that holds the last key issued. |
Transform
Section titled “Transform”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Aggregate | Full | Table Extract with aggregation; Pivot when the gem pivots | Groups and aggregates, applies the HAVING condition, and pivots into one column per pivot value; propagate-all carries every other input column. |
| Bulk Column Expressions | Full | Expand to one expression per column in a Table Extract step | One expression per listed column in a Table Extract step; the conversion warns that the gem’s layout is assumed. |
| Bulk Column Rename | Full | Column rename in a Table Extract step | One rename per listed column in a Table Extract step; the conversion warns that the gem’s layout is assumed. |
| Deduplicate | Full | Table Extract with distinct or a row-number filter | Keeps one row per key: distinct, first or last by your sort order, any, or only the keys that occur once. |
| Dynamic Replace | Full | Dynamic replace transform, configured by hand | Converts to replacement rules on the dynamic replace transform; the conversion warns that the gem’s layout is assumed. |
| Dynamic Select | Full | Column select in a Table Extract step resolved by type | Resolves columns by type at conversion time into an explicit select. |
| Filter | Full | Table Extract with a filter expression | Converts to a Table Extract step with a filter expression; NULL conditions drop the row. |
| Flatten Schema | Full | JSON Parse and Table Extract | Flattens nested fields into columns named with the gem’s delimiter and explodes arrays into one row per element. |
| Fuzzy Match | Full | Table Extract and User Defined Transform | Pairs the records, scores each match field by the gem’s method, averages the scores, and keeps the pairs at or above your threshold. |
| Group By Rollup | Full | Union of Table Extract aggregation steps per rollup level | Expands the rollup into one aggregate per grouping level unioned together with NULL subtotals. |
| Limit | Full | Table Extract with a row limit | Converts to a row limit; config references become project variables. |
| Normalize | Full | Table Extract with an explode expression | Repeats each record by the length expression, computes the output columns per index, and stops at the first index where the finished condition holds. |
| Order By | Full | Sort on a Table Extract step | Converts to a sort with the same keys, direction and null placement. |
| Reformat | Full | Table Extract with column expressions | Converts to a Table Extract step with one expression per output column; unlisted columns are dropped. |
| Sample Rows | Full | Table Extract with a row limit or random sample | Keeps a random fraction or count of rows; the rows chosen differ from the Spark run because sampling is random. |
| Schema Transform | Full | Chain of Table Extract steps applied in order | Applies each operation in order: add or replace a column, drop, rename, add if missing with its default, and add a project business rule. |
| Set Operation | Full | Union All; Anti Join for except; Inner Join for intersect | Union all stacks the inputs; except all and intersect all keep the right number of duplicate rows. |
| TransposeAgg | Full | Table Extract with aggregation | Converts the gem’s grouping, aggregates and pivot as an Aggregate, with a warning to review it. |
| Unpivot | Full | Melt | Stacks the selected columns into name and value columns. |
| Watermark | Full | No step needed | Dropped: a batch run reads all the data present, so there is no late-data cutoff to apply. |
| Window Function | Full | Window expressions in a Table Extract step | Adds window columns with the gem’s partition, order and frame, rows or range, with unbounded, current-row or offset bounds. |
Join/Split
Section titled “Join/Split”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Compare Columns | Full | Full outer join plus per-column aggregates | Compares the columns through a full outer join of the unpivoted data, with match and mismatch counts. |
| Join | Full | Inner Join, Multi-Table Join, Anti Join or Cross Join | Builds the join of each type, inner, left, right, full, semi, anti, cross and natural, on any join condition, applies the where clause, and writes the select list. |
| Repartition | Full | No step needed | Dropped: PlaidCloud manages partitioning, so the next step reads the upstream table directly. |
| Row Distributor | Full | One Table Extract filter per output | Converts to one filter per output; rows can land in multiple outputs. |
| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Column Parser | Full | JSON Parse / XML Parse | Converts to a JSON or XML parse of the named column; the conversion warns that the gem’s layout is assumed. |
Custom
Section titled “Custom”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Custom Gem (Gem Builder) | Full | User Defined Transform running the gem’s generated code | The gem’s generated PySpark code from the pipeline export runs in a Python step, with a warning to review it. |
| CustomReformatBitbucket | Full | Table Extract | Converts as the Reformat whose expressions the gem carries, with a warning to review it. |
| Data Quality Check | Full | Union All of one result row per check; Raise Workflow Error to stop on failures | Evaluates each check over the input and writes one result row per check with its status and message. The data passes through, and the run stops when the gem’s failure threshold is reached. |
| Deequ | Full | Table Extract checks | Passes data through and builds the constraint results table for the isComplete, isUnique, isNonNegative and isContainedIn rules. |
| Delta Table Operations | Full | Table In-Place Delete, Table Update, Table Drop, Table Restore to Timestamp | Delete, update, drop and restore to a timestamp map to table steps; optimize, vacuum and restore by version pass data through with a warning. |
| Directory | Full | Directory Listing | Lists the files of the bound document folder in the gem’s columns; with the sheet-name option, each workbook gives one row per sheet. A missing folder gives zero rows. |
| Full | Notify Via Email | Sends the email, with an optional data attachment, through a bound email connection. | |
| Email Data | Full | Notify Via Email | Emails the input rows as a table and passes the input through unchanged. |
| File Operation | Full | Copy Document File or Copy Document Directory | Copies, moves or syncs files from the bound document folder into the converted workflow’s document folder; a move removes the source files. |
| Log | Full | Table Extract | Writes the gem’s event row (time, gem name, event type and text) to its output table. |
| REST API Enrich | Full | REST Request | Calls the API once per row and adds the response columns. |
| Script | Full | Python step with frame-access rewrite | Embeds the script body in a python step, with data-frame access rewritten; Scala bodies carried with a note. |
| Soda Data Quality Check | Full | Table Extract checks | Builds the check results table from the Soda checks. |
| SQL Statement | Full | SQL step per output tab, text translated to the tenant dialect | Embeds each tab as a SQL step with inputs bound as temp tables; translated to the tenant dialect. |
| Todo | Full | Raise Workflow Error or Notify via Log | Marks logic still to be written: a log message when nothing reads it, or a step that stops the run with the gem’s message when a later step depends on it. |
Report
Section titled “Report”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Tableau | Full | Publish to Tableau | Writes a Hyper extract and publishes it to the bound Tableau connection. |
| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Bedrock | Full | Text NLP AI | Embeds the text column through the workspace AI connection. |
| OpenAI | Full | Text NLP AI | Answers questions or embeds text through the workspace AI connection. |
| Pinecone Lookup | Full | REST Request | Queries the Pinecone index for each row and adds pinecone_matches and pinecone_error columns. |
| SageMaker Endpoint | Full | Text NLP AI | Answers questions through the workspace AI connection. |
| Text Processing | Full | Table Extract for chunk splitting; REST Request for web scraping | Chunk splitting becomes a character-chunk expression; web scraping fetches each row’s URL and joins the page content back onto the row. |
Subgraph/Control
Section titled “Subgraph/Control”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Control Flow Input | Full | Dropped (boundary marker only) | Dropped; it only marks the iterator sub-workflow input boundary. |
| Control Flow Output | Full | Dropped (boundary marker only) | Dropped; it only marks the iterator sub-workflow output boundary. |
| Subgraph (Basic) | Full | Canvas container / nested workflow from inline processes | Converts the inline inner graph into a nested workflow (or inlined steps) with in/out table parameters. |
| Table Iterator | Full | Workflow Loop | Runs the loop body once per input row, setting each configuration value from its column, up to maxIteration. |
| While Iterator | Full | Workflow Loop | Runs the loop body once per pass, passing the iteration number, until the body returns no rows or the gem’s maximum is reached. |
SQL / Analyst gems
Section titled “SQL / Analyst gems”Source/Target
Section titled “Source/Target”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| External table Source/Target (BigQuery, Databricks, SAP HANA, MongoDB, MSSQL, Oracle, Postgres, Redshift, Salesforce, Snowflake, Synapse) | Full | External Data import/export step with the matching database connection | Reads or writes a table in an external database through a PlaidCloud connection. |
| File storage Source/Target (ADLS, Databricks Volumes, GCS, OneDrive, S3, SFTP, SharePoint, Smartsheet) | Full | Import/Export file steps against a document account (ADLS/GCS/S3/SFTP/SharePoint/OneDrive) | Reads or writes files in cloud/SFTP storage with wildcard union, trigger, delete/move and encryption options. |
| Model | Full | Upstream step output table | Reads another model as the table produced by the referenced upstream step. |
| Seed | Full | Import: CSV | Seed CSV is imported into a PlaidCloud table. |
| Source | Full | Import step chosen by the dataset format (see Connectors) | Each Source becomes an import step into a table named after the dataset; a format the converter does not recognise is read as a CSV file uploaded to the workflow’s document folder, with a warning. |
| Target Model / Table (Target) | Full | Table Extract, Upsert, Copy Table and Union All | Writes the model’s final select to a PlaidCloud table as replace, append, upsert, delete-and-insert, SCD2 or replace by predicate. |
| Upload Files | Full | Import: CSV | Uploaded local files become a table via an import step. |
Prepare
Section titled “Prepare”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Data Cleansing | Full | Column expressions in a Table Extract step | Converts null fill, trim and case options into column expressions; cleaned columns move last. |
| Data Masking | Full | Column expressions for character masking or hashing | Masks or hashes selected string columns in place or as new columns. |
| Find Duplicates | Full | Row-number window expression plus filter | Returns unique, duplicate or count/position-selected rows per key. |
| Generate Rows | Full | Sequence table joined and filtered per input row | Generates rows per input row from init, condition and loop expressions. |
| Imputation | Full | Aggregate plus join-back and column expression | Replaces nulls or sentinel values with average, median, mode or a fixed value. |
| Multi-Column Edit | Full | Per-column expressions expanded from the template | Applies one expression template to every selected column. |
| Multi-Column Rename | Full | Column rename mapping on a select step | Renames selected columns by prefix, suffix or name expression. |
| Record ID | Full | Row-number or uuid column expression | Adds a UUID or sequential record id column. |
| Sample | Full | Row limit or sample option on extract | Takes first/last/random/Nth/percent rows overall or per group. |
| Tile | Full | Window expressions for tile number and sequence | Assigns tile numbers by equal sum, count, std-dev bands, distinct values or manual cutoffs. |
Transform
Section titled “Transform”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Aggregate | Full | Table Extract with aggregation; Pivot when the gem pivots | Groups and aggregates, applies the HAVING condition, and pivots into one column per pivot value; propagate-all carries every other input column. |
| Count Records | Full | Aggregate step with count functions | Returns total, non-null or distinct counts as one row. |
| Data Encoder/Decoder | Full | Column expressions for base64/hex encode and decode | Encodes or decodes selected columns; AES only on Databricks. |
| Deduplicate | Full | Table Extract with distinct or a row-number filter | Keeps one row per key: distinct, first or last by your sort order, any, or only the keys that occur once. |
| Dynamic Select | Full | Column select in a Table Extract step resolved by type | Resolves columns by type at conversion time into an explicit select. |
| Filter | Full | Table Extract with a filter expression | Converts to a Table Extract step with a filter expression; NULL conditions drop the row. |
| Finance | Full | Column expression implementing the financial function | Appends a financial function result (FV, PV, NPV, IRR, …) as a new column. |
| Flatten Schema | Full | JSON Parse and Table Extract | Flattens nested fields into columns named with the gem’s delimiter and explodes arrays into one row per element. |
| Fuzzy Match | Full | Table Extract and User Defined Transform | Pairs the records, scores each match field by the gem’s method, averages the scores, and keeps the pairs at or above your threshold. |
| Limit | Full | Table Extract with a row limit | Converts to a row limit; config references become project variables. |
| Order By | Full | Sort on a Table Extract step | Converts to a sort with the same keys, direction and null placement. |
| Pivot | Full | Pivot | Turns the selected pivot values into columns of aggregates; a Simple-mode value list fetched at design time is converted as the listed values. |
| Reformat | Full | Table Extract with column expressions | Converts to a Table Extract step with one expression per output column; unlisted columns are dropped. |
| Running Total | Full | Cumulative-sum window expression column | Appends cumulative sums per selected column. |
| Snowflake Unpivot (third-party) | Full | Melt | Unpivots columns via Snowflake UNPIVOT. |
| Transpose | Full | Melt | Stacks data columns into name and string value columns. |
| TransposeAgg | Full | Table Extract with aggregation | Converts the gem’s grouping, aggregates and pivot as an Aggregate, with a warning to review it. |
| Unpivot | Full | Melt | Stacks data columns into name and value columns. |
| Weighted Average | Full | Aggregate step with weighted-average expression | Computes a weighted mean overall or per group. |
| Window Function | Full | Window expressions in a Table Extract step | Adds window columns with the gem’s partition, order and frame, rows or range, with unbounded, current-row or offset bounds. |
Join/Split
Section titled “Join/Split”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Except | Full | Anti Join on all columns plus a distinct Table Extract | Keeps rows of the first input that appear in none of the other inputs. |
| Intersect | Full | Inner Join on all columns plus Distinct | Keeps rows present in every input. |
| Join | Full | Inner Join, Multi-Table Join, Anti Join or Cross Join | Builds the join of each type, inner, left, right, full, semi, anti, cross and natural, on any join condition, applies the where clause, and writes the select list. |
| Union | Full | Union All | Stacks the inputs by position, removing duplicates when the gem does not keep them. |
| Union By Name | Full | Union All | Becomes a Union All that aligns columns by name and null-fills missing ones. |
| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| JSON Parse | Full | JSON Parse | Parses a JSON text column into structured fields. |
| Regex | Full | Regex expressions in a Table Extract / Transform step | Replace, parse, match or tokenize a text column with a regex. |
| Text To Columns | Full | Split-column / explode steps with regex delimiter | Splits a delimited column into columns or rows using a regex delimiter. |
| XML Parse | Full | XML Parse | Parses an XML text column into structured fields. |
Custom
Section titled “Custom”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Directory | Full | Directory Listing | Lists the files of the bound document folder in the gem’s columns; with the sheet-name option, each workbook gives one row per sheet. A missing folder gives zero rows. |
| DynamicInput | Full | SQL step for query templates; workflow loop for multi-file mode | Query-template mode embeds the template as a SQL step; multi-file mode maps with assumptions. |
| Full | Notify Via Email | Sends the email, with an optional data attachment, through a bound email connection. | |
| RestAPI | Full | REST Request | Sends one request per input row with your URL, headers, query, body and retries, keeps the response, and polls an asynchronous request until it is done. Credentials come from the connection you bind. |
| Script | Full | Python step with frame-access rewrite | Embeds the script body in a python step, with data-frame access rewritten; Scala bodies carried with a note. |
| SQL Statement | Full | SQL step per output tab, text translated to the tenant dialect | Embeds each tab as a SQL step with inputs bound as temp tables; translated to the tenant dialect. |
| StoredProcedure | Full | Agent Remote Execution of SQL | Calls the procedure on its warehouse through the connection you choose: once per input row with each argument bound from its column, or once when it takes no arguments. The input columns pass through. |
| Table Operations | Full | Table In-Place Delete, Table Update, Table Drop | Update, delete and drop map to table steps; vacuum and optimize become table optimize or a no-op with a warning. |
| ToDo | Full | Raise Workflow Error or Notify via Log | Marks logic still to be written: a log message when nothing reads it, or a step that stops the run with the gem’s message when a later step depends on it. |
Spatial
Section titled “Spatial”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Buffer | Full | Spatial Buffer | Grows each geometry by the configured distance in miles or kilometers. |
| CreatePoint | Full | Table Extract with geom_point |
Builds point geometry from longitude and latitude columns. |
| Distance | Full | Table Extract | Computes distance, cardinal direction and bearing between two WKT points. |
| FindNearest | Full | Table Extract with a cross join | Finds N nearest target points for each source point within a max distance. |
| HeatMap | Full | Heat Map (macro) | Bins points into H3 hexagons at the gem’s resolution, spreads the heat over the grid distance with its decay, and gives each hexagon’s density and outline. |
| PolyBuild | Full | Table Extract and a Python step | Builds polygon or polyline WKT from ordered points per group. |
| Simplify | Full | Spatial Generalize | Simplifies WKT geometries by a tolerance in miles/kms. |
| SpatialMatch | Full | Spatial Match | Matches source and target geometries by intersects/contains/within/touches/envelope. |
Report
Section titled “Report”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| PowerBIWrite | Full | REST Request | Creates the Power BI dataset and pushes the rows through the Power BI REST API. |
| TableauWrite | Full | Publish to Tableau | Writes a Hyper extract and publishes it to the bound Tableau connection. |
| Visualize | Full | Copy Table | Passes the rows on unchanged; build the chart as a PlaidCloud dashboard on the same table. |
Subgraph/Control
Section titled “Subgraph/Control”| Gem | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Condition | Full | Cross Join and Table Extract | Evaluates the conditions over the whole input and sends every row to the first output whose condition holds, else to the last output. |
| Macro | Full | SQL step with the macro body expanded | Embeds the macro body as a SQL step translated to the tenant dialect, inputs bound as temp tables. |
| Pipeline (trigger) | Full | Workflow Loop | Runs the named pipeline once per input row with your parameters and trigger condition, continues past a failed run, and returns one row per run. |
| Subgraph (Basic) | Full | Canvas container / nested workflow from inline processes | Converts the inline inner graph into a nested workflow (or inlined steps) with in/out table parameters. |
Connectors
Section titled “Connectors”Dataset formats and connectors are source and target settings of a gem, not gems. Each format converts to the import or export step shown, and conversion never contacts the source or target system.
| Format | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Avro | Full | Import: Avro; Export: Avro | Reads and writes Avro files. |
| BigQuery | Full | External database connection with a table import | Reads and writes through a connection. |
| Binary files | Full | Directory Listing with file contents; Alteryx Executor to write | Reads each file of the folder with its path, modification time, length and content; writes each row’s content to the file its path names. |
| Catalog table | Full | Existing PlaidCloud table binding | Binds to a table by name; reads and every write mode convert. |
| Cosmos DB | Full | Import and export: REST Request | Reads and writes the container’s documents through your Cosmos DB connection. |
| CSV | Full | Import: CSV; Export: CSV | Reads and writes delimited files with header, separator, quote, escape, and encoding options. |
| Data Generator | Full | Table Faker | Generates the dataset’s columns with the same kinds of fake data. |
| DataHub table | Full | Import SQL | Reads the table the dataset names through your connection; a write lands in a PlaidCloud table. |
| Delta | Full | Existing PlaidCloud table binding | Reads and writes the table in every write mode; a read of an earlier version reads it through the Databricks connection you bind. |
| Excel (XLSX) | Full | Import: Excel; Export: Excel | Reads and writes workbook sheets and ranges. |
| Fixed format | Full | Import: Fixed Width; Export: CSV | Reads fixed-width files laid out from the dataset’s column widths; writes a delimited file. |
| Hive | Full | Import: Import SQL; Export: External Tables | Reads and writes the table through your Hive connection. |
| Iceberg | Full | Existing PlaidCloud table binding | Reads and writes the table. |
| JDBC | Full | External database connection with a table import | Reads a table or query and writes in replace or append mode through your connection; a post-write statement runs after the write. |
| JSON | Full | JSON Parse | Loads records and flattens nested fields. |
| Kafka | Full | Import: Import Kafka; Export: Export Kafka | Reads a topic as one batch of messages and writes rows as messages. |
| MongoDB | Full | Import: MongoDB; Export: MongoDB | Reads and writes the collection through your MongoDB connection. |
| ORC | Full | Import: ORC; Export: ORC | Reads and writes ORC files. |
| Parquet | Full | Import: Parquet; Export: Parquet | Reads and writes Parquet files. |
| Pinecone | Full | Import and export: REST Request | Reads an index’s records and upserts rows as vectors through the Pinecone REST API. |
| Redshift | Full | External database connection with a table import | Reads and writes through a connection. |
| Salesforce | Full | Import: Query Salesforce Object; Export: REST Request | Reads an object or SOQL query and writes records through your Salesforce connection. |
| Seed (inline) | Full | Import: CSV | Loads the inline rows as a table. |
| Slack | Full | Import: Singer; Export: REST Request | Reads channel messages and posts each row as a message through your Slack connection. |
| Smartsheet | Full | Import and export: REST Request | Reads and writes the sheet’s rows through your Smartsheet connection. |
| Snowflake | Full | External database connection with a table import | Reads and writes through a connection. |
| SQL file types (CSV, Excel, JSON, Parquet, Text, XML, fixed-width) | Full | Import and export file steps against a document account | Reads and writes the file type; options without a PlaidCloud equivalent are flagged. |
| Text | Full | Import: CSV as a single column | Reads each line as one column. |
| Web page | Full | REST Request | Reads the page at the dataset’s URL into a table. |
| XML | Full | XML Parse | Loads records by row tag. |
Expressions and Project Structure
Section titled “Expressions and Project Structure”| Area | Status | PlaidCloud Equivalent | Converts |
|---|---|---|---|
| Expressions | Full | Spark and warehouse SQL expressions translated function by function | Functions and operators convert to PlaidCloud expressions; a function without an equivalent is flagged by name. |
| Project and graph structure | Full | Project loader | Every Prophecy export shape loads and normalizes into one model. |
Known Edge Cases
Section titled “Known Edge Cases”| Case | What happens |
|---|---|
| Code or data the export does not include | A Gem Builder or other custom gem whose code is not in the export, and a call to a macro of a dbt package the export leaves out, become a step that names what is missing; export the project with its pipeline code and dbt packages, or rebuild the logic in a Python or SQL step. Ab Initio Error Detection and MTime gems pass their input through, because the rule they apply is not in the export. Soda checks read from a file outside the project, and Soda checks that compare with earlier scans or with another dataset or test a format Soda names, have no row in the results table; compute them in a later step. |
| Results only known when the pipeline runs | A REST API gem that flattens its response puts the whole response in one column when nothing declares the field names, which only a run reveals; extract the fields you need in a later step. A dbt macro that queries the warehouse as dbt compiles becomes a step that names it; replace the query with the values it returns. A Spark StoredProcedure gem runs, but its OUT values are not returned; the output is the pass-through columns. input_file_name() returns an empty string over a table or Delta source, which has no file to name, and hash() / xxhash64() inside an aggregate, a window or a gem with several inputs are computed by the warehouse, so they can differ from Spark’s. |
| Values written into a step when the project converts | A parameter value holding a quote, backslash or brace, a pipeline value a SQL step reads, and the database a Delta table operation names through configuration are written into the step as fixed values, so changing the workflow variable does not change them; edit the step when the configuration changes. |
| Scala code | A Scala script, Scala expression, or Scala user-defined function or aggregate function becomes a step that names it; rewrite it in SQL or Python. |
| Kafka Kerberos sign-in | A Kafka source or target that signs in with Kerberos (GSSAPI) converts without the Kerberos service name and sign-in mechanism; set them on the PlaidCloud connection. |
| Streaming | Streaming sources, targets and pipelines run as batch workflows over the data present when they run; schedule the workflow to run as often as you need. |
| Job components | A Spark job converts its one pipeline and schedule; its other components, run conditions and configuration overrides are not converted. |
| Uncommon calls | A user-defined aggregate function called from a rollup or a window, and a dbt macro that uses Jinja beyond if, for, set and the common filters, become a step that names the gem; call the function from an Aggregate gem, or inline the macro. |
Validating a Converted Workflow
Section titled “Validating a Converted Workflow”Compare the converted workflow’s output against the output of your own Prophecy pipeline: schema, row count, and row values. Ignore row order unless the workflow depends on it.