Skip to content
PlaidCloud

Import Delta Lake

Description

Section titled “Description”

Imports a Delta Lake table stored in PlaidCloud Document into a project table. Point the step at the table’s folder, the one that holds its _delta_log folder and data files. The table is read at its latest version.

The step form has the same tabs as Import Parquet: Import Parameters picks the Source File, the table’s folder, and the Target Table; Table Data Selection maps the columns; Data Filters filters the rows. Populate reads the columns and their types from the table’s _delta_log, with any partition columns last, as text. When the folder is missing or empty, Populate answers Nothing to populate with a warning naming the path and keeps the mapping you have.

Migrated KNIME workflows use this step for Delta Table Reader nodes.

Configuration

Section titled “Configuration”
  • input_file — the Document account and the path of the Delta table folder. A search finds several tables: with schema_mismatch set to union (default), each table is loaded by column name, columns a table lacks are NULL, and columns the mapping does not declare are left out; error stops the step, naming the table and the columns. Set schema_mismatch with an AI assistant or the REST API.
  • target_frame — the table the rows are written to.
  • source_columns and target_columns — the column mapping.
  • source_where, having, aggregate, aggregation_type and distinct — filter and summarize the rows, as on other import steps.
  • use_target_slicer, limit_target_start and limit_target_end — keep only a slice of the result.
  • delete_file — delete the imported files after the import.
  • ascii_only — remove non-ASCII characters.
  • skip_if_unchanged — on by default: a workflow run skips the import when the source files and the step’s settings are unchanged since the last successful run.
Section titled “Related”