System requirements for Databricks

Supported versions of Databricks

Tonic Structural supports Spark 2.4.x, Spark 3.0.x, and Spark 3.1.x. Spark 2.4.2 is not supported.

Any version of Databricks that runs one of those Spark versions should be compatible. Structural specifically has been tested against Databricks versions 9.1, and 10.4.

Supported providers

Structural supports the following data providers:

Source Provider	Output Provider
Parquet	Parquet
CSV	Parquet
Avro	Avro
JSON	JSON
ORC	ORC
Delta	Delta

Supported table types

Databricks supports both MANAGED and EXTERNAL tables.

• MANAGED tables store all of their data within Databricks.

• EXTERNAL tables store their data on a separate file system (often S3).

Structural can read from both table types. When it writes output data, Structural only writes to EXTERNAL tables.