System requirements for Databricks
Supported versions of Databricks
Tonic Structural supports Spark 2.4.x, Spark 3.0.x, and Spark 3.1.x. Spark 2.4.2 is not supported.
Any version of Databricks that runs one of those Spark versions should be compatible. Structural specifically has been tested against Databricks versions 9.1, and 10.4.
Supported providers
Structural supports the following data providers:
| Source Provider | Output Provider |
|---|---|
Parquet | Parquet |
CSV | Parquet |
Avro | Avro |
JSON | JSON |
ORC | ORC |
Delta | Delta |
Supported table types
Databricks supports both MANAGED and EXTERNAL tables.
MANAGED tables store all of their data within Databricks.
EXTERNAL tables store their data on a separate file system (often S3).
Structural can read from both table types. When it writes output data, Structural only writes to EXTERNAL tables.
Last updated