Structural process overview for Spark with Livy

The following high-level diagram describes how Tonic Structural data generation is processed for Spark with Livy.

Overview diagram of the Tonic Structural process for Spark with Livy

For a Spark with Livy workspace, the source data comes from a Hive catalog.

Apache Livy coordinates the running of the data generation job on the Spark cluster.

The destination data is written to a location on HDFS.