Structural differences and limitations with Spark with Livy

Required license: Professional or Enterprise

Not available on Tonic Structural Cloud.

No data science mode

Spark with Livy is only available for data generation workspaces. You cannot use Spark with Livy for data science mode workspaces.

No workspace inheritance

Spark with Livy workspaces do not support workspace inheritance.

Table mode limitations

You can only assign the De-Identify or Truncate table modes.

For Truncate mode, the table is ignored completely. The table does not exist in the destination database.

Generator limitations

Spark 2.4.x and higher

For Spark 2.4.x and higher, Spark with Livy workspaces cannot use the following generators:

  • AI Synthesizer

  • Algebraic

  • Array Character Scramble

  • Array JSON Mask

  • Array Regex Mask

  • Cross-Table Sum

  • CSV Mask

  • Event Timestamps

  • HTML Mask

  • JSON Mask

  • SIN

The following generators are supported, but with restrictions:

  • Character Scramble is only supported for text columns.

  • Timestamp Shift is only supported on date column types.

Spark 2.3.x and Spark 2.4.2

For Spark 2.3.x and 2.4.2, Spark with Livy workspaces only support the following generators:

  • Address

  • Categorical

  • Character Scramble

  • Company Name

  • Constant

  • Continuous

  • Custom Categorical

  • Date Truncation

  • Email

  • HIPAA Address

  • Integer Key

  • JSON Mask

  • MAC Address

  • Name

  • Noise Generator

  • Null

  • Random Hash

  • Random Integer

  • Random UUID

  • Regex Mask

  • SSN

  • Struct Mask

  • Timestamp Shift Generator

  • UUID Key

No subsetting, but support for table filtering

Amazon EMR workspaces do not support subsetting.

However, for tables that use the De-Identify table mode, you can provide a WHERE clause to filter the table. For details, go to Using table filtering for data warehouses and Spark-based data connectors.

No upsert

Spark with Livy workspaces do not support upsert.

No output to container artifacts

For Spark with Livy workspaces, you cannot write the destination data to container artifacts.

No output to an Ephemeral snapshot

For Spark with Livy workspaces, you cannot write the destination data to an Ephemeral snapshot.

Last updated