Structural differences and limitations with Spark with Livy
Required license: Professional or Enterprise
Not available on Tonic Structural Cloud.
No data science mode
Spark with Livy is only available for data generation workspaces. You cannot use Spark with Livy for data science mode workspaces.
No workspace inheritance
Spark with Livy workspaces do not support workspace inheritance.
Table mode limitations
You can only assign the De-Identify or Truncate table modes.
For Truncate mode, the table is ignored completely. The table does not exist in the destination database.
Generator limitations
Spark 2.4.x and higher
For Spark 2.4.x and higher, Spark with Livy workspaces cannot use the following generators:
AI Synthesizer
Algebraic
Array Character Scramble
Array JSON Mask
Array Regex Mask
Cross-Table Sum
CSV Mask
Event Timestamps
HTML Mask
JSON Mask
SIN
The following generators are supported, but with restrictions:
Character Scramble is only supported for text columns.
Timestamp Shift is only supported on date column types.
Spark 2.3.x and Spark 2.4.2
For Spark 2.3.x and 2.4.2, Spark with Livy workspaces only support the following generators:
Address
Categorical
Character Scramble
Company Name
Constant
Continuous
Custom Categorical
Date Truncation
Email
HIPAA Address
Integer Key
JSON Mask
MAC Address
Name
Noise Generator
Null
Random Hash
Random Integer
Random UUID
Regex Mask
SSN
Struct Mask
Timestamp Shift Generator
UUID Key
No subsetting, but support for table filtering
Amazon EMR workspaces do not support subsetting.
However, for tables that use the De-Identify table mode, you can provide a WHERE clause to filter the table. For details, go to Using table filtering for data warehouses and Spark-based data connectors.
No upsert
Spark with Livy workspaces do not support upsert.
No output to container artifacts
For Spark with Livy workspaces, you cannot write the destination data to container artifacts.
No output to an Ephemeral snapshot
For Spark with Livy workspaces, you cannot write the destination data to an Ephemeral snapshot.
Last updated