Required license: Professional or Enterprise
Not available on Tonic Structural Cloud.
Spark with Livy is only available for data generation workspaces. You cannot use Spark with Livy for data science mode workspaces.
Spark with Livy workspaces do not support workspace inheritance.
You can only assign the De-Identify or Truncate table modes.
For Truncate mode, the table is ignored completely. The table does not exist in the destination database.
For Spark 2.4.x and higher, Spark with Livy workspaces cannot use the following generators:
AI Synthesizer
Algebraic
Array Character Scramble
Array JSON Mask
Array Regex Mask
Cross-Table Sum
CSV Mask
Event Timestamps
HTML Mask
JSON Mask
SIN
The following generators are supported, but with restrictions:
Character Scramble is only supported for text columns.
Timestamp Shift is only supported on date column types.
For Spark 2.3.x and 2.4.2, Spark with Livy workspaces only support the following generators:
Address
Categorical
Character Scramble
Company Name
Constant
Continuous
Custom Categorical
Date Truncation
HIPAA Address
Integer Key
JSON Mask
MAC Address
Name
Noise Generator
Null
Random Hash
Random Integer
Random UUID
Regex Mask
SSN
Struct Mask
Timestamp Shift Generator
UUID Key
Amazon EMR workspaces do not support subsetting.
However, for tables that use the De-Identify table mode, you can provide a WHERE
clause to filter the table. For details, go to Using table filtering for data warehouses and Spark-based data connectors.
Spark with Livy workspaces do not support upsert.
For Spark with Livy workspaces, you cannot write the destination data to container artifacts.
For Spark with Livy workspaces, you cannot write the destination data to an Ephemeral snapshot.