The following high-level diagram describes how Tonic Structural orchestrates the processing and moving of data in Snowflake on Azure.

Structural orchestrates the moving and transforming of data between Snowflake databases that are hosted on Azure. Structural uses Azure Blob Storage for interim storage of files that contain the source and destination data.

At a high level, the data generation process is:

1. Structural copies the table data from the Snowflake database to files in Azure Blob Storage. You specify the container path in the Structural workspace configuration. Structural places the files in an input folder within the container path.

2. Structural applies the configured generators to the data in the files, then writes the resulting files to an output folder in the container path.

3. As it finishes processing each file, Structural copies the data from the container path’s output folder into the Snowflake destination database.

In the workspace configuration, under Connection Type, select Snowflake.

In the Source Settings section, under Snowflake Type, click Azure.

Connecting to the source database

In the Source Settings section, provide the details about connecting to the source database.

Providing the connection details

To connect to the source database, you can either populate the connection fields or use a connection string.

You can also opt to use key pair authentication to connect to the database.

Populating the connection fields

By default, Use connection string is off, and you provide the connection values in the individual fields:

1. In the Server field, provide the server where the database is located. You must provide the full path to the server. https:// is optional. So the format of the server value can be either: <account>.<region>.snowflakecomputing.com https://<account>.<region>.snowflakecomputing.com For example: abc123456.us-east-1.snowflakecomputing.com or https://abc123456.us-east-1.snowflakecomputing.com

2. In the Database field, provide the name of the database.

3. In the Username field, provide the username for the account to use to connect to the database.

4. In the Password field, provide the password for the specified username.

Using a connection string

To use a connection string to connect to the source database:

1. Toggle Use connection string to the on position.

2. In the Connection String field, provide the connection string.

3. In the Password field, provide the password for the user.

4. To test the connection to the source database, click Test Source Connection.

The connection string uses the following format:

account=<account>;host=<account>.<region>.snowflakecomputing.com;user=<username>;db=<database>

Using key pair authentication to connect to the database

Instead of providing a password, you can instead use key pair authentication.

To do this:

1. Toggle Use Key Pair Authentication to the on position.

2. Expand the Key Pair Authentication Settings.

3. For RSA Private Key, click Browse, then select the key file.

4. If the key is encrypted, then in the Encrypted Key Passphrase field, provide the passphrase to use to decrypt the key.

Indicating whether to trust the server certificate

To trust the server certificate, and ignore the certificate authority's revocation list, toggle Trust Server Certificate to the on position.

This option can be useful when your Tonic Structural instance cannot connect to the certificate authority.

Enabling a proxy connection

You can use a proxy server to connect to the source database.

To use a proxy server to connect to the source database:

1. Toggle Enable proxy connection to the on position.

2. In the Proxy Host field, provide the host name for the proxy connection.

3. In the Proxy Port field, provide the port for the proxy connection.

4. Optionally, in the Proxy User field, provide the name of the user for the proxy connection.

5. If you provide a proxy user, then in the Proxy Password field, provide the password for the specified user.

6. Optionally, in the Non-Proxy Hosts field, provide the list of hosts for which to bypass the proxy server and connect to directly.

Use a pipe symbol (|) to separate the host names. For example, host1|host2|host3.

You can also use an asterisk (*) as a wildcard. For example, to connect directly to all hosts whose host names start with myhost, use myhost*.

Limiting the included schemas

By default, the source database includes all of the schemas. To specify a list of specific schemas to either include or exclude:

1. Toggle Limit Schemas to the on position.

2. From the filter option dropdown list, select whether to include or exclude the listed schemas.

3. In the field, provide the list of schemas to either include or exclude. Use commas or semicolons to separate the schemas.

Do not exclude schemas that are referred to by included schemas, unless you create those schemas manually outside of Structural.

Testing the source connection

To test the connection to the source database, click Test Source Connection.

Blocking data generation on all schema changes

By default, data generation is not blocked as long as schema changes do not conflict with your workspace configuration.

To block data generation when there are any schema changes, regardless of whether they conflict with your workspace configuration, toggle Block data generation on schema changes to the on position.

Connecting to the destination database

In the Destination Settings section, you specify the connection information for the destination database.

If the destination database is in the same location as the source database, then you can copy the connection and authentication details from the source database. The copied details include the proxy connection configuration.

If the destination database is in a different location, then you can either populate the connection fields or use a connection string.

You can also opt to use key pair authentication to connect to the database.

Copying the source database connection details

To copy the connection details from the source database:

1. Click Copy Settings from Source.

2. In the Password field, provide the password.

3. To test the connection to the destination database, click Test Destination Connection.

Providing destination database connection details

If you don't copy the details from the source database, then you can either populate the connection fields or use a connection string.

Populating the connection fields

By default, Use connection string is off, and you provide the connection values in the individual fields:

1. In the Server field, provide the server where the database is located. You must provide the full path to the server. https:// is optional. So the format of the server value can be either: <account>.<region>.snowflakecomputing.com https://<account>.<region>.snowflakecomputing.com For example: abc123456.us-east-1.snowflakecomputing.com or https://abc123456.us-east-1.snowflakecomputing.com

2. In the Database field, provide the name of the database.

3. In the Username field, provide the username for the account to use to connect to the database.

4. In the Password field, provide the password for the specified username.

Using a connection string

To use a connection string to connect to the destination database:

1. Toggle Use connection string to the on position.

2. In the Connection String field, provide the connection string.

3. In the Password field, provide the password for the user.

The connection string uses the following format:

account=<account>;host=<account>.<region>.snowflakecomputing.com;user=<username>;db=<database>

Indicating whether to trust the server certificate

To trust the server certificate, and ignore the certificate authority's revocation list, toggle Trust Server Certificate to the on position.

This option can be useful when your Structural instance cannot connect to the certificate authority.

Enabling a proxy connection

You can use a proxy server to connect to the destination database.

To enable and configure the proxy connection:

1. Toggle Enable proxy connection to the on position.

2. In the Proxy Host field, provide the host name for the proxy connection.

3. In the Proxy Port field, provide the port for the proxy connection.

4. Optionally, in the Proxy User field, provide the name of the user for the proxy connection.

5. If you provide a proxy user, then in the Proxy Password field, provide the password for the specified user.

6. Optionally, in the Non-Proxy Hosts field, provide the list of hosts for which to bypass the proxy server and connect to directly.

Use a pipe symbol (|) to separate the host names. For example, host1|host2|host3.

You can also use an asterisk (*) as a wildcard. For example, to connect directly to all hosts whose host names start with myhost, use myhost*.

Using key pair authentication to connect to the database

Instead of providing a password, you can instead use key pair authentication.

To do this:

1. Toggle Use Key Pair Authentication to the on position.

2. Expand the Key Pair Authentication Settings.

3. For RSA Private Key, click Browse, then select the key file.

4. If the key is encrypted, then in the Encrypted Key Passphrase field, provide the passphrase to use to decrypt the key.

Testing the destination database connection

To test the connection to the destination database, click Test Destination Connection.

Setting the storage location for temporary files

During data generation, Structural uses temporary CSV files to load and unload Snowflake tables.

To store these temporary files, you can use either:

• Azure Blob Storage

• External stages

During data generation, Structural copies CSV files containing the source data to an input folder in the storage location. After the generators are applied, Structural copies CSV files containing the destination data to an output folder.

Selecting the type of storage

Under File Storage Options, by default, the Use External Stage toggle is off, which indicates to use Azure Blob Storage for the temporary files.

To instead use external stages, toggle Use External Stage to the on position.

Setting the Azure Blob Storage location for temporary files

When Use External Stage is off, in the Source Azure Storage Account Name field, provide the name of the account that Structural uses to load and unload Snowflake data in Azure Blob Storage. Do not provide the URI. You only need the account name.

In the Source Azure Blob Storage Path field, provide the container path to the Azure Blob Storage location where Structural places temporary CSV files that it uses to load and unload Snowflake tables.

Setting the external stage location for temporary files

When Use External Stage is on, in the Source Snowflake External Stage Name field, provide the path to the external stage to use for temporary files.

The format is:

<database>.<schema>.<stage>

Where:

• <database> is the name of the database where the stage is located.

• <schema> is the name of the schema that contains the stage.

• <stage> is the name of the stage.

Configuring the source and destination users

Tonic.ai recommends that you create separate accounts for the source and destination databases, even if the databases are in the same Snowflake account. This allows each account to have the minimum permissions needed in the source and destination databases.

If you use the same account, then you must combine the relevant permissions for the source and destination databases into a single role. Snowflake only allows each account to have a single active primary role.

User permissions on the source database

The below permissions create a role with the necessary permissions to act as the source database user. A user is then created and assigned into that role.

The below permissions give read-only data access to a specific database on all current and future schemas, tables, and sequences.

It also grants the role access to a specified warehouse.

CREATE ROLE TONIC_SOURCE_DATABASE_ROLE;

GRANT USAGE ON WAREHOUSE <warehouse name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;

GRANT USAGE ON DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;

GRANT USAGE ON ALL SCHEMAS IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;
GRANT USAGE ON FUTURE SCHEMAS IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;

GRANT SELECT ON FUTURE TABLES IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;
GRANT SELECT ON ALL TABLES IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;

GRANT USAGE ON FUTURE SEQUENCES IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;
GRANT USAGE ON ALL SEQUENCES IN DATABASE <source database name> TO ROLE TONIC_SOURCE_DATABASE_ROLE;

CREATE USER <user name> password='<password>' default_role = TONIC_SOURCE_DATABASE_ROLE default_warehouse = <warehouse name>;

GRANT ROLE TONIC_SOURCE_DATABASE_ROLE TO USER <user name>;


-- Structural requires this parameter to be set at either the account or user level.
ALTER USER <user name> set QUOTED_IDENTIFIERS_IGNORE_CASE = false;

User permissions on the destination database

The destination database must exist before Structural can connect to it. The user that you provide to Tonic Structural for connecting to the destination database should be the owner of the database and of all of the objects in the database.

We suggest that you first create a specific Structural destination database user. Then create the destination database from that user's account. If you create the database with another account such as ACCOUNTADMIN, then you must transfer ownership of the database and all of its objects to the new account.

As part of a Structural data generation job, this user must be able to:

• DROP and then create schemas on the output database

• Copy data from Azure Blob Storage into tables in the database

The below permissions create a role with the necessary permissions to act as the destination database user. A user is then created and assigned into that role.

The following permissions gives full access to the destination database. They also grant the role access to a specified warehouse. To accommodate a situation where the database was created by another user such as ACCOUNTADMIN, this includes the required transfer of ownership to the role.

CREATE ROLE TONIC_DESTINATION_DATABASE_ROLE;

GRANT OWNERSHIP ON DATABASE <destination database name> TO TONIC_DESTINATION_DATABASE_ROLE;
GRANT USAGE ON WAREHOUSE <warehouse name> TO TONIC_DESTINATION_DATABASE_ROLE;

CREATE USER <user name> password='<password>' default_role = TONIC_DESTINATION_DATABASE_ROLE default_warehouse = <warehouse name>;
GRANT ROLE TONIC_DESTINATION_DATABASE_ROLE TO USER <user name>;

-- If the destination database was already created with another user, 
-- existing schemas (including default PUBLIC) must have ownership transferred
-- to the Structural user role.
GRANT OWNERSHIP on SCHEMA <destination database name>.PUBLIC to TONIC_DESTINATION_DATABASE_ROLE;
GRANT OWNERSHIP on SCHEMA <destination database name>.<additional schemas> to TONIC_DESTINATION_DATABASE_ROLE;


-- Structural requires this parameter to be set at either the account or user level.
ALTER USER <user name> set QUOTED_IDENTIFIERS_IGNORE_CASE = false;

Additional user configuration

Structural requires that the parameter QUOTED_IDENTIFIERS_IGNORE_CASE = false at either the account or user level.

To set it at the user level, run:

ALTER USER <user name> set QUOTED_IDENTIFIERS_IGNORE_CASE = false;

Setting the access key for the Azure Blob Storage account

Structural uses an Azure storage account to load and unload data to and from Azure Blob Storage. You provide the storage account name in the workspace configuration.

You provide the access key for the storage account as the value of the environment setting TONIC_AZURE_BLOB_STORAGE_ACCOUNT_KEY. See Configuring environment settings.

You can obtain the access key from the storage account portal under Access keys. Structural uses this key to authorize API calls when it unloads and loads data to and from Azure Blob Storage.

Configuring whether Structural creates the destination database schema

By default, during each data generation job, Structural creates the database schema for the destination database tables, then populates the database tables based on the workspace configuration.

If you prefer to manage the destination database schema yourself, then set the environment setting TONIC_SNOWFLAKE_SKIP_CREATE_DB to true. You can add this setting manually to the Environment Settings list on Structural Settings.

When TONIC_SNOWFLAKE_SKIP_CREATE_DBis true, then Structural does not create the destination database schema. Before you run data generation, you must create the destination database with the full schema.

During data generation, Structural deletes the data from the destination database tables, except for tables that use Preserve Destination mode. It then populates the tables with the new destination data.

Table mode limitations

Snowflake on Azure workspaces cannot use the following table modes:

• Scale

• Incremental

Generator limitations

Snowflake on Azure workspaces cannot use the following generators:

• AI Synthesizer

• Algebraic

• Array JSON Mask

• Array Regex Mask

• Cross Table Sum

• Current Date

• Event Timestamps

• Geo

• Sequential Integer

Support for table filtering

Snowflake on Azure workspaces support both subsetting and table filtering.

No upsert

Snowflake on Azure workspaces do not support upsert.

No output to container artifacts

For Snowflake on Azure workspaces, you cannot write the destination data to container artifacts.

No output to an Ephemeral snapshot

For Snowflake on Azure workspaces, you cannot write the destination data to an Ephemeral snapshot.

Snowflake is a cloud-based data warehousing platform.

Tonic Structural supports moving data from one database to another within a single Snowflake instance. Structural can also move data between Snowflake instances.

For the Snowflake on Azure data connector, Structural uses Azure Blob Storage as an intermediate stage to host both the original data and the masked data.