Configuring Snowflake on AWS workspace data connections
Last updated
Was this helpful?
Last updated
Was this helpful?
In the workspace configuration, under Connection Type, select Snowflake.
In the Source Settings section, under Snowflake Type, click AWS.
In the Source Settings section, provide the details for the connection to the source database.
To connect to the source database, you can either:
Populate the connection fields.
Use a connection string.
You can also use key pair authentication instead of a password.
By default, Use connection string is off, and you provide the connection values in the individual fields:
In the Server field, provide the server where the database is located.
You must provide the full path to the server. https:// is optional.
So the format of the server value can be either:
<account>.<region>.snowflakecomputing.com
https://<account>.<region>.snowflakecomputing.com
For example: abc123456.us-east-1.snowflakecomputing.com or https://abc123456.us-east-1.snowflakecomputing.com
In the Database field, provide the name of the database.
In the Username field, provide the username for the account to use to connect to the database.
For the user password, you can either specify the password manually, or you can select a secret name from a secrets manager. The selected secret must store a password. The secrets manager option only displays if at least one secrets manager is configured. For information about configuring the available secrets managers, go to Configuring secrets managers for database connections. To enter the password manually:
Click Provide Password.
In the password field, enter the password.
To use a secret name from a secrets manager:
Click Use Secrets Manager.
From the secrets manager dropdown list, select the secrets manager. Structural connects to the secrets manager and retrieves a list of available secret names.
From the secret name dropdown list, select the secret name.
To use a connection string to connect to the source database:
Toggle Use connection string to the on position.
In the Connection String field, provide the connection string.
For the password, you can either specify the password manually, or you can select a secret name from a secrets manager. The selected secret must store a password. The secrets manager option only displays if at least one secrets manager is configured. For information about configuring the available secrets managers, go to Configuring secrets managers for database connections. To enter the password manually:
Click Provide Password.
In the password field, enter the password.
To use a secret name from a secrets manager:
Click Use Secrets Manager.
From the secrets manager dropdown list, select the secrets manager. Structural connects to the secrets manager and retrieves a list of available secret names.
From the secret name dropdown list, select the secret name.
The connection string uses the following format:
Instead of providing a password, you can use key pair authentication.
To do this:
Toggle Use Key Pair Authentication to the on position.
Expand the Key Pair Authentication Settings.
For RSA Private Key, click Browse, then select the key file.
If the key is encrypted, then in the Encrypted Key Passphrase field, provide the passphrase to use to decrypt the key.
To trust the server certificate, and ignore the certificate authority's revocation list, toggle Trust Server Certificate to the on position.
This option can be useful when your Tonic Structural instance cannot connect to the certificate authority.
You can use a proxy server to connect to the source database.
To use a proxy server to connect to the source database:
Toggle Enable proxy connection to the on position.
In the Proxy Host field, provide the host name for the proxy connection.
In the Proxy Port field, provide the port for the proxy connection.
Optionally, in the Proxy User field, provide the name of the user for the proxy connection.
If you provide a proxy user, then in the Proxy Password field, provide the password for the specified user.
Optionally, in the Non-Proxy Hosts field, provide the list of hosts for which to bypass the proxy server and connect to directly.
Use a pipe symbol (|) to separate the host names. For example, host1|host2|host3.
You can also use an asterisk (*) as a wildcard. For example, to connect directly to all hosts with host names that start with myhost, use myhost*.
By default, the source database includes all of the schemas. To specify a list of specific schemas to either include or exclude:
Toggle Limit Schemas to the on position.
From the filter option dropdown list, select whether to include or exclude the listed schemas.
In the field, provide the list of schemas to either include or exclude. Use commas or semicolons to separate the schemas.
Do not exclude schemas that are referred to by included schemas, unless you create those schemas manually outside of Structural.
To test the connection to the source database, click Test Source Connection.
By default, data generation is not blocked for schema changes that do not conflict with your workspace configuration.
To block data generation when there are any schema changes, regardless of whether they conflict with your workspace configuration, toggle Block data generation on schema changes to the on position.
The default data generation process for Snowflake on AWS cannot scale to extremely large volumes of data. For volumes of hundreds of gigabytes or larger, you must use the Lambda-based processing.
To enable Lambda processing, toggle Enable Lambda generation to the on position.
In the Destination Settings section, you specify the connection information for the destination database.
If the destination database is in the same location as the source database, then you can copy the connection and authentication details from the source database. The copied details include the proxy connection configuration.
If the destination database is in a different location, then you can either:
Populate the connection fields.
Use a connection string.
You can also use key pair authentication instead of a password.
To copy the connection details from the source database:
Click Copy Settings from Source.
For the user password, you can either specify the password manually, or you can select a secret name from a secrets manager. The selected secret must store a password. The secrets manager option only displays if at least one secrets manager is configured. For information about configuring the available secrets managers, go to Configuring secrets managers for database connections. To enter the password manually:
Click Provide Password.
In the password field, enter the password.
To use a secret name from a secrets manager:
Click Use Secrets Manager.
From the secrets manager dropdown list, select the secrets manager. Structural connects to the secrets manager and retrieves a list of available secret names.
From the secret name dropdown list, select the secret name.
To test the connection to the destination database, click Test Destination Connection.
If you do not copy the details from the source database, then you can either populate the connection fields or use a connection string.
By default, Use connection string is off, and you provide the connection values in the individual fields:
In the Server field, provide the server where the database is located.
You must provide the full path to the server. The https:// is optional.
So the format of the server value can be either:
<account>.<region>.snowflakecomputing.com
https://<account>.<region>.snowflakecomputing.com
For example: abc123456.us-east-1.snowflakecomputing.com or https://abc123456.us-east-1.snowflakecomputing.com
In the Database field, provide the name of the database.
In the Username field, provide the username for the account to use to connect to the database.
For the user password, you can either specify the password manually, or you can select a secret name from a secrets manager. The selected secret must store a password. The secrets manager option only displays if at least one secrets manager is configured. For information about configuring the available secrets managers, go to Configuring secrets managers for database connections. To enter the password manually:
Click Provide Password.
In the password field, enter the password.
To use a secret name from a secrets manager:
Click Use Secrets Manager.
From the secrets manager dropdown list, select the secrets manager. Structural connects to the secrets manager and retrieves a list of available secret names.
From the secret name dropdown list, select the secret name.
To use a connection string to connect to the destination database:
Toggle Use connection string to the on position.
In the Connection String field, provide the connection string.
For the password, you can either specify the password manually, or you can select a secret name from a secrets manager. The selected secret must store a password. The secrets manager option only displays if at least one secrets manager is configured. For information about configuring the available secrets managers, go to Configuring secrets managers for database connections. To enter the password manually:
Click Provide Password.
In the password field, enter the password.
To use a secret name from a secrets manager:
Click Use Secrets Manager.
From the secrets manager dropdown list, select the secrets manager. Structural connects to the secrets manager and retrieves a list of available secret names.
From the secret name dropdown list, select the secret name.
The connection string uses the following format:
Instead of providing a password, you can use key pair authentication.
To do this:
Toggle Use Key Pair Authentication to the on position.
Expand the Key Pair Authentication Settings.
For RSA Private Key, click Browse, then select the key file.
If the key is encrypted, then in the Encrypted Key Passphrase field, provide the passphrase to use to decrypt the key.
To test the connection to the destination database, click Test Destination Connection.
To trust the server certificate, and ignore the certificate authority's revocation list, toggle Trust Server Certificate to the on position.
This option can be useful when your Structural instance cannot connect to the certificate authority.
You can use a proxy server to connect to the destination database.
To enable and configure the proxy connection:
Toggle Enable proxy connection to the on position.
In the Proxy Host field, provide the host name for the proxy connection.
In the Proxy Port field, provide the port for the proxy connection.
Optionally, in the Proxy User field, provide the name of the user for the proxy connection.
If you provide a proxy user, then in the Proxy Password field, provide the password for the specified user.
Optionally, in the Non-Proxy Hosts field, provide the list of hosts for which to bypass the proxy server and connect to directly.
Use a pipe symbol (|) to separate the host names. For example, host1|host2|host3.
You can also use an asterisk (*) as a wildcard. For example, to connect directly to all hosts whose host names start with myhost, use myhost*.
During data generation, Structural uses temporary CSV files to load and unload Snowflake tables.
For Lambda processing, you specify a single S3 bucket path.
If you do not use Lambda processing, then you can either:
Use external stages instead of S3 buckets.
Provide separate paths for the source and destination files.
By default, the temporary files are stored in S3 buckets.
To instead use external stages, toggle Use External Stage to the on position.
The Use External Stage toggle does not display if Enable Lambda Generation is on.
By default, you provide a single S3 bucket path or external stage. Within that path:
Structural copies the files that contain the source data into an input folder.
After it applies the generators, Structural copies the files that contain the destination data into an output folder.
To instead provide separate paths for the source and destination files, toggle Use Separate Destination Location to the on position.
The Use Separate Destination Location toggle does not display if Enable Lambda Generation is on.
If Use Separate Destination Location is off, then in the S3 Bucket Path field, specify the S3 bucket.
If Use Separate Destination Location is on, then:
In the Source S3 Bucket field, enter the path to the S3 bucket to use for the source files.
In the Destination S3 Bucket field, enter the path to the S3 bucket to use for the destination files.
If Use External Stage is on, then you provide external stage locations instead of S3 buckets. For each stage, the format is:
<database>.<schema>.<stage>
Where:
<database> is the name of the database where the stage is located.
<schema> is the name of the schema that contains the stage.
<stage> is the name of the stage.
If Use Separate Destination Location is off, then in the Source Snowflake External Stage Name field, enter the external stage.
If Use Separate Destination Location is on, then:
in the Source Snowflake External Stage Name field, enter the external stage to use for the source files.
In the Destination Snowflake External Stage Name field, enter the external stage to use for the destination files.
For each S3 bucket or external stage, you can optionally provide specific AWS credentials.
If you do not provide credentials in the workspace configuration, then Structural uses either:
TONIC_AWS_ACCESS_KEY_ID - An AWS access key that is associated with an IAM user or role.
TONIC_AWS_SECRET_ACCESS_KEY - The secret key that is associated with the access key.
TONIC_AWS_REGION - The AWS Region to send the authentication request to.
The credentials for the IAM role on the host machine.
The credentials in a credentials file.
To provide the credentials:
For the S3 bucket or external stage, to display the credentials field, click AWS Credentials.
In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.
In the AWS Secret Key field, enter the secret key that is associated with the access key.
From the AWS Region dropdown list, select the AWS Region to send the authentication request to.
If you enable Lambda processing, make sure that you , and .
The credentials set in the following :