AWS - Setting the storage location for temporary files

During data generation, Structural uses temporary CSV files to load and unload Snowflake tables.

To store the temporary files, you can either:

• Use external stages instead of S3 buckets.
• Provide separate paths for the source and destination files.

Setting the type of storage to use

By default, the temporary files are stored in S3 buckets.

To instead use external stages, toggle Use External Stage to the on position.

Enabling separate paths for source and destination files

By default, you provide a single S3 bucket path or external stage. Within that path:

• Structural copies the files that contain the source data into an input folder.
• After it applies the generators, Structural copies the files that contain the destination data into an output folder.

To instead provide separate paths for the source and destination files, toggle Use Separate Destination Location to the on position.

Setting S3 bucket locations

When Use External Stage is off, then you provide the S3 buckets to use.

If Use Separate Destination Location is off, then in the S3 Bucket Path field, specify the S3 bucket.

If Use Separate Destination Location is on, then:

1. In the Source S3 Bucket field, enter the path to the S3 bucket to use for the source files.
2. In the Destination S3 Bucket field, enter the path to the S3 bucket to use for the destination files.

Setting external stage locations

If Use External Stage is on, then you provide external stage locations instead of S3 buckets. For each stage, the format is:

<database>.<schema>.<stage>

Where:

• <database> is the name of the database where the stage is located.
• <schema> is the name of the schema that contains the stage.
• <stage> is the name of the stage.

If Use Separate Destination Location is off, then in the Source Snowflake External Stage Name field, enter the external stage.

If Use Separate Destination Location is on, then:

1. in the Source Snowflake External Stage Name field, enter the external stage to use for the source files.
2. In the Destination Snowflake External Stage Name field, enter the external stage to use for the destination files.

Providing AWS credentials for storage locations

For each S3 bucket or external stage, you provide the credentials to use connect to it.

Selecting the type of credentials to use

Under AWS Credentials, click the type of credentials to use. The options are:

• Environment - Only available on self-hosted instances. Indicates to use either:

  • The credentials for the IAM role on the host machine.
  • The credentials set in the following environment settings:

    • TONIC_AWS_ACCESS_KEY_ID - An AWS access key that is associated with an IAM user or role.
    • TONIC_AWS_SECRET_ACCESS_KEY - The secret key that is associated with the access key.
    • TONIC_AWS_REGION - The AWS Region to send the authentication request to.
  • The credentials in a credentials file.
• Assume role - Indicates to use a specified assumed role.
• User credentials - Indicates to use the provided user credentials.

Providing an assumed role

To provide an assumed role, click Assume Role, then:

1. In the Role ARN field, provide the Amazon Resource Name (ARN) for the role.
2. In the Session Name field, provide the role session name. If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with TonicStructural.
3. In the Duration (in seconds) field, provide the maximum length in seconds of the session. The default is 3600, indicating that the session can be active for up to 1 hour. The provided value must be less than the maximum session duration that is allowed for the role.
4. From the AWS Region dropdown list, select the AWS Region to send the authentication request to.

For each assumed role, Structural generates the external ID that is used in the assume role request. Your role’s trust policy must be configured to condition on your unique external ID.

Here is an example trust policy:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<originating-account-id>"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "7aab6da2-62ed-fae9-efb5-03b3b4daed8a"
      }
    }
  }
}

Providing AWS user credentials

To provide AWS credentials, click User Credentials, then:

1. In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.
2. In the AWS Secret Key field, enter the secret key that is associated with the access key.
3. From the AWS Region dropdown list, select the AWS Region to send the authentication request to.

Testing the storage location connection

To verify that Structural is able to use the provided credentials to connect to the storage location, click Test Source Object Storage.