Configuring the file connector storage type and output options

On the workspace details view for a file connector workspace, you:

• Identify the type of storage. After you add a file group to the workspace, you cannot change the storage type.

• Indicate where to write the transformed files.

• If needed, provide credentials to access the cloud storage.

Identifying the type of storage

On the workspace creation view:

1. Under Connection Type, under File/Blob Storage, click Files.

2. Select the type of file storage where the source files are located.

   • To choose files from Amazon S3, click Amazon S3.

   • To choose files from MinIO, make sure that the TONIC_AWS_S3_OVERRIDE_URL environment setting points to your MinIO endpoint, then click Amazon S3.

   • To choose files from GCS, click Google Cloud Storage.

   • To upload files from a local file system, click Local Filesystem.

   • To choose files from a local file mount, click File Mount. The file mount option is not available on Structural Cloud. If you configured a single file mount path, then that path is used. You cannot specify the path. Otherwise, in the Source File Mount Path field, provide the file mount path where the source files are located. The file mount path must be accessible by the container that runs the Structural application.

   After you add a file group to the workspace, you cannot change the storage type.

Selecting the location for the transformed files

Local files

When the source files come from a local file system, Tonic Structural writes the output files to the large file store in the Structural application database. You can then download the most recently generated files.

Cloud storage

For cloud storage workspaces, in the Output location field, provide the path to the folder where Structural writes the transformed files.

File mount

For files that come from a local file mount, you can write the output files to one of the following:

• An S3 bucket

• Google Cloud Storage

• A file mount

S3 bucket or Google Cloud Storage

For S3 buckets and Google Cloud Storage, in the Output location field, provide the path to the folder where Structural writes the transformed files.

File mount - single file mount path

For a file mount, if you configured a single file mount path, then in the Output location field, provide the location within the file mount where Structural writes the transformed files.

File mount - no single file mount path

If you did not configure a single file mount path:

1. By default, the files are written to the same file mount path where the source files are located. To use a different file mount path:

   1. Toggle Set different mount for output to the on position.

   2. In the Destination File Mount Path field, provide the file mount path. The file mount path must be accessible by the container that runs the Structural application.

2. In the Output location field, provide the location within the file mount where Structural writes the transformed files.

Providing credentials to access AWS

For a file connector workspace that writes files to Amazon S3, under AWS Credentials, you configure how Structural obtains the credentials to connect to Amazon S3.

Selecting the type of credentials to use

Under AWS Credentials, click the type of credentials to use. The options are:

• Environment - Only available on self-hosted instances. Indicates to use either:

  • The credentials for the IAM role on the host machine.

  • The credentials set in the following environment settings:

    • TONIC_AWS_ACCESS_KEY_ID - An AWS access key that is associated with an IAM user or role.

    • TONIC_AWS_SECRET_ACCESS_KEY - The secret key that is associated with the access key.

    • TONIC_AWS_REGION - The AWS Region to send the authentication request to.

• Assumed role - Indicates to use the specified assumed role.

• User credentials - Indicates to use the provided user credentials.

Providing an assumed role

To provide an assumed role, click Assume role, then:

1. In the Role ARN field, provide the Amazon Resource Name (ARN) for the role.

2. In the Session Name field, provide the role session name. If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with TonicStructural.

3. In the Duration (in seconds) field, provide the maximum length in seconds of the session. The default is 3600, indicating that the session can be active for up to 1 hour. The provided value must be less than the maximum session duration that is allowed for the role.

4. By default, Structural uses the same assumed role to both retrieve the source files and write the output files. To provide a different assumed role for the output location:

   1. Toggle Set different credentials for output to the on position.

   2. In the Role ARN field, provide the ARN for the role.

   3. In the Session Name field, provide the role session name. If you do not provide a session name, then Structural automatically generates a default unique value. The generated value begins with TonicStructural.

   4. In the Duration (in seconds) field, provide the maximum length in seconds of the session. The default is 3600, indicating that the session can be active for up to 1 hour. The provided value must be less than the maximum session duration that is allowed for the role.

For each assumed role, Structural generates the external ID that is used in the assume role request. Your role’s trust policy must be configured to condition on your unique external ID.

Here is an example trust policy:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<originating-account-id>"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "<external-id>"
      }
    }
  }
}

Providing the AWS credentials

To provide the credentials, under AWS Credentials:

1. In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.

2. In the AWS Secret Key field, enter the secret key that is associated with the access key.

3. From the AWS Region dropdown list, select the AWS Region to send the authentication request to.

4. By default, Structural uses the same AWS credentials to both retrieve the source files and write the output files. To provide different AWS credentials for the output location:

   1. Toggle Set different credentials for output to the on position.

   2. In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.

   3. In the AWS Secret Key field, enter the secret key that is associated with the access key.

   4. From the AWS Region dropdown list, select the AWS Region to send the authentication request to.

5. In the AWS Session Token field, you can optionally provide a session token for a temporary set of credentials. You can provide a session token regardless of whether you use the same or different credentials for the source and output.

Providing credentials to access Google Cloud Storage

To write files to a folder in Google Cloud Storage, you must provide Google Cloud Platform credentials in the workspace configuration.

Under GCP Credentials:

1. For Service Account File, select the service account file (JSON file) for the source files.

2. In the GCP Project ID field, provide the identifier of the project that contains the source files.

Providing credentials to access MinIO

When the environment setting TONIC_AWS_S3_OVERRIDE_URL points to a MinIO endpoint, then when you select Amazon S3 as the source, you create a MinIO workspace.

Under AWS credentials, you provide the MinIO credentials. The MinIO credentials consist of an access key and a secret key.

To provide the credentials, you can either:

• (Self-hosted only) Use the credentials set in the following environment settings:

  • TONIC_AWS_ACCESS_KEY_ID - A MinIO access key

  • TONIC_AWS_SECRET_ACCESS_KEY - The secret key that is associated with the access key

• Provide the access key and secret key manually

To use the credentials from the environment settings, under AWS Credentials, click Environment.

To provide the credentials manually:

1. Under AWS Credentials, click User credentials.

2. In the AWS Access Key field, enter the MinIO access key.

3. In the AWS Secret Key field, enter the secret key that is associated with the access key.

4. By default, Structural uses the same credentials to both retrieve the source files and write the output files. To provide different MinIO credentials for the output location:

   1. Toggle Set different credentials for output to the on position.

   2. In the AWS Access Key field, enter the MinIO access key.

   3. In the AWS Secret Key field, enter the secret key that is associated with the access key.