Before you create a file connector workspace

For a file connector workspace that reads files from and writes files to Amazon S3 or Google Cloud Storage, make sure to set up the appropriate permissions so that Tonic Structural can locate the source files and write the destination files.

You can also set up permissions to protect buckets that contain files that you do not want used in a workspace.

If you have a custom gateway endpoint configured for Amazon S3, then you must identify that endpoint to Structural.

You can also enable MinIO instead of Amazon S3 as a source of file connector files.

Amazon S3 configuration

Amazon S3 permissions

On Structural Cloud, in the workspace configuration, you must configure either an assumed role or AWS credentials.

On a self-hosted instance, you can also have Structural get the credentials from the environment. Structural uses either:

• The credentials set in the following environment settings:

  • TONIC_AWS_ACCESS_KEY_ID - An AWS access key that is associated with an IAM user or role.

  • TONIC_AWS_SECRET_ACCESS_KEY - The secret key that is associated with the access key

  • TONIC_AWS_REGION - The AWS Region to send the authentication request to

• The credentials for the IAM role on the host machine.

The policy that is associated with your IAM role or IAM user must have the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        ## Lists all buckets that are tied to given AWS account.
        ## Allows Structural to view the list of buckets during file group creation.
        ## If not granted, then users must type bucket names manually.
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        ## Allows Structural to list all objects within a specified bucket.
        ## Needed to list files during file group creation.
        ## Use Resource to restrict the displayed buckets to a specific prefix 
        ## or bucket name.
        ## Use Condition to restrict the listed files to a specific prefix.
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Condition": {
                 "StringLike": {
                     "s3:prefix": "*"
                 }
             }
        },
        ## File level permissions for reading and writing.
        ## Can use Resource to restrict access based on bucket name or prefix.
        ## For example: arn:aws:s3:::bucket-prefix-*/object/prefix/*
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
}

If the source and destination S3 buckets are in different accounts, or are in an account that is different from the account or instance profile that Structural uses, then the configuration must include cross-account permissions. For assistance with this, contact support@tonic.ai.

Providing your custom Amazon S3 gateway endpoint

If you configured a custom gateway endpoint from Amazon S3, then you set that endpoint as the value of the TONIC_AWS_S3_OVERRIDE_URL environment setting.

When you configure a custom URL, then you can also configure Structural to trust the server certificate. To do this, set the TONIC_AWS_S3_TRUST_SERVER_CERT environment setting to true.

You can add these settings to the Environment Settings list on Structural Settings.

Google Cloud Storage permissions

To get access to Google Cloud Storage, Structural uses the Google Cloud Platform credentials that you provide in the workspace configuration.

The service account that you specify must have the following permissions.

• storage.buckets.list - This allows Structural to see the list of buckets when it creates a file group. If the service account does not have this permission, then on the file group creation panel, users must type the name of the bucket where a file is located.

• For buckets that contain source files, the following permissions allow Structural to get the list of files within buckets, and to retrieve the actual files and file content.

  • storage.objects.get

  • storage.objects.list

  If the permissions are assigned globally, then Structural can list and retrieve files from any bucket. If the permissions are assigned to individual buckets, the file group creation view displays a list of all buckets. However, if you select a bucket for which the service account does not have the permissions, Structural returns an error.

• For buckets that contain destination files, the following permissions allow Structural to see and get access to the bucket content and to create the generated files. This includes deleting and overwriting existing files that are regenerated.

  • storage.buckets.get

  • storage.objects.get

  • storage.objects.list

  • storage.objects.create

  • storage.objects.delete

  If the permissions are assigned globally, then Structural can write files to any bucket. If the permissions are assigned to individual buckets, then Structural can only write files to those buckets.

MinIO configuration

To use MinIO as a source for file connector files, you set the TONIC_AWS_S3_OVERRIDE_URL environment setting to your MinIO endpoint.

When you set a MinIO endpoint URL, you can also configure Structural to trust the server certificate. To do this, set the TONIC_AWS_S3_TRUST_SERVER_CERT environment setting to true.

You can add these settings to the Environment Settings list on Structural Settings.

Note that if you configure TONIC_AWS_S3_OVERRIDE_URL to point to a MinIO endpoint, then you cannot create a file connector workspace that connects to Amazon S3.