Required AWS instance profile permissions for Amazon Redshift

When it uses Amazon Redshift, Tonic Structural orchestrates the creation, usage, and deletion of AWS components.

The required permissions to do so are taken from the instance profile role of the machine that runs Structural's server. This role (EC2) needs the permissions listed below.

These policies allows Structural to properly orchestrate jobs in your AWS infrastructure. It assumes that you use default names for objects in AWS, and that your source and destination S3 buckets begin with the tonic- prefix.

Note that these permissions are starting point. Based on your exact AWS setup, you might need to add additional permissions.

Required permissions for Data Pipeline V2 data generation

By default, Amazon Redshift workspaces use the Data Pipeline V2 data generation process. For this process, the required permissions are as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification"
            ],
            "Resource": "arn:aws:s3:::tonic-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": "arn:aws:s3:::tonic-*/*"
        }
    ]
}

Required permissions for the previous data generation process

For workspaces that use the previous data generation process, which uses Amazon SQS and AWS Lambda, the required permissions are as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "lambda:CreateEventSourceMapping",
                "lambda:DeleteEventSourceMapping",
                "lambda:GetEventSourceMapping",
                "lambda:ListFunctions",
                "ecr:DescribeRepositories",
                "ecr:ListImages"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ListDeadLetterSourceQueues",
                "sqs:ListQueueTags",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SetQueueAttributes",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:PutBucketNotification",
                "lambda:CreateFunction",
                "lambda:GetFunctionConfiguration",
                "lambda:UpdateFunctionConfiguration"
            ],
            "Resource": [
                "arn:aws:sqs:*:*:tonic-*",
                "arn:aws:s3:::tonic-*",
                "arn:aws:lambda:*:*:function:tonic-*"
​
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                <ARN to Structural Lambda Role>
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": "arn:aws:s3:::tonic-*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogStreams",
                "logs:GetQueryResults",
                "logs:GetLogEvents",
                "logs:FilterLogEvents",
                "logs:StartQuery",
                "logs:StopQuery"
            ],
            "Resource": ["arn:aws:logs:${region}:${account_id}:log-group:/aws/lambda/tonic-*"]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::tonic-*",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "*"
                    ]
                }
            }
        }
    ]
}

Note that if you use AWS KMS on your S3 buckets, then you might need to grant AWS Key Management Service (AWS KMS) access. Go to the instructions on how to use AWS KMS encryption with Structural's programmatically generated Amazon SQS queues.