1 of 4

Before you create an Amazon EMR workspace

Before you create a workspace that uses the Amazon EMR data connector, you need to complete the following configuration.

Creating IAM roles for Structural and Amazon EMR

Tonic Structural with Amazon EMR uses the following IAM roles:

A role that is used by the Structural server
A role that is used by the Amazon EC2 instance profile for the Amazon EMR cluster

These roles must have the required permissions in order for Structural to run successfully on Amazon EMR data. Each role must be assigned to the appropriate profile.

If the Glue catalog is under a different AWS account from the one used for Structural and Amazon EMR, then use the instructions in Configuration for cross-account setups.

Structural server role for Amazon EMR

Identifying the profile that has the Structural server role

By default, Structural uses the IAM profile that is attached to the instance where Structural runs.

If you do not want to use that IAM profile, then to identify the profile to use:

Set the environment setting TONIC_AWS_ACCESS_KEY_ID to the AWS access key that is associated with the IAM profile.
Set the environment setting TONIC_AWS_SECRET_ACCESS_KEY to the secret key that is associated with the access key.

For information about how to configure Structural environment settings, go to Configuring environment settings.

Required permissions for the Structural server role

The Structural server role must have the the following permissions:

/*Ability to launch jobs via EMR Steps API and also to view clusters*/
{
    "Sid": "EmrListClustersPerms",
    "Effect": "Allow",
    "Action": "elasticmapreduce:ListClusters",
    "Resource": "*"
},
{
    "Sid": "EmrPerms",
    "Effect": "Allow",
    "Action": [
        "elasticmapreduce:DescribeStep",
        "elasticmapreduce:AddJobFlowSteps",
        "elasticmapreduce:DescribeCluster"
    ],
    "Resource": [
        "arn:aws:elasticmapreduce:<region>:<account-id>:cluster/<emr-cluster-id>"
    ]
},

/*Ability to query Glue catalog*/
{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:database/*",
        "arn:aws:glue:<region>:<account-id>:table/*"
    ]
},
{
    "Sid": "S3SourcePermissions",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<source-s3-bucket>",
        "arn:aws:s3:::<source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestPermissions",
    "Effect": "Allow",
    "Action": [
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
}

/*Ability to issue queries against Athena*/
{
    "Sid": "AthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "athena:GetWorkGroup"
    ],
    "Resource": "arn:aws:athena:<region>:<account-id>:workgroup/tonic-emr-workgroup"
},
{
    "Sid": "AthenaQueryResultPermissions",
    "Effect": "Allow",
    "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload",
        "s3:CreateBucket",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<athena-query-results-bucket>",
        "arn:aws:s3:::<athena-query-results-bucket>/*"
    ]
}

Amazon EC2 instance profile role

Identifying the Amazon EC2 instance profile

The profile is the Amazon EC2 instance profile that you assigned as the value of EC2 instance profile when you created the Amazon EMR cluster.

Required permissions for the Amazon EC2 instance profile role

By default, a new Amazon EMR cluster is assigned the role EMR_EC2_DefaultRole, which contains all of the required permissions, plus additional permissions.

However, AWS recommends that you create a custom IAM role for your Amazon EMR cluster's Amazon EC2 instance profile role.

The following permissions reflect the minimum permissions needed for Structural data generation:

{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:database/*",
        "arn:aws:glue:<region>:<account-id>:table/*"
    ]
},
{
    "Sid": "S3SourceBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<source-s3-bucket>",
        "arn:aws:s3:::<source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>",
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "S3EmrLogBucketPerms",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": [
        "arn:aws:s3:::<emr-logs-s3-bucket>/*"
    ]
}

For Amazon EMR, the Glue catalog must contain a default database. If the default database does not exist, then Amazon EMR attempts to create it.

Before you run a Structural data generation, you must either:

Ensure that the default catalog exists
Add glue:CreateDatabase to the list of permissions that are granted to this role

Structural does not otherwise require this permission, and does not explicitly attempt to create a database.

Decrypt and encrypt permissions

You must add decrypt and encrypt permissions to your account on both the source and destination paths.

If you use the Amazon EMR Steps API, then the Amazon EMR role that is assigned to your cluster must have Decrypt access to the AWS KMS key that is used on the output bucket.

Creating Athena workgroups

Tonic Structural uses Amazon Athena to run queries against your AWS Glue catalog. These queries are typically for powering the front-end experience. They are also used in conjunction with gathering data used by certain generators during a data generation.

To use Athena, you must set up the required permissions, and also create an Athena Workgroup. You can do this from the Athena homepage in the AWS Console.

By default, Structural expects the WorkGroup name to be tonic-emr-workgroup. To override the default value, configure the following environment setting:

TONIC_EMR_ATHENA_WORKGROUP

If you do override this value, you must override it in both the Structural web server container and on the Amazon EMR cluster.

Configuration for cross-account setups

Tonic Structural supports operating on AWS Glue catalogs in AWS accounts different from where Structural and Amazon EMR are configured.

For the instructions in this topic, we'll use the following example:

AWS Account A contains the Amazon EMR Cluster, Athena workgroup, and destination S3 bucket.
AWS Account B contains the AWS Glue data catalog and source S3 bucket.

The following instructions explain how to set up each required AWS component. For cross-account setups, you use these instructions instead of the instructions in Creating IAM roles for Structural and Amazon EMR.

These instructions assume that both accounts reside in the same region. If your accounts belong to different regions, then see the Amazon documentation for instructions on how to set up a VPC for cross-account access.

Granting access to the required resources

The account that has Structural and Amazon EMR must be granted accesses to the resources for the account that has the AWS Glue catalog.

To continue our example, you must first grant Account A access to Account B's resources. To do this, set up the following resource-based policies for Account B's AWS Glue data catalog and source S3 bucket.

Account B Glue data catalog resource policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account-A-id>:role/<tonic-role>",
                    "arn:aws:iam::<account-A-id>:role/<emr-ec2-instance-profile-role>"
                ]
            },
            "Action": [
                "glue:GetUserDefinedFunctions",
                "glue:BatchGetPartition",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetTableVersion",
                "glue:GetTableVersions"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account-B-id>:catalog",
                "arn:aws:glue:<region>:<account-B-id>:database/*",
                "arn:aws:glue:<region>:<account-B-id>:table/*"
            ]
        }
    ]
}

Register Account B's glue data catalog as an Athena data source inside Account A's Athena console. For instructions, see the AWS documentation.

Account B source S3 bucket bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account-A-id>:role/<tonic-role>",
                    "arn:aws:iam::<account-A-id>:role/<emr-ec2-instance-profile-role>"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<account-B-source-bucket>",
                "arn:aws:s3:::<account-B-source-bucket>/*"
            ]
        }
    ]
}

Account A Amazon EMR cluster

When you create your Amazon EMR cluster, make sure to enable the Use AWS Glue Data Catalog for table metadata option. This allows you to set a default catalog ID that points to Account B.

You must set the following configuration for all instance groups in the Amazon EMR cluster:

[
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
      "hive.metastore.glue.catalogid": "<account-B-id>"
    }
  }
]

Structural server role

Identifying the profile that has the Structural server role

By default, Structural uses the IAM profile that is attached to the instance where Structural runs.

If you do not want to use that IAM profile, then to identify the profile to use:

Set the environment setting TONIC_AWS_ACCESS_KEY_ID to the AWS access key that is associated with the IAM profile.
Set the environment setting TONIC_AWS_SECRET_ACCESS_KEY to the secret key that is associated with the access key.

For information about how to configure Structural environment settings, go to Configuring environment settings.

Required permissions for the Structural server role

The Structural server role must have the the following permissions:

{
    "Sid": "EmrListClustersPerms",
    "Effect": "Allow",
    "Action": "elasticmapreduce:ListClusters",
    "Resource": "*"
},
{
    "Sid": "EmrPerms",
    "Effect": "Allow",
    "Action": [
        "elasticmapreduce:DescribeStep",
        "elasticmapreduce:AddJobFlowSteps",
        "elasticmapreduce:DescribeCluster"
    ],
    "Resource": [
        "arn:aws:elasticmapreduce:<region>:<account-A-id>:cluster/<cluster-id>"
    ]
},
{
    "Sid": "CrossAccountGluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-B-id>:catalog",
        "arn:aws:glue:<region>:<account-B-id>:database/*",
        "arn:aws:glue:<region>:<account-B-id>:table/*"
    ]
},
{
    "Sid": "CrossAccountS3SourcePerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<account-B-source-s3-bucket>",
        "arn:aws:s3:::<account-B-source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestPerms",
    "Effect": "Allow",
    "Action": [
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "AthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults"
    ],
    "Resource": "arn:aws:athena:<region>:<account-A-id>:workgroup/tonic-emr-workgroup"
},
{
    "Sid": "CrossAccountAthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:GetDataCatalog"
    ],
    "Resource": "arn:aws:athena:<region>:<account-A-id>:datacatalog/<catalog-name>"
},
{
    "Sid": "AthenaQueryResultPerms",
    "Effect": "Allow",
    "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload",
        "s3:CreateBucket",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<athena-query-results-bucket>",
        "arn:aws:s3:::<athena-query-results-bucket>/*"
    ]
}

Amazon EC2 instance profile role

Identifying the profile that has the Amazon EC2 instance role

The profile is the Amazon EC2 instance profile that you assigned as the value of EC2 instance profile when you created the Amazon EMR cluster.

Required permissions for the Amazon EC2 instance role

By default, a new Amazon EMR cluster is assigned the role EMR_EC2_DefaultRole, which contains all of the required permissions, plus additional permissions.

However, AWS recommends that you create a custom IAM role for your Amazon EMR cluster's Amazon EC2 instance profile role.

The following permissions reflect the minimum permissions needed for Structural data generation:

{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:CreateDatabase",
        "glue:UpdateDatabase",
        "glue:DeleteDatabase",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:CreateTable",
        "glue:UpdateTable",
        "glue:DeleteTable",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersions",
        "glue:CreatePartition",
        "glue:BatchCreatePartition",
        "glue:UpdatePartition",
        "glue:DeletePartition",
        "glue:BatchDeletePartition",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:BatchGetPartition",
        "glue:CreateUserDefinedFunction",
        "glue:UpdateUserDefinedFunction",
        "glue:DeleteUserDefinedFunction",
        "glue:GetUserDefinedFunction",
        "glue:GetUserDefinedFunctions"
    ],
    "Resource": "*"
},
{
    "Sid": "CrossAccountS3SourceBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<account-B-source-s3-bucket>",
        "arn:aws:s3:::<account-B-source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>",
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "S3EmrLogBucketPerms",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": [
        "arn:aws:s3:::<s3-emr-log-bucket>/*"
    ]
}

For Amazon EMR, the Glue catalog must contain a default database. If the default database does not exist, then Amazon EMR attempts to create it.

Before you run a Structural data generation, you must either:

Ensure that the default catalog exists
Add glue:CreateDatabase to the list of permissions that are granted to this role

Structural does not otherwise require this permission, and does not explicitly attempt to create a database.

Configuration for cross-account setups

Tonic Structural supports operating on AWS Glue catalogs in AWS accounts different from where Structural and Amazon EMR are configured.

For the instructions in this topic, we'll use the following example:

AWS Account A contains the Amazon EMR Cluster, Athena workgroup, and destination S3 bucket.
AWS Account B contains the AWS Glue data catalog and source S3 bucket.

Granting access to the required resources

The account that has Structural and Amazon EMR must be granted accesses to the resources for the account that has the AWS Glue catalog.

Account B Glue data catalog resource policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account-A-id>:role/<tonic-role>",
                    "arn:aws:iam::<account-A-id>:role/<emr-ec2-instance-profile-role>"
                ]
            },
            "Action": [
                "glue:GetUserDefinedFunctions",
                "glue:BatchGetPartition",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetTableVersion",
                "glue:GetTableVersions"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account-B-id>:catalog",
                "arn:aws:glue:<region>:<account-B-id>:database/*",
                "arn:aws:glue:<region>:<account-B-id>:table/*"
            ]
        }
    ]
}

Register Account B's glue data catalog as an Athena data source inside Account A's Athena console. For instructions, see the AWS documentation.

Account B source S3 bucket bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account-A-id>:role/<tonic-role>",
                    "arn:aws:iam::<account-A-id>:role/<emr-ec2-instance-profile-role>"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<account-B-source-bucket>",
                "arn:aws:s3:::<account-B-source-bucket>/*"
            ]
        }
    ]
}

Account A Amazon EMR cluster

When you create your Amazon EMR cluster, make sure to enable the Use AWS Glue Data Catalog for table metadata option. This allows you to set a default catalog ID that points to Account B.

You must set the following configuration for all instance groups in the Amazon EMR cluster:

[
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
      "hive.metastore.glue.catalogid": "<account-B-id>"
    }
  }
]

Structural server role

Identifying the profile that has the Structural server role

By default, Structural uses the IAM profile that is attached to the instance where Structural runs.

If you do not want to use that IAM profile, then to identify the profile to use:

Set the environment setting TONIC_AWS_ACCESS_KEY_ID to the AWS access key that is associated with the IAM profile.
Set the environment setting TONIC_AWS_SECRET_ACCESS_KEY to the secret key that is associated with the access key.

For information about how to configure Structural environment settings, go to Configuring environment settings.

Required permissions for the Structural server role

The Structural server role must have the the following permissions:

{
    "Sid": "EmrListClustersPerms",
    "Effect": "Allow",
    "Action": "elasticmapreduce:ListClusters",
    "Resource": "*"
},
{
    "Sid": "EmrPerms",
    "Effect": "Allow",
    "Action": [
        "elasticmapreduce:DescribeStep",
        "elasticmapreduce:AddJobFlowSteps",
        "elasticmapreduce:DescribeCluster"
    ],
    "Resource": [
        "arn:aws:elasticmapreduce:<region>:<account-A-id>:cluster/<cluster-id>"
    ]
},
{
    "Sid": "CrossAccountGluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-B-id>:catalog",
        "arn:aws:glue:<region>:<account-B-id>:database/*",
        "arn:aws:glue:<region>:<account-B-id>:table/*"
    ]
},
{
    "Sid": "CrossAccountS3SourcePerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<account-B-source-s3-bucket>",
        "arn:aws:s3:::<account-B-source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestPerms",
    "Effect": "Allow",
    "Action": [
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "AthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults"
    ],
    "Resource": "arn:aws:athena:<region>:<account-A-id>:workgroup/tonic-emr-workgroup"
},
{
    "Sid": "CrossAccountAthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:GetDataCatalog"
    ],
    "Resource": "arn:aws:athena:<region>:<account-A-id>:datacatalog/<catalog-name>"
},
{
    "Sid": "AthenaQueryResultPerms",
    "Effect": "Allow",
    "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload",
        "s3:CreateBucket",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<athena-query-results-bucket>",
        "arn:aws:s3:::<athena-query-results-bucket>/*"
    ]
}

Amazon EC2 instance profile role

Identifying the profile that has the Amazon EC2 instance role

The profile is the Amazon EC2 instance profile that you assigned as the value of EC2 instance profile when you created the Amazon EMR cluster.

Required permissions for the Amazon EC2 instance role

By default, a new Amazon EMR cluster is assigned the role EMR_EC2_DefaultRole, which contains all of the required permissions, plus additional permissions.

However, AWS recommends that you create a custom IAM role for your Amazon EMR cluster's Amazon EC2 instance profile role.

The following permissions reflect the minimum permissions needed for Structural data generation:

{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:CreateDatabase",
        "glue:UpdateDatabase",
        "glue:DeleteDatabase",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:CreateTable",
        "glue:UpdateTable",
        "glue:DeleteTable",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersions",
        "glue:CreatePartition",
        "glue:BatchCreatePartition",
        "glue:UpdatePartition",
        "glue:DeletePartition",
        "glue:BatchDeletePartition",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:BatchGetPartition",
        "glue:CreateUserDefinedFunction",
        "glue:UpdateUserDefinedFunction",
        "glue:DeleteUserDefinedFunction",
        "glue:GetUserDefinedFunction",
        "glue:GetUserDefinedFunctions"
    ],
    "Resource": "*"
},
{
    "Sid": "CrossAccountS3SourceBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<account-B-source-s3-bucket>",
        "arn:aws:s3:::<account-B-source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>",
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "S3EmrLogBucketPerms",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": [
        "arn:aws:s3:::<s3-emr-log-bucket>/*"
    ]
}

For Amazon EMR, the Glue catalog must contain a default database. If the default database does not exist, then Amazon EMR attempts to create it.

Before you run a Structural data generation, you must either:

Ensure that the default catalog exists
Add glue:CreateDatabase to the list of permissions that are granted to this role

Structural does not otherwise require this permission, and does not explicitly attempt to create a database.

Creating IAM roles for Structural and Amazon EMR

Tonic Structural with Amazon EMR uses the following IAM roles:

A role that is used by the Structural server
A role that is used by the Amazon EC2 instance profile for the Amazon EMR cluster

These roles must have the required permissions in order for Structural to run successfully on Amazon EMR data. Each role must be assigned to the appropriate profile.

If the Glue catalog is under a different AWS account from the one used for Structural and Amazon EMR, then use the instructions in Configuration for cross-account setups.

Structural server role for Amazon EMR

Identifying the profile that has the Structural server role

By default, Structural uses the IAM profile that is attached to the instance where Structural runs.

If you do not want to use that IAM profile, then to identify the profile to use:

Set the environment setting TONIC_AWS_ACCESS_KEY_ID to the AWS access key that is associated with the IAM profile.
Set the environment setting TONIC_AWS_SECRET_ACCESS_KEY to the secret key that is associated with the access key.

For information about how to configure Structural environment settings, go to Configuring environment settings.

Required permissions for the Structural server role

The Structural server role must have the the following permissions:

/*Ability to launch jobs via EMR Steps API and also to view clusters*/
{
    "Sid": "EmrListClustersPerms",
    "Effect": "Allow",
    "Action": "elasticmapreduce:ListClusters",
    "Resource": "*"
},
{
    "Sid": "EmrPerms",
    "Effect": "Allow",
    "Action": [
        "elasticmapreduce:DescribeStep",
        "elasticmapreduce:AddJobFlowSteps",
        "elasticmapreduce:DescribeCluster"
    ],
    "Resource": [
        "arn:aws:elasticmapreduce:<region>:<account-id>:cluster/<emr-cluster-id>"
    ]
},

/*Ability to query Glue catalog*/
{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:database/*",
        "arn:aws:glue:<region>:<account-id>:table/*"
    ]
},
{
    "Sid": "S3SourcePermissions",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<source-s3-bucket>",
        "arn:aws:s3:::<source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestPermissions",
    "Effect": "Allow",
    "Action": [
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
}

/*Ability to issue queries against Athena*/
{
    "Sid": "AthenaPerms",
    "Effect": "Allow",
    "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "athena:GetWorkGroup"
    ],
    "Resource": "arn:aws:athena:<region>:<account-id>:workgroup/tonic-emr-workgroup"
},
{
    "Sid": "AthenaQueryResultPermissions",
    "Effect": "Allow",
    "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload",
        "s3:CreateBucket",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<athena-query-results-bucket>",
        "arn:aws:s3:::<athena-query-results-bucket>/*"
    ]
}

Amazon EC2 instance profile role

Identifying the Amazon EC2 instance profile

The profile is the Amazon EC2 instance profile that you assigned as the value of EC2 instance profile when you created the Amazon EMR cluster.

Required permissions for the Amazon EC2 instance profile role

By default, a new Amazon EMR cluster is assigned the role EMR_EC2_DefaultRole, which contains all of the required permissions, plus additional permissions.

However, AWS recommends that you create a custom IAM role for your Amazon EMR cluster's Amazon EC2 instance profile role.

The following permissions reflect the minimum permissions needed for Structural data generation:

{
    "Sid": "GluePerms",
    "Effect": "Allow",
    "Action": [
        "glue:GetUserDefinedFunctions",
        "glue:BatchGetPartition",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
    ],
    "Resource": [
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:database/*",
        "arn:aws:glue:<region>:<account-id>:table/*"
    ]
},
{
    "Sid": "S3SourceBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::<source-s3-bucket>",
        "arn:aws:s3:::<source-s3-bucket>/*"
    ]
},
{
    "Sid": "S3DestBucketPerms",
    "Effect": "Allow",
    "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<destination-s3-bucket>",
        "arn:aws:s3:::<destination-s3-bucket>/*"
    ]
},
{
    "Sid": "S3EmrLogBucketPerms",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": [
        "arn:aws:s3:::<emr-logs-s3-bucket>/*"
    ]
}

For Amazon EMR, the Glue catalog must contain a default database. If the default database does not exist, then Amazon EMR attempts to create it.

Before you run a Structural data generation, you must either:

Ensure that the default catalog exists
Add glue:CreateDatabase to the list of permissions that are granted to this role

Structural does not otherwise require this permission, and does not explicitly attempt to create a database.

Decrypt and encrypt permissions

You must add decrypt and encrypt permissions to your account on both the source and destination paths.

If you use the Amazon EMR Steps API, then the Amazon EMR role that is assigned to your cluster must have Decrypt access to the AWS KMS key that is used on the output bucket.