System requirements for Amazon EMR

Supported versions of Spark and Amazon EMR

The following table lists the supported versions of Amazon EMR, along with the corresponding versions of Spark, based on this information in the Amazon EMR documentation.

We recommend EMR-6.1.0 with Spark 3.0.0 or EMR-6.2.0 with Spark 3.0.1.

Amazon EMR version	Spark version
6.2.x	3.0.1
6.1.x	3.0.0
6.0.x	2.4.4
5.36.x	2.4.8
5.35.x	2.4.8
5.34.0	2.4.8
5.33.x	2.4.7
5.32.x	2.4.7
5.31.x	2.4.6
5.30.x	2.4.5
5.29.0	2.4.4
5.28.x	2.4.4

Supported providers

Structural supports the following data providers:

Source Provider	Output Provider
Parquet	Parquet
CSV	CSV
Avro	Avro
JSON	JSON
ORC	ORC

Metadata catalog

Structural requires a metadata catalog when connecting to your data. Currently only AWS Glue is supported when working with Amazon EMR.

Structural writes data to Amazon S3 only. Structural does not write output data back into a catalog.

Amazon S3 server side encryption requirements

If your S3 buckets have server side encryption enabled via AWS KMS, then your Spark cluster must have Hadoop 2.8.1+ installed.