Azure - Setting the storage location for temporary files

During data generation, Structural uses temporary CSV files to load and unload Snowflake tables.

To store these temporary files, you can use either:

• Azure Blob Storage
• External stages

You can also provide separate locations for the source and destination files.

Selecting the type of storage to use

Under File Storage Options, by default, the Use External Stage toggle is off, which indicates to use Azure Blob Storage for the temporary files.

To instead use external stages, toggle Use External Stage to the on position.

Enabling separate paths for source and destination files

By default, you provide a single Azure Blob Storage or external stage. Within that path:

• During data generation, Structural copies source data CSV files to an input folder in the storage location.
• After it applies the generators, Structural copies destination data CSV files to an output folder.

To instead provide separate paths for the source and destination files, toggle Use Separate Destination Location to the on position.

Setting the Azure Blob Storage location for temporary files

When Use External Stage is off, if Use Separate Destination Location is off, then:

1. In the Source Azure Storage Account Name field, provide the name of the account that Structural uses to load and unload Snowflake data in Azure Blob Storage. Do not provide the URI. You only need the account name.
2. In the Source Azure Blob Storage Path field, provide the container path to the Azure Blob Storage location where Structural places temporary CSV files that it uses to load and unload Snowflake tables.

If Use Separate Destination Location is on, then:

1. In the Source Azure Storage Account Name field, provide the name of the account that Structural uses to load and unload source Snowflake data in Azure Blob Storage. Do not provide the URI. You only need the account name.
2. In the Source Azure Blob Storage Path field, provide the container path to the Azure Blob Storage location where Structural places temporary source CSV files that it uses to load and unload Snowflake tables.
3. In the Destination Azure Storage Account Name field, provide the name of the account that Structural uses to write and read the temporary destination files.
4. In the Destination Azure Blob Storage Path field, provide the container path to the Azure Blob Storage location where Structural places the temporary destination files.

Setting the external stage location for temporary files

When Use External Stage is on, you provide the path to the external stage or stages to use for temporary files.

The format is:

<database>.<schema>.<stage>

Where:

• <database> is the name of the database where the stage is located.
• <schema> is the name of the schema that contains the stage.
• <stage> is the name of the stage.

If Use Separate Destination Location is off, then in the Source Snowflake External Stage Name field, provide the path to the external stage.

If Use Separate Destination Location is on, then:

1. In the Source Snowflake External Stage Name field, provide the path to the external stage to use for the source files.
2. In the Destination Snowflake External Stage Name field, provide the path to the external stage use for the destination files.

Providing the connection credentials

For the connection to the temporary file storage, under Azure Credentials, you configure the connection credentials.

You can either:

• On a self-hosted instance, use an environment setting value.
• Use a storage account key.
• Use a service principal. The service principal must be assigned the Storage Blob Data Contributor role in Azure. For more information, go to the Azure documentation.

Environment setting

On a self-hosted setting, to use the storage account key that is set as the value of the environment setting TONIC_AZURE_BLOB_STORAGE_ACCOUNT_KEY, click Environment.

Storage account key

To provide a storage account key to use for the connection:

1. Click Storage Account Key.
2. In the Azure Blob Storage Account Key field, provide the key.

Service principal

To use a service principal for the connection:

1. Click Service Principal.
2. In the Tenant ID field, provide your Azure tenant identifier.
3. In the Client ID field, provide your Azure client identifier.
4. In the Client Secret field, provide your Azure client secret.

Testing the storage location connection

To verify that Structural is able to use the provided credentials to connect to the storage location, click Test Source Object Storage.