Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The Tonic Structural file connector allows you to use files as the source data, and to write transformed versions of the files as destination data.
The files can be selected from either
An S3 bucket
Google Cloud Storage (GCS)
A local file system
You can also view a video tutorial that provides an overview of the file connector and how to manage file groups.
Structural differences and limitations
Features that are unavailable or work differently for the file connector.
Configure required cloud storage permissions
Set up the required permissions for Amazon S3 or Google Cloud Storage.
Configure workspace file storage and output
Identify where to find the source files and where to write the destination files.
Manage file groups
Configure the file groups. Each file group is treated as a table.
Download generated files
For files from a local file system, download the most recently generated files.
On the workspace details view for a file connector workspace, you:
Identify the type of storage. After you add a file group to the workspace, you cannot change the storage type.
Indicate where to write the transformed files
If needed, provide credentials to access the cloud storage.
On the workspace creation view:
Under Connection Type, under File/Blob Storage, click Files.
Select the type of file storage where the source files are located.
To choose files from Amazon S3, click Amazon S3. Note that this option is not supported on Tonic Cloud.
To choose files from GCS, click Google Cloud Storage.
To upload files from a local file system, click Local Filesystem.
After you add a file group to the workspace, you cannot change the storage type.
For Amazon S3 and Google Cloud Storage, under Output location, provide the path to the folder where Structural writes the transformed files.
When the source files come from a local file system, Tonic Structural writes the output files to the large file store in the Structural application database. You can then download the most recently generated files.
On Structural Cloud, to write files to Amazon S3, you must provide the AWS credentials in the workspace configuration.
On a self-hosted instance, if you provide AWS credentials in the workspace configuration, then Structural uses those credentials.
If you do not provide credentials in the workspace configuration, then Structural uses either:
The credentials set in the following environment settings:
TONIC_AWS_ACCESS_KEY_ID
- An AWS access key that is associated with an IAM user or role.
TONIC_AWS_SECRET_ACCESS_KEY
- The secret key that is associated with the access key
TONIC_AWS_REGION
- The AWS Region to send the authentication request to
For more information, go to Configuring environment settings.
The credentials for the IAM role on the host machine.
To provide the credentials, under AWS Credentials:
In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.
In the AWS Secret Key field, enter the secret key that is associated with the access key.
From the AWS Region dropdown list, select the AWS Region to send the authentication request to.
By default, Structural uses the same AWS credentials to both retrieve the source files and write the output files.
To provide different AWS credentials for the output location:
Toggle Set different credentials for output to the on position.
In the AWS Access Key field, enter the AWS access key that is associated with an IAM user or role.
In the AWS Secret Key field, enter the secret key that is associated with the access key.
From the AWS Region dropdown list, select the AWS Region to send the authentication request to.
To write files to a folder in Google Cloud Storage, you must provide Google Cloud Platform credentials in the workspace configuration.
Under GCP Credentials:
For Service Account File, select the service account file (JSON file) for the source files.
In the GCP Project ID field, provide the identifier of the project that contains the source files.
Required license: Professional or Enterprise
File connector workspaces do not support workspace inheritance.
In a file connector workspace, you can only use the following table modes:
De-Identify
Truncate
When a file group is assigned Truncate mode, Tonic Structural ignores that file group during data generation.
The Conditional generator can only be used in a file group that contains CSV files.
Otherwise, the available generators are based on the assigned data type:
For CSV data, each column is assigned the text
data type
For JSON, the table contains a single column that is assigned the json
data type
For XML and HTML, the table contains a single column that is assigned the xml
data type
File connector workspaces do not support subsetting.
File connector workspaces do not support upsert.
For file connector workspaces, you cannot write the destination data to container artifacts.
For file connector workspaces, you cannot write the destination data to an Ephemeral snapshot.
For file connector workspaces, there is no option to run post-job scripts after a job.
Required workspace permission: Manage file connector file groups
To identify the source files to transform, you create file groups. A file group is a set of source files that have an identical format and structure.
For local files, you always select each file individually.
For cloud storage (Amazon S3 or Google Cloud Storage), you can select individual files and folders. You can also filter by file extension and automatically select files based on prefix patterns.
A file group can contain files that contain CSV, XML, JSON, or Parquet content, including:
.csv, .tsv, .xml, .json, and .parquet files.
.txt files that contain CSV, XML, or JSON content.
.gzip files that contain compressed CSV, XML, or JSON content. .gzip files are only supported in workspaces that use files from cloud storage. They are not supported in workspaces that use local files.
For Parquet files:
The files must use plain encoding.
The files must be uncompressed. For example, you cannot select a .snappy.parquet file.
You cannot select the following Parquet file types:
HalfFloat
Struct
Union
Dictionary
Map
List
FixedSizeList
Arrays of any type
The file connector can read files that are ASCII encoded.
Within a file group:
All files must use the same format. For example, you cannot have both CSV and XML content in the same file group.
All files must have the same structure. For example, for a file group that contains CSV files, the content in all of the files must contain the same columns and use the same delimiters.
You can combine .txt and .gzip files with other files, as long as the file content in all of the files has the same format and structure. For example, a file group can contain a .txt file, a .csv file, and a .gzip file, as long they all contain CSV content that has the same structure.
After you add the first file to the file group, Tonic Structural does not allow you to select files that do not have the same format.
On the workspace management view for a file connector workspace, to display the list of file groups, click File Groups.
For each file group, the file group list includes:
The name of the file group.
The type of file that the file group contains.
The number of files in the file group.
For cloud storage, the number of prefix patterns.
When the file group was most recently modified.
To filter the file group list, begin typing the file group name. As you type, the list is filtered to only include the matching file groups.
On File Groups view, to display the details for a file group, click the file group name.
The file group details view includes:
The type of content in the files.
For cloud storage files, any file type filters.
A list of files that were manually selected. For local file workspaces, you select all files individually. For cloud storage workspaces, you can select files individually or use prefix filters. The Individual Files tab lists the individually selected files.
For cloud storage files, a Prefix Patterns tab that contains the list of prefix patterns.
To filter the file or prefix path list, in the filter field, begin to type the name. As you type, the list is updated to only include matching items.
On File Groups view, to create and populate a new file group:
Click Create group.
Under File group name, enter a name for the file group.
Select the file group files:
Click Save.
You can add files to an existing file group. The added files must have the same format and structure as the files that are already in the group.
To add files to a file group:
Either:
On the file group details view, click Add Files.
On File Groups view, click the + icon for the file group.
Change the file selection:
Click Save.
When you select an individual file to add, Structural automatically displays a preview of the file content. You can use the preview to verify the file content.
Note that for Parquet files, there is no file preview.
To hide the preview, click Hide Preview. To restore the preview, click Show Preview.
For local file file groups, you cannot see a preview for existing files that you added previously. For cloud storage file groups, you can preview existing files.
For a local files file group, you can delete any file from the group.
For a cloud storage file group, you can only delete files that you selected manually. You cannot delete files that were added by a prefix pattern. You can only change or remove the prefix pattern.
To remove a file from a file group, on the file group details view, click the delete icon for the file.
To remove multiple files:
Check the checkbox for each file to remove.
Click Actions, then click Delete Files.
For local files, when you delete the file from the file group, Structural also deletes the file from the Structural application database.
To delete a file group, on the File Groups view, click the delete icon for the file group.
For local files, when you delete a file group, Structural also deletes the file group's files from the Structural application database.
For local files, you always select the individual files to include in the file group.
To add files to the file group from a local file system, either:
Drag and drop files from your file system to the add files panel.
Click Select files to upload, then navigate to and select the files.
To remove a selected file, click its delete icon.
For a cloud storage (Amazon S3 or GCS) file group, you can select individual files and folders. When you select a folder, the file group automatically includes the files in that folder.
You can filter the available files based on file type. You can also automatically select files and folders based on their fully qualified path.
Finally, you can configure whether the data generation process only transforms files that were added since the previous data generation.
To select a file, click its name or the file checkbox.
To select a folder, click the folder checkbox. When you select a folder, Structural automatically adds a prefix filter for the folder.
You use the File types dropdown list to filter the file extensions for the files to include. You can select multiple file types, as long as files of the selected types can be compatible with each other. For example, you cannot filter the files to include both .json and .csv files.
By default, the selected file type is All file types, and the files are not filtered by file extension.
To add a file extension filter:
Click the File extensions dropdown list.
Click each file type to include.
Prefix patterns allow you to automatically select files based on paths. A prefix pattern is a fully qualified path. When you select a folder, Structural automatically adds the folder as a prefix pattern.
You can specify more than one prefix pattern.
To add a prefix pattern:
In the Prefix pattern field, type the path for which to include folders and files.
Click the add icon.
To remove a prefix pattern, click its delete icon.
When you add a prefix pattern, Structural automatically selects the files that match both the prefix pattern and any file extension filters.
The first time you generate data from a file connector workspace, Structural processes all of the files.
For subsequent data generations, the Only process new files configuration indicates whether Structural processes all of the files in the file group, or only processes files that were added to the file group since the most recent data generation.
For example, for a folder that is in the file group, you might have a regular process that adds new files on a regular basis. In that case, you would only want Structural to process new files, and ignore all of the files that it processed before.
By default, Only process new files is toggled to the on position, and Structural only processes new files. To always process all of the files in the file group, toggle Only process new files to the off position.
For files that contain CSV content, you use the delimiter and file settings fields to provide information about the file structure:
Whether the file contains a header row. If the file contains a header row, then toggle First row is column header to the on position.
Quote spaces - Whether to enclose spaces in quotes in the output files.
Trim whitespace - Whether to trim whitespace from before or after the values when the file is uploaded.
Column Delimiter - The file delimiter. The default is a comma.
Escape Character - The character that is used to escape characters. The default is a double quote.
Quoting Character - The character that is used to quote text. The default is the double quote.
Null Character - How null values are indicated. The default is \N
.
Skip First N Rows - The number of rows to omit from the beginning of the file.
Skip Last N Rows - The number of rows to omit fro the end of the file.
Tonic uses these settings to read and write the files.
After you save the file group, you cannot change most of these settings. You can change the Quote spaces setting.
Files that have a different delimiter configuration must be in a different file group.
You can create that are triggered by data generation jobs.
For CSV content, .
For files that contain CSV content, the preview also can help you to check that the are correct. If the settings are correct, the preview should show a readable table with the correct columns.
Under Select folders and files to add to file group, navigate to the folder in Amazon S3 or GCS where the file is located. You can only view and select from buckets and files that the associated IAM user (for Amazon S3) or Google Cloud Platform credentials (for GCS) is granted access to. For more information about the required permissions, go to . You can use the search field to search for a particular bucket, folder, or file. For the bucket list, you can search based on any text in the bucket name. For a folder or file, the search text is matched against the beginning of the folder or file name. The file browser can only display a limited number of items. Structural warns you when the number of items reaches the limit. You must use the file filter to locate items that are not displayed.
For a file connector workspace that reads files from and writes files to Amazon S3 or Google Cloud Storage, make sure to set up the appropriate permissions so that Tonic Structural can locate the source files and write the destination files.
You can also set up permissions to protect buckets that contain files that you do not want used in a workspace.
For a self-hosted instance, to get access to Amazon S3, Structural uses either:
The credentials set in the following environment settings:
TONIC_AWS_ACCESS_KEY_ID
- An AWS access key that is associated with an IAM user or role.
TONIC_AWS_SECRET_ACCESS_KEY
- The secret key that is associated with the access key
TONIC_AWS_REGION
- The AWS Region to send the authentication request to
For more information, go to Configuring environment settings.
The credentials for the IAM role on the host machine.
On Structural Cloud, you must provide the AWS credentials in the workspace configuration.
The IAM user that is associated with the credentials must have the following permissions:
If the source and destination S3 buckets are in different accounts, or are in an account that is different from the account or instance profile that Structural uses, then the configuration must include cross-account permissions. For assistance with this, contact support@tonic.ai.
To get access to Google Cloud Storage, Structural uses the Google Cloud Platform credentials that you provide in the workspace configuration.
The service account that you specify must have the following permissions.
storage.buckets.list
- This allows Structural to see the list of buckets when it creates a file group.
If the service account does not have this permission, then on the file group creation panel, users must type the name of the bucket where a file is located.
For buckets that contain source files, the following permissions allow Structural to get the list of files within buckets, and to retrieve the actual files and file content.
storage.objects.get
storage.objects.list
If the permissions are assigned globally, then Structural can list and retrieve files from any bucket. If the permissions are assigned to individual buckets, the file group creation view displays a list of all buckets. However, if you select a bucket for which the service account does not have the permissions, Structural returns an error.
For buckets that contain destination files, the following permissions allow Structural to see and get access to the bucket content and to create the generated files. This includes deleting and overwriting existing files that are regenerated.
storage.buckets.get
storage.objects.get
storage.objects.list
storage.objects.create
storage.objects.delete
If the permissions are assigned globally, then Structural can write files to any bucket. If the permissions are assigned to individual buckets, then Structural can only write files to those buckets.
For a file group that is based on files from a local file system, you can download the most recently generated files.
The download is a .zip file that contains the generated files.
To download all of the most recently generated files for a file group, either:
On the File Groups view, click the download icon for the file group.
On the file group details view, to download all of the generated files, click Export Files.
To download the most recent generated files for specific source files:
On the file group details view, check the checkbox for each file to include.
Click Actions, then click Export Files.
You also can download generated files for a specific data generation job.
On the job details view, when files are available to download, the Data available for file groups panel displays.
To download the files for that job for a selected file group:
Click Download Results.
From the list, select the file group. Use the filter field to filter the list by the file group name.
The file connector uses text files for the source and destination data.
The recommended option is to have the files in cloud storage. The files can be in either:
Amazon S3
Google Cloud Storage (GCS)
Within a file connector workspace, you create file groups. A file group is a set of files that have the same format and structure. The file group points to the files in the cloud storage location.
Tonic Structural treats each file group as a table. Within the file groups, you configure the generators to apply to the columns.
When you run data generation, Structural writes the output files to the configured destination file storage location for the workspace.
In the destination location, each source file is represented by a corresponding destination file that has the same name.
You can also use files from a local file system.
For a file connector workspace that uses local files, when you add files to a file group, Structural encrypts the selected files. It then stores the files in the large file store of the Structural application database.
When you remove a file from a file group, remove an entire file group, or delete the workspace, Structural also removes those stored files.
When you run data generation, Structural also writes the output files to the large file store of the Structural application database. You can then download the most recently generated files.