Links

Managing file groups in a file connector workspace

Required workspace permission: Manage file connector file groups
To identify the source files to transform, you create file groups. A file group is a set of source files that have an identical format and structure.
A file group can contain files that contain CSV, XML, or JSON content, including:
  • .csv, .tsv, .xml, and .json files.
  • .txt files that contain CSV, XML, or JSON content.
  • .gzip files that contain compressed CSV, XML, or JSON content. .gzip files are only supported in workspaces that use files from Amazon S3 or Google Cloud Storage. They are not supported in workspaces that use local files.
The file connector can read files that are ASCII encoded.
Within a file group:
  • All files must use the same format. For example, you cannot have both CSV and XML content in the same file group.
  • All files must have the same structure. For example, for a file group that contains CSV files, the content in all of the files must contain the same columns and use the same delimiters.
  • You can combine .txt and .gzip files with other files, as long as the file content in all of the files has the same format and structure. For example, a file group can contain a .txt file, a .csv file, and a .gzip file, as long they all contain CSV content that has the same structure.
After you add the first file to the file group, Tonic does not allow you to select files that do not match the format and structure of that file.

Viewing the list of file groups

On the workspace management view for a file connector workspace, to display the list of file groups, click File Groups.
File Groups view
For each file group, the file group list includes:
  • The name of the file group.
  • The type of file that the file group contains.
  • The number of files in the file group.
  • When the file group was most recently modified.
To filter the file group list, begin typing the file group name. As you type, the list is filtered to only include the matching file groups.

Creating a file group

On File Groups view, to create and populate a new file group:
  1. 1.
    Click Create group.
  2. 2.
    Under File group name, enter a name for the file group.

Adding files from Amazon S3 or GCS

To add files to the group from Amazon S3 or GCS:
File group creation from cloud storage
  1. 1.
    Under Available Files, navigate to the folder in Amazon S3 or GCS where the file is located. You can only view and select from buckets and files that the associated IAM user (for Amazon S3) or Google Cloud Platform credentials (for GCS) is granted access to. For more information about the required permissions, go to Before you create a file connector workspace. You can use the search field to search for a particular bucket, folder, or file. For the bucket list, you can search based on any text in the bucket name. For a folder or file, the search text is matched against the beginning of the folder or file name. The file browser can only display up to 1000 items. If a folder contains more than 1,000 items, then you must use the file filter to locate items that are not displayed.
  2. 2.
    Click the file name. The file is added to the list of selected files. You can only add files that end with one of the following file extensions: .csv, .tsv, .xml, .json, .txt, .gzip. All of the selected files must contain data that has the same format (CSV, XML, or JSON) and structure. To remove a selected file, click its delete icon.

Adding files from a local file system

To add files to the file group from a local file system, either:
  • Drag and drop files from your file system to the add files panel.
  • Click Select files to upload, then navigate to and select the files.
To remove a selected file, click its delete icon.
File group creation from local files

Configuring delimiters and file settings for .csv files

For files that contain CSV content, you use the delimiter and file settings fields to provide information about the file structure:
Delimiter and file settings for .csv files
  • Whether the file contains a header row. If the file contains a header row, then toggle First row is column header to the on position.
  • Quote spaces - Whether to enclose spaces in quotes in the output files.
  • Trim whitespace - Whether to trim whitespace from before or after the values when the file is uploaded.
  • Column Delimiter - The file delimiter. The default is a comma.
  • Escape Character - The character that is used to escape characters. The default is a double quote.
  • Quoting Character - The character that is used to quote text. The default is the double quote.
  • Null Character - How null values are indicated. The default is \N.
  • Skip First N Rows - The number of rows to omit from the beginning of the file.
  • Skip Last N Rows - The number of rows to omit fro the end of the file.
Tonic uses these settings to read and write the files.
After you save the file group, you cannot change most of these settings. You can change the Quote spaces setting.
Files that have a different delimiter configuration must be in a different file group.

Previewing added files

For each added file, Tonic displays a preview of the file content. To hide the previews, click Hide File Preview. To restore the preview, click Show Preview.
For files that contain CSV content, the preview can help you to check that the CSV delimiter settings are correct. If the settings are correct, the preview should show a readable table with the correct columns.

Saving the new file group

To save the file group, click Save.

Displaying details for a file group

On File Groups view, to display the details for a file group, click the file group name.
The file group details view contains the list of files in the file group.
File group details
For each file in the file group, the list includes:
  • The file name
  • The type of file
To filter the file list, in the filter field, begin to type the file name. As you type, the list is updated to only include matching files.

Adding files to a file group

You can add files to an existing file group. The added files must have the same format and structure as the files that are already in the group.
To add files to a file group, either:
  • On the file group details view, click Add Files.
  • On File Groups view, click the actions menu for the file group, then select Add Files.
Actions menu in the file group list

Adding files from cloud storage

To add files from Amazon S3 or GCS, on the file selection panel, under Available Files, use the file browser to locate the folder in Amazon S3 or GCS where the file is located, then click the file name. The file is added to the list.
The file browser can only display up to 1000 items. If a folder contains more than 1,000 items, then you must use the file filter to locate items that are not displayed.
If the AWS or GCS credentials do not have permission to display the list of buckets, then in the filter field, type the name of the bucket that contains the file.

Adding files from a local file system

To add files from a local file system, either drag and drop the files from the file system to the file selection panel, or click Select files to upload.

Previewing the new files

When you select a file to add, Tonic automatically displays a preview of the file content. You can use the preview to verify the file content.
To hide the preview, click Hide Preview. To restore the preview, click Show Preview.
You cannot see a preview for existing files that you added previously.

Saving the new files

To add the files and return to the file group details, click Save.

Removing files from a file group

To remove a file from a file group, on the file group details view, click the delete icon for the file.
To remove multiple files:
  1. 1.
    Check the checkbox for each file to remove.
  2. 2.
    Click Actions, then click Delete Files.
Actions menu on file group details
For files that you uploaded from a local file system, when you remove the file, Tonic also deletes the file from the Tonic application database.

Deleting a file group

To delete a file group, on the File Groups view, click the actions menu for the file group, then select Delete.
Actions menu in the file group list
For files that you uploaded from a local file system, when you delete a file group, Tonic also deletes the files in that group from the Tonic application database.
Last modified 1mo ago