# Managing file groups in a file connector workspace

{% hint style="info" %}
**Required workspace permission:** Manage file connector file groups
{% endhint %}

To identify the source files to transform, you create file groups. A file group is a set of source files that have an identical format and structure.

For local files, you always select each file individually.

For cloud storage and file mounts, you can select individual files and folders. You can also filter by file extension and automatically select files based on prefix patterns.

## File group restrictions <a href="#file-group-requirements-restrictions" id="file-group-requirements-restrictions"></a>

The selected files must use a [supported file and content type](file-connector-file-content-types).

Within a file group:

* All files must use the same format. For example, you cannot have both CSV and XML content in the same file group.
* All files must have the same structure. For example, for a file group that contains CSV files, the content in all of the files must contain the same columns and use the same delimiters.
* You can combine .txt and .gzip files with other files, as long as the file content in all of the files has the same format and structure. For example, a file group can contain a .txt file, a .csv file, and a .gzip file, as long they all contain CSV content that has the same structure.

After you add the first file to the file group, Tonic Structural does not allow you to select files that do not have the same format.

## Viewing the list of file groups

On the workspace management view for a file connector workspace, to display the list of file groups, click **File Groups**.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FPIl4vE6awPRusZTCKCFN%2FFileConnector_FileGroupsPopulated.png?alt=media&#x26;token=ca7a03e3-a14a-4b0d-9f7f-3fcf64a18fd5" alt=""><figcaption><p>File Groups view</p></figcaption></figure>

For each file group, the file group list includes:

* The name of the file group.
* The type of file that the file group contains.
* The number of files in the file group.
* For cloud storage, the number of prefix patterns.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FhJOhWgCk1ksj7C8J5vkR%2FFileConnector_CloudGroupsColumns.png?alt=media&#x26;token=18173603-3147-4e03-9b85-393a2f5c602e" alt=""><figcaption><p>File type, file count, and prefix pattern count columns for a cloud storage workspace</p></figcaption></figure>

* When the file group was most recently modified.

To filter the file group list, begin to type the file group name. As you type, the list is filtered to only include the matching file groups.

## Displaying details for a file group

On **File Groups** view, to display the details for a file group, click the file group name.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FPKBa98BrSWunmCMfXU6P%2FFileConnector_FileGroupDetails.png?alt=media&#x26;token=782bdbb9-74d1-4d95-b8d0-911c3dfd314e" alt=""><figcaption><p>File group details for a local files workspace</p></figcaption></figure>

The file group details view includes:

* The type of content in the files.
* For cloud storage files, any file type filters.
* A list of files that were manually selected.\
  \
  For local file workspaces, you select all files individually.\
  \
  For cloud storage workspaces, you can select files individually or use prefix filters. The **Individual Files** tab lists the individually selected files.
* For cloud storage files, a **Prefix Patterns** tab that contains the list of prefix patterns.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F4qNNLAOiA3TdpAfiIuWA%2FFileConnector_FileGroupDetailsCloud.png?alt=media&#x26;token=9f0d2593-1635-4e15-8842-04438385140c" alt=""><figcaption><p>File group details for a cloud storage workspace</p></figcaption></figure>

To filter the file or prefix path list, in the filter field, begin to type the name. As you type, the list is updated to only include matching items.

## Creating, editing, and deleting file groups <a href="#file-group-create-edit-delete" id="file-group-create-edit-delete"></a>

### Creating a file group

On **File Groups** view, to create and populate a new file group:

1. Click **Create group**.
2. Under **File group name**, enter a name for the file group.
3. Select the file group files:
   * [#file-group-select-files-local](#file-group-select-files-local "mention")
   * [#file-group-file-select-cloud](#file-group-file-select-cloud "mention")
4. For CSV content, [configure the delimiters and file settings](#configuring-delimiters-and-file-settings-for-.csv-files).
5. Click **Save.**

### Adding files to a file group

You can add files to an existing file group. The added files must have the same format and structure as the files that are already in the group.

To add files to a file group:

1. Either:
   * On the file group details view, click **Add Files**.
   * On **File Groups** view, click the + icon for the file group.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F90zBlYy01XsQooV3fHjS%2FFileGroupOptionsColumn.png?alt=media&#x26;token=7ec024dd-c719-4456-b221-6bc5540e9b61" alt=""><figcaption><p>Options column in the file group list</p></figcaption></figure>

2. Change the file selection:
   * [#file-group-select-files-local](#file-group-select-files-local "mention")
   * [#file-group-file-select-cloud](#file-group-file-select-cloud "mention")
3. Click **Save**.

### **Previewing added files**

When you select an individual file to add, Structural automatically displays a preview of the file content. You can use the preview to verify the file content.

For files that contain CSV content, the preview also can help you to check that the [CSV delimiter settings](#configuring-delimiters-and-file-settings-for-.csv-files) are correct. If the settings are correct, the preview should show a readable table with the correct columns.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F72ARME1CsOLwVDoslP8J%2FFileConnector_FilePreview.png?alt=media&#x26;token=13d4da89-7f08-4f4d-9330-f5e950f96649" alt=""><figcaption><p>File preview for a newly added file</p></figcaption></figure>

Note that for Parquet files, there is no file preview.

To hide the preview, click **Hide Preview**. To restore the preview, click **Show Preview**.

For local file file groups, there are no previews for existing files that you added previously. For cloud storage file groups, you can preview existing files.

### Deleting files from a file group

For a local files file group, you can delete any file from the group.

For a cloud storage file group, you can only delete files that you selected manually. You cannot delete files that were added by a prefix pattern. You can only change or remove the prefix pattern.

To remove a file from a file group, on the file group details view, click the delete icon for the file.

To remove multiple files:

1. Check the checkbox for each file to remove.
2. Click **Actions**, then click **Delete Files**.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F89y1w20N6JGXJ8uXcZ9z%2FFileConnector_FileGroupActionsMenu.png?alt=media&#x26;token=0c2dda64-caeb-41fe-8834-0ff5d30a15a8" alt=""><figcaption><p>Actions menu on file group details</p></figcaption></figure>

For local files, when you delete the file from the file group, Structural also deletes the file from the Structural application database.

### Deleting a file group

To delete a file group, on the **File Groups** view, click the delete icon for the file group.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F90zBlYy01XsQooV3fHjS%2FFileGroupOptionsColumn.png?alt=media&#x26;token=7ec024dd-c719-4456-b221-6bc5540e9b61" alt=""><figcaption><p>Options column in the file group list</p></figcaption></figure>

For local files, when you delete a file group, Structural also deletes the file group's files from the Structural application database.

## **Selecting local files** <a href="#file-group-select-files-local" id="file-group-select-files-local"></a>

For local files, you always select the individual files to include in the file group.

To add files to the file group from a local file system, either:

* Drag and drop files from your file system to the add files panel.
* Click **Select files to upload**, then navigate to and select the files.

To remove a selected file, click its delete icon.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FoWFgJn4zhMyeL1fmpqoY%2FFileConnector_CreateFileGroup_CSV.png?alt=media&#x26;token=0fc99205-0dba-452b-b9ca-b01f1c90a6ff" alt=""><figcaption><p>File group file selection from local files</p></figcaption></figure>

## **Selecting cloud storage or file mount files** <a href="#file-group-file-select-cloud" id="file-group-file-select-cloud"></a>

For a cloud storage or file mount file group, you can select individual files and folders. When you select a folder, the file group automatically includes the files in that folder.

You can filter the available files based on file type. You can also automatically select files and folders based on their fully qualified path.

Finally, you can configure whether the data generation process only transforms files that were added since the previous data generation.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FxOiRSH48p7YLJAyOkeKE%2FFileConnector_CreateFileGroup_Cloud.png?alt=media&#x26;token=84427521-fb14-448f-9105-41dcfcf90c76" alt=""><figcaption><p>Flie group file selection from cloud storage</p></figcaption></figure>

### Navigating to and selecting files and folders <a href="#file-select-cloud-storage-individual" id="file-select-cloud-storage-individual"></a>

Under **Select folders and files to add to file group**, navigate to the folder where the file is located.\
\
For cloud storage, you can only view and select from buckets and files that the associated IAM user (for Amazon S3) or Google Cloud Platform credentials (for GCS) is granted access to. For more information about the required permissions, go to [connectors-file-before-workspace](connectors-file-before-workspace "mention").\
\
You can use the search field to search for a particular bucket, folder, or file. For the bucket list, you can search based on any text in the bucket name. For a folder or file, the search text is matched against the beginning of the folder or file name.\
\
The file browser can only display a limited number of items. Structural warns you when the number of items reaches the limit. You must use the file filter to locate items that are not displayed.

To select a file, click its name or the file checkbox.

To select a folder, click the folder checkbox. When you select a folder, Structural automatically adds a prefix filter for the folder.

### Filtering files by file extension <a href="#file-selection-cloud-storage-file-extension-filter" id="file-selection-cloud-storage-file-extension-filter"></a>

You use the **File types** dropdown list to filter the file extensions for the files to include. You can select multiple file types, as long as files of the selected types can be compatible with each other. For example, you cannot filter the files to include both .json and .csv files.

By default, the selected file type is **All file types**, and the files are not filtered by file extension.

To add a file extension filter:

1. Click the **File extensions** dropdown list.
2. Click each file type to include.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FhRi7TpnN00NlqzoG37pY%2FFileConnector_FileGroupExtensionFilter.png?alt=media&#x26;token=595a5f88-155e-4ce3-94ca-17349ea1e1ca" alt=""><figcaption><p>File extensions filter for a cloud storage file group</p></figcaption></figure>

### Using prefix patterns for automatic file and folder selection <a href="#file-selection-cloud-storage-prefix-filters" id="file-selection-cloud-storage-prefix-filters"></a>

Prefix patterns allow you to automatically select files based on paths. A prefix pattern is a fully qualified path. When you select a folder, Structural automatically adds the folder as a prefix pattern.

You can specify more than one prefix pattern.

To add a prefix pattern:

1. In the **Prefix pattern** field, type the path for which to include folders and files.
2. Click the add icon.

To remove a prefix pattern, click its delete icon.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FBom9iPFpjyAw9VDZ8Fzb%2FFileConnector_FileGroupPrefixFilters.png?alt=media&#x26;token=068b3284-f2b5-427f-8d29-ef07b9e0f62a" alt=""><figcaption><p>Prefix patterns for a cloud storage file group</p></figcaption></figure>

When you add a prefix pattern, Structural automatically selects the files that match both the prefix pattern and any file extension filters.

### Configuring whether to only process new files <a href="#file-connector-cloud-storage-new-files-only" id="file-connector-cloud-storage-new-files-only"></a>

The first time you generate data from a file connector workspace, Structural processes all of the files.

For subsequent data generations, the **Only process new files** configuration indicates whether Structural processes all of the files in the file group, or only processes files that were added to the file group since the most recent data generation.

For example, for a folder that is in the file group, you might have a regular process that adds new files on a regular basis. In that case, you would only want Structural to process new files, and ignore all of the files that it processed before.

By default, **Only process new files** is toggled to the on position, and Structural only processes new files. To always process all of the files in the file group, toggle **Only process new files** to the off position.

## **Configuring delimiters and file settings for .csv files**

For files that contain CSV content, you use the delimiter and file settings fields to provide information about the file structure:

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2F9LDY7JczU7NI79HqxHPj%2FFileConnector_CSVDelimiterOptions.png?alt=media&#x26;token=5b2cc88d-fc83-498d-ae89-fe816efe1df7" alt=""><figcaption><p>Delimiter and file settings for a CSV file group</p></figcaption></figure>

Structural uses these settings to read and write the files.

After you save the file group, you cannot change most of these settings. You can change the **Quote spaces** setting.

Files that have a different delimiter configuration must be in a different file group.

### File header row <a href="#file-group-csv-header-row" id="file-group-csv-header-row"></a>

If the file contains a header row, then toggle **First row is column header** to the on position.

### Managing spaces <a href="#file-group-csv-space-handling" id="file-group-csv-space-handling"></a>

To configure how to process spaces:

1. The **Quote spaces** setting indicates whether to enclose spaces in quotes in the output files. You can change this setting after you save the file group.
2. The **Trim whitespace** setting indicates whether to trim whitespace from before or after the values when the file is uploaded.&#x20;

### Encoding <a href="#file-group-csv-encoding" id="file-group-csv-encoding"></a>

To specify the type of encoding that the file uses:

1. Toggle **Specify encoding** to the on position.
2. From the dropdown list, select the type of encoding to use.

<figure><img src="https://3378426797-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LSQCLFQ4bslJ-HYc8c3%2Fuploads%2FchjImFpBwDRDRL6qHfaT%2FFileConnector_FileGroupCSVEncoding.png?alt=media&#x26;token=7131cf0a-e73d-40a7-9622-1ca8a11d1fde" alt=""><figcaption><p>Encoding configuration for a CSV file group</p></figcaption></figure>

If you do not specify the encoding, then Structural attempts to determine the encoding automatically. If Structural cannot identify the encoding, then the default encoding is UTF-8.

### Delimiters and special characters <a href="#file-group-csv-delimiter-special-characters" id="file-group-csv-delimiter-special-characters"></a>

The following configurations identify the delimiter and special characters for the file group.

* **Column Delimiter** - The file delimiter. The default is a comma.
* **Escape Character** - The character that is used to escape characters. The default is a double quote.
* **Quoting Character** - The character that is used to quote text. The default is the double quote.
* **Null Character** - How null values are indicated. The default is `\N`.

### Options to skip first and last rows <a href="#file-group-csv-skip-rows" id="file-group-csv-skip-rows"></a>

The following options allow you to omit rows from the beginning or end of the file. By default, Structural does not omit any rows.

* **Skip First N Rows -** The number of rows to omit from the beginning of the file.
* **Skip Last N Rows** - The number of rows to omit from the end of the file.
