Creating and editing pipelines
Last updated
Was this helpful?
Last updated
Was this helpful?
To create a pipeline, on the Pipelines page, click Create a New Pipeline.
On the Create A New Pipeline panel:
In the Name field, type the name of the pipeline.
Under Files Source, select the location of the source files.
To upload files from a local file system, click File upload, then click Save.
To select files from and write output to Amazon S3, click Amazon S3.
To select files from and write output to Databricks, click Databricks.
To select files from and write output to Azure Blob Storage, click Azure.
To select files from and write output to Sharepoint, click Sharepoint.
Click Save.
If you selected Amazon S3, provide the credentials to use to connect to Amazon S3.
In the Access Secret field, provide the secret key that is associated with the access key.
From the Region dropdown list, select the AWS Region to send the authentication request to.
In the Session Token field, provide the session token to use for the authentication request.
To test the credentials, click Test AWS Connection.
By default, connections to Amazon S3 use Amazon S3 encryption. To instead use AWS KMS encryption:
Click Show Advanced Options.
In the Server-side Encryption AWS KMS ID field, provide the KMS key ID. Note that if the KMS key doesn't exist in the same account that issues the command, you must provide the full key ARN instead of the key ID.
Click Save.
On the Pipeline Settings page, provide the rest of the pipeline configuration. For more information, go to Configuring an Amazon S3 pipeline.
Click Save.
If you selected Databricks, provide the connection information:
In the Databricks URL field, provide the URL to the Databricks workspace.
In the Access Token field, provide the access token to use to get access to the volume.
To test the connection, click Test Databricks Connection.
Click Save.
On the Pipeline Settings page, provide the rest of the pipeline configuration. For more information, go to Configuring a Databricks pipeline.
Click Save.
If you selected Azure, provide the connection information:
In the Account Name field, provide the name of your Azure account.
In the Account Key field, provide the access key for your Azure account.
To test the connection, click Test Azure Connection.
Click Save.
On the Pipeline Settings page, provide the rest of the pipeline configuration. For more information, go to Configuring an Azure pipeline.
Click Save.
If you selected Sharepoint, provide the credentials for the Entra ID application.
The credentials must have the following application permissions (not delegated permissions):
Files.Read.All - To see the Sharepoint files
Files.ReadWrite.All -To write redacted files and metadata back to Sharepoint
Sites.ReadWrite.All - To view and modify the Sharepoint sites
In the Tenant ID field, provide the Sharepoint tenant identifier for the Sharepoint site.
In the Client ID field, provide the client identifier for the Sharepoint site.
In the Client Secret field, provide the secret to use to connect to the Sharepoint site.
To test the connection, click Test Sharepoint Connection.
On the Pipeline Settings page, provide the rest of the pipeline configuration. For more information, go to Configuring a Sharepoint pipeline.
Click Save.
To update a pipeline configuration:
On the pipeline details page, click the settings icon. For cloud storage pipelines, the settings icon is next to the Run Pipeline option. For uploaded file pipelines, the settings icon is next to the Upload Files option.
On the Pipeline Settings page, update the configuration. For all pipelines, you can change the pipeline name, and whether to also create redacted versions of the original files. For cloud storage pipelines, you can change the file selection. For more information, go to:
For uploaded file pipelines, you do not manage files from the Pipeline Settings page. For information about uploading files, go to Selecting files for an uploaded file pipeline.
Click Save.
To delete a pipeline, on the Pipeline Settings page, click Delete Pipeline.
In the Access Key field, provide an AWS access key that is associated with an IAM user or role. For an example of a role that has the required permissions for an Amazon S3 pipeline, go to .
From the Server-Side Encryption Type dropdown list, select AWS KMS. Note that after you save the new pipeline, you cannot change the encryption type.
