Create and manage pipelines
Last updated
Was this helpful?
Last updated
Was this helpful?
Textual uses pipelines to transform file text into a format that can be used in an LLM system.
You can use the Textual SDK to create and manage pipelines and to retrieve pipeline run results.
Before you perform these tasks, remember to .
To create a pipeline, use the pipeline creation method for the type of pipeline to create"
- Creates an uploaded file pipeline.
- Creates an Amazon S3 pipeline.
- Creates an Azure pipeline.
- Creates a Databricks pipeline.
When you create the pipeline, you can also:
If needed, provide the credentials to use to connect to Amazon S3, Azure, or Databricks.
Indicate whether to also generate redacted files. By default, pipelines do not generate redacted files. To generate redacted files, set synthesize_files to True.
For example, to create an uploaded file pipeline that also creates redacted files:
The response contains the pipeline object.
For an Amazon S3 pipeline, you can configure the output location for the processed files. You can also identify the files and folders for the pipeline to process:
For an Azure pipeline, you can configure the output location for the processed files. You can also identify the files and folders for the pipeline to process:
The response contains a list of pipeline objects.
The response contains a single pipeline object.
The pipeline identifier is displayed on the pipeline details page. To copy the identifier, click the copy icon.
The response contains the job identifier.
The response contains a list of pipeline run objects.
The response is an enumerator of file parse result objects.
generator_config is a dictionary that specifies whether to redact, synthesize, or do neither for each entity type in the dictionary.
For a list of the entity types that Textual detects, go to Entity types that Textual detects.
For each entity type, you provide the handling type:
Redaction indicates to replace the value with a token that represents the entity type.
Synthesis indicates to replace the value with a realistic value.
Off indicates to keep the value as is.
generator_default indicates how to process values for entity types that were not included in the generator_config list.
The response contains the list of entities. For each value, the list includes:
Entity type
Where the value starts in the source file
Where the value ends in the source file
The original text of the entity
The response contains the Markdown files, with the detected entities processed as specified in generator_config and generator_default.
In the request, you set the maximum number of characters in each chunk.
You can also provide generator_config and generator_default to configure how to present the detected entities in the text chunks.
The response contains the list of text chunks, with the detected entities processed as specified in generator_config and generator_default.
To delete a pipeline, use .
To change whether a pipeline also generates synthesized files, use .
To a a file to an uploaded file pipeline, use .
To identify the output location for the processed files, use .
To identify individual files for the pipeline to process, use .
To identify prefixes - folders for which the pipeline processes all applicable files - use .
To identify the output location for the processed files, use .
To identify individual files for the pipeline to process, use .
To identify prefixes - folders for which the pipeline processes all applicable files - use .
To get the list of pipelines, use .
To use the pipeline identifier to get a single pipeline, use .
To run a pipeline, use .
To get the list of pipeline runs, use .
Once you have the pipeline, to get an enumerator of the files in the pipeline from the most recent pipeline run, use .
To get a list of entities that were detected in a file, use . For example, to get the detected entities for all of the files in a pipeline:
To provide a list entity types and how to process them, use :
To get the Markdown output of a pipeline file, use . In the request, you can provide generator_config and generator_default to configure how to present the detected entities in the output file.
To split a pipeline file into text chunks that can be imported into an LLM, use .
