When Textual processes pipeline files, it produces JSON output that provides access to the Markdown content and that identifies entities that were detected in the file.
Common elements in the JSON output
Information about the entire file
All JSON output files contain the following elements that contain information for the entire file:
{"fileType":"<file type>","content": {"text":"<Markdown file content>","hash":"<hashed file content>","entities": [ //Entry for each entity in the file {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}
For specific file types, the JSON output includes additional objects and properties to reflect the file structure.
Hashed and Markdown content
The JSON output contains hashed and Markdown content for the entire file and for individual file components.
Entities
The JSON output contains entities arrays for the entire file and for individual file components.
Each entity in the entities array has the following properties:
Plain text files
For plain text files, the JSON output only contains the information for the entire file.
{"fileType":"<file type>","content": {"text":"<Markdown content>","hash":"<hashed content>","entities": [ //Entry for each entity in the file {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}
.csv files
For .csv files, the structure contains a tables array.
The tables array contains a table object that contains header and data arrays..
For each row in the file, the data array contains a row array.
For each value in a row, the row array contains a value object.
The value object contains the entities, hashed content, and Markdown content for the value.
{"tables": [ {"tableName":"csv_table","header": [//Columns that contain heading info (col_0, col_1, and so on)"<column identifier>" ],"data": [ //Entry for each row in the file [ //Entry for each value in the row { "entities": [ //Entry for each entity in the value {"start": <start location>,,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed value content>","text":"<Markdown value content>" } ] ] } ],"fileType":"<file type>","content": {"text":"<Markdown file content>","hash":"<hashed file content>","entities": [ ///Entry for each entity in the file {"start": <start location>,"end": <end location>"label": "<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}
.xlsx files
For .xlsx files, the structure contains a tables array that provides details for each worksheet in the file.
For each worksheet, the tables array contains a worksheet object.
For each row in a worksheet, the worksheet object contains a header array and a data array. The data array contains a row array.
For each cell in a row, the row array contains a cell object.
Each cell object contains the entities, hashed content, and Markdown content for the cell.
{"tables": [ //Entry for each worksheet {"tableName":"<Name of the worksheet>","header": [ //Columns that contain heading info (col_0, col_1, and so on)"<column identifier>" ],"data": [ //Entry for each row [ //Entry for each cell in the row {"entities": [ //Entry for each entity in the cell {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed cell content>","text":"<Markdown cell content>" } ] ] } ],"fileType":"<file type>","content": {"text":"<Markdown file content>","hash":"<hashed file content>","entities": [ //Entry for each entity in the file {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}
.docx files
For .docx files, the JSON output structure adds:
A footnotes array for content in footnotes.
An endnotes array for content in endnotes.
A header object for content in the page headers. Includes separate objects for the first page header, even page header, and odd page header.
A footer object for content in the page footers. Includes separate objects for the first page footer, even page footer, and odd page footer.
These arrays and objects contain the entities, hashed content, and Markdown content for the notes, headers, and footers.
{"footNotes": [ //Entry for each footnote {"entities": [ //Entry for each entity in the footnote {"start": <start location>,"end": <end location>,"pythonStart": <start location in Python>,"pythonEnd": <end location in Python>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>""exampleRedaction": null } ],"hash":"<hashed footnote content>","text":"<Markdown footnote content>" } ],"endNotes": [ //Entry for each endnote {"entities": [ //Entry for each entity in the endnote {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed endnote content>","text":"<Markdown endnote content>" } ],"header": {"first": {"entities": [ //Entry for each entity in the first page header {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed first page header content>","text":"<Markdown first page header content>" },"even": {"entities": [ //Entry for each entity in the even page header {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed even page header content>","text":"<Markdown even page header content>" },"odd": {"entities": [ //Entry for each entity in the odd page header {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed odd page header content>","text":"<Markdown odd page header content>" } },"footer": {"first": {"entities": [ //Entry for each entity in the first page footer {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed first page footer content>","text":"<Markdown first page footer content>" },"even": {"entities": [ //Entry for each entity in the even page footer {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed even page footer content>","text":"<Markdown even page footer content>" },"odd": {"entities": [ //Entry for each entity in the odd page footer {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed odd page footer content>","text":"<Markdown odd page footer content>" } },"fileType":"<file type>","content": {"text":"<Markdown file content>","hash":"<hashed file content>","entities": [ //Entry for each entity in the file {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}
PDF and image files
PDF and image files use the same structure. Textual extracts and scans the text from the files.
For PDF and image files, the JSON output structure adds the following content.
pages array
The pages array contains all of the content on the pages. This includes content in tables and key-value pairs, which are also listed separately in the output.
For each page in the file, the pages array contains a page array.
For each component on the page - such as paragraphs, headings, headers, and footers - the page array contains a component object.
Each component object contains the component entities, hashed content, and Markdown content.
tables array
The tables array contains content that is in tables.
For each table in the file, the tables array contains a table array.
For each row in a table, the table array contains a row array.
For each cell in a row, the row array contains a cell object.
Each cell object identifies the type of cell (header or content). It also contains the entities, hashed content, and Markdown content for the cell.
keyValuePairs array
The keyValuePairs array contains key-value pair content. For example, for a PDF of a form with fields, a key-value pair might represent a field label and field value.
For each key-value pair, the keyValuePairs array contains a key-value pair object.
The key-value pair object contains:
An automatically incremented identifier. For example, id for the first key-value pair is 1, for the second key-value pair is 2, and so on.
The start and end position of the key-value pair
The text of the key
The entities, hashed content, and Markdown content for the value
PDF and image JSON outline
{"pages": [ //Entry for each page in the file [ //Entry for each component on the page {"type":"<page component type>","content": {"entities": [ //Entry for each entity in the component {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed component content>","text":"<Markdown component content>" } } ],"tables": [ //Entry for each table in the file [ //Entry for each row in the table [ //Entry for each cell in the row {"type":"<content type>",//ColumnHeader or Content"content": {"entities": [ //Entry for each entity in the cell {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed cell text>","text":"<Markdown cell text>" } } ] ] ],"keyValuePairs": [ //Entry for each key-value pair in the file {"id": <incremented identifier>,"key":"<key text>","value": {"entities": [ //Entry for each entity in the value {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ],"hash":"<hashed value text>","text":"<Markdown value text> }, "start": <start location of the key-value pair>, "end": <end location of the key-value pair> } ], "fileType": "<file type>", "content": { "text": "<Markdown file content>", "hash": "<hashed file content>", "entities": [ ///Entry for each entity in the file { "start": <start location>, "end": <end location>, "label": "<value type>", "text": "<value text>", "score": <confidence score>, "language": "<language code>" } ] }, "schemaVersion": <integer schema version>}
.eml and .msg files
For email message files, the JSON output structure adds the following content.
Email message identifiers
The JSON output includes the following email message identifiers:
The identifier of the current message
If the message was a reply to another message, the identifier of that message
An array of related email messages. This includes the email message that the message replied to, as well as any other messages in an email message thread.
Recipients
The JSON output includes the email address and display name of the message recipients. It contains separate lists for the following:
Recipients in the To line
Recipients in the CC line
Recipients in the BCC line
Subject line
The subject object contains the message subject line. It includes:
Markdown and hashed versions of the message subject line.
The entities that were detected in the subject line.
Message timestamp
sentDate provides the timestamp when the message was sent.
Message body
The plainTextBodyContent object contains the body of the email message.
It contains:
Markdown and hashed versions of the message body.
The entities that were detected in the message body.
Message attachments
The attachments array provides information about any attachments to the email message. For each attached file, it includes:
The identifier of the message that the file is attached to.
The identifier of the attachment.
The JSON output for the file.
The count of words in the original file.
The count of words in the redacted version of the file.
Email message JSON outline
{"messageId":"<email message identifier>","inReplyToMessageId": <message that this message replied to>,"messageIdReferences": [<related email messages>],"senderAddress": {"address":"<sender email address>","displayName":"<sender display name>" },"toAddresses": [ //Entry for each recipient in the To list {"address":"<recipient email address>","displayName":"<recipient display name>" } ],"ccAddresses": [ //Entry for each recipient in the CC list {"address":"<recipient email address>","displayName":"<recipient display name>" } ],"bccAddresses": [ //Entry for each recipient in the BCC list {"address":"<recipient email address>","displayName":"<recipient display name>" } ],"sentDate":"<timestamp when the message was sent>","subject": {"text":"<Markdown version of the subject line>","hash":"<hashed version of the subject line>","entities": [ //Entry for each entity in the subject line {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"plainTextBodyContent": {"text":"<Markdown version of the message body>","hash":"<hashed version of the message body>","entities": [ //Entry for each entity in the message body {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"attachments": [ //Entry for each attached file {"parentMessageId":"<the message that the file is attached to>","contentId":"<identifier of the attachment>","fileName":"<name of the attachment file>","document": {<pipeline JSON for the attached file>},"wordCount": <number of words in the attachment>,"redactedWordCount": <number of words in the redacted attachment> } ],"fileType":"<file type>","content": {"text":"<Markdown file content>","hash":"<hashed file content>","entities": [ //Entry for each entity in the file {"start": <start location>,"end": <end location>,"label":"<value type>","text":"<value text>","score": <confidence score>,"language":"<language code>" } ] },"schemaVersion": <integer schema version>}