LogoLogo
Release notesAPI docsDocs homeStructural CloudTonic.ai
  • Tonic Structural User Guide
  • About Tonic Structural
    • Structural data generation workflow
    • Structural deployment types
    • Structural implementation roles
    • Structural license plans
  • Logging into Structural for the first time
  • Getting started with the Structural free trial
  • Managing your user account
  • Frequently Asked Questions
  • Tutorial videos
  • Creating and managing workspaces
    • Managing workspaces
      • Viewing your list of workspaces
      • Creating, editing, or deleting a workspace
      • Workspace configuration settings
        • Workspace identification and connection type
        • Data connection settings
        • Configuring secrets managers for database connections
        • Data generation settings
        • Enabling and configuring upsert
        • Writing output to Tonic Ephemeral
        • Writing output to a container repository
        • Advanced workspace overrides
      • About the workspace management view
      • About workspace inheritance
      • Assigning tags to a workspace
      • Exporting and importing the workspace configuration
    • Managing access to workspaces
      • Sharing workspace access
      • Transferring ownership of a workspace
    • Viewing workspace jobs and job details
  • Configuring data generation
    • Privacy Hub
    • Database View
      • Viewing and configuring tables
      • Viewing the column list
      • Displaying sample data for a column
      • Configuring an individual column
      • Configuring multiple columns
      • Identifying similar columns
      • Commenting on columns
    • Table View
    • Working with document-based data
      • Performing scans on collections
      • Using Collection View
    • Identifying sensitive data
      • Running the Structural sensitivity scan
      • Manually indicating whether a column is sensitive
      • Built-in sensitivity types that Structural detects
      • Creating and managing custom sensitivity rules
    • Table modes
    • Generator information
      • Generator summary
      • Generator reference
        • Address
        • Algebraic
        • Alphanumeric String Key
        • Array Character Scramble
        • Array JSON Mask
        • Array Regex Mask
        • ASCII Key
        • Business Name
        • Categorical
        • Character Scramble
        • Character Substitution
        • Company Name
        • Conditional
        • Constant
        • Continuous
        • Cross Table Sum
        • CSV Mask
        • Custom Categorical
        • Date Truncation
        • Email
        • Event Timestamps
        • File Name
        • Find and Replace
        • FNR
        • Geo
        • HIPAA Address
        • Hostname
        • HStore Mask
        • HTML Mask
        • Integer Key
        • International Address
        • IP Address
        • JSON Mask
        • MAC Address
        • Mongo ObjectId Key
        • Name
        • Noise Generator
        • Null
        • Numeric String Key
        • Passthrough
        • Phone
        • Random Boolean
        • Random Double
        • Random Hash
        • Random Integer
        • Random Timestamp
        • Random UUID
        • Regex Mask
        • Sequential Integer
        • Shipping Container
        • SIN
        • SSN
        • Struct Mask
        • Timestamp Shift Generator
        • Unique Email
        • URL
        • UUID Key
        • XML Mask
      • Generator characteristics
        • Enabling consistency
        • Linking generators
        • Differential privacy
        • Partitioning a column
        • Data-free generators
        • Supporting uniqueness constraints
        • Format-preserving encryption (FPE)
      • Generator types
        • Composite generators
        • Primary key generators
    • Generator assignment and configuration
      • Reviewing and applying recommended generators
      • Assigning and configuring generators
      • Document View for file connector JSON columns
      • Generator hints and tips
      • Managing generator presets
      • Configuring and using Structural data encryption
      • Custom value processors
    • Subsetting data
      • About subsetting
      • Using table filtering for data warehouses and Spark-based data connectors
      • Viewing the current subsetting configuration
      • Subsetting and foreign keys
      • Configuring subsetting
      • Viewing and managing configuration inheritance
      • Viewing the subset creation steps
      • Viewing previous subsetting data generation runs
      • Generating cohesive subset data from related databases
      • Other subsetting hints and tips
    • Viewing and adding foreign keys
    • Viewing and resolving schema changes
    • Tracking changes to workspaces, generator presets, and sensitivity rules
    • Using the Privacy Report to verify data protection
  • Running data generation
    • Running data generation jobs
      • Types of data generation
      • Data generation process
      • Running data generation manually
      • Scheduling data generation
      • Issues that prevent data generation
    • Managing data generation performance
    • Viewing and downloading container artifacts
    • Post-job scripts
    • Webhooks
  • Installing and Administering Structural
    • Structural architecture
    • Using Structural securely
    • Deploying a self-hosted Structural instance
      • Deployment checklist
      • System requirements
      • Deploying with Docker Compose
      • Deploying on Kubernetes with Helm
      • Enabling the option to write output data to a container repository
        • Setting up a Kubernetes cluster to use to write output data to a container repository
        • Required access to write destination data to a container repository
      • Entering and updating your license key
      • Setting up host integration
      • Working with the application database
      • Setting up a secret
      • Setting a custom certificate
    • Using Structural Cloud
      • Structural Cloud notes
      • Setting up and managing a Structural Cloud pay-as-you-go subscription
      • Structural Cloud onboarding
    • Managing user access to Structural
      • Structural organizations
      • Determining whether users can create accounts
      • Creating a new account in an existing organization
      • Single sign-on (SSO)
        • Structural user authentication with SSO
        • Enabling and configuring SSO on Structural Cloud
        • Synchronizing SSO groups with Structural
        • Viewing the list of SSO groups in Tonic Structural
        • AWS IAM Identity Center
        • Duo
        • GitHub
        • Google
        • Keycloak
        • Microsoft Entra ID (previously Azure Active Directory)
        • Okta
        • OpenID Connect (OIDC)
        • SAML
      • Managing Structural users
      • Managing permissions
        • About permission sets
        • Built-in permission sets
        • Available permissions
        • Viewing the lists of global and workspace permission sets
        • Configuring custom permission sets
        • Selecting default permission sets
        • Configuring access to global permission sets
        • Setting initial access to all global permissions
        • Granting Account Admin access for a Structural Cloud organization
    • Structural monitoring and logging
      • Monitoring Structural services
      • Performing health checks
      • Downloading the usage report
      • Tracking user access and permissions
      • Redacted and diagnostic (unredacted) logs
      • Data that Tonic.ai collects
      • Verifying and enabling telemetry sharing
    • Configuring environment settings
    • Updating Structural
  • Connecting to your data
    • About data connectors
    • Overview for database administrators
    • Data connector summary
    • Amazon DynamoDB
      • System requirements and limitations for DynamoDB
      • Structural differences and limitations with DynamoDB
      • Before you create a DynamoDB workspace
      • Configuring DynamoDB workspace data connections
    • Amazon EMR
      • Structural process overview for Amazon EMR
      • System requirements for Amazon EMR
      • Structural differences and limitations with Amazon EMR
      • Before you create an Amazon EMR workspace
        • Creating IAM roles for Structural and Amazon EMR
        • Creating Athena workgroups
        • Configuration for cross-account setups
      • Configuring Amazon EMR workspace data connections
    • Amazon Redshift
      • Structural process overview for Amazon Redshift
      • Structural differences and limitations with Amazon Redshift
      • Before you create an Amazon Redshift workspace
        • Required AWS instance profile permissions for Amazon Redshift
        • Setting up the AWS Lambda role for Amazon Redshift
        • AWS KMS permissions for Amazon SQS message encryption
        • Amazon Redshift-specific Structural environment settings
        • Source and destination database permissions for Amazon Redshift
      • Configuring Amazon Redshift workspace data connections
    • Databricks
      • Structural process overview for Databricks
      • System requirements for Databricks
      • Structural differences and limitations with Databricks
      • Before you create a Databricks workspace
        • Granting access to storage
        • Setting up your Databricks cluster
        • Configuring the destination database schema creation
      • Configuring Databricks workspace data connections
    • Db2 for LUW
      • System requirements for Db2 for LUW
      • Structural differences and limitations with Db2 for LUW
      • Before you create a Db2 for LUW workspace
      • Configuring Db2 for LUW workspace data connections
    • File connector
      • Overview of the file connector process
      • Supported file and content types
      • Structural differences and limitations with the file connector
      • Before you create a file connector workspace
      • Configuring the file connector storage type and output options
      • Managing file groups in a file connector workspace
      • Downloading generated file connector files
    • Google BigQuery
      • Structural differences and limitations with Google BigQuery
      • Before you create a Google BigQuery workspace
      • Configuring Google BigQuery workspace data connections
      • Resolving schema changes for de-identified views
    • MongoDB
      • System requirements for MongoDB
      • Structural differences and limitations with MongoDB
      • Configuring MongoDB workspace data connections
      • Other MongoDB hints and tips
    • MySQL
      • System requirements for MySQL
      • Before you create a MySQL workspace
      • Configuring MySQL workspace data connections
    • Oracle
      • Known limitations for Oracle schema objects
      • System requirements for Oracle
      • Structural differences and limitations with Oracle
      • Before you create an Oracle workspace
      • Configuring Oracle workspace data connections
    • PostgreSQL
      • System requirements for PostgreSQL
      • Before you create a PostgreSQL workspace
      • Configuring PostgreSQL workspace data connections
    • Salesforce
      • System requirements for Salesforce
      • Structural differences and limitations with Salesforce
      • Before you create a Salesforce workspace
      • Configuring Salesforce workspace data connections
    • Snowflake on AWS
      • Structural process overviews for Snowflake on AWS
      • Structural differences and limitations with Snowflake on AWS
      • Before you create a Snowflake on AWS workspace
        • Required AWS instance profile permissions for Snowflake on AWS
        • Other configuration for Lambda processing
        • Source and destination database permissions for Snowflake on AWS
        • Configuring whether Structural creates the Snowflake on AWS destination database schema
      • Configuring Snowflake on AWS workspace data connections
    • Snowflake on Azure
      • Structural process overview for Snowflake on Azure
      • Structural differences and limitations with Snowflake on Azure
      • Before you create a Snowflake on Azure workspace
      • Configuring Snowflake on Azure workspace data connections
    • Spark SDK
      • Structural process overview for the Spark SDK
      • Structural differences and limitations with the Spark SDK
      • Configuring Spark SDK workspace data connections
      • Using Spark to run de-identification of the data
    • SQL Server
      • System requirements for SQL Server
      • Before you create a SQL Server workspace
      • Configuring SQL Server workspace data connections
    • Yugabyte
      • System requirements for Yugabyte
      • Structural differences and limitations with Yugabyte
      • Before you create a Yugabyte workspace
      • Configuring Yugabyte workspace data connections
      • Troubleshooting Yugabyte data generation issues
  • Using the Structural API
    • About the Structural API
    • Getting an API token
    • Getting the workspace ID
    • Using the Structural API to perform tasks
      • Configure environment settings
      • Manage generator presets
        • Retrieving the list of generator presets
        • Structure of a generator preset
        • Creating a custom generator preset
        • Updating an existing generator preset
        • Deleting a generator preset
      • Manage custom sensitivity rules
      • Create a workspace
      • Connect to source and destination data
      • Manage file groups in a file connector workspace
      • Assign table modes and filters to source database tables
      • Set column sensitivity
      • Assign generators to columns
        • Getting the generator IDs and available metadata
        • Updating generator configurations
        • Structure of a generator assignment
        • Generator API reference
          • Address (AddressGenerator)
          • Algebraic (AlgebraicGenerator)
          • Alphanumeric String Key (AlphaNumericPkGenerator)
          • Array Character Scramble (ArrayTextMaskGenerator)
          • Array JSON Mask (ArrayJsonMaskGenerator)
          • Array Regex Mask (ArrayRegexMaskGenerator)
          • ASCII Key (AsciiPkGenerator)
          • Business Name (BusinessNameGenerator)
          • Categorical (CategoricalGenerator)
          • Character Scramble (TextMaskGenerator)
          • Character Substitution (StringMaskGenerator)
          • Company Name (CompanyNameGenerator)
          • Conditional (ConditionalGenerator)
          • Constant (ConstantGenerator)
          • Continuous (GaussianGenerator)
          • Cross Table Sum (CrossTableAggregateGenerator)
          • CSV Mask (CsvMaskGenerator)
          • Custom Categorical (CustomCategoricalGenerator)
          • Date Truncation (DateTruncationGenerator)
          • Email (EmailGenerator)
          • Event Timestamps (EventGenerator)
          • File Name (FileNameGenerator)
          • Find and Replace (FindAndReplaceGenerator)
          • FNR (FnrGenerator)
          • Geo (GeoGenerator)
          • HIPAA Address (HipaaAddressGenerator)
          • Hostname (HostnameGenerator)
          • HStore Mask (HStoreMaskGenerator)
          • HTML Mask (HtmlMaskGenerator)
          • Integer Key (IntegerPkGenerator)
          • International Address (InternationalAddressGenerator)
          • IP Address (IPAddressGenerator)
          • JSON Mask (JsonMaskGenerator)
          • MAC Address (MACAddressGenerator)
          • Mongo ObjectId Key (ObjectIdPkGenerator)
          • Name (NameGenerator)
          • Noise Generator (NoiseGenerator)
          • Null (NullGenerator)
          • Numeric String Key (NumericStringPkGenerator)
          • Passthrough (PassthroughGenerator)
          • Phone (USPhoneNumberGenerator)
          • Random Boolean (RandomBooleanGenerator)
          • Random Double (RandomDoubleGenerator)
          • Random Hash (RandomStringGenerator)
          • Random Integer (RandomIntegerGenerator)
          • Random Timestamp (RandomTimestampGenerator)
          • Random UUID (UUIDGenerator)
          • Regex Mask (RegexMaskGenerator)
          • Sequential Integer (UniqueIntegerGenerator)
          • Shipping Container (ShippingContainerGenerator)
          • SIN (SINGenerator)
          • SSN (SsnGenerator)
          • Struct Mask (StructMaskGenerator)
          • Timestamp Shift (TimestampShiftGenerator)
          • Unique Email (UniqueEmailGenerator)
          • URL (UrlGenerator)
          • UUID Key (UuidPkGenerator)
          • XML Mask (XmlMaskGenerator)
      • Configure subsetting
      • Check for and resolve schema changes
      • Run data generation jobs
      • Schedule data generation jobs
    • Example script: Starting a data generation job
    • Example script: Polling for a job status and creating a Docker package
Powered by GitBook
On this page
  • Viewing the list of jobs
  • Information in the job list
  • Job statuses
  • Filtering the job list
  • Sorting the job list
  • Viewing details for a selected job
  • Workspace information
  • Job Log
  • Privacy Report
  • Ephemeral output details
  • Copying the job identifier
  • Canceling a job
  • Downloading job information
  • Job logs
  • Privacy Report for data generation
  • Additional logs for output to a container repository
  • CloudWatch logs for data generation
  • Oracle SQL Loader log files
  • Transformed files for file connector data generation
  • Performance metrics for data generation

Was this helpful?

Export as PDF
  1. Creating and managing workspaces

Viewing workspace jobs and job details

Last updated 2 months ago

Was this helpful?

Tonic Structural runs the following types of jobs on a workspace:

  • Sensitivity scans, which analyze the source database to identify sensitive data.

  • Collection scans, which analyze the source data for a MongoDB workspace to determine the available fields in each collection, the field types, and how prevalent the fields are.

  • Data generation, data pipeline generation, and containerized generation jobs, which generate the destination data from the source data.

  • Upsert data generation jobs, which generate the intermediate database from the source database.

  • Upsert jobs, which use data from the intermediate database to add new rows to and update changed rows in the destination database. If the migration process is enabled, then it is a step in the upsert job.

  • SDK table statistics jobs. These jobs only run when you use the SDK to generate data in a Spark workspace, and the assigned generators require the statistics.

You can view a list of jobs that ran on the workspace, and view details for individual jobs.

Viewing the list of jobs

The Jobs view displays the list of jobs that ran on the workspace. The list includes the 100 most recent jobs.

To display the Jobs view:

  • On the workspace management view, in the workspace navigation bar, click Jobs.

  • On Workspaces view, from the dropdown menu in the Name column, select Jobs.

Information in the job list

For each job, the job list includes the following information:

  • Job ID - The identifier of the job. To copy the job identifier, click the icon at the left of the row.

  • Type - The type of job.

  • Submitted - The date and time when the job was submitted.

  • Completed - The date and time when the job finished running.

Job statuses

A job can have one of the following statuses:

  • Queued - The job is queued to run, but has not yet started. A job is queued for one of the following reasons:

    • Another job is currently running on the same workspace. For example, you cannot run a sensitivity scan and a data generation, or multiple data generations, at the same time on the same workspace. This is true regardless of the number of workers on the instance. On Structural Cloud, there is also a limit on the number of concurrent running jobs for each organization. When that maximum is reached, a new job remains queued until a current running job completes.

    • There isn't an available worker on the instance to run the job. A Structural instance with one worker can only run one job at a time. If a job from one workspace is currently running, a job from another workspace cannot start until the first job is finished.

    To view information about why a job is queued, click the status value.

  • Running - The job is in progress.

  • Canceled - The job is canceled.

  • Completed - The job completed successfully.

  • Failed - The job failed to complete.

Each of these statuses has a corresponding "with warnings" status. For example, Running with warnings, Completed with warnings. A "with warnings" status indicates that the job had at least one warning at the time of the request.

Filtering the job list

You can filter the list by either the type or the status.

To filter the list by the job type:

  1. Click the filter icon in the Type column heading. By default, all types are included, and none of the checkboxes are checked.

  2. To only include specific types of jobs, check the checkbox next to each type to include. Checking all of the checkboxes has the same effect as unchecking all of the checkboxes.

To filter the list by the job status:

  1. Click the filter icon in the Status column heading. The status panel displays all of the statuses that are currently in the list. For example, if there are no Queued jobs, then the Queued status is not in the list. By default, all of the statuses are included, and none of the checkboxes are checked.

  2. To only include jobs that have specific statuses, check the checkbox next to each status to include. Checking all of the checkboxes has the same effect as unchecking all of the checkboxes.

Sorting the job list

You can sort the jobs by either the submission or completion timestamp.

To sort by submission date, click the Submitted column heading. To reverse the sort order, click the heading again.

To sort by completion date, click the Completed column heading. To reverse the sort order, click the heading again.

Viewing details for a selected job

For jobs other than Queued jobs, you can display details about the workspace and the job progress.

From the Jobs view, to display the details for a job, click the job row.

Workspace information

The left side of the job details view contains the workspace information.

For a sensitivity scan, the workspace information is limited to the owner, database type, and worker version.

For a data generation job, the workspace information also includes:

  • Whether subsetting, post-job scripts, or webhooks are used.

  • The number of schemas, tables, and columns in the source database.

  • The number of schemas, tables, and columns in the destination database.

Job Log

The Job Log tab shows the start date, start time, and duration of the job, followed by the list of job process steps.

Privacy Report

For data generation jobs, the Privacy Report tab displays the number of at-risk, protected, and not sensitive columns in the source database.

At-risk columns contain sensitive data, but still have Passthrough as the assigned generator.

Protected columns have an assigned generator other than Passthrough.

Not sensitive columns have Passthrough as the assigned generator, but do not contain sensitive data.

Ephemeral output details

For data generation jobs that write to Ephemeral, the Data available in Tonic Ephemeral panel displays. It contains a link to Ephemeral, and access to either the snapshot or the database.

Snapshot details

When the temporary database is not preserved, the Data available in Tonic Ephemeral panel provides access to the snapshot.

To navigate to Ephemeral and view the details for an Ephemeral snapshot, click View Snapshot in Tonic Ephemeral.

Database connection details

When the temporary database is preserved, the Data available in Tonic Ephemeral panel provides access to the database.

To display the connection details for the Ephemeral database, click View connection info.

For an Ephemeral database, the connection details include the database location and credentials. Each field contains a copy icon to allow you to copy the value.

Copying the job identifier

The job identifier is a unique identifier for the job. To copy the job identifier, either:

Canceling a job

You can cancel Queued or Running jobs.

For jobs with those statuses, the rightmost column in the job list contains a cancel icon.

To cancel the job, click the icon.

Downloading job information

For workspaces that are configured to write destination data to a container repository, the Jobs view also provides access to the generated artifacts. For more information, go to Viewing and downloading container artifacts.

Job logs

Required workspace permission: Download job logs

To download diagnostic logs, you must have the Enable diagnostic logging global permission.

For all jobs, the job logs provide detailed information about the job processing. Tonic.ai support might request the job logs to help diagnose issues.

For a failed data generation to Ephemeral, the job logs include the Ephemeral logs and the destination database pod logs.

For upsert jobs where the migration process is enabled, and you configured the GET Schema Change Logs endpoint, the upsert job logs include the migration process logs.

Where to download the job logs

You can download the job logs from the Jobs view or the job details view. The download includes up to 1MB of log entries.

On the Jobs view, to download the logs for a job, click the download icon in the rightmost column.

On the job details view, to download the logs for a job, click Reports and Logs, then select Job Logs.

Downloading diagnostic logs

By default, Structural redacts sensitive values from the job logs. To help support troubleshooting, you can configure data connectors or an individual data generation job to create unredacted versions of the log files, referred to as diagnostic logs. For more information, go to Redacted and diagnostic (unredacted) logs.

To access diagnostic log files, you must have the Enable diagnostic logging global permission.

If you do not have the Enable diagnostic logging global permission, then you cannot download the logs for that job. The download option is disabled.

Privacy Report for data generation

Required workspace permission: View and download Privacy Report

From the job details view, you can download a Privacy Report file that provides an overview of the current protection status of the database columns based on the workspace configuration at the time that the job ran.

You can download either:

  • The Privacy Report .csv file, which provides details about the table columns, the column content, and the current protection configuration.

  • The Privacy Report PDF file, which provides charts that summarize the privacy ranking scores for the table columns. It also includes the table from the .csv file.

To display the download options, click Reports and Logs. In the menu:

  • To download the Privacy Report .csv file, click Privacy Report CSV.

  • To download the Privacy Report PDF file, click Privacy Report PDF.

For more information about the Privacy Report files and their content, go to Using the Privacy Report to verify data protection.

Additional logs for output to a container repository

  • Database logs - Logs for the database container that is used as the destination.

  • Datapacker logs - Logs for creating the OCI artifact and uploading it to an OCI registry.

To download these logs for a data generation job, on the job details view, click Reports and Logs, then select Database Logs or Datapacker Logs.

CloudWatch logs for data generation

For workspaces that are connected to Amazon Redshift or Snowflake on AWS databases, the data generation job requires multiple calls to a Lambda function. For these data generation jobs, the CloudWatch logs monitor the progress of and display errors for these Lambda function calls.

To download the CloudWatch logs for a data generation job, on the job details view, click Reports and Logs, then select CloudWatch Logs.

The CloudWatch Logs option only displays for Amazon Redshift and Snowflake on AWS data generation jobs.

Oracle SQL Loader log files

Required workspace permission: Download SqlLdr Files

For an Oracle data generation, if both of the following are true:

  • The data generation job ran SQL Loader (sqlldr).

  • sqlldr either failed or succeeded with errors.

Then to download the sqlldr log files, click Reports and Logs, then select sqlldr Logs.

Transformed files for file connector data generation

For a data generation from a file connector workspace that uses local files, you can download the transformed files for that job.

The download is a .zip file that contains the files for a selected file group.

On the job details view, when files are available to download, the Data available for file groups panel displays.

To download the files for a file group:

  1. Click Download Results.

  2. From the list, select the file group. Use the filter field to filter the list by the file group name.

Performance metrics for data generation

Required workspace permission: Download job logs

On the job details view, to download the performance metrics for the job, click Reports and Logs, then click Performance Metrics.

Status - The current status of the job, and how long ago the job reached that status. When you hover over the status, a tooltip displays the actual timestamp for the status change, and the length of time that the job ran. For queued jobs, to display a panel with information about why the job is queued, click the status value.

A workspace can , with an option to preserve the temporary Ephemeral database that is used to create the snapshot.

From the Jobs view, click the copy () icon in the leftmost column.

From the job details view, click the copy () icon next to the job identifier.

For a workspace that , the job includes the following additional logs:

For workspaces that use the newer data generation processing, users can configure a data generation job to also . This is usually done for troubleshooting purposes.

write output to a Tonic Ephemeral snapshot
writes the output to a container repository
Jobs view
Job type filter options for Jobs view
Job status filter options for Jobs view
Job details page for a data generation job
Privacy Report tab on the job details page
Data available in Tonic Ephemeral panel when the temporary database is not preserved
Data available in Tonic Ephemeral panel when the temporary database is preserved
Job list with a Running job that can be canceled
Reports and Logs menu for a data generation job
Reports and Logs dropdown list for generation to a container repository
Job details option to download transformed file connector files
generate performance metrics