LogoLogo
Release notesAPI docsDocs homeStructural CloudTonic.ai
  • Tonic Structural User Guide
  • About Tonic Structural
    • Structural data generation workflow
    • Structural deployment types
    • Structural implementation roles
    • Structural license plans
  • Logging into Structural for the first time
  • Getting started with the Structural free trial
  • Managing your user account
  • Frequently Asked Questions
  • Tutorial videos
  • Creating and managing workspaces
    • Managing workspaces
      • Viewing your list of workspaces
      • Creating, editing, or deleting a workspace
      • Workspace configuration settings
        • Workspace identification and connection type
        • Data connection settings
        • Configuring secrets managers for database connections
        • Data generation settings
        • Enabling and configuring upsert
        • Writing output to Tonic Ephemeral
        • Writing output to a container repository
        • Advanced workspace overrides
      • About the workspace management view
      • About workspace inheritance
      • Assigning tags to a workspace
      • Exporting and importing the workspace configuration
    • Managing access to workspaces
      • Sharing workspace access
      • Transferring ownership of a workspace
    • Viewing workspace jobs and job details
  • Configuring data generation
    • Privacy Hub
    • Database View
      • Viewing and configuring tables
      • Viewing the column list
      • Displaying sample data for a column
      • Configuring an individual column
      • Configuring multiple columns
      • Identifying similar columns
      • Commenting on columns
    • Table View
    • Working with document-based data
      • Performing scans on collections
      • Using Collection View
    • Identifying sensitive data
      • Running the Structural sensitivity scan
      • Manually indicating whether a column is sensitive
      • Built-in sensitivity types that Structural detects
      • Creating and managing custom sensitivity rules
    • Table modes
    • Generator information
      • Generator summary
      • Generator reference
        • Address
        • Algebraic
        • Alphanumeric String Key
        • Array Character Scramble
        • Array JSON Mask
        • Array Regex Mask
        • ASCII Key
        • Business Name
        • Categorical
        • Character Scramble
        • Character Substitution
        • Company Name
        • Conditional
        • Constant
        • Continuous
        • Cross Table Sum
        • CSV Mask
        • Custom Categorical
        • Date Truncation
        • Email
        • Event Timestamps
        • File Name
        • Find and Replace
        • Finnish Personal Identity Code
        • FNR
        • Geo
        • HIPAA Address
        • Hostname
        • HStore Mask
        • HTML Mask
        • Integer Key
        • International Address
        • IP Address
        • JSON Mask
        • MAC Address
        • Mongo ObjectId Key
        • Name
        • Noise Generator
        • Null
        • Numeric String Key
        • Passthrough
        • Phone
        • Random Boolean
        • Random Double
        • Random Hash
        • Random Integer
        • Random Timestamp
        • Random UUID
        • Regex Mask
        • Sequential Integer
        • Shipping Container
        • SIN
        • SSN
        • Struct Mask
        • Timestamp Shift Generator
        • Unique Email
        • URL
        • UUID Key
        • XML Mask
      • Generator characteristics
        • Enabling consistency
        • Linking generators
        • Differential privacy
        • Partitioning a column
        • Data-free generators
        • Supporting uniqueness constraints
        • Format-preserving encryption (FPE)
      • Generator types
        • Composite generators
        • Primary key generators
    • Generator assignment and configuration
      • Reviewing and applying recommended generators
      • Assigning and configuring generators
      • Document View for file connector JSON columns
      • Generator hints and tips
      • Managing generator presets
      • Configuring and using Structural data encryption
      • Custom value processors
    • Subsetting data
      • About subsetting
      • Using table filtering for data warehouses and Spark-based data connectors
      • Viewing the current subsetting configuration
      • Subsetting and foreign keys
      • Configuring subsetting
      • Viewing and managing configuration inheritance
      • Viewing the subset creation steps
      • Viewing previous subsetting data generation runs
      • Generating cohesive subset data from related databases
      • Other subsetting hints and tips
    • Viewing and adding foreign keys
    • Viewing and resolving schema changes
    • Tracking changes to workspaces, generator presets, and sensitivity rules
    • Using the Privacy Report to verify data protection
  • Running data generation
    • Running data generation jobs
      • Types of data generation
      • Data generation process
      • Running data generation manually
      • Scheduling data generation
      • Issues that prevent data generation
    • Managing data generation performance
    • Viewing and downloading container artifacts
    • Post-job scripts
    • Webhooks
  • Installing and Administering Structural
    • Structural architecture
    • Using Structural securely
    • Deploying a self-hosted Structural instance
      • Deployment checklist
      • System requirements
      • Deploying with Docker Compose
      • Deploying on Kubernetes with Helm
      • Enabling the option to write output data to a container repository
        • Setting up a Kubernetes cluster to use to write output data to a container repository
        • Required access to write destination data to a container repository
      • Entering and updating your license key
      • Setting up host integration
      • Working with the application database
      • Setting up a secret
      • Setting a custom certificate
    • Using Structural Cloud
      • Structural Cloud notes
      • Setting up and managing a Structural Cloud pay-as-you-go subscription
      • Structural Cloud onboarding
    • Managing user access to Structural
      • Structural organizations
      • Determining whether users can create accounts
      • Creating a new account in an existing organization
      • Single sign-on (SSO)
        • Structural user authentication with SSO
        • Enabling and configuring SSO on Structural Cloud
        • Synchronizing SSO groups with Structural
        • Viewing the list of SSO groups in Tonic Structural
        • AWS IAM Identity Center
        • Duo
        • GitHub
        • Google
        • Keycloak
        • Microsoft Entra ID (previously Azure Active Directory)
        • Okta
        • OpenID Connect (OIDC)
        • SAML
      • Managing Structural users
      • Managing permissions
        • About permission sets
        • Built-in permission sets
        • Available permissions
        • Viewing the lists of global and workspace permission sets
        • Configuring custom permission sets
        • Selecting default permission sets
        • Configuring access to global permission sets
        • Setting initial access to all global permissions
        • Granting Account Admin access for a Structural Cloud organization
    • Structural monitoring and logging
      • Monitoring Structural services
      • Performing health checks
      • Downloading the usage report
      • Tracking user access and permissions
      • Redacted and diagnostic (unredacted) logs
      • Data that Tonic.ai collects
      • Verifying and enabling telemetry sharing
    • Configuring environment settings
    • Updating Structural
  • Connecting to your data
    • About data connectors
    • Overview for database administrators
    • Data connector summary
    • Amazon DynamoDB
      • System requirements and limitations for DynamoDB
      • Structural differences and limitations with DynamoDB
      • Before you create a DynamoDB workspace
      • Configuring DynamoDB workspace data connections
    • Amazon EMR
      • Structural process overview for Amazon EMR
      • System requirements for Amazon EMR
      • Structural differences and limitations with Amazon EMR
      • Before you create an Amazon EMR workspace
        • Creating IAM roles for Structural and Amazon EMR
        • Creating Athena workgroups
        • Configuration for cross-account setups
      • Configuring Amazon EMR workspace data connections
    • Amazon Redshift
      • Structural process overview for Amazon Redshift
      • Structural differences and limitations with Amazon Redshift
      • Before you create an Amazon Redshift workspace
        • Required AWS instance profile permissions for Amazon Redshift
        • Setting up the AWS Lambda role for Amazon Redshift
        • AWS KMS permissions for Amazon SQS message encryption
        • Amazon Redshift-specific Structural environment settings
        • Source and destination database permissions for Amazon Redshift
      • Configuring Amazon Redshift workspace data connections
    • Databricks
      • Structural process overview for Databricks
      • System requirements for Databricks
      • Structural differences and limitations with Databricks
      • Before you create a Databricks workspace
        • Granting access to storage
        • Setting up your Databricks cluster
        • Configuring the destination database schema creation
      • Configuring Databricks workspace data connections
    • Db2 for LUW
      • System requirements for Db2 for LUW
      • Structural differences and limitations with Db2 for LUW
      • Before you create a Db2 for LUW workspace
      • Configuring Db2 for LUW workspace data connections
    • File connector
      • Overview of the file connector process
      • Supported file and content types
      • Structural differences and limitations with the file connector
      • Before you create a file connector workspace
      • Configuring the file connector storage type and output options
      • Managing file groups in a file connector workspace
      • Downloading generated file connector files
    • Google BigQuery
      • Structural differences and limitations with Google BigQuery
      • Before you create a Google BigQuery workspace
      • Configuring Google BigQuery workspace data connections
      • Resolving schema changes for de-identified views
    • MongoDB
      • System requirements for MongoDB
      • Structural differences and limitations with MongoDB
      • Configuring MongoDB workspace data connections
      • Other MongoDB hints and tips
    • MySQL
      • System requirements for MySQL
      • Before you create a MySQL workspace
      • Configuring MySQL workspace data connections
    • Oracle
      • Known limitations for Oracle schema objects
      • System requirements for Oracle
      • Structural differences and limitations with Oracle
      • Before you create an Oracle workspace
      • Configuring Oracle workspace data connections
      • Troubleshooting Oracle permissions
    • PostgreSQL
      • System requirements for PostgreSQL
      • Before you create a PostgreSQL workspace
      • Configuring PostgreSQL workspace data connections
    • Salesforce
      • System requirements for Salesforce
      • Structural differences and limitations with Salesforce
      • Before you create a Salesforce workspace
      • Configuring Salesforce workspace data connections
    • Snowflake on AWS
      • Structural process overviews for Snowflake on AWS
      • Structural differences and limitations with Snowflake on AWS
      • Before you create a Snowflake on AWS workspace
        • Required AWS instance profile permissions for Snowflake on AWS
        • Other configuration for Lambda processing
        • Source and destination database permissions for Snowflake on AWS
        • Configuring whether Structural creates the Snowflake on AWS destination database schema
      • Configuring Snowflake on AWS workspace data connections
    • Snowflake on Azure
      • Structural process overview for Snowflake on Azure
      • Structural differences and limitations with Snowflake on Azure
      • Before you create a Snowflake on Azure workspace
      • Configuring Snowflake on Azure workspace data connections
    • Spark SDK
      • Structural process overview for the Spark SDK
      • Structural differences and limitations with the Spark SDK
      • Configuring Spark SDK workspace data connections
      • Using Spark to run de-identification of the data
    • SQL Server
      • System requirements for SQL Server
      • Before you create a SQL Server workspace
      • Configuring SQL Server workspace data connections
    • Yugabyte
      • System requirements for Yugabyte
      • Structural differences and limitations with Yugabyte
      • Before you create a Yugabyte workspace
      • Configuring Yugabyte workspace data connections
      • Troubleshooting Yugabyte data generation issues
  • Using the Structural API
    • About the Structural API
    • Getting an API token
    • Getting the workspace ID
    • Using the Structural API to perform tasks
      • Configure environment settings
      • Manage generator presets
        • Retrieving the list of generator presets
        • Structure of a generator preset
        • Creating a custom generator preset
        • Updating an existing generator preset
        • Deleting a generator preset
      • Manage custom sensitivity rules
      • Create a workspace
      • Connect to source and destination data
      • Manage file groups in a file connector workspace
      • Assign table modes and filters to source database tables
      • Set column sensitivity
      • Assign generators to columns
        • Getting the generator IDs and available metadata
        • Updating generator configurations
        • Structure of a generator assignment
        • Generator API reference
          • Address (AddressGenerator)
          • Algebraic (AlgebraicGenerator)
          • Alphanumeric String Key (AlphaNumericPkGenerator)
          • Array Character Scramble (ArrayTextMaskGenerator)
          • Array JSON Mask (ArrayJsonMaskGenerator)
          • Array Regex Mask (ArrayRegexMaskGenerator)
          • ASCII Key (AsciiPkGenerator)
          • Business Name (BusinessNameGenerator)
          • Categorical (CategoricalGenerator)
          • Character Scramble (TextMaskGenerator)
          • Character Substitution (StringMaskGenerator)
          • Company Name (CompanyNameGenerator)
          • Conditional (ConditionalGenerator)
          • Constant (ConstantGenerator)
          • Continuous (GaussianGenerator)
          • Cross Table Sum (CrossTableAggregateGenerator)
          • CSV Mask (CsvMaskGenerator)
          • Custom Categorical (CustomCategoricalGenerator)
          • Date Truncation (DateTruncationGenerator)
          • Email (EmailGenerator)
          • Event Timestamps (EventGenerator)
          • File Name (FileNameGenerator)
          • Find and Replace (FindAndReplaceGenerator)
          • Finnish Personal Identity Code (FinnishPicGenerator)
          • FNR (FnrGenerator)
          • Geo (GeoGenerator)
          • HIPAA Address (HipaaAddressGenerator)
          • Hostname (HostnameGenerator)
          • HStore Mask (HStoreMaskGenerator)
          • HTML Mask (HtmlMaskGenerator)
          • Integer Key (IntegerPkGenerator)
          • International Address (InternationalAddressGenerator)
          • IP Address (IPAddressGenerator)
          • JSON Mask (JsonMaskGenerator)
          • MAC Address (MACAddressGenerator)
          • Mongo ObjectId Key (ObjectIdPkGenerator)
          • Name (NameGenerator)
          • Noise Generator (NoiseGenerator)
          • Null (NullGenerator)
          • Numeric String Key (NumericStringPkGenerator)
          • Passthrough (PassthroughGenerator)
          • Phone (USPhoneNumberGenerator)
          • Random Boolean (RandomBooleanGenerator)
          • Random Double (RandomDoubleGenerator)
          • Random Hash (RandomStringGenerator)
          • Random Integer (RandomIntegerGenerator)
          • Random Timestamp (RandomTimestampGenerator)
          • Random UUID (UUIDGenerator)
          • Regex Mask (RegexMaskGenerator)
          • Sequential Integer (UniqueIntegerGenerator)
          • Shipping Container (ShippingContainerGenerator)
          • SIN (SINGenerator)
          • SSN (SsnGenerator)
          • Struct Mask (StructMaskGenerator)
          • Timestamp Shift (TimestampShiftGenerator)
          • Unique Email (UniqueEmailGenerator)
          • URL (UrlGenerator)
          • UUID Key (UuidPkGenerator)
          • XML Mask (XmlMaskGenerator)
      • Configure subsetting
      • Check for and resolve schema changes
      • Run data generation jobs
      • Schedule data generation jobs
    • Example script: Starting a data generation job
    • Example script: Polling for a job status and creating a Docker package
Powered by GitBook
On this page
  • Self-hosted Structural data collection
  • Data that Tonic.ai does NOT collect, process, or store
  • Analytics telemetry - end-user interactions
  • Application delivery
  • Customer support and account management
  • Debugging and application performance management
  • Structural Cloud analytics data collection
  • Customer data
  • Additional analytics from Structural Cloud
  • Structural log data
  • Schema information
  • Usage information
  • Performance data
  • What is NOT in the log data
  • Viewing the Structural logs that are sent to Tonic.ai

Was this helpful?

Export as PDF
  1. Installing and Administering Structural
  2. Structural monitoring and logging

Data that Tonic.ai collects

Tonic.ai collects telemetry data from the Tonic Structural application. Structural telemetry provides information about how the application is being used. It is primarily used to generate analytics for application usage and performance, but can also be used for debugging, tracking, and troubleshooting.

Structural telemetry includes:

Type
What it includes
How we collect it

Analytics

Data about end-user interactions with our application to understand how the application is used.

Used for product research, roadmap development, debugging, and account management.

Amplitude

Sentry

Logs

Information generated by the application to record its progress and status as it performs its functions.

Includes information such as completed tasks and errors.

Generally used for tracking and troubleshooting.

Amazon Web Services

Sentry

In addition to telemetry data, Tonic.ai collects other information during the course of its interactions with customers.

The following information provides more detail about the types of data that Tonic.ai does and does not collect.

Self-hosted Structural data collection

Most customers self-host the Structural application in their own VPC. Customer data does not leave the customer's environment.

The Structural application transmits telemetry data to Tonic.ai to enable us to perform the following tasks:

  • Manage our accounts

  • Accurately invoice for usage

  • Provide customer support

  • Investigate errors within our application

  • Understand usage to improve product development

Data that Tonic.ai does NOT collect, process, or store

On self-hosted instances, Tonic.ai never sees the following data:

  • Customer data

    • The content of source, destination, and application databases that support the Structural application

  • Datastore credentials

    • URI or IP address of the datastore

    • Credentials (password)

    • Proxy information

Analytics telemetry - end-user interactions

Tonic.ai collects data about end-user interactions with our application to understand how the application is used. We use this data for product research, roadmap development, debugging, and account management.

Tonic.ai collects the following information about end-user interactions:

  • End-user identity

    • First and last name

    • Email address

  • End-user interaction with the Structural application:

    • Last seen

    • First seen

    • Usage time

    • Total sessions

    • Total number of events initiated. Events can include jobs, configuration updates, downloads, database views, and interactions with the workspace.

  • Application environment

    • Database type

    • Features enabled

    • Application version

    • License tier

  • Location - derived from GeoIP

    • Country

    • City

    • Region (state, province, county)

    • Designated market area (DMA)

  • Language

  • Browser used to access the application

    • Platform (iOS, Android, Web)

    • Operating system

    • Device family (iPhone, Samsung Galaxy, Windows)

    • Device type (iPhone 13, MacBook Pro)

    • Carrier (AT&T, Verizon)

  • Network and technical identifiers

    • IP address

    • Unique device identifier

Application delivery

To build and deploy software, Tonic.ai uses a container registry that is run by Quay.io. This container registry maintains information about access to these containers.

The registry maintains a list of authorized users (organizational accounts). It maintains, collects, and stores the following information:

  • Network and technical identifiers

    • IP address

    • Unique device identifier

    • Operating system

Customer support and account management

Tonic.ai collects, processes, and stores information about end users:

  • When they interact with our customer support and success staff during account implementation (scoping sessions, implementation calls).

  • Throughout the life of the account, during customer support interactions (support emails, shared Slack channels).

Tonic.ai uses several tools to allow our customers to get the support they need quickly, including:

  • Chat support

  • Video training and implementation calls over web conferencing solutions

  • Email support

We aggregate requests from these tools into our Customer Management System (CMS) and our centralized customer support management portal. Aggregating these requests helps us to ensure responsiveness and quality, and to more easily integrate requests into our development process.

We collect the following information related to customer requests:

  • End-user identity

    • First and last name

    • Email address

    • Title

    • Avatar image

    • Images, video, or audio from participating in live training over a video or audio conference

    • Other personal information that the service provider collects and shares. For example, Google Mail collects voluntary directory information that it shares with email recipients. For an email interaction, Tonic.ai receives any information that is configured to be shared externally. Slack has configurable profiles that contain additional personal information such as pronouns and honorifics.

  • Network and technical identifiers

    • IP address

    • Unique device identifier

This data is collected from your organization and users through communication with our staff. The Structural application does not collect this data.

Debugging and application performance management

Tonic.ai engineers monitor the application performance and errors. They use this information to maintain, repair, and improve the application.

For these purposes, Tonic.ai collects the following information:

  • End-user identity

    • First and last name

    • Email address

  • Environment details

    • Name

    • Application version

  • Requests made by the application

    • URLs

    • Header information

    • HTTP POST parameters

    URL query parameters in exception messages are redacted when they are captured. The capturing agent replaces them with "". They are never sent to Tonic.ai.

  • Stack traces and exceptions

    • Method arguments

    • Classes called

    • Processing time

    • CPU usage

  • Location of error (application, file, and line)

    • Database queries

    • Database

    • Database table and names

    • Relationships between columns and tables

    WHERE clause literals are redacted when they are captured. The capturing agent replaces them with "". They are never sent to Tonic.ai.

  • Network and technical identifiers

    • IP address

    • Hostname

    • Unique device identifier

  • Operating system logs

Structural Cloud analytics data collection

Customers who do not self-host Structural can use the hosted option, Structural Cloud.

Structural Cloud collects, processes, and stores data to support the Structural application.

Structural Cloud stores information about end users, configuration, hashed passwords, and datastore connections.

Customer data

Structural Cloud does not store data from source databases. It does process customer data in memory during scans and jobs.

Structural Cloud collects the following customer data:

  • End-user identity

    • First and last name

    • Email address

    • Job title

    • Avatar image

  • Application environment

    • Database type

    • Features enabled

    • Application version being run

    • License tier

  • Location - Derived from the GeoIP

    • Country

    • City

    • Region (state, province, county)

    • Designated market area (DMA)

  • Language

  • Browser used to access the application

    • Platform (iOS, Android, Web)

    • Operating system

    • Device family (iPhone, Samsung Galaxy, Windows)

    • Device type (iPhone 13, MacBook Pro)

    • Carrier (AT&T, Verizon)

    • Network and technical identifiers

    • IP address

    • Unique device identifier

  • Datastore credentials

    • URI or IP address of the datastore

    • Credentials (password)

    • Proxy information

Additional analytics from Structural Cloud

Organizations in our hosted environment may also have additional analytics data collected, processed, and stored. This additional data allows Tonic.ai to replay their user sessions to better understand usage patterns.

Sensitive data is redacted from these collections on the end-user device.

This data is not collected from self-hosted customers.

Structural Cloud collects the following additional analytics data:

  • Usage patterns

  • Clicks

  • Mouse movements

  • Scrolling

  • Typing - Excludes data that is typed in sensitive fields such as password fields

  • Navigation

  • Pages visited

  • Referrers

  • URL parameters

  • Session duration

Structural log data

Structural log files are stored in an S3 bucket for one year.

Tonic.ai uses a log aggregator to make the log files searchable. On the log aggregator, job logs are deleted after six months. API logs are deleted after 60 days.

Schema information

Structural logs include detailed schema information for your database, including:

  • Table, schema, and column names

  • Data types

  • Table sizes

Usage information

Structural logs include many types of usage information, including information related to:

  • Actions in the user interface, from web requests that the web server sees.

  • Details related to data generation.

  • Workspace configuration details, such as the generators that are applied to each column.

Performance data

Tonic.ai collects detailed performance data for the generation process, including data transfer rates and code profiler results.

What is NOT in the log data

Structural has strong safeguards in place to ensure that actual data does not leak into logs.

Logs that are shared with Tonic.ai are always redacted. Structural does not send diagnostic logs to Tonic.ai.

Structural does not log information related to the database connection, such as the database username, password, and host.

Viewing the Structural logs that are sent to Tonic.ai

Structural writes all logs to STDOUT. To view the exact logs that are collected and shared, view what is written to STDOUT.

If the Structural container runs in Docker, you can run:

docker logs tonic_worker

Last updated 26 days ago

Was this helpful?

Tonic Structural log architecture