Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Required global permission: Update Tonic Structural
On the System Status tab of Structural Settings view, the Application Status section displays the list of Tonic Structural services that run on your Structural instance.
To be able to view the Structural service status and download service logs, you must set up Structural host integration. Host integration also enables you to update Structural.
Structural services include:
Structural web server
Structural workers
Structural PII detection
Structural machine learning
Structural notifications
Structural application database, if the database is deployed in a Docker container
For each service, the list includes
Name of the service
The current status of the service. The status indicates whether the service is currently running. If it is running, then the status information includes the amount of time that the service has been running.
You can sort the list by either the service name or status. To sort by a column, click the column header. To reverse the sort order, click the column header again.
From the service list, you can download the service logs. To download the logs, click the Download Logs button for the service.
Tonic Structural job logs can contain sensitive data values. To avoid leaking these values, by default Structural redacts sensitive values from the logs. The logs indicate when data is redacted.
For data generation, you can optionally have Structural generate logs that are not redacted, referred to in Structural as diagnostic logs. Diagnostic logs are encrypted in the Structural application database, and are not shared with Tonic.ai.
You typically use diagnostic logging to help troubleshoot issues with data generation.
When you set any of the following environment settings to true, the corresponding logs are always considered diagnostic logs:
For Amazon Redshift and PostgreSQL:
NPGSQL_LOGGING_ENABLED - Whether to enable diagnostic logging. Increases the logging of the Npgsql database driver.
NPGSQL_LOG_PARAMETERS - Whether to de-anonymize the parameter values in the logged queries.
For Db2 for LUW:
TONIC_DB2_SQL_LOGGING_ENABLED - Whether to enable diagnostic logging.
TONIC_DB2_SQL_LOG_PARAMETERS - Whether to de-anonymize the parameter values in the logged queries.
For MongoDB:
MONGO_MQL_LOGGING_ENABLED - Whether to enable diagnostic logging. Increases the logging of the MongoDB database driver.
For MySQL:
MYSQL_CONNECTOR_LOGGING_ENABLED - Whether to enable diagnostic logging. Increases the logging of the MySqlConnector database driver.
For Oracle:
TONIC_ORACLE_PLSQL_LOGGING_ENABLED - Whether to enable diagnostic logging. Increases the logging of the Oracle database driver.
TONIC_ORACLE_PLSQL_LOG_PARAMETERS - Whether to de-anonymize the parameter values in the logged queries.
TONIC_ORACLE_ENABLE_SQLLDR_ERROR_LOGGING - Whether to generate Oracle sqlldr logs. Note that access to the sqlldr logs is controlled by the Download Sqlldr Files workspace permission, and not the Enable diagnostic logging global permission.
For SQL Server:
TONIC_SQL_SERVER_TSQL_LOGGING_ENABLED - Whether to enable diagnostic logging. Increases the logging of the SQL Server SMO database driver.
TONIC_SQL_SERVER_TSQL_LOG_PARAMETERS - Whether to de-anonymize the parameter values in logged queries.
Required global permission: Enable diagnostic logging
If the Structural instance is not configured to generate diagnostic logs for the workspace data connector, then you can configure whether to use diagnostic logging for an individual job.
If diagnostic logging is enabled because of an environment setting, then you cannot disable it.
For details, go to #data-gen-confirm-details-diagnostic-logging.
Tonic.ai collects telemetry data from the Tonic Structural application. Structural telemetry provides information about how the application is being used. It is primarily used to generate analytics for application usage and performance, but can also be used for debugging, tracking, and troubleshooting.
Structural telemetry includes:
In addition to telemetry data, Tonic.ai collects other information during the course of its interactions with customers.
The following information provides more detail about the types of data that Tonic.ai does and does not collect.
Most customers self-host the Structural application in their own VPC. Customer data does not leave the customer's environment.
The Structural application transmits telemetry data to Tonic.ai to enable us to perform the following tasks:
Manage our accounts
Accurately invoice for usage
Provide customer support
Investigate errors within our application
Understand usage to improve product development
On self-hosted instances, Tonic.ai never sees the following data:
Customer data
The content of source, destination, and application databases that support the Structural application
Datastore credentials
URI or IP address of the datastore
Credentials (password)
Proxy information
Tonic.ai collects data about end-user interactions with our application to understand how the application is used. We use this data for product research, roadmap development, debugging, and account management.
Tonic.ai collects the following information about end-user interactions:
End-user identity
First and last name
Email address
End-user interaction with the Structural application:
Last seen
First seen
Usage time
Total sessions
Total number of events initiated. Events can include jobs, configuration updates, downloads, database views, and interactions with the workspace.
Application environment
Database type
Features enabled
Application version
License tier
Location - derived from GeoIP
Country
City
Region (state, province, county)
Designated market area (DMA)
Language
Browser used to access the application
Platform (iOS, Android, Web)
Operating system
Device family (iPhone, Samsung Galaxy, Windows)
Device type (iPhone 13, MacBook Pro)
Carrier (AT&T, Verizon)
Network and technical identifiers
IP address
Unique device identifier
To build and deploy software, Tonic.ai uses a container registry that is run by Quay.io. This container registry maintains information about access to these containers.
The registry maintains a list of authorized users (organizational accounts). It maintains, collects, and stores the following information:
Network and technical identifiers
IP address
Unique device identifier
Operating system
Tonic.ai collects, processes, and stores information about end users:
When they interact with our customer support and success staff during account implementation (scoping sessions, implementation calls).
Throughout the life of the account, during customer support interactions (support emails, shared Slack channels).
Tonic.ai uses several tools to allow our customers to get the support they need quickly, including:
Chat support
Video training and implementation calls over web conferencing solutions
Email support
We aggregate requests from these tools into our Customer Management System (CMS) and our centralized customer support management portal. Aggregating these requests helps us to ensure responsiveness and quality, and to more easily integrate requests into our development process.
We collect the following information related to customer requests:
End-user identity
First and last name
Email address
Title
Avatar image
Images, video, or audio from participating in live training over a video or audio conference
Other personal information that the service provider collects and shares. For example, Google Mail collects voluntary directory information that it shares with email recipients. For an email interaction, Tonic.ai receives any information that is configured to be shared externally. Slack has configurable profiles that contain additional personal information such as pronouns and honorifics.
Network and technical identifiers
IP address
Unique device identifier
This data is collected from your organization and users through communication with our staff. The Structural application does not collect this data.
Tonic.ai engineers monitor the application performance and errors. They use this information to maintain, repair, and improve the application.
For these purposes, Tonic.ai collects the following information:
End-user identity
First and last name
Email address
Environment details
Name
Application version
Requests made by the application
URLs
Header information
HTTP POST parameters
URL query parameters in exception messages are redacted when they are captured. The capturing agent replaces them with "". They are never sent to Tonic.ai.
Stack traces and exceptions
Method arguments
Classes called
Processing time
CPU usage
Location of error (application, file, and line)
Database queries
Database
Database table and names
Relationships between columns and tables
WHERE clause literals are redacted when they are captured. The capturing agent replaces them with "". They are never sent to Tonic.ai.
Network and technical identifiers
IP address
Hostname
Unique device identifier
Operating system logs
Customers who do not self-host Structural can use the hosted option, Structural Cloud.
Structural Cloud collects, processes, and stores data to support the Structural application.
Structural Cloud stores information about end users, configuration, hashed passwords, and datastore connections.
Structural Cloud does not store data from source databases. It does process customer data in memory during scans and jobs.
Structural Cloud collects the following customer data:
End-user identity
First and last name
Email address
Job title
Avatar image
Application environment
Database type
Features enabled
Application version being run
License tier
Location - Derived from the GeoIP
Country
City
Region (state, province, county)
Designated market area (DMA)
Language
Browser used to access the application
Platform (iOS, Android, Web)
Operating system
Device family (iPhone, Samsung Galaxy, Windows)
Device type (iPhone 13, MacBook Pro)
Carrier (AT&T, Verizon)
Network and technical identifiers
IP address
Unique device identifier
Datastore credentials
URI or IP address of the datastore
Credentials (password)
Proxy information
Organizations in our hosted environment may also have additional analytics data collected, processed, and stored. This additional data allows Tonic.ai to replay their user sessions to better understand usage patterns.
Sensitive data is redacted from these collections on the end-user device.
This data is not collected from self-hosted customers.
Structural Cloud collects the following additional analytics data:
Usage patterns
Clicks
Mouse movements
Scrolling
Typing - Excludes data that is typed in sensitive fields such as password fields
Navigation
Pages visited
Referrers
URL parameters
Session duration
Structural log files are stored in an S3 bucket for one year.
Tonic.ai uses a log aggregator to make the log files searchable. On the log aggregator, job logs are deleted after six months. API logs are deleted after 60 days.
Structural logs include detailed schema information for your database, including:
Table, schema, and column names
Data types
Table sizes
Structural logs include many types of usage information, including information related to:
Actions in the user interface, from web requests that the web server sees
Details related to data generation
Workspace configuration details, such as the generators that are applied to each column
Tonic.ai collects detailed performance data for the generation process, including data transfer rates and code profiler results.
Structural has strong safeguards in place to ensure that actual data does not leak into logs.
Logs that are shared with Tonic.ai are always redacted. Structural does not send diagnostic logs to Tonic.ai.
Structural does not log information related to the database connection, such as the database username, password, and host.
Structural writes all logs to STDOUT. To view the exact logs that are collected and shared, view what is written to STDOUT.
If the Structural container runs in Docker, you can run:
Monitor Structural services
Track the current status of Structural services, including Structural workers and the Structural application database.
Perform health checks
Check the current health of the API server and Structural workers.
Download a usage report
Track the tables that Structural processed during data generation.
Track user access and permissions
Use an API request to retrieve lists of user events.
Share logs with Tonic.ai
Tonic.ai uses log information to help troubleshoot issues.
Data that Tonic.ai collects
Learn about the types of data that Tonic.ai collects from your instance.
Analytics
Data about end-user interactions with our application to understand how the application is used.
Used for product research, roadmap development, debugging, and account management.
Amplitude
Sentry
Logs
Information generated by the application to record its progress and status as it performs its functions.
Includes information such as completed tasks and errors.
Generally used for tracking and troubleshooting.
Amazon Web Services
Sentry
Fullstory
Sharing of analytics telemetry is required and enforced. Log sharing is also enabled by default. If you do not share logs with Tonic.ai, then identifying problems and troubleshooting issues becomes a slower, more manual process.
For details about all of the data that Tonic.ai collects from Structural, go to Data that Tonic.ai collects.
To ensure that Tonic.ai receives telemetry from your instance, verify the following.
To ensure that Tonic.ai can receive telemetry from Structural, allowlist the telemetry URLs.
Log sharing is enabled on each individual Structural container.
If log sharing becomes disabled, then to enable log sharing in a container, add the following environment settings to the Structural web server container:
Tonic.ai continuously verifies that it receives analytics telemetry from Structural.
If Tonic.ai cannot receive analytics telemetry, then:
After it does not receive analytics telemetry for 5 days, Structural displays warnings as a banner and on the Confirm Generation panel. The warnings include the number of days until data generation is disabled, if analytics telemetry sharing is not enabled.
After it does not receive analytics telemetry for 15 days, Structural disables data generation. The warnings change to errors.
When Tonic.ai begins to receive analytics telemetry, data generation is re-enabled.
Required global permission: Update Tonic Structural
From Structural Settings, you can check the status of telemetry sharing with Tonic.ai.
On the System Status tab, under Data Sharing:
The Telemetry panel indicates whether:
Analytics telemetry sharing is enabled.
If it is enabled, then whether Structural can make the connection to send the analytics telemetry data.
The Logging panel indicates whether:
Log sharing is enabled.
If it is enabled, then whether Structural can make the connection to send the logs.
Required global permission: View summary usage metrics
The usage report summarizes the number of table rows and cells that data generation jobs applied generators to.
The usage report was introduced in V876. The report does not include jobs that ran before V876 was released.
To download the report, on Structural Settings view, click Download Usage Report.
The report is downloaded as usage_report.csv.
Each entry in the report represents a table that was processed by a data generation job.
The report contains the following columns:
Date - The date and time when the data generation job completed.
WorkspaceName - The name of the Tonic Structural workspace.
WorkspaceId - The identifier of the Structural workspace.
JobId - The identifier of the data generation job.
TableMode - How the table was processed.
PassThrough indicates that none of the columns had assigned generators.
Masked indicates that the data generation job applied generators to table columns.
Truncated indicates that the table mode was Truncate. No generators were applied.
TableName - The name of the table.
RowCount - The number of rows in the table that were processed. Indicates that generators were applied or that the rows were included in a subset.
CellCount - The number of processed rows multiplied by the total number of columns in the table.
The environment setting USAGE_ROWS_COUNT_LIMIT determines the maximum number of rows to include in the report. When the number of rows exceeds this limit, Structural removes rows from the oldest jobs one job at a time until the number of rows is within the limit.
To determine the value, use the following formula:
<number of days> * <number of jobs per day> * <number of tables>
The default value is 450,000. This value was calculated based on 90 days with 5 jobs per day that each process 1000 tables.
For information on how to change an environment setting value, go to Configuring environment settings.
Tonic Structural provides an API endpoint to track the following events related to user access and permissions:
A user account is created.
A user account is removed.
A user logs in to Structural.
A permission is added to or removed from a permission set.
A permission set is assigned to or removed from a user. This might be a global permission set, or a workspace permission set in the context of a specific workspace.
A generator preset is added, updated, or removed.
To request the access and permission events, use the following endpoint:
The returned event details include:
When the event occurred.
The type of event.
Whether the event succeeded or failed.
The Structural user who was affected by the action.
Other details about the event.
The API server and the worker both have health check endpoints that you can use to check whether your Tonic Structural instance is available.
If the service is up and all dependent resources can be reached, the response is Healthy with a status code of 200.
Otherwise, the response is Unhealthy with a status code of 503. The logs for that container indicate the specific resources that could not be reached.