Install

A Tonic Structural Helm chart is located at: https://github.com/TonicAI/tonic_helm_charts.

To use the Helm chart, you can either:

• Use the OCI-based registry that Tonic hosts on Quay.io.

• Fork or clone the repository and then maintain it locally.

The Structural images are hosted on Quay.io. During the onboarding period, you are provided access credentials to our repositories. If you require new credentials, or you experience issues accessing the repository, contact support@tonic.ai.

Configure

Before you deploy Structural, you create a values.yaml file with the configuration for your instance.

For details about the required and optional configuration options, go to the repository readme.

Deploy

To deploy and validate access to Structural from the forked repository, follow the instructions in the repository readme.

To use the OCI-based registry, run:

helm install structural oci://quay.io/tonicai/structural -f values.yaml -n structural --create-namespace

Update

Structural notifies you when the current version is more than 10 versions behind the most recent release. The notification is on the System Status tab of Structural Settings view.

To get the latest Structural version, users with the Update Tonic global permission can use the in-app update feature.

Alternatively, if you need to specify a particular version of Structural to use, set tonicVersion, then run the following:

$ helm upgrade <name_of_release> -n <namespace_name> <path-to-helm-chart>

To upgrade using the OCI-based repository, run:

helm upgrade -n structural structural oci://quay.io/tonicai/structural -f values.yaml --version "1.${TONIC_RELEASE}.0"

Tonic Structural stores and encrypts your database credentials.

To use your own encryption key, during initial installation, add the environment setting TONIC_SECRET to both the Structural worker and Structural web server containers.

TONIC_SECRET must have the same value in both containers. It can be any string.

TONIC_SECRET is encrypted using AES. By default, the encryption uses a 128-bit key. It can also support a 256-bit key.

A Docker instance of Tonic Structural does not automatically support the option to write destination data to a container artifact.

To enable this option, you can set up a separate Kubernetes cluster. You then configure Structural environment settings to enable Structural to use that Kubernetes cluster as the destination location.

Installing the Kubernetes cluster

You can install the Kubernetes cluster on the same server where Docker is installed, or on a remote host that has network access to the Docker server.

You can use any compatible Kubernetes distribution. Here are links to the installation instructions for a few different options that will work:

• minikube

• K3s

• K0s

The Structural service account must have the permissions listed in Required access to write destination data to container artifacts.

Setting the host location in the mounted kubeconfig

In the kubeconfig file, you must change the server property value from localhost to either:

• If the cluster is remote, the Kubernetes host IP address or hostname

• If the cluster is on the same host, to host.docker.internal

clusters:
- cluster:
certificate-authority-data: ...
server: <IP address or hostname, or host.docker.internal>
name: ..

Configuring Structural environment settings

To allow Structural to connect to the Kubernetes cluster and to write destination data to it, you must configure the following environment settings.

You can add these settings manually to the list on the Environment Settings tab of Structural Settings.

Specifying the kubeconfig path

To allow Structural to write output data to the Kubernetes cluster, Structural also needs the path where kubeconfig is mounted to the Structural worker.

In the Docker Compose file, to specify the kubeconfig path, add the KUBECONFIG environment variable to the tonic_worker environment section.

tonic_worker:
  environment:
    KUBECONFIG: <fully qualified kubeconfig path>

For a self-hosted Tonic Structural instance, you deploy Structural to a public cloud account (for example, AWS, GCP, or Azure) or data center.

Use this checklist to prepare to install Structural. Structural architecture includes a diagram of the Structural components and how they are connected.

Structural application server

Provision a server that meets the required specifications.

Kubernetes or Docker environment

You deploy Structural to either a Kubernetes cluster or a Docker container. Ensure that the Kubernetes or Docker environment meets the required specifications:

• Kubernetes environment requirements

• Docker environment requirements

Structural application database

Provision a PostgreSQL database that meets the required specifications.

Determine whether to host the Structural application database in Docker.

Determine whether to select a different schema for the application database. By default, the schema is public.

Required network access

To ensure a smooth installation and configuration process, all of the Structural components must have the appropriate network configurations.

Source and destination databases

Source databases contain the original data for Structural data generation or data science mode. For Structural data generation, Structural writes the transformed data to a destination database.

Overview for database administrators contains an overview of the requirements for Structural source and destination databases.

For PostgreSQL and MySQL workspaces, you can configure Tonic Structural to write destination data to a container artifact instead of to a database server. For more information, go to Writing output to a container repository.

If Structural is deployed on Kubernetes, then the option is supported automatically.

If Structural is deployed on Docker, then to enable the option, you can set up a separate Kubernetes cluster to use specifically for that purpose.

For a self-hosted instance of Tonic Structural, you install Structural in your own environment.

You install Tonic Structural either on in a Docker container or on a Kubernetes cluster.

You cannot deploy Structural on Mac computers with Apple silicon (M1, M2).

Host environment

The server or cluster that you deploy Structural to must at a minimum have access to the following resources:

If your source database is larger than 500 GB and/or contains "large" (~1+ MB values) data such as JSONB, XML, NVARCHAR(max), then add 16GB to the minimum memory.

If you have questions about the number of resources to allocate based on your source databases, contact support@tonic.ai.

Docker environment

For Docker, our recommendation is to use Linux with Docker and Docker Compose installed.

When you deploy Structural using Docker:

• Both Docker and Docker Compose must be installed on the machine.

• The Docker Daemon must be running.

The minimum required Docker version for future Structural compatibility is 20.10.10.

• Installation instructions for Docker on Ubuntu

Note that by default, Docker sets its MTU (maximum transmission unit) to 1500. Make sure that the MTU for the Docker network matches your environment. Some networks, such as the Google Cloud Platform (GCP) VPCs, have a lower default MTU (1460 in the case of GCP VPCs), which causes network problems for your Structural instance.

To change the Docker MTU setting, you must:

• Change it in the Docker daemon configuration

• On the Structural Docker network, set the driver bridge option com.docker.network.driver.mtu

Kubernetes environment

You can deploy Structural manually to a Kubernetes cluster. For a manual deployment, you must have:

• A cluster created and configured.

• A namespace to deploy Structural.

Both kubectl and helm must be installed on the machine. The minimum acceptable versions are:

• kubectl: 1.17+

• helm: 3+

Amazon ECS

You can run Structural on Amazon Elastic Container Service (Amazon ECS) on either Fargate or Amazon Elastic Compute Cloud (Amazon EC2) hosts.

Depending on your requirements, this can be either as a single or multiple task definitions.

For more information about Structural in Amazon ECS or example task definition files, contact Tonic.ai support.

Access to quay.io

The Structural images are obtained from quay.io. For manual deployment, Structural provides the required credentials for you to use.

If possible, allowlist *.quay.io.

If you cannot use a wildcard, then you can allowlist specific quay.io URLs. For example: quay.io, cdn.quay.io, cdn01.quay.io, cdn02.quay.io, cdn03.quay.io. The actual URLs are controlled by Quay. The following information on configuring a firewall includes a list of quay.io URLs.

If you cannot allowlist based on DNS names, you can allowlist the IP addresses. To get the IP addresses for a URL, run the URL through nslookup (for example, nslookup cdn01.quay.io). You then allowlist those IP addresses.

Structural application database

The Structural application database (sometimes referred to as the metadata database) is a PostgreSQL database that stores the workspace and Structural configuration.

External database server

In most cases, the Structural application database is an external database that is hosted on a separate server.

Instead of an external database, a deployment to Docker also provides an option to run PostgreSQL in a Docker container.

For an external database, one small host (for example, an RDS t3.small on AWS with at least 100 GB of storage) can serve as the PostgreSQL server.

If you plan to create file connector workspaces that use files from a local file system, make sure that the storage space can accommodate the uploaded and generated files.

To prevent the loss of Structural metadata, keep regular backups of the PostgreSQL instance.

PostgreSQL version

For the Structural application database, the current minimum supported version is PostgreSQL 12+. The current recommended version is 14+.

You should keep your PostgreSQL version relatively up-to-date with the current PostgreSQL LTS.

Tonic.ai might periodically conduct a campaign to request updates of self-hosted PostgreSQL instances before a scheduled update in the minimum supported version.

Database user permissions

The user credentials that you provide to Structural for the application database must have permission to create a database, create tables, insert, and select.

uuid-ossp extension

The Structural application database requires the uuid-ossp extension

You must either:

• Install the uuid-ossp extension.

• Grant the account that Structural uses the necessary permissions to create the extension.

Docker container option for the Structural application database

For a deployment to Docker, you have the option to run PostgreSQL in a Docker container on the Structural application server.

If you use this configuration, mount the data directory for the PostgreSQL container on the host machine and schedule regular backups.

To enable this, uncomment the optional section in the docker-compose.yaml file that Structural provides.

Schema selection for the Structural application database

By default, Structural uses the public schema. You can optionally select a different schema.

Network access for Structural updates

For a Kubernetes deployment, Structural Settings view includes an option to update Structural to the latest version.

To enable this option, Structural requires access to temp-tonic-release.s3.amazonaws.com.

Install

A Tonic Structural Docker Compose repository is located here: .

To make future updates easier, fork this repository.

The repository readme includes more detail on how to set environment settings. It also provides information on how to determine which containers are required for your deployment. For example, whether you use Docker to deploy the Structural application PostgreSQL database.

When you sign up with Structural, you are provided access credentials to our image repository. If you require new credentials, or you are unable to access the repository, contact to get access to our Quay.io Docker repository.

On the machine where you plan to deploy Structural, log in to Quay.io with credentials that Structural provides:

Configure

Review the .

Deploy

To deploy and validate access to Structural, follow the .

Update

Structural notifies you when the current version is more than 10 versions behind the most recent release. The notification is on the System Status tab of Structural Settings view.

At a minimum, to update Structural, run the following:

To free additional disk space before you complete the update, you can optionally include commands to remove unused images and volumes.

Host integration allows Tonic Structural to reach outside of the the Structural software so that it can interact with the system that runs Structural.

You need to configure host integration so that you can:

• For Kubernetes, use the one-click option to . In addition to configuring the host integration properties, your instance must have at least one user that has the Update Tonic global permission.

Kubernetes

In your file, under tonicai.web_server.features, set the following properties:

Docker

In your , uncomment and set the following properties:

For information about setting these values, go to .

In docker-compose.yaml, make sure that services.tonic_web_server.environment refers to the values set in the environment file.

Tonic Structural uses a PostgreSQL database to store its internal state. This database does not store customer data. It only stores data that Structural needs to operate.

Selecting the schema for the Structural application database

By default, Structural creates the application database tables in the public schema.

To use a different schema, set the environment setting TONIC_DB_SCHEMA to the schema name.

Note that you cannot set this setting from the Environment Settings tab on Structural Settings. You must set it from a .yaml file and then restart Structural.

Also note that if you change the schema, you should do it when you first install Structural.

If you change the setting on a running instance, it deploys the application schema objects into the new schema beside the existing public schema, but it does not migrate the data from public.

When the application launches after the schema change, it looks like a new instance of Structural. For example, it has none of your user accounts or workspaces. To restore the data, you must manually move it from the public schema to the new schema.

Connecting to the Structural application database

When you initially set up Structural, you provide Structural with connection details to a PostgreSQL database. How to connect depends on where you set up your PostgreSQL database.

Inside a Docker container

Some customers set up a PostgreSQL database inside the same Docker network as the other Structural containers. In that case, you should ensure that:

• Port 5432 is exposed on the PostgreSQL Docker container.

• Port 5432 is properly mapped.

To do this, add the following section to the PostgreSQL service section of your docker-compose file:

#This will expose Port 5432 on the docker container and map it to
#Port 5432 on the host machine

ports:
    - 5432:5432

Standalone PostgreSQL or cloud installation

If you installed PostgreSQL in a standalone fashion, or you use a cloud service such as Amazon RDS, then make sure that the firewall settings and security groups allow a connection on the appropriate port.

Making the connection

You can use any PostgreSQL client. The following example uses psql, the PostgreSQL command line client.

#Format
#psql -h <HOST> -p <PORT> -U <user> -d <Tonic DB name>

#Example
psql -h localhost -p 5432 -U pguser -d allos

Other Structural application database tips and troubleshooting

How can I reduce the size of the Structural application database?

The Structural application database includes a ColumnSchema table that contains the column schema information for your workspace source databases.

By default, when you delete a workspace, Structural does a soft delete. It marks the workspace as deleted, but does not remove the associated rows from the ColumnSchema table.

To help prevent the table from growing too large, you can configure Structural to instead do a hard delete, and remove the ColumnSchema rows that are associated with a deleted workspace.

To have Structural remove the ColumnSchema rows for deleted workspaces, set the environment setting TONIC_DELETE_COLUMN_SCHEMA_ON_WORKSPACE_DELETE to true. You can add this setting manually to the Environment Settings tab on Structural Settings.

Tonic Structural provides a certificate for https traffic, but can also use a user-provided certificate. The certificate must use the the PFX format and be named tonic.pfx.

To use your own certificate, you must:

• Add the TONIC_PFX_PASSWORD environment setting.

• Use a volume mount to provide the certificate file. Structural uses volume mounting to give the Structural containers access to the certificate.

You must apply the changes to both the Structural web server and Structural worker containers.

Docker

To use your own certificate, you make the following changes to the docker-compose.yml file.

Environment setting

You must add the environment setting TONIC_PFX_PASSWORD, which contains the certificate password.

Volume mount

You place the certificate on the host machine, then share it to the containers as a volume.

You must map the certificate to /certificates on the containers.

volumes:
        ...
        - /my-host-path:/certificates

Kubernetes

Environment setting

You must add the environment setting TONIC_PFX_PASSWORD, which contains the certificate password.

Volume mount

You can use any volume type that is allowed within your environment. It must provide at least ReadOnlyMany access.

You must map the certificate to /certificates on the containers. Within your web server and worker deployment YAML files, the entry should be similar to the following:

    volumeMounts:
    - name: <my-volume-name>
      mountPath: /certificates

When you start your Tonic Structural instance for the first time, you must provide your license key in order to activate Structural.

You also update the license key when it expires or when you upgrade to a new license tier.

Activating a new Structural instance

On a new instance of Structural, to provide the Structural license key and activate Structural:

1. On the Welcome to Tonic panel, click Get Started.

2. On the Input License Key panel, in the License Key text area, paste your license key.

3. Click Activate.

Structural verifies the license key. If the license key is valid, the Structural login screen is displayed.

Updating a license key

When your license key expires, Structural displays a banner across the top of the application. The banner contains a message to indicate that the license is expired.

The expired license banner contains an Update Tonic License button.

The Update Tonic License button also displays on the System Status tab of Structural Settings view. You can use this option to update an expired license key or to provide a new key when you upgrade to a higher license plan.

You obtain your updated license key from Tonic.ai. To update the license key:

1. Click Update Tonic License.

2. On the Update License Key panel, you can either:

   1. Paste the new license key into the license key text area.

   2. Upload a file that contains the license key. To search for and select the file, click Browse.

3. Click Update.

Structural verifies the license key. If the license is valid, the license key is updated, and the expired license banner is removed.

Using the TONIC_LICENSE environment setting (if no users have permission to update the license)

The Structural Settings option to update the license key is only available to users who have the Update the Tonic Structural license key global permission. All self-hosted instances should have at least one user with this permission.

If your instance does not have a user with the Update the Tonic Structural license key global permission, then you can set the Structural license key as the value of the TONIC_LICENSE environment setting.

If your instance has a user with the Update the Tonic Structural license key global permission, then Structural ignores the TONIC_LICENSE environment setting. You must use the Structural Settings option to update the license.

To enable Tonic Structural to write destination data to container artifacts, the Structural service account requires specific levels of access to Kubernetes.

The required access applies both on a Kubernetes cluster where Structural is deployed and, for Docker instances, on the .

Rolebinding with required access

On the Kubernetes cluster, the Structural service account must be granted a rolebinding that grants the following access to the Structural Kubernetes cluster:

On a Kubernetes instance of Structural, you can allow Structural to create the rolebinding automatically. In the Structural Helm chart, the following setting determines whether to have Structural automatically create and grant the rolebinding. By default, the setting is true.

If your access management method does not allow you to use this default configuration, then:

1. Change the setting to false.

2. Create and grant the rolebinding.

Additional rbac grant for a separate Kubernetes cluster

For a separate Kubernetes cluster, the environment setting CONTAINERIZATION_MANAGE_NAMESPACE indicates whether to allow Structural to manage the remote namespace.

If the setting is true, then you must add the following rbac grant to enable the Structural service account to manage namespaces.