Version: 3.0.0

Speech Platform Virtual Appliance

The Speech Platform Virtual Appliance is a distribution of the Phonexia Speech Platform in the form of a virtual image. Presently, it exclusively supports the OVF format.

Installation

This section describes how to install virtual appliance into your virtualization platform.

Prerequisites

Currently we support only virtualbox and VMWare.

It will probably work on other virtualization platforms but we haven't tested it yet.

Minimal HW requirements

50GB of disk space
4 CPU cores
16GB of memory

Minimal requirements mean that you are able to process single technology (speaker identification, enhanced speech-to-text built on Whisper or speech-to-text by Phonexia) for evaluation purposes. We recommend to disable all non-needed (not evaluated) technologies to save the resources.

Resource usage per technology

Speaker identification - 1 CPU core and 2GB memory
Speech-to-text by phonexia - 1 CPU core and 4GB memory per language
Enhanced Speech-to-text built on whisper - 8 CPU cores and 8GB memory or 1 CPU core and 8GB memory and GPU card

Note: Running enhanced speech-to-text built on Whisper on CPU is slow. We recommend to use at least 8 CPU cores to run our built-in examples in reasonable time.

GPU

GPU is not required to make virtual appliance work but you will suffer serious performance degradation for enhanced speech-to-text built on Whisper functionality.

If you decide to use GPU, then make sure that

Server HW (especially BIOS) has support for IOMMU.
Host OS can pass GPU device to virtualization platform (== Host OS can be configured to NOT use the GPU device)
Virtualization platform can pass GPU device to guest OS.

Installation guide

Download virtual appliance
Import virtual appliance to your virtualization platform (For Hyper-V deployment, please refer to section 'How to modify OVF to Hyper-V compatible VM')
Run virtual appliance

Post-installation steps

Virtual appliance is configured to obtain IP address from DHCP server. If you are not using DHCP server for IP allocation or prefer to set up static IP, then you have to reconfigure the OS.

SSH server

SSH server is deployed and enabled in virtual appliance. Use following credentials:

login: root
password: InVoiceWeTrust

We recommend to change the root password and disable password authentication via SSH for root user in favor of key-based authentication.

Open ports

List of open ports:

SSH (22) - for convenient access to OS
HTTP (80) - Speech platform is accessible via HTTP protocol
HTTPS (443) - Speech platform is also accessible via HTTPS protocol
HTTPS (6443) - Kubernetes API
HTTPS (10250) - Metrics server

K3s check

K3s (kubernetes distribution) is started automatically by systemd when virtual appliance is started. You can verify whether k3s is running or not with this command:

systemctl status k3s

Kubernetes check

When k3s service is started, it takes some time until application (== kubernetes pods) is started. Usually it takes around 2 minutes. To check if application is up and running, execute following command:

kubectl -n speech-platform get pods

When all pods are running, output looks like:

[root@speech-platform ~]# kubectl -n speech-platform get pods
NAME                                                                       READY   STATUS             RESTARTS      AGE
speech-platform-docs-57dcd49f9f-q97w4                                      1/1     Running            0             2m10s
speech-platform-envoy-759c9b49d9-99vp7                                     1/1     Running            0             2m10s
speech-platform-frontend-7f4566dbc6-jhprh                                  1/1     Running            0             2m10s
speech-platform-assets-5697b4c86-8sh9k                                     1/1     Running            0             2m9s
speech-platform-media-conversion-7d8f884f9-zh75g                           1/1     Running            0             2m9s
speech-platform-api-69bc7d4d5b-6kv7x                                       1/1     Running            0             2m9s
enhanced-speech-to-text-built-on-whisper-74548494c866mrz                   0/1     CrashLoopBackOff   4 (29s ago)   2m10s
speech-platform-voiceprint-extraction-68d646d449-9br8m                     0/1     CrashLoopBackOff   4 (33s ago)   2m10s
speech-platform-voiceprint-comparison-76948b4947-xjw92                     0/1     CrashLoopBackOff   4 (15s ago)   2m10s

Voiceprint-extraction, voiceprint-comparison and enhanced-speech-to-text-built-on-whisper microservices (pods) are failing initially. This is expected and it is caused by missing license. You can either add license to microservices or disable them if you don't plan to use them.

Optionally you can check if all other system and auxiliary applications are running:

kubectl get pods -A

All pods should be running or completed, like this:

[root@speech-platform ~]# kubectl get pods -A
NAMESPACE         NAME                                                                       READY   STATUS             RESTARTS        AGE
kube-system       local-path-provisioner-8d98546c4-9pq8p                                     1/1     Running            0               6m44s
kube-system       coredns-94bcd45cb-rp6zx                                                    1/1     Running            0               6m44s
kube-system       metrics-server-754ff994c9-pczpx                                            1/1     Running            0               6m44s
kube-system       svclb-ingress-nginx-controller-baed713a-nzwcc                              2/2     Running            0               5m24s
kube-system       helm-install-ingress-nginx-wpwk4                                           0/1     Completed          0               6m45s
kube-system       helm-install-filebrowser-fd569                                             0/1     Completed          0               6m45s
kube-system       helm-install-nginx-28rll                                                   0/1     Completed          0               6m45s
kube-system       helm-install-speech-platform-7k6qf                                         0/1     Completed          0               6m45s
ingress-nginx     ingress-nginx-controller-679f97c77d-rdssr                                  1/1     Running            0               5m24s
nginx             nginx-6ddd78f789-f9lq2                                                     1/1     Running            0               5m39s
filebrowser       filebrowser-7476f7c65c-rk9d5                                               1/1     Running            0               5m39s
gpu               nfd-58s4x                                                                  2/2     Running            0               5m44s
speech-platform   speech-platform-docs-57dcd49f9f-q97w4                                      1/1     Running            0               5m38s
speech-platform   speech-platform-envoy-759c9b49d9-99vp7                                     1/1     Running            0               5m38s
speech-platform   speech-platform-frontend-7f4566dbc6-jhprh                                  1/1     Running            0               5m38s
speech-platform   speech-platform-assets-5697b4c86-8sh9k                                     1/1     Running            0               5m37s
speech-platform   speech-platform-media-conversion-7d8f884f9-zh75g                           1/1     Running            0               5m37s
speech-platform   speech-platform-api-69bc7d4d5b-6kv7x                                       1/1     Running            0               5m37s
speech-platform   speech-platform-voiceprint-extraction-68d646d449-9br8m                     0/1     CrashLoopBackOff   5 (2m33s ago)   5m38s
speech-platform   enhanced-speech-to-text-built-on-whisper-74548494c866mrz                   0/1     CrashLoopBackOff   5 (2m32s ago)   5m38s
speech-platform   speech-platform-voiceprint-comparison-76948b4947-xjw92                     0/1     CrashLoopBackOff   5 (2m20s ago)   5m38s

Application check

Access virtual appliance welcome page on virtual appliance to see IP address or hostname from your local computer. If you are able to access the welcome page, applications should work.

Components

This is the list of components virtual appliance is composed of.

Operating system

There is Rocky Linux 9.3 under the hood.

GPU support

Virtual appliance has all necessary prerequisites pre-baked to allow run GPU-powered workloads (especially enhanced-speech-to-text-built-on-whisper). This means that NVIDIA drivers and container toolkit are already installed.

Kubernetes

There is k3s kubernetes distribution deployed inside.

Ingress controller

We use ingress-nginx ingress controller. This component is serving as reverse proxy and loadbalancer.

Speech platform

This is the application for solving various voice-related problems like speaker identification, speech-to-text transcription and many more. Speech platform is accessible via web browser or API.

File Browser

File Browser is web-based file browser/editor used to work with data on data disk.

Prometheus

Prometheus is a tool for providing monitoring information about kubernetes components.

Grafana

Grafana is a tool for visualization of prometheus metrics.

Disks

Virtual appliance comes with system disk and data disk.

System disk

Operating system is installed on system disk. You should not modify system disk unless you know what you are doing.

List of component stored on system disk:

NVIDIA drivers
Container images for microservices
Packaged helm charts

Data disk

Data disk is used as persistent storage. Unlike system disk, data disk is intended to contain files which can be viewed/modified by the user. Data disk is created with PHXDATADISK label and system is instructed to mount filesystem with this label to /data directory.

List of components stored on data disk:

Logs (/data/logs) of the system, k3s and individual containers
Configuration for ingress controller (/data/ingress-nginx/ingress-nginx-values.yaml)
Configuration for speech platform (/data/speech-platform/speech-platform-values.yaml)
Models for individual microservices (/data/models/)
Custom images (/data/images/)
Prometheus persistent storage (/data/storage/prometheus)

Configuration

Following section describes various configuration use cases.

Upload Speech to Text Phonexia and Time Analysis models

Upload provided licensed-models.zip archive to virtual appliance via filebrowser or via scp:
```
$ scp licensed-models.zip root@<virtual-appliance-ip>:/data/
```
Unzip archive. Model is extracted to directory per language:
```
$ unzip licensed-models.zip
```

Content of the /data/models should look like:

$ find /data/models
/data/models
/data/models/speech_to_text_phonexia
/data/models/speech_to_text_phonexia/en_us_6-3.61.0.model
/data/models/speech_to_text_phonexia/tr_tr_6-3.61.0.model
/data/models/time_analysis/
/data/models/time_analysis/generic-3.61.0.model

If model is already configured, nothing else needs to be done. In case it is not, you need to configure it as described in load Speech to Text Phonexia and Time Analysis model from data disk section.

Insert license keys

Virtual appliance is distributed without license. Speech platform does not work without valid license. If you haven't received any licence, please contact Phonexia support. License must be inserted into each microservice.

Insert license into enhanced-speech-to-text-built-on-whisper microservice

Get license for speech-to-text model. License looks like eyJ2...In0=.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser. You can access the file browser via URL http://<IP_address_of_virtual_appliance>/filebrowser/
Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.config.license.value
Change content of the value key from "<put your license for enhanced-speech-to-text-built-on-whisper model here>" to license key.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper-enhanced:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        license:
          value: "eyJ2...In0="

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Insert license into voiceprint-extraction microservice

Get license for speaker-identification model. License looks like eyJ2...In0=.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.voiceprint-extraction.config.license.value
Change content of the value key from "<put your license for speaker-identification model here>" to license key.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    voiceprint-extraction:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        license:
          value: "eyJ2...In0="

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Insert license into voiceprint-comparison microservice

Get license for speaker-identification model. License looks like eyJ2...In0=.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.voiceprint-comparison.config.license.value
Change content of the value key from "<put your license for speaker-identification model here>" to license key.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    voiceprint-comparison:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        license:
          value: "eyJ2...In0="

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Insert license into speech-to-text-phonexia microservice

Get license for speech-to-text-phonexia. License looks like:

SERVER license.phonexia.com/lic
USE_TIME

PRODUCT SPE_v3 ACB46...
2uJ...M9A==
PRODUCT STT-tech F23B6...
jXu...K7A=

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-to-text-phonexia.config.license.value
Change content of the value key from "<put your license for speech-to-text-phonexia here>" to the license key.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        license:
          value: |
            SERVER license.phonexia.com/lic
            USE_TIME

            PRODUCT SPE_v3 ACB46...
            2uJ...M9A==
            PRODUCT STT-tech F23B6...
            jXu...K7A=

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Insert license into time-analysis microservice

Get license for time-analysis. License looks like:

SERVER license.phonexia.com/lic
USE_TIME

PRODUCT SPE_v3 ACB46...
2uJ...M9A==
PRODUCT TAE-tech D3118...
Ba+4...eg==

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.time-analysis.config.license.value
Change content of the value key from "<put your license for time-analysis here>" to the license key.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    time-analysis:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        license:
          value: |
            SERVER license.phonexia.com/lic
            USE_TIME

            PRODUCT SPE_v3 ACB46...
            2uJ...M9A==
            PRODUCT TAE-tech D3118...
            Ba+4...eg==

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Set DNS name for speech platform virtual appliance

Speech platform is accessible on http://<IP_address_of_virtual_appliance>. We recommend to create DNS record to make access more comfortable for users. Consult your DNS provider to get more information how to add corresponding DNS record.

Use HTTPS certificate

Speech platform is also accessible via HTTPS protocol on https://<IP_address_of_virtual_appliance>. If you prefer secure communication you might need to use your own TLS certificate for securing the communication.

To do so, follow this guide:

Prepare the TLS certificate beforehand.
Put certificate private key in file named cert.key.
Put certificate into file named cert.crt.

Create kubernetes secret manifest storing the certificate and private key:

kubectl create -n ingress-nginx secret tls default-ssl-certificate --key cert.key --cert cert.crt -o yaml --dry-run > /tmp/certificate-secret.yaml

Copy manifest (resulting file) to /data/ingress-nginx/certificate-server.yaml.
Open text file /data/ingress-nginx/ingress-nginx-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.controller.extraArgs.default-ssl-certificate
Uncomment the line.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
    <Not significant lines omitted>
      extraArgs:
      <Not significant lines omitted>
        default-ssl-certificate: "ingress-nginx/default-ssl-certificate"

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Extend disks

Following section describes how to extend both system and data disk.

At first run lsblk to see all disk devices in the system:

$ lsblk

This is example output:

[root@speech-platform ~]# lsblk
NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1
│    259:0    0    40G  0 disk
├─nvme0n1p1
│    259:4    0     1G  0 part /boot
├─nvme0n1p2
│    259:5    0     4G  0 part /
├─nvme0n1p3
│    259:6    0  33.3G  0 part /var/lib/kubelet/pods/c699b49a-d5cb-4e63-8555-0a11f4204bb6/volume-subpaths/capabilities/frontend/0
│                              /var/lib/kubelet/pods/29a7020e-1b24-4720-a94d-b458e9297fbe/volume-subpaths/pvc-cf413864-ffa6-4e3b-bfb6
│                              /var/lib/kubelet/pods/1093b1d0-4dfb-48fa-910b-cc04a553b155/volume-subpaths/sc-dashboard-provider/grafana/4
│                              /var/lib/kubelet/pods/1093b1d0-4dfb-48fa-910b-cc04a553b155/volume-subpaths/config/grafana/2
│                              /var/lib/kubelet/pods/1093b1d0-4dfb-48fa-910b-cc04a553b155/volume-subpaths/config/grafana/0
│                              /var/lib/kubelet/pods/7e28800d-ad39-4d92-8d9c-cd24fce8f861/volume-subpaths/config/filebrowser/0
│                              /var
├─nvme0n1p4
│    259:7    0     1K  0 part
└─nvme0n1p5
     259:8    0  1023M  0 part [SWAP]
nvme1n1
│    259:1    0    10G  0 disk
└─nvme1n1p1
     259:3    0    10G  0 part /var/log
                               /var/lib/rancher/k3s/server/manifests/speech-platform/values
                               /var/lib/rancher/k3s/server/manifests/ingress-nginx/values
                               /data

System disk is the one with /, /boot and /var mountpoints. In this example it is nvme0n1. Data disk is the one with /data mountpoint. In this example it is nvme1n1.

Extend system disk

System disk is nvme0n1 and partition with /var mountpoint (which is nvme0n1p3) needs to be extended. At first extend the system disk in the virtualization platform. Then verify that you can see extended disk inside the virtual appliance.

lsblk --nodeps
NAME     MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1  259:0   0  50G  0 disk
nvme1n1  259:1   0  20G  0 disk

System disk was extended from 40GB to 50GB in the virtualization platform. Now you can extend the system disk in the virtual appliance itself.

Recreate partition 3 with all disk space:

echo ", +" | sfdisk --force -N 3 /dev/nvme0n1

Run partprobe to for system to use new partition:

partprobe

Run lsblk to check if system sees the resized partition:

lsblk /dev/nvme0n1

Extend xfs filesystem:

xfs_growfs /var

Check that disk was resized:

df -h /var

Extend data disk

Data disk is nvme1n1 and partition with /data mountpoint (which is /dev/nvme1n1p1) needs to be extended.

At first extend the data disk in the virtualization platform. Then verify that you can see extended disk inside the virtual appliance.

lsblk --nodeps
NAME     MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1  259:0   0  50G  0 disk
nvme1n1  259:1   0  20G  0 disk

Data disk was extended from 10GB to 20GB in the virtualization platform. Now you can extend the data disk in the virtual appliance itself.

Recreate partition 1 with all disk space:

echo ", +" | sfdisk --force -N 1 /dev/nvme1n1

Run partprobe to for system to use new partition:

partprobe

Run lsblk to check if system sees the resized partition:

lsblk /dev/nvme1n1

Extend xfs filesystem:

xfs_growfs /data

Check that disk was resized:

df -h /data

Disable unneeded microservices

Virtual appliance comes with all microservices enabled by default. You may decide to disable microservice if you do not plan to use it. Disabled microservice does not consume any compute resources.

Find out which microservices you want to disable - voiceprint-extraction, voiceprint-comparison, enhanced-speech-to-text-built-on-whisper or speech-to-text-phonexia.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.<microservice>.enabled
Change the value from true to false.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    <microservice>:
    <Not significant lines omitted>
      enabled: false

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Phonexia speech to text microservice

This section describes configuration specific to phonexia speech to text microservice.

Permanent vs onDemand instances

Permanent instance is started and running (and consuming resources) all the time. OnDemand instance is started only when corresponding task is queued. Instance is stopped when all tasks were processed.

All instances are onDemand by default. Any instance can be reconfigured to be permanent. Use following guide to reconfigure instance from onDemand to permanent one:

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-to-text-phonexia.config.instances.

Corresponding section looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
            .
            .
            .
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            onDemand:
              enabled: true
            .
            .
            .

Delete onDemand key and its subkeys.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
            .
            .
            .
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            .
            .
            .

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Configure languages in speech-to-text-phonexia microservice

This microservice consists of multiple instances. Each instance corresponds to a single language. All instances are listed in the configuration file.

Note: Docker images for any language are not included in the virtual appliance. This means that virtual appliance needs to access the internet to download the docker image when speech-to-text-phonexia microservice is used! As a workaround you can put custom image into virtual appliance.

By default all languages/instances are enabled. List of languages:

ar_kw_6
ar_xl_6
bn_6
cs_cz_6
de_de_6
en_us_6
es_6
fa_6
fr_fr_6
hr_hr_6
hu_hu_6
it_it_6
ka_ge_6
kk_kz_6
nl_6
pl_pl_6
ps_6
ru_ru_6
sk_sk_6
sr_rs_6
sv_se_6
tr_tr_6
uk_ua_6
vi_vn_6
zh_cn_6

How to disable all language instances except of cs_cz_6 and en_us_6:

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-to-text-phonexia.config.instances.

Corresponding section looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: ark
            imageTag: 3.59.0-stt-ar_kw_6
            onDemand:
              enabled: true
          - name: arx
            imageTag: 3.59.0-stt-ar_xl_6
            onDemand:
              enabled: true
          - name: bn
            imageTag: 3.59.0-stt-bn_6
            onDemand:
              enabled: true
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            onDemand:
              enabled: true
          - name: de
            imageTag: 3.59.0-stt-de_de_6
            onDemand:
              enabled: true
          - name: en
            imageTag: 3.59.0-stt-en_us_6
            onDemand:
              enabled: true
            .
            .
            .
          - name: vi
            imageTag: 3.59.0-stt-vi_vn_6
            onDemand:
              enabled: true
          - name: zh
            imageTag: 3.59.0-stt-zh_cn_6
            onDemand:
              enabled: true

Comment out all the instances except (cs_cz_6 and en_us_6).

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          #- name: ark
          #  imageTag: 3.59.0-stt-ar_kw_6
          #  onDemand:
          #    enabled: true
          #- name: arx
          #  imageTag: 3.59.0-stt-ar_xl_6
          #  onDemand:
          #    enabled: true
          #- name: bn
          #  imageTag: 3.59.0-stt-bn_6
          #  onDemand:
          #    enabled: true
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            onDemand:
              enabled: true
          #- name: de
          #  imageTag: 3.59.0-stt-de_de_6
          #  onDemand:
          #    enabled: true
          - name: en
            imageTag: 3.59.0-stt-en_us_6
            onDemand:
              enabled: true
            .
            .
            .
          #- name: vi
          #  imageTag: 3.59.0-stt-vi_vn_6
          #  onDemand:
          #    enabled: true
          #- name: zh
          #  imageTag: 3.59.0-stt-zh_cn_6
          #  onDemand:
          #    enabled: true

Or you can even delete the instances you are not interested in.

Then updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            onDemand:
              enabled: true
          - name: en
            imageTag: 3.59.0-stt-en_us_6
            onDemand:
              enabled: true

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Modify replicas for permanent language instances

Each language instance has only one replica by default. This means that only one request/audiofile can be processed at once. To process multiple requests/audiofiles in parallel you have to increase replicas for corresponding language instance.

Note: We do not recommend increasing replicas for any microservice when virtual appliance is running with default resources (4CPU, 16GB memory)! Note: OnDemand instance has always only one replica.

Find out which language instance you want to configure replicas for.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-to-text-phonexia.config.instances.<language instance>.replicaCount.
Change the value to desired amount of replicas.
Updated file should look like:

Corresponding section looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6
            replicaCount: 2

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Modify parallelism for instances

Each instance is able to process only one request at the time, unless the parallelism is overridden. Value of parallelism means the maximum number of requests processed by one instance. Parallelism is set globally for all instances of technology, however each instance can override the value. To override parallelism for speech-to-text-phonexia or time-analysis these steps needs to be followed:

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Find the key, depending on technology (speech-to-text-phonexia, time-analysis) for which parallelism should be overridden: .spec.valuesContent.<technology>.parallelism
Change the value to desired number of requests processed in parallel

Corresponding section looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      # Global value of parallelism for all instances
      parallelism: 2
      config:
      <Not significant lines omitted>
        instances:
          - name: cs
            imageTag: 3.61.0
          - name: en
            imageTag: 3.61.0
            # Override of parallelism for en instance
            parallelism: 4

Save the file

Add custom images

This section describes how to add custom image into virtual appliance. Typical use case is to add speech to text images to Speech Engine for languages you want to use. These images needs to be added to the data disk in order to make phonexia speech to text work offline. In the example I will add two images - phonexia speech to text english and czech languages.

[Virtual appliance] Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
[Virtual appliance] Locate key .spec.valuesContent.speech-to-text-phonexia.config.instances.

[Virtual appliance] Choose which images you want to add. Use imageTag key to find out which image tag(s) to use:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: cs
            imageTag: 3.59.0-stt-cs_cz_6 <- This is the image tag
            onDemand:
              enabled: true
          - name: en
            imageTag: 3.59.0-stt-en_us_6 <- This is the image tag
            onDemand:
              enabled: true

[PC] Pull all images:

docker pull phonexia/spe:3.60.1-stt-en_us_6
docker pull phonexia/spe:3.60.1-stt-cs_cz_6

[PC] Save all images to single tar archive:

docker save -o images.tar phonexia/spe:3.60.1-stt-cs_cz_6 phonexia/spe:3.60.1-stt-en_us_6

[PC] Copy images.tar file into virtual appliance via ssh or filebrowser to /data/images.
```
scp images.tar root@<IP of virtual appliance>:/data/images
```
[Virtual appliance] Restart virtual appliance to load the images or load them manually with:
```
ctr image import /data/images/images.tar
```

Modify microservice replicas

Each microservice has only one replica by default. This means that only one request/audiofile can be processed at once. To process multiple requests/audiofiles in parallel, you have to increase replicas for corresponding microservices.

Note: We do not recommend increasing replicas for any microservice when virtual appliance is running with default resources (4CPU, 16GB memory)!

Find out which microservices you want to modify replicas - voiceprint-extraction, voiceprint-comparison and enhanced-speech-to-text-built-on-whisper-enhanced.
Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.<microservice>.replicaCount
Change the value to desired amount of replicas.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    <microservice>:
    <Not significant lines omitted>
      replicaCount: 2

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Run enhanced-speech-to-text-built-on-whisper microservice on GPU

At first make sure virtual appliance can see the GPU device(s). Use nvidia-smi to list all the devices. If device is present and visible to the system, then output should look like:

[root@speech-platform ~]# nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 980 (UUID: GPU-1fb957fa-a6fc-55db-e76c-394e9a67b7f5)

If the GPU is visible, then you can reconfigure the enhanced-speech-to-text-built-on-whisper to use GPU for the processing.

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate enhanced-speech-to-text-built-on-whisper section .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.
Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.config.device.

Uncomment the line so that it looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
       # Uncomment this to force whisper to run on GPU
       device: cuda

Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.resources.

Request GPU resources for the processing so that it looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      # Uncomment this to grant access to GPU on whisper pod
      resources:
        limits:
          nvidia.com/gpu: "1"

Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.runtimeClassName.

Set runtimeClassName so that it looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      # Uncomment this to run whisper on GPU
      runtimeClassName: "nvidia"

Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.updateStrategy.

Set type to Recreate to allow seamless updates so that it looks like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      # Uncomment this to allow seamless updates on single GPU machine
      updateStrategy:
        type: Recreate

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        device: cuda

      <Not significant lines omitted>
      resources:
        limits:
          nvidia.com/gpu: "1"

      <Not significant lines omitted>
      runtimeClassName: "nvidia"

      <Not significant lines omitted>
      updateStrategy:
        type: Recreate

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Change model used in a microservice

Each microservice needs a model to do its job properly. We provide more models for some microservices, for example enhanced-speech-to-text-built-on-whisper. Usually we pre-configure microservices with the most accurate (and slowest model). Typically users use different model to speed up processing in favor of less accurate results.

License you have received with the virtual appliance is valid only for default model. If you change the model, you have to change the license as well.

Change model in enhanced-speech-to-text-built-on-whisper microservice

We offer following models for enhanced-speech-to-text-built-on-whisper microservice:

large-v3 - next-gen most accurate multilingual model.
large-v2 - most accurate multilingual model. This is the default model.
medium - less accurate but faster than large-v2.
base - less accurate but faster than medium.
small - less accurate but faster than base.

Ask Phonexia to provide you desired model and license. You will receive link(s) which results into zip archive (zip file) when downloaded.

Upload archive to virtual appliance.

$ scp licensed-models.zip root@<virtual-appliance-ip>:/data/

Unzip archive. Models are extracted to directory per microservice:
```
$ unzip licensed-models.zip
```

Content of the /data/models should look like:

$ find /data/models
/data/models/
/data/models/enhanced_speech_to_text_built_on_whisper
/data/models/enhanced_speech_to_text_built_on_whisper/small-1.0.0.model
/data/models/enhanced_speech_to_text_built_on_whisper/enhanced_speech_to_text_built_on_whisper-base-1.0.0-license.key.txt
/data/models/enhanced_speech_to_text_built_on_whisper/base-1.0.0.model
/data/models/enhanced_speech_to_text_built_on_whisper/enhanced_speech_to_text_built_on_whisper-small-1.0.0-license.key.txt
/data/models/speaker_identification
/data/models/speaker_identification/xl-5.0.0.model
/data/models/speaker_identification/speaker_identification-xl-5.0.0-license.key.txt

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.enhanced-speech-to-text-built-on-whisper.config.model
Change content of the file key from "large_v2-1.0.0.model" to file you've just uploaded ("small-1.0.0.model").

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      config:
        model:
          <Not significant lines omitted>
          file: "small-1.0.0.model"

Change the license because you have changed the model. See above how to do it.
Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Load Speech to Text Phonexia and Time Analysis model from data disk

To keep up with the latest version of application, load models from virtual appliance volume is possible. For using the image without the model and load existing models from data volume, instance in config file need to be setup as follows:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          . . .
          - name: en
            imageTag: 3.61.0
          . . .
  <Not significant lines omitted>
    time-analysis:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          . . .
          - name: tae
            imageTag: 3.61.0
          . . .

As a default we count with that the model will be located on path /data/models/speech_to_text_phonexia/en_us_6-3.61.0.model. This folder structure is ensured by unziping provided licensed-models.zip archive in /models/ path. Additionally if the path to the model is different, or the version of model is not matching with the image, it can be specified in instances config as a:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          . . .
          - name: cs
            imageTag: 3.61.0
            model:
              hostPath: /data/models/speech_to_text_phonexia/en_us_6-3.61.0.model
          . . .
  <Not significant lines omitted>
    time-analysis:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          . . .
          - name: tae
            imageTag: 3.61.0
            model:
              hostPath: /data/models/time_analysis/generic-3.61.0.model
          . . .

So far model loading from data disk is supported only by the Speech to Text Phonexia and Time Analysis technologies.

Process patented audio codecs with media-conversion

By default media conversion can work only with patent-free audio codecs.

We cannot include and distribute patented codecs with virtual appliance. If you need to process audiofiles encoded with patented codecs, you have to use different version of media-conversion. Media-conversion service image is located on dockerhub.

Pull Media Conversion image on the fly

This is handy if you don't mind pulling images from the internet. Image is pulled only if it is not present yet.

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.media-conversion.image

Change content of the repository, registry, tag and tagSuffix to

media-conversion:
  image:
    registry: docker.io
    repository: phonexia/media-conversion
    tag: 1.0.0
    tagSuffix: ""

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    media-conversion:
    <Not significant lines omitted>
      image:
        registry: docker.io
        repository: phonexia/media-conversion
        tag: 1.0.0
        tagSuffix: ""

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Put Media Conversion image into virtual appliance

This approach is needed if your deployment is completely offline and access to internet from virtual appliance is forbidden.

[PC] Pull media-conversion image locally:

$ docker pull phonexia/media-conversion:1.0.0

[PC] Save Media Conversion image to tar archive:

$ docker save --output images.tar phonexia/media-conversion:1.0.0

[PC] Copy images.tar file into virtual appliance via ssh or filebrowser to /data/images.
```
scp images.tar root@<IP of virtual appliance>:/data/images
```
[Virtual appliance] Restart virtual appliance to load the images or load them manually with:
```
ctr image import /data/images/images.tar
```
Reconfigure the speech-platform to use locally downloaded image as mentioned above.

Limits

This section describe what are virtual appliance limits and how to modify them.

API limits

Following limits are applied for the API itself.

Name	Unit	Default	Description
`taskExpirationTime`	seconds	`300`	Time when finished tasks are expired. API holds the information about finished tasks (both successfully finished and failed). These information are discarded after `taskExpirationTime`. Client usually polls on the task id. Client must retrieve the task status before it is expired. Maximum value is `3600`.
`taskGrpcTimeout`	seconds	`120`	Maximum time API waits for any task to complete. If you process big audio files, you probably need to increase this limit.
`inputStorageSize`	variable	`1GiB`	Size of the input storage. When audio file is POSTed to the API, whole file must be stored on the disk. If you process big files or multiple files in parallel, then this limit must be probably increased.
`internalStorageSize`	variable	`1GiB`	Size of the internal storage. Each audiofile is converted into wav format before processing. Converted audio is stored on the disk. If you process big files or multiple files in parallel, then this limit must be probably increased. Also note the `internalStorageSize` must be greater or equal to the `inputStorageSize`.
`singleFileUploadTimeout`	second	`120`	Maximum allowed time for uploading single file to the API. If you process big files or having a poor network connection, then this limit must be increased.
`singleFileUploadSize`	bytes	`104857600` (== 100MB)	Maximum allowed size of an audio file to upload. If you process big files then this limit must be increased. Note that this API/ingress limit not the UI limit!

How to change the API limits

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.api.config
Change the value of the corresponding limit to a new value.
```
api:
  config:
    taskExpirationTime: 1200
```

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    api:
    <Not significant lines omitted>
      config:
        taskExpirationTime: 1200

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

UI limits

Following limits are applied for the UI itself.

Name	Unit	Default	Description
`taskParallelism`		`4`	UI post task to the API and polls for the task until it is finished. This controls how many tasks can be processed in parallel.
`taskPollingInterval`	seconds	`1`	Duration between poll attempts.
`taskPollingTimeout`	seconds	`3600`	How long the UI polls for the task. How long is the UI willing to wait until the task is finished.

Speaker Identification UI limits

Limits in config section .spec.valuesContent.frontend.config.limits.speakerIdentification are applicable only for speaker identification.

Name	Unit	Default	Description
`maxFileSize`	bytes	`5000000` (== 5MB)	Maximum allowed size of an audio file to upload. Note that this must be lower than or equal to the `singleFileUploadSize` API limit.
`maxFilesCount`		`100`	Maximum number of files to be uploaded.
`maxVoiceRecorderDuration`	seconds	`300`	Maximum duration of the record captured by voice recorder.

Speech to text UI limits

Limits in config section .spec.valuesContent.frontend.config.limits.speechToText are applicable only for Speech to Text. Limits are applicable for both Enhanced Speech to Text built on Whisper and Speech to Text by Phonexia 6th Gen.

Name	Unit	Default	Description
`maxFileSize`	bytes	`5000000` (== 5MB)	Maximum allowed size of an audio file to upload. Note that this must be lower than or equal to the `singleFileUploadSize` API limit.
`maxFilesCount`		`100`	Maximum number of files to be uploaded.
`maxVoiceRecorderDuration`	seconds	`300`	Maximum duration of the record captured by voice recorder.

How to change the UI limits

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.frontend.config.limits

Change the value of the corresponding limit to a new value.

frontend:
  config:
    limits:
      taskParallelism: 2

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    frontend:
    <Not significant lines omitted>
      config:
        limits:
          taskParallelism: 2

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Pod count limits

Currently, the platform is limited by the number of pods that can be created inside the Kubernetes cluster. The maximum number of pods is set to 300.

How to change the pod count limits

Pod count limits can be overridden by editing the /etc/k3s/rancher/config.yaml file. To override the maximal number of pods max-pods parameter needs to be added/edited. Example:

debug: true
system-default-registry: airgapped.phonexia.com
disable:
  - traefik
  - cloud-controller
kubelet-arg:
  - "kube-reserved=cpu=500m,memory=1Gi,ephemeral-storage=2Gi"
  - "system-reserved=cpu=500m, memory=1Gi,ephemeral-storage=2Gi"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10%"
  - "max-pods=350"

After editing configuration, virtual machine needs to be restarted (stop and start) to apply changes.

Admin backends limits

Following limits are applied to admin backends (filebrowser, grafana, prometheus).

Name	Unit	Default	Description
`singleFileUploadTimeout`	seconds	`120`	Maximum allowed time for uploading.
`singleFileUploadSize`	bytes	`5368709120` (== 5GB)	Maximum allowed size of an audio file to upload.

How to change admin backends limits

Open text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.ingressAdmin
Change the value of the corresponding limit to a new value.
```
ingressAdmin:
  singleFileUploadTimeout: 300
```

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    ingressAdmin:
    <Not significant lines omitted>
      singleFileUploadTimeout: 300

Save the file
Application automatically recognizes that file was updated and redeploys itself with updated configuration.

Admin console

Admin console is a simple web page containing links to various admin-related tools. Console is located at http://<IP_of_virtual_appliance>/admin. It contains links to

filebrowser
prometheus
grafana

Grafana

Grafana is tool for visualizing application and kubernetes metrics. List of most useful dashboards available in the grafana:

Envoy Clusters - See envoy cluster statistics
Kubernetes / Compute Resources / Pod - See resource consumption of individual pods
NGINX Ingress controller - See ingress controller stats
NVIDIA DCGM Exporter Dashboard - See GPU device stats
Node Exporter / Nodes - See stats about virtual appliance
Speech Platform API capacity - See metrics about speech platform itself

Troubleshooting

This section contains information about individual components of the speech platform and request flow

Speech platform components

List of the components:

frontend - simple webserver serving static html, css, javascript and image files
docs - simple webserver serving documentation
assets - simple webserver hosting examples
api - python component providing REST API interface
envoy - router and loadbalancer for GRPC messages
media-conversion - python component used for ** converting audio files from various formats to simple wav format ** splitting multi-channel audio into multiple single-channel files
technology microservices ** enhanced-speech-to-text-built-on-whisper - transcribes speech to text ** speech-to-text-phonexia - transcribes speech to text ** voiceprint-extraction - extracts voiceprint from audio file ** voiceprint-comparison - compares multiple voiceprints

Request flow

User POST request (for example transcribe speech to text) to API.
API creates task for processing and output task id to the user.
From this point user can poll on the task to get the result.
API calls media-conversion via envoy.
Media conversion converts the audiofile to wav format and possibly splits it into multiple mono-channel files.
API gets converted audiofile from media-conversion.
API calls enhanced-speech-to-text-built-on-whisper via envoy.
Enhanced-speech-to-text-built-on-whisper transcribes the audiofile.
API gets the transcription.
User can retrieve the task result.

Check node status

Check node status with:

[root@speech-platform ~]# kubectl get nodes
NAME                          STATUS   ROLES                  AGE   VERSION
speech-platform.localdomain   Ready    control-plane,master   9s    v1.27.6+k3s1

If node is not in ready state, there is usually something wrong.

Note: Node list can be empty (No resources found) or node can be in notReady state if virtual appliance is starting up. This is normal and shoud be fixed in a few moments.

Also node has to have enough free disk and memory capacity. When this is not true, pressure events are emitted. Run following command to see the node conditions:

[root@speech-platform disks]# kubectl describe node | grep -A 6 Conditions:
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 08:06:45 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletReady                 kubelet is posting ready status

Disk pressure

Disk pressure node event is emitted, when kuberentes is running out of disk capacity in the /var filesystem. Node conditions looks like this:

[root@speech-platform disks]# kubectl describe node | grep -A 6 Conditions:
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 08:06:45 +0000   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 29 Apr 2024 08:13:54 +0000   Mon, 29 Apr 2024 07:46:39 +0000   KubeletReady                 kubelet is posting ready status

Follow the procedure for extending the disks.

Memory pressure

Memory pressure node event is emitted, when kuberentes is running out of free memory. Node conditions looks like this:

[root@speech-platform disks]# kubectl describe node | grep -A 6 Conditions:
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                         Message
  ----             ------  -----------------                 ------------------                ------                         -------
  MemoryPressure   True    Mon, 29 Apr 2024 08:50:50 +0000   Mon, 29 Apr 2024 08:50:50 +0000   KubeletHasInsufficientMemory   kubelet has insufficient memory available
  DiskPressure     False   Mon, 29 Apr 2024 08:50:50 +0000   Mon, 29 Apr 2024 08:33:08 +0000   KubeletHasNoDiskPressure       kubelet has no disk pressure
  PIDPressure      False   Mon, 29 Apr 2024 08:50:50 +0000   Mon, 29 Apr 2024 08:33:08 +0000   KubeletHasSufficientPID        kubelet has sufficient PID available
  Ready            True    Mon, 29 Apr 2024 08:50:50 +0000   Mon, 29 Apr 2024 08:33:08 +0000   KubeletReady                   kubelet is posting ready status

You need to grant more memory to the virtual appliance or disable unneeded microservices.

View pod logs

Logs are stored in /data/log/pods/ or in /data/logs/containers. You can view them via filebrowser if needed.

Alternatively you can display logs with kubectl command:

[root@speech-platform ~]# kubectl -n speech-platform logs -f speech-platform-voiceprint-extraction-7867578b97-w7bzd
[2024-04-29 08:59:10.250] [Configuration] [info] model: /models/xl-5.0.0.model
[2024-04-29 08:59:10.250] [Configuration] [info] port: 8080
[2024-04-29 08:59:10.250] [Configuration] [info] device: cpu
[2024-04-29 08:59:10.250] [critical] base64_decode: invalid character ''<''

View status of a helm chart

Changing the file /data/speech-platform/speech-platform-values.yaml triggers update (== redeployment) of the speech platform helm chart.

View status of all helm-install pods:

[root@speech-platform disks]# kubectl get pods -n kube-system | grep -i helm-install
helm-install-filebrowser-2b7pn                  0/1     Completed   0             51m
helm-install-ingress-nginx-m87d4                0/1     Completed   0             51m
helm-install-nginx-nrcvk                        0/1     Completed   0             51m
helm-install-dcgm-exporter-fjqzz                0/1     Completed   0             51m
helm-install-kube-prometheus-stack-jn5bz        0/1     Completed   0             51m
helm-install-keda-vsn95                         0/1     Completed   0             51m
helm-install-speech-platform-9l9vj              0/1     Error       4 (46s ago)   6m15s

View logs of failed helm-install pod:

[root@speech-platform disks]# kubectl logs -f helm-install-speech-platform-9l9vj -n kube-system
...
...
...
Upgrading speech-platform
+ helm_v3 upgrade --namespace speech-platform speech-platform https://10.43.0.1:443/static/phonexia-charts/speech-platform-0.0.0-3356586-helm.tgz --values /config/values-10_HelmChartConfig.yaml
Error: UPGRADE FAILED: template: speech-platform/templates/envoy/deployment.yaml:20:28: executing "speech-platform/templates/envoy/deployment.yaml" at <include (print $.Template.BasePath "/envoy/configmap.yaml") .>: error calling include: template: speech-platform/templates/envoy/configmap.yaml:122:24: executing "speech-platform/templates/envoy/configmap.yaml" at <include (printf "%s.fullname" $ms) (index $.Subcharts $ms)>: error calling include: template: no template "voiceprint-extraction.fullname" associated with template "gotpl"

This might help you to pinpoint the problem. Usually is something wrong in the /data/speech-platform/speech-platform-values.yaml.

Upgrade guide

This section describes manual steps which need to be done prior upgrading. There are various changes in the configuration which must be reflected before upgrade. We suggest to always use configuration file bundled with new version of the virtual appliance and update it to suit your needs (insert licenses, enable/disable service, set replicas, ...). If you are not willing to do this, then you must modify your current configuration file to work with new version of the virtual appliance.

This section describes how to perform upgrade of the virtual appliance.

Upgrade and retain data disk

This upgrade approach retains all the data and configuration stored on the data disk.

Pros:

No need to configure virtual appliance from scratch
Prometheus metrics are kept

Cons:

You have to do version-specific upgrade steps

Import new version of virtual appliance (version X+1) into your virtualization platform
Stop current version of virtual appliance (version X)
Detach data disk from current version of virtual appliance (version X)
Attach data disk to new version of virtual appliance (version X+1)
Start new version of virtual appliance (version X+1)
Delete old version of virtual appliance (version X)
Follow version-specific upgrade steps

Upgrade and discard data disk

This upgrade approach discard current data disk and uses new one.

Pros:

Easier upgrade procedure
No version-specific upgrade steps
No accumulated disarray on the data disk

Cons:

You have to configure virtual appliance from the scratch:
- Disable unneeded services
- Insert license keys
- Insert models

Import new version of virtual appliance (version X+1) into your virtualization platform
Stop current version of virtual appliance (version X)
Start new version of virtual appliance (version X+1)
Delete old version of virtual appliance (version X)
Configure virtual appliance from scratch

Upgrade to 3.0.0

This section describes the manual steps which need to be done prior to upgrading to 3.0.0.

Rename Whisper microservice

Due to licensing reasons we had to rename the speech-to-text-whisper-enhanced microservice. The new name is enhanced-speech-to-text-built-on-whisper. This change must be reflected in the values file.

Step by step upgrade guide to 3.0.0

This section describes how to upgrade virtual appliance from 2.1.0 to 3.0.0 with retaining data disk content.

Rename whisper microservice in currently running version of virtual appliance.
Open the text file /data/speech-platform/speech-platform-values.yaml either directly from inside the virtual appliance or via a file browser.
Locate .spec.valuesContent.speech-to-text-whisper-enhanced.
Replace all occurences of speech-to-text-whisper-enhanced with enhanced-speech-to-text-built-on-whisper.
Replace all occurences of speech_to_text_whisper_enhanced with enhanced_speech_to_text_built_on_whisper.
The updated file should look like this:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    enhanced-speech-to-text-built-on-whisper:
    <Not significant lines omitted>
      image:
        repository: phonexia/dev/technologies/microservices/enhanced-speech-to-text-built-on-whisper/main
      <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        model:
          volume:
            hostPath:
              path: /data/models/enhanced_speech_to_text_built_on_whisper

Save the file
Rename the directory with Whisper models with the following command:

mv /data/models/speech_to_text_whisper_enhanced /data/models/enhanced_speech_to_text_built_on_whisper

Import new version of virtual appliance (version X+1) into your virtualization platform
Stop current version of virtual appliance (version X)
Detach data disk from current version of virtual appliance (version X)
Attach data disk to new version of virtual appliance (version X+1)
Start new version of virtual appliance (version X+1)
Delete old version of virtual appliance (version X)

Upgrade to 2.1.0

This section describes manual steps which need to be done prior upgrading to 2.1.0.

Load Speech to Text Phonexia and Time Analysis model from data disk instead of image

In new version, default way how to load models for Speech to Text Phonexia and Time Analysis will change. Before, models were loaded from image which lead to lot of duplicity in images. From now on, we will consider loading models from data disk as a default. However, the old way of loading models from image will still work.

Upgrade to load models from data disk (/data/models) require to update speech platform values file:

Open the text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser
Locate .spec.valuesContent.speech-to-text-phonexia.config.instances or .spec.valuesContent.time-analysis.config.instances key
Define versions of images (imageTag) without model (e.g. 3.61.0)
Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: ar-kw
            imageTag: 3.61.0
            onDemand:
              enabled: true
          - name: ar-kx
            imageTag: 3.61.0
            onDemand:
              enabled: true
    time-analysis:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: tae
            imageTag: 3.61.0
            onDemand:
              enabled: true

Locate .spec.valuesContent.speech-to-text-phonexia.image or .spec.valuesContent.time-analysis.image key to uncomment the image section.
Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      image:
        registry: airgapped.phonexia.com
      <Not significant lines omitted>

    time-analysis:
    <Not significant lines omitted>
      image:
        registry: airgapped.phonexia.com
      <Not significant lines omitted>

Save the file

Add ingressAdmin section

Open the text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate the key .spec.valuesContent.ingress.extraBackends
Remove the extraBackends scope with all of its contents
Add new ingressAdmin scope on the same indentation as the ingress scope. The resulting file should look like this:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
    ingress:
      <Not significant lines omitted>

    ingressAdmin:
      enabled: true
      annotations: {}
      singleFileUploadSize: "5368709120"
      singleFileUploadTimeout: 120
    <Not significant lines omitted>

Save the file
Proceed with upgrade

Fix permission for prometheus storage

This is post-upgrade task. Must be run when virtual appliance is upgraded to 2.1.0.

Run following command in the virtual appliance to fix permissions of the prometheus storage:

$ chmod -R a+w /data/storage/prometheus/prometheus-db/

Upgrade to 2.0.0

This section describes manual steps which need to be done prior upgrading to 2.0.0.

Rename speech-engine subchart to speech-to-text-phonexia

Due to renaming speech-engine subchart you have to update speech platform values file before upgrading:

Open the new text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-engine.
Rename speech-engine to speech-to-text-phonexia.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>

Save the file

Rename speech-to-text-phonexia instances

Open the new text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.speech-to-text-phonexia.config.instances.
Remove stt- prefix from the name of each instance.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    speech-to-text-phonexia:
    <Not significant lines omitted>
      config:
      <Not significant lines omitted>
        instances:
          - name: ar-kw
            imageTag: 3.60.1-stt-ar_kw_6
            onDemand:
              enabled: true
          - name: ar-kx
            imageTag: 3.60.1-stt-ar_xl_6
            onDemand:
              enabled: true

Save the file

Add proper tag suffix for Media Conversion

Open the new text file /data/speech-platform/speech-platform-values.yaml either directly from inside virtual appliance or via file browser.
Locate key .spec.valuesContent.media-conversion.image.
Change the value of the tagSuffix key to -free.

Updated file should look like:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: speech-platform
  namespace: kube-system
spec:
  valuesContent: |-
  <Not significant lines omitted>
    media-conversion:
      <Not significant lines omitted>
      image:
        <Not significant lines omitted>
        tagSuffix: "-free"
    <Not significant lines omitted>

Save the file
Proceed with upgrade

Update path to models

Default model location was changed from /data/models to /data/models/<microservice>. If you plan to upgrade and keep current data disk, no steps are needed. Model are loaded from old location which is /data/models. If you plan to upgrade from scratch (discarding the current data disk), no steps are needed as well - models are loaded from new location which is /data/models/<microservice>.

How to modify OVF to Hyper-V compatible VM

Both of existing virtual HDDs (.vmdk) need to be converted to Hyper-V compatible HDDs (.vhdx). Do it through this program: Starwind V2V Converter.
Create new VM in Hyper-V.
IMPORTANT: Use Generation 1 VM - Generation 2 doesn't work.
Enable networking/make sure it is enabled.
OPTIONAL: Disable options like DVD drive or SCSI controller since they are not needed.
Set Memory to at least 16GB and CPUs to at least 8 cores.
Attach HDDs, preferably onto one IDE controller.
Start the VM.
After it starts, check IP address either printed out on a login screen. Wait for the entire engine to start.
Go to the IP from the previous step and verify that the entire VM works as it should.

Load balancing

The performance of a single instance of the virtual appliance is of course limited by the HW resources and by the number of concurrent tasks the API component can handle. To work around these limitations, we can advise you to deploy multiple instances of the virtual appliance and put a load balancer before them.

How the load balancer works

The load balancer (LB) must ensure that the requests for the same task are routed to the same instance of the virtual appliance. This is called a stateful session. It can be achieved with a session cookie or with a session header.

The request flow is then following:

The client POSTs a task to the LB.
The LB picks a virtual appliance instance (depending on an LB algorithm) and sends the request there.
The API in the virtual appliance accepts the task and sends a response back to the LB.
The LB adds a session cookie or a session header to the response and sends it back to the client.
The client extracts the task id and the session cookie or session header from the response.
The client polls for the task. It sends a GET request with the session cookie or session header to the LB.
The LB routes the request to the proper instance of the virtual appliance based on the session cookie or session header.

In the following example, have used Envoy as the load balancer. Any other load balancer can be used if it supports stateful sessions.

Envoy configuration

This is the example Envoy configuration:

static_resources:
  listeners:
    - address:
        socket_address:
          # Load balancer address and port
          # This is where Envoy accepts the incoming traffic
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                      log_format:
                        text_format_source:
                          inline_string: >
                            [%START_TIME%] "%REQ(:METHOD)%
                            %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
                            %RESPONSE_CODE% %RESPONSE_FLAGS%
                            %RESPONSE_CODE_DETAILS%
                            %UPSTREAM_REQUEST_ATTEMPT_COUNT% %BYTES_RECEIVED%
                            %BYTES_SENT% %DURATION%
                            %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%
                            "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
                            "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%"
                            "%UPSTREAM_HOST%" "%REQ(REQUEST-ID)%"
                            "%REQ(CORRELATION-ID)%" "%REQ(session-header)%"
                            "%RESP(session-header)%"
                codec_type: AUTO
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/api/"
                          route:
                            cluster: speech-platform-virtual-appliance
                            retry_policy:
                              retry_on: "retriable-status-codes"
                              # Retry request on a different upstream when the 429 response is received
                              # This should happen when POSTing a request/task but max concurrent tasks limit is reached
                              # This ensures that task is accepted in the other (== less busy) instance of the virtual appliance
                              retriable_status_codes:
                                - 429
                              # How many times is the request retried
                              # Should be # of virtual appliance instances minus 1
                              num_retries: 1

                http_filters:
                  - name: envoy.filters.http.stateful_session
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.stateful_session.v3.StatefulSession
                      strict: true
                      session_state:
                        name: envoy.http.stateful_session.header
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.http.stateful_session.header.v3.HeaderBasedSessionState
                          # Name of the session header
                          # Contains base64 encoded upstream_address:port
                          # This tells Envoy to which upstream server it should send the request
                          name: session-header
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: speech-platform-virtual-appliance
      connect_timeout: 0.5s
      type: STATIC
      dns_lookup_family: V4_ONLY
      lb_policy: RANDOM
      load_assignment:
        cluster_name: speech-platform-virtual-appliance
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      # IP address of the first instance of the virtual appliance
                      address: 1.2.3.4
                      # Port of the first instance of the virtual appliance
                      port_value: 80
              - endpoint:
                  address:
                    socket_address:
                      # IP address of the second instance of the virtual appliance
                      address: 1.2.3.5
                      # Port of the second instance of the virtual appliance
                      port_value: 80
      health_checks:
        - timeout: 2s
          interval: 60s
          interval_jitter: 1s
          unhealthy_threshold: 3
          healthy_threshold: 3
          http_health_check:
            # Healthcheck uri of the speech api inside virtual appliance
            path: /api/system/status

# Admin interface for looking at things
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9090

Access API with LB

Here is an example script to show how to work with a header-based stateful session using curl:

#!/bin/bash

# URL of the virtual appliance or load balancer
platform_url=http://localhost:8080

# URI to POST the task to
uri="/api/technology/speech-to-text?language=en"

# Path to audio file for processing
voice_file=/tmp/audio.wav

# Proccess this many tasks in parallel
parallel=100

# End when this many tasks are processed
total_tasks=400

# Post tasks to the API so that we still have $parallel tasks running
post_tasks() {
  local count=$1
  local tmpfile_task=/tmp/task.${task_counter}.json
  local tmpfile_headers=/tmp/headers.${task_counter}.txt

  for i in $(seq 1 $count); do
    # POST single task
    echo "[${task_counter}] Task $i of ${count}"
    curl \
      -L -s -X POST  \
      -H 'Content-Type: multipart/form-data' \
      -H 'Accept: application/json' \
      -F file=@"${voice_file}" \
      --output ${tmpfile_task} \
      --dump-header ${tmpfile_headers} \
      "${platform_url}${uri}"

    rv=$?

    # Parse session header
    session_header=$(grep session-header ${tmpfile_headers} | cut -d ':' -f 2)
    echo "[${task_counter}] Curl response code is: ${rv}"
    echo "[${task_counter}] Session header is: ${session_header}"
    echo "[${task_counter}] $(cat ${tmpfile_task})"


    task_id=$(jq -r '.task.task_id' ${tmpfile_task})
    # Store task id
    current_tasks+=($task_id)
    # Store session header for each task id
    taskToHeader["${task_id}"]="${session_header}"
    task_counter=$((${task_counter} +1))
  done
}

# Poll for all running tasks
poll_tasks() {
  local counter_done=0
  local counter_rejected=0
  local counter_running=0
  local counter_pending=0
  local counter_unknown=0

  # Poll status of each task
  for task_id in ${current_tasks[@]}; do
    local tmpfile_task=/tmp/task-id-${task_id}

    # Add session header
    curl -s --header "session-header:${taskToHeader[${task_id}]}" -L -o ${tmpfile_task} "${platform_url}/api/task/${task_id}"
    rv=$?

    echo "[${task_id}] Curl response code is: ${rv}"
    task_status=$(jq -r '.state' ${tmpfile_task})
    echo "[${task_id}] Task is still ${task_status}..."

    # Evaluate task status
    case $task_status in
      pending)
        counter_pending=$((${counter_pending} +1))
      ;;

      running)
        counter_running=$((${counter_running} +1))
      ;;

      rejected)
        counter_rejected=$((${counter_rejected} +1))
      ;;

      done)
        counter_done=$((${counter_done} +1))
        counter_total_done=$((${counter_total_done} +1))
        finished_tasks+=(${task_id})
      ;;

      *)
        counter_unknown=$((${counter_unknown} +1))
      ;;
    esac
  done
  echo "Summary: Done: ${counter_done}, Rejected: ${counter_rejected}, Running: ${counter_running}, Pending: ${counter_pending}, Unknown: ${counter_unknown}"
}

rm -f /tmp/task-id-*

if [ ! -f ${voice_file} ]; then
  echo "Voicefile does not exists!"
  exit 1
fi

current_tasks=()
declare -A taskToHeader
task_counter=1
start_time=$(date '+%s')
counter_total_done=0

# Control loop
while true; do
  finished_tasks=()
  poll_tasks

  # Remove finished tasks
  for del in ${finished_tasks[@]}
  do
    current_tasks=(${current_tasks[@]/$del})
  done

  echo "Task counter: ${task_counter}, Finished tasks: ${counter_total_done}"
  if [ ${counter_total_done} -ge ${total_tasks} ]; then
    echo "Reached ${total_tasks} finished tasks."
    echo "Start time: ${start_time}"
    end_time=$(date '+%s')
    echo "End time: ${end_time}"
    echo "Duration: $(( ${end_time} - ${start_time} ))"
    echo "Voicefile: ${voice_file}"
    echo "task parallelism: ${parallel}"
    break
  fi

  # POST tasks to have $parallel tasks running all the time
  if [ ${#current_tasks[@]} -le $parallel ]; then
    post_tasks $(($parallel - ${#current_tasks[@]}))
  fi

  sleep 2
done

Installation​

Prerequisites​

Minimal HW requirements​

Resource usage per technology​

GPU​

Installation guide​

Post-installation steps​

SSH server​

Open ports​

K3s check​

Kubernetes check​

Application check​

Components​

Operating system​

GPU support​

Kubernetes​

Ingress controller​

Speech platform​

File Browser​

Prometheus​

Grafana​

Disks​

System disk​

Data disk​

Configuration​

Upload Speech to Text Phonexia and Time Analysis models​

Insert license keys​

Insert license into enhanced-speech-to-text-built-on-whisper microservice​

Insert license into voiceprint-extraction microservice​

Insert license into voiceprint-comparison microservice​

Insert license into speech-to-text-phonexia microservice​

Insert license into time-analysis microservice​

Set DNS name for speech platform virtual appliance​

Use HTTPS certificate​

Extend disks​

Extend system disk​

Extend data disk​

Disable unneeded microservices​

Phonexia speech to text microservice​

Permanent vs onDemand instances​

Configure languages in speech-to-text-phonexia microservice​

Modify replicas for permanent language instances​

Modify parallelism for instances​

Add custom images​

Modify microservice replicas​

Run enhanced-speech-to-text-built-on-whisper microservice on GPU​

Change model used in a microservice​

Change model in enhanced-speech-to-text-built-on-whisper microservice​

Load Speech to Text Phonexia and Time Analysis model from data disk​

Process patented audio codecs with media-conversion​

Pull Media Conversion image on the fly​

Put Media Conversion image into virtual appliance​

Limits​

API limits​

How to change the API limits​

UI limits​

Speaker Identification UI limits​

Speech to text UI limits​

How to change the UI limits​

Pod count limits​

How to change the pod count limits​

Admin backends limits​

How to change admin backends limits​

Admin console​

Grafana​

Troubleshooting​

Speech platform components​

Request flow​

Check node status​

Disk pressure​

Memory pressure​

View pod logs​

View status of a helm chart​

Upgrade guide​

Upgrade and retain data disk​

Upgrade and discard data disk​

Upgrade to 3.0.0​

Rename Whisper microservice​

Step by step upgrade guide to 3.0.0​

Upgrade to 2.1.0​

Installation

Prerequisites

Minimal HW requirements

Resource usage per technology

GPU

Installation guide

Post-installation steps

SSH server

Open ports

K3s check

Kubernetes check

Application check

Components

Operating system

GPU support

Kubernetes

Ingress controller

Speech platform

File Browser

Prometheus

Grafana

Disks

System disk

Data disk

Configuration

Upload Speech to Text Phonexia and Time Analysis models

Insert license keys

Insert license into enhanced-speech-to-text-built-on-whisper microservice

Insert license into voiceprint-extraction microservice

Insert license into voiceprint-comparison microservice

Insert license into speech-to-text-phonexia microservice

Insert license into time-analysis microservice

Set DNS name for speech platform virtual appliance

Use HTTPS certificate

Extend disks

Extend system disk

Extend data disk

Disable unneeded microservices

Phonexia speech to text microservice

Permanent vs onDemand instances

Configure languages in speech-to-text-phonexia microservice

Modify replicas for permanent language instances

Modify parallelism for instances

Add custom images

Modify microservice replicas

Run enhanced-speech-to-text-built-on-whisper microservice on GPU

Change model used in a microservice

Change model in enhanced-speech-to-text-built-on-whisper microservice

Load Speech to Text Phonexia and Time Analysis model from data disk

Process patented audio codecs with media-conversion

Pull Media Conversion image on the fly

Put Media Conversion image into virtual appliance

Limits

API limits

How to change the API limits

UI limits

Speaker Identification UI limits

Speech to text UI limits

How to change the UI limits

Pod count limits

How to change the pod count limits

Admin backends limits

How to change admin backends limits

Admin console

Grafana

Troubleshooting

Speech platform components

Request flow

Check node status

Disk pressure

Memory pressure

View pod logs

View status of a helm chart

Upgrade guide

Upgrade and retain data disk

Upgrade and discard data disk

Upgrade to 3.0.0

Rename Whisper microservice

Step by step upgrade guide to 3.0.0

Upgrade to 2.1.0