For AI agents: a documentation index is available at /llms.txt. Markdown versions of all pages can be requested by appending `.md` to the URL, or by setting the `Accept` header to `text/markdown`.
Skip to main content
DeploymentsContainer

GPU Speech to text container (Melia 1)

Learn about the Speechmatics Melia 1 GPU container system

Melia 1 is a multilingual, GPU-only model, generally available in Batch. For an overview of the model, see Models. Real-time (streaming) inference is available as an early preview — see Real-time inference.

Prerequisites​

System requirements​

The system must have:

  • Nvidia GPU(s) with at least 16GB of GPU memory
  • Nvidia drivers (see below for supported versions)
  • CUDA compute capability of 7.5-12.1 inclusive, which corresponds to the Turing, Ampere, Lovelace, Hopper, Blackwell architectures. Cards with the Volta architecture or below are not able to run the models
  • 24 GB RAM
  • The nvidia-container-toolkit installed
  • Docker version > 19.03

See Performance and cost for more information on the performance and cost of the container.

Nvidia drivers​

  • The GPU Inference Container is based on CUDA 13.0.1, which requires NVIDIA Driver release 580 or later.

Driver installation can be validated by running nvidia-smi. This command should return the Nvidia driver version and show additional information about the GPU(s).

Cloud instances​

The GPU node can be provisioned in the cloud.

Running the image​

Currently, each GPU Inference Container can only run on a single GPU. If a system has more than one GPU, the device must be specified using CUDA_VISIBLE_DEVICES or selecting the device using the --gpus argument. See Nvidia/CUDA documentation for details.

Pull the Melia 1 inference server image as described in Accessing images, then run it:

docker run --rm -it \
  -v $PWD/license.json:/license.json \
  --gpus '"device=0"' \
  -e CUDA_VISIBLE_DEVICES \
  -p 8001:8001 \
  speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.3.0

When the Container starts you should see output similar to this, indicating that the server has started and is ready to serve requests.

I0705 08:12:55.419608 1 server.cc:709]
+--------------+---------+--------+
| Model        | Version | Status |
+--------------+---------+--------+
| aed          | 1       | READY  |
| body         | 1       | READY  |
| detokenizer  | 1       | READY  |
| dz           | 1       | READY  |
| ensemble     | 1       | READY  |
| preprocessor | 1       | READY  |
+--------------+---------+--------+
...
I0705 08:12:55.515473 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0705 08:12:55.515940 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0705 08:12:55.598672 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"

Batch inference​

The Melia 1 inference server processes whole files and returns the transcript at the end. For streaming, see Real-time inference below.

Once the inference server is running, run the Melia 1 transcriber with the SM_INFERENCE_ENDPOINT environment variable set to the gRPC endpoint of the inference server. The transcriber offloads inference to the server, so it does not require a GPU. The transcriber is a single unified image for both batch and real-time, so the SM_ASR_MODE environment variable must also be set, to batch or rt:

docker run --rm -it \
  -e SM_INFERENCE_ENDPOINT=<server>:<port> \
  -e SM_ASR_MODE=batch \
  -v $PWD/license.json:/license.json \
  -v $PWD/example.wav:/input.audio \
  speechmaticspublic.azurecr.io/sm-asr-transcriber-melia-1:${smVariables.latestMelia1ContainerVersion}

To accept multiple jobs over an HTTP API without restarting the container between jobs, run the transcriber as a batch persistent worker.

Real-time inference​

Real-time (streaming) inference for Melia 1 is an early preview for evaluation only, and is not intended for production use. Feature coverage matches the Melia 1 Realtime Preview.

The Melia 1 inference server can run in two modes: batch or real-time. The default mode is batch. To configure the server for real-time, set the SM_STREAMING environment variable to true when starting the inference server:

docker run --rm -it \
  -v $PWD/license.json:/license.json \
  --gpus '"device=0"' \
  -e CUDA_VISIBLE_DEVICES \
  -e SM_STREAMING=true \
  -p 8001:8001 \
  speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.3.0

When the Container starts you should see output similar to this, indicating that the server has started and is ready to serve requests.

I0911 11:04:02.321449 1 server.cc:709] 
+--------------------+---------+--------+
| Model              | Version | Status |
+--------------------+---------+--------+
| aed                | 1       | READY  |
| body               | 1       | READY  |
| preprocessor       | 1       | READY  |
| streaming          | 1       | READY  |
| streaming_ensemble | 1       | READY  |
+--------------------+---------+--------+
...
I0911 11:04:02.449550 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0911 11:04:02.449744 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0911 11:04:02.491582 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"

The server can only support one of these modes at once.

Once the inference server is running in real-time mode, run the Melia 1 transcriber as normal, with the SM_INFERENCE_ENDPOINT environment variable set to the gRPC endpoint of the inference server. Unlike the Standard and Enhanced GPU container, Melia 1 uses the same transcriber image for both batch and real-time — there are no separate rt- and batch- prefixed images. Set SM_ASR_MODE=rt to run the transcriber in real-time mode:

docker run --rm -it \
  -e SM_INFERENCE_ENDPOINT=<server>:<port> \
  -e SM_ASR_MODE=rt \
  -v $PWD/license.json:/license.json \
  -p 9000:9000 \
  speechmaticspublic.azurecr.io/sm-asr-transcriber-melia-1:${smVariables.latestMelia1ContainerVersion}

Connect over WebSocket as described in Realtime transcription.

Monitoring the server​

The inference server is based on Nvidia's Triton architecture and as such can be monitored using Triton's inbuilt Prometheus metrics, or the GRPC/HTTP APIs. To expose these, configure an external mapping for port 8002(Prometheus) or 8000(HTTP).

Docker compose example​

This Docker Compose file will create a Speechmatics Melia 1 GPU Inference Server:

(assumes your license.json file is in the current working directory)

docker-compose.yml
version: "3.8"

networks:
  transcriber:
    driver: bridge

services:
  triton:
    image: speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.3.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              ### Limit to N GPUs
              # count: 1
              ### Pick specific GPUs by device ID
              # device_ids:
              #   - 0
              #   - 3
              capabilities:
                - gpu
    container_name: triton
    networks:
      - transcriber
    expose:
      - 8000/tcp
      - 8001/tcp
      - 8002/tcp
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all
      - CUDA_VISIBLE_DEVICES=0
    volumes:
      - $PWD/license.json:/license.json:ro