Audio Transcription

MiniBot can transcribe audio sent via Telegram using faster-whisper.

Setup

  1. Install the stt extra:

    poetry install --extras stt

  2. Ensure ffmpeg is available on the host.

  3. Enable file storage and transcription in config.toml:

[tools.file_storage]
enabled = true
root_dir = "./data/files"

[tools.audio_transcription]
enabled = true
model = "medium"
device = "cpu"
compute_type = "int8"
beam_size = 5
vad_filter = true

  1. Send an audio file to your Telegram bot as a document/file attachment (.mp3, .wav, .m4a, etc.).

  2. Ask the bot to transcribe it: “transcribe this audio”.

Notes:

  • Telegram voice and audio message types are ingested automatically, as well as file/document uploads.

  • If channels.telegram.allowed_document_mime_types is set, include your audio MIME types.

  • In the Docker yolo profile, Whisper model assets are downloaded lazily on first use and cached under /app/data/.cache.

Model Selection

[tools.audio_transcription].model is a faster-whisper model name, not an LLM. Valid values include tiny, base, small (the default), medium, large-v3 and turbo (.en variants exist for English-only use). Larger models are more accurate but slower.

A CPU-only host is good enough for typical use — model = "medium" with device = "cpu" and compute_type = "int8" transcribes Spanish reliably and avoids CUDA setup entirely. Use device = "auto" to prefer CUDA when available and fall back to CPU; see GPU Runtime Dependencies below when running on GPU.

The automatic-ingest knobs default to transcribing short incoming voice/audio messages on arrival:

  • auto_transcribe_short_incoming — transcribe short Telegram voice/audio attachments automatically (default: true).

  • auto_transcribe_max_duration_seconds — cap for automatic transcription (default: 45); longer files must be transcribed explicitly.

Remote whisper.cpp Server

Set server_url to a whisper.cpp server /inference endpoint to transcribe there instead of loading faster-whisper locally. The stt extra is not needed, and model, device, compute_type and vad_filter are ignored (the server’s own model and flags apply); beam_size, language and translate mode are forwarded. Requests wait up to 10 minutes for a result.

[tools.audio_transcription]
enabled = true
server_url = "http://whisper:8080/inference"

GPU Runtime Dependencies

If STT fails with Library libcublas.so.12 is not found, your CUDA runtime libraries are missing from the loader path.

Debian / Ubuntu

sudo apt update
sudo apt install -y nvidia-driver nvidia-cuda-toolkit libcudnn9 libcudnn9-cuda-12

echo '/usr/local/cuda/lib64' | sudo tee /etc/ld.so.conf.d/cuda.conf
sudo ldconfig
ldconfig -p | grep libcublas.so.12

Arch / Manjaro

sudo pacman -Syu cuda cudnn
echo '/opt/cuda/lib64' | sudo tee /etc/ld.so.conf.d/cuda.conf
sudo ldconfig
ldconfig -p | grep libcublas.so.12

Alternative: CUDA runtime libs inside Poetry venv

Useful when system CUDA versions do not match your Python wheel:

poetry run pip install -U nvidia-cublas-cu12 nvidia-cudnn-cu12
export SP=$(poetry run python -c "import site; print(next(p for p in site.getsitepackages() if 'site-packages' in p))")
export LD_LIBRARY_PATH="$SP/nvidia/cublas/lib:$SP/nvidia/cudnn/lib:${LD_LIBRARY_PATH}"