Secure Voice Recognition in Telegram
Secure Voice Recognition in Telegram

Xoxo, everyone — it’s Reznik, and today we’re going to recognize voice messages…

Let’s step away from affiliate networks for a bit and build a bot to help out with tech support. Its job is to recognize the content of voice messages and video messages in Telegram group chats. We’ll use Python as our language of choice. We’ll recognize speech using the free, open-source vosk library, and add punctuation using a model from Silero. The same mechanism is what powers our own AlterCPA Talk Bot.

A bot like this has one crucial advantage: speech recognition happens locally and doesn’t need to touch the Yandex or Google API. It’s completely secure and perfect for paranoids and classified information alike. Even a junior developer or DevOps engineer without deep specialized knowledge can get this bot running.

Step 1. Setting Up the Bot in Telegram

First, we need to create a bot and get its token so we can work with it.

  1. Open Telegram and find @BotFather
  2. Send the bot the /newbot command to create a new bot
  3. Give our bot a name and its future address
  4. The bot is created — copy the token you get and jot it down somewhere
  5. Send the bot the /mybots command to show the list of bots
  6. Pick the bot you just created from the list
  7. Click Bot Settings
  8. Click Allow groups and make sure it says Groups are currently enabled for bot

The bot is now ready and cleared to work in group chats. That way, it’ll be able to automatically recognize messages in groups.

Step 2. Setting Up the Server

As you’ve probably guessed, we’re going to need a virtual server. My bots run on the Scarlett and Danny plans from Timeweb. The OS of choice is plain Debian 10. Pretty much any OS will do, in theory, but step 5 might look different on a different one. Installing the software will take up about 1.5 GB of space, and the language models will need another 2-2.5 GB on top of that.

Before we start, make sure to update the system:

apt update && apt upgrade -y

Install Python and the other software we’ll need:

apt install -y python3 python3-dev python3-pip unzip ffmpeg

Install the modules we need: torch, vosk, PyTelegramBotAPI, and their dependencies:

pip3 install PyTelegramBotAPI
pip3 install --no-cache-dir vosk
pip3 install --no-cache-dir wave
pip3 install --no-cache-dir numpy
pip3 install --no-cache-dir torch
pip3 cache purge

Keep in mind that some of these modules are pretty hefty — the install might take a while.

Step 3. Downloading the Speech Recognition Models

The recognition models will be stored locally. Let’s put them in the /home/ml folder. They’ll need about 3 GB of free space. You can find the available languages and recognition models on the vosk site, in the Models section. As of this writing, the latest model is vosk-model-ru-0.22, and that’s the one we’ll be using.

The punctuation model comes from Silero’s GitHub repository, where we’ll need the file models.yml. In it, find te_models and its package under latest. That’s the link we’re after — mine led right here.

It’s best to run these commands one at a time — downloading the models can take a ton of time.

mkdir /home/ml
cd /home/ml
wget https://alphacephei.com/vosk/models/vosk-model-ru-0.22.zip
unzip vosk-model-ru-0.22.zip
rm -f vosk-model-ru-0.22.zip
wget https://models.silero.ai/te_models/v2_4lang_q.pt

The models are downloaded and ready to go. Keep in mind that by the time you’re reading this, the links might have changed. Use the latest versions of the files.

Step 4. Writing the Bot

You can download the finished bot file, voxy.py, from our repository on GitLab. Just toss it onto your server and enjoy the process — that process is covered in the next step. In this one, let’s break down its code piece by piece.

At the top of the file, we specify the version of Python we’re working with. Yours is probably a lot newer — check it with which python3.

#!/usr/bin/python3
# coding: utf-8

Import all the modules we downloaded in step two.

import telebot
import pathlib
import requests
import subprocess
import os
import json
import wave
import torch
from vosk import Model, KaldiRecognizer

Specify our bot’s token from step one.

TOKEN = '12345:AAAA-BBBBBB_CCC'

Specify the paths to the models and the language we’re using. If you’ve been following the instructions exactly, these paths won’t change.

MODEL = r"/home/ml/vosk-model-ru-0.22"
TEMODEL = "/home/ml/v2_4lang_q.pt"
LANG = 'ru'

Set up the bot and the models.

WORKDIR = str(pathlib.Path(__file__).parent.absolute())
bot = telebot.TeleBot( TOKEN )
model = Model( MODEL )
voska = KaldiRecognizer( model, 16000 )
tmodel = torch.package.PackageImporter( TEMODEL ).load_pickle( "te_model", "model" )

We’re going to intercept every message of type “voice” and “video message”.

@bot.message_handler(content_types=["voice","video_note"])
def voice_decoder(message):

What follows is the code for the nested function — don’t forget about indentation. First, let’s check the message type.

    if ( message.voice != None ):
        file =  message.voice
    elif ( message.video_note != None ):
        file =  message.video_note
    else:
        return False

Download the file attached to the message.

    finfo = bot.get_file(file.file_id)
    try:
        contents = requests.get( 'https://api.telegram.org/file/bot{0}/{1}'.format(TOKEN, finfo.file_path) )
    except Exception:
        return False

Save the file right next to the bot. We already found the WORKDIR path during initialization.

    downpath = WORKDIR + "/" + file.file_unique_id
    with open( downpath, 'wb' ) as dest:
        dest.write(contents.content)

Convert the file with a magic command we’ll show you below.

    path = audioconvert( downpath )
    if ( path == False ):
        return False

Turn the file into text with a simple call to the recognition model. If it worked, run the text through the enhancement model.

    text = speech2text( path )
    os.remove( path )
    if ( text == False or text == "" or text == " " ):
        return False
    else:
        text = tmodel.enhance_text( text, LANG )

Send the message back as a reply — and that’s the end of the recognition work.

    bot.reply_to(message, text)

We’ll use FFmpeg for audio conversion, because I honestly don’t know of any other options. We need a file at a 16k bitrate, PCM format, mono.

def audioconvert(path):
    out_path = path + ".wav"
    command = [
        r'/usr/bin/ffmpeg',
        '-i', path,
        '-acodec', 'pcm_s16le',
        '-ac', '1',
        '-ar', '16000',
        out_path
    ]
    result = subprocess.run(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL )
    os.remove( path )
    if ( result.returncode ):
        os.remove( out_path )
        return False
    else:
        return out_path

To extract the text through vosk, we’ll use some ready-made magic from its developers. I don’t have the faintest idea how it works or why it’s built this way. But it works.

def speech2text(path):
    wf = wave.open(path, "rb")
    result = ''
    last_n = False
    while True:
        data = wf.readframes(16000)
        if len(data) == 0:
            break
        if voska.AcceptWaveform(data):
            res = json.loads(voska.Result())
            if res['text'] != '':
                result += f" {res['text']}"
                last_n = False
            elif not last_n:
                result += 'n'
                last_n = True
    res = json.loads(voska.FinalResult())
    result += f" {res['text']}"
    return result

Last step — start the bot up in active polling mode.

if __name__ == '__main__':
    bot.infinity_polling()

Step 5. Running the Bot as a Service

Let’s place the bot in the /home/bot folder and run it from there.

mkdir /home/bot
cd /home/bot
wget https://gitlab.com/altervision/altercpa-voxy/-/raw/main/voxy.py
chmod a+x voxy.py
nano voxy.py

Enter the bot’s token from step 1, save (Ctrl + O, Enter, Ctrl + X). Let’s try starting the bot up and see if it runs.

./voxy.py

You’ll see a few loading messages, and the bot will be ready within a few seconds. Send it a voice message and wait for a reply. In theory, nothing should go wrong at this step. If it does, go ahead and tear your hair out, throw a tantrum, and file your complaints into the void.

Let’s create a file that’ll be responsible for running our service. We’ll call the service voxybot.

nano /lib/systemd/system/voxybot.service

The file’s contents will look something like this.

[Unit]
Description=VoxyBot

[Service]
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/home/bot/voxy.py

[Install]
WantedBy=multi-user.target

Reload the services, enable the bot, and start the service.

systemctl daemon-reload
systemctl enable voxybot.service
service voxybot start

Step 6. Testing It Out

Your bot is now running. Open a chat with it and send it a voice message. Then a video message. If there’s no reply, you know the drill: panic, denial, anger, bargaining, depression, and finally accepting the inevitable — contacting support, or learning Python. If you do get a reply — enjoy.

TL;DR: The Easy Install

Once you’ve got your bot’s token, log into your server and run the following commands:

wget https://gitlab.com/altervision/altercpa-voxy/-/raw/main/setup-ru.sh
bash setup-ru.sh YOURBOTTOKEN

Replace YOURBOTTOKEN with the token you got from BotFather. The install might take a couple of hours — the script needs to download around 3.5 GB of files. Once it’s done, the bot will start working on its own.

That’s it from Reznik — xoxo, everyone!