tutorials/ASR.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ac0c14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SPDX-FileCopyrightText: Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
    "# SPDX-License-Identifier: MIT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c97d9d54",
   "metadata": {},
   "source": [
    "# ASR API tutorial\n",
    "\n",
    "This tutorial demonstates how to use Python Riva API.\n",
    "\n",
    "## <font color=\"blue\">Server</font>\n",
    "\n",
    "Before running client part of Riva, please set up a server. The simplest\n",
    "way to do this is to follow\n",
    "[quick start guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html#local-deployment-using-quick-start-scripts).\n",
    "\n",
    "\n",
    "## <font color=\"blue\">Authentication</font>\n",
    "\n",
    "Before using Riva services you will need to establish connection with a server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd57f561",
   "metadata": {},
   "outputs": [],
   "source": [
    "import riva.client\n",
    "\n",
    "uri = \"localhost:50051\"  # Default value\n",
    "\n",
    "auth = riva.client.Auth(uri=uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5319eae",
   "metadata": {},
   "source": [
    "## <font color=\"blue\">Setting up service</font>\n",
    "\n",
    "To instantiate a service pass `riva.client.Auth` instance to a constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb75ba38",
   "metadata": {},
   "outputs": [],
   "source": [
    "asr_service = riva.client.ASRService(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4f6a1c",
   "metadata": {},
   "source": [
    "For speech recognition you will need to create a recognition config (an instance of `riva.client.RecognitionConfig`). \n",
    "A detailed description of config fields is available in Riva \n",
    "[documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/reference/protos/riva_asr.proto.html?highlight=max%20alternatives#riva-proto-riva-asr-proto).\n",
    "If you intend to use streaming recognition, an offline config has to wrapped into `riva.client.StreamingRecognitionConfig`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94e0e2c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import deepcopy\n",
    "offline_config = riva.client.RecognitionConfig(\n",
    "    encoding=riva.client.AudioEncoding.LINEAR_PCM,\n",
    "    max_alternatives=1,\n",
    "    enable_automatic_punctuation=True,\n",
    "    verbatim_transcripts=False,\n",
    ")\n",
    "streaming_config = riva.client.StreamingRecognitionConfig(config=deepcopy(offline_config), interim_results=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e975f38",
   "metadata": {},
   "source": [
    "You also need to a set frame rate and number of channels of audio which is going to be processed. If you'd like to process file `data/examples/en-US_AntiBERTa_for_word_boosting_testing.wav`, then your code will be"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3723b2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_wav_file = '../data/examples/en-US_AntiBERTa_for_word_boosting_testing.wav'\n",
    "riva.client.add_audio_file_specs_to_config(offline_config, my_wav_file)\n",
    "riva.client.add_audio_file_specs_to_config(streaming_config, my_wav_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8175e1d",
   "metadata": {},
   "source": [
    "If you intent to use word boosting, then use convenience method `riva.client.add_word_boosting_to_config()` to add boosting parameters to config."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6aff022",
   "metadata": {},
   "outputs": [],
   "source": [
    "boosted_lm_words = ['AntiBERTa', 'ABlooper']\n",
    "boosted_lm_score = 20.0\n",
    "riva.client.add_word_boosting_to_config(offline_config, boosted_lm_words, boosted_lm_score)\n",
    "riva.client.add_word_boosting_to_config(streaming_config, boosted_lm_words, boosted_lm_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7818f6f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(offline_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cdfc64a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(streaming_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c87b82b",
   "metadata": {},
   "source": [
    "## <font color=\"blue\">Offline</font>\n",
    "\n",
    "To run offline speech recognition read data from a file and pass to a service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "539c95f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(my_wav_file, 'rb') as fh:\n",
    "    data = fh.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f41894",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = asr_service.offline_recognize(data, offline_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35834e64",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffdd9348",
   "metadata": {},
   "source": [
    "To extract a transcript you may use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76532974",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(response.results[0].alternatives[0].transcript)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f39e2c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(response.results[0].alternatives[0].confidence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83feec76",
   "metadata": {},
   "source": [
    "### <font color=\"green\">Asynchronous calls</font>\n",
    "\n",
    "You can recognize speech asynchronously by setting `future=True` in `ASRService.offline_recognize()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cd9037d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time\n",
    "\n",
    "num_repeats = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c42dce3",
   "metadata": {},
   "outputs": [],
   "source": [
    "sync_transcripts = []\n",
    "start_time = time()\n",
    "for _ in range(num_repeats):\n",
    "    sync_transcripts.append(\n",
    "        asr_service.offline_recognize(data, offline_config).results[0].alternatives[0].transcript\n",
    "    )\n",
    "print(f\"Time spent on synchronous recognition: {time() - start_time:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e75d008",
   "metadata": {},
   "outputs": [],
   "source": [
    "async_transcripts = []\n",
    "start_time = time()\n",
    "futures = []\n",
    "for _ in range(num_repeats):\n",
    "    futures.append(asr_service.offline_recognize(data, offline_config, future=True))\n",
    "for f in futures:\n",
    "    async_transcripts.append(f.result().results[0].alternatives[0].transcript)\n",
    "print(f\"Time spent on async recognition: {time() - start_time:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56dfdde1",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert sync_transcripts == async_transcripts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5254e8",
   "metadata": {},
   "source": [
    "## <font color=\"blue\">Streaming</font>\n",
    "\n",
    "To imitate audio streaming use `riva.client.AudioChunkFileIterator`. You can imitate realtime audio by providing a delay callback to the iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eb7d7a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wav_parameters = riva.client.get_wav_file_parameters(my_wav_file)\n",
    "# correponds to 1 second of audio\n",
    "chunk_size = wav_parameters['framerate']\n",
    "with riva.client.AudioChunkFileIterator(\n",
    "    my_wav_file, chunk_size, delay_callback=riva.client.sleep_audio_length,\n",
    ") as audio_chunk_iterator:\n",
    "    for i, chunk in enumerate(audio_chunk_iterator):\n",
    "        print(i, len(chunk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b146f07",
   "metadata": {},
   "source": [
    "Then audio chunks are passed to `ASRService.streaming_response_generator()` and response generator is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a6cce30",
   "metadata": {},
   "outputs": [],
   "source": [
    "audio_chunk_iterator = riva.client.AudioChunkFileIterator(my_wav_file, 4800)\n",
    "response_generator = asr_service.streaming_response_generator(audio_chunk_iterator, streaming_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c8ee10",
   "metadata": {},
   "source": [
    "You may find description of streaming response (`StreamingRecognizeResponse`) fields in Riva [documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/reference/protos/riva_asr.proto.html?highlight=max%20alternatives#riva-proto-riva-asr-proto)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f401240",
   "metadata": {},
   "outputs": [],
   "source": [
    "streaming_response = next(response_generator)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b9df74c",
   "metadata": {},
   "source": [
    "For showing streaming results it is convenient to use function `riva.client.print_streaming()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ecdc5b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "riva.client.print_streaming(response_generator, additional_info='time')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffdc7afb",
   "metadata": {},
   "source": [
    "If you set a delay callback in audio chunk iterator and `show_intermediate=True` in `riva.client.print_streaming()`, then you will be able watch transcript forming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "380b2ef3",
   "metadata": {},
   "outputs": [],
   "source": [
    "audio_chunk_iterator = riva.client.AudioChunkFileIterator(my_wav_file, 4800, riva.client.sleep_audio_length)\n",
    "response_generator = asr_service.streaming_response_generator(audio_chunk_iterator, streaming_config)\n",
    "riva.client.print_streaming(response_generator, show_intermediate=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1aaea8f",
   "metadata": {},
   "source": [
    "It is also possible to print streaming results in several places, e.g. in STDOUT and a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f531607b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "output_file = \"my_results.txt\"\n",
    "audio_chunk_iterator = riva.client.AudioChunkFileIterator(my_wav_file, 4800)\n",
    "response_generator = asr_service.streaming_response_generator(audio_chunk_iterator, streaming_config)\n",
    "riva.client.print_streaming(response_generator, additional_info='confidence', output_file=[sys.stdout, output_file])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3339c08c",
   "metadata": {},
   "source": [
    "Showing file and clean up in bash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f205c5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat $output_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a948a2dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm $output_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9addd43e",
   "metadata": {},
   "source": [
    "Showing file and clean up in cmd.exe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ea3158",
   "metadata": {},
   "outputs": [],
   "source": [
    "!type $output_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02a8efd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!del $output_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b40e15",
   "metadata": {},
   "source": [
    "## <font color=\"blue\">Audio input/output</font>\n",
    "\n",
    "For using audio input and output you need to install PyAudio.\n",
    "\n",
    "```bash\n",
    "conda install -c anaconda pyaudio\n",
    "```\n",
    "\n",
    "### <font color=\"green\">Playing audio during transcribing</font>\n",
    "\n",
    "For playing audio simultaneously with transcribing, provide an instance of `riva.client.audio_io.SoundCallBack` as a `delay_callback` to `riva.client.AudioChunkFileIterator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9097d307",
   "metadata": {},
   "outputs": [],
   "source": [
    "import riva.client.audio_io"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "530af21d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# show available output devices\n",
    "riva.client.audio_io.list_output_devices()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70f52ead",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_device = None  # use default device\n",
    "wav_parameters = riva.client.get_wav_file_parameters(my_wav_file)\n",
    "sound_callback = riva.client.audio_io.SoundCallBack(\n",
    "    output_device, wav_parameters['sampwidth'], wav_parameters['nchannels'], wav_parameters['framerate'],\n",
    ")\n",
    "audio_chunk_iterator = riva.client.AudioChunkFileIterator(my_wav_file, 4800, sound_callback)\n",
    "response_generator = asr_service.streaming_response_generator(audio_chunk_iterator, streaming_config)\n",
    "riva.client.print_streaming(response_generator, show_intermediate=True)\n",
    "sound_callback.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aee8c861",
   "metadata": {},
   "source": [
    "### <font color=\"green\">Streaming from microphone</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "251ed66e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "riva.client.audio_io.list_input_devices()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57c5a1ad",
   "metadata": {},
   "source": [
    "Run code below and then say something in English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb174fe0",
   "metadata": {
    "tags": [
     "do_not_test"
    ]
   },
   "outputs": [],
   "source": [
    "input_device = None  # default device\n",
    "with riva.client.audio_io.MicrophoneStream(\n",
    "    rate=streaming_config.config.sample_rate_hertz,\n",
    "    chunk=streaming_config.config.sample_rate_hertz // 10,\n",
    "    device=input_device,\n",
    ") as audio_chunk_iterator:\n",
    "    riva.client.print_streaming(\n",
    "        responses=asr_service.streaming_response_generator(\n",
    "            audio_chunks=audio_chunk_iterator,\n",
    "            streaming_config=streaming_config,\n",
    "        ),\n",
    "        show_intermediate=True,\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}