docs/samples/components/wordfreq.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "@file:DependsOn(\"com.londogard:nlp:1.2.0-BETA2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "First thing to do is import the required dependencies and then we'll fetch some word frequencies to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordFrequency of 'hej'=2.9512093E-4 and 'och'=0.025118863"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import com.londogard.nlp.wordfreq.WordFrequencies\n",
    "import com.londogard.nlp.utils.LanguageSupport\n",
    "\n",
    "val hej = WordFrequencies.wordFrequency(\"hej\", LanguageSupport.sv)\n",
    "val och = WordFrequencies.wordFrequency(\"och\", LanguageSupport.sv)\n",
    "\n",
    "\"WordFrequency of 'hej'=$hej and 'och'=$och\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "What about using the Zipf Frequency instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ZipfFrequency of 'hej'=5.4700003 and 'och'=7.4"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val hej = WordFrequencies.zipfFrequency(\"hej\", LanguageSupport.sv)\n",
    "val och = WordFrequencies.zipfFrequency(\"och\", LanguageSupport.sv)\n",
    "\n",
    "\"ZipfFrequency of 'hej'=$hej and 'och'=$och\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Non existing words are handled to default to 0, using `OrNull` suffix you'll retrieve a null instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordFrequency of 'hraihaodjasmdiamo' (non-word) using `wordFrequency` 0.0 and using `wordFrequencyOrNull` null"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val weird = WordFrequencies.wordFrequency(\"hraihaodjasmdiamo\", LanguageSupport.sv)\n",
    "val weirdOrNull = WordFrequencies.wordFrequencyOrNull(\"hraihaodjasmdiamo\", LanguageSupport.sv)\n",
    "\n",
    "\"WordFrequency of 'hraihaodjasmdiamo' (non-word) using `wordFrequency` $weird and using `wordFrequencyOrNull` $weirdOrNull\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Trying to get frequencies from a `LanguageSupport` that is not supported will result in a failure and exception thrown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "runCatching { WordFrequencies.wordFrequency(\"hello\", LanguageSupport.af) }.isFailure // WordFrequencies does not support af"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Again, using `OrNull` suffix will help in returning `null` rather than throwing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "WordFrequencies.wordFrequencyOrNull(\"hello\", LanguageSupport.af) == null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally it's possible to retrieve all Word Frequencies for a language, this will return a `Map<String, Float>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WordFrequencies.wordFrequencyOrNull(\"hello\", LanguageSupport.af) == null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally it's possible to retrieve all Word Frequencies for a language, this will return a `Map<String, Float>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[är=0.037153527, det=0.031622775, att=0.026302677]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WordFrequencies.getAllWordFrequenciesOrNull(LanguageSupport.sv)?.entries?.take(3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Kotlin",
   "language": "kotlin",
   "name": "kotlin"
  },
  "language_info": {
   "codemirror_mode": "text/x-kotlin",
   "file_extension": ".kt",
   "mimetype": "text/x-kotlin",
   "name": "kotlin",
   "nbconvert_exporter": "",
   "pygments_lexer": "kotlin",
   "version": "1.7.20-dev-1299"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}