examples/hello-world/hello_world.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello World! NeMo-Run Style!\n",
    "\n",
    "Let's start with a simple notebook that demonstrates how to use NeMo-Run to configure and launch your Python functions. In this example, we take simple addition functions that look like:\n",
    "\n",
    "```python\n",
    "def add(a: int, b: int) -> int:\n",
    "    print(f\"Adding {a} to {b} returns {a + b}\")\n",
    "    return a + b\n",
    "```\n",
    "\n",
    "and use NeMo-Run to configure and launch it. This basic notebook demonstrates many of the building blocks in our NeMo-Run library.\n",
    "\n",
    "As described in the introduction, NeMo-Run is a tool that allows you to:\n",
    "1. Configure your functions or scripts in a Pythonic way.\n",
    "2. Launch them on any supported remote cluster directly from your local workstation.\n",
    "3. Manage them using `run.Experiment`.\n",
    "\n",
    "Let's get into it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
    "# SPDX-License-Identifier: Apache-2.0\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "# Set up and imports\n",
    "import logging\n",
    "\n",
    "import nemo_run as run\n",
    "from simple.add import add\n",
    "\n",
    "logging.basicConfig(level=logging.INFO, format=\"%(asctime)s %(message)s\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the Python Function\n",
    "\n",
    "The first step in using NeMo-Run is to configure your Python function. As mentioned above, we're trying to configure the `add` function. Configuration is similar to Python's native [functools.partial](https://docs.python.org/3/library/functools.html#functools.partial). In fact, for most functions, you can replace `functools.partial` with `run.Partial` and it should still work. Configuration just ties your function and your arguments together to create a `run.Partial` object which can executed at a later time. This is done by building the `run.Partial` object which will recursively build any configured objects and then return a [`functools.partial` object](https://docs.python.org/3/library/functools.html#partial-objects).\n",
    "\n",
    "This brings us to the first building blocks of NeMo-Run: `Partial` and `Config`. These are buildables that allow you to configure functions, arguments, and objects in a Pythonic way. Under the hood, we use [fiddle](https://fiddle.readthedocs.io/en/latest/) to manage the configuration. Our `Partial` and `Config` classes are subclasses of `fiddle.Partial` and `fiddle.Config`, respectively, with additional features to enhance UX and programmability. We take some inspiration from [Praxis](https://github.com/google/praxis/blob/main/praxis/pax_fiddle.py#L72), but also have custom addons that should improve the user experience.\n",
    "\n",
    "We already discussed `run.Partial`. Similarly, `run.Config` takes a `fn` or `class` as the first argument, followed by the `fn`'s or `class`\\' `*args`, and `**kwargs` as the subsequent arguments. However, on building `run.Config`, it actually calls the underlying `fn` or, in case of a `class`, its `__init__` method with the arguments tied into the `Config`. For example:\n",
    "\n",
    "\n",
    "```python\n",
    "import fiddle as fdl\n",
    "import nemo_run as run\n",
    "def hello_world(msg: str):\n",
    "    print(f\"Hello World! {msg}\")\n",
    "\n",
    "cfg = run.Config(hello_world, msg=\"How are you?\")\n",
    "partial = run.Partial(hello_world, msg=\"How are you?\")\n",
    "\n",
    "fn = fdl.build(partial)\n",
    "fn()\n",
    "#>>> Hello World! How are you?\n",
    "\n",
    "built_cfg = fdl.build(cfg)\n",
    "#>>> Hello World! How are you?\n",
    "\n",
    "print(built_cfg is None)\n",
    "#>>> True\n",
    "```\n",
    "\n",
    "The `Partial` and `Config` classes also come with utilities that provide a visual representation of the object (if you have `graphviz` installed). Try running the cell below to configure the `add` function and visualize the configured `Partial`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.1.2 (20240928.0832)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"76pt\" height=\"68pt\"\n",
       " viewBox=\"0.00 0.00 76.00 68.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 64)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-64 72,-64 72,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#ffc0cb\" stroke=\"none\" points=\"0,-40 0,-60 68,-60 68,-40 0,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"0,-40 0,-60 68,-60 68,-40 0,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"4\" y=\"-47.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Partial:</text>\n",
       "<text text-anchor=\"start\" x=\"40\" y=\"-47.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> add</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" stroke-dasharray=\"5,2\" points=\"0,-20 0,-40 31,-40 31,-20 0,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-20 0,-40 31,-40 31,-20 0,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">a</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"31,-20 31,-40 68,-40 68,-20 31,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"35\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">5</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,0 0,-20 31,-20 31,0 0,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,0 0,-20 31,-20 31,0 0,0\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">b</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"31,0 31,-20 68,-20 68,0 31,0\"/>\n",
       "<text text-anchor=\"start\" x=\"35\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Partial[add(a=5, b=10)]>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn = run.Partial(add, a=5, b=10)\n",
    "fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The configured function is now ready for execution. Before we proceed, let's look at configuring complex functions that take non-primitive types. Additionally, we will look at another utility that NeMo-Run provides: `run.autoconvert`. This decorator allows you to automatically convert functions that return regular Python objects into functions that return a `run.Config` of the underlying object. To demonstrate this, let's consider an `add_object` function that looks like:\n",
    "\n",
    "```python\n",
    "@dataclass\n",
    "class SomeObject:\n",
    "    value_1: int\n",
    "    value_2: int\n",
    "    value_3: int\n",
    "\n",
    "\n",
    "def add_object(obj_1: SomeObject, obj_2: SomeObject) -> SomeObject:\n",
    "    result = SomeObject(\n",
    "        value_1=obj_1.value_1 + obj_2.value_1,\n",
    "        value_2=obj_1.value_2 + obj_2.value_2,\n",
    "        value_3=obj_1.value_3 + obj_2.value_3,\n",
    "    )\n",
    "    print(f\"{result = }\")\n",
    "\n",
    "    return result\n",
    "```\n",
    "\n",
    "To configure this function, you can use `run.Partial`. Note, however, that you need to ensure the arguments to the function are `run.Config` or `run.Partial` instances. This is necessary for serialization and remote execution, but more on that later. For now, a basic configuration for add_object looks like:\n",
    "\n",
    "```python\n",
    "run.Partial(\n",
    "    add_object,\n",
    "    obj_1=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    "    obj_2=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    ")\n",
    "```\n",
    "\n",
    "Now, let's say you have a regular Python function that returns `SomeObject` and you want to use it as one of the arguments to `add_object`. You can decorate it using `run.autoconvert` as follows:\n",
    "```python\n",
    "@run.autoconvert\n",
    "def commonly_used_object() -> SomeObject:\n",
    "    return SomeObject(\n",
    "        value_1=5,\n",
    "        value_2=10,\n",
    "        value_3=15,\n",
    "    )\n",
    "```\n",
    "\n",
    "The `run.autoconvert` decorator uses fiddle's autoconfig and parses the AST so that you get the following:\n",
    "\n",
    "```python\n",
    "commonly_used_object() == run.Config(SomeObject, value_1=5, value_2=10, value_3=15)\n",
    "```\n",
    "\n",
    "\n",
    "You can then use the decorated function as follows:\n",
    "```python\n",
    "run.Partial(\n",
    "    add_object,\n",
    "    obj_1=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    "    obj_2=commonly_used_object(),\n",
    ")\n",
    "```\n",
    "\n",
    "Additionally, you can also use args in the function. Note, however, that `run.autoconvert` currently doesn't support control flow and may be unreliable for complex code. In that case, you can define a function to return the `run.Config` directly, as follows:\n",
    "\n",
    "```python\n",
    "def commonly_used_config() -> run.Config[SomeObject]:\n",
    "    config = run.Config(\n",
    "        SomeObject,\n",
    "        value_1=5,\n",
    "        value_2=10,\n",
    "        value_3=15,\n",
    "    )\n",
    "\n",
    "    for i in range(10):\n",
    "        config.value_1 *= i\n",
    "        config.value_2 += i\n",
    "        config.value_3 -= i\n",
    "\n",
    "    return config\n",
    "```\n",
    "\n",
    "> 📝 `autoconvert` returns a `run.Config` of the underlying object by default, whereas `@autoconvert(partial=True)` returns a `run.Partial` of the underlying object.\n",
    "\n",
    "One additional thing is that the arguments of a `run.Partial` or `run.Config` can be mutated via dot access, as shown below:\n",
    "```python\n",
    "fn.obj_1.value_1 = 100\n",
    "fn.obj_2.value_2 *= 2\n",
    "...\n",
    "```\n",
    "\n",
    "As you can see, our tool is designed to provide a lot of flexibility. Run the cells shown below to experiment with these APIs and learn about the different ways to configure your task or function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.1.2 (20240928.0832)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"236pt\" height=\"184pt\"\n",
       " viewBox=\"0.00 0.00 236.00 184.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 180)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-180 232,-180 232,4 -4,4\"/>\n",
       "<!-- 1 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"3.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"35.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-40 0,-60 67,-60 67,-40 0,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-40 0,-60 67,-60 67,-40 0,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,-40 67,-60 105,-60 105,-40 67,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-20 0,-40 67,-40 67,-20 0,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-20 0,-40 67,-40 67,-20 0,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,-20 67,-40 105,-40 105,-20 67,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">20</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,0 0,-20 67,-20 67,0 0,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,0 0,-20 67,-20 67,0 0,0\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,0 67,-20 105,-20 105,0 67,0\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">30</text>\n",
       "</g>\n",
       "<!-- 0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#ffc0cb\" stroke=\"none\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<text text-anchor=\"start\" x=\"29.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Partial:</text>\n",
       "<text text-anchor=\"start\" x=\"65.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> add_object</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" stroke-dasharray=\"5,2\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-142.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-136 91.5,-156 135.5,-156 135.5,-136 91.5,-136\"/>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-122.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-116 91.5,-136 135.5,-136 135.5,-116 91.5,-116\"/>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;1:c</title>\n",
       "<path fill=\"none\" stroke=\"#73be73\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M110.72,-140.26C104.47,-129.61 88.82,-102.91 75.31,-79.89\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"126.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"158.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-40 123,-60 190,-60 190,-40 123,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-40 123,-60 190,-60 190,-40 123,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,-40 190,-60 228,-60 228,-40 190,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-20 123,-40 190,-40 190,-20 123,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-20 123,-40 190,-40 190,-20 123,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,-20 190,-40 228,-40 228,-20 190,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">20</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,0 123,-20 190,-20 190,0 123,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,0 123,-20 190,-20 190,0 123,0\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,0 190,-20 228,-20 228,0 190,0\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">30</text>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;2:c</title>\n",
       "<path fill=\"none\" stroke=\"#cbc6b0\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M116.83,-120.48C122.62,-112.65 135.04,-95.81 146.98,-79.64\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Partial[add_object(\n",
       "  obj_1=<Config[SomeObject(value_1=10, value_2=20, value_3=30)]>,\n",
       "  obj_2=<Config[SomeObject(value_1=10, value_2=20, value_3=30)]>)]>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from simple.add import add_object, commonly_used_object, commonly_used_object_2, SomeObject\n",
    "\n",
    "fn = run.Partial(\n",
    "    add_object,\n",
    "    obj_1=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    "    obj_2=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    ")\n",
    "fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.1.2 (20240928.0832)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"236pt\" height=\"184pt\"\n",
       " viewBox=\"0.00 0.00 236.00 184.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 180)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-180 232,-180 232,4 -4,4\"/>\n",
       "<!-- 1 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"3.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"35.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-40 0,-60 64,-60 64,-40 0,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-40 0,-60 64,-60 64,-40 0,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"18\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"64,-40 64,-60 105,-60 105,-40 64,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"68\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">100</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-20 0,-40 64,-40 64,-20 0,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-20 0,-40 64,-40 64,-20 0,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"18\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"64,-20 64,-40 105,-40 105,-20 64,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"68\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,0 0,-20 64,-20 64,0 0,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,0 0,-20 64,-20 64,0 0,0\"/>\n",
       "<text text-anchor=\"start\" x=\"18\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"64,0 64,-20 105,-20 105,0 64,0\"/>\n",
       "<text text-anchor=\"start\" x=\"68\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">15</text>\n",
       "</g>\n",
       "<!-- 0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#ffc0cb\" stroke=\"none\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<text text-anchor=\"start\" x=\"29.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Partial:</text>\n",
       "<text text-anchor=\"start\" x=\"65.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> add_object</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" stroke-dasharray=\"5,2\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-142.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-136 91.5,-156 135.5,-156 135.5,-136 91.5,-136\"/>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-122.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-116 91.5,-136 135.5,-136 135.5,-116 91.5,-116\"/>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;1:c</title>\n",
       "<path fill=\"none\" stroke=\"#73be73\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M110.72,-140.26C104.47,-129.61 88.82,-102.91 75.31,-79.89\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"126.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"158.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-40 123,-60 184,-60 184,-40 123,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-40 123,-60 184,-60 184,-40 123,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"138\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"184,-40 184,-60 228,-60 228,-40 184,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"188\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">500</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-20 123,-40 184,-40 184,-20 123,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-20 123,-40 184,-40 184,-20 123,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"138\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"184,-20 184,-40 228,-40 228,-20 184,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"188\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">1000</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,0 123,-20 184,-20 184,0 123,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,0 123,-20 184,-20 184,0 123,0\"/>\n",
       "<text text-anchor=\"start\" x=\"138\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"184,0 184,-20 228,-20 228,0 184,0\"/>\n",
       "<text text-anchor=\"start\" x=\"188\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">1500</text>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;2:c</title>\n",
       "<path fill=\"none\" stroke=\"#cbc6b0\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M116.83,-120.48C122.62,-112.65 135.04,-95.81 146.98,-79.64\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Partial[add_object(\n",
       "  obj_1=<Config[SomeObject(value_1=100, value_2=10, value_3=15)]>,\n",
       "  obj_2=<Config[SomeObject(value_1=500, value_2=1000, value_3=1500)]>)]>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn = run.Partial(add_object, obj_1=commonly_used_object(), obj_2=commonly_used_object_2())\n",
    "fn.obj_1.value_1 = 100\n",
    "fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.1.2 (20240928.0832)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"236pt\" height=\"184pt\"\n",
       " viewBox=\"0.00 0.00 236.00 184.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 180)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-180 232,-180 232,4 -4,4\"/>\n",
       "<!-- 1 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-60 0,-80 105,-80 105,-60 0,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"3.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"35.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-40 0,-60 67,-60 67,-40 0,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-40 0,-60 67,-60 67,-40 0,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,-40 67,-60 105,-60 105,-40 67,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">5</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-20 0,-40 67,-40 67,-20 0,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-20 0,-40 67,-40 67,-20 0,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,-20 67,-40 105,-40 105,-20 67,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,0 0,-20 67,-20 67,0 0,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,0 0,-20 67,-20 67,0 0,0\"/>\n",
       "<text text-anchor=\"start\" x=\"21\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"67,0 67,-20 105,-20 105,0 67,0\"/>\n",
       "<text text-anchor=\"start\" x=\"71\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">15</text>\n",
       "</g>\n",
       "<!-- 0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#ffc0cb\" stroke=\"none\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"25.5,-156 25.5,-176 135.5,-176 135.5,-156 25.5,-156\"/>\n",
       "<text text-anchor=\"start\" x=\"29.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Partial:</text>\n",
       "<text text-anchor=\"start\" x=\"65.5\" y=\"-163.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> add_object</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" stroke-dasharray=\"5,2\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-136 25.5,-156 91.5,-156 91.5,-136 25.5,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-142.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-136 91.5,-156 135.5,-156 135.5,-136 91.5,-136\"/>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-140 95.5,-152 131.5,-152 131.5,-140 95.5,-140\"/>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"25.5,-116 25.5,-136 91.5,-136 91.5,-116 25.5,-116\"/>\n",
       "<text text-anchor=\"start\" x=\"57.5\" y=\"-122.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">obj_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"91.5,-116 91.5,-136 135.5,-136 135.5,-116 91.5,-116\"/>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"95.5,-120 95.5,-132 131.5,-132 131.5,-120 95.5,-120\"/>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;1:c</title>\n",
       "<path fill=\"none\" stroke=\"#73be73\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M110.72,-140.26C104.47,-129.61 88.82,-102.91 75.31,-79.89\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"#fff8dc\" stroke=\"none\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-60 123,-80 228,-80 228,-60 123,-60\"/>\n",
       "<text text-anchor=\"start\" x=\"126.75\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"158.25\" y=\"-67.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> SomeObject</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-40 123,-60 190,-60 190,-40 123,-40\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-40 123,-60 190,-60 190,-40 123,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_1</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,-40 190,-60 228,-60 228,-40 190,-40\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-46.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">10</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,-20 123,-40 190,-40 190,-20 123,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,-20 123,-40 190,-40 190,-20 123,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_2</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,-20 190,-40 228,-40 228,-20 190,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-26.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">20</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"123,0 123,-20 190,-20 190,0 123,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"123,0 123,-20 190,-20 190,0 123,0\"/>\n",
       "<text text-anchor=\"start\" x=\"144\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">value_3</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"190,0 190,-20 228,-20 228,0 190,0\"/>\n",
       "<text text-anchor=\"start\" x=\"194\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">30</text>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;2:c</title>\n",
       "<path fill=\"none\" stroke=\"#cbc6b0\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M116.83,-120.48C122.62,-112.65 135.04,-95.81 146.98,-79.64\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Partial[add_object(\n",
       "  obj_1=<Config[SomeObject(value_1=5, value_2=10, value_3=15)]>,\n",
       "  obj_2=<Config[SomeObject(value_1=10, value_2=20, value_3=30)]>)]>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn = run.Partial(\n",
    "    add_object,\n",
    "    obj_1=commonly_used_object(),\n",
    "    obj_2=run.Config(SomeObject, value_1=10, value_2=20, value_3=30),\n",
    ")\n",
    "fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute the Configured Function\n",
    "\n",
    "The previous section concludes the core features of the NeMo-Run configuration. Now, let's move on to execution. The core building block to execute a single function is `run.run`.\n",
    "\n",
    "The easiest way to execute a configured function is to execute it directly, just like you would execute a normal Python function. You can do this using the `direct` option in `run.run` as shown below:\n",
    "```python\n",
    "# You can also do a dryrun to see what's getting executed where.\n",
    "run.run(fn, direct=True, dryrun=True)\n",
    "run.run(fn, direct=True)\n",
    "```\n",
    "\n",
    "> 📝 As of now, the `run` function doesn't recreate the return value, so it is the user's responsibility to manage the artifacts from the function. For example, if you are launching a function to train an ML model, make sure you configure a job directory that you can access and inspect later. We are working on improving the management capabilities of a run and open to feedback, so please reach out to us if you have any ideas.\n",
    "\n",
    "Try it out in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">Dry run for task simple.add:add_object</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36mDry run for task simple.\u001b[0m\u001b[1;36madd:add\u001b[0m\u001b[1;36m_object\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Resolved Arguments</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;32mResolved Arguments\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Argument Name        </span>┃<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Resolved Value                                               </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> obj_1                </span>│ <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SomeObject</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">value_1</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #808000; text-decoration-color: #808000\">value_2</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>, <span style=\"color: #808000; text-decoration-color: #808000\">value_3</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span><span style=\"font-weight: bold\">)</span>                │\n",
       "│<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> obj_2                </span>│ <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">SomeObject</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">value_1</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>, <span style=\"color: #808000; text-decoration-color: #808000\">value_2</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #808000; text-decoration-color: #808000\">value_3</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span><span style=\"font-weight: bold\">)</span>               │\n",
       "└──────────────────────┴──────────────────────────────────────────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1;35m \u001b[0m\u001b[1;35mArgument Name       \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mResolved Value                                              \u001b[0m\u001b[1;35m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[2m \u001b[0m\u001b[2mobj_1               \u001b[0m\u001b[2m \u001b[0m│ \u001b[1;35mSomeObject\u001b[0m\u001b[1m(\u001b[0m\u001b[33mvalue_1\u001b[0m=\u001b[1;36m5\u001b[0m, \u001b[33mvalue_2\u001b[0m=\u001b[1;36m10\u001b[0m, \u001b[33mvalue_3\u001b[0m=\u001b[1;36m15\u001b[0m\u001b[1m)\u001b[0m                │\n",
       "│\u001b[2m \u001b[0m\u001b[2mobj_2               \u001b[0m\u001b[2m \u001b[0m│ \u001b[1;35mSomeObject\u001b[0m\u001b[1m(\u001b[0m\u001b[33mvalue_1\u001b[0m=\u001b[1;36m10\u001b[0m, \u001b[33mvalue_2\u001b[0m=\u001b[1;36m20\u001b[0m, \u001b[33mvalue_3\u001b[0m=\u001b[1;36m30\u001b[0m\u001b[1m)\u001b[0m               │\n",
       "└──────────────────────┴──────────────────────────────────────────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now real run\n",
      "result = SomeObject(value_1=15, value_2=30, value_3=45)\n"
     ]
    }
   ],
   "source": [
    "# You can also do a dryrun to see what's getting executed where.\n",
    "run.run(fn, direct=True, dryrun=True)\n",
    "print(\"Now real run\")\n",
    "run.run(fn, direct=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Direct runs are not that interesting, as you wouldn't need to use `run` if you were only running the function directly. Beyond direct runs, we support a few executors out of the box:\n",
    "1. Local executor: This will run the function locally, but in a separate process. It is useful when you are using components like `torchrun` to launch your job/run.\n",
    "1. Slurm executor: This will run the function on a remote Slurm cluster. Currently, only Slurm clusters that have [Pyxis](https://github.com/NVIDIA/pyxis) are supported, but we plan to add support for other types of Slurm clusters in the future. Reach out to us if you have a specific request.\n",
    "1. Skypilot executor: This will run the function on any cloud supported by [Skypilot](https://skypilot.readthedocs.io/en/latest/).\n",
    "\n",
    "To learn more about executors, go to the execution guide linked in [README](../../README.md).\n",
    "\n",
    "> 📝 Currently, we do not pickle the function, but rely on Python module references for serializing the configured function. This means that executing a function defined in this notebook (or in the same script calling `run.run`) will probably not work. During packaging, we currently use `git archive` for our remote Slurm executor. You can specify a subpath which will be set as the working directory for the execution. To execute `add_object` function in this example, you need to provide a subpath of `examples/hello-world`, assuming `examples` is at the root of the repository. This will ensure that, during remote execution, imports of the style `from simple.add import add_object` work properly. We are looking to improve this user experience and open to suggestions, so please reach out to us if you have any ideas. We also plan to add `cloudpickle` support for arbitrarily defined functions.\n",
    "\n",
    "Under the hood, we are using [TorchX](https://pytorch.org/torchx/latest/) to manage the execution. However, this is fairly abstracted away from the user and we can potentially support more standalone executor libraries or add custom schedulers for TorchX in the future.\n",
    "\n",
    "Let's see the local executor in action. The local executor is the simplest executor we have and can be initialized without any arguments. Once we have an instance of `run.LocalExecutor`, we can pass it to `run.run` to execute the same configured function on the local executor, which will run the function in a separate process. This demonstrates the ease of use provided by NeMo-Run. You can configure a function once and execute it on any supported remote cluster seamlessly. Later, we will also explore `run.Experiment` which allows you to combine and manage multiple runs, providing additional flexibility to the user.\n",
    "\n",
    "Execute the cells below to run your function on the local executor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.1.2 (20240928.0832)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"173pt\" height=\"252pt\"\n",
       " viewBox=\"0.00 0.00 173.00 252.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 248)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-248 169,-248 169,4 -4,4\"/>\n",
       "<!-- 1 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"72,-20 72,-40 165,-40 165,-20 72,-20\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"72,-20 72,-40 165,-40 165,-20 72,-20\"/>\n",
       "<text text-anchor=\"start\" x=\"75.75\" y=\"-27.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"107.25\" y=\"-27.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> Packager</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"72,0 72,-20 118,-20 118,0 72,0\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"72,0 72,-20 118,-20 118,0 72,0\"/>\n",
       "<text text-anchor=\"start\" x=\"84\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">debug</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"118,0 118,-20 165,-20 165,0 118,0\"/>\n",
       "<text text-anchor=\"start\" x=\"122\" y=\"-6.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">False</text>\n",
       "</g>\n",
       "<!-- 0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"#ffc0cb\" stroke=\"none\" points=\"0,-224 0,-244 139,-244 139,-224 0,-224\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-224 0,-244 139,-244 139,-224 0,-224\"/>\n",
       "<text text-anchor=\"start\" x=\"11.75\" y=\"-231.5\" font-family=\"Courier,monospace\" font-size=\"8.00\">Config:</text>\n",
       "<text text-anchor=\"start\" x=\"43.25\" y=\"-231.5\" font-family=\"Courier,monospace\" font-size=\"10.00\"> LocalExecutor</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-204 0,-224 98,-224 98,-204 0,-204\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-204 0,-224 98,-224 98,-204 0,-204\"/>\n",
       "<text text-anchor=\"start\" x=\"46\" y=\"-210.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">packager</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-204 98,-224 139,-224 139,-204 98,-204\"/>\n",
       "<polygon fill=\"#90ee90\" stroke=\"none\" points=\"102,-208 102,-220 135,-220 135,-208 102,-208\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"102,-208 102,-220 135,-220 135,-208 102,-208\"/>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-184 0,-204 98,-204 98,-184 0,-184\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-184 0,-204 98,-204 98,-184 0,-184\"/>\n",
       "<text text-anchor=\"start\" x=\"46\" y=\"-190.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">launcher</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-184 98,-204 139,-204 139,-184 98,-184\"/>\n",
       "<text text-anchor=\"start\" x=\"102\" y=\"-190.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">None</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-156 0,-184 98,-184 98,-156 0,-156\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-156 0,-184 98,-184 98,-156 0,-156\"/>\n",
       "<text text-anchor=\"start\" x=\"46\" y=\"-166.88\" font-family=\"Courier,monospace\" font-size=\"10.00\">env_vars</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-156 98,-184 139,-184 139,-156 98,-156\"/>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"102,-159.25 102,-180 135,-180 135,-159.25 102,-159.25\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"102,-159.25 102,-180 135,-180 135,-159.25 102,-159.25\"/>\n",
       "<text text-anchor=\"start\" x=\"106.5\" y=\"-166.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">dict</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-136 0,-156 98,-156 98,-136 0,-136\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-136 0,-156 98,-156 98,-136 0,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"52\" y=\"-142.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">retries</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-136 98,-156 139,-156 139,-136 98,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"102\" y=\"-142.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">0</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-116 0,-136 98,-136 98,-116 0,-116\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-116 0,-136 98,-136 98,-116 0,-116\"/>\n",
       "<text text-anchor=\"start\" x=\"16\" y=\"-122.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">experiment_id</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-116 98,-136 139,-136 139,-116 98,-116\"/>\n",
       "<text text-anchor=\"start\" x=\"102\" y=\"-122.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">None</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-96 0,-116 98,-116 98,-96 0,-96\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-96 0,-116 98,-116 98,-96 0,-96\"/>\n",
       "<text text-anchor=\"start\" x=\"52\" y=\"-102.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">job_dir</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-96 98,-116 139,-116 139,-96 98,-96\"/>\n",
       "<text text-anchor=\"start\" x=\"102\" y=\"-102.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">&#39;&#39;</text>\n",
       "<polygon fill=\"#eeeeee\" stroke=\"none\" points=\"0,-76 0,-96 98,-96 98,-76 0,-76\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-76 0,-96 98,-96 98,-76 0,-76\"/>\n",
       "<text text-anchor=\"start\" x=\"4\" y=\"-82.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">ntasks_per_node</text>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"98,-76 98,-96 139,-96 139,-76 98,-76\"/>\n",
       "<text text-anchor=\"start\" x=\"102\" y=\"-82.5\" font-family=\"Courier,monospace\" font-size=\"10.00\">1</text>\n",
       "</g>\n",
       "<!-- 0&#45;&#45;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0:c&#45;&#45;1:c</title>\n",
       "<path fill=\"none\" stroke=\"#73be73\" stroke-width=\"3\" stroke-opacity=\"0.501961\" d=\"M118.5,-208.25C118.5,-184.98 118.5,-82.33 118.5,-39.68\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "LocalExecutor(packager=Packager(debug=False), launcher=None, env_vars={}, retries=0, experiment_id=None, job_dir='', experiment_dir='', _launcher_setup=False, ntasks_per_node=1)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "executor = run.LocalExecutor()\n",
    "executor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">─────────────── </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Entering Experiment simple.add.add_object with id: simple.add.add_object_1730126761</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ───────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m─────────────── \u001b[0m\u001b[1;35mEntering Experiment simple.add.add_object with id: simple.add.add_object_1730126761\u001b[0m\u001b[92m ───────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Log directory is: /Users/andreyma/.nemo_run/experiments/simple.add.add_object/simple.add.add_object_1730126761/simple.add.add_object\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[15:46:01] </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">Launching job simple.add.add_object for experiment simple.add.add_object</span>               <a href=\"file:///Users/andreyma/workspace/nvidia/NeMo-Run/src/nemo_run/run/experiment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">experiment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/andreyma/workspace/nvidia/NeMo-Run/src/nemo_run/run/experiment.py#660\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">660</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[15:46:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;36mLaunching job simple.add.add_object for experiment simple.add.add_object\u001b[0m               \u001b]8;id=741146;file:///Users/andreyma/workspace/nvidia/NeMo-Run/src/nemo_run/run/experiment.py\u001b\\\u001b[2mexperiment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=572721;file:///Users/andreyma/workspace/nvidia/NeMo-Run/src/nemo_run/run/experiment.py#660\u001b\\\u001b[2m660\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Log directory is: /Users/andreyma/.nemo_run/experiments/simple.add.add_object/simple.add.add_object_1730126761/simple.add.add_object\n",
      "Launched app: local_persistent://nemo_run/simple.add.add_object-ccgx567ghm3nwd\n",
      "AppStatus:\n",
      "    State: RUNNING\n",
      "    Num Restarts: 0\n",
      "    Roles: \n",
      "    Msg: <NONE>\n",
      "    Structured Error Msg: <NONE>\n",
      "    UI URL: file:///Users/andreyma/.nemo_run/experiments/simple.add.add_object/simple.add.add_object_1730126761/simple.add.add_object/nemo_run/simple.add.add_object-ccgx567ghm3nwd\n",
      "    \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">──────────────────────── </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Waiting for Experiment simple.add.add_object_1730126761 to finish</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m──────────────────────── \u001b[0m\u001b[1;35mWaiting for Experiment simple.add.add_object_1730126761 to finish\u001b[0m\u001b[92m ────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Experiment Status for</span> <span style=\"color: #ffaf00; text-decoration-color: #ffaf00; font-weight: bold\">simple.add.add_object_1730126761</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;32mExperiment Status for\u001b[0m \u001b[1;38;5;214msimple.add.add_object_1730126761\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Task 0</span>: <span style=\"color: #ffaf00; text-decoration-color: #ffaf00; font-weight: bold\">simple.add.add_object</span>\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Status</span>: RUNNING\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Executor</span>: LocalExecutor\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Job id</span>: simple.add.add_object-ccgx567ghm3nwd\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Local Directory</span>: /Users/andreyma/.nemo_run/experiments/simple.add.add_object/simple.add.add_object_1730126761/simple.add.add_object\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;32mTask 0\u001b[0m: \u001b[1;38;5;214msimple.add.add_object\u001b[0m\n",
       "- \u001b[1;32mStatus\u001b[0m: RUNNING\n",
       "- \u001b[1;32mExecutor\u001b[0m: LocalExecutor\n",
       "- \u001b[1;32mJob id\u001b[0m: simple.add.add_object-ccgx567ghm3nwd\n",
       "- \u001b[1;32mLocal Directory\u001b[0m: /Users/andreyma/.nemo_run/experiments/simple.add.add_object/simple.add.add_object_1730126761/simple.add.add_object\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Waiting for job simple.add.add_object-ccgx567ghm3nwd to finish [log=True]...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "add_object/0 result = SomeObject(value_1=15, value_2=30, value_3=45)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job simple.add.add_object-ccgx567ghm3nwd finished: SUCCEEDED\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"background-color: #272822\">                                                                                                                   </span>\n",
       "<span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># The experiment was run with the following tasks: ['simple.add.add_object']</span><span style=\"background-color: #272822\">                                       </span>\n",
       "<span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># You can inspect and reconstruct this experiment at a later point in time using:</span><span style=\"background-color: #272822\">                                  </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">experiment </span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">=</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\"> run</span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">.</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">Experiment</span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">.</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">from_id(</span><span style=\"color: #e6db74; text-decoration-color: #e6db74; background-color: #272822\">\"simple.add.add_object_1730126761\"</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">)</span><span style=\"background-color: #272822\">                                            </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">experiment</span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">.</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">status() </span><span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># Gets the overall status</span><span style=\"background-color: #272822\">                                                                      </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">experiment</span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">.</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">logs(</span><span style=\"color: #e6db74; text-decoration-color: #e6db74; background-color: #272822\">\"simple.add.add_object\"</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">) </span><span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># Gets the log for the provided task</span><span style=\"background-color: #272822\">                                      </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">experiment</span><span style=\"color: #ff4689; text-decoration-color: #ff4689; background-color: #272822\">.</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">cancel(</span><span style=\"color: #e6db74; text-decoration-color: #e6db74; background-color: #272822\">\"simple.add.add_object\"</span><span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">) </span><span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># Cancels the provided task if still running</span><span style=\"background-color: #272822\">                            </span>\n",
       "<span style=\"background-color: #272822\">                                                                                                                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[48;2;39;40;34m                                                                                                                   \u001b[0m\n",
       "\u001b[38;2;149;144;119;48;2;39;40;34m# The experiment was run with the following tasks: ['simple.add.add_object']\u001b[0m\u001b[48;2;39;40;34m                                       \u001b[0m\n",
       "\u001b[38;2;149;144;119;48;2;39;40;34m# You can inspect and reconstruct this experiment at a later point in time using:\u001b[0m\u001b[48;2;39;40;34m                                  \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m=\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mrun\u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m.\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mExperiment\u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m.\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mfrom_id\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m(\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34msimple.add.add_object_1730126761\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m)\u001b[0m\u001b[48;2;39;40;34m                                            \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m.\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mstatus\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m(\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m)\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;149;144;119;48;2;39;40;34m# Gets the overall status\u001b[0m\u001b[48;2;39;40;34m                                                                      \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m.\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mlogs\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m(\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34msimple.add.add_object\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m)\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;149;144;119;48;2;39;40;34m# Gets the log for the provided task\u001b[0m\u001b[48;2;39;40;34m                                      \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;255;70;137;48;2;39;40;34m.\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mcancel\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m(\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34msimple.add.add_object\u001b[0m\u001b[38;2;230;219;116;48;2;39;40;34m\"\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m)\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;149;144;119;48;2;39;40;34m# Cancels the provided task if still running\u001b[0m\u001b[48;2;39;40;34m                            \u001b[0m\n",
       "\u001b[48;2;39;40;34m                                                                                                                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"background-color: #272822\">                                                                                                                   </span>\n",
       "<span style=\"color: #959077; text-decoration-color: #959077; background-color: #272822\"># You can inspect this experiment at a later point in time using the CLI as well:</span><span style=\"background-color: #272822\">                                  </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">nemo experiment status simple.add.add_object_1730126761</span><span style=\"background-color: #272822\">                                                            </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">nemo experiment logs simple.add.add_object_1730126761 </span><span style=\"color: #ae81ff; text-decoration-color: #ae81ff; background-color: #272822\">0</span><span style=\"background-color: #272822\">                                                            </span>\n",
       "<span style=\"color: #f8f8f2; text-decoration-color: #f8f8f2; background-color: #272822\">nemo experiment cancel simple.add.add_object_1730126761 </span><span style=\"color: #ae81ff; text-decoration-color: #ae81ff; background-color: #272822\">0</span><span style=\"background-color: #272822\">                                                          </span>\n",
       "<span style=\"background-color: #272822\">                                                                                                                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[48;2;39;40;34m                                                                                                                   \u001b[0m\n",
       "\u001b[38;2;149;144;119;48;2;39;40;34m# You can inspect this experiment at a later point in time using the CLI as well:\u001b[0m\u001b[48;2;39;40;34m                                  \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mnemo\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mstatus\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34msimple.add.add_object_1730126761\u001b[0m\u001b[48;2;39;40;34m                                                            \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mnemo\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mlogs\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34msimple.add.add_object_1730126761\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;174;129;255;48;2;39;40;34m0\u001b[0m\u001b[48;2;39;40;34m                                                            \u001b[0m\n",
       "\u001b[38;2;248;248;242;48;2;39;40;34mnemo\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mexperiment\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34mcancel\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34msimple.add.add_object_1730126761\u001b[0m\u001b[38;2;248;248;242;48;2;39;40;34m \u001b[0m\u001b[38;2;174;129;255;48;2;39;40;34m0\u001b[0m\u001b[48;2;39;40;34m                                                          \u001b[0m\n",
       "\u001b[48;2;39;40;34m                                                                                                                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "run.run(fn, executor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you notice the logs, it has mentions of Experiment. Each `run.run` internally creates an experiment with a single task to provide management capabilities for a run out of the box. We also make the `run.Experiment` API available publicly to create custom experiments and workflows. Check out our next tutorial in this series named [`hello_experiments.ipynb`](hello_experiments.ipynb) to learn more about experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way, you can execute your configured function on a remote Slurm cluster or remote Skypilot cluster as well. \n",
    "\n",
    "Below, we show an example of defining a `run.SlurmExecutor`. NeMo-Run also sets up an `ssh` tunnel automatically for you to connect to the Slurm cluster and handle the packaging of the code so you never have to leave your local workstation. You can configure the Slurm executor as below:\n",
    "\n",
    "```python\n",
    "tunnel_cfg = run.Config(\n",
    "    TunnelConfig,\n",
    "    host=os.environ[\"SLURM_HOST\"],\n",
    "    user=os.environ[\"SLURM_USER\"],\n",
    "    job_dir=os.environ[\"SLURM_JOBDIR\"],\n",
    ")\n",
    "packager = run.Config(\n",
    "    GitArchivePackager,\n",
    "    use_torchrun=False,\n",
    "    subpath=\"examples/hello-world\"\n",
    ")\n",
    "\n",
    "executor = run.Config(\n",
    "    SlurmExecutorConfig,\n",
    "    account=os.environ[\"SLURM_ACCT\"],\n",
    "    partition=os.environ[\"SLURM_PARTITION\"],\n",
    "    nodes=1,\n",
    "    ntasks_per_node=1,\n",
    "    tunnel=tunnel,\n",
    "    container_image=os.environ[\"BASE_IMAGE\"],\n",
    "    time=\"00:30:00\",\n",
    "    packager=packager,\n",
    ")\n",
    "```\n",
    "\n",
    "More details about configuring the executors can be found in the `API Reference` portion of our docs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}