examples/tutorials/flare_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e129ede5",
   "metadata": {},
   "source": [
    "   # Hello FLARE API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf7e391",
   "metadata": {},
   "source": [
    "In this notebook, we go through the different commands of the FLARE API to show the syntax and usage of each."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbca0050",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 1. Install NVIDIA FLARE and Provision and Start an FL System\n",
    "\n",
    "For this notebook, we will need a running NVFLARE project that we can connect to. Follow the [Installation](https://nvflare.readthedocs.io/en/main/getting_started.html#installation) instructions to set up an environment that has NVIDIA FLARE installed if you do not have one already.\n",
    "\n",
    "To provision and start an FL system, you can use [POC mode](setup_poc.ipynb) to quickly get started. Feel free to use an existing **provisioned** NVFLARE project if you have that available.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fe3165d",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2. Connect to the FL System with the FLARE API and Get System Info\n",
    "\n",
    "Use `new_secure_session()` to initiate a session connecting to the FL Server with the FLARE API. The necessary arguments are the username of the admin user you are using and the corresponding startup kit location.\n",
    "\n",
    "In the code example below, we get the `admin_user_dir` by using the default project location for POC mode. Please change the workspace location and the project name to what applies for your environment if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3dbde69",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from nvflare.fuel.flare_api.flare_api import new_secure_session\n",
    " \n",
    "username = \"admin@nvidia.com\"\n",
    "admin_user_dir = os.path.join(\"/tmp/nvflare/poc/example_project/prod_00\", username)\n",
    "sess = new_secure_session(\n",
    "    username=username,\n",
    "    startup_kit_location=admin_user_dir\n",
    ")\n",
    "print(sess.get_system_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fb361e5",
   "metadata": {},
   "source": [
    "Note that if debug mode is not enabled, there is no output after initiating a session successfully, so we print the output of `get_system_info()`.\n",
    "\n",
    "The `get_system_info()` command does not take any arguments and returns a SystemInfo object consisting of server_info (server status and start time), client_info (each connected client and the last connect time for that client), and job_info (the list of current jobs with the job_id and app_name).\n",
    "\n",
    "If you are unable to connect and initiate a session, make sure that your FL Server is running and that the configurations are correct with the right path to the admin startup kit directory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79fd45a2-4bc9-4b2e-b501-dda1b43de557",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3. Prepare Examples\n",
    "\n",
    "We are going to use hello-numpy-sag for an example job to submit. The following command copies the example to the place jobs are expected assuming you are using the default poc workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d3395a6-e902-4068-9c36-f858f3583ffa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cp -r  ../hello-world/hello-numpy-sag /tmp/nvflare/poc/example_project/prod_00/admin@nvidia.com/transfer/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b162ce0e-e219-475d-a014-9a7a23145239",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! ls -al /tmp/nvflare/poc/example_project/prod_00/admin@nvidia.com/transfer/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405edb37",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4. Submit Job\n",
    "With a session successfully connected, you can use `submit_job()` to submit your job. You can change `path_to_example_job` to the location of the job you are submitting (here, we are using the example job we copied to our transfer directory in the previous step). Upon successful submission, the job_id is returned as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8f08cef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "path_to_example_job = \"hello-numpy-sag/jobs/hello-numpy-sag\"\n",
    "job_id = sess.submit_job(path_to_example_job)\n",
    "print(job_id + \" was submitted\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42317cf3",
   "metadata": {},
   "source": [
    "### 5. Monitor Job\n",
    "\n",
    "The command `monitor_job()` allows you to follow a job until the job is done.\n",
    "\n",
    "By default, `monitor_job()` only has one required arguement, the `job_id` of the job you are waiting for, and the default behavior is to wait until the job is complete before returning a Return Code of `JOB_FINISHED`.\n",
    "\n",
    "In order to follow along and see a more meaningful result and demonstrate some of the possibilities of using this function, the following cell contains a `sample_cb()` callback that keeps track of the number of times the callback is run and prints the `job_meta` the first three times and the final time before `monitor_job()` completes with every other call just printing a dot to save output space. This callback is just an example of what can be done with additional arguments and the `job_meta` information of the job that is being monitored. You can use logic to return a value other than True to conditionally stop `monitor_job()` and return `MonitorReturnCode.ENDED_BY_CB`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03fd93d0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from nvflare.fuel.flare_api.flare_api import Session\n",
    "\n",
    "def sample_cb(\n",
    "        session: Session, job_id: str, job_meta, *cb_args, **cb_kwargs\n",
    "    ) -> bool:\n",
    "    if job_meta[\"status\"] == \"RUNNING\":\n",
    "        if cb_kwargs[\"cb_run_counter\"][\"count\"] < 3:\n",
    "            print(job_meta)\n",
    "            print(cb_kwargs[\"cb_run_counter\"])\n",
    "        else:\n",
    "            print(\".\", end=\"\")\n",
    "    else:\n",
    "        print(\"\\n\" + str(job_meta))\n",
    "    \n",
    "    cb_kwargs[\"cb_run_counter\"][\"count\"] += 1\n",
    "    return True\n",
    "\n",
    "sess.monitor_job(job_id, cb=sample_cb, cb_run_counter={\"count\":0})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a1fdfc",
   "metadata": {},
   "source": [
    "### 6. Get Job Meta\n",
    "\n",
    "To get the job meta information, you can use the `get_job_meta()` command. The only argument required for this command is the job id for the job you are getting the meta for. The job meta information will be returned as an object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a9b0b73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sess.get_job_meta(job_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "272bccab",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 7. List Jobs\n",
    "\n",
    "To get the information for what jobs have been submitted to the server, you can use the `list_jobs()` command. If no arguments are included, the defaults are False for both \"detailed\" and \"reverse\".\n",
    "\n",
    "Setting \"detailed\" to True will return more detailed information about each job.\n",
    "\n",
    "Setting \"reverse\" to True will return the jobs in reverse order by submission time.\n",
    "\n",
    "The arg \"limit\" can be set to specify the maximum number of jobs to return, with 0 or None meaning return all jobs (the default is None to show all).\n",
    "\n",
    "The args \"id_prefix\" and \"name_prefix\" can be used to further filter the jobs returned to have an id or name beginning with the string set for the respective argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fe50268",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def format_json( data: dict): \n",
    "    print(json.dumps(data, sort_keys=True, indent=4,separators=(',', ': ')))\n",
    "\n",
    "list_jobs_output = sess.list_jobs()\n",
    "print(format_json(list_jobs_output))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae0f5c25",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "list_jobs_output_detailed = sess.list_jobs(detailed=True)\n",
    "print(format_json(list_jobs_output_detailed))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ade87b",
   "metadata": {},
   "source": [
    "### 8. Download Job Result\n",
    "\n",
    "The `download_job_result()` command downloads the job result to the \"download_dir\" (this is usually set in fed_admin.json in the startup directory of the admin_user_dir used when launching the FLARE API Session, and this value is relative to the admin_user_dir). This command only has one required arguement, the `job_id` of the job result to download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d43556bf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sess.download_job_result(job_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7dced59",
   "metadata": {},
   "source": [
    "### 9. Clone Jobs\n",
    "\n",
    "To clone an existing job, you can use the `clone_job()` command. The `clone_job()` command only has one required arguement, the `job_id` of the job to clone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "512f7dc2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sess.clone_job(job_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bfbcf71",
   "metadata": {},
   "source": [
    "### 10. Abort Job\n",
    "\n",
    "If training gets stuck or there is another reason to stop a running job, you can use the `abort_job()` command. The `abort_job()` command only has one required arguement, the `job_id` of the job to abort, and None is returned upon successfully issuing the command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "187cb413",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(job_id)\n",
    "sess.abort_job(job_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4110775e",
   "metadata": {},
   "source": [
    "### 11. Delete Job\n",
    "\n",
    "You should be able to see the output in the terminals where you are running your FL Server and Clients when you submitted the job. You can also use `monitor_job()` to follow along and give you updates on the progress until the job is done.\n",
    "\n",
    "By default, `delete_job()` only has one required argument, the `job_id` of the job you are waiting for, and the default behavior is to return nothing if the command is successful. The command will raise a `JobNotDone` exception if the job is still running. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "178a860b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sess.delete_job(job_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f43f4831",
   "metadata": {},
   "source": [
    "### 12. All Other Commands\n",
    "\n",
    "For all other commands that do not have a specific FLARE API implementation, using ``api.do_command()`` should provide what the FLAdminAPI would have returned in the raw result."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nvflare_example",
   "language": "python",
   "name": "nvflare_example"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}