roadmap.ipynb

{
    "cells": [
        {
            "source": [
                "A list of features for a Notebook **client**.\n",
                "\n",
                "# Top-level structure\n",
                "\n",
                "- Rendering\n",
                "- Editor features\n",
                "- Cell manipulation \n",
                "  - Commands/Keybindings\n",
                "- Accessibility\n",
                "- Language features\n",
                "- Debug\n",
                "- MISC\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "Annotations\n",
                "\n",
                "| Mark | Description | \n",
                "| --- | --- |\n",
                "| 🏃 | work in progress |\n",
                "| ✔️ | supported |\n",
                "| 1️⃣ | p1 |\n",
                "| 2️⃣ | p2 |\n",
                "| 🐍 | supported in Python extension | "
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Rendering\n",
                "\n",
                "A notebook consists of an ordered list of cells. Each cell can be markdown content or source code with executed outputs. Currently we use `marked.js` to render markdown content and a full size monaco editor for source code.\n",
                "\n",
                "Currently there are still two three missing pieces:\n",
                "\n",
                "- [x] 🐍 MIME Types. Python ships with `nteract`.\n",
                "  - [x] LaTeX. Supported by customn renderers.\n",
                "  - [x] `application/json`.\n",
                "  - [x] `image/*`. We render PNG and JEPG in core but svg and gifs should be rendered in webview/iframe. (GIF support is a must as we don't support videos).\n",
                "  - [x] `application/scripts`.\n",
                "- Metadata\n",
                "  - [ ] Cell metadata which controls the renderings of cells, like `editable`, `execution_count`\n",
                "  - [ ] Output metadata, like whether the output should be rendered in an isolated context.\n",
                "  - [ ] Metadata **editing**\n",
                "- Comm between output and extension\n",
                "  - [ ] 2️⃣  API for notebook extensions to talk to the output rendered in webview\n",
                "\n",
                "Fulls lists of cell types, output mime types and metadata we may want to support are listed below.\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "## Cell\n",
                "\n",
                "- Code Cell\n",
                "  - [x] Render source code in regular editor\n",
                "  - [x] 🐍 1️⃣ Editor height grow with content.\n",
                "    - Note: make sure word wrapping and folding (one example for content widgets) work properly.\n",
                "  - [ ] View output in seperate view. For example view output in fullscreen (contributed by notebook extensions)\n",
                "- Markdown Cell\n",
                "  - [x] Live Preview\n",
                "    - [x] Editor/Preview splited vertically\n",
                "    - [ ] Side by side. Ref [Google Colab](https://colab.research.google.com/drive/16RMW8h7h2zcfuamLkhWucf2ooo82yky1#scrollTo=0h0u01uSB4nT). (The preview can be in an another editor group, similar to markdown preview to the side)\n",
                "  - [ ] Markdown engine\n",
                "    - [x] Commonmark\n",
                "    - [x] GFM\n",
                "    - [ ] 🐍 1️⃣ LaTeX\n",
                "    - [ ] HTML (we have HTML support with marked.js but not full HTML)\n",
                "    - [ ] Video. Note that VS Code doesn't ship with ffmpeg\n",
                "    - [ ] Attachments. References to local resources.\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "## Output \n",
                "\n",
                "- [x] Now we delegate custom mimetype rendering to extensions so we will only implement limited mime type rendering in the core.\n",
                "\n",
                "🐍 Python supports all Mimetypes which are supported by `nteract`.\n",
                "Refs:\n",
                "\n",
                "* [jupyterlab](https://jupyterlab.readthedocs.io/en/stable/user/file_formats.html)\n",
                "* [ipython](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.formatters.html?highlight=text%2Flatex#IPython.core.formatters.DisplayFormatter)\n",
                "\n",
                "The following MIME types are usually implemented by Notebook client. We track all mime types we support with [test files](https://github.com/rebornix/notebook-test/blob/master/samples/mimetypes.ipynb).\n",
                "\n",
                "- Stream\n",
                "  - [x] Text\n",
                "  - [x] Error/ANSI\n",
                "- Display Data\n",
                "  - [x] text/plain\n",
                "  - [x] text/markdown\n",
                "  - [x] text/html\n",
                "  - [x] image/png\n",
                "  - [x] image/jpeg\n",
                "  - [x] 1️⃣ image/svg+xml\n",
                "  - [x] 1️⃣ application/javascript\n",
                "  - [x] 1️⃣ application/json\n",
                "\n",
                "Contributed by extensions\n",
                "\n",
                "  - [x] Interactive JavaScript wigets. Contributed by extensions, like ipywidget or vega/vega-lite\n",
                "  - [ ] 1️⃣ text/latex\n",
                "  - [ ] 1️⃣ image/gif\n",
                "  - [ ] 1️⃣ image/bmp\n",
                "  - [ ] application/pdf\n",
                "  - [ ] **Custom Mime Types** (`application/vnd*`). Similar to above, might require API extensibility for contributing custom vendor mime types handler\n",
                "\n",
                "Notebook outputs might have mutiple mimetypes and notebook clients will choose the richest mime type they can render. The display order for mime types can be as below\n",
                "\n",
                "* application/json\n",
                "* application/javascript\n",
                "* text/html\n",
                "* image/svg+xml\n",
                "* text/markdown\n",
                "* text/latex\n",
                "* image/svg+xml\n",
                "* image/gif\n",
                "* image/png\n",
                "* image/jpeg\n",
                "* application/pdf\n",
                "* text/plain\n",
                "\n",
                "Internally a notebook client should maintain two lists, one for mime types it can render and one for the display order.\n",
                "\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "## Cell metadata\n",
                "\n",
                "Cell metadata is used to control the rendering of a cell, for example we can disable editing of markdown cells by setting `editable` to `false`.\n",
                "\n",
                "Refs: [nbformat](https://nbformat.readthedocs.io/en/latest/format_description.html#cell-metadata)\n",
                "\n",
                "- [ ] 1️⃣ editable/readonly\n",
                "- [ ] 1️⃣ deletable\n",
                "- [ ] 1️⃣ collapsed\n",
                "- [ ] scrolled\n",
                "- [ ] name\n",
                "- [ ] tags\n",
                "- [ ] format\n",
                "- [ ] jupyter (namespace)\n",
                "  - source_hidden\n",
                "  - outputs_hidden\n",
                "- [ ] execution (namespace)\n",
                "\n",
                "## Output metadata\n",
                "\n",
                "Output metadata can provide information of how to render an output.\n",
                "\n",
                "Refs: [nbformat](https://nbformat.readthedocs.io/en/latest/format_description.html#output-metadata)\n",
                "\n",
                "- [ ] 1️⃣ isolated. isolated output should be isolated into an iframe\n",
                "- [ ] 1️⃣ dimensions. `\"metadata\" : { \"image/png\": { \"width\": 640, \"height\": 480, } }`\n",
                "- [ ] needs_background. `light/dark`\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Editor Features\n",
                "\n",
                "Notebooks contain text (markdown) and source code, so users would expect [code editing features](https://code.visualstudio.com/docs/editor/codebasics) from the notebook editor. \n",
                "\n",
                "- View/Model States\n",
                "  - [x] 1️⃣ Save\n",
                "  - [ ] 🐍 1️⃣ SaveAs\n",
                "  - [ ] 🐍 1️⃣ Auto Save\n",
                "  - [ ] 🐍 Hot Exit\n",
                "  - [ ] View States\n",
                "    - [ ] 1️⃣ Cursor/Selection. Cursor states should be tracked when users scroll the notebook (our virtualization optimization leads to state loss now).\n",
                "    - [ ] 1️⃣ Move editor across editor groups and persist states\n",
                "- Basic Editing ([ref](https://code.visualstudio.com/docs/editor/codebasics))\n",
                "  - [ ] 🐍 1️⃣ Find in File. Users might want to F&R in editable markdown cells too.\n",
                "  - [ ] 1️⃣ Replace in File.\n",
                "  - [ ] Find & Replace in Cell \n",
                "  - [ ] Folding\n",
                "  - [ ] Multi cursor across cells\n",
                "  - [ ] Undo/Redo across cells. Cell operations should be saved to Undo stack as well.\n",
                "  - [ ] 1️⃣ Cursor movement across cells.\n",
                "  - [ ] Minimap for the whole notebook document\n",
                "  - [ ] Snippets. Code snippets and cell snippets\n",
                "  - [ ] Diff. \n",
                "    - [ ] Text based diff. This one might require API works as notebook documents might not be persistent on disk.\n",
                "    - [ ] Rich Diff. Similar to NBViewer, the core needs to diff and render rich diff.\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Cell Manipulation\n",
                "\n",
                "Currently we put all cell related actions in the context menu but it's not easily accessibable, we may want to have a better UX to ensure users can be **productive**.\n",
                "\n",
                "Refs: [jupyterlab api for cell management](https://jupyterlab.readthedocs.io/en/stable/developer/notebook.html)\n",
                "\n",
                "- UX polish\n",
                "  - [ ] 🐍 Run code cell, Edit\n",
                "  - [ ] 🐍 Toolbar for other actions\n",
                "- Commands\n",
                "  - [x] Create new markdown cell\n",
                "  - [x] Create new code cell\n",
                "  - [ ] 🐍 1️⃣ Move Cell\n",
                "  - [x] Delete Cell\n",
                "  - [ ] Drag and Drop. Supported in JupyterLab\n",
                "  - [ ] Expand/Collapse outputs\n",
                "  - [ ] 🐍 1️⃣ Undo/Redo cell manipulation\n",
                "  - [x] Execute code cell\n",
                "  - [ ] Cell selection\n",
                "  - [ ] 1️⃣ Clear output\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "## Commands/Keybindings\n",
                "\n",
                "Jupyter notebook web app has two modes: Command mode and Edit mode, which is similar to Vim's mode system. In Command mode, users can use keybindings to manipulate cells while Edit mode is used to edit content.\n",
                "\n",
                "Keybindings in Edit mode are pretty close to VS Code's builtin keybindings however Command mode has many conflicts with the core. We are going to implement the commands and users (or we) can build keymaps on top of them.\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "### Command Mode\n",
                "\n",
                "Command Mode (press Esc to enable)\n",
                "\n",
                "| Shortcut | Command | Status | Python | \n",
                "| --- | --- | --- | --- | \n",
                "| F | find and replace | | |\n",
                "| ↩ | enter edit mode | | |\n",
                "| ⌘⇧F | open the command palette | | |\n",
                "| ⌘⇧P | open the command palette | ✔️ | |\n",
                "| P | open the command palette | | |\n",
                "| ⇧↩ | run cell, select below | 1️⃣  | |\n",
                "| ⌃↩ | run selected cells | 1️⃣ | |\n",
                "| ⌥↩ | run cell and insert below | 1️⃣ | |\n",
                "| Y | change cell to code | 1️⃣ | |\n",
                "| M | change cell to markdown | 1️⃣ | |\n",
                "| R | change cell to raw | | |\n",
                "| 1 | change cell to heading 1 | | |\n",
                "| 2 | change cell to heading 2 | | |\n",
                "| 3 | change cell to heading 3 | | |\n",
                "| 4 | change cell to heading 4 | | |\n",
                "| 5 | change cell to heading 5 | | |\n",
                "| 6 | change cell to heading 6 | | |\n",
                "| K | select cell above | | |\n",
                "| ↑ | select cell above | 1️⃣ | |\n",
                "| ↓ | select cell below | 1️⃣ | |\n",
                "| J | select cell below | | |\n",
                "| ⇧K | extend selected cells above | | |\n",
                "| ⇧↑ | extend selected cells above | | |\n",
                "| ⇧↓ | extend selected cells below | | |\n",
                "| ⇧J | extend selected cells below | | |\n",
                "| A | insert cell above | 1️⃣ | |\n",
                "| B | insert cell below | 1️⃣ | |\n",
                "| X | cut selected cells | 1️⃣ | |\n",
                "| C | copy selected cells | 1️⃣ | |\n",
                "| ⇧V | paste cells above | 1️⃣ | |\n",
                "| V | paste cells below | 1️⃣ | |\n",
                "| Z | undo cell deletion | 1️⃣ | |\n",
                "| D,D | delete selected cells | | |\n",
                "| ⇧M | merge selected cells, or current cell with cell below if only one cell is selected | | |\n",
                "| ⌘S | Save and Checkpoint | | |\n",
                "| S | Save and Checkpoint | 1️⃣ | |\n",
                "| L | toggle line numbers | | |\n",
                "| O | toggle output of selected cells | | |\n",
                "| ⇧O | toggle output scrolling of selected cells | | |\n",
                "| H | show keyboard shortcuts | | |\n",
                "| I,I | interrupt the kernel | | |\n",
                "| 0,0 | restart the kernel (with dialog) | | |\n",
                "| Esc | close the pager | | |\n",
                "| Q | close the pager | | |\n",
                "| ⇧L | toggles line numbers in all cells, and persist the setting | | |\n",
                "| ⇧␣ | scroll notebook up | | |\n",
                "| ␣ | scroll notebook down | | |\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "### Edit Mode\n",
                "\n",
                "Edit Mode (press Enter to enable)\n",
                "\n",
                "| Shortcut | Command | Status | Python |\n",
                "| --- | --- | --- | --- |\n",
                "| ⇥ | code completion or indent | ✔️  | |\n",
                "| ⇧⇥ | tooltip | | |\n",
                "| ⌘] | indent | ✔️  | |\n",
                "| ⌘[ | dedent | ✔️ | |\n",
                "| ⌘A | select all | ✔️ | |\n",
                "| ⌘Z | undo | ✔️ | |\n",
                "| ⌘/ | comment | ✔️ | |\n",
                "| ⌘D | delete whole line | | |\n",
                "| ⌘U | undo selection | ✔️ | |\n",
                "| Insert | toggle overwrite flag | ✔️ | |\n",
                "| ⌘↑ | go to cell start | ✔️ | |\n",
                "| ⌘↓ | go to cell end | ✔️ | |\n",
                "| ⌥← | go one word left | ✔️ | |\n",
                "| ⌥→ | go one word right | ✔️ | |\n",
                "| ⌥⌫ | delete word before | ✔️ | |\n",
                "| ⌥⌦ | delete word after | ✔️ | |\n",
                "| ⌘⇧Z | redo | ✔️ | |\n",
                "| ⌘⇧U | redo selection | | |\n",
                "| ⌃K | emacs-style line kill | ✔️ | |\n",
                "| ⌘⌫ | delete line left of cursor | ✔️ | |\n",
                "| ⌘⌦ | delete line right of cursor | ✔️ | |\n",
                "| ⌃M | enter command mode | | |\n",
                "| Esc | enter command mode | | |\n",
                "| ⌘⇧F | open the command palette | | |\n",
                "| ⌘⇧P | open the command palette | ✔️ | |\n",
                "| ⇧↩ | run cell, select below | | |\n",
                "| ⌃↩ | run selected cells | | |\n",
                "| ⌥↩ | run cell and insert below | | |\n",
                "| ⌃⇧Minus | split cell at cursor | | |\n",
                "| ⌘S | Save and Checkpoint | | |\n",
                "| ↓ | move cursor down | | |\n",
                "| ↑ | move cursor up | | |\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Accessibility\n",
                "\n",
                "Classic Jupyter Notebook is lack of good accessibility support for code editing and navigations. While building the native notebook editor in core, we should make sure it's equally accessible as the text editor.\n",
                "\n",
                "Users from community already provides concrete examples of accessibility issues in classic Jupyter Notebook.\n",
                "\n",
                "Refs: \n",
                "  * https://github.com/jupyter/notebook/issues/1801\n",
                "  * https://github.com/jupyterlab/jupyterlab/issues/4878"
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Language Features\n",
                "\n",
                "Source code in code cells in a notebook are loosely coupled. You can import a module in one code cell and then use it directly in another code cell, however it doesn't mean that the `import` cell has to be layed before the other cell. As long as the `import` cell is executed first, the other cell is valid.\n",
                "\n",
                "When the notebook is not connected to a kernel or the execution sequence is unknown (which requires users' input), the language service for code cells needs to be fuzzy. The language service for the notebook may want to analyze all code cells and provide hints/suggestions based on heuristics.\n",
                "\n",
                "Language features can fall into following groups by their requirements\n",
                "\n",
                "## Definiton/Suggestion\n",
                "\n",
                "To support **Completions**, **Parameter hints**, **Hover** and **Diagnostics**, language service will read live content from every code cell\n",
                "\n",
                "- [ ] API: Expose code cells contents and event listeners for content change\n",
                "- [ ] Extension register themselves as language features providers\n",
                "\n",
                "🐍 Completions, Parameter Hints, Hover. Code cells are concatenated from top to bottom. \n",
                "\n",
                "## Navigation\n",
                "\n",
                "Code navigation includes **Go to Definition/Implmenation**, **References/Peek View** and **Call hierarchy**. The challenge here is jumping between code cells in a notebook, instead of opening a regular code editor.\n",
                "\n",
                "- [ ] Core: Support opening editor/models from a notebook document\n",
                "\n",
                "## Edits across cells\n",
                "\n",
                "Features like **Refactoring** and **Rename** might require Workspace-like Edits.\n",
                "\n",
                "- [ ] Core & API: Support edits for multiple code cells\n",
                "\n",
                "\n",
                "## Embedded model support\n",
                "\n",
                "**Formatters** and **Symbols** can work seamlessly on individual code cells but we may also want a Notebook level (between normal document and workspace) formaters or symbols provider\n",
                "\n",
                "- [ ] Format notebook (format cells with different languages)\n",
                "- [ ] Notebook symbols (combination of symbols from different cells)\n",
                "\n",
                "\n",
                "## Cell agnostic\n",
                "\n",
                "Following features can be cell agnostic and we need to make sure the code cells are labeled as the right language\n",
                "\n",
                "- [ ] Syntax highlighting\n",
                "- [ ] Link detection\n",
                "- [ ] Color\n",
                "- [ ] Folding\n",
                "- [ ] Codelens\n",
                "\n",
                "\n",
                "Refs:\n",
                "\n",
                "* [Google Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb#recent=true)\n",
                "\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# Debug\n",
                "\n",
                "In Jupyter Notebook, you can run code cell line by line and the kernel behaves like a debugger. In theory, debugging a code cell is similar to debugging a file. The notebook provider provides content from the active code cell in current notebook document, and initializes the debugging session with the kernel. The provider is also the debug adapter in this case.\n",
                "\n",
                "We need to validate if this can work and investigate how debugging should work when the kernel is not connected.\n",
                "\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        },
        {
            "source": [
                "# MISC\n",
                "\n",
                "Bugs / Polish items\n",
                "\n",
                "- [x] Notebook file not loaded from extension when opened the second time\n",
                "- [x] Activation events for notebook provider. Activate extension when a notebook file is opened.\n",
                "- [ ] Editing in nested code editor, cursor should always be visible.\n",
                "- [x] Vim Escape Key \n",
                "  * Currently workaround the issue by adding keybinding\n",
                "  ```json\n",
                "  {\n",
                "        \"key\": \"escape\",\n",
                "        \"command\": \"extension.vim_escape\",\n",
                "        \"when\": \"editorTextFocus && vim.active && !inDebugRepl && vim.mode != 'Normal'\"\n",
                "    },\n",
                "    {\n",
                "        \"key\": \"escape\",\n",
                "        \"command\": \"-extension.vim_escape\",\n",
                "        \"when\": \"editorTextFocus && vim.active && !inDebugRepl\"\n",
                "    }\n",
                "  ```\n",
                "- [x] Python extension activation slows down notebook file opening. (Mitigated when text models in code cells are lazily resolved.)\n",
                "- [ ] Saving: `activeNotebookDocument` is `undefined` if there is no active code cell being edited. `ExtHostNotebookController._documents` is empty when that happens.\n",
                "- [ ] Markdown preview and text editor somethings out of sync\n",
                ""
            ],
            "metadata": {
                "language_info": {
                    "name": "python"
                }
            },
            "cell_type": "markdown"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.3"
        },
        "nteract": {
            "version": "0.15.0"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 0
}