Learning Interpretability Tool Release Notes

Release 1.3.1

This is a minor update to fix issues with running the LIT Gemma Colab.

Release 1.3

This release updates how the Learning Interpretability Tool (LIT) can be deployed on Google Cloud. You can now use LIT to interpret foundation models—including Gemini, Gemma, Llama, and Mistral—using LIT's prompt debugging workflows. LIT now provides public container images to make it easier to deploy on your hosting platform of choice, with an updated tutorial for deploying LIT with Cloud Run.

New Stuff

LIT on GCP - 1075325, 1acc868, 55bfc99, 180f68a, 64114d5, 2488aa7, 9baac29, 60bdc7c, 7681476, 4c81182, 4e5e8e2, b9a0b82, 424adce, 1d019c7, f4436a2,

Non-breaking Changes, Bug Fixes, and Enhancements

Upgrade LIT to MobX v6. - c1f5055
Fix indexing issue in Sequence Salience module. - 58b1d2
Load multiple model wrappers with shared model. - ba4d975
Add the custom model and dataset loaders to prompt debugging notebook. - 338c6b
Convert hosted demos images to multi-stage builds. - 4bf1f8
Adding testing instructions to README. - f24b841
More LIT documentation updates. - 2e9d267

Release 1.2

This release covers clean-ups on various obsolete demos, as well as improved packaging and isolated dependencies on the GLUE, Penguin, Prompt Debugging with Sequence Salience and TyDi demos for easier launch.

New Stuff

Improved packaging and instructions for launching Prompt Debugging with Sequence Salience demo, as well as minor bug fixes - 08289df, 675ca2d, 15eccb1, e0e35c3, c7970fb, cee3b58
Clean up of obsolete demos - b16059f, f4c0990, 6aa2eb6, c2fb41b, dd196e9, 72fd772, 71d88fb, aa49340, fc7b0d0, 2475b3b, a59641c, 1ed82d4, 7d5ef58, 992823b, 3dad2b0, 0656386, 27d7a84, 8863019, 71cbdba, 416d573
Python requirements update and isolated setup for individual demos - bcc481e, bb29f43, fbd8874, b3c120b, 5188c8c, 5639e3b
Documentation cleanup and updates - afd51fe, 7dda659, 79ada6e, 1c8d6a0, 2e9d267

Non-breaking Changes, Bug Fixes, and Enhancements

Refactor DataService reactions - 483082d
Add warm_start option to LitWidget - a5265a4
Pretty-printing of Model objects - 4fb3bde
Avoid equivalent shuffles in Scrambler - 0d8c0d9
Updated gunicorn config for demos running in Docker - b14e3b1
Disable embeddings for TyDi - 7ff377f
Cast embeddings to float32 before computing distances - 5456011
Update colab examples to include installation of the lit-nlp package - 48b029c

Release 1.1.1

This release covers various improvements for sequence salience, including new features in the UI module, support of more LLMs, and detailed tutorial and documentation on how to use the sequence salience module for prompt engineering.

New stuff

New features in the sequence salience UI module - 62f18b2, f0417c9, fe5a705, 1ec8626, 15184a1, 84af141, 27cafd8, 3591e61, d108b59, 309c4f2, 99821d3, c8ee224
Support of more models (GPT2, Gemma, Llama, Mistral) on deep learning frameworks (Tensorflow, Pytorch) for Keras and Hugging Face - b26256a, 45887d3, b9941ed, 5ee7064, 8ea325b
A tutorial to use sequence salience at our website and documentation updates - 962faaa, 96eff29, f731e6d, f4d7cac, 49e7736

Non-breaking Changes, Bug Fixes, and Enhancements

Py typing fix - d70e3d3
Improvements on the curves UI module - 3d61a09 2efe62b
Support model-column search in LIT Data Table - 525bf5e
Obsolete code cleanup - 82abec6

Release 1.1

This release provides the capabilities to interpret and debug the behaviors of Generative AI models in LIT. Specifically, we added sequence salience, which explains the impact of the preceding tokens on the generated tokens produced by the GenAI models. Major changes include:

An LM salience module in the LIT UI that computes generations, tokenization, and sequence salience on-demand;
Computation of sequence salience at different granularities, from the smallest possible level of tokens, to more interpretable larger spans, such as words, sentences, lines, or paragraphs.
Support of OSS modeling frameworks, including KerasNLP and Hugging Face Transformers for sequence salience computation. This release would not have been possible without the work of our contributors. Many thanks to: Ryan Mullins, Ian Tenney, Bin Du, and Cibi Arjun.

New Stuff

LM salience module in the LIT UI - ab294bd 5cffc4d 40bb57a d3980cc 406fbc7 77583e7 a758f98
Sequence salience for decoder-only LM, with support for GPT-2 and KerasNLP - 27e6901 80cf699 1df3ba8 b6ab352 c97a710
Prompt examples for sequence salience - 4f19891 000c844 34aa110 ca032ff

Non-breaking Changes, Bug Fixes, and Enhancements

Improvements to display various fields and their default ranges - 8a3f366 e63b674 d274508
Allow only displaying the UI layouts provided by users - a219863
Internal dependency changes - f254fa8 724bdee 2138bd9
Fix issues with adding more than one example from counterfactual generators - d4302bd
Fix issues with loading SimpleSentimentModel - ac8ed59
Notebook widget improvements - cdf79eb
Docs updates

Release 1.0

This is a major release, covering many new features and API changes from the dev branch since the v0.5 release over 8 months ago. This release includes a variety of breaking changes meant to simplify various aspects of the LIT API and visual changes to improve usability. This release includes over 250 commits. Major changes include:

Refactored python code to remove _with_metadata methods from all component and model classes.
Refactored Model and BatchedModel python classes to remove predict_minibatch method.
Reworked UI and backend logic for dynamic loading of new datasets and models from the UI. This makes use of the new init_spec methods for datasets and model classes.
- Added a blank demo with no models or datasets preloaded which allows for dynamic loading of models and datasets through the UI.
Refactored to upgrade metrics calculation from a type of interpreter to its own top-level concept.
Updated front-end layout code to default to a new layout that includes a full height side-panel on the left side to complement the existing top and bottom panels, providing for more customization of module layouts.
Added automatic metrics calculations for multilabel models.
Added target selector dropdown for saliency methods.
A visual redesign of the Salience Clustering module.
Improved searching capabilities in the Data Table module.
Improved the Data Table module's display of long strings through a "Show more" capability.
Updated to Python 3.10.
Updated to Node 18 and Typescript 5.0.
Improved documentation pages, now at https://pair-code.github.io/lit/documentation/

This release would not have been possible without the work of our new contributors in 2023. Many thanks to Minsuk Kahng, Nada Hussein, Oscar Wahltinez, Bin Du, and Cibi Arjun for your support and contributions to this project! A full list of contributors to this repo can be found at https://github.com/PAIR-code/lit/graphs/contributors.

Breaking Changes

Adds init_spec() capability to models and datasets for dynamic loading - d28eec3, d624562, 7bb60b2, f74798a, db51d9d, f3b0d6e, 0f133cf, 9eebe57, bcc6c09, 99e78ff
Simplify Model spec code - 16b72f7
Promote Metrics to top-level property of LitMetadata - f019279, 6ba1db8, c1777ea
Remove _with_metadata and batched methods from models and components - cb4f6b0, e020faa, e9ce692, 5f1a971, 061973a, ad65fd9, bc6f82b, 7888c66, 9767670, 0ec1527, e30e59a, b29d1f3, 5ed93bd, 5047bdd, a15cc88, 0146d5f, 50fc3a4, 6fdcbfe, ce38565
Simplifications and refactors in layout system - 4a5c0cb, 5f6a46a, 2551c2c, cc7bfd5, fb2467d
Update LIT to Node 18 and TypeScript 5.0 - 7b96a6d
Update LIT to Python 3.10 - 8bce86a

New Stuff

Add three-panel layout configuration option - a95ed67
Add output embeddings and attention options to GlueConfig - 6e0df41
Allow downloading/copying data from the slice editor - 57fac3a
Use new custom tooltip elemement in various places - d409900, bd0f7fc, 6c25619, 7d30408, 6779a4b, 9179c73
Add multi-label metrics to LIT - c0e3663
Improved UI for dynamic loading of models and datasets - abc8d1a, b7ce560
Replace conda installation instructions with pip de23ceb
Add a new Blank demo for dynamic loading of models and datasets - 22b0dea
Add target-selector dropdowns to salience map module - 4c9a7ec, 10926ea, f635ea7, fe121ca, 8cb965a

Non-breaking Changes, Bug Fixes, and Enhancements

Fixes Scalars Module resize bug affecting datasets with scalar fields - 453461a
Moves Model-Dataset compatibility checks to Model class instead of ModelSpec - c268ce4
Updates to the Salience Clustering module - 3a3aad3, 7d3f235, ff759ad, 20ec052
Data Table module improvements - c7fa619, 7301b28, dd23083, 42d189a, 1cc6964, ab7da61, ee54333, d74f2d6, 35487fa, ea25e75, 8c4bf1f, ddf8e52
Various styling fixes, bug fixes, and code cleanup efforts
Docs, FAQ, and README updates

Release 0.5

This is a major release, covering many new features from the dev branch since the v0.4 release nearly 11 months ago. Most notably, we're renaming! It's still LIT, but now the L stands for "Learning" instead of "Language", to better reflect the scope of LIT and support for non-text modalities like images and tabular data. Additionally, we've made lots of improvements, including:

New modules including salience clustering, tabular feature attribution, and a new Dive module for data exploration (inspired by our prior work on Facets Dive).
New demos and tutorials for input salience comparison and tabular feature attribution.
Many UI improvements, with better consistency across modules and shared functionality for colors, slicing, and faceting of data.
Better performance on large datasets (up to 100k examples), as well as improvements to the type system and new validation routines (--validate) for models and datasets.
Download data as CSV directly from tables in the UI, and in notebook mode access selected examples directly from Python.
Update to Python 3.9 and TypeScript 4.7.

This release would not have been possible without the work of many new contributors in 2022. Many thanks to Crystal Qian, Shane Wong, Anjishnu Mukherjee, Aryan Chaurasia, Animesh Okhade, Daniel Levenson, Danila Sinopalnikov, Deepak Ramachandran, Rebecca Chen, Sebastian Ebert, and Yilei Yang for your support and contributions to this project!

Breaking Changes

Upgraded to Python 3.9 – 17bfabd
Upgraded to Typescript 4.7 – 10e2548
Layout definitions moved to Python – 05824c8, d3d19d2, 2994d7e, b78c962, 0eacdd0
Moving classification and regression results to Interpreters – 2b4e622, bcdbb80, dad8edb
Use a Pinning construct instead of comparison mode – 05bfc90, d7bdc65, 6a4ca00, 0fe3c79, 5b2b737
Parallel, class-based Specs and LitTypes in Python and TypeScript code
- Prep work – db1ef3d, c85e556, 660b8ef, db58fa4, c020d25, eb02465, 72edd26, 65c5b8a, abb8889, 4c93b62, 40d14e5, 9ec5324, 40a661e
- Breaking changes to front-end typing infrastructure – 8c6ac11, 2522e4f, 0f8ff8e, 58970de, ef72bfc, ccbb72c, a5b9f65, ab1e06a, 853edd0, cb528f1, a36a936, 74b5dbb, e811359
- Build fixes – 948adb3
Minimizing duplication in modules
- Classification Results – 4f2b53d
- Scalars, including its migration to Megaplot – 353b96e, ed07199, 184c8c6, 14f82d5, 764674a
Changes to component is_compatible() signature
- Added checks to some generators – 9b2de92, db94849
- Added Dataset parameter to all checks – ecd3a66
Adds core components library to encapsulate default interpreters, generators, and metrics – 9ea4ab2
Removed the Color module – b18d887
Removed the Slice module – 7db22ae
Moved star button to Data Table module – cd14f35
Salience Maps now inside of expansion panels with popup controls – 1994425
Metrics
- Promotion to a major component type – de7d8ba
- Improved compatibility checks – 0d8341d

New Stuff

Common Color Legend element – f846772, 7a1e26a, 0cc934c
Common Expansion Panel element – 2d67ce
Common Faceting Control – 0f46e16, b109f9b, 8993f9b, 670abeb
Common Popup element – 1994425, cca3511
A new Dive module for exploring your data – 155e0c4, 1d17ca2, a0da9cf
Copy or download data from Table elements – d23ecfc
Training Data Attribution module – 5ff9102, c7398f8
Tabular Feature Attribution module with a heatmap mode and SHAP interpreter – 45e526c, 76379ad
Salience Clustering module – 8f3c26c, fb795e8, 49faa00, e35d8d8, 7505861, f970958
Selection state syncing in Python notebooks – 08abc2c, 06613b9
Unified DataService – 9bdc23e, 00749fc
AUC ROC and AUC PR Curve interpreters and module – 51842ba, 0f9fd4d, 0558ef5, 4efd58e
Splash screen documentation – 1f09ae9, cfabe78, aca35d8
Added a GeneratedURL type that displays in the Generated Text module – bb06368
Added new built-in ROUGE and Exact Match metrics – 6773927, eac9382
Input Salience demo – a98edce, 75ff835, 55579de
Model and Dataset validation – 0fef77a
Tutorials written by our Google Summer of Code contributor, Anjishnu Mukherjee
- Using LIT for Tabular Feature Attribution – 2c0703d
- Making Sense of Salience Maps – 4159527

Non-breaking Changes, Bug Fixes, and Enhancements

Added Dataset embeddings to Embeddings projector – 78e2e9c, 3c0929f, e7ac98b
Added a “sparse” mode to Classification Results – 20a8f31
Added “Show only generated” option to Data Table module – 4851c9d
Added threshold property for MulticlassPreds that allows for default threshold values other than 0.5 – 5e91b19
Added toggle for module duplication direction – 4e05a75
Clickable links in the Generated Images module – 8cf8119
Constructor parameters for salience interpreters – ab057b5
Image upload in Datapoint Editor – a23b146
Markdown support in LIT component descriptions – 0eaa00c
Selection updates based on interactions in Metrics module – c3b6a0c
Support for many. new types of inputs in the Datapoint editor, including GeneratedText, GeneratedTextCandidates, MultiSegmentAnnotation, Tokens, SparseMultilabel, and SparseMultilabelPreds
Various styling fixes and code cleanup efforts
Docs, FAQ, and README updates

Release 0.4.1

This is a bug fix release aimed at improving visual clarity and common workflows.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

Adds "open in new tab" feature to LIT Notebook widget
Adds support for SparseMultilabelPreds to LIME
Improves color consistency across the UI
Switching NumPy instead of SciKit Learn for PCA
Ensuring all built-in demos are compatible with the Docker
Updating the Dockerfile to support run-time DEMO_NAME and DEMO_PORT args
Fixed a rendering bug in the Confusion Matrix related column and row spans when "hide empty labels" is turned on

Release 0.4

This release adds a lot of new features. The website and documentation have been updated accordingly.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

Support for Google Cloud Vertex AI notebooks.
Preliminary support for tabular and image data, in addition to NLP models.
Addition of TCAV global interpretability method.
New counterfactual generators for ablating or flipping text tokens for minimal changes to flip predictions.
New counterfactual generator for tabular data for minimal changes to flip predictions.
Partial dependence plots for tabular input features.
Ability to set binary classification thresholds separately for different facets of the dataset
Controls to find optimal thresholds across facets given different fairness constraints, such as demographic parity or equal opportunity.

Release 0.3

This release adds the ability to use LIT directly in colab and jupyter notebooks. The website and documentation have been updated accordingly.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added. Notable changes include:

Notebook mode added.
New annotated text visualization module added.
Allow saving/loading of generated datapoints, and dynamic adding of new datasets by path in the UI.
Added synchronized scrolling between duplicated modules when comparing datapoints or models.
Added a focus service for visually linking focus (i.e. hover) states between components.
Allow layouts to be specified on LIT creation in python.

Release 0.2

This release of LIT coincides with the EMNLP 2020 conference, where the LIT paper was presented, and the publication of the LIT website, including tutorials and hosted demos.

The UI has been slightly revamped, bugs have been fixed, and new capabilities have been added.

Release 0.1.1

This release of LIT adds a pip package for easy installation, cleans up some of the code and documentation, and adds more examples.

Release 0.1

This is the initial release of LIT.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!