EBOOK

A Compact Guide to
Large Language Models
A Compact Guide to Large Language Models 2

S ECT I O N 1

Introduction

Definition of large language models (LLMs)

Large language models are AI systems that are designed to process and analyze
vast amounts of natural language data and then use that information to generate
responses to user prompts. These systems are trained on massive data sets
using advanced machine learning algorithms to learn the patterns and structures
of human language, and are capable of generating natural language responses to
a wide range of written inputs. Large language models are becoming increasingly
important in a variety of applications such as natural language processing,
machine translation, code and text generation, and more.

While this guide will focus on language models, it’s important to understand that
they are only one aspect under a larger generative AI umbrella. Other noteworthy
generative AI implementations include projects such as art generation from text,
audio and video generation, and certainly more to come in the near future.
A Compact Guide to Large Language Models 3

Extremely brief historical background and development of LLMs

1950s–1990s 2018
Initial attempts are made to map hard rules around languages and Google introduces BERT (Bidirectional Encoder Representations
follow logical steps to accomplish tasks like translating a sentence from Transformers), which is a big leap in architecture and paves
from one language to another. the way for future large language models.

While this works sometimes, strictly defined rules only work for
2020
concrete, well-defined tasks that the system has knowledge about.
OpenAI releases GPT-3, which becomes the largest model at
175B parameters and sets a new performance benchmark for
1990s language-related tasks.
Language models begin evolving into statistical models and
language patterns start being analyzed, but larger-scale projects
2022
are limited by computing power.
ChatGPT is launched, which turns GPT-3 and similar models into
a service that is widely accessible to users through a web interface
2000s and kicks off a huge increase in public awareness of LLMs and
Advancements in machine learning increase the complexity of generative AI.
language models, and the wide adoption of the internet sees an
enormous increase in available training data.
2023
Open source LLMs begin showing increasingly impressive results
2012 with releases such as Dolly 2.0, LLaMA, Alpaca and Vicuna.
Advancements in deep learning architectures and larger data sets GPT-4 is also released, setting a new benchmark for both parameter
lead to the development of GPT (Generative Pre-trained Transformer). size and performance.
A Compact Guide to Large Language Models 4

S ECT I O N 2

Understanding Large Language Models

What are language models and how do they work? INCREASED ACCESSIBILITY
The release of ChatGPT opened the door for anyone with internet access
Large language models are advanced artificial intelligence systems that take to interact with one of the most advanced LLMs through a simple web
some input and generate humanlike text as a response. They work by first interface. This brought the impressive advancements of LLMs into the
analyzing vast amounts of data and creating an internal structure that models spotlight, since previously these more powerful LLMs were only available
the natural language data sets that they’re trained on. Once this internal to researchers with large amounts of resources and those with very deep
structure has been developed, the models can then take input in the form of technical knowledge.
natural language and approximate a good response.
G R O W I N G C O M P U TAT I O N A L P O W E R
The availability of more powerful computing resources, such as graphics
If they’ve been around for so many years, why are they just now processing units (GPUs), and better data processing techniques allowed
making headlines? researchers to train much larger models, improving the performance of
these language models.
A few recent advancements have really brought the spotlight to generative AI
and large language models: I M P R O V E D T R A I N I N G DATA
As we get better at collecting and analyzing large amounts of data, the
A D VA N C E M E N T S I N T E C H N I Q U E S model performance has improved dramatically. In fact, Databricks showed
Over the past few years, there have been significant advancements in the that you can get amazing results training a relatively small model with a
techniques used to train these models, resulting in big leaps in performance. high-quality data set with Dolly 2.0 (and we released the data set as well
Notably, one of the largest jumps in performance has come from integrating with the databricks-dolly-15k data set).
human feedback directly into the training process.
A Compact Guide to Large Language Models 5

So what are organizations using large language models for? L A N G U AG E T R A N S L AT I O N

Globalize all your content without hours of painstaking work by simply
Here are just a few examples of common use cases for large language models: feeding your web pages through the proper LLMs and translating them to
different languages. As more LLMs are trained in other languages, quality
C H AT B O T S A N D V I R T U A L A S S I S TA N T S and availability will continue to improve.
One of the most common implementations, LLMs can be used by
organizations to provide help with things like customer support, S U M M A R I Z AT I O N A N D PA R A P H R A S I N G
troubleshooting, or even having open-ended conversations with user- Entire customer calls or meetings could be efficiently summarized so that
provided prompts. others can more easily digest the content. LLMs can take large amounts of
text and boil it down to just the most important bytes.
C O D E G E N E R AT I O N A N D D E B U G G I N G
LLMs can be trained on large amounts of code examples and give C O N T E N T G E N E R AT I O N
useful code snippets as a response to a request written in natural language. Start with a detailed prompt and have an LLM develop an outline for you.
With the proper techniques, LLMs can also be built in a way to reference Then continue on with those prompts and LLMs can generate a good first
other relevant data that it may not have been trained with, such as a draft for you to build off. Use them to brainstorm ideas, and ask the LLM
company’s documentation, to help provide more accurate responses. questions to help you draw inspiration from.

S E N T I M E N T A N A LY S I S Note: Most LLMs are not trained to be fact machines. They know how to use
Often a hard task to quantify, LLMs can help take a piece of text and gauge language, but they might not know who won the big sporting event last year.
emotion and opinions. This can help organizations gather the data and It’s always important to fact check and understand the responses before
feedback needed to improve customer satisfaction. using them as a reference.

T E X T C L A S S I F I C AT I O N A N D C L U S T E R I N G
The ability to categorize and sort large volumes of data enables the
identification of common themes and trends, supporting informed
decision-making and more targeted strategies.
A Compact Guide to Large Language Models 6

S ECT I O N 3

Applying Large Language Models

There are a few paths that one can take when looking to apply large language and require you to send your data to their servers in order to interact with their
models for their given use case. Generally speaking, you can break them down language models. This raises privacy and security concerns, and also subjects
into two categories, but there’s some crossover between each. We’ll briefly cover users to “black box” models, whose training and guardrails they have no control
the pros and cons of each and what scenarios fit best for each. over. Also, due to the compute required, these services are not free beyond a
very limited use, so cost becomes a factor in applying these at scale.

Proprietary services In summary: Proprietary services are great to use if you have very complex tasks,
are okay with sharing your data with a third party, and are prepared to incur
As the first widely available LLM powered service, OpenAI’s ChatGPT was the costs if operating at any significant scale.
explosive charge that brought LLMs into the mainstream. ChatGPT provides
a nice user interface (or API) where users can feed prompts to one of many
models (GPT-3.5, GPT-4, and more) and typically get a fast response. These are Open source models
among the highest-performing models, trained on enormous data sets, and are
The other avenue for language models is to go to the open source community,
capable of extremely complex tasks both from a technical standpoint, such as
where there has been similarly explosive growth over the past few years.
code generation, as well as from a creative perspective like writing poetry in a
Communities like Hugging Face gather hundreds of thousands of models
specific style.
from contributors that can help solve tons of specific use cases such as text
The downside of these services is the absolutely enormous amount of compute generation, summarization and classification. The open source community has
required not only to train them (OpenAI has said GPT-4 cost them over $100 been quickly catching up to the performance of the proprietary models, but
million to develop) but also to serve the responses. For this reason, these ultimately still hasn’t matched the performance of something like GPT-4.
extremely large models will likely always be under the control of organizations,
A Compact Guide to Large Language Models 7

It does currently take a little bit more work to grab an open source model and Conclusion and general guidelines
start using it, but progress is moving very quickly to make them more accessible
to users. On Databricks, for example, we’ve made improvements to open source Ultimately, every organization is going to have unique challenges to overcome,
frameworks like MLflow to make it very easy for someone with a bit of Python and there isn’t a one-size-fits-all approach when it comes to LLMs. As the world
experience to pull any Hugging Face transformer model and use it as a Python becomes more data driven, everything, including LLMs, will be reliant on having
object. Oftentimes, you can find an open source model that solves your specific a strong foundation of data. LLMs are incredible tools, but they have to be used
problem that is orders of magnitude smaller than ChatGPT, allowing you to bring and implemented on top of this strong data foundation. Databricks brings both
the model into your environment and host it yourself. This means that you can that strong data foundation as well as the integrated tools to let you use and
keep the data in your control for privacy and governance concerns as well as fine-tune LLMs in your domain.
manage your costs.

Another huge upside to using open source models is the ability to fine-tune
them to your own data. Since you’re not dealing with a black box of a proprietary
service, there are techniques that let you take open source models and train
them to your specific data, greatly improving their performance on your
specific domain. We believe the future of language models is going to move
in this direction, as more and more organizations will want full control and
understanding of their LLMs.
A Compact Guide to Large Language Models 8

S ECT I O N 4

So What Do I Do Next If I Want to Start Using LLMs?

That depends where you are on your journey! Fortunately, we have a few paths
for you. Getting started with NLP using Hugging Face
transformers pipelines
If you want to go a little deeper into LLMs but aren’t quite ready to do it yourself,
you can watch one of Databricks’ most talented developers and speakers go
over these concepts in more detail during the on-demand talk “How to Build
Your Own Large Language Model Like Dolly.”
Fine-Tuning Large Language Models with
If you’re ready to dive a little deeper and expand your education and
Hugging Face and DeepSpeed
understanding of LLM foundations, we’d recommend checking out our
course on LLMs. You’ll learn how to develop production-ready LLM applications
and dive into the theory behind foundation models.

If your hands are already shaking with excitement and you already have some Introducing AI Functions: Integrating Large
working knowledge of Python and Databricks, we’ll provide some great examples Language Models with Databricks SQL
with sample code that can get you up and running with LLMs right away!
About Databricks
Databricks is the data and AI company. More than 9,000
organizations worldwide — including Comcast, Condé Nast and
over 50% of the Fortune 500 — rely on the Databricks Lakehouse
Platform to unify their data, analytics and AI. Databricks is
headquartered in San Francisco, with offices around the globe.
Founded by the original creators of Apache Spark™, Delta Lake
and MLflow, Databricks is on a mission to help data teams solve
the world’s toughest problems. To learn more, follow Databricks on
Twitter, LinkedIn and Facebook.

S TA R T YO U R F R E E T R I A L

Contact us for a personalized demo:

databricks.com/contact

© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation. Privacy Policy | Terms of Use