Data Analysis with Kibana: Agenda
● Getting Started
● Search your Data
● Visualize your Data
● Analyze your Data
● Present your Data
● Analyze your Data with Machine Learning
● Advanced Kibana
● Anomaly Hunt
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Search your Data
Module 2
Topics
● Discover and Data Visualizer
● KQL and Filters
● Field Focus
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover and Data
Visualizer
Module 2 Lesson 1
Overview
● Elasticsearch documents and data types
○ Numeric
○ Text, keywords
○ Dates and more
● Data Visualizer
○ Examine existing data in detail
○ Based on index patterns
● Discover
○ Explore data in Elasticsearch
○ Slice and dice (analyze) data
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Documents
● In the Elastic Stack, data is stored in Elasticsearch
● Elasticsearch is a document store
○ it stores data as JSON objects, called documents
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Fields and values
● Documents have fields
● Every field:
○ can have 0 or more values
○ has a data type
{
"category": [
"Women's Clothing",
"Women's Shoes"
],
"currency": "EUR",
"customer_full_name": "Pia Carr",
"customer_id": 45,
"order_date": "2022-12-09T11:11:02+00:00",
"order_id": 569968,
...
}
Values
Fields
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Data types
● Kibana labels fields with an icon based on
the type
● Dates, text, numeric, IP, boolean and geo
points are the key types
● The data type influences the way the field
can be used within Kibana
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Data Visualizer
Query bar Fields list Time filter
Histogram
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Quickly view any field
● Filter for fields in the fields list by name or type
● Field type, name, percentage of docs with the value set, and distinct
values are shown in a simple view
○ Expand the field to get a list of the top values
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover Query bar Fields list Toolbar
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover Index pattern Time filter
Histogram
Doc table
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Context: time and pattern
● One of the first issues new users run into is having no results
● Always check the time filter and the index pattern
○ The combination of these is your context
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Working with fields
● Search for a field by name
● Filter by field type
● Click on a field for top values
● Click the + to add the field to the
document table
○ Field will be added to Popular
list
○ Selected fields list will be
created
● Columns in the document table can
be moved, resized and sorted
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Working with documents
● From the document table, click the
expand icon ( ) for details
● Can view as a table (shown) or as
raw JSON
● Click Single document to open the
view to the full browser window
● Click Surrounding documents to
view documents with a similar
timestamp
○ Selected fields will still be
shown
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Interactive histogram
● The time filter can be visually changed by:
○ click and dragging across the histogram
○ clicking on a single bar
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Summary:
Discover and Data
Visualizer
Module 2 Lesson 1
Summary
● Data types influence how fields may be used in Kibana
● The Data Visualizer can be used to inspect an index pattern as a
whole
● Discover can be used to drill down into specific documents
● The time filter and index pattern in Discover form the context of the
view
○ If no data is visible, check the time and pattern
● The document table can be customized to show selected fields
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Quiz
1. In Discover, which two settings determine the scope or context?
2. True or False: The Data Visualizer tool can be used to examine
specific documents
3. What data type is represented by this icon:
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover and Data
Visualizer
Lab 2.1
Explore logs data with Discover and Data
Visualizer
KQL and filters
Module 2 Lesson 2
Search is everywhere
● Elasticsearch is a search engine
○ Kibana can be used to search documents in Elasticsearch
● A search is executed by sending a query to Elasticsearch
○ A query can answer many different types of questions:
ᐨ Who are the customers with the first name Selena?
ᐨ What are the names of customers in France?
ᐨ Are there any orders for Men’s Clothing?
● In Kibana, a search can be executed from the query bar
○ We will focus on KQL, the Kibana Query Language
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Querying
● Crafting a good question gets good results
● Let’s say we want to find all the orders for the customer “Selena
Simmons”
● First, let’s get our context:
○ All the orders means we need
to select a wide time range
○ All the orders means we should
select the orders index pattern
● Now let’s search . . .
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Better queries, better results
● Our search returns a lot of results
○ But not the exact results
● By default, the query logic is going to look in all fields, and for any
values, leading to results like . . .
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Boolean operators
● By default, KQL uses or logic
○ This is why our results included customers with either “Selena”
or “Simmons”
● KQL supports and, or and not operators
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Search a specific field
● Searching every field is time consuming
● Speed up searching by using field names as well
○ KQL auto-complete will help
○ The : means equals for a text field
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Wildcards and grouping
● Just one wildcard: *
○ This query finds all the first names starting with “S”:
● Quotes for grouping search terms
○ These two queries work differently!
The full name will be searched for The full name has to be
Selena or Simmons Selena Simmons
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Querying numeric fields
● Let’s ask a more complex question:
○ Find all the orders where the customer’s first name is “Selena”
and the order total price is greater than or equal to $50
● Numbers allow for range queries
○ Greater than (>) or equal (>=), less than (<) or equal (<=)
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Query bar limitations
● Let’s take our example and expand it
○ customer_first_name:Selena
○ customer_last_name:Simmons
○ taxful_total_price>=50
○ geoip.city_name:Los Angeles
○ category:Shoes
● You may want to use different combinations of these clauses
● With the query bar, you will have to do a lot of typing and deleting
● Filters are sticky queries
○ Individual query clauses that can be turned on and off
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Defining a filter
● There are two ways to define a filter from Discover
○ The + or - symbol on any list creates a filter for that value
○ The Add filter link will open a dialog
as well
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Filter operations
● Once defined, a filter can be:
○ pinned
○ edited
○ negated
○ disabled
○ deleted
● Filters can be collectively
managed via the ( ) icon
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Filter customization
● Internally filters are transformed into a query
● You can change the filter by editing the query
● You can add a custom a label to the filter to quickly identify it
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Filters and the query bar
● You can use filters and the query bar together
● Use some broad filters to get a large slice of your data
○ Enable, include, exclude as needed
● Refine your results with a query
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Saved searches
● Any search can be saved in Kibana
○ Saved searches can be used
as the starting point for
Dashboards and other tools
○ Saved searches can be shared
between Spaces
● To save a search, simply click
Save in Discover and give the
search a name
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Saved queries
● Any query can be saved in Kibana
○ Saved queries can opened in
any tool that uses the Query
Bar
○ Saved queries can also be
shared between Spaces
● To save a query, click the disk icon
in the query bar, and then Save
current query
● Click the disk icon to load a saved
query as well
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Saved queries vs. saved searches
● What’s the difference between a saved query and a saved search?
○ Searches include the index pattern, queries do not
○ Searches can be added to dashboards
○ Queries may be loaded from any query bar
● What is similar between them?
○ Both can store the KQL and filters, as well as the time filter
settings
○ Both may be shared between spaces
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Summary:
KQL and filters
Module 2 Lesson 2
Summary
● Kibana filters and the query bar are complementary
● Kibana filters provide an easy way to explore data by
○ enabling and disabling them
○ pinning and having them follow to different parts of Kibana
● The query bar can be used to search all the data inside Elasticsearch
● The KQL language supports and, or and not boolean operators
● KQL provides auto-completion for writing queries
● Queries and searches can be saved for later reuse.
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Quiz
1. True or False: You can only use a single filter at a time in Kibana
2. Name three actions that you can perform on a filter
3. What are the three different boolean operators?
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
KQL and filters
Lab 2.2
Query and filter the logs data with KQL
Field focus
Module 2 Lesson 3
The shortest path to visualization
● Visualizations can be created
directly from Discover or Data
Visualizer
● Select a field, and click Visualize
[Discover] or the icon [Data
Visualizer]
● Geo point fields will open in Maps
● All other field types will open in
Lens
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Focus with Lens
● We will use Lens to focus on a single field: geoip.city_name
● If we Visualize this field, we are presented with this view . . .
● Vertical bar chart
● Simple count of records
● Split by city name
● Sorted descending
● Top 5 shown
● With an “Other”
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Change the visual
● Bar charts are nice, but sometimes it helps to see a proportion
● Change the view to a Donut
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Change the values
● Maybe we don’t want Other, or want more than 5 cities
● In the layer pane, click Slice by to adjust the slices
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Get suggestions
● Everyone loves donuts, but maybe a tree map looks better
● See a preview in the Suggestions panel
● Select the view that works
best for you
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Map your data
● Visualize a geo point field to open the Map editor
● We will spend more time with Maps in later lessons
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Using visualizations
● Visualizations can be saved
○ And automatically added to a dashboard
● Lens and Maps visualizations can create filters
○ Filters can be pinned and used in Discover
● Click and drag in time based visualizations to change the time filter
○ Just like the Discover histogram
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Summary:
Field focus
Module 2 Lesson 3
Summary
● The Visualize link in Discover creates Lens or Maps visualizations
● Lens enables you to visualize a field
● Maps visualizations link documents to points on a map
● Visualizations can be used to create filters and change the time
filter
● Visualizations can be saved to dashboards
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Quiz
1. True or False: Visualizing geo point data opens the Lens editor by
default
2. What part of Lens is used to change the displayed values?
3. True or False: It is difficult to change visualization styles in Lens
Copyright Elasticsearch BV 2015-2022 Copying, publishing and/or distributing without written permission is strictly prohibited
Field focus
Lab 2.3
Create visualizations from Discover
