The 4 Levels of REST Debugging

Find that annoying bug by spying on the data being exchanged in four di erent ways

Nassos Michas
Dec 20, 2019 · 11 min read

Image by Mohamed Hassan from Pixabay

Irrespective of whether you’re on an Angular, React, Vue, Backbone, or JavaScript front

end, at some point you’ll probably need to communicate with a back-end service. But
what happens when your back end replies with an undesired non2xx reply? Did you
transmit the exact payload your back end expected? Did you include the precise HTTP
headers needed? Is there a problem at the protocol level maybe?
In this piece, we’ll review some of the common approaches to efficiently debug REST
calls: The JavaScript console, the network monitor, a man-in-the-middle server, and
network-traffic analysis.

. . .

Basic Elements of a REST Request

If you’re curious about how and when REST was defined as a concept, Wikipedia has
an interesting page about it¹.

In summary, REST, the representational state transfer, is a “software architectural style

that defines a set of constraints to be used for creating Web services.” It was first
defined back in 2000 by Roy Fielding² in his PhD dissertation “Architectural Styles and
the Design of Network-based Software Architectures.”³

Basic elements of a REST request (image by author)

In the interest of the debugging methods and tools we’re about to discuss next, let’s
quickly recap the basic elements comprising a REST request:

Endpoint: This is the location, or address, where the server-side recipient of your
REST call exists. It’s defined as a URL, and in its fullest form may look like
protocol://domain/path?queryParameters .

Method: The HTTP request method⁴ to indicate the desired action to be performed
for a given resource. Also known as the HTTP verb, this is one of the following:
GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, and PATCH.
 Headers: HTTP headers⁵ consist of additional information accompanying an HTTP
request passed from the client to the server and vice versa. Your request may or
may not have headers associated with it.

Body: The HTTP message body is any additional data transmitted in an HTTP
transaction message immediately following the headers. The body is optional, and
not all requests need to have one.

Your preferred back-end framework’s controller logic has to match all the elements of
an incoming REST request in order to serve it.

So what can go wrong? If you’re dynamically creating your endpoint, your processing
might result in a vital part of the URL going missing. The HTTP method used in your
request may mismatch with the method expected by your back end. Or an important
header might not be injected or may be misspelt.

In the following sections, we’ll examine four different ways of increasing complexity to
debug client-side REST calls when the back end refuses to serve them.

. . .

Level 1: Browser Console

Your internet browser’s console, although the poor man’s choice for debugging, can be
helpful if you carefully plan your messages.

Sure, even if you work with a front-end framework that hot reloads your code,
changing the code by adding output statements is probably not the most efficient way
to debug.

However, you can still leverage your browser’s console to display debug messages to
provide you with some immediate insight into your code being executed — as well as
insight into the critical data structures being generated and exchanged as part of your
REST calls.

As a rule of thumb, try to keep logs using the browser’s console minimised. Scattering
debug messages in your front-end code becomes a technical debt, and unless you have
a static-code analysis tool, like SonarQube, to point them out for you, they may be
forgotten.
So if the browser console is your only option, at least consider using a logging
framework. A front-end logging framework allows you to separate your debug
messages in groups according to their level and, possibly, other additional attributes.

According to which part of your application you need to further debug, you can
selectively enable and disable specific component-level logging groups, shedding light
into the specific piece of code you need. When you’re ready to roll to production, you
can disable all nonessential logging by flicking a central configuration variable.

There are many logging frameworks to choose from, according to the framework you
use in your front end, so no need reinventing the wheel here. Here’s a quick selection
you might want to try:

JavaScript/NodeJS: ulog, winston

Angular: NGX Logger
Vue: vuejs-logger
React: react-logger-lib

You can, of course, use the browser console for some fancy or preventive notifications,
just like Facebook and CNN do:

Facebook browser console (image by author)

CNN.com browser console (image by author)

 . . .

Level 2: Browser Network Monitor/Debugger

If the debugging functionality of the browser console isn’t enough to figure out what
goes wrong and your REST call is not accepted by your back end, it’s time for your
browser’s network monitor/debugger.

The network debugger tab is one of the tabs in your JavaScript or developers console
found in all browsers. Although it may look slightly different between various
browsers, its essential functionality remains the same. It allows you to view every
single network request made by the browser to fetch external resources.

To examine some of the interesting features of the network tab, we’ll use the online
Petstore sample application provided by Swagger:

Petstore sample application (image by author)

Swagger’s intuitive design allows us to pick an API endpoint and test it right from the
browser. Let’s do that to issue a GET request for /pet/1 using Chrome:

Open the Petstore application at https://petstore.swagger.io

 Click on the blue GET button next to /pet/{petId}

Click on “Try it out”

Enter “1” as the pet ID, and click on the blue “Execute” button

Your Chrome’s network tab should now look similar to this:

Chrome’s default Network tab (image by author)

There are several buttons and configuration options here. Some of the most useful ones
are the Clear button (1), the request-type selection (2), and the Network settings
button (3).

Using the options under the Network setting, you can turn your network output to
something more compact, like this:

A compact version of Chrome’s Network tab (image by author)

By clicking on the request on the left, you can see the details of this request on the
right. There are five main sections of details depicting the headers exchanged, a
preview of the response received (nicely formatted if it’s in one of the supported
formats), the raw response received, the HTTP cookies used, as well as the various
information regarding the timing of the request.

The Network tab provides a plethora of information, essentially shedding ample light
behind the scenes of your REST calls. Browsing through the captured requests, the
Network tab enables you to investigate what exactly your client is sending to effectively
cross-check it with what your back end is expecting.

Before we move to Level 3, let’s quickly highlight an interesting feature of the Network
tab.

When you need to quickly replay a REST call, instead of going through your UI
repeating your actions, you can just right-click on the request and choose “Copy as
cURL.”

Copying a request as cURL (image by author)

You can then open your favourite command line interface (CLI)and paste the request to
replay it:
 Replaying a request with cURL (image by author)

. . .

Level 3: Man-in-the-Middle Proxy

The browser JavaScript console and Network tabs are very convenient when you want
to debug browser-generated calls. What if you have a back-end application employing
an embedded REST client? Here’s where man-in-the-middle⁶ comes into play.

In cryptography and computer science, man-in-time-middle (MITM) is a term

associated with the word attack. It’s used to describe an attack where the attacker
secretly relays and possibly alters the communications between two parties who believe
they’re directly communicating with each other.

For our debugging purposes, we’ll borrow the concept of MITM but not to perform an
attack or alter the content of a request — just to record it. By sitting in the middle of the
REST client and the REST server, we can observe traffic from both directions and
provide insight and logging to third parties.

Let’s see how it works next.

DIRECT request-reply
The simplest DIRECT request-reply sequence involves just the client and the REST
server. You make a DIRECT request on the server address and port, and you receive an
answer back:
 A DIRECT request-reply call (image by author)

There is, obviously, no man-in-the-middle here, so let’s move on to introduce one.

MITM request-reply
In a MITM scenario, we introduce an extra part in the architecture. This is a component
sitting between the REST client and the REST endpoint:

A request-reply with a MITM proxy (image by author)

The MITM proxy listens for incoming requests on a specific port and then rerequests all
captured requests to a target server and port.

Once the target server provides a reply, the MITM proxy forwards the reply back to the
original REST client while locally logging all traffic.

Your client remains unaware of all this taking place behind the scenes, and you don’t
need to change any of your code except then introducing the proxy URL. According to
the underlying framework and architecture of your application, this might be a
command-line option you need to pass or an in-application configuration option.

Using a MITM proxy

You can effortlessly launch your local MITM proxy using an excellent open-source
project called mitmproxy. This proxy tool is available for all major OSs — as well as in a
Docker image. For this hands-on part, we’ll use the Docker image of mitmproxy , and
we’ll also expose its web interface so we can review collected logs using an internet
browser.

To start mitmproxy using Docker, you can issue the following command:
 docker run --rm -it \
-p 8080:8080 \
-p 8081:8081 \
mitmproxy/mitmproxy mitmweb \
--web-iface 0.0.0.0

This will start the proxy listening on port 8080, exposing its web interface on port
8081. We can now proceed to retry a previous REST request using cURL, this time
using the mitmproxy as our proxy server:

curl -k --proxy http://localhost:8080

http://petstore.swagger.io/v2/pet/1

Let’s open the web interface of mitmproxy to examine what was captured:

The web interface of mitmproxy (image by author)

On the left, we see the original request captured, and on the right we see the data
exchanged for this request.

This interface seems similar to the Network tab of the browser we examined in Level 1
above; however, mitmproxy provides many more options to intercept requests, save
HTTP conversations, replay client-side and server-side responses, filter and manipulate
content, etc.

. . .

Level 4: Network-Traffic Analysis

Congratulations, you’ve made it to Level 4. In all of the previous levels, debugging was
taking place by examining information already interpreted by another application layer
between us and the data. On this level, we move down to the network-representation
level so we can debug information directly at the packet level.

Mind you this isn’t a typical level on which REST calls need to be debugged. Usually,
the problem is somewhere higher up your stack. However, debugging on this level
allows you to discover deeper problems at the protocol level and connectivity issues,
security issues, packet drops, latency, etc. that the earlier debugging levels can’t help
you with.

So let’s capture some network data.

Network-traffic analysis requires a way to capture the packets being transmitted by

your network interface. The most well-known, free tool in that category is Wireshark (a
fork of Ethereal). Wireshark is the world’s foremost and widely used network-protocol
analyzer and lets you see what’s happening on your network at a microscopic level.

Wireshark is such an impressive, comprehensive network-traffic analysis tool that it’d

require several standalone posts to explain it. Therefore, I’ll only attempt to scratch the
surface and quickly show you how to capture and visualise your HTTP traffic.

Network capturing and packet filtering

If you start Wireshark’s capture, you’ll see many, many packets being captured. The
first thing you want to do is to filter those packets to the ones of interest. Reusing the
Petstore example from above, we can see the server we’re about to communicate with
is petstore.swagger.io , so we need to filter network packets from and to this address.

Setting such a filter in Wireshark requires just a couple of steps:

Go to View > Name resolution, and enable all resolve options.

Use the following filter: ip.host == petstore.swagger.io

You can start network capturing by clicking on the blue fin icon. You should get
something similar to the following output:
 A Wireshark capture (image by author)

The above capture includes all network packets transmitted and received while your
host was communicating with petstore.swagger.io and executing the REST call for
pet/1 . You can view the full details of each packet by clicking on a specific
request/response in the upper part in green.

A more convenient way to visualise a complete exchange is by right-clicking on a

request/response row and then selecting Follow > TCP Stream :
 Following a TCP stream in Wireshark (image by author)

Wireshark might not be the most intuitive tool to use and takes time to master;
however, it can help you find problems at the lowest possible level.

. . .

Conclusion
Sometimes, when we develop a REST client we find that the REST back end refuses to
serve an appropriate reply back.

In this piece, we examined four different levels of increasing complexity. allowing us to

shed light into the REST data exchange.

The JavaScript console, the network monitor, a man-in-the-middle server, and

network-traffic analysis are valuable tools in the hands of the front-end software
developer to identify issues.

. . .

References
[1] Wikipedia. 2005. Representational state transfer [Online Encyclopedia].
https://en.wikipedia.org/wiki/Representational_state_transfer (Accessed 2019–Dec).
[2] Wikipedia. 2005. Roy Thomas Fielding [Online Encyclopedia].
https://en.wikipedia.org/wiki/Roy_Fielding (Accessed 2019–Dec).
[3] Fielding, R. 2000. Architectural Styles and the Design of Network-based Software
Architectures [PhD dissertation].
http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm (Accessed
2019–Dec).
[4] MDN web docs. 2016. HTTP request methods [Website].
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods (Accessed 2019–
 Dec).
[5] MDN web docs. 2010. HTTP headers [Website]. https://developer.mozilla.org/en-
US/docs/Web/HTTP/Headers (Accessed 2019–Dec).
[6] Wikipedia. 2005. Man-in-the-middle attack [Online Encyclopedia].
https://en.wikipedia.org/wiki/Man-in-the-middle_attack (Accessed 2019–Dec)
[7] Wikipedia. 2005. Domain Name System [Online Encyclopedia].
https://en.wikipedia.org/wiki/Domain_Name_System (Accessed 2019–Dec)
)

Some rights reserved

Programming JavaScript Software Development Front End Development API

About Help Legal