Compilation, Execution, and Errors

For instance, let us consider math.sin. How should it behave when called on a table? Suppose it returns
an error code. If we need to check for errors, we would have to write something like this:

local res = math.sin(x)

if not res then -- error?
error-handling code

However, we could as easily check this exception before calling the function:

if not tonumber(x) then -- x is not a number?

error-handling code

Frequently we check neither the argument nor the result of a call to sin; if the argument is not a number,
it means that probably there is something wrong in our program. In such situations, the simplest and most
practical way to handle the exception is to stop the computation and issue an error message.

On the other hand, let us consider io.open, which opens a file. How should it behave when asked to open
a file that does not exist? In this case, there is no simple way to check for the exception before calling the
function. In many systems, the only way of knowing whether a file exists is by trying to open it. Therefore,
if io.open cannot open a file because of an external reason (such as “file does not exist” or “permission
denied”), it returns false, plus a string with the error message. In this way, we have a chance to handle the
situation in an appropriate way, for instance by asking the user for another file name:

local file, msg

repeat
print "enter a file name:"
local name = io.read()
if not name then return end -- no input
file, msg = io.open(name, "r")
if not file then print(msg) end
until file

If we do not want to handle such situations, but still want to play safe, we simply use assert to guard
the operation:

file = assert(io.open(name, "r"))

--> stdin:1: no-file: No such file or directory

This is a typical Lua idiom: if io.open fails, assert will raise an error. Notice how the error message,
which is the second result from io.open, goes as the second argument to assert.

Error Handling and Exceptions

For many applications, we do not need to do any error handling in Lua; the application program does this
handling. All Lua activities start from a call by the application, usually asking Lua to run a chunk. If there
is any error, this call returns an error code, so that the application can take appropriate actions. In the case
of the stand-alone interpreter, its main loop just prints the error message and continues showing the prompt
and running the given commands.

However, if we want to handle errors inside the Lua code, we should use the function pcall (protected
call) to encapsulate our code.

Suppose we want to run a piece of Lua code and to catch any error raised while running that code. Our
first step is to encapsulate that piece of code in a function; more often than not, we use an anonymous
function for that. Then, we call that function through pcall:

127
 Compilation, Execution, and Errors

local ok, msg = pcall(function ()

some code
if unexpected_condition then error() end
some code
print(a[i]) -- potential error: 'a' may not be a table
some code
end)

if ok then -- no errors while running protected code

regular code
else -- protected code raised an error: take appropriate action
error-handling code
end

The function pcall calls its first argument in protected mode, so that it catches any errors while the
function is running. The function pcall never raises any error, no matter what. If there are no errors,
pcall returns true, plus any values returned by the call. Otherwise, it returns false, plus the error message.

Despite its name, the error message does not have to be a string; a better name is error object, because
pcall will return any Lua value that we pass to error:

local status, err = pcall(function () error({code=121}) end)

print(err.code) --> 121

These mechanisms provide all we need to do exception handling in Lua. We throw an exception with
error and catch it with pcall. The error message identifies the kind of error.

Error Messages and Tracebacks

Although we can use a value of any type as an error object, usually error objects are strings describing
what went wrong. When there is an internal error (such as an attempt to index a non-table value), Lua
generates the error object, which in that case is always a string; otherwise, the error object is the value
passed to the function error. Whenever the object is a string, Lua tries to add some information about
the location where the error happened:

local status, err = pcall(function () error("my error") end)

print(err) --> stdin:1: my error

The location information gives the chunk's name (stdin, in the example) plus the line number (1, in
the example).

The function error has an additional second parameter, which gives the level where it should report the
error. We use this parameter to blame someone else for the error. For instance, suppose we write a function
whose first task is to check whether it was called correctly:

function foo (str)

if type(str) ~= "string" then
error("string expected")
end
regular code
end

Then, someone calls this function with a wrong argument:

128
 Compilation, Execution, and Errors

foo({x=1})

As it is, Lua points its finger to foo —after all, it was it who called error— and not to the real culprit,
the caller. To correct this problem, we inform error that the error it is reporting occurred on level two
in the calling hierarchy (level one is our own function):

function foo (str)

if type(str) ~= "string" then
error("string expected", 2)
end
regular code
end

Frequently, when an error happens, we want more debug information than only the location where the error
occurred. At least, we want a traceback, showing the complete stack of calls leading to the error. When
pcall returns its error message, it destroys part of the stack (the part that goes from it to the error point).
Consequently, if we want a traceback, we must build it before pcall returns. To do this, Lua provides the
function xpcall. It works like pcall, but its second argument is a message handler function. In case of
error, Lua calls this message handler before the stack unwinds, so that it can use the debug library to gather
any extra information it wants about the error. Two common message handlers are debug.debug, which
gives us a Lua prompt so that we can inspect by ourselves what was going on when the error happened;
and debug.traceback, which builds an extended error message with a traceback. The latter is the
function that the stand-alone interpreter uses to build its error messages.

Exercises
Exercise 16.1: Frequently, it is useful to add some prefix to a chunk of code when loading it. (We saw an
example previously in this chapter, where we prefixed a return to an expression being loaded.) Write a
function loadwithprefix that works like load, except that it adds its extra first argument (a string)
as a prefix to the chunk being loaded.

Like the original load, loadwithprefix should accept chunks represented both as strings and as
reader functions. Even in the case that the original chunk is a string, loadwithprefix should not
actually concatenate the prefix with the chunk. Instead, it should call load with a proper reader function
that first returns the prefix and then returns the original chunk.

Exercise 16.2: Write a function multiload that generalizes loadwithprefix by receiving a list of
readers, as in the following example:

f = multiload("local x = 10;",
io.lines("temp", "*L"),
" print(x)")

In the above example, multiload should load a chunk equivalent to the concatenation of the string
"local...", the contents of the temp file, and the string "print(x)". Like loadwithprefix,
from the previous exercise, multiload should not actually concatenate anything.

Exercise 16.3: The function stringrep, in Figure 16.2, “String repetition”, uses a binary multiplication
algorithm to concatenate n copies of a given string s.

129
 Compilation, Execution, and Errors

Figure 16.2. String repetition

function stringrep (s, n)
local r = ""
if n > 0 then
while n > 1 do
if n % 2 ~= 0 then r = r .. s end
s = s .. s
n = math.floor(n / 2)
end
r = r .. s
end
return r
end

For any fixed n, we can create a specialized version of stringrep by unrolling the loop into a sequence
of instructions r = r .. s and s = s .. s. As an example, for n = 5 the unrolling gives us
the following function:

function stringrep_5 (s)

local r = ""
r = r .. s
s = s .. s
s = s .. s
r = r .. s
return r
end

Write a function that, given n, returns a specialized function stringrep_n. Instead of using a closure,
your function should build the text of a Lua function with the proper sequence of instructions (a mix of r
= r .. s and s = s .. s) and then use load to produce the final function. Compare the performance
of the generic function stringrep (or of a closure using it) with your tailor-made functions.

Exercise 16.4: Can you find any value for f such that the call pcall(pcall, f) returns false as its
first result? Why is this relevant?

130
Chapter 17. Modules and Packages
Usually, Lua does not set policies. Instead, Lua provides mechanisms that are powerful enough for groups
of developers to implement the policies that best suit them. However, this approach does not work well
for modules. One of the main goals of a module system is to allow different groups to share code. The
lack of a common policy impedes this sharing.

Starting in version 5.1, Lua has defined a set of policies for modules and packages (a package being a
collection of modules). These policies do not demand any extra facility from the language; programmers
can implement them using what we have seen so far. Programmers are free to use different policies. Of
course, alternative implementations may lead to programs that cannot use foreign modules and modules
that cannot be used by foreign programs.

From the point of view of the user, a module is some code (either in Lua or in C) that can be loaded through
the function require and that creates and returns a table. Everything that the module exports, such as
functions and constants, it defines inside this table, which works as a kind of namespace.

As an example, all standard libraries are modules. We can use the mathematical library like this:

local m = require "math"

print(m.sin(3.14)) --> 0.0015926529164868

However, the stand-alone interpreter preloads all standard libraries with code equivalent to this:

math = require "math"

string = require "string"
...

This preloading allows us to write the usual notation math.sin, without bothering to require the module
math.

An obvious benefit of using tables to implement modules is that we can manipulate modules like any
other table and use the whole power of Lua to create extra facilities. In most languages, modules are not
first-class values (that is, they cannot be stored in variables, passed as arguments to functions, etc.); those
languages need special mechanisms for each extra facility they want to offer for modules. In Lua, we get
extra facilities for free.

For instance, there are several ways for a user to call a function from a module. The usual way is this:

local mod = require "mod"

mod.foo()

The user can set any local name for the module:

local m = require "mod"

m.foo()

She can also provide alternative names for individual functions:

local m = require "mod"

local f = m.foo
f()

She can also import only a specific function:

local f = require "mod".foo -- (require("mod")).foo

131
 Modules and Packages

f()

The nice thing about these facilities is that they involve no special support from Lua. They use what the
language already offers.

The Function require

Despite its central role in the implementation of modules in Lua, require is a regular function, with
no special privileges. To load a module, we simply call it with a single argument, the module name.
Remember that, when the single argument to a function is a literal string, the parentheses are optional,
and it is customary to omit them in regular uses of require. Nevertheless, the following uses are all
correct, too:

local m = require('math')

local modname = 'math'

local m = require(modname)

The function require tries to keep to a minimum its assumptions about what a module is. For it, a module
is just any code that defines some values, such as functions or tables containing functions. Typically, that
code returns a table comprising the module functions. However, because this action is done by the module
code, not by require, some modules may choose to return other values or even to have side effects (e.g.,
by creating global variables).

The first step of require is to check in the table package.loaded whether the module is already
loaded. If so, require returns its corresponding value. Therefore, once a module is loaded, other calls
requiring the same module simply return the same value, without running any code again.

If the module is not loaded yet, require searches for a Lua file with the module name. (This search
is guided by the variable package.path, which we will discuss later.) If it finds such a file, it loads
it with loadfile. The result is a function that we call a loader. (The loader is a function that, when
called, loads the module.)

If require cannot find a Lua file with the module name, it searches for a C library with that name.1 (In
that case, the search is guided by the variable package.cpath.) If it finds a C library, it loads it with
the low-level function package.loadlib, looking for a function called luaopen_modname. The
loader in this case is the result of loadlib, which is the C function luaopen_modname represented
as a Lua function.

No matter whether the module was found in a Lua file or a C library, require now has a loader for
it. To finally load the module, require calls the loader with two arguments: the module name and
the name of the file where it got the loader. (Most modules just ignore these arguments.) If the loader
returns any value, require returns this value and stores it in the package.loaded table, to return
the same value in future calls for this same module. If the loader returns no value, and the table entry
package.loaded[@rep{modname}] is still empty, require behaves as if the module returned
true. Without this correction, a subsequent call to require would run the module again.

To force require into loading the same module twice, we can erase the library entry from
package.loaded:

package.loaded.modname = nil

The next time the module is required, require will do all its work again.
1
In the section called “C Modules”, we will discuss how to write C libraries.

132
 Modules and Packages

A common complaint against require is that it cannot pass arguments to the module being loaded. For
instance, the mathematical module might have an option for choosing between degrees and radians:

-- bad code
local math = require("math", "degree")

The problem here is that one of the main goals of require is to avoid loading a module multiple times.
Once a module is loaded, it will be reused by whatever part of the program that requires it again. There
would be a conflict if the same module were required with different parameters. In case you really want
your module to have parameters, it is better to create an explicit function to set them, like here:

local mod = require "mod"

mod.init(0, 0)

If the initialization function returns the module itself, we can write that code like this:

local mod = require "mod".init(0, 0)

In any case, remember that the module itself is loaded only once; it is up to it to handle conflicting ini-
tializations.

Renaming a module
Usually, we use modules with their original names, but sometimes we must rename a module to avoid
name clashes. A typical situation is when we need to load different versions of the same module, for
instance for testing. Lua modules do not have their names fixed internally, so usually it is enough to
rename the .lua file. However, we cannot edit the object code of a C library to correct the name of
its luaopen_* function. To allow for such renamings, require uses a small trick: if the module
name contains a hyphen, require strips from the name its suffix after the hyphen when creating the
luaopen_* function name. For instance, if a module is named mod-v3.4, require expects its open
function to be named luaopen_mod, instead of luaopen_mod-v3.4 (which would not be a valid
C name anyway). So, if we need to use two modules (or two versions of the same module) named mod,
we can rename one of them to mod-v1, for instance. When we call m1 = require "mod-v1",
require will find the renamed file mod-v1 and, inside this file, the function with the original name
luaopen_mod.

Path searching
When searching for a Lua file, the path that guides require is a little different from typical paths. A
typical path is a list of directories wherein to search for a given file. However, ISO C (the abstract platform
where Lua runs) does not have the concept of directories. Therefore, the path used by require is a list
of templates, each of them specifying an alternative way to transform a module name (the argument to
require) into a file name. More specifically, each template in the path is a file name containing optional
question marks. For each template, require substitutes the module name for each question mark and
checks whether there is a file with the resulting name; if not, it goes to the next template. The templates
in a path are separated by semicolons, a character seldom used for file names in most operating systems.
For instance, consider the following path:

?;?.lua;c:\windows\?;/usr/local/lua/?/?.lua

With this path, the call require "sql" will try to open the following Lua files:

sql
sql.lua
c:\windows\sql
/usr/local/lua/sql/sql.lua

133
 Modules and Packages

The function require assumes only the semicolon (as the component separator) and the question mark;
everything else, including directory separators and file extensions, is defined by the path itself.

The path that require uses to search for Lua files is always the current value of the variable
package.path. When the module package is initialized, it sets this variable with the value of the en-
vironment variable LUA_PATH_5_3; if this environment variable is undefined, Lua tries the environment
variable LUA_PATH. If both are unefined, Lua uses a compiled-defined default path.2 When using the
value of an environment variable, Lua substitutes the default path for any substring ";;". For instance, if
we set LUA_PATH_5_3 to "mydir/?.lua;;", the final path will be the template "mydir/?.lua"
followed by the default path.

The path used to search for a C library works exactly in the same way, but its value comes from the variable
package.cpath, instead of package.path. Similarly, this variable gets its initial value from the
environment variables LUA_CPATH_5_3 or LUA_CPATH. A typical value for this path in POSIX is like
this:

./?.so;/usr/local/lib/lua/5.2/?.so

Note that the path defines the file extension. The previous example uses .so for all templates; in Windows,
a typical path would be more like this one:

.\?.dll;C:\Program Files\Lua502\dll\?.dll

The function package.searchpath encodes all those rules for searching libraries. It takes a module
name and a path, and looks for a file following the rules described here. It returns either the name of the
first file that exists or nil plus an error message describing all files it unsuccessfully tried to open, as in
the next example:

> path = ".\\?.dll;C:\\Program Files\\Lua502\\dll\\?.dll"

> print(package.searchpath("X", path))
nil
no file '.\X.dll'
no file 'C:\Program Files\Lua502\dll\X.dll'

As an interesting exercise, in Figure 17.1, “A homemade package.searchpath” we implement a

function similar to package.searchpath.

Figure 17.1. A homemade package.searchpath

function search (modname, path)
modname = string.gsub(modname, "%.", "/")
local msg = {}
for c in string.gmatch(path, "[^;]+") do
local fname = string.gsub(c, "?", modname)
local f = io.open(fname)
if f then
f:close()
return fname
else
msg[#msg + 1] = string.format("\n\tno file '%s'", fname);
end
end
return nil, table.concat(msg) -- not found
end
2
Since Lua 5.2, the stand-alone interpreter accepts the command-line option -E to prevent the use of those environment variables and force the
default.

134
 Modules and Packages

The first step is to substitute the directory separator, assumed to be a slash in this example, for any dots.
(As we will see later, a dot has a special meaning in a module name.) Then the function loops over all
components of the path, wherein each component is a maximum expansion of non-semicolon characters.
For each component, the function substitutes the module name for the question marks to get the final file
name, and then it checks whether there is such a file. If so, the function closes the file and returns its name.
Otherwise, it stores the failed name for a possible error message. (Note the use of a string buffer to avoid
creating useless long strings.) If no file is found, then it returns nil plus the final error message.

Searchers
In reality, require is a little more complex than we have described. The search for a Lua file and the
search for a C library are just two instances of a more general concept of searchers. A searcher is simply a
function that takes the module name and returns either a loader for that module or nil if it cannot find one.

The array package.searchers lists the searchers that require uses. When looking for a module,
require calls each searcher in the list passing the module name, until one of them finds a loader for the
module. If the list ends without a positive response, require raises an error.

The use of a list to drive the search for a module allows great flexibility to require. For instance, if we
want to store modules compressed in zip files, we only need to provide a proper searcher function for that
and add it to the list. In its default configuration, the searcher for Lua files and the searcher for C libraries
that we described earlier are respectively the second and the third elements in the list. Before them, there
is the preload searcher.

The preload searcher allows the definition of an arbitrary function to load a module. It uses a table, called
package.preload, to map module names to loader functions. When searching for a module name, this
searcher simply looks for the given name in the table. If it finds a function there, it returns this function
as the module loader. Otherwise, it returns nil. This searcher provides a generic method to handle some
non-conventional situations. For instance, a C library statically linked to Lua can register its luaopen_
function into the preload table, so that it will be called only when (and if) the user requires that module.
In this way, the program does not waste resources opening the module if it is not used.

The default content of package.searchers includes a fourth function that is relevant only for sub-
modules. We will discuss it at the section called “Submodules and Packages”.

The Basic Approach for Writing Modules in Lua

The simplest way to create a module in Lua is really simple: we create a table, put all functions we want
to export inside it, and return this table. Figure 17.2, “A simple module for complex numbers” illustrates
this approach.

135
 Modules and Packages

Figure 17.2. A simple module for complex numbers

local M = {} -- the module

-- creates a new complex number

local function new (r, i)
return {r=r, i=i}
end

M.new = new -- add 'new' to the module

-- constant 'i'
M.i = new(0, 1)

function M.add (c1, c2)

return new(c1.r + c2.r, c1.i + c2.i)
end

function M.sub (c1, c2)

return new(c1.r - c2.r, c1.i - c2.i)
end

function M.mul (c1, c2)

return new(c1.r*c2.r - c1.i*c2.i, c1.r*c2.i + c1.i*c2.r)
end

local function inv (c)

local n = c.r^2 + c.i^2
return new(c.r/n, -c.i/n)
end

function M.div (c1, c2)

return M.mul(c1, inv(c2))
end

function M.tostring (c)

return string.format("(%g,%g)", c.r, c.i)
end

return M

Note how we define new and inv as private functions simply by declaring them local to the chunk.

Some people do not like the final return statement. One way of eliminating it is to assign the module table
directly into package.loaded:

local M = {}
package.loaded[...] = M
as before, without the return statement

Remember that require calls the loader passing the module name as the first argument. So, the vararg
expression ... in the table index results in that name. After this assignment, we do not need to return M
at the end of the module: if a module does not return a value, require will return the current value of
package.loaded[modname] (if it is not nil). Anyway, I find it clearer to write the final return. If we
forget it, any trivial test with the module will detect the error.

136
 Modules and Packages

Another approach to write a module is to define all functions as locals and build the returning table at the
end, as in Figure 17.3, “Module with export list”.

Figure 17.3. Module with export list

local function new (r, i) return {r=r, i=i} end

-- defines constant 'i'

local i = complex.new(0, 1)

other functions follow the same pattern

return {
new = new,
i = i,
add = add,
sub = sub,
mul = mul,
div = div,
tostring = tostring,
}

What are the advantages of this approach? We do not need to prefix each name with M. or something
similar; there is an explicit export list; and we define and use exported and internal functions in the same
way inside the module. What are the disadvantages? The export list is at the end of the module instead of
at the beginning, where it would be more useful as a quick documentation; and the export list is somewhat
redundant, as we must write each name twice. (This last disadvantage may become an advantage, as it
allows functions to have different names inside and outside the module, but I think programmers seldom
do this.)

Anyway, remember that no matter how we define a module, users should be able to use it in a standard way:

local cpx = require "complex"

print(cpx.tostring(cpx.add(cpx.new(3,4), cpx.i)))
--> (3,5)

Later, we will see how we can use some advanced Lua features, such as metatables and environments, for
writing modules. However, except for a nice technique to detect global variables created by mistake, I use
only the basic approach in my modules.

Submodules and Packages

Lua allows module names to be hierarchical, using a dot to separate name levels. For instance, a module
named mod.sub is a submodule of mod. A package is a complete tree of modules; it is the unit of dis-
tribution in Lua.

When we require a module called mod.sub, the function require will query first the ta-
ble package.loaded and then the table package.preload, using the original module name
"mod.sub" as the key. Here, the dot is just a character like any other in the module name.

However, when searching for a file that defines that submodule, require translates the dot into another
character, usually the system's directory separator (e.g., a slash for POSIX or a backslash for Windows).
After the translation, require searches for the resulting name like any other name. For instance, assume
the slash as the directory separator and the following path:

137
 Modules and Packages

./?.lua;/usr/local/lua/?.lua;/usr/local/lua/?/init.lua

The call require "a.b" will try to open the following files:

./a/b.lua
/usr/local/lua/a/b.lua
/usr/local/lua/a/b/init.lua

This behavior allows all modules of a package to live in a single directory. For instance, if a package has
modules p, p.a, and p.b, their respective files can be p/init.lua, p/a.lua, and p/b.lua, with
the directory p within some appropriate directory.

The directory separator used by Lua is configured at compile time and can be any string (remember,
Lua knows nothing about directories). For instance, systems without hierarchical directories can use an
underscore as the “directory separator”, so that require "a.b" will search for a file a_b.lua.

Names in C cannot contain dots, so a C library for submodule a.b cannot export a function
luaopen_a.b. Here, require translates the dot into another character, an underscore. So, a C library
named a.b should name its initialization function luaopen_a_b.

As an extra facility, require has one more searcher for loading C submodules. When it cannot find either
a Lua file or a C file for a submodule, this last searcher searches again the C path, but this time looking
for the package name. For example, if the program requires a submodule a.b.c this searcher will look
for a. If it finds a C library for this name, then require looks into this library for an appropriate open
function, luaopen_a_b_c in this example. This facility allows a distribution to put several submodules
together, each with its own open function, into a single C library.

From the point of view of Lua, submodules in the same package have no explicit relationship. Requiring
a module does not automatically load any of its submodules; similarly, requiring a submodule does not
automatically load its parent. Of course, the package implementer is free to create these links if she wants.
For instance, a particular module may start by explicitly requiring one or all of its submodules.

Exercises
Exercise 17.1: Rewrite the implementation of double-ended queues (Figure 14.2, “A double-ended queue”)
as a proper module.

Exercise 17.2: Rewrite the implementation of the geometric-region system (the section called “A Taste of
Functional Programming”) as a proper module.

Exercise 17.3: What happens in the search for a library if the path has some fixed component (that is, a
component without a question mark)? Can this behavior be useful?

Exercise 17.4: Write a searcher that searches for Lua files and C libraries at the same time. For instance,
the path used for this searcher could be something like this:

./?.lua;./?.so;/usr/lib/lua5.2/?.so;/usr/share/lua5.2/?.lua

(Hint: use package.searchpath to find a proper file and then try to load it, first with loadfile
and next with package.loadlib.)

138
Part III. Lua-isms
Table of Contents
18. Iterators and the Generic for ........................................................................................ 142
Iterators and Closures ............................................................................................... 142
The Semantics of the Generic for ............................................................................... 143
Stateless Iterators ..................................................................................................... 145
Traversing Tables in Order ........................................................................................ 146
True Iterators .......................................................................................................... 147
19. Interlude: Markov Chain Algorithm ............................................................................... 149
20. Metatables and Metamethods ........................................................................................ 152
Arithmetic Metamethods ........................................................................................... 152
Relational Metamethods ............................................................................................ 155
Library-Defined Metamethods .................................................................................... 155
Table-Access Metamethods ....................................................................................... 156
The __index metamethod ............................................................................... 156
The __newindex metamethod ......................................................................... 157
Tables with default values ................................................................................. 158
Tracking table accesses ..................................................................................... 159
Read-only tables .............................................................................................. 160
21. Object-Oriented Programming ...................................................................................... 162
Classes ................................................................................................................... 163
Inheritance .............................................................................................................. 165
Multiple Inheritance ................................................................................................. 166
Privacy ................................................................................................................... 168
The Single-Method Approach .................................................................................... 170
Dual Representation ................................................................................................. 170
22. The Environment ........................................................................................................ 173
Global Variables with Dynamic Names ....................................................................... 173
Global-Variable Declarations ..................................................................................... 174
Non-Global Environments ......................................................................................... 176
Using _ENV ............................................................................................................ 177
Environments and Modules ....................................................................................... 180
_ENV and load ...................................................................................................... 181
23. Garbage .................................................................................................................... 183
Weak Tables ........................................................................................................... 183
Memorize Functions ................................................................................................. 184
Object Attributes ..................................................................................................... 185
Revisiting Tables with Default Values ......................................................................... 186
Ephemeron Tables ................................................................................................... 187
Finalizers ................................................................................................................ 188
The Garbage Collector .............................................................................................. 190
Controlling the Pace of Collection .............................................................................. 191
24. Coroutines ................................................................................................................. 194
Coroutine Basics ...................................................................................................... 194
Who Is the Boss? .................................................................................................... 196
Coroutines as Iterators .............................................................................................. 198
Event-Driven Programming ....................................................................................... 200
25. Reflection .................................................................................................................. 205
Introspective Facilities .............................................................................................. 205
Accessing local variables .................................................................................. 207
Accessing non-local variables ............................................................................ 208
Accessing other coroutines ................................................................................ 209
Hooks .................................................................................................................... 210

140
 Lua-isms

Profiles .................................................................................................................. 211

Sandboxing ............................................................................................................. 212
26. Interlude: Multithreading with Coroutines ....................................................................... 217

141
Chapter 18. Iterators and the Generic
for
We have been using the generic for for several tasks through the book, such as reading the lines of a file
or traversing the matches of a pattern over a subject. However, we still do not know how to create our own
iterators. In this chapter, we will fill this gap. Starting with simple iterators, we will learn how to use all
the power of the generic for to write all kinds of iterators.

Iterators and Closures

An iterator is any construction that allows us to iterate over the elements of a collection. In Lua, we
typically represent iterators by functions: each time we call the function, it returns the “next” element from
the collection. A typical example is io.read: each time we call it, it returns the next line of the standard
input file, returning nil when there are no more lines to be read.

Any iterator needs to keep some state between successive calls, so that it knows where it is and how
to proceed from there. For io.read, C keeps that state in its stream structure. For our own iterators,
closures provide an excellent mechanism for keeping state. Remember that a closure is a function that
accesses one or more local variables from its enclosing environment. These variables keep their values
across successive calls to the closure, allowing the closure to remember where it is along a traversal. Of
course, to create a new closure we must also create its non-local variables. Therefore, a closure construction
typically involves two functions: the closure itself and a factory, the function that creates the closure plus
its enclosing variables.

As an example, let us write a simple iterator for a list. Unlike ipairs, this iterator does not return the
index of each element, only its value:

function values (t)

local i = 0
return function () i = i + 1; return t[i] end
end

In this example, values is the factory. Each time we call this factory, it creates a new closure (the iterator
itself). This closure keeps its state in its external variables t and i, which are also created by values.
Each time we call the iterator, it returns a next value from the list t. After the last element the iterator
returns nil, which signals the end of the iteration.

We can use this iterator in a while loop:

t = {10, 20, 30}

iter = values(t) -- creates the iterator
while true do
local element = iter() -- calls the iterator
if element == nil then break end
print(element)
end

However, it is easier to use the generic for. After all, it was designed for this kind of iteration:

t = {10, 20, 30}

for element in values(t) do

142
 Iterators and the Generic for

print(element)
end

The generic for does all the bookkeeping for an iteration loop: it keeps the iterator function internally, so
that we do not need the iter variable; it calls the iterator for each new iteration; and it stops the loop
when the iterator returns nil. (In the next section, we will see that the generic for does even more than that.)

As a more advanced example, Figure 18.1, “Iterator to traverse all words from the standard input” shows
an iterator to traverse all the words from the standard input.

Figure 18.1. Iterator to traverse all words from the standard input
function allwords ()
local line = io.read() -- current line
local pos = 1 -- current position in the line
return function () -- iterator function
while line do -- repeat while there are lines
local w, e = string.match(line, "(%w+)()", pos)
if w then -- found a word?
pos = e -- next position is after this word
return w -- return the word
else
line = io.read() -- word not found; try next line
pos = 1 -- restart from first position
end
end
return nil -- no more lines: end of traversal
end
end

To do this traversal, we keep two values: the contents of the current line (variable line), and where we
are on this line (variable pos). With this data, we can always generate the next word. The main part of the
iterator function is the call to string.match, which searches for a word in the current line starting at the
current position. It describes a “word” using the pattern '%w+', which matches one or more alphanumeric
characters. If it finds a word, it captures and returns the word and the position of the first character after it
(with an empty capture). The function then updates the current position and returns this word. Otherwise,
the iterator reads a new line and repeats the search. If there are no more lines, it returns nil to signal the
end of the iteration.

Despite its complexity, the use of allwords is straightforward:

for word in allwords() do

print(word)
end

This is a common situation with iterators: they may not be easy to write, but they are easy to use. This
is not a big problem; more often than not, end users programming in Lua do not define iterators, but just
use those provided by the application.

The Semantics of the Generic for

One drawback of those previous iterators is that we need to create a new closure to initialize each new
loop. For many situations, this is not a real problem. For instance, in the allwords iterator, the cost of
creating one single closure is negligible compared to the cost of reading a whole file. However, in some

143
 Iterators and the Generic for

situations this overhead can be inconvenient. In such cases, we can use the generic for itself to keep the
iteration state. In this section, we will see the facilities that the generic for offers to hold state.

We saw that the generic for keeps the iterator function internally, during the loop. Actually, it keeps three
values: the iterator function, an invariant state, and a control variable. Let us see the details now.

The syntax for the generic for is as follows:

for var-list in exp-list do

body
end

Here, var-list is a list of one or more variable names, separated by commas, and exp-list is a list of
one or more expressions, also separated by commas. Usually the expression list has only one element, a call
to an iterator factory. In the next code, for instance, the list of variables is k,v and the list of expressions
has the single element pairs(t):

for k, v in pairs(t) do print(k, v) end

We call the first (or only) variable in the list the control variable. Its value is never nil during the loop,
because when it becomes nil the loop ends.

The first thing the for does is to evaluate the expressions after the in. These expressions should result in
the three values kept by the for: the iterator function, the invariant state, and the initial value for the control
variable. Like in a multiple assignment, only the last (or the only) element of the list can result in more
than one value; and the number of values is adjusted to three, extra values being discarded or nils added
as needed. For instance, when we use simple iterators, the factory returns only the iterator function, so the
invariant state and the control variable get nil.

After this initialization step, the for calls the iterator function with two arguments: the invariant state and
the control variable. From the standpoint of the for construct, the invariant state has no meaning at all.
The for only passes the state value from the initialization step to all calls to the iterator function. Then the
for assigns the values returned by the iterator function to the variables declared by its variable list. If the
first returned value (the one assigned to the control variable) is nil, the loop terminates. Otherwise, the for
executes its body and calls the iteration function again, repeating the process.

More precisely, a construction like

for var_1, ..., var_n in explist do block end

is equivalent to the following code:

do
local _f, _s, _var = explist
while true do
local var_1, ... , var_n = _f(_s, _var)
_var = var_1
if _var == nil then break end
block
end
end

So, if our iterator function is f, the invariant state is s, and the initial value for the control variable is a0,
the control variable will loop over the values a1 = f(s, a0), a2 = f(s, a1), and so on, until ai is nil. If the for
has other variables, they simply get the extra values returned by each call to f.

144
 Iterators and the Generic for

Stateless Iterators
As the name implies, a stateless iterator is an iterator that does not keep any state by itself. Therefore, we
can use the same stateless iterator in multiple loops, avoiding the cost of creating new closures.

As we just saw, the for loop calls its iterator function with two arguments: the invariant state and the control
variable. A stateless iterator generates the next element for the iteration using only these two values. A
typical example of this kind of iterator is ipairs, which iterates over all elements of a sequence:

a = {"one", "two", "three"}

for i, v in ipairs(a) do
print(i, v)
end

The whole state of the iteration comprises the table being traversed (the invariant state, which does not
change during the loop), plus the current index (the control variable). Both ipairs (the factory) and the
iterator are quite simple; we could write them in Lua as follows:

local function iter (t, i)

i = i + 1
local v = t[i]
if v then
return i, v
end
end

function ipairs (t)

return iter, t, 0
end

When Lua calls ipairs(t) in a for loop, it gets three values: the function iter as the iterator, the table
t as the invariant state, and zero as the initial value for the control variable. Then, Lua calls iter(t,
0), which results in 1,t[1] (unless t[1] is already nil). In the second iteration, Lua calls iter(t,
1), which results in 2,t[2], and so on, until the first nil element.

The function pairs, which iterates over all elements of a table, is similar, except that its iterator function
is next, which is a primitive function in Lua:

function pairs (t)

return next, t, nil
end

The call next(t, k), where k is a key of the table t, returns a next key in the table, in an arbitrary
order, plus the value associated with this key as a second return value. The call next(t, nil) returns
a first pair. When there are no more pairs, next returns nil.

We might use next directly, without calling pairs:

for k, v in next, t do
loop body
end

Remember that the for loop adjusts its expression list to three results, so that it gets next, t, and nil; this
is exactly what it gets when it calls pairs(t).

145
 Iterators and the Generic for

Another interesting example of a stateless iterator is one to traverse a linked list. (Linked lists are not
frequent in Lua, but sometimes we need them.) A first thought could be to use only the current node as
the control variable, so that the iterator function could return its next node:

local function getnext (node)

return node.next
end

function traverse (list)

return getnext, nil, list
end

However, this implementation would skip the first node. Instead, we can use the following code:

local function getnext (list, node)

if not node then
return list
else
return node.next
end
end

function traverse (list)

return getnext, list, nil
end

The trick here is to use the first node as the invariant state (the second value returned by traverse),
besides the current node as the control variable. The first time the iterator function getnext is called,
node will be nil, and so the function will return list as the first node. In subsequent calls, node will
not be nil, and so the iterator will return node.next, as expected.

Traversing Tables in Order

A common confusion happens when programmers try to order the entries of a table. In a table, the entries
form a set, and have no order whatsoever. If we want to order them, we have to copy the keys to an array
and then sort the array.

We saw an example of this technique in the “Most Frequent Words” program, in Chapter 11, Interlude:
Most Frequent Words. Let us see here another example. Suppose that we read a source file and build a
table that gives, for each function name, the line where this function is defined; something like this:

lines = {
["luaH_set"] = 10,
["luaH_get"] = 24,
["luaH_present"] = 48,
}

Now we want to print these function names in alphabetical order. If we traverse this table with pairs,
the names appear in an arbitrary order. We cannot sort them directly, because these names are keys of the
table. However, when we put them into an array, then we can sort them. First, we must create an array
with these names, then sort it, and finally print the result:

a = {}
for n in pairs(lines) do a[#a + 1] = n end
table.sort(a)

146
 Iterators and the Generic for

for _, n in ipairs(a) do print(n) end

Some people get confused here. After all, for Lua, arrays also have no order (they are tables, after all).
But we know how to count! So, we impose the order, when we access the array with ordered indices. That
is why we should always traverse arrays with ipairs, rather than pairs. The first function imposes
the key order 1, 2, etc., whereas the latter uses the natural arbitrary order of the table (which may not be
what we need, even though usually it is).

Now we are ready to write an iterator that traverses a table following the order of its keys:

function pairsByKeys (t, f)

local a = {}
for n in pairs(t) do -- create a list with all keys
a[#a + 1] = n
end
table.sort(a, f) -- sort the list
local i = 0 -- iterator variable
return function () -- iterator function
i = i + 1
return a[i], t[a[i]] -- return key, value
end
end

The factory function pairsByKeys first collects the keys into an array, then it sorts the array, and finally
it returns the iterator function. At each step, the iterator returns the next key and value from the original
table, following the order in the array a. An optional parameter f allows the specification of an alternative
order.

With this function, it is easy to solve our initial problem of traversing a table in order:

for name, line in pairsByKeys(lines) do

print(name, line)
end

As usual, all the complexity is hidden inside the iterator.

True Iterators
The name “iterator” is a little misleading, because our iterators do not iterate: what iterates is the for loop.
Iterators only provide the successive values for the iteration. Maybe a better name would be “generator” —
which generates elements for the iteration— but “iterator” is already well established in other languages,
such as Java.

However, there is another way to build iterators wherein iterators actually do the iteration. When we use
such iterators, we do not write a loop; instead, we simply call the iterator with an argument that describes
what the iterator must do at each iteration. More specifically, the iterator receives as argument a function
that it calls inside its loop.

As a concrete example, let us rewrite once more the allwords iterator using this style:

function allwords (f)

for line in io.lines() do
for word in string.gmatch(line, "%w+") do
f(word) -- call the function
end

147
 Iterators and the Generic for

end
end

To use this iterator, we must supply the loop body as a function. If we want only to print each word, we
simply use print:

allwords(print)

Often, we use an anonymous function as the body. For instance, the next code fragment counts how many
times the word “hello” appears in the input file:

local count = 0
allwords(function (w)
if w == "hello" then count = count + 1 end
end)
print(count)

The same task, written with the previous iterator style, is not very different:

local count = 0
for w in allwords() do
if w == "hello" then count = count + 1 end
end
print(count)

True iterators were popular in older versions of Lua, when the language did not have the for statement.
How do they compare with generator-style iterators? Both styles have approximately the same overhead:
one function call per iteration. On the one hand, it is easier to write the iterator with true iterators (although
we can recover this easiness with coroutines, as we will see in the section called “Coroutines as Iterators”).
On the other hand, the generator style is more flexible. First, it allows two or more parallel iterations.
(For instance, consider the problem of iterating over two files comparing them word by word.) Second,
it allows the use of break and return inside the iterator body. With a true iterator, a return returns from
the anonymous function, not from the function doing the iteration. For these reasons, overall I usually
prefer generators.

Exercises
Exercise 18.1: Write an iterator fromto such that the next loop becomes equivalent to a numeric for:

for i in fromto(n, m) do
body
end

Can you implement it as a stateless iterator?

Exercise 18.2: Add a step parameter to the iterator from the previous exercise. Can you still implement
it as a stateless iterator?

Exercise 18.3: Write an iterator uniquewords that returns all words from a given file without repetitions.
(Hint: start with the allwords code in Figure 18.1, “Iterator to traverse all words from the standard
input”; use a table to keep all words already reported.)

Exercise 18.4: Write an iterator that returns all non-empty substrings of a given string.

Exercise 18.5: Write a true iterator that traverses all subsets of a given set. (Instead of creating a new table
for each subset, it can use the same table for all its results, only changing its contents between iterations.)

148
Chapter 19. Interlude: Markov Chain
Algorithm
Our next complete program is an implementation of the Markov chain algorithm, described by Kernighan
& Pike in their book The Practice of Programming (Addison-Wesley, 1999).

The program generates pseudo-random text based on what words can follow a sequence of n previous
words in a base text. For this implementation, we will assume that n is two.

The first part of the program reads the base text and builds a table that, for each prefix of two words,
gives a list of the words that follow that prefix in the text. After building the table, the program uses it to
generate random text, wherein each word follows two previous words with the same probability as in the
base text. As a result, we have text that is very, but not quite, random. For instance, when applied to this
book, the output of the program has pieces like this: “Constructors can also traverse a table constructor,
then the parentheses in the following line does the whole file in a field n to store the contents of each
function, but to show its only argument. If you want to find the maximum element in an array can return
both the maximum value and continues showing the prompt and running the code. The following words
are reserved and cannot be used to convert between degrees and radians.”

To use a two-word prefix as a key in tables, we will represent it by the two words concatenated with a
space in between:

function prefix (w1, w2)

return w1 .. " " .. w2
end

We use the string NOWORD (a newline) to initialize the prefix words and to mark the end of the text. For
instance, for the text "the more we try the more we do" the table of following words would
be like this:

{ ["\n \n"] = {"the"},

["\n the"] = {"more"},
["the more"] = {"we", "we"},
["more we"] = {"try", "do"},
["we try"] = {"the"},
["try the"] = {"more"},
["we do"] = {"\n"},
}

The program keeps its table in the variable statetab. To insert a new word in a list in this table, we
use the following function:

function insert (prefix, value)

local list = statetab[prefix]
if list == nil then
statetab[prefix] = {value}
else
list[#list + 1] = value
end
end

It first checks whether that prefix already has a list; if not, it creates a new one with the new value. Other-
wise, it inserts the new value at the end of the existing list.

149
 Interlude: Markov Chain Algorithm

To build the statetab table, we keep two variables, w1 and w2, with the last two words read. We
read the words using the iterator allwords, from the section called “Iterators and Closures”, but we
adapted the definition of “word” to include optional punctuation marks, such as commas and periods (see
Figure 19.1, “Auxiliary definitions for the Markov program”). For each new word read, we add it to the
list associated with w1–w2 and then update w1 and w2.

After building the table, the program starts to generate a text with MAXGEN words. First, it re-initializes
variables w1 and w2. Then, for each prefix, it chooses a next word randomly from the list of valid next
words, prints this word, and updates w1 and w2. Figure 19.1, “Auxiliary definitions for the Markov pro-
gram” and Figure 19.2, “The Markov program” show the complete program.

Figure 19.1. Auxiliary definitions for the Markov program

function allwords ()
local line = io.read() -- current line
local pos = 1 -- current position in the line
return function () -- iterator function
while line do -- repeat while there are lines
local w, e = string.match(line, "(%w+[,;.:]?)()", pos)
if w then -- found a word?
pos = e -- update next position
return w -- return the word
else
line = io.read() -- word not found; try next line
pos = 1 -- restart from first position
end
end
return nil -- no more lines: end of traversal
end
end

function prefix (w1, w2)

return w1 .. " " .. w2
end

local statetab = {}

function insert (prefix, value)

local list = statetab[prefix]
if list == nil then
statetab[prefix] = {value}
else
list[#list + 1] = value
end
end

150
 Interlude: Markov Chain Algorithm

Figure 19.2. The Markov program

local MAXGEN = 200
local NOWORD = "\n"

-- build table
local w1, w2 = NOWORD, NOWORD
for nextword in allwords() do
insert(prefix(w1, w2), nextword)
w1 = w2; w2 = nextword;
end
insert(prefix(w1, w2), NOWORD)

-- generate text
w1 = NOWORD; w2 = NOWORD -- reinitialize
for i = 1, MAXGEN do
local list = statetab[prefix(w1, w2)]
-- choose a random item from list
local r = math.random(#list)
local nextword = list[r]
if nextword == NOWORD then return end
io.write(nextword, " ")
w1 = w2; w2 = nextword
end

Exercises
Exercise 19.1: Generalize the Markov-chain algorithm so that it can use any size for the sequence of
previous words used in the choice of the next word.

151
Chapter 20. Metatables and
Metamethods
Usually, each value in Lua has a quite predictable set of operations. We can add numbers, we can con-
catenate strings, we can insert key–value pairs into tables, and so on. However, we cannot add tables, we
cannot compare functions, and we cannot call a string. Unless we use metatables.

Metatables allow us to change the behavior of a value when confronted with an unknown operation. For
instance, using metatables, we can define how Lua computes the expression a + b, where a and b are
tables. Whenever Lua tries to add two tables, it checks whether either of them has a metatable and whether
this metatable has an __add field. If Lua finds this field, it calls the corresponding value —the so-called
metamethod, which should be a function— to compute the sum.

We can think about metatables as a restricted kind of classes, in object-oriented terminology. Like class-
es, metatables define the behavior of its instances. However, metatables are more restricted than classes,
because they can only give behavior to a predefined set of operations; also, metatables do not have in-
heritance. Nevertheless, we will see in Chapter 21, Object-Oriented Programming how to build a quite
complete class system on top of metatables.

Each value in Lua can have a metatable. Tables and userdata have individual metatables; values of other
types share one single metatable for all values of that type. Lua always creates new tables without metat-
ables:

t = {}
print(getmetatable(t)) --> nil

We can use setmetatable to set or change the metatable of a table:

t1 = {}
setmetatable(t, t1)
print(getmetatable(t) == t1) --> true

From Lua, we can set the metatables only of tables; to manipulate the metatables of values of other types we
must use C code or the debug library. (The main reason for this restriction is to curb excessive use of type-
wide metatables. Experience with older versions of Lua has shown that those global settings frequently
lead to non-reusable code.) The string library sets a metatable for strings; all other types by default have
no metatable:

print(getmetatable("hi")) --> table: 0x80772e0

print(getmetatable("xuxu")) --> table: 0x80772e0
print(getmetatable(10)) --> nil
print(getmetatable(print)) --> nil

Any table can be the metatable of any value; a group of related tables can share a common metatable,
which describes their common behavior; a table can be its own metatable, so that it describes its own
individual behavior. Any configuration is valid.

Arithmetic Metamethods
In this section, we will introduce a running example to explain the basics of metatables. Suppose we have
a module that uses tables to represent sets, with functions to compute set union, intersection, and the like,
as shown in Figure 20.1, “A simple module for sets”.

152
 Metatables and Metamethods

Figure 20.1. A simple module for sets

local Set = {}

-- create a new set with the values of a given list

function Set.new (l)
local set = {}
for _, v in ipairs(l) do set[v] = true end
return set
end

function Set.union (a, b)

local res = Set.new{}
for k in pairs(a) do res[k] = true end
for k in pairs(b) do res[k] = true end
return res
end

function Set.intersection (a, b)

local res = Set.new{}
for k in pairs(a) do
res[k] = b[k]
end
return res
end

-- presents a set as a string

function Set.tostring (set)
local l = {} -- list to put all elements from the set
for e in pairs(set) do
l[#l + 1] = tostring(e)
end
return "{" .. table.concat(l, ", ") .. "}"
end

return Set

Now, we want to use the addition operator to compute the union of two sets. For that, we will arrange for
all tables representing sets to share a metatable. This metatable will define how they react to the addition
operator. Our first step is to create a regular table, which we will use as the metatable for sets:

local mt = {} -- metatable for sets

The next step is to modify Set.new, which creates sets. The new version has only one extra line, which
sets mt as the metatable for the tables that it creates:

function Set.new (l) -- 2nd version

local set = {}
setmetatable(set, mt)
for _, v in ipairs(l) do set[v] = true end
return set
end

After that, every set we create with Set.new will have that same table as its metatable:

153
 Metatables and Metamethods

s1 = Set.new{10, 20, 30, 50}

s2 = Set.new{30, 1}
print(getmetatable(s1)) --> table: 0x00672B60
print(getmetatable(s2)) --> table: 0x00672B60

Finally, we add to the metatable the metamethod __add, a field that describes how to perform the addition:

mt.__add = Set.union

After that, whenever Lua tries to add two sets, it will call Set.union, with the two operands as argu-
ments.

With the metamethod in place, we can use the addition operator to do set unions:

s3 = s1 + s2
print(Set.tostring(s3)) --> {1, 10, 20, 30, 50}

Similarly, we may set the multiplication operator to perform set intersection:

mt.__mul = Set.intersection

print(Set.tostring((s1 + s2)*s1)) --> {10, 20, 30, 50}

For each arithmetic operator there is a corresponding metamethod name. Besides addition and multiplica-
tion, there are metamethods for subtraction (__sub), float division (__div), floor division (__idiv),
negation (__unm), modulo (__mod), and exponentiation (__pow). Similarly, there are metamethods for
all bitwise operations: bitwise AND (__band), OR (__bor), exclusive OR (__bxor), NOT (__bnot),
left shift (__shl), and right shift (__shr). We may define also a behavior for the concatenation operator,
with the field __concat.

When we add two sets, there is no question about what metatable to use. However, we may write an
expression that mixes two values with different metatables, for instance like this:

s = Set.new{1,2,3}
s = s + 8

When looking for a metamethod, Lua performs the following steps: if the first value has a metatable with
the required metamethod, Lua uses this metamethod, independently of the second value; otherwise, if the
second value has a metatable with the required metamethod, Lua uses it; otherwise, Lua raises an error.
Therefore, the last example will call Set.union, as will the expressions 10 + s and "hello" + s
(because both numbers and strings do not have a metamethod __add).

Lua does not care about these mixed types, but our implementation does. If we run the s = s + 8
example, we will get an error inside the function Set.union:

bad argument #1 to 'pairs' (table expected, got number)

If we want more lucid error messages, we must check the type of the operands explicitly before attempting
to perform the operation, for instance with code like this:

function Set.union (a, b)

if getmetatable(a) ~= mt or getmetatable(b) ~= mt then
error("attempt to 'add' a set with a non-set value", 2)
end
as before

Remember that the second argument to error (2, in this example) sets the source location in the error
message to the code that called the operation.

154
 Metatables and Metamethods

Relational Metamethods
Metatables also allow us to give meaning to the relational operators, through the metamethods __eq
(equal to), __lt (less than), and __le (less than or equal to). There are no separate metamethods for
the other three relational operators: Lua translates a ~= b to not (a == b), a > b to b < a,
and a >= b to b <= a.

In older versions, Lua translated all order operators to a single one, by translating a <= b to not (b
< a). However, this translation is incorrect when we have a partial order, that is, when not all elements
in our type are properly ordered. For instance, most machines do not have a total order for floating-point
numbers, because of the value Not a Number (NaN). According to the IEEE 754 standard, NaN represents
undefined values, such as the result of 0/0. The standard specifies that any comparison that involves NaN
should result in false. This means that NaN <= x is always false, but x < NaN is also false. It also
means that the translation from a <= b to not (b < a) is not valid in this case.

In our example with sets, we have a similar problem. An obvious (and useful) meaning for <= in sets is
set containment: a <= b means that a is a subset of b. With this meaning, it is possible that a <= b
and b < a are both false. Therefore, we must implement both __le (less or equal, the subset relation)
and __lt (less than, the proper subset relation):

mt.__le = function (a, b) -- subset

for k in pairs(a) do
if not b[k] then return false end
end
return true
end

mt.__lt = function (a, b) -- proper subset

return a <= b and not (b <= a)
end

Finally, we can define set equality through set containment:

mt.__eq = function (a, b)

return a <= b and b <= a
end

After these definitions, we are ready to compare sets:

s1 = Set.new{2, 4}
s2 = Set.new{4, 10, 2}
print(s1 <= s2) --> true
print(s1 < s2) --> true
print(s1 >= s1) --> true
print(s1 > s1) --> false
print(s1 == s2 * s1) --> true

The equality comparison has some restrictions. If two objects have different basic types, the equality
operation results in false, without even calling any metamethod. So, a set will always be different from a
number, no matter what its metamethod says.

Library-Defined Metamethods
So far, all the metamethods we have seen are for the Lua core. It is the virtual machine who detects that the
values involved in an operation have metatables with metamethods for that particular operation. However,

155
 Metatables and Metamethods

because metatables are regular tables, anyone can use them. So, it is a common practice for libraries to
define and use their own fields in metatables.

The function tostring provides a typical example. As we saw earlier, tostring represents tables in
a rather simple format:

print({}) --> table: 0x8062ac0

The function print always calls tostring to format its output. However, when formatting any value,
tostring first checks whether the value has a __tostring metamethod. In this case, tostring
calls the metamethod to do its job, passing the object as an argument. Whatever this metamethod returns
is the result of tostring.

In our example with sets, we have already defined a function to present a set as a string. So, we need only
to set the __tostring field in the metatable:

mt.__tostring = Set.tostring

After that, whenever we call print with a set as its argument, print calls tostring that calls
Set.tostring:

s1 = Set.new{10, 4, 5}
print(s1) --> {4, 5, 10}

The functions setmetatable and getmetatable also use a metafield, in this case to protect metat-
ables. Suppose we want to protect our sets, so that users can neither see nor change their metatables. If we
set a __metatable field in the metatable, getmetatable will return the value of this field, whereas
setmetatable will raise an error:

mt.__metatable = "not your business"

s1 = Set.new{}
print(getmetatable(s1)) --> not your business
setmetatable(s1, {})
stdin:1: cannot change protected metatable

Since Lua 5.2, pairs also have a metamethod, so that we can modify the way a table is traversed and
add a traversal behavior to non-table objects. When an object has a __pairs metamethod, pairs will
call it to do all its work.

Table-Access Metamethods
The metamethods for arithmetic, bitwise, and relational operators all define behavior for otherwise erro-
neous situations; they do not change the normal behavior of the language. Lua also offers a way to change
the behavior of tables for two normal situations, the access and modification of absent fields in a table.

The __index metamethod

We saw earlier that, when we access an absent field in a table, the result is nil. This is true, but it is not
the whole truth. Actually, such accesses trigger the interpreter to look for an __index metamethod: if
there is no such method, as usually happens, then the access results in nil; otherwise, the metamethod will
provide the result.

The archetypal example here is inheritance. Suppose we want to create several tables describing windows.
Each table must describe several window parameters, such as position, size, color scheme, and the like.
All these parameters have default values and so we want to build window objects giving only the non-

156
 Metatables and Metamethods

default parameters. A first alternative is to provide a constructor that fills in the absent fields. A second
alternative is to arrange for the new windows to inherit any absent field from a prototype window. First,
we declare the prototype:

-- create the prototype with default values

prototype = {x = 0, y = 0, width = 100, height = 100}

Then, we define a constructor function, which creates new windows sharing a metatable:

local mt = {} -- create a metatable

-- declare the constructor function
function new (o)
setmetatable(o, mt)
return o
end

Now, we define the __index metamethod:

mt.__index = function (_, key)

return prototype[key]
end

After this code, we create a new window and query it for an absent field:

w = new{x=10, y=20}
print(w.width) --> 100

Lua detects that w does not have the requested field, but has a metatable with an __index field. So,
Lua calls this __index metamethod, with arguments w (the table) and "width" (the absent key). The
metamethod then indexes the prototype with the given key and returns the result.

The use of the __index metamethod for inheritance is so common that Lua provides a shortcut. Despite
being called a method, the __index metamethod does not need to be a function: it can be a table, instead.
When it is a function, Lua calls it with the table and the absent key as its arguments, as we have just seen.
When it is a table, Lua redoes the access in this table. Therefore, in our previous example, we could declare
__index simply like this:

mt.__index = prototype

Now, when Lua looks for the metatable's __index field, it finds the value of prototype, which
is a table. Consequently, Lua repeats the access in this table, that is, it executes the equivalent of
prototype["width"]. This access then gives the desired result.

The use of a table as an __index metamethod provides a fast and simple way of implementing single
inheritance. A function, although more expensive, provides more flexibility: we can implement multiple
inheritance, caching, and several other variations. We will discuss these forms of inheritance in Chapter 21,
Object-Oriented Programming, when we will cover object-oriented programming.

When we want to access a table without invoking its __index metamethod, we use the function rawget.
The call rawget(t, i) does a raw access to table t, that is, a primitive access without considering
metatables. Doing a raw access will not speed up our code (the overhead of a function call kills any gain
we could have), but sometimes we need it, as we will see later.

The __newindex metamethod

The __newindex metamethod does for table updates what __index does for table accesses. When we
assign a value to an absent index in a table, the interpreter looks for a __newindex metamethod: if there

157
 Metatables and Metamethods

is one, the interpreter calls it instead of making the assignment. Like __index, if the metamethod is a
table, the interpreter does the assignment in this table, instead of in the original one. Moreover, there is a
raw function that allows us to bypass the metamethod: the call rawset(t, k, v) does the equivalent
to t[k] = v without invoking any metamethod.

The combined use of the __index and __newindex metamethods allows several powerful constructs in
Lua, such as read-only tables, tables with default values, and inheritance for object-oriented programming.
In this chapter, we will see some of these uses. Object-oriented programming has its own chapter, later.

Tables with default values

The default value of any field in a regular table is nil. It is easy to change this default value with metatables:

function setDefault (t, d)

local mt = {__index = function () return d end}
setmetatable(t, mt)
end

tab = {x=10, y=20}

print(tab.x, tab.z) --> 10 nil
setDefault(tab, 0)
print(tab.x, tab.z) --> 10 0

After the call to setDefault, any access to an absent field in tab calls its __index metamethod,
which returns zero (the value of d for this metamethod).

The function setDefault creates a new closure plus a new metatable for each table that needs a default
value. This can be expensive if we have many tables that need default values. However, the metatable has
the default value d wired into its metamethod, so we cannot use a single metatable for tables with different
default values. To allow the use of a single metatable for all tables, we can store the default value of each
table in the table itself, using an exclusive field. If we are not worried about name clashes, we can use a
key like "___" for our exclusive field:

local mt = {__index = function (t) return t.___ end}

function setDefault (t, d)
t.___ = d
setmetatable(t, mt)
end

Note that now we create the metatable mt and its corresponding metamethod only once, outside SetDe-
fault.

If we are worried about name clashes, it is easy to ensure the uniqueness of the special key. All we need
is a new exclusive table to use as the key:

local key = {} -- unique key

local mt = {__index = function (t) return t[key] end}
function setDefault (t, d)
t[key] = d
setmetatable(t, mt)
end

An alternative approach for associating each table with its default value is a technique I call dual represen-
tation, which uses a separate table where the indices are the tables and the values are their default values.
However, for the correct implementation of this approach, we need a special breed of table called weak
tables, and so we will not use it here; we will return to the subject in Chapter 23, Garbage.

158
 Metatables and Metamethods

Another alternative is to memorize metatables in order to reuse the same metatable for tables with the same
default. However, that needs weak tables too, so that again we will have to wait until Chapter 23, Garbage.

Tracking table accesses

Suppose we want to track every single access to a certain table. Both __index and __newindex are
relevant only when the index does not exist in the table. So, the only way to catch all accesses to a table
is to keep it empty. If we want to monitor all accesses to a table, we should create a proxy for the real
table. This proxy is an empty table, with proper metamethods that track all accesses and redirect them to
the original table. The code in Figure 20.2, “Tracking table accesses” implements this idea.

Figure 20.2. Tracking table accesses

function track (t)
local proxy = {} -- proxy table for 't'

-- create metatable for the proxy

local mt = {
__index = function (_, k)
print("*access to element " .. tostring(k))
return t[k] -- access the original table
end,

__newindex = function (_, k, v)

print("*update of element " .. tostring(k) ..
" to " .. tostring(v))
t[k] = v -- update original table
end,

__pairs = function ()
return function (_, k) -- iteration function
local nextkey, nextvalue = next(t, k)
if nextkey ~= nil then -- avoid last value
print("*traversing element " .. tostring(nextkey))
end
return nextkey, nextvalue
end
end,

__len = function () return #t end

}

setmetatable(proxy, mt)

return proxy
end

The following example illustrates its use:

> t = {} -- an arbitrary table

> t = track(t)
> t[2] = "hello"
--> *update of element 2 to hello
> print(t[2])

159
 Metatables and Metamethods

--> *access to element 2

--> hello

The metamethods __index and __newindex follow the guidelines that we set, tracking each access
and then redirecting it to the original table. The __pairs metamethod allows us to traverse the proxy as
if it were the original table, tracking the accesses along the way. Finally, the __len metamethod gives
the length operator through the proxy:

t = track({10, 20})
print(#t) --> 2
for k, v in pairs(t) do print(k, v) end
--> *traversing element 1
--> 1 10
--> *traversing element 2
--> 2 20

If we want to monitor several tables, we do not need a different metatable for each one. Instead, we can
somehow map each proxy to its original table and share a common metatable for all proxies. This problem
is similar to the problem of associating tables to their default values, which we discussed in the previous
section, and allows the same solutions. For instance, we can keep the original table in a proxy's field, using
an exclusive key, or we can use a dual representation to map each proxy to its corresponding table.

Read-only tables
It is easy to adapt the concept of proxies to implement read-only tables. All we have to do is to raise an
error whenever we track any attempt to update the table. For the __index metamethod, we can use a
table —the original table itself— instead of a function, as we do not need to track queries; it is simpler
and rather more efficient to redirect all queries to the original table. This use demands a new metatable for
each read-only proxy, with __index pointing to the original table:

function readOnly (t)

local proxy = {}
local mt = { -- create metatable
__index = t,
__newindex = function (t, k, v)
error("attempt to update a read-only table", 2)
end
}
setmetatable(proxy, mt)
return proxy
end

As an example of use, we can create a read-only table for weekdays:

days = readOnly{"Sunday", "Monday", "Tuesday", "Wednesday",

"Thursday", "Friday", "Saturday"}

print(days[1]) --> Sunday

days[2] = "Noday"
--> stdin:1: attempt to update a read-only table

Exercises
Exercise 20.1: Define a metamethod __sub for sets that returns the difference of two sets. (The set a -
b is the set of elements from a that are not in b.)

160
 Metatables and Metamethods

Exercise 20.2: Define a metamethod __len for sets so that #s returns the number of elements in the set s.

Exercise 20.3: An alternative way to implement read-only tables might use a function as the __index
metamethod. This alternative makes accesses more expensive, but the creation of read-only tables is cheap-
er, as all read-only tables can share a single metatable. Rewrite the function readOnly using this ap-
proach.

Exercise 20.4: Proxy tables can represent other kinds of objects besides tables. file as array Write a function
fileAsArray that takes the name of a file and returns a proxy to that file, so that after a call t =
fileAsArray("myFile"), an access to t[i] returns the i-th byte of that file, and an assignment
to t[i] updates its i-th byte.

Exercise 20.5: Extend the previous example to allow us to traverse the bytes in the file with pairs(t)
and get the file length with #t.

161
Chapter 21. Object-Oriented
Programming
A table in Lua is an object in more than one sense. Like objects, tables have a state. Like objects, tables
have an identity (a self) that is independent of their values; specifically, two objects (tables) with the same
value are different objects, whereas an object can have different values at different times. Like objects,
tables have a life cycle that is independent of who created them or where they were created.

Objects have their own operations. Tables also can have operations, as in the following fragment:

Account = {balance = 0}
function Account.withdraw (v)
Account.balance = Account.balance - v
end

This definition creates a new function and stores it in field withdraw of the object Account. Then,
we can call it like here:

Account.withdraw(100.00)

This kind of function is almost what we call a method. However, the use of the global name Account
inside the function is a horrible programming practice. First, this function will work only for this particular
object. Second, even for this particular object, the function will work only as long as the object is stored
in that particular global variable. If we change the object's name, withdraw does not work any more:

a, Account = Account, nil

a.withdraw(100.00) -- ERROR!

Such behavior violates the principle that objects have independent life cycles.

A more principled approach is to operate on the receiver of the operation. For that, our method would need
an extra parameter with the value of the receiver. This parameter usually has the name self or this:

function Account.withdraw (self, v)

self.balance = self.balance - v
end

Now, when we call the method we have to specify the object that it has to operate on:

a1 = Account; Account = nil

...
a1.withdraw(a1, 100.00) -- OK

With the use of a self parameter, we can use the same method for many objects:

a2 = {balance=0, withdraw = Account.withdraw}

...
a2.withdraw(a2, 260.00)

This use of a self parameter is a central point in any object-oriented language. Most OO languages have
this mechanism hidden from the programmer, so that she does not have to declare this parameter (although
she still can use the name self or this inside a method). Lua also can hide this parameter, with the colon

162
 Object-Oriented Programming

operator. Using it, we can rewrite the previous method call as a2:withdraw(260.00) and the pre-
vious definition as here:

function Account:withdraw (v)

self.balance = self.balance - v
end

The effect of the colon is to add an extra argument in a method call and to add an extra hidden parameter
in a method definition. The colon is only a syntactic facility, although a convenient one; there is nothing
really new here. We can define a function with the dot syntax and call it with the colon syntax, or vice-
versa, as long as we handle the extra parameter correctly:

Account = { balance=0,
withdraw = function (self, v)
self.balance = self.balance - v
end
}

function Account:deposit (v)

self.balance = self.balance + v
end

Account.deposit(Account, 200.00)
Account:withdraw(100.00)

Classes
So far, our objects have an identity, state, and operations on this state. They still lack a class system,
inheritance, and privacy. Let us tackle the first problem: how can we create several objects with similar
behavior? Specifically, how can we create several accounts?

Most object-oriented languages offer the concept of class, which works as a mold for the creation of objects.
In such languages, each object is an instance of a specific class. Lua does not have the concept of class;
the concept of metatables is somewhat similar, but using it as a class would not lead us too far. Instead, we
can emulate classes in Lua following the lead from prototype-based languages like Self. (Javascript also
followed that path.) In these languages, objects have no classes. Instead, each object may have a prototype,
which is a regular object where the first object looks up any operation that it does not know about. To
represent a class in such languages, we simply create an object to be used exclusively as a prototype for
other objects (its instances). Both classes and prototypes work as a place to put behavior to be shared by
several objects.

In Lua, we can implement prototypes using the idea of inheritance that we saw in the section called “The
__index metamethod”. More specifically, if we have two objects A and B, all we have to do to make
B a prototype for A is this:

setmetatable(A, {__index = B})

After that, A looks up in B for any operation that it does not have. To see B as the class of the object A
is not much more than a change in terminology.

Let us go back to our example of a bank account. To create other accounts with behavior similar to Ac-
count, we arrange for these new objects to inherit their operations from Account, using the __index
metamethod.

163
 Object-Oriented Programming

local mt = {__index = Account}

function Account.new (o)

o = o or {} -- create table if user does not provide one
setmetatable(o, mt)
return o
end

After this code, what happens when we create a new account and call a method on it, like this?

a = Account.new{balance = 0}
a:deposit(100.00)

When we create the new account, a, it will have mt as its metatable. When we call
a:deposit(100.00), we are actually calling a.deposit(a, 100.00); the colon is only syntac-
tic sugar. However, Lua cannot find a "deposit" entry in the table a; hence, Lua looks into the __in-
dex entry of the metatable. The situation now is more or less like this:

getmetatable(a).__index.deposit(a, 100.00)

The metatable of a is mt, and mt.__index is Account. Therefore, the previous expression evaluates
to this one:

Account.deposit(a, 100.00)

That is, Lua calls the original deposit function, but passing a as the self parameter. So, the new account
a inherited the function deposit from Account. By the same mechanism, it inherits all fields from
Account.

We can make two small improvements on this scheme. The first one is that we do not need to create a new
table for the metatable role; instead, we can use the Account table itself for that purpose. The second
one is that we can use the colon syntax for the new method, too. With these two changes, method new
becomes like this:

function Account:new (o)

o = o or {}
self.__index = self
setmetatable(o, self)
return o
end

Now, when we call Account:new(), the hidden parameter self gets Account as its value, we make
Account.__index also equal to Account, and set Account as the metatable for the new object. It
may seem that we do not gained much with the second change (the colon syntax); the advantage of using
self will become apparent when we introduce class inheritance, in the next section.

The inheritance works not only for methods, but also for other fields that are absent in the new account.
Therefore, a class can provide not only methods, but also constants and default values for its instance
fields. Remember that, in our first definition of Account, we provided a field balance with value 0.
So, if we create a new account without an initial balance, it will inherit this default value:

b = Account:new()
print(b.balance) --> 0

When we call the deposit method on b, it runs the equivalent of the following code, because self is b:

164
 Object-Oriented Programming

b.balance = b.balance + v

The expression b.balance evaluates to zero and the method assigns an initial deposit to b.balance.
Subsequent accesses to b.balance will not invoke the index metamethod, because now b has its own
balance field.

Inheritance
Because classes are objects, they can get methods from other classes, too. This behavior makes inheritance
(in the usual object-oriented meaning) quite easy to implement in Lua.

Let us assume we have a base class like Account, in Figure 21.1, “the Account class”.

Figure 21.1. the Account class

Account = {balance = 0}

function Account:new (o)

o = o or {}
self.__index = self
setmetatable(o, self)
return o
end

function Account:deposit (v)

self.balance = self.balance + v
end

function Account:withdraw (v)

if v > self.balance then error"insufficient funds" end
self.balance = self.balance - v
end

From this class, we want to derive a subclass SpecialAccount that allows the customer to withdraw
more than his balance. We start with an empty class that simply inherits all its operations from its base class:

SpecialAccount = Account:new()

Up to now, SpecialAccount is just an instance of Account. The magic happens now:

s = SpecialAccount:new{limit=1000.00}

SpecialAccount inherits new from Account, like any other method. This time, however, when new
executes, its self parameter will refer to SpecialAccount. Therefore, the metatable of s will be
SpecialAccount, whose value at field __index is also SpecialAccount. So, s inherits from
SpecialAccount, which inherits from Account. Later, when we evaluate s:deposit(100.00),
Lua cannot find a deposit field in s, so it looks into SpecialAccount; it cannot find a deposit
field there, too, so it looks into Account; there it finds the original implementation for a deposit.

What makes a SpecialAccount special is that we can redefine any method inherited from its super-
class. All we have to do is to write the new method:

function SpecialAccount:withdraw (v)

165
 Object-Oriented Programming

if v - self.balance >= self:getLimit() then

error"insufficient funds"
end
self.balance = self.balance - v
end

function SpecialAccount:getLimit ()
return self.limit or 0
end

Now, when we call s:withdraw(200.00), Lua does not go to Account, because it finds the new
withdraw method in SpecialAccount first. As s.limit is 1000.00 (remember that we set this
field when we created s), the program does the withdrawal, leaving s with a negative balance.

An interesting aspect of objects in Lua is that we do not need to create a new class to specify a new
behavior. If only a single object needs a specific behavior, we can implement that behavior directly in
the object. For instance, if the account s represents some special client whose limit is always 10% of her
balance, we can modify only this single account:

function s:getLimit ()
return self.balance * 0.10
end

After this declaration, the call s:withdraw(200.00) runs the withdraw method from Spe-
cialAccount, but when withdraw calls self:getLimit, it is this last definition that it invokes.

Multiple Inheritance
Because objects are not primitive in Lua, there are several ways to do object-oriented programming in
Lua. The approach that we have seen, using the index metamethod, is probably the best combination
of simplicity, performance, and flexibility. Nevertheless, there are other implementations, which may be
more appropriate for some particular cases. Here we will see an alternative implementation that allows
multiple inheritance in Lua.

The key to this implementation is the use of a function for the metafield __index. Remember that, when
a table's metatable has a function in the __index field, Lua will call this function whenever it cannot find
a key in the original table. Then, __index can look up for the missing key in how many parents it wants.

Multiple inheritance means that a class does not have a unique superclass. Therefore, we should not use a
(super)class method to create subclasses. Instead, we will define an independent function for this purpose,
createClass, which has as arguments all superclasses of the new class; see Figure 21.2, “An imple-
mentation of multiple inheritance”. This function creates a table to represent the new class and sets its
metatable with an __index metamethod that does the multiple inheritance. Despite the multiple inheri-
tance, each object instance still belongs to one single class, where it looks for all its methods. Therefore,
the relationship between classes and superclasses is different from the relationship between instances and
classes. Particularly, a class cannot be the metatable for its instances and for its subclasses at the same
time. In Figure 21.2, “An implementation of multiple inheritance”, we keep the class as the metatable for
its instances, and create another table to be the metatable of the class.

166
 Object-Oriented Programming

Figure 21.2. An implementation of multiple inheritance

-- look up for 'k' in list of tables 'plist'
local function search (k, plist)
for i = 1, #plist do
local v = plist[i][k] -- try 'i'-th superclass
if v then return v end
end
end

function createClass (...)

local c = {} -- new class
local parents = {...} -- list of parents

-- class searches for absent methods in its list of parents

setmetatable(c, {__index = function (t, k)
return search(k, parents)
end})

-- prepare 'c' to be the metatable of its instances

c.__index = c

-- define a new constructor for this new class

function c:new (o)
o = o or {}
setmetatable(o, c)
return o
end

return c -- return new class

end

Let us illustrate the use of createClass with a small example. Assume our previous class Account
and another class, Named, with only two methods: setname and getname.

Named = {}
function Named:getname ()
return self.name
end

function Named:setname (n)

self.name = n
end

To create a new class NamedAccount that is a subclass of both Account and Named, we simply call
createClass:

NamedAccount = createClass(Account, Named)

To create and to use instances, we do as usual:

account = NamedAccount:new{name = "Paul"}

print(account:getname()) --> Paul

Now let us follow how Lua evaluates the expression account:getname(); more specifically, let us
follow the evaluation of account["getname"]. Lua cannot find the field "getname" in account;

167
 Object-Oriented Programming

so, it looks for the field __index on the account's metatable, which is NamedAccount in our exam-
ple. But NamedAccount also cannot provide a "getname" field, so Lua looks for the field __index
of NamedAccount's metatable. Because this field contains a function, Lua calls it. This function then
looks for "getname" first in Account, without success, and then in Named, where it finds a non-nil
value, which is the final result of the search.

Of course, due to the underlying complexity of this search, the performance of multiple inheritance is not
the same as single inheritance. A simple way to improve this performance is to copy inherited methods
into the subclasses. Using this technique, the index metamethod for classes would be like this:

setmetatable(c, {__index = function (t, k)

local v = search(k, parents)
t[k] = v -- save for next access
return v
end})

With this trick, accesses to inherited methods are as fast as to local methods, except for the first access.
The drawback is that it is difficult to change method definitions after the program has started, because
these changes do not propagate down the hierarchy chain.

Privacy
Many people consider privacy (also called information hiding) to be an integral part of an object-oriented
language: the state of each object should be its own internal affair. In some object-oriented languages, such
as C++ and Java, we can control whether a field (also called an instance variable) or a method is visible
outside the object. Smalltalk, which popularized object-oriented languages, makes all variables private and
all methods public. Simula, the first ever object-oriented language, did not offer any kind of protection.

The standard implementation of objects in Lua, which we have shown previously, does not offer privacy
mechanisms. Partly, this is a consequence of our use of a general structure (tables) to represent objects.
Moreover, Lua avoids redundancy and artificial restrictions. If you do not want to access something that
lives inside an object, just do not do it. A common practice is to mark all private names with an underscore
at the end. You immediately feel the smell when you see a marked name being used in public.

Nevertheless, another aim of Lua is to be flexible, offering the programmer meta-mechanisms that enable
her to emulate many different mechanisms. Although the basic design for objects in Lua does not offer
privacy mechanisms, we can implement objects in a different way, to have access control. Although pro-
grammers do not use this implementation frequently, it is instructive to know about it, both because it
explores some interesting aspects of Lua and because it is a good solution for more specific problems.

The basic idea of this alternative design is to represent each object through two tables: one for its state
and another for its operations, or its interface. We access the object itself through the second table, that
is, through the operations that compose its interface. To avoid unauthorized access, the table representing
the state of an object is not kept in a field of the other table; instead, it is kept only in the closure of the
methods. For instance, to represent our bank account with this design, we could create new objects running
the following factory function:

function newAccount (initialBalance)

local self = {balance = initialBalance}

local withdraw = function (v)

self.balance = self.balance - v
end

168
 Object-Oriented Programming

local deposit = function (v)

self.balance = self.balance + v
end

local getBalance = function () return self.balance end

return {
withdraw = withdraw,
deposit = deposit,
getBalance = getBalance
}
end

First, the function creates a table to keep the internal object state and stores it in the local variable self.
Then, the function creates the methods of the object. Finally, the function creates and returns the external
object, which maps method names to the actual method implementations. The key point here is that these
methods do not get self as an extra parameter; instead, they access self directly. Because there is no
extra argument, we do not use the colon syntax to manipulate such objects. We call their methods just
like regular functions:

acc1 = newAccount(100.00)
acc1.withdraw(40.00)
print(acc1.getBalance()) --> 60

This design gives full privacy to anything stored in the self table. After the call to newAccount returns,
there is no way to gain direct access to this table. We can access it only through the functions created
inside newAccount. Although our example puts only one instance variable into the private table, we can
store all private parts of an object in this table. We can also define private methods: they are like public
methods, but we do not put them in the interface. For instance, our accounts may give an extra credit of
10% for users with balances above a certain limit, but we do not want the users to have access to the details
of that computation. We can implement this functionality as follows:

function newAccount (initialBalance)

local self = {
balance = initialBalance,
LIM = 10000.00,
}

local extra = function ()

if self.balance > self.LIM then
return self.balance*0.10
else
return 0
end
end

local getBalance = function ()

return self.balance + extra()
end

as before

Again, there is no way for any user to access the function extra directly.

169
 Object-Oriented Programming

The Single-Method Approach

A particular case of the previous approach for object-oriented programming occurs when an object has
a single method. In such cases, we do not need to create an interface table; instead, we can return this
single method as the object representation. If this sounds a little weird, it is worth remembering iterators
like io.lines or string.gmatch. An iterator that keeps state internally is nothing more than a sin-
gle-method object.

Another interesting case of single-method objects occurs when this single-method is actually a dispatch
method that performs different tasks based on a distinguished argument. A prototypical implementation
for such an object is as follows:

function newObject (value)

return function (action, v)
if action == "get" then return value
elseif action == "set" then value = v
else error("invalid action")
end
end
end

Its use is straightforward:

d = newObject(0)
print(d("get")) --> 0
d("set", 10)
print(d("get")) --> 10

This unconventional implementation for objects is quite effective. The syntax d("set", 10), although
peculiar, is only two characters longer than the more conventional d:set(10). Each object uses one
single closure, which is usually cheaper than one table. There is no inheritance, but we have full privacy:
the only way to access an object state is through its sole method.

Tcl/Tk uses a similar approach for its widgets. The name of a widget in Tk denotes a function (a widget
command) that can perform all kinds of operations over the widget, according to its first argument.

Dual Representation
Another interesting approach for privacy uses a dual representation. Let us start seeing what a dual rep-
resentation is.

Usually, we associate attributes to tables using keys, like this:

table[key] = value

However, we can use a dual representation: we can use a table to represent a key, and use the object itself
as a key in that table:

key = {}
...
key[table] = value

A key ingredient here is the fact that we can index tables in Lua not only with numbers and strings, but
with any value —in particular with other tables.

170
 Object-Oriented Programming

As an example, in our Account implementation, we could keep the balances of all accounts in a table
balance, instead of keeping them in the accounts themselves. Our withdraw method would become
like this:

function Account.withdraw (self, v)

balance[self] = balance[self] - v
end

What we gain here? Privacy. Even if a function has access to an account, it cannot directly access its
balance unless it also has access to the table balance. If the table balance is kept in a local inside
the module Account, only functions inside the module can access it and, therefore, only those functions
can manipulate account balances.

Before we go on, I must discuss a big naivety of this implementation. Once we use an account as a key in
the balance table, that account will never become garbage for the garbage collector. It will be anchored
there until some code explicitly removes it from that table. That may not be a problem for bank accounts
(as an account usually has to be formally closed before going away), but for other scenarios that could
be a big drawback. In the section called “Object Attributes”, we will see how to solve this problem. For
now, we will ignore it.

Figure 21.3, “Accounts using a dual representation” shows again an implementation for accounts, this time
using a dual representation.

Figure 21.3. Accounts using a dual representation

local balance = {}

Account = {}

function Account:withdraw (v)

balance[self] = balance[self] - v
end

function Account:deposit (v)

balance[self] = balance[self] + v
end

function Account:balance ()
return balance[self]
end

function Account:new (o)

o = o or {} -- create table if user does not provide one
setmetatable(o, self)
self.__index = self
balance[o] = 0 -- initial balance
return o
end

We use this class just like any other one:

a = Account:new{}
a:deposit(100.00)
print(a:balance())

171
 Object-Oriented Programming

However, we cannot tamper with an account balance. By keeping the table balance private to the mod-
ule, this implementation ensures its safety.

Inheritance works without modifications. This approach has a cost quite similar to the standard one, both
in terms of time and of memory. New objects need one new table and one new entry in each private table
being used. The access balance[self] can be slightly slower than self.balance, because the
latter uses a local variable while the first uses an external variable. Usually this difference is negligible.
As we will see later, it also demands some extra work from the garbage collector.

Exercises
Exercise 21.1: Implement a class Stack, with methods push, pop, top, and isempty.

Exercise 21.2: Implement a class StackQueue as a subclass of Stack. Besides the inherited methods,
add to this class a method insertbottom, which inserts an element at the bottom of the stack. (This
method allows us to use objects of this class as queues.)

Exercise 21.3: Reimplement your Stack class using a dual representation.

Exercise 21.4: A variation of the dual representation is to implement objects using proxies (the section
called “Tracking table accesses”). Each object is represented by an empty proxy table. An internal table
maps proxies to tables that carry the object state. This internal table is not accessible from the outside,
but methods use it to translate their self parameters to the real tables where they operate. Implement the
Account example using this approach and discuss its pros and cons.

172
Chapter 22. The Environment
Global variables are a necessary evil of most programming languages. On one hand, the use of global
variables can easily lead to complex code, entangling apparently unrelated parts of a program. On the other
hand, the judicious use of global variables can better express truly global aspects of a program; moreover,
global constants are innocuous, but dynamic languages like Lua have no way to distinguish constants
from variables. An embedded language like Lua adds another ingredient to this mix: a global variable is
a variable that is visible in the whole program, but Lua has no clear concept of a program, having instead
pieces of code (chunks) called by the host application.

Lua solves this conundrum by not having global variables, but going to great lengths to pretend it has. In
a first approximation, we can think that Lua keeps all its global variables in a regular table, called the
global environment. Later in this chapter, we will see that Lua can keep its “global” variables in several
environments. For now, we will stick to that first approximation.

The use of a table to store global variables simplifies the internal implementation of Lua, because there
is no need for a different data structure for global variables. Another advantage is that we can manipulate
this table like any other table. To help such manipulations, Lua stores the global environment itself in the
global variable _G. (As a result, _G._G is equal to _G.) For instance, the following code prints the names
of all the variables defined in the global environment:

for n in pairs(_G) do print(n) end

Global Variables with Dynamic Names

Usually, assignment is enough for accessing and setting global variables. However, sometimes we need
some form of meta-programming, such as when we need to manipulate a global variable whose name is
stored in another variable or is somehow computed at run time. To get the value of such a variable, some
programmers are tempted to write something like this:

value = load("return " .. varname)()

If varname is x, for example, the concatenation will result in "return x", which when run achieves
the desired result. However, this code involves the creation and compilation of a new chunk, which is
somewhat expensive. We can accomplish the same effect with the following code, which is more than an
order of magnitude more efficient than the previous one:

value = _G[varname]

Because the environment is a regular table, we can simply index it with the desired key (the variable name).

In a similar way, we can assign a value to a global variable whose name is computed dynamically by writing
_G[varname] = value. Beware, however: some programmers get a little excited with these facilities
and end up writing code like _G["a"] = _G["b"], which is just a complicated way to write a = b.

A generalization of the previous problem is to allow fields in the dynamic name, such as "io.read"
or "a.b.c.d". If we write _G["io.read"], clearly we will not get the field read from the table
io. But we can write a function getfield such that getfield("io.read") returns the expected
result. This function is mainly a loop, which starts at _G and evolves field by field:

function getfield (f)

local v = _G -- start with the table of globals

173
 The Environment

for w in string.gmatch(f, "[%a_][%w_]*") do

v = v[w]
end
return v
end

We rely on gmatch to iterate over all identifiers in f.

The corresponding function to set fields is a little more complex. An assignment like a.b.c.d = v is
equivalent to the following code:

local temp = a.b.c

temp.d = v

That is, we must retrieve up to the last name and then handle this last name separately. The function
setfield, in Figure 22.1, “The function setfield”, does the task and also creates intermediate tables
in a path when they do not exist.

Figure 22.1. The function setfield

function setfield (f, v)

local t = _G -- start with the table of globals
for w, d in string.gmatch(f, "([%a_][%w_]*)(%.?)") do
if d == "." then -- not last name?
t[w] = t[w] or {} -- create table if absent
t = t[w] -- get the table
else -- last name
t[w] = v -- do the assignment
end
end
end

The pattern there captures the field name in the variable w and an optional following dot in the variable d.
If a field name is not followed by a dot, then it is the last name.

With the previous functions in place, the next call creates a global table t, another table t.x, and assigns
10 to t.x.y:

setfield("t.x.y", 10)

print(t.x.y) --> 10
print(getfield("t.x.y")) --> 10

Global-Variable Declarations
Global variables in Lua do not need declarations. Although this behavior is handy for small programs,
in larger programs a simple typo can cause bugs that are difficult to find. However, we can change this
behavior if we like. Because Lua keeps its global variables in a regular table, we can use metatables to
detect when Lua accesses non-existent variables.

A first approach simply detects any access to absent keys in the global table:

setmetatable(_G, {

174
 The Environment

__newindex = function (_, n)

error("attempt to write to undeclared variable " .. n, 2)
end,
__index = function (_, n)
error("attempt to read undeclared variable " .. n, 2)
end,
})

After this code, any attempt to access a non-existent global variable will trigger an error:

> print(a)
stdin:1: attempt to read undeclared variable a

But how do we declare new variables? One option is to use rawset, which bypasses the metamethod:

function declare (name, initval)

rawset(_G, name, initval or false)
end

(The or with false ensures that the new global always gets a value different from nil.)

A simpler option is to restrict assignments to new global variables only inside functions, allowing free
assignments in the outer level of a chunk.

To check whether an assignment is in the main chunk, we must use the debug library. The call
debug.getinfo(2, "S") returns a table whose field what tells whether the function that called the
metamethod is a main chunk, a regular Lua function, or a C function. (We will see debug.getinfo
in more detail in the section called “Introspective Facilities”.) Using this function, we can rewrite the
__newindex metamethod like this:

__newindex = function (t, n, v)

local w = debug.getinfo(2, "S").what
if w ~= "main" and w ~= "C" then
error("attempt to write to undeclared variable " .. n, 2)
end
rawset(t, n, v)
end

This new version also accepts assignments from C code, as this kind of code usually knows what it is doing.

If we need to test whether a variable exists, we cannot simply compare it to nil because, if it is nil, the
access will raise an error. Instead, we use rawget, which avoids the metamethod:

if rawget(_G, var) == nil then

-- 'var' is undeclared
...
end

As it is, our scheme does not allow global variables with nil values, as they would be automatically con-
sidered undeclared. But it is not difficult to correct this problem. All we need is an auxiliary table that keeps
the names of declared variables. Whenever a metamethod is called, it checks in this table whether the vari-
able is undeclared. The code can be like the one in Figure 22.2, “Checking global-variable declaration”.

175
 The Environment

Figure 22.2. Checking global-variable declaration

local declaredNames = {}

setmetatable(_G, {
__newindex = function (t, n, v)
if not declaredNames[n] then
local w = debug.getinfo(2, "S").what
if w ~= "main" and w ~= "C" then
error("attempt to write to undeclared variable "..n, 2)
end
declaredNames[n] = true
end
rawset(t, n, v) -- do the actual set
end,

__index = function (_, n)

if not declaredNames[n] then
error("attempt to read undeclared variable "..n, 2)
else
return nil
end
end,
})

Now, even an assignment like x = nil is enough to declare a global variable.

The overhead for both solutions is negligible. With the first solution, the metamethods are never called
during normal operation. In the second, they can be called, but only when the program accesses a variable
holding a nil.

The Lua distribution comes with a module strict.lua that implements a global-variable check that
uses essentially the code in Figure 22.2, “Checking global-variable declaration”. It is a good habit to use
it when developing Lua code.

Non-Global Environments
In Lua, global variables do not need to be truly global. As I already hinted, Lua does not even have global
variables. That may sound strange at first, as we have been using global variables all along this text. As
I said, Lua goes to great lengths to give the programmer an illusion of global variables. Now we will see
how Lua builds this illusion.1

First, let us forget about global variables. Instead, we will start with the concept of free names. A free
name is a name that is not bound to an explicit declaration, that is, it does not occur inside the scope of a
corresponding local variable. For instance, both x and y are free names in the following chunk, but z is not:

local z = 10
x = y + z

Now comes the important part: The Lua compiler translates any free name x in the chunk to _ENV.x. So,
the previous chunk is fully equivalent to this one:

local z = 10
_ENV.x = _ENV.y + z
1
This mechanism was one of the parts of Lua that changed most from version 5.1 to 5.2. Very little of the following discussion applies to Lua 5.1.

176
 The Environment

But what is this new _ENV variable?

_ENV cannot be a global variable; we just said that Lua has no global variables. Again, the compiler does
the trick. I already mentioned that Lua treats any chunk as an anonymous function. Actually, Lua compiles
our original chunk as the following code:

local _ENV = some value

return function (...)
local z = 10
_ENV.x = _ENV.y + z
end

That is, Lua compiles any chunk in the presence of a predefined upvalue (an external local variable) called
_ENV. So, any variable is either local, if it is a bounded name, or a field in _ENV, which itself is a local
variable (an upvalue).

The initial value for _ENV can be any table. (Actually, it does not need to be a table; more about that later.)
Any such table is called an environment. To preserve the illusion of global variables, Lua keeps internally a
table that it uses as a global environment. Usually, when we load a chunk, the function load initializes this
predefined upvalue with that global environment. So, our original chunk becomes equivalent to this one:

local _ENV = the global environment

return function (...)
local z = 10
_ENV.x = _ENV.y + z
end

The result of all these arrangements is that the x field of the global environment gets the value of the y
field plus 10.

At first sight, this may seem a rather convoluted way to manipulate global variables. I will not argue that
it is the simplest way, but it offers a flexibility that is difficult to achieve with a simpler implementation.

Before we go on, let us summarize the handling of global variables in Lua:

• The compiler creates a local variable _ENV outside any chunk that it compiles.

• The compiler translates any free name var to _ENV.var.

• The function load (or loadfile) initializes the first upvalue of a chunk with the global environment,
which is a regular table kept internally by Lua.

After all, it is not that complicated.

Some people get confused because they try to infer extra magic from these rules. There is no extra magic.
In particular, the first two rules are done entirely by the compiler. Except for being predefined by the
compiler, _ENV is a plain regular variable. Outside the compiler, the name _ENV has no special meaning
at all to Lua.2 Similarly, the translation from x to _ENV.x is a plain syntactic translation, with no hidden
meanings. In particular, after the translation, _ENV will refer to whatever _ENV variable is visible at that
point in the code, following the standard visibility rules.

Using _ENV
In this section, we will see some ways to explore the flexibility brought by _ENV. Keep in mind that we
must run most examples in this section as a single chunk. If we enter code line by line in interactive mode,
2
To be completely honest, Lua uses that name for error messages, so that it reports an error involving a variable _ENV.x as being about global x.

177
 The Environment

each line becomes a different chunk and therefore each will have a distinct _ENV variable. To run a piece
of code as a single chunk, we can either run it from a file or enclose it in a do--end block.

Because _ENV is a regular variable, we can assign to and access it as any other variable. The assignment
_ENV = nil will invalidate any direct access to global variables in the rest of the chunk. This can be
useful to control what variables our code uses:

local print, sin = print, math.sin

_ENV = nil
print(13) --> 13
print(sin(13)) --> 0.42016703682664
print(math.cos(13)) -- error!

Any assignment to a free name (a “global variable”) will raise a similar error.

We can write the _ENV explicitly to bypass a local declaration:

a = 13 -- global
local a = 12
print(a) --> 12 (local)
print(_ENV.a) --> 13 (global)

We can do the same with _G:

a = 13 -- global
local a = 12
print(a) --> 12 (local)
print(_G.a) --> 13 (global)

Usually, _G and _ENV refer to the same table but, despite that, they are quite different entities. _ENV is a
local variable, and all accesses to “global variables” in reality are accesses to it. _G is a global variable with
no special status whatsoever. By definition, _ENV always refers to the current environment; _G usually
refers to the global environment, provided it is visible and no one changed its value.

The main use for _ENV is to change the environment used by a piece of code. Once we change the envi-
ronment, all global accesses will use the new table:

-- change current environment to a new empty table

_ENV = {}
a = 1 -- create a field in _ENV
print(a)
--> stdin:4: attempt to call global 'print' (a nil value)

If the new environment is empty, we have lost all our global variables, including print. So, we should
first populate it with some useful values, for instance with the global environment:

a = 15 -- create a global variable

_ENV = {g = _G} -- change current environment
a = 1 -- create a field in _ENV
g.print(_ENV.a, g.a) --> 1 15

Now, when we access the “global” g (which lives in _ENV, not in the global environment) we get the
global environment, wherein Lua will find the function print.

We can rewrite the previous example using the name _G instead of g:

a = 15 -- create a global variable

178
 The Environment

_ENV = {_G = _G} -- change current environment

a = 1 -- create a field in _ENV
_G.print(_ENV.a, _G.a) --> 1 15

The only special status of _G happens when Lua creates the initial global table and makes its field _G points
to itself. Lua does not care about the current value of this variable. Nevertheless, it is customary to use this
same name whenever we have a reference to the global environment, as we did in the rewritten example.

Another way to populate our new environment is with inheritance:

a = 1
local newgt = {} -- create new environment
setmetatable(newgt, {__index = _G})
_ENV = newgt -- set it
print(a) --> 1

In this code, the new environment inherits both print and a from the global one. However, any assign-
ment goes to the new table. There is no danger of changing a variable in the global environment by mistake,
although we still can change them through _G:

-- continuing the previous chunk

a = 10
print(a, _G.a) --> 10 1
_G.a = 20
print(_G.a) --> 20

Being a regular variable, _ENV follows the usual scoping rules. In particular, functions defined inside a
chunk access _ENV as they access any other external variable:

_ENV = {_G = _G}

local function foo ()
_G.print(a) -- compiled as '_ENV._G.print(_ENV.a)'
end
a = 10
foo() --> 10
_ENV = {_G = _G, a = 20}
foo() --> 20

If we define a new local variable called _ENV, references to free names will bind to that new variable:

a = 2
do
local _ENV = {print = print, a = 14}
print(a) --> 14
end
print(a) --> 2 (back to the original _ENV)

Therefore, it is not difficult to define a function with a private environment:

function factory (_ENV)

return function () return a end
end

f1 = factory{a = 6}
f2 = factory{a = 7}
print(f1()) --> 6

179
 The Environment

print(f2()) --> 7

The factory function creates simple closures that return the value of their “global” a. When the closure
is created, its visible _ENV variable is the parameter _ENV of the enclosing factory function; therefore,
each closure will use its own external variable (as an upvalue) to access its free names.

Using the usual scoping rules, we can manipulate environments in several other ways. For instance, we
may have several functions sharing a common environment, or a function that changes the environment
that it shares with other functions.

Environments and Modules

In the section called “The Basic Approach for Writing Modules in Lua”, when we discussed how to write
modules, I mentioned that one drawback of those methods was that it was all too easy to pollute the
global space, for instance by forgetting a local in a private declaration. Environments offer an interesting
technique for solving that problem. Once the module main chunk has an exclusive environment, not only
all its functions share this table, but also all its global variables go to this table. We can declare all public
functions as global variables and they will go to a separate table automatically. All the module has to do
is to assign this table to the _ENV variable. After that, when we declare a function add, it goes to M.add:

local M = {}
_ENV = M
function add (c1, c2)
return new(c1.r + c2.r, c1.i + c2.i)
end

Moreover, we can call other functions from the same module without any prefix. In the previous code,
add gets new from its environment, that is, it calls M.new.

This method offers a good support for modules, with little extra work for the programmer. It needs no
prefixes at all. There is no difference between calling an exported function and a private one. If the pro-
grammer forgets a local, he does not pollute the global namespace; instead, a private function simply be-
comes public.

Nevertheless, currently I still prefer the original basic method. It may need more work, but the resulting
code states clearly what it does. To avoid creating a global by mistake, I use the simple method of assigning
nil to _ENV. After that, any assignment to a global name will raise an error. This approach has the extra
advantage that it works without changes in older versions of Lua. (In Lua 5.1, the assignment to _ENV
will not prevent errors, but it will not cause any harm, either.)

To access other modules, we can use one of the methods we discussed in the previous section. For instance,
we can declare a local variable that holds the global environment:

local M = {}
local _G = _G
_ENV = nil

We then prefix global names with _G and module names with M.

A more disciplined approach is to declare as locals only the functions we need or, at most, the modules
we need:

-- module setup
local M = {}

180
 The Environment

-- Import Section:
-- declare everything this module needs from outside
local sqrt = math.sqrt
local io = io

-- no more external access after this point

_ENV = nil

This technique demands more work, but it documents the module dependencies better.

_ENV and load

As I mentioned earlier, load usually initializes the _ENV upvalue of a loaded chunk with the global
environment. However, load has an optional fourth parameter that allows us to give a different initial
value for _ENV. (The function loadfile has a similar parameter.)

For an initial example, consider that we have a typical configuration file, defining several constants and
functions to be used by a program; it can be something like this:

-- file 'config.lua'
width = 200
height = 300
...

We can load it with the following code:

env = {}
loadfile("config.lua", "t", env)()

The whole code in the configuration file will run in the empty environment env, which works as a kind
of sandbox. In particular, all definitions will go into this environment. The configuration file has no way
to affect anything else, even by mistake. Even malicious code cannot do much damage. It can do a denial
of service (DoS) attack, by wasting CPU time and memory, but nothing else.

Sometimes, we may want to run a chunk several times, each time with a different environment table. In
that case, the extra argument to load is not useful. Instead, we have two other options.

The first option is to use the function debug.setupvalue, from the debug library. As its name implies,
setupvalue allows us to change any upvalue of a given function. The next fragment illustrates its use:

f = load("b = 10; return a")

env = {a = 20}
debug.setupvalue(f, 1, env)
print(f()) --> 20
print(env.b) --> 10

The first argument in the call to setupvalue is the function, the second is the upvalue index, and the
third is the new value for the upvalue. For this kind of use, the second argument is always one: when a
function represents a chunk, Lua assures that it has only one upvalue and that this upvalue is _ENV.

A small drawback of this option is its dependence on the debug library. This library breaks some usual
assumptions about programs. For instance, debug.setupvalue breaks Lua's visibility rules, which
ensures that we cannot access a local variable from outside its lexical scope.

Another option to run a chunk with several different environments is to twist the chunk a little when loading
it. Imagine that we add the following line just before the chunk:

181
 The Environment

_ENV = ...;

Remember that Lua compiles any chunk as a variadic function. So, that extra line of code will assign to
the _ENV variable the first argument passed to the chunk, thereby setting that argument as the environ-
ment. The following code snippet illustrates the idea, using the function loadwithprefix that you
implemented in Exercise 16.1:

prefix = "_ENV = ...;"

f = loadwithprefix(prefix, io.lines(filename, "*L"))
...
env1 = {}
f(env1)
env2 = {}
f(env2)

Exercises
Exercise 22.1: The function getfield that we defined in the beginning of this chapter is too forgiving,
as it accepts “fields” like math?sin or string!!!gsub. Rewrite it so that it accepts only single dots
as name separators.

Exercise 22.2: Explain in detail what happens in the following program and what it will print.

local foo
do
local _ENV = _ENV
function foo () print(X) end
end
X = 13
_ENV = nil
foo()
X = 0

Exercise 22.3: Explain in detail what happens in the following program and what it will print.

local print = print

function foo (_ENV, a)
print(a + b)
end

foo({b = 14}, 12)

foo({b = 10}, 1)

182
Chapter 23. Garbage
Lua does automatic memory management. Programs can create objects (tables, closures, etc.), but there
is no function to delete objects. Lua automatically deletes objects that become garbage, using garbage
collection. This frees us from most of the burden of memory management and, more importantly, frees us
from most of the bugs related to this activity, such as dangling pointers and memory leaks.

In an ideal world, the garbage collector would be invisible to the programmer, like a good cleaner that
does not interfere with other workers. However, sometimes even the smarter collector needs our help. We
may need to stop it at some performance-critical periods or allow it to work only in some specific times.
Moreover, a garbage collector can collect only what it can be sure is garbage; it cannot guess what we
consider garbage. No garbage collector allows us to forget all worries about resource management, such
as hoarding memory and external resources.

Weak tables, finalizers, and the function collectgarbage are the main mechanisms that we can use in
Lua to help the garbage collector. Weak tables allow the collection of Lua objects that are still accessible
to the program; finalizers allow the collection of external objects that are not directly under control of the
garbage collector. The function collectgarbage allows us to control the pace of the collector. In this
chapter, we will discuss these mechanisms.

Weak Tables
As we said, a garbage collector cannot guess what we consider garbage. A typical example is a stack,
implemented with an array and an index to the top. We know that the valid part of the array goes only up
to the top, but Lua does not. If we pop an element by simply decrementing the top, the object left in the
array is not garbage to Lua. Similarly, any object stored in a global variable is not garbage to Lua, even
if our program will never use it again. In both cases, it is up to us (i.e., our program) to assign nil to these
positions so that they do not lock an otherwise disposable object.

However, simply cleaning our references is not always enough. Some constructions need extra collabora-
tion between the program and the collector. A typical example happens when we want to keep a list of
all live objects of some kind (e.g., files) in our program. This task seems simple: all we have to do is to
insert each new object into the list. However, once the object is part of the list, it will never be collected!
Even if nothing else points to it, the list does. Lua cannot know that this reference should not prevent the
reclamation of the object, unless we tell Lua about this fact.

Weak tables are the mechanism that we use to tell Lua that a reference should not prevent the reclamation
of an object. A weak reference is a reference to an object that is not considered by the garbage collector.
If all references pointing to an object are weak, the collector will collect the object and delete these weak
references. Lua implements weak references through weak tables: a weak table is a table whose entries
are weak. This means that, if an object is held only in weak tables, Lua will eventually collect the object.

Tables have keys and values, and both can contain any kind of object. Under normal circumstances, the
garbage collector does not collect objects that appear as keys or as values of an accessible table. That is,
both keys and values are strong references, as they prevent the reclamation of objects they refer to. In a
weak table, both keys and values can be weak. This means that there are three kinds of weak tables: tables
with weak keys, tables with weak values, and tables where both keys and values are weak. Irrespective of
the kind of table, when a key or a value is collected the whole entry disappears from the table.

The weakness of a table is given by the field __mode of its metatable. The value of this field, when
present, should be a string: if this string is "k", the keys in the table are weak; if this string is "v", the
values in the table are weak; if this string is "kv", both keys and values are weak. The following example,
although artificial, illustrates the basic behavior of weak tables:

183
 Garbage

a = {}
mt = {__mode = "k"}
setmetatable(a, mt) -- now 'a' has weak keys
key = {} -- creates first key
a[key] = 1
key = {} -- creates second key
a[key] = 2
collectgarbage() -- forces a garbage collection cycle
for k, v in pairs(a) do print(v) end
--> 2

In this example, the second assignment key = {} overwrites the reference to the first key. The call to
collectgarbage forces the garbage collector to do a full collection. As there is no other reference
to the first key, Lua collects this key and removes the corresponding entry in the table. The second key,
however, is still anchored in the variable key, so Lua does not collect it.

Notice that only objects can be removed from a weak table. Values, such as numbers and Booleans, are
not collectible. For instance, if we insert a numeric key in the table a (from our previous example), the
collector will never remove it. Of course, if the value corresponding to a numeric key is collected in a table
with weak values, then the whole entry is removed from the table.

Strings present a subtlety here: although strings are collectible from an implementation point of view, they
are not like other collectible objects. Other objects, such as tables and closures, are created explicitly. For
instance, whenever Lua evaluates the expression {}, it creates a new table. However, does Lua create a
new string when it evaluates "a".."b"? What if there is already a string "ab" in the system? Does
Lua create a new one? Can the compiler create this string before running the program? It does not matter:
these are implementation details. From the programmer's point of view, strings are values, not objects.
Therefore, like a number or a Boolean, a string key is not removed from a weak table unless its associated
value is collected.

Memorize Functions
A common programming technique is to trade space for time. We can speed up a function by memorizing its
results so that, later, when we call the function with the same argument, the function can reuse that result.1

Imagine a generic server that takes requests in the form of strings with Lua code. Each time it gets a request,
it runs load on the string, and then calls the resulting function. However, load is an expensive function,
and some commands to the server may be quite frequent. Instead of calling load repeatedly each time it
receives a common command like "closeconnection()", the server can memorize the results from
load using an auxiliary table. Before calling load, the server checks in the table whether the given string
already has a translation. If it cannot find a match, then (and only then) the server calls load and stores
the result into the table. We can pack this behavior in a new function:

local results = {}
function mem_loadstring (s)
local res = results[s]
if res == nil then -- result not available?
res = assert(load(s)) -- compute new result
results[s] = res -- save for later reuse
end
return res
end
1
Although the established English word “memorize” describes precisely what we want to do, the programming community created a new word,
memoize, to describe this technique. I will stick to the original word.

184
 Garbage

The savings with this scheme can be huge. However, it may also cause unsuspected waste. Although some
commands repeat over and over, many other commands happen only once. Gradually, the table results
accumulates all commands the server has ever received plus their respective codes; after enough time, this
behavior will exhaust the server's memory.

A weak table provides a simple solution to this problem. If the results table has weak values, each
garbage-collection cycle will remove all translations not in use at that moment (which means virtually
all of them):

local results = {}
setmetatable(results, {__mode = "v"}) -- make values weak
function mem_loadstring (s)
as before

Actually, because the indices are always strings, we can make this table fully weak, if we want:

setmetatable(results, {__mode = "kv"})

The net result is the same.

The memorization technique is useful also to ensure the uniqueness of some kind of object. For instance,
assume a system that represents colors as tables, with fields red, green, and blue in some range. A
naive color factory generates a new color for each new request:

function createRGB (r, g, b)

return {red = r, green = g, blue = b}
end

Using memorization, we can reuse the same table for the same color. To create a unique key for each color,
we simply concatenate the color indices with a separator in between:

local results = {}
setmetatable(results, {__mode = "v"}) -- make values weak
function createRGB (r, g, b)
local key = string.format("%d-%d-%d", r, g, b)
local color = results[key]
if color == nil then
color = {red = r, green = g, blue = b}
results[key] = color
end
return color
end

An interesting consequence of this implementation is that the user can compare colors using the primitive
equality operator, because two coexistent equal colors are always represented by the same table. Any
given color can be represented by different tables at different times, because from time to time the garbage
collector clears the results table. However, as long as a given color is in use, it is not removed from
results. So, whenever a color survives long enough to be compared with a new one, its representation
also has survived long enough to be reused by the new color.

Object Attributes
Another important use of weak tables is to associate attributes with objects. There are endless situations
where we need to attach some attribute to an object: names to functions, default values to tables, sizes
to arrays, and so on.

185
 Garbage

When the object is a table, we can store the attribute in the table itself, with an appropriate unique key. (As
we saw before, a simple and error-proof way to create a unique key is to create a new table and use it as the
key.) However, if the object is not a table, it cannot keep its own attributes. Even for tables, sometimes we
may not want to store the attribute in the original object. For instance, we may want to keep the attribute
private, or we do not want the attribute to disturb a table traversal. In all these cases, we need an alternative
way to map attributes to objects.

Of course, an external table provides an ideal way to map attributes to objects. It is what we called a dual
representation in the section called “Dual Representation”. We use the objects as keys, and their attributes
as values. An external table can keep attributes of any type of object, as Lua allows us to use any type
of object as a key. Moreover, attributes kept in an external table do not interfere with other objects, and
can be as private as the table itself.

However, this seemingly perfect solution has a huge drawback: once we use an object as a key in a table,
we lock the object into existence. Lua cannot collect an object that is being used as a key. For instance,
if we use a regular table to map functions to its names, none of these functions will ever be collected.
As you might expect, we can avoid this drawback by using a weak table. This time, however, we need
weak keys. The use of weak keys does not prevent any key from being collected, once there are no other
references to it. On the other hand, the table cannot have weak values; otherwise, attributes of live objects
could be collected.

Revisiting Tables with Default Values

In the section called “Tables with default values”, we discussed how to implement tables with non-nil
default values. We saw one particular technique and commented that two other techniques needed weak
tables, so we postponed them. Now it is time to revisit the subject. As we will see, these two techniques for
default values are actually particular applications of the two general techniques that we have just discussed:
dual representation and memorization.

In the first solution, we use a weak table to map each table to its default value:

local defaults = {}
setmetatable(defaults, {__mode = "k"})
local mt = {__index = function (t) return defaults[t] end}
function setDefault (t, d)
defaults[t] = d
setmetatable(t, mt)
end

This is a typical use of a dual representation, where we use defaults[t] to represent t.default. If
the table defaults did not have weak keys, it would anchor all tables with default values into permanent
existence.

In the second solution, we use distinct metatables for distinct default values, but we reuse the same metat-
able whenever we repeat a default value. This is a typical use of memorization:

local metas = {}
setmetatable(metas, {__mode = "v"})
function setDefault (t, d)
local mt = metas[d]
if mt == nil then
mt = {__index = function () return d end}
metas[d] = mt -- memorize
end
setmetatable(t, mt)

186
 Garbage

end

In this case, we use weak values to allow the collection of metatables that are not being used anymore.

Given these two implementations for default values, which is best? As usual, it depends. Both have similar
complexity and similar performance. The first implementation needs a few memory words for each table
with a default value (an entry in defaults). The second implementation needs a few dozen memory
words for each distinct default value (a new table, a new closure, plus an entry in the table metas). So, if
your application has thousands of tables with a few distinct default values, the second implementation is
clearly superior. On the other hand, if few tables share common defaults, then you should favor the first
implementation.

Ephemeron Tables
A tricky situation occurs when, in a table with weak keys, a value refers to its own key.

This scenario is more common than it may seem. As a typical example, consider a constant-function
factory. Such a factory takes an object and returns a function that, whenever called, returns that object:

function factory (o)

return (function () return o end)
end

This factory is a good candidate for memorization, to avoid the creation of a new closure when there is one
already available. Figure 23.1, “Constant-function factory with memorization” shows this improvement.

Figure 23.1. Constant-function factory with memorization

do
local mem = {} -- memorization table
setmetatable(mem, {__mode = "k"})
function factory (o)
local res = mem[o]
if not res then
res = (function () return o end)
mem[o] = res
end
return res
end
end

There is a catch, however. Note that the value (the constant function) associated with an object in mem
refers back to its own key (the object itself). Although the keys in that table are weak, the values are not.
From a standard interpretation of weak tables, nothing would ever be removed from that memorizing table.
Because values are not weak, there is always a strong reference to each function. Each function refers to
its corresponding object, so there is always a strong reference to each key. Therefore, these objects would
not be collected, despite the weak keys.

This strict interpretation, however, is not very useful. Most people expect that a value in a table is only
accessible through its respective key. We can think of the above scenario as a kind of cycle, where the
closure refers to the object that refers back (through the memorizing table) to the closure.

Lua solves the above problem with the concept of ephemeron tables.2 In Lua, a table with weak keys
and strong values is an ephemeron table. In an ephemeron table, the accessibility of a key controls the
2
Ephemeron tables were introduced in Lua 5.2. Lua 5.1 still has the problem we described.

187
 Garbage

accessibility of its corresponding value. More specifically, consider an entry (k,v) in an ephemeron table.
The reference to v is only strong if there is some other external reference to k. Otherwise, the collector will
eventually collect k and remove the entry from the table, even if v refers (directly or indirectly) to k.

Finalizers
Although the goal of the garbage collector is to collect Lua objects, it can also help programs to release
external resources. For that purpose, several programming languages offer finalizers. A finalizer is a func-
tion associated with an object that is called when that object is about to be collected.

Lua implements finalizers through the metamethod __gc, as the following example illustrates:

o = {x = "hi"}
setmetatable(o, {__gc = function (o) print(o.x) end})
o = nil
collectgarbage() --> hi

In this example, we first create a table and give it a metatable that has a __gc metamethod. Then we
erase the only link to the table (the global variable o) and force a complete garbage collection. During the
collection, Lua detects that the table is no longer accessible, and therefore calls its finalizer —the __gc
metamethod.

A subtlety of finalizers in Lua is the concept of marking an object for finalization. We mark an object for
finalization by setting a metatable for it with a non-null __gc metamethod. If we do not mark the object,
it will not be finalized. Most code we write works naturally, but some strange cases can occur, like here:

o = {x = "hi"}
mt = {}
setmetatable(o, mt)
mt.__gc = function (o) print(o.x) end
o = nil
collectgarbage() --> (prints nothing)

Here, the metatable we set for o does not have a __gc metamethod, so the object is not marked for
finalization. Even if we later add a __gc field to the metatable, Lua does not detect that assignment as
something special, so it will not mark the object.

As we said, this is seldom a problem; it is not usual to change metamethods after setting a metatable. If you
really need to set the metamethod later, you can provide any value for the __gc field, as a placeholder:

o = {x = "hi"}
mt = {__gc = true}
setmetatable(o, mt)
mt.__gc = function (o) print(o.x) end
o = nil
collectgarbage() --> hi

Now, because the metatable has a __gc field, o is properly marked for finalization. There is no problem
if you do not set a metamethod later; Lua only calls the finalizer if it is a proper function.

When the collector finalizes several objects in the same cycle, it calls their finalizers in the reverse order
that the objects were marked for finalization. Consider the next example, which creates a linked list of
objects with finalizers:

mt = {__gc = function (o) print(o[1]) end}

188
 Garbage

list = nil
for i = 1, 3 do
list = setmetatable({i, link = list}, mt)
end
list = nil
collectgarbage()
--> 3
--> 2
--> 1

The first object to be finalized is object 3, which was the last to be marked.

A common misconception is to think that links among objects being collected can affect the order that they
are finalized. For instance, one can think that object 2 in the previous example must be finalized before
object 1 because there is a link from 2 to 1. However, links can form cycles. Therefore, they do not impose
any order to finalizers.

Another tricky point about finalizers is resurrection. When a finalizer is called, it gets the object being
finalized as a parameter. So, the object becomes alive again, at least during the finalization. I call this a
transient resurrection. While the finalizer runs, nothing stops it from storing the object in a global variable,
for instance, so that it remains accessible after the finalizer returns. I call this a permanent resurrection.

Resurrection must be transitive. Consider the following piece of code:

A = {x = "this is A"}
B = {f = A}
setmetatable(B, {__gc = function (o) print(o.f.x) end})
A, B = nil
collectgarbage() --> this is A

The finalizer for B accesses A, so A cannot be collected before the finalization of B. Lua must resurrect
both B and A before running that finalizer.

Because of resurrection, Lua collects objects with finalizers in two phases. The first time the collector
detects that an object with a finalizer is not reachable, the collector resurrects the object and queues it to
be finalized. Once its finalizer runs, Lua marks the object as finalized. The next time the collector detects
that the object is not reachable, it deletes the object. If we want to ensure that all garbage in our program
has been actually released, we must call collectgarbage twice; the second call will delete the objects
that were finalized during the first call.

The finalizer for each object runs exactly once, due to the mark that Lua puts on finalized objects. If an
object is not collected until the end of a program, Lua will call its finalizer when the entire Lua state
is closed. This last feature allows a form of atexit functions in Lua, that is, functions that will run
immediately before the program terminates. All we have to do is to create a table with a finalizer and
anchor it somewhere, for instance in a global variable:

local t = {__gc = function ()

-- your 'atexit' code comes here
print("finishing Lua program")
end}
setmetatable(t, t)
_G["*AA*"] = t

Another interesting technique allows a program to call a given function every time Lua completes a col-
lection cycle. As a finalizer runs only once, the trick here is to make each finalization create a new object
to run the next finalizer, as in Figure 23.2, “Running a function at every GC cycle”.

189
 Garbage

Figure 23.2. Running a function at every GC cycle

do
local mt = {__gc = function (o)
-- whatever you want to do
print("new cycle")
-- creates new object for next cycle
setmetatable({}, getmetatable(o))
end}
-- creates first object
setmetatable({}, mt)
end

collectgarbage() --> new cycle

collectgarbage() --> new cycle
collectgarbage() --> new cycle

The interaction of objects with finalizers and weak tables also has a subtlety. At each cycle, the collector
clears the values in weak tables before calling the finalizers, but it clears the keys after it. The rationale for
this behavior is that frequently we use tables with weak keys to hold properties of an object (as we discussed
in the section called “Object Attributes”), and therefore finalizers may need to access those attributes.
However, we use tables with weak values to reuse live objects; in this case, objects being finalized are
not useful anymore.

The Garbage Collector

Up to version 5.0, Lua used a simple mark-and-sweep garbage collector (GC). This kind of collector is
sometimes called a “stop-the-world” collector. This means that, from time to time, Lua would stop running
the main program to perform a whole garbage-collection cycle. Each cycle comprises four phases: mark,
cleaning, sweep, and finalization.

The collector starts the mark phase by marking as alive its root set, which comprises the objects that Lua
has direct access to. In Lua, this set is only the C registry. (As we will see in the section called “The
registry”, both the main thread and the global environment are predefined entries in this registry.)

Any object stored in a live object is reachable by the program, and therefore is marked as alive too. (Of
course, entries in weak tables do not follow this rule.) The mark phase ends when all reachable objects
are marked as alive.

Before starting the sweep phase, Lua performs the cleaning phase, where it handles finalizers and weak
tables. First, it traverses all objects marked for finalization looking for non-marked objects. Those objects
are marked as alive (resurrected) and put in a separate list, to be used in the finalization phase. Then,
Lua traverses its weak tables and removes from them all entries wherein either the key or the value is
not marked.

The sweep phase traverses all Lua objects. (To allow this traversal, Lua keeps all objects it creates in a
linked list.) If an object is not marked as alive, Lua collects it. Otherwise, Lua clears its mark, in preparation
for the next cycle.

Finally, in the finalization phase, Lua calls the finalizers of the objects that were separated in the cleaning
phase.

The use of a real garbage collector means that Lua has no problems with cycles among object references.
We do not need to take any special action when using cyclic data structures; they are collected like any
other data.

190
 Garbage

In version 5.1, Lua got an incremental collector. This collector performs the same steps as the old one, but
it does not need to stop the world while it runs. Instead, it runs interleaved with the interpreter. Every time
the interpreter allocates some amount of memory, the collector runs a small step. (This means that, while
the collector is working, the interpreter may change an object's reachability. To ensure the correctness of
the collector, some operations in the interpreter have barriers that detect dangerous changes and correct
the marks of the objects involved.)

Lua 5.2 introduced emergency collection. When a memory allocation fails, Lua forces a full collection
cycle and tries again the allocation. These emergencies can occur any time Lua allocates memory, including
points where Lua is not in a consistent state to run code; so, these collections are unable to run finalizers.

Controlling the Pace of Collection

The function collectgarbage allows us to exert some control over the garbage collector. It is actually
several functions in one: its optional first argument, a string, specifies what to do. Some options have an
integer as a second argument, which we call data.

The options for the first argument are:

"stop": stops the collector until another call to collectgarbage with the option
"restart".

"restart": restarts the collector.

"collect": performs a complete garbage-collection cycle, so that all unreachable objects are
collected and finalized. This is the default option.

"step": performs some garbage-collection work. The second argument, data, specifies the
amount of work, which is equivalent to what the collector would do after allocating
data bytes.

"count": returns the number of kilobytes of memory currently in use by Lua. This result is a
floating-point number that multiplied by 1024 gives the exact total number of bytes.
The count includes dead objects that have not yet been collected.

"setpause": sets the collector's pause parameter. The data parameter gives the new value in
percentage points: when data is 100, the parameter is set to 1 (100%).

"setstepmul": sets the collector's step multiplier (stepmul) parameter. The new value is given
by data, also in percentage points.

The two parameters pause and stepmul control the collector's character. Any garbage collector trades
memory for CPU time. At one extreme, the collector might not run at all. It would spend zero CPU time,
at the price of a huge memory consumption. At the other extreme, the collector might run a complete cycle
after every single assignment. The program would use the minimum memory necessary, at the price of
a huge CPU consumption. The default values for pause and stepmul try to find a balance between
those two extremes and are good enough for most applications. In some scenarios, however, it is worth
trying to optimize them.

The pause parameter controls how long the collector waits between finishing a collection and starting
a new one. A pause of zero makes Lua start a new collection as soon as the previous one ends. A pause
of 200% waits for memory usage to double before restarting the collector. We can set a lower pause if
we want to trade more CPU time for lower memory usage. Typically, we should keep this value between
0 and 200%.

191
 Garbage

The step-multiplier parameter (stepmul) controls how much work the collector does for each kilobyte of
memory allocated. The higher this value the less incremental the collector. A huge value like 100000000%
makes the collector work like a non-incremental collector. The default value is 200%. Values lower than
100% make the collector so slow that it may never finish a collection.

The other options of collectgarbage give us control over when the collector runs. Again, the default
control is good enough for most programs, but some specific applications may benefit from a manual
control. Games often need this kind of control. For instance, if we do not want any garbage-collection
work during some periods, we can stop it with a call collectgarbage("stop") and then restart it
with collectgarbage("restart"). In systems where we have periodic idle phases, we can keep
the collector stopped and call collectgarbage("step", n) during the idle time. To set how much
work to do at each idle period, we can either choose experimentally an appropriate value for n or call
collectgarbage in a loop, with n set to zero (meaning minimal steps), until the idle period expires.

Exercises
Exercise 23.1: Write an experiment to determine whether Lua actually implements ephemeron tables.
(Remember to call collectgarbage to force a garbage collection cycle.) If possible, try your code
both in Lua 5.1 and in Lua 5.2/5.3 to see the difference.

Exercise 23.2: Consider the first example of the section called “Finalizers”, which creates a table with a
finalizer that only prints a message when activated. What happens if the program ends without a collection
cycle? What happens if the program calls os.exit? What happens if the program ends with an error?

Exercise 23.3: Imagine you have to implement a memorizing table for a function from strings to strings.
Making the table weak will not do the removal of entries, because weak tables do not consider strings as
collectable objects. How can you implement memorization in that case?

Exercise 23.4: Explain the output of the program in Figure 23.3, “Finalizers and memory”.

Figure 23.3. Finalizers and memory

local count = 0

local mt = {__gc = function () count = count - 1 end}

local a = {}

for i = 1, 10000 do
count = count + 1
a[i] = setmetatable({}, mt)
end

collectgarbage()
print(collectgarbage("count") * 1024, count)
a = nil
collectgarbage()
print(collectgarbage("count") * 1024, count)
collectgarbage()
print(collectgarbage("count") * 1024, count)

Exercise 23.5: For this exercise, you need at least one Lua script that uses lots of memory. If you do not
have one, write it. (It can be as simple as a loop creating tables.)

• Run your script with different values for pause and stepmul. How they affect the performance and
memory usage of the script? What happens if you set the pause to zero? What happens if you set the

192
 Garbage

pause to 1000? What happens if you set the step multiplier to zero? What happens if you set the step
multiplier to 1000000?

• Adapt your script so that it keeps full control over the garbage collector. It should keep the collector
stopped and call it from time to time to do some work. Can you improve the performance of your script
with this approach?

193
Chapter 24. Coroutines
We do not need coroutines very often, but when we do, it is an unparalleled feature. Coroutines can literally
turn upside-down the relationship between callers and callees, and this flexibility solves what I call the
"who-is-the-boss" (or "who-has-the-main-loop") problem in software architecture. This is a generalization
of several seemingly unrelated problems, such as entanglement in event-driven programs, building iterators
through generators, and cooperative multithreading. Coroutines solve all these problems in simple and
efficient ways.

A coroutine is similar to a thread (in the sense of multithreading): it is a line of execution, with its own
stack, its own local variables, and its own instruction pointer; it shares global variables and mostly anything
else with other coroutines. The main difference between threads and coroutines is that a multithreaded
program runs several threads in parallel, while coroutines are collaborative: at any given time, a program
with coroutines is running only one of its coroutines, and this running coroutine suspends its execution
only when it explicitly requests to be suspended.

In this chapter we will cover how coroutines work in Lua and how we can use them to solve a diverse
set of problems.

Coroutine Basics
Lua packs all its coroutine-related functions in the table coroutine. The function create creates new
coroutines. It has a single argument, a function with the code that the coroutine will run (the coroutine
body). It returns a value of type "thread", which represents the new coroutine. Often, the argument to
create is an anonymous function, like here:

co = coroutine.create(function () print("hi") end)

print(type(co)) --> thread

A coroutine can be in one of four states: suspended, running, normal, and dead. We can check the state
of a coroutine with the function coroutine.status:

print(coroutine.status(co)) --> suspended

When we create a coroutine, it starts in the suspended state; a coroutine does not run its body automatically
when we create it. The function coroutine.resume (re)starts the execution of a coroutine, changing
its state from suspended to running:

coroutine.resume(co) --> hi

(If you run this code in interactive mode, you may want to finish the previous line with a semicolon, to
suppress the display of the result from resume.) In this first example, the coroutine body simply prints
"hi" and terminates, leaving the coroutine in the dead state:

print(coroutine.status(co)) --> dead

Until now, coroutines look like nothing more than a complicated way to call functions. The real power of
coroutines stems from the function yield, which allows a running coroutine to suspend its own execution
so that it can be resumed later. Let us see a simple example:

co = coroutine.create(function ()
for i = 1, 10 do

194
 Coroutines

print("co", i)
coroutine.yield()
end
end)

Now, the coroutine body does a loop, printing numbers and yielding after each print. When we resume
this coroutine, it starts its execution and runs until the first yield:

coroutine.resume(co) --> co 1

If we check its status, we can see that the coroutine is suspended and, therefore, can be resumed:

print(coroutine.status(co)) --> suspended

From the coroutine's point of view, all activity that happens while it is suspended is happening inside
its call to yield. When we resume the coroutine, this call to yield finally returns and the coroutine
continues its execution until the next yield or until its end:

coroutine.resume(co) --> co 2
coroutine.resume(co) --> co 3
...
coroutine.resume(co) --> co 10
coroutine.resume(co) -- prints nothing

During the last call to resume, the coroutine body finishes the loop and then returns, without printing
anything. If we try to resume it again, resume returns false plus an error message:

print(coroutine.resume(co))
--> false cannot resume dead coroutine

Note that resume runs in protected mode, like pcall. Therefore, if there is any error inside a coroutine,
Lua will not show the error message, but instead will return it to the resume call.

When a coroutine resumes another, it is not suspended; after all, we cannot resume it. However, it is not
running either, because the running coroutine is the other one. So, its own status is what we call the normal
state.

A useful facility in Lua is that a pair resume–yield can exchange data. The first resume, which has no
corresponding yield waiting for it, passes its extra arguments to the coroutine main function:

co = coroutine.create(function (a, b, c)
print("co", a, b, c + 2)
end)
coroutine.resume(co, 1, 2, 3) --> co 1 2 5

A call to coroutine.resume returns, after the true that signals no errors, any arguments passed to
the corresponding yield:

co = coroutine.create(function (a,b)
coroutine.yield(a + b, a - b)
end)
print(coroutine.resume(co, 20, 10)) --> true 30 10

Symmetrically, coroutine.yield returns any extra arguments passed to the corresponding resume:

195
 Coroutines

co = coroutine.create (function (x)

print("co1", x)
print("co2", coroutine.yield())
end)
coroutine.resume(co, "hi") --> co1 hi
coroutine.resume(co, 4, 5) --> co2 4 5

Finally, when a coroutine ends, any values returned by its main function go to the corresponding resume:

co = coroutine.create(function ()
return 6, 7
end)
print(coroutine.resume(co)) --> true 6 7

We seldom use all these facilities in the same coroutine, but all of them have their uses.

Although the general concept of coroutines is well understood, the details vary considerably. So, for those
that already know something about coroutines, it is important to clarify these details before we go on. Lua
offers what we call asymmetric coroutines. This means that it has a function to suspend the execution of a
coroutine and a different function to resume a suspended coroutine. Some other languages offer symmetric
coroutines, where there is only one function to transfer control from one coroutine to another.

Some people call asymmetric coroutines semi-coroutines. However, other people use the same term se-
mi-coroutine to denote a restricted implementation of coroutines, where a coroutine can suspend its exe-
cution only when it is not calling any function, that is, when it has no pending calls in its control stack. In
other words, only the main body of such semi-coroutines can yield. (A generator in Python is an example
of this meaning of semi-coroutines.)

Unlike the difference between symmetric and asymmetric coroutines, the difference between coroutines
and generators (as presented in Python) is a deep one; generators are simply not powerful enough to im-
plement some of the most interesting constructions that we can write with full coroutines. Lua offers full,
asymmetric coroutines. Those that prefer symmetric coroutines can implement them on top of the asym-
metric facilities of Lua (see Exercise 24.6).

Who Is the Boss?

One of the most paradigmatic examples of coroutines is the producer–consumer problem. Let us suppose
that we have a function that continually produces values (e.g., reading them from a file) and another
function that continually consumes these values (e.g., writing them to another file). These two functions
could look like this:

function producer ()
while true do
local x = io.read() -- produce new value
send(x) -- send it to consumer
end
end

function consumer ()
while true do
local x = receive() -- receive value from producer
io.write(x, "\n") -- consume it
end

196