2004 IEEE International Conference on Systems, Man and Cybernetics

Visualizing cognitive systems:

Getting past block diagrams*
Pieter Jan Sappers J o h n M. Flach
Delff University of Technology Wright State University
Delft, The Netherlands Dayton, OH, USA
@.j .stappers@io.tudelft.nl) (john.flach@wight.edu)

drawings, CAD); others are seemingly loose and artistic

Abstract- Visualisotions play an important part in the (such as collages and moodboards); yet others are vague
development of ideas. They make ingredients and and incomplete, such as thought diagrams, and concept
relations explicit, guide thinking processes of the sketches. Communicative visualization is a substantial
designer or scientist, and support communications, open part of a designers’ training. Visualisations of various
across the boundaries of disciplines. In the fields of kinds are key tools of thought.
cognitive psychology and human-machine interaction, In representing cognitive systems, the dominant
block diagrams have been a dominant means for visualization used to organize our thinking on cognitive
representing cognitive systems. However, we believe that processes has been the block diagram, an example of
this form of representation may constrain how we think which is shown in Fig. 1. Block diagrams depict
about cognition in undesirable ways. This form of successive stages of processing, and highlight
representation biases viewers to see cognition as a interdependencies between them. Although these models
sequential, step-by-step process, underemphasizing the provide a relatively straightforward decomposition using
dynamical properties of closed-loop, adaptive processes. a spatial framework that is familiar to many disciplines,
Block diagrams emphasize activity and internal mental we believe they can lead to difficulties wben attempting
operations (awareness) and occlude the ecological or to visualize dynamic phenomena. We believe that the
work domain (situational) constraints. What would block diagrams actually hinder thinking appropriately
happen if we applied the principles of ecological about dynamic phenomena, because they invite the
interface design to the domain of the cognitive scientist reader to think of essentially parallel processes as
or cognitive engineer? Are there alternative sequential stages, promote causal chain thinking, and
representations that will make the role of ecological actually ignore the important contribution of the
constraints more clear and that might lead to deeper environmental context. In the diagram of Fig. 1, the
insights about the nature of cognitive dynamics? whole complex feedback of the environment is
summarized into a single arrow, whereas all the
Keywords: Cognitive Systems Engineering, diagram’s structure and nuance lies ‘in the mind of the
visualisation, communication, dynamic systems, actor.’ Often in control theoretical analysis of human-
diagrams. machine interaction, a single box ‘plant’ is included to
represent the machine (e.g., a pilot’s airplane), but even
this is only a small part of the environment, disregarding
1 Introduction the complexities surrounding the airplane. These
‘Visualisations play an important part in both science diagrams promote a focusing of the analysis totally on
and design. They stand at the birth of new ideas, play an the actor’s mind.
important role in summarizing complex theories, help to For example, we regularly see that students, when
structure ingredients and relations, and serve in confronted with a block diagram, will follow it from left
communicating ideas to others. Especially in new fields, to right, following the arrows, as if they were pointing
or multidisciplinary areas, visualizations can help us to with their finger. One step follows another in a causal
cross the gaps created by the different abstractions and chain, each step seemingly being finished before the next
jargon. The visualization can provide a concrete one begins: After ‘encoding’ is complete, ‘pattern
framework for manipulating ideas and exploring recognition’ can start; meanwhile, ‘decision’ has to stand
alternatives. by, do nothing, and wait. Finally, ‘motor’ control will
Especially in product design, a rich variety of produce the ‘response’. The arrow pointing back seems
different visualizations are used to help designers only there to remind the reader that this happens not
communicate with clients, experts, users, and other once, but time and again, each time with a new stimulus.
designers. Some of these visualizations address technical It’s like a chain of domino pieces with a reset action.
issues and are highly formalized (e.g. measurement What gets easily overlooked in this form is that (a) the
processes all run in parallel, they don’t pop into and out
of action, (b) the feedback loop introduces a circular
* 0-7803-8566-7/04/$20.0002004 IEEE causality in which perception and action are dynamically

021
 Figure 1. Example of a box diagram with a single feedback loop.

coupled, and (c) the structure of the environment is Peters projection on the other hand1 gives a ‘true’
overlooked (i.e., it gets no boxes). Even though the representation of area, at the price of distorting angles.
professional will acknowledge the importance of these The makers of maps not only choose a projection, but
three points, the form of the diagram occludes these also what to depict and how to depict it; mapmaking is
properties 60m the dialogue. toolmaking, and the form of a map tells its purpose, or
Moreover, it can be claimed that professionals also should do so. The shape and choice of elements can be
are not immune to the directions of thought suggested by chosen to lead or, in the case of representation of military
the visualization. In automotive design, great care is secrets on civilian maps, to mislead [4].
taken to ‘rough up’ clay models of concept designs prior A telling example is Harry Beck‘s famous design for
to presentation to management, because a polished the London Transport Underground map. Previously,
model suggests a finished design. Similar problems travelers had to find their way on topographic maps,
occurred with computer visualizations in the 90s: showing the city proportioned in bird’s eye perspective.
because the pictures were ‘too fioished, too closed-off’, This meant that innercity areas (with many connections)
they presented a barrier for respondents at focus groups were cluttered, whereas further away, most of the map
to provide criticism. was empty space. Beck realized that the biggest problem
In both of these cases viewers ignored explicit verbal for travelers was not to find their destination on the map,
statements that these were rough concepts. Visualisations but to determine their joumey plan, i.e., at which stations
guide us in subtler ways than we are consciously aware. to switch trains, and how many stops there were between
The field needs a richer set of visualizations. How stations. To the underground traveler, wayfinding
can we augment and improve the current ones? First we doesn’t involve precise localization from street pattems
look at visualizations as tools in general, then retum to or landscape elements. Not surprisingly (Beck was an
box diagrams in particular, and cognitive systems in electrical engineer), the resulting map (and the maps of
general. almost every subway system afterward) followed the
aesthetics of electrical switching diagrams, emphasizing
2 Varieties of visualisations the functional elements of lines, stops and connections,
while sacrificing visual similitude or geometric scale:
In this section, we will use a few examples to time and action were the metrics of the user’s process,
illustrate how different forms of visualization constrain not distance or angle.
exploration. For other examples, see Rasmussen ([l],
[2]). He reviewed how representations are used in 2.2 Mathematics
science relative to the concept of ecological interface
design. Similar examples are known from the development of
graphical and verbal formalisms in mathematics and
2.1 Cartographers physics, such as the forms of mathematical expressions
(which promote substitution and pattem matching, giving
Cartographers use a wide variety of projections, and visual prominence to balanced equations), algebra for
are eminently aware that maps are inshuments which quantum physics (e.g., Dirac’s bracket notation versus
promote certain (useful) perspectives on a subject matter, vector notation) and set theory [ 5 ] . In each case, the
but at the sametime distort other equally valid notation systems that best supported the operations and
perspectives. Although each of these projections may transformations current in the discipline, or which gave
cany basically the same information, and one may be the most fruitful suggestions for approaching a new
mathematically reconstructed from the other, they greatly problem survived. Tufte, in his seminal books on
vary in the tasks that are easily carried out with them, visualizing statistical data, describes the striving for
and the actions they suggest to their users [3]. For graphical excellence as a search to bring across as much
instance, Mercator’s projection is eminently suited for relevant information through as much ink on the
sailing around the world, because the directions on the available paper as possible, and discusses examples of
map correspond to directions you measure with a how graphical form can deceive or inform [ 6 ] . In this
compass or by measuring angles on the horizon. The area especially, there’s considerable awareness of the
price however, is that area is greatly distorted, making interpretations and confusions engendered by different
Greenland seem many times the size of Africa. The ways of visualizing data. Tufie’s books especially

022
discuss several ways in which visual depictions may 3 Mending the box diagram?
distort data, or may intentionally or unwittingly lead or
mislead readers. Most standard mathematics software If we want to improve the block diagram, we should
(e.g., Mathematica, SPSS) now has visualisation first ask what it is good for, what functions it brings to
modules which incorporate these findings, but the same the practitioner as a tool. These functions include
awareness is not always found in the diagrams
professionals draw to support their thinking and to list the elements that are considered
communication. important.
to indicate the relations between the elements
2.3 Design to indicate the boundaries of the system
The mapmakers’ realization that visualizations are tools In particular, the box diagram lists as elements
to support activities is reflected in the uses that engineers processes (the boxes) and signals (the arrows). Relations
make of sketching, as distinguished by Ferguson ([7]): a) are in the connections of processes by signals. The
to support thinking (getting new ideas); b) to support processes are named, the signals often are not, which
talking (communicating and elaborating ideas); and c) to suggests that the focus of the practitioner should be on
support prescription (settling ideas into a fixed form). the processes. Only the arrows in and out (stimulus and
Likewise, the way people use diagrams to reason has response) are typically labeled. They represent the
gained attention in the past decade. For instance, if we interface with the environment. But one has to look very
look at a series of visualizations that product designers precisely if one is to spot the properties of the
make in the early phases of a design project, we see environment, which are hidden inside the returning
collages, diagrams, sketches, models, situation sketches, arrow. If taken as a model of human-machine interaction,
and storyboards, along with verbal documents (Fig. 2, the diagram invites the scientist to look for explanations
see [SI). of phenomena only inside the human (or often: inside the
The visualizations made for thinking can be quite mind of the human), and to disregard the environment.
personal in form, and prescriptive drawings can be Similarly, the engineer is invited to design systems by
highly formalized or expressive (see, e.g., the variability paying attention to how humans process information, but
of architects’ sketches in [9]). The visualizations that not to what information there is, or how the human
support talking are less extreme in form, because they actions feed back to constrain perceptions.
must be understood by the different participants, and the Also, the linear arrangement suggests that the
participants need to be able to modify the visualizations processes in the boxes are a succession of isolated events
to bring across their points, That’s why they have an where the output fiom one stage becomes the input to the
easy-to-draw, diagrammatic form. next: the domino-chain model. A different spatial
reordering, as shown in Fig. 3, already helps to change
the appearance and the suggested directions. Although it
is semantically equivalent to Fig. l., it already suggests a
different approach to conceiving the closed-loop
dynamics. Pragmatically, Fig. 3 leads to different
interpretations of the dynamic.
In this diagram, the dominant visual element is the
cyclical ordering of the processes: the feedback loop.
Also, stimulus and response are shown close together,

Figure 2. Sketches, situation views, and storyboards are Figure 3. Flach‘s reordering of the diagram in Figure I .
among the visualizations used in interaction design. This form emphasizes the closed-loop dynamic and the
associated coupling between perception and action (or
between stimulus and response).

823
 It

Figure 4. A variety of diagrams of cognitive systems produced by participants in the CSE spring course.

reflecting that action and reaction are often joined at the visualizations which could be entered into a powerpoint
interface between human and environment. Instead of the presentation). In the second meeting of the course, the
train-of-dominoes, the diagram rather resembles the participants explained their diagram, and the teacher
spokes of a wheel, suggesting that the different processes reflected on common elements and differences between
are concurrent and interdependent, and that the the visualizations. Nine visualizations are shown in Fig.
phenomena are an ongoing process, not a series of 4. Here we summarize a few observations:
reactions to discrete stimuli.
But the reordering does not solve all the criticisms Most of the pictures depict elements (boxes,
raised above. Still, the diagram underemphasizes the nodes, people) and relations (arrows).
environment, and guides its user to focus on processing Many of the pictures show a distinct
stages rather than on the interaction as a whole. In ‘environment’ (the wiggly line in picture 1, the
dealing with cognitive dynamics we need other outer circle in the picture 2, the ‘outside’ in
visualizations above, or next to, the block diagram. In the picture 8) with which a central ‘actor’ interacts.
next section we explore some other forms of Some pictures(4,7) depict people in action.
representation. Many pictures explicitly show closed loops
(e.g., picture 3,7,8), often depicting perception
4 Adding to the box diagram? and (motor) action @icture i.2; p i & 6-with
boxes instead of arrows).
To illustrate the scope and limitations of Elements of psychological theory, viz. ‘mental
practitioner’s thoughts on cognitive systems we asked a models’, are explicitly depicted in picture 3.
number of researchers to produce the sketch they would Picture 6 ‘lists’ elements such as rules or
use to explain their view of the cognitive system they memory.
dealt with. * Picture.5 took a special place. The designer who
made it explained ‘in the diagram I show both
Participants were researchers and Ph.D. students ffom the user and product, but they are immersed in,
engineering, humanities, and design. The assignment was partially behind, the environment, and you
part of a course on cognitive systems engineering, and cannot separate them’.
followed the introductory lecture. Very few constraints There are representations of phase spaces,
were given in the assignment (it should be one or two depicting how parameters of the system state

824
 evolve in the form of a spatial map, where time the environment and its structure;
and causality are implicit in the path of the people’s goals and intentions;
system, rather than explicit in signals or spatial
ordering (picture 9). In the course, the comparison of visualizations greatly
helped in forming a mutual understanding in the
All these people were practitioners studying related participants of each other’s approaches and values.
phenomena, but their views are very different. Most Visualizations which contained depictions of the
importantly, some of the things they consider important, situation under study quickly opened up the forum for
and give a place in their visualizations, are not readily discussion. This is similar to what is found in design
visible in the classic block diagrams. Again some practice, where many half-abstracted visualizations are
examples used to help the designer organize his thoughts, develop
solution concepts, and negotiate with different
activities, such as problem solving, learning; stakeholders.
elements from cognitive theory such as rules
and memory; these are often referred to, but are
difficult to integrate in these schemes.
be valuable to explore the problem from both
5 Beyond man and machine: perspectives.
Activity can be represented as a trace in the state
interaction space. For example, the path associated with a particular
Situation-sketches and storyboard techniques are one landing can be traced in either the energy space or the
counterweight against the problems of the block altitudeispeed space. However, time is less explicit in
diagram mentioned above, in that they explicitly draw this representation and the boundaries and surfaces are
attention to issues of context and ongoing time’ time independent for stationary processes. For
development. The danger of these techniques may be nonstationary processes, (i.e., the dynamic itself
that they invite the reader to think exclusively in terms changes over time) the changing dynamic constraints
of activity. would be represented as changes in the boundaries or
A fundamentally different perspective is provided in surfaces themselves. For example, the limits on the rate
‘phase space diagrams’, also called ‘state spaces’. In of energy change available to the pilot might change as
phase space diagrams, processing stages and activities the result of losing an engine or damage to a flight
are ignored and constraints are emphasized. See Fig. 5 . surface. In the state representation this would be
The basic metaphor of state diagrams is neither that of represented by a change in the performance boundaries.
the train-of-dominoes, expressing temporal or causal Figure. 5 illustrates how phase spaces might shape
succesion, nor that of the wheel-and-spokes diagram, discussions about cognitive systems. Instead of
expressing parallel cyclic dependence. Rather, it’s partitioning the phenomenon into different ‘processes’ it
metaphor is that of the cartographer’s map, showing a is partitioned into different classes of constraint. For
landscape with roads and pitfalls, mountains and rivers, example, values or goals might represent one class of
i.e., opportunities and threats, but leaving the reader’s constraint. The physical laws of motion might represent
mind free to roam and imagine developing states and another class of constraints. And the information
possible routes through the territory. available to guide action might represent a third class of
In a phase space representation parameters of the constraint. The dialogue about the dynamic then might
system dynamic are chosen as the dimensions of the focus on what constraints should fall within each class
space and constraints on the dynamic are represented as and how those constraints come together or interact to
boundaries or surfaces in the space. The choice of shape system behavior. The figure illustrates this with a
parameters or state variables will naturally vary as a simple example: the anival of a train slowing down to
function of the dynamic being represented and different align with the end of the platform. State parameters are
state variables can be chosen to represent the same distance to the end and speed of approach; some of the
dynamic. For example, the state of an aircraft with concerns shown in the diagram are the operating speed,
regard to the vertical flight path can be represented in the braking power, and the expected time for
terms of altitude and air speed or in terms of kinetic and completion. These concerns derive from different
potential energy. Either choice is sufficient to fully sources: speed is limited on the lower side by the
characterize the vertical path and there is a mapping technical capabilities of the engine, on the higher side
fiom one space to the other. One choice might be better by traffic regulations. Braking is at least limited by the
for visualizing the aerodynamic constraints on landing. maximal braking ability: the diagram shows clearly
The other choice might he better for visualizing the which combinations of speed and distance lead to
information constraints (what is displayed to the pilots). dangerous situations, in which the driver is no longer
When exploring the safe boundaries of operation, it may able to brake sufficiently strongly. The time dimension

825
 References
[I] J. Rasmussen, Information Processing and Human-
Machine Interaction. New York: North Holland,
1986.

[2] J. Rasmussen, Ecological Interface Design for

Complex Systems: An Example: SEAD-UAV
Systems. Final Report, EOARD F617’08-97-W0211,
April 1998.

[3] E. Hutchins (1995) Cognition in the Wild. MIT

Press, Cambridge, MA

[4] Monmonier, M. (1991) How to lie with maps.

University of Chicago Press: Chicago.

t [5] L.Carrol1 (1986) Lewis Carroll’s Symbolic Logic,

Clarkson N. Porter, Publishers.

[6] Tufle, E.R. (1983) The visual display of quantitative

e for the arrival of a train at a station information. Graphics Press, Cheshire, CT.

serves to evaluate whether arrival fits in with the [7] Ferguson E. S. (1992). Engineering and the mind’s
schedule, constraints of another source, reflecting eye. Cambridge, MA: MIT Press.
demands of travel connections and passenger
expectations. A well-designed state space representation [SI Keller, Hoeben, & Stappers (2000) Aesthetics,
should handle all these, just as a well-designed control interaction, and usability in ‘sketchy‘ design tools.
interface should support the driver in satisfying his Exchange Online Journal, 1(1), (refereed intemet
goals within this set of constraints. journal,
hrto://w\h?K.media.uwe.ac.uWexchanaronline/
6 Conclusions
[9] B.Lacy (1991) 100 Contemporary Architects,
Our notations guide what we do and think, in many Drawings & Sketches. Thames and Hudson, London.
ways. They make some aspects of a problem salient and
hide other facets. In order to free our view we need to
develop a richer set of visualizations, which can in their
turn help communication in this highly interdisciplinary
area. State spaces are one especially promising way to
help us organize, conceive, and discuss cognitive
systems. The major advantage of this form of
representation is that it draws attention to the problem
space. The state representation provides a frame for
visualizing the constraints that shape the behavior of
cognitive systems. We think that this is an important
tool for building situated models of cognition that
compliment the “process” centered perspective that has
shaped the discussion over the last fifty years.

826