Chapter 9: Why code is more important than flat design

The last chapter described more accessible ways to interact with code and data, working differently to established programming languages, and potentially looking nothing like them. Machine learning algorithms have a part to play in this new generation of tools, but Moral Codes must still be open, learnable and controllable. Inventing new codes is not easy, and this chapter explains some of the mistakes that are holding us back. In particular, it’s important to understand that Moral Codes are diagrams, not pictures.

The invention of the GUI, as described in chapter 6, was a breakthrough that used pictorial design to depict files, folders, and other operating system elements as paper objects in a physical office, known ever since as the “desktop metaphor". In the 1980’s and 90’s this pictorial resemblance between the icons on a computer screen and familiar physical objects was believed to be the single most important element of design for usability. Design guidelines from Apple and Microsoft^[1], and countless textbooks on user interface design^[2], used to say that the first step in designing a new user interface was the selection of an appropriate real-world metaphor, so that users could understand the abstract world of the computer in relation to familiar objects. The idea was that, if a user was unsure how a particular operation could be achieved, or what the effects of their actions might be, they could think about this by analogy to the real-world objects shown on the screen, and use that analogy to make the new digital product more intuitive.

However, this principle of usability by analogy to the physical world has severe limitations, and received vigorous opposition from the very people who were credited with inventing it. In 1990, Joy Mountford of Apple commissioned a book on The Art of Human-Computer Interface Design that set out a comprehensive design research agenda following the commercial success of the Macintosh. Hypertext pioneer Ted Nelson, in his contribution to that book, said: 

What I object to is severalfold: first, these mnemonic gimmicks are not very useful for presenting the ideas in the first place; second, their resemblance to any real objects in the world is so tenuous that it gets in the way more than it helps; and third . . . the metaphor becomes a dead weight . . . The visualizations become locked to some sort of continuing relation to the mnemonic. It becomes like a lie or a large government project: more and more things have to be added to it.^[3]

My own PhD started by following textbook advice^[4], but ended up challenging the myth of the metaphor. My experimental results found that, while iconic pictures can be useful mnemonics (slightly more than a gimmick, as Ted Nelson claimed, though not very much more), the complex analogy did little to support understanding. In fact, the mnemonic was surprisingly effective even when people completely misunderstood the analogy^[5]. One person thought the “folder” icon was a briefcase to carry documents around, and I recently heard someone describe the “cut” from cut and paste as a hole in her finger to suck up text, ignoring the reference to old-style paper editing with scissors and glue^[6].

Skeuomorphism

Making user interfaces resemble older technologies is not unique to software. In the early days of the motor car, there was huge diversity in operator controls. Arrangements for steering included bicycle handlebars, a tiller handle as on a boat, and even reins that could be used like steering a horse. Today we expect cars to have a steering wheel, but we can imagine how reins might have seemed a good idea at the time. People buying motor cars already rode horses, so would intuitively understand the steering arrangement. The steering wheel, on the other hand, was a new invention. People who had never seen one before might take a little while to work it out.

This is a constant challenge for the designers of innovative user interfaces. Many companies hope that their new products will be seen as familiar or “intuitive.” Businesses are anxious about possible market resistance if customers need to adopt new habits or understand something new. As a result, new products are often made to look like old ones.

Designing new products to resemble familiar but obsolete technical material is called skeuomorphism^[7]. It can result in inappropriate designs, like a motor car steered with reins, or just silly ones, like plastic laundry baskets moulded to resemble woven cane, making them harder to clean and easier to break. Although originally an obscure term, the word skeuomorphism became very familiar to readers of design blogs and gadget magazines around the time that the Apple design department gave the iPhone a makeover with the transition to “flat design”. Someone is sure to mention skeuomorphism, whenever the apparently obsolete 1990’s textbook wisdom about realistic metaphor gets repeated in a technical setting today. 

In the decades before HCI learned to say skeuomorphism, the widespread enthusiasm for analogies to the physical world was responsible for some dramatic failures. Microsoft “Bob” was originally launched as the home-computing alternative to business operating systems Windows 95 and Windows NT. Bob replaced the files, folders and menus of the desktop with a cosy-looking den, lined with books and a fire burning by the leather armchair. Despite that faithful reproduction of a particular kind of (American, male, middle-class) dream, Bob was a dramatic failure, quickly withdrawn from the market among much embarrassment. It was just too clunky to access the useful parts of the system among the distracting pictures, and the way that Bob hid the business functionality (not least behind “Rover” - a dog intended to act as a faithful AI helper and companion) was seen as patronising rather than informative.

Microsoft survived that experiment into excessive metaphor, despite the embarrassment of a product launch and marketing campaign for a product very few people wanted. But other companies suffered greater damage by following the advice from HCI experts. The most intriguing case was “Magic Cap”, one of the very first graphical user interfaces for a mobile device. Created by a company called General Magic, the Magic Cap operating system was (briefly) released as a product by Sony. Sony’s MagicLink remarkably offered many of the capabilities of Apple's iPhone, but in 1994 - thirteen years before the iPhone was launched. The cell phone, internet access and productivity applications were all accessed via supposedly intuitive illustrations of an office desk, with a hallway leading to functionality in other rooms. The problem with this user interface, as with Microsoft’s Bob, was that it held too rigidly to the textbook prescriptions that novel software capabilities must resemble physical objects to be intuitive. 

I’m not suggesting that Magic Cap failed only because of the user interface design, and there were certainly other vendors of popular personal digital assistant devices (called PDAs at the time) that failed to continue into the iPhone era, including other licensees of the Magic Cap system. Nevertheless, the artistic renderings of real-world desks, office doors and hallways that you had to navigate seem particularly clunky and annoying, by comparison to a simple screen of abstract icons. In the decades since then, we all seem to have adjusted quite comfortably to touch-screen phones, without elaborate analogies to physical objects.

This was classic skeuomorphism. The idea that a computer screen should look like an office or a den is not much more sensible than steering a car with reins or carrying laundry in an imitation-wicker plastic basket. Appealing to the user interface of older technologies may seem superficially attractive, but is quickly counterproductive when capabilities are genuinely new. Perhaps the widespread (at present) interest in speech interfaces using natural language is just another example of skeuomorphism? Speech was good for many information processing tasks at one time, especially when those tasks had to be done by humans. Now that computers are available, we may not need speech any more, just as we don’t really need reins when there are no horses involved.

The escape from physical analogy and the desktop metaphor has been liberating. Flat design no longer attempts to imitate 3D objects, and the more modern style is highly popular. Nobody is calling for the return of 3D renderings on their computer screens, apart perhaps from the VR display vendors who would like people to do email, debug spreadsheets or type word documents while wearing a “metaverse” headset. Although popular with gamers, research projects seeking more mainstream deployment of VR continue to struggle against the practical utility of 2D notations.

What did we lose in flat design?

The problem with flat design, as in the current generation of smartphone interfaces, is that it is still not a good representation to support coding. Each app icon on a phone is designed separately from every other app, and the removal of the “file” so central to desktop computing (with the exception of images in the phone’s photo gallery) has made it harder to combine their functionality. This places annoying constraints on practical work, as if you were doing a complicated carpentry project with a Swiss army knife. If you can only use one tool at a time, your work is hugely constrained by comparison to a workbench with multiple tools that can be combined in different ways, including temporary arrangements for clamping and assembling. A smartphone that supported Moral Codes would be more like a workbench, allowing software elements to be related to each other and combined, rather than a collection of flatly isolated features with no unifying logic or representation.

Alan Kay’s contribution to The Art of Interface Design - the same book where Ted Nelson made his diatribe against bureaucratic metaphor - presents an impressive manifesto for the technical steps that he suggested should come next^[8]. Many of his predictions for the growth of hardware and online media are now everyday reality, but the evolution of the GUI metaphor into an iconic flat design is almost the reverse of what he proposed. Kay agreed that Apple’s adoption of metaphor was the wrong metaphor for what is needed, but suggested that interfaces should be more magical, not like physical paper, coordinating work by intelligent agents.

Alan Kay has always been a pioneer and advocate of the design priorities that I am calling Moral Codes. Even in 1990, the need to integrate AI with HCI was apparent to Kay, just as to Allen Cypher, and to others using machine learning methods to make programmable human-centric “agents” such as Henry Lieberman at the MIT AI Lab and Media Lab^[9].

All of these researchers saw an opportunity for sharing of responsibility between machine learning and human intelligence, and Kay’s personal view on user interface made many analogies to human uses of visual representation in the past, suggesting what kinds of cognitive capacity might be exploited in future. The role of imagination, inventing types of code that go beyond existing standards and products, is essential, just as his original proposal for the Dynabook introduced it as a science fiction fantasy, rather than a technical proposal. The failure of a company called General Magic is ironic, especially since Kay himself had advocated magic as the way to avoid the tyranny of literalism. The failure of General Magic, as with many who fail to recognise the true opportunities from AI, was not that it was too magical, but that it was too literal.

We might even think of the desktop metaphor as partly responsible for the rise of surveillance capitalism. In the typical iconic GUI, whether realistic 3D or flat design, the system watches you moving objects around, but there is no way to say what you would like done with those observations. Looking “inside” or “behind” an object to see its invisible data exhaust requires some kind of transparency that breaks the metaphor. The commercial advocates of metaphor, such as Apple’s Bruce Tognazzini, explained in his book Tog on Interface^[10] that it was essential to maintain the illusion, and that the company should never let the user into the technical “engine room” where the real machinery could be found. Unfortunately, this is the problem with any society governed by magic – the person who defines and controls the magic gets to exercise a kind of power that can’t be questioned using the surface logic of non-magical everyday appearances.

The worst problem with metaphor was that it disguised the potential of abstract notation. If everything on the screen works like a physical object, there is no way to describe what ought to be done in the future (the future is an abstraction), or a policy that must be applied in multiple places (a class of situations is an abstraction). As I explained in chapter 7, Interacting with physical objects is so easy to understand, requiring minimal attention investment, because every action has only one effect. But minimal investment offers minimal return. If an icon is interpreted as a physical object rather than as an abstract notation, it expresses no computational power. If every object is just the way it is, it can never change in response to different contexts. Operations in the real, physical world are unavoidably repetitious, precisely because each action on a single object affects only that object. These are the things we lose, by treating the screen only as a mirror of the physical world, not capturing the “magical” potential of notational codes.

Although this chapter might have seemed like distant history, an urgent question for today is whether the emphasis on imitating human conversation in AI chatbots and LLMs might be a new kind of skeuomorphism. Thirty years ago, people were incredibly impressed by technology that delivered sharp pictures on a mobile device screen. They thought it was obvious that this graphical realism would lead to intuitive user interfaces. But as it turned out, imitating the physical world at an unprecedented level just made the computer more clunky. Today, we have reached equivalent levels of hype in delivering text that realistically simulates human conversation. Just as before, everybody is assuming that realistic simulation of something we are already familiar with will make a product more intuitive - the classic error of skeuomorphism. What if imitating real human conversation is actually a clunky way of interacting with computers, just as imitating real-world scenery was?

The pictures on the Magic Cap screen were technically impressive at the time, and briefly entertaining, but the additional demands on user attention were not justified by the amount of entertainment being offered. In the same way, interacting with a chatbot is often more time consuming and error-prone than simply using a standard search engine. Searching effectively does involve some investment of attention: forming an abstract concept of what you want, selecting the most effective keywords, navigating the thicket of advertisers and so on. But at least these abstractions are visible and manipulable - in fact, a kind of Moral Code. Do we really want to lose all this, by surrendering to the agreeable illusions of language models as our AI friends?

This chapter has zoomed in to look at notation-in-the-small, considering how even the design of icons and home screens reflects a set of policies that determine how regular users are able to invest their attention. The next chapter zooms right out, to see how some of the same principles apply even in huge software development projects, staffed by hundreds of programmers, writing thousands or millions of lines of code.

^[1] Apple Computer, Inc. Macintosh Human Interface Guidelines. (Reading, MA: Addison Wesley, 1992); Microsoft Corporation. The Windows Interface Guidelines for Software Design. (Redmond, WA: Microsoft Corporation, 1995).

^[2] e.g. Stephen. A Hill, Practical Introduction to the Human Computer Interface in a Semester. (London: DP  Publications, 1995), 22; Susan Weinschenk, Pamela Jamar, and Sarah C. Yeo, GUI design essentials: for Windows 95, Windows 3.1, World Wide Web. (New York: Wiley, 1997), 60; Wilbert O. Galitz, The essential guide to user interface design: an introduction to GUI design principles and techniques (New York: Wiley, 2007), 84; Theo Mandel, The elements of user interface design. (New York: Wiley, 1997), 69; Alan Dix, Janet Finlay, Gregory D. Abowd, and Russell Beale. Human Computer Interaction, 2nd ed. (Hemel Hempstead, UK: Prentice Hall, 1998), 149; Christine Faulkner, The Essence of Human-Computer Interaction. (Hemel Hempstead, UK: Prentice Hall, 1998), 89.

^[3] Ted H. Nelson, "The right way to think about software design," in The Art of Human-Computer Interface Design, ed. Brenda Laurel. (Reading, MA: Addison Wesley, 1990), 235–243.

^[4] Alan F. Blackwell, "Chasing the intuition of an industry: Can pictures really help us think?" in Proceedings of the first Psychology of Programming Interest Group Postgraduate Student Workshop, (1996), 13-24

^[5] Alan F. Blackwell, Metaphor in Diagrams. Unpublished PhD Thesis, Cambridge University, 1998; Alan F. Blackwell and Thomas R.G. Green, "Does metaphor increase visual language usability?" in Proceedings 1999 IEEE Symposium on Visual Languages VL'99 (1999), 246-253.

^[6] Alan F. Blackwell, "The reification of metaphor as a design tool," ACM Transactions on Computer-Human Interaction (TOCHI) 13 no. 4 (2006): 490-530.

^[7]  George Basalla, The evolution of technology. (Cambridge, UK: Cambridge University Press, 1988)

^[8] Alan C. Kay, "User interface: A personal view," in The Art of Human-Computer Interface Design. ed. Brenda Laurel. (Reading, MA: Addison Wesley, 1990), 191-207.

^[9] Henry Lieberman, "An example based environment for beginning programmers," Instructional Science 14 no. 3 (1986): 277-292; Henry Lieberman "Integrating user interface agents with conventional applications," Knowledge-Based Systems 11 no. 1 (1998): 15-23; Henry Lieberman, Bonnie A. Nardi, and David Wright, "Training agents to recognize text by example," in Proceedings of the third annual conference on Autonomous Agents (1999, April), 116-122; Henry Lieberman, ed., Your wish is my command: Programming by example. (San Francisco, CA: Morgan Kaufmann, 2001).

^[10] Bruce Tognazzini, Tog on Interface. (Reading, MA: Addison-Wesley, 1992).