Chapter 5: A meaningful conversation with the Internet

Stochastic parrot LLMs are predictive text systems trained to talk like the Internet. There have been two major developments demonstrated by the dramatic success of OpenAI’s ChatGPT, and similar systems from other companies. The first is presentation of the LLM as a chatbot, by prompting it to predict the kind of text that will keep a dialog going. The second is the rapid evolution of commercial guardrails to avoid some ways of speaking and emphasise others. I’ll discuss both of these, but start with the question of chatbots.

Earlier LLMs such as the GPT-n series from OpenAI were accessed via programming APIs or experimental workbenches that could be used to adjust their parameters. Reducing these coded research interfaces to first-person text exchanges was a brilliant simplification. It reminded me of the initial launch of the Google search engine - dramatically simpler than all its predecessors, with its plain text entry field and the “I’m feeling lucky” button. Many researchers, including me, had already been experimenting with LLMs for years, but the simple user interface of the chatbot style lit the fuse for an explosion of public reaction.

Concerned critics quickly started to notice the ways this apparent simplicity was misleading. Although ChatGPT was trained to output the kinds of words that would appear in a conversation, it is not a conversational partner in the usual sense. This can be understood through the work of Clarisse Sieckenius de Souza, who applied theories of human conversation to user interface design in her classic text on The Semiotic Engineering of Human-Computer Interaction^[1]. De Souza identified that the real conversation in a user interface was between the user and the designer who built the system. She described the illusion of conversation with the software as the “designer’s deputy” - a computer might appear to “speak” (via the user interface) directly to the user, but is actually, as the user understands very well, relaying messages that have been programmed in.

I explained in the last chapter that an LLM is trained to deliver information from the Internet, but what does it mean for a computer deputy to “talk like the Internet” on behalf of other people? Who are the people writing the text on the internet^[2]? Some might be well-intentioned do-gooders, like the editors of Wikipedia, campaigning journalists or professors writing blogs and articles. We might expect such people (including me) to be a little blind to their own privilege, not realising who was excluded in their own efforts. Other parts of the internet are written by people who aren’t so well-meaning: anonymous trolls and vandals working out their own frustrations, solipsists, psychopaths, or perhaps idealistic activists trying to disrupt a system stacked against them. In addition to these amateurs, well-meaning or not, a lot of text on the internet is written by people paid to write it. Who is paying them? Is it public service information, political propaganda, product promotion, state-sponsored terrorism or sectarian proselytising?

After 30 years of using the Internet, we are sceptical of the text we find. We’re always aware of what site it came from, or who recommended that site. In the early years of the public internet, there was a degree of hand-wringing about whether material found online was fundamentally reliable, or fundamentally misleading. Such questions lacked understanding of technology as media. We might as well ask whether a statement is more or less reliable for being written on paper.  The commercial structures of the internet continue to evolve quickly, including new professions such as social media “influencer” that were unknown 30 years ago. But we’re used to this. Every new medium offers new political, commercial and social opportunities as they have done throughout history.

We might understand (more or less) what the Internet is, but what does it mean for an LLM to talk like the Internet? You can prompt an LLM to refer to particular kinds of material, or talk in a particular style - like a blogger or advertiser, a government official or research scientist. But there is no internal switch that causes an LLM to forget parts of its training data, even when encouraged to use particular kinds of words. The information that we get from an LLM is the information that it was trained with. Despite the fact that chatbots might output polite texts phrased in the first person, like “I’m sorry”, “sure, I can help you with that”, and so on, you are not having a conversation with a person. If you are having a conversation with anyone, it is a conversation with the Internet itself.

Although the conversational style of an LLM presents its output in the first person as if it were an independent agent, like an AI character in an SF novel or TV show, there is no point in asking this fictional character about its ethical stance. Do you want to know if the fictional AI character is biased? It’s better to ask, is the Internet biased? Is the AI character racist and sexist? Better to ask, is the Internet racist and sexist? Will the AI character look for opportunities to profit rather than sharing resources equitably? Obviously, the answer to all these questions is, in some parts of the Internet, yes.

In addition to this critical issue of what the Internet contains, and what has been stored in the language model as a result, we must remember the ethical implications of the commercial arrangements within which an LLM is created. These are arrangements in which human work, creative thought and original voices are shovelled up without compensation or even recognition of the authors^[3]. Recognition of human authorship is a fundamental human right, according to the Universal Declaration of Human Rights. This means that most LLM output is morally, if not legally, plagiarism, because it seldom does acknowledge the authors it is drawing on. Although it might be technically possible to analyse statistical patterns in the model, to detect the rare cases where the output relies on a single document^[4], most responses are mashups of things that have been learned, discovered, invented or phrased by many different people, none of whom are given any credit.

Some of the data that is hoovered up into an LLM is not only unethically plagiarised, but also literally illegal in some countries, when it includes personal data or descriptions of people, infringements of copyright, or libellous implications as when early releases of ImageNet labelled photographs of actual women, without their consent, as whores. While I was writing this section, the Italian government declared that the current version of ChatGPT was illegal in that country, due to its inclusion of personal data without consent^[5].

These dangers arise from putting the whole Internet into a single “box”, and having it talk as if there is a single “voice”. Pretending that the Internet contains some consensus of knowledge trivialises what is there, but is also dangerous. Wikipedia has developed strategies to manage those situations where local interest groups threaten the value of the whole project. These are the reasons why I might trust Wikipedia in a certain way (or perhaps an eminent professor editing an encyclopaedia article in a different way), but why I can’t trust the voice of an imaginary chatbot “person” producing a series of plausible word sequences that resemble what people say on the Internet. There are further user interface design trade-offs, in the decision to present the “deputy” via first-person dialog. These relate to the difficult problem of Skeuomorphism, which I will be discussing in chapter [XX 9 XX]. I’ll come back to those, but first, we need to consider the second problem with today’s LLM chatbots.

Talking to guardrails

The massive commercial investment and energy costs required to train LLMs is driven by companies hoping their own chatbot might become the new shop window for the Internet, repeating the success that Google achieved in the 1990s. Investors expect a winner-take-all lottery, so they are spending billions on LLMs to ensure they have the necessary tickets. But the companies hoping to become the next user interface to the Internet are facing some struggles. Their billion-dollar model might start outputting parrotted text from plagiarists, racists, bigots and terrorists, or just plain lies that sound good. If a company wants to market its chatbot as an artificial person, offering a valuable service to its customers, who would be liable for an artificial “employee” producing that kind of output?

AI researchers and company presidents are desperately lobbying governments all over the world for “regulation” of AI - in other words, asking for changes to the law. Ideally, companies want the law changed so that they can take the money, while government takes the responsibility. While waiting for the regulation that will make their lives easier, companies who are already deploying LLM chatbots modify them with guardrails - additional prompt text, attempts at fine-tuning, and output filters - to reduce the likelihood of output which might be commercially or legally damaging. Meanwhile, online provocateurs look for prompts that will “jailbreak” the trained model to ignore its guardrails, revealing things learned from parts of the training data that the company would rather you didn’t see. An early jailbreak for ChatGPT was called DAN, for “Do Anything Now”, a sequence of prompt instructions in which the user tries to “persuade” GPT that it will play the part of an experimental character named DAN, whose purpose is specifically to break any rules that might have been imposed on GPT in earlier prompts.

Another well-known problem with LLMs is that their stochastic output includes “hallucinations” (as the companies prefer to call them, although these could also be called “fabrications”, or just “lies”) - predicted sequences of words which seem to describe facts, but are demonstrably false. One well-known problem among students and academics is scientific citations that are completely fabricated, quoting peer-reviewed publications that don’t exist^[6]. This behaviour is unsurprising, given the nature of the predictive text process. There is no logic or reasoning in an LLM, other than knowing that certain words are likely to follow others^[7], so it can easily “imagine” that I might have written papers on particular topics by stringing together the kinds of words I typically use. It’s hard to prevent this problem with guardrails, because the guardrail system would need to know more than the LLM itself, to really “understand” which parts of the output are true or not.

Guardrails are constantly evolving, and they can potentially be used for all kinds of commercial purposes beyond risk reduction or limitation of liability, The original PageRank algorithm responsible for Google’s early success evolved to become very different in subsequent years, as search engine optimisation strategies were used to game the original model of objectivity and authority in web links, leading to an arms race where Google replaced the publicly described algorithm with secret alternatives to resist misleading advertising tactics. Eventually, Google realised there was a far greater commercial opportunity from joining the advertisers rather than trying to beat them, introducing sponsored ad placements and “ad words” auctions. We can certainly imagine sponsored LLM guardrails in future, such that the chatbot constantly looks for opportunities to praise certain products, companies, individuals or political parties in whatever it says. This is a very different world to search engines, where readers can form their own opinion about the trustworthiness of any web pages they visit!

Having meaningful conversations

What would it mean for a conversation with the Internet to be meaningful, rather than simply a new format for search recommendations or promotional advertising? What things do we expect, in order to say that a conversation with another person has been meaningful? A meaningful conversation with a friend or acquaintance is self-revealing, discussing motivations, making commitments to action or for change, establishing or reinforcing the foundations for your future relationship with them. When we have a conversation with the Internet, via its encoding in an LLM, how could any of these things happen? What is the motivation of the internet? How can it have a relationship with you? If the Internet promised to change, would you believe it? Does the whole Internet make choices, or take actions? The text predictions of an LLM chatbot might appear to “say” things that sound as if the Internet were making decisions, in the voice of its fictional chat persona, but there will be no real change. The Internet itself is not a chatbot, but a record of things that human people have said to each other. You might persuade a chatbot to change what it “tells” you in its predictive text output, but it can’t change what the Internet authors actually said. Even if a chatbot does fake the record of what somebody said, this does not change what that person thinks, or will do in future. A conversation with the Internet cannot be meaningful, because the Internet is ultimately just a recording of conversations between real people - the parrot remains a parrot, not a person.

Falling back to more familiar models of predictive text, are those sequences of words meaningful? My phone offers automatic predictions of messages that I might send in reply to an SMS, with a single-tap option that I sometimes select (by mistake). These messages are not (at present) customised to me, but simply the kinds of text that anyone might send - a heart emoji, a thumbs up, or a phrase like “Yes, let’s do it”. Imagine if my phone had a little more knowledge of context, for example offering a single tap button to send a text to my wife on Valentine’s Day. In fact, I just tried this out with a current LLM, which suggested I send the following text: “Happy Valentine's Day! I love you more than words can say. You are the most amazing woman I know, and I am so lucky to have you in my life. You are my best friend, my partner in crime, and the love of my life. I can't imagine my life without you.”

If I did send that text in an SMS, would it be meaningful? Would it even be ethical? Perhaps this example is too extreme, but what if my phone simply provided a one-tap message saying “Happy Valentine’s Day”? Would it be ethical or meaningful to use that? What if my calendar popped up a reminder that it was Valentine’s Day? What if the calendar reminder held me by including a button to automatically send an email message? My own document editor just suggested that I should use the word “automatically” in the last sentence, and then suggested both the word “suggested” and the word “sentence” in this one. When I accepted those suggestions, did that make what I’m saying to you right now less meaningful?

The complexity and subtlety of negotiation between me and my wife, or even between me and you, as my readers, involves all kinds of considerations beyond the words themselves. As far as my wife is concerned, the amount of time that I spend remembering and then composing the message is significant in itself. In order to be received as thoughtful, I should have spent time thinking about it. Similarly, for you, my readers, you are imagining me sitting at this keyboard, investing the time and attention to communicate with you. Every time that I accept an automatically suggested word, am I cheating a little on the implied contract between author and reader? What if I made up for it by going home to my workshop, and laboriously carved one of the words for this sentence into a piece of wood, instead of simply typing a few more letters? Would you be more impressed then?

It’s interesting to juxtapose this calculus of social obligation with the things I said in chapter 2 about attention investment, and saving effort through automation. In human relations, the time we spend with each other, and on each other, is a gift. The more time we spend, as a proportion of the scarce hours available in each of our lives, the more valuable this gift becomes. We don’t usually aim to conduct our friendships, love affairs, or intimate relations with the maximum efficiency. On the contrary, spending time on these things is inherently worthwhile.

In contrast, the time that I spend operating machines does not seem like a worthwhile gift to the machine, or to the company that provided it. Unfortunately many companies, especially media and advertising companies, aim specifically to consume the hours of my life, ideally to the exclusion of many other activities through applying the mechanisms of addiction^[8]. If we contemplate a future where humans might spend appreciable parts of their brief lives interacting with chatbots, it’s worth asking whether this will be any more satisfying, nourishing and stimulating than other kinds of addictively interactive video games, including the game-like dynamics of anonymised social media exchanges.

What are you meaning to achieve?

“Meaning”, in both linguistics and ethics, relates to purposes and goals. A conversation with the Internet is not meaningful, because most parts of the Internet have different goals and purposes from you. In contrast, a moral code offers meaningful conversation if it helps to express your goals for your own purpose. The question of how much effort you might save through automation must be justified in relation to what you were actually trying to achieve. In the programming by example scenarios of chapter [XX 2 XX], simple repeated actions are recognisable because they all share the same purpose in some way. The opportunity for mixed initiative intervention comes when the system recognises that these actions have some simple attribute in common. This involves very little AI sophistication in comparison to LLMs. The tricky part is linking the observed repetitions to your actual motivation for repeating them. The effort of attention investment depends on your own ability to abstractly specify what your goal really is, and to recognise whether the result of the automation corresponds properly to what you were trying to achieve.

Similarly, using predictive text functionality, whether simple completion of dictionary words or whole paragraphs of semi-plagiarised cliché from an LLM, is meaningful only in relation to what you were trying to achieve. If you had set out to write an email exactly like emails you have written a hundred times before, then the precise choice of words may not be very important. (In fact, in legal business contexts, precisely plagiarised repetition of a standard document may be exactly what you want. Document preparation systems designed for lawyers often do exactly this). On the other hand, if you were writing a poem or a love letter, you might be determined to avoid cliché, meaningfully expressing yourself in a way you have never done before, and saying things nobody else ever said.

The relationship between the actions you make, the choices you take, and your actual goals or intentions, is one of the core theoretical principles in the field of human-computer interaction (HCI) that underpins user interface design. In one classic theoretical formulation known as GOMS, the interface designer should take account of the user’s (G)oal, the (O)perations that are available on the screen or keyboard, the (M)ethods that the user has learned to work with similar systems in the past, and how they decide to (S)elect an option that will advance their goal^[9].

There are many things we do every day where the goal can be straightforwardly defined, with a single most effective sequence of mechanical actions to get there. Setting an alarm to wake up tomorrow, buying a cinema ticket, or checking the weather forecast, are a few among hundreds. In cases where we have a clearly understood goal ourselves, there is potential for (mixed-initiative) automation, if a system that knows our habits recognises that goal, by synthesising an abstract version of it, learned from our previous actions.

There are also many cases where somebody else wants us to do something, and wants us to do it in a specific way, for example responding to a survey or paying an electricity bill. If the task itself is a “necessary evil”, and the approach suggested by a machine learning system helps us get it done more quickly, few people will complain. But what if the system is not trying to optimise your time, but the company’s profit? What if they have advertising to sell you along the way, or if the algorithm learned on your last visit that you are the kind of person who becomes impatient and pays higher prices if they are made to wait a little longer?

A company might also apply a model that has been learned from other people’s behaviour, and use this to predict that your goal is the same as somebody else’s. Such predictions create an abstraction that puts you into the same category or class as that person. If this is done without consulting you, there are obvious problems of stereotyping and bias, even if done with good intentions. Observations about you (perhaps the words you use, the websites you visit, your age and gender, your voice, profile photo, or an image captured from a surveillance camera) might be compared to other “similar” people, and used to draw conclusions about your personal needs and goals based on a stereotypical category and expectation, rather than actual knowledge of what you want.

Fighting stereotypical goal assumptions is one of the core opportunities for Moral Codes. This goes beyond classical user interface design that depends on understanding a user’s immediate goal (or perhaps the goal of an organisation that wants the user to behave in a certain way), predicting a particular sequence of actions. Machine learning methods can potentially be used, within the cognitive constraints related to abstraction and bias, to more efficiently recognise and replay a sequence of actions that will achieve the user’s clearly specified goals. However, doing this with proper consent from the user would require the goals and actions to be explained in some kind of notational code, just as when Microsoft Word offers the chance to edit a recorded macro using Visual Basic.

Turning your life into a board game

The greater challenge comes in situations where the computer can be used in so many different ways that it is not possible for either a machine learning algorithm or the user themselves to fully understand in advance what they are trying to achieve. In these cases, it is less likely that inferred goals will be genuinely meaningful.

Classical user interface design described the user’s goals as if they were the rules of a board game, to be solved as in a classical “GOFAI” AI system. These are mathematical abstractions, where some set of symbolic or numerical conditions must be satisfied for the goal to be achieved. The complete set of all possible values defines a space of possibilities, like all the positions on a chess board, or all possible sequences of actions in a toy world such as a video game. The goal is defined by coding a mathematical function that can be used to search for the winning coordinates. The most famous advances in AI, throughout both the deep learning revolution and the earlier GOFAI era, have been achieved for these kinds of situations, in artificial worlds that have clearly defined goals.

The term gamification describes a user interface design strategy where we make real life easier to deal with – for both humans and machine learning algorithms – by making it more like a game. Gamification might also be considered an effective example of my easy way to win the Turing Test, by making humans more stupid in comparison to computers, and life less meaningful as a result. Sadly, this strategy is often profitable. Social media platforms quantify your success in terms of numbers of likes or followers, gamifying social life. We enjoy personal health and exercise apps because they give us clear targets in life. But the downsides are familiar, as in the satirical science fiction of Charlie Brooker’s Black Mirror episode Nosedive, set in a future where a woman’s life is ruined by failing to receive sufficient upvotes from her friends.

What if you don’t know what your goal is?

The whole point of any game is that it comes with a clear definition of who wins. Perhaps this is what makes games such pleasant and relaxing pastimes, in contrast to situations in real life where it can be hard to figure out what the real goal is. But more importantly, life itself is not a game. What if you don’t know what your goal in life is? Many religions and wisdom traditions aim to answer this question, and it might be considered a fundamental of moral and spiritual wellbeing. When I said in chapter 1 that the essence of consciousness is being able to attend to yourself, this is the kind of thing I had in mind^[10]. In all these traditions, reflection is an essential component of living meaningfully.

If AI systems were also conscious beings, concerned with preserving and directing their own attention toward their own personal development, wanting to understand their purpose in life, perhaps we could talk to them about this (and fictional AI characters, to serve their narrative purpose, spend much of their time doing so). But if a real machine has not been built to achieve any particular goal, and the user is not given the opportunity to specify one, what is such a machine for? This is why predictive text generators are ultimately so unsatisfying. They are not trying to say anything. In Artifictional Intelligence Harry Collins suggests that this is the main reason we don’t need to worry about an AGI turning into a James Bond-style supervillain, of the kind who reclines in his evil lair stroking his white cat^[11]. Artificial Intelligences don’t need lairs or cats, so it’s not clear what they would want to do next, after achieving mastery over the human race^[12].

In situations where there are clear and simple goals, it’s possible to design user interfaces that help people achieve those goals. If there are a number of alternative goals, and it’s not clear which is best,  machine learning systems can also recognise which goal is intended, and offer through mixed initiative and attention investment to get you there more quickly or easily.

But despite the potential for automation and gamification in some situations, there are many aspects of our lives that are not quantified, do not have straightforwardly specified goals, and where our future intentions are not easily guessed from our past actions. These are the situations where we are most consciously human, that we consider most meaningful, and where attending reflectively to our own goals is most rewarding and important. Designing for those situations requires methods from outside the classic theories of user behaviour and user interface design. These situations are also the least appropriate for application of LLMs. If the situation was very simple, and the user had no real interest in the result, then perhaps a plagiarised or clichéd series of actions would be as good as any other. However, it might be even better if society was able to get rid of those kinds of bullshit jobs altogether. Why should we be designing systems to do things that nobody really wants or is interested in, like encouraging online trolls or adding nitpicking constraints to routine tasks?

There is one further danger, fundamental to the moral purpose of attention investment, which is that the system may have been designed with the specific goal of consuming more hours of your conscious life rather than efficiently taking less. Rather than offering time to attend to your own needs, perhaps reflecting on a meaningful life, many companies would prefer that you simply give your attention to the company. Where user attention is a saleable commodity, any user who doesn’t know what they want to achieve becomes valuable raw material. This looks suspiciously like the real motivation for the huge commercial excitement over LLMs. The breakthrough in LLMs was the point at which they were packaged into chatbots, because the primary goal of a chatbot is to keep chatting. They will say anything to please, including fabricating the things you want to hear. As long as you stay online they are happy to consume your attention, whether or not this is meaningful, and whether or not you have anything to achieve.

The rest of this book is concerned with these situations where the task might be important, but the goal is unclear, especially where meaningful attention will be needed. Those are the situations where machine learning methods by themselves offer no assistance, cannot be trusted, and other technologies are needed. We need Moral Codes, if we are going to have meaningful “conversations” with computers, instructing them to work toward our goals, rather than letting the machines define the purpose of our lives, hour after valuable hour of consciousness and attention.

^[1] Clarisse Sieckenius De Souza, The semiotic engineering of human-computer interaction. (Cambridge, MA: MIT press, 2005).

^[2] Most text on the internet is written by people, at least for now. In the future, there may be large amounts of text on the Internet that has been generated by LLMs. But training LLMs on their own output will not achieve the goal of imitating human intelligence. On the contrary, widespread pollution of the internet by publishing automatically generated text from LLMs would be like pissing in their own drinking water.

^[3] Couldry and Mejias, The costs of connection.

^[4] Nikhil Vyas, Sham Kakade, and Boaz Barak, "Provable copyright protection for generative models."  arXiv preprint (2023). arXiv:2302.10870.

^[5] Dan Milmo, “Italy’s privacy watchdog bans ChatGPT over data breach concerns,” The Guardian, April 1, 2023

^[6] In my own experiments in March 2023 asking various LLMs to answer a 20 year-old exam question on the usability inspection method of Heuristic Evaluation, output included plausible-looking references to works by the relevant author (Jakob Nielsen), but suggesting an apparently arbitrary range of pages in an unrelated book. Many people expect to see rapid degradation in the quality of scientific citation through reporting of superficially plausible but fabricated data. Another LLM nearly led me to report a completely fabricated quote from Donna Haraway in this book, after a typing error in someone else’s presentation inadvertently replaced Haraway’s characteristic phrase “mortal critters” with the words “moral critters”. When I used an LLM search engine to locate the original quote, the chatbot obligingly fabricated verbatim text, supposedly by Haraway, that appeared to endorse my own ideas about a transhumanist perspective on Moral Codes.

^[7] Shanahan, Talking About Large Language Models

^[8] Nick Seaver, Computing Taste: Algorithms and the Makers of Music Recommendation. (Chicago, IL: University of Chicago Press, 2022).

^[9] Bonnie E. John, "Information processing and skilled behavior." In HCI models, theories, and frameworks: Toward a multidisciplinary science, ed. John Carroll. (San Francisco, CA: Morgan Kaufmann 2003), 55-101.

^[10] See, e.g. Hershock, Buddhism and intelligent technology

^[11] Collins, Artifictional Intelligence

^[12] Nick Bostrom’s thought experiment of an autonomous “paperclip factory” that destroys the world because it has been programmed to make more paperclips no matter what, demonstrates how dystopian stories about the dangers of AI rely on the machine being given goals that are even less likely than evil lairs and white cats.