Appendix: Further Reading

This book deals with topics that I have worked on for over 40 years. There has only been space for brief introductions to some subjects where I have published far more comprehensive scientific and technical descriptions. This appendix suggests further reading, for people who would like to understand a specific topic in more detail.

Design of visual notations

The “patterns of user experience” design framework presented in chapter 14 is explained in greater detail in a textbook chapter for design students: A pattern language for the design of diagrams^[1]. I’ve spent many years puzzling over the best way to communicate human factors considerations to programming language designers, building on the agenda of my PhD supervisor Thomas Green, whose Cognitive Dimensions of Notations framework was previously described in textbook chapters such as Notational systems - the Cognitive Dimensions of Notations framework^[2]. These frameworks concentrate on the way that we interact with representations, but the linguistic/semiotic analysis of representation is an essential element, as developed in my work with Yuri Engelhardt, and introduced in an encyclopedia article on Visual Representation^[3].

Several chapters of this book have drawn on the history of user interface development, especially work done at MIT by Ivan Sutherland, at Stanford by Doug Engelbart, at Xerox PARC by Alan Kay, and by Smalltalk gurus Kent Beck and Ward Cunningham. More extensive descriptions of that historical material can be found in a critical history describing The reification of metaphor as a design tool^[4], an editors' introduction to the online version of Sutherland’s Sketchpad, A Man-Machine Graphical Communication System^[5], and historical background to the creation of PUX: Patterns of User Experience^[6].

I’ve spent a considerable amount of time arguing with programming language designers who would like to think that purely mathematical or “evidence-based” approaches are adequate to address the problems of instructing computers, rather than the cognitive, semiotic and phenomenological approaches to design that I consider necessary. Some of these campaigning arguments include the prize-winning Metacognitive Theories of Visual Programming: What do we think we are doing?^[7] A later, more light-hearted comparison of programming language researchers to advocates of medieval Latin, was 6,000 years of programming language design: a meditation on Eco’s perfect language^[8].

Understanding coding as a human practice

The campaign to make programming languages that are more usable and accessible to a wide range of people is most often described as “end-user” programming. A comprehensive programme directed by Margaret Burnett for the U.S. National Science Foundation introduced many factors that continue to be relevant to future research, as surveyed in The State of the Art in End-User Software Engineering^[9]. My own contributions to this field have focused on studies of the human factors, with contrasting perspectives on universal considerations and varieties of difference: Psychological issues in end-user programming^[10] and End-user developers - what are they like^[11]?.

All of this work is informed by communities dedicated to the psychology of programming and empirical studies of programmers, as surveyed with Luke Church and Marian Petre in Fifty years of the Psychology of Programming^[12]. Luke has been an especially close collaborator for many years, and his own PhD research has addressed the challenges of influencing programming language design within some of the largest companies in the global software industry.

An emphasis throughout this book has been that we need a variety of codes, not a single way of programming computers. Investigations of this variety have included comparisons of different professional styles in The Practices of Programming^[13], and also the effect of biases and stereotyped expectations in Tinkering and gender in end-user programmers' debugging^[14]. In recent work, I have considered the special requirements of programming in creative contexts, building on work done by PhD student Chris Nash to explore Liveness and Flow in Notation Use^[15].

At the time of writing, I have a large portfolio of projects under way investigating more usable “probabilistic programming languages” - a class of language that is being widely used for machine learning, data science and AI research. Members of my group are working on designs for many kinds of novel notation, using the methods described in this book, and following an agenda set out with leading technical researchers in the field as Usability of Probabilistic Programming Languages^[16].

Prototypes and case studies

My research philosophy is that design can only be properly understood by designing. Now that I am a university professor rather than a professional designer of programming languages, I seldom have time to build whole new languages. However, I have been fortunate to work with students and collaborators to explore many of the ideas described in this book. The more detailed descriptions of those projects (some of which have been mentioned in the main part of the book, especially chapter 8), are far better presentations of the problems they were intended to solve, and the technical realisation of working languages. 

As I already discussed, Mariana Mărăşoiu’s Cuscus allowed visualisations to be programmed in a kind of spreadsheet, as described in Cuscus: An end user programming tool for data visualisation^[17]. Advait Sarkar created a whole series of prototypes that challenged the distinction between programming and visualisation in exploratory data science, including Visual discovery and model-driven explanation of time series patterns^[18].

Hanna Wallach was one of a series of students who built experimental systems as undergraduates, investigating Diagrammatic integration of abstract operations into software work contexts^[19]. The years of experiments “layering” abstraction over depiction led eventually to my own Palimpsest: A layered language for exploratory image processing^[20].

Alistair Stead, following experiments in live music notation on whiteboards that were inspired by the work of live coding pioneer Alex McLean, went on to create a teaching environment that turned children’s drawings into the code for animated web pages: Learning Syntax as Notational Expertise when using DrawBridge^[21]. Cecily Morrison and her team at Microsoft Research Cambridge built an amazing product for teaching programming via sound and touch: Torino: A tangible programming language inclusive of children with visual disabilities^[22].

My own understanding of the mathematics of machine learning is owed wholly to my friend David MacKay, who was a transformative figure in this and many fields. His life was far too short, and we can only imagine what else he might have given the world. It is truly astonishing that his Dasher - a Data Entry Interface Using Continuous Gestures and Language Models^[23], which I worked on together with David Ward 25 years ago, is still an effective introduction to the usability problems of predictive language models in my graduate classes today.

Luke Church has built even more languages than I have (and certainly been more successful in persuading companies to deploy them). He has been doing this ever since his undergraduate dissertation, which built on David MacKay’s ideas to apply language-model approaches to visual programming in Structured text modification using guided inference^[24].

Coding for creative arts

I’ve had the privilege of a front-row seat (and occasionally a seat onstage) working with the creative innovators who have developed programming languages and methods into the performance art of live coding. A compendium of practice and philosophy has recently appeared in Live Coding: A User’s Manual^[25]. 

Years of collaboration with Sam Aaron led to the development and publication of his open source Sonic Pisystem, now one of the most popular languages for live-coded music worldwide. Sam and I published many papers on this work, including one that reflects on the process of designing such languages in a creative way: Craft Practices of Live Coding Language Design^[26].

Dance research offers an especially stringent critique of disembodied intelligence, and Scott deLahunta has been a long-term collaborator, especially in our series of projects with Wayne MacGregor’s Random Dance company. One especially ambitious project, working with some of the world’s leading artist-programmers, was Sketching by programming in the Choreographic Language Agent^[27].

Critical understanding of AI

As I said at the start of this book, I have not set out to write about “AI Ethics,” a field in which there are increasing numbers of specialists, including philosophers and legal scholars, who are dedicating their careers to that topic. My own awareness of the philosophical issues discussed in this book has developed more slowly, and usually in addressing my own discomfort about contradictions or inconsistencies that I saw in the work of the computer scientists around me.

Some of the core questions underpinning my critique of AI have occasionally been presented in papers for more specialist audiences, such as The two kinds of Artificial Intelligence, or how not to confuse objects and subjects^[28] and Wonders without number: The information economy of data and its subjects^[29].

Questions about the economic and political significance of machine learning methods are analysed in more detail in papers drawing on collaboration with colleagues in other countries and disciplines, building on conversations with Tesfa Tegegne and Addisu Damena in Inventing artificial intelligence in Ethiopia^[30], and ideas developed with Leonardo Impett in Artificial intelligence and the abstraction of cognitive labour^[31].

My use of Claude Shannon’s information theory to understand creativity owes a lot to casual conversations with David MacKay, but also to very early analysis of this question by media scholar Umberto Eco, as discussed in What does digital content mean? Umberto Eco and The Open Work^[32].

The Aarhus conferences in 1975, 1985, 1995 and 2005 have been at the leading edge of thinking about sociotechnical and design approaches to computing. As a member of the organising committee for the 2015 Aarhus conference, I felt it was important to investigate the imminent challenges of machine learning in Interacting with an inferred world: The challenge of machine learning for humane computer interaction^[33]. Some years later, I looked in a more focused way at the project that had kicked off the deep learning revolution, and the ethical issues that had been there from the start, in The Age of ImageNet’s Discovery^[34].

The problems that arise when AI methods are used to classify and simulate human emotions are now widely understood to be challenging. Some years before this came to public attention, I noticed the contradictions between assumptions being made by the machine learning experts who were building these systems, and philosophers of mind with whom I was discussing more diverse perspectives on human experience, in When systemizers meet empathizers: Universalism and the prosthetic imagination^[35].

Cognitive theory of attention investment

The idea that programming is fundamentally different to direct manipulation, and that this difference relates to how we choose to spend our time, is something that I came to understand only gradually, over more than 10 years, starting with a suggestion made by my PhD supervisor Thomas Green.

The most comprehensive ‘traditional’ HCI experimental investigation of these experiences was presented in the context of domestic automation: How do we program the home? Gender, attention investment, and the psychology of programming at home^[36]. Conversations with a friend in the BBC led to an early understanding that the attention economy was related to its programmability: Programming culture in the 2nd-generation attention economy^[37].

There were a couple of papers in which I first managed to explain for a computer science audience how significant this concept might be (and indeed I later won a prize for the influence that this work had): First steps in programming: A rationale for Attention Investment models^[38] and What is programming^[39]? The basic idea was there a few years earlier, developed during my PhD studies with Thomas Green, although I had struggled to explain how it was relevant to design problems: Investment of Attention as an Analytic Approach to Cognitive Dimensions^[40] and Ironies of Abstraction^[41]. With advice from David MacKay, I did build a prototype that used simple machine learning methods to help the user recognise and create abstractions for a repeated task: See What You Need: Helping end users to build abstractions^[42].

Working across disciplines

In much of my research, the challenge has been to bring together ideas from different disciplines in order to get a more holistic understanding of a fundamental problem that does not seem to be properly addressed from any single academic tradition. Working with different disciplines is hard to do well, despite the popularity of the principle, and I’ve published a fair amount of advice on how to do it, as well as some historical study of what happens when this is needed in HCI research.

I was commissioned to lead a UK policy research project, investigating how interdisciplinary methods can be made to work and when they are most helpful, with a policy report resulting: Creating value across boundaries: Maximising the return from interdisciplinary innovation^[43]. A more personal manifesto, including historical anecdotes from Cambridge, is set out in Design research and academic disciplines^[44]. A longer and more detailed report includes case studies of leading UK practitioners, with analyses of their experiences in relation to national research policy: Radical innovation: crossing knowledge boundaries with interdisciplinary teams^[45].

The field of human-computer interaction (HCI) has always had a delicate relationship with computer science, not only because it has to draw on other disciplines, but also because much of what it has to say is critical of things that computer scientists would rather believe. I’ve contributed to debates about the status of HCI in HCI as an inter-discipline^[46] and Filling the big hole in HCI research^[47]. I also investigated these questions through a historical case study of the collision between social science and technical innovation, including the AI agenda of the 1980s, in Undisciplined disciples: everything you always wanted to know about ethnomethodology but were afraid to ask Yoda^[48].

Design always crosses academic disciplines, because every design project must engage with both human and technical problems. However, design itself is far from homogeneous, and it’s important to understand how design looks different (or the same) depending on what you are designing, as explored in the project Witnesses to design: A phenomenology of comparative design^[49]. That project included a theme specifically related to the visual representations used by designers as part of their creative process, in the context of AI systems: Sketching across design domains: Roles and formalities^[50].

^[1] Blackwell, “A pattern language for the design of diagrams”

^[2] Alan F. Blackwell and Thomas R. G. Green, "Notational systems - the Cognitive Dimensions of Notations framework," in HCI Models, Theories and Frameworks: Toward a multidisciplinary science, ed. John M. Carroll. (San Francisco: Morgan Kaufmann, 2003), 103-134.

^[3] Alan F. Blackwell, "Visual Representation," in The Encyclopedia of Human-Computer Interaction, 2nd Ed, ed. Mads Soegaard and Rikke Friis Dam. (Aarhus, Denmark: The Interaction Design Foundation, 2003). Available online at https://www.interaction-design.org/encyclopedia/visual_representation.html

^[4] Blackwell, “The reification of metaphor as a design tool”.

^[5] Sutherland, Sketchpad

^[6] Blackwell and Fincher, “PUX: Patterns of User Experience”.

^[7] Blackwell, “Metacognitive Theories of Visual Programming”

^[8] Alan F. Blackwell, "6,000 years of programming language design: a meditation on Eco’s perfect language". In Conversations around semiotic engineering. ed. Simone Diniz Junqueira Barbosa and Karin Breitman (Berlin: Springer, 2017), 31-39.

^[9] Ko, Abraham et al, “The State of the Art in End-User Software Engineering”

^[10] Alan F. Blackwell, "Psychological issues in end-user programming," in End User Development, ed. Henry Lieberman, Fabio Paternò and Volker Wulf, (Dordrecht: Springer, 2006), 9-30.

^[11] Blackwell, “End-user developers - what are they like?”.

^[12] Alan F. Blackwell, Marian Petre, and Luke Church, "Fifty years of the psychology of programming," International Journal of Human-Computer Studies 131 (2019): 52-63.

^[13] Bergstrom and Blackwell, “The Practices of Programming”

^[14] Beckwith, Kissinger et al, “Tinkering and gender in end-user programmers' debugging”

^[15] Nash and Blackwell, “Liveness and Flow in Notation Use”

^[16] Alan F. Blackwell, Luke Church, Martin Erwig, James Geddes, Andy Gordon, Maria Gorinova, Atilim Güneş Baydin, Bradley Gram-Hansen, Tobias Kohn, Neil Lawrence, Vikash K. Mansinghka, Brooks Paige, Tomas Petricek, Diana Robinson, Advait Sarkar and Oliver Strickson, “Usability of Probabilistic Programming Languages”. In Proceedings of the Psychology of Programming Interest Group (PPIG), 2019.

^[17] Mărăşoiu, “Cuscus: An end user programming tool for data visualisation”

^[18]Advait Sarkar, Martin Spott, Alan F. Blackwell, and Mateja Jamnik, "Visual discovery and model-driven explanation of time series patterns," in Proceedings of IEEE Visual Languages and Human-Centric Computing (VL/HCC), (2016), 78-86.

^[19] Alan F. Blackwell and Hanna Wallach, "Diagrammatic integration of abstract operations into software work contexts," in Diagrammatic Representation and Inference: Second International Conference (Diagrams), (2002), 191-205.

^[20] Blackwell, “Palimpsest”

^[21] Alistair Stead and Alan F. Blackwell, "Learning syntax as notational expertise when using Drawbridge," in Proceedings of the Psychology of Programming Interest Group (PPIG), 2014, 41-52.

^[22] Cecily Morrison, Nicolas Villar, Anja Thieme, Zahra Ashktorab, Eloise Taysom, Oscar Salandin, Daniel Cletheroe, Greg Saul, Alan F. Blackwell, Darren Edge, Martin Grayson, and Haiyan Zhang, "Torino: A tangible programming language inclusive of children with visual disabilities," Human–Computer Interaction 35, no. 3 (2020): 191-239. DOI: 10.1080/07370024.2018.1512413.

^[23] Ward, Blackwell and MacKay, “Dasher”

^[24] Luke Church, and Alan F. Blackwell, "Structured text modification using guided inference," in Proceedings of the 20th annual workshop of the Psychology of Programming Interest Group (PPIG) (2008), 83-94.

^[25] Blackwell, Cocker, et al Live Coding: A User’s Manual.

^[26] Alan F. Blackwell and Samuel Aaron, "Craft Practices of Live Coding Language Design," in Proceedings of the First International Conference on Live Coding (ICLC) (2015). Zenodo. http://doi.org/10.5281/zenodo.19318

^[27] Luke Church, Nick Rothwell, Marc Downie, Scott DeLahunta, and Alan F. Blackwell, "Sketching by Programming in the Choreographic Language Agent," in Proceedings of the Psychology of Programming Interest Group Annual Conference (PPIG) (2012), 163-174.

^[28] Alan F. Blackwell, "The two kinds of artificial intelligence, or how not to confuse objects and subjects," Interdisciplinary Science Reviews 48, no. 1 (2023): 2-11. DOI: 10.1080/03080188.2022.2158258.

^[29] Blackwell, “Wonders without number”

^[30] Blackwell, Damena and Tegegne, “Inventing artificial intelligence in Ethiopia”

^[31] Alan F. Blackwell, "Artificial intelligence and the abstraction of cognitive labour," in Marx200: The significance of Marxism in the 21st century, ed. Mary Davis. (London: Praxis Press, 2019), 59-68.

^[32] Alan F. Blackwell, "What does digital content mean? Umberto Eco and The Open Work," in Critical theory and interaction design, ed. Jeffrey Bardzell, Shaowen Bardzell, and Mark Blythe. (Cambridge, MA: MIT Press, 2018), 167-185.

^[33] Alan F. Blackwell, "Interacting with an inferred world: The challenge of machine learning for humane computer interaction," in Proceedings of Critical Alternatives: The 5th Decennial Aarhus Conference, (2015), 169-180.

^[34] Blackwell, "The Age of ImageNet’s Discovery"

^[35] Alan F. Blackwell, "When systemizers meet empathizers: Universalism and the prosthetic imagination," Interdisciplinary Science Reviews 35, no. 3-4 (2010): 387-403.

^[36] Blackwell, Rode, and Toye, “How do we program the home?”

^[37] Alan F. Blackwell and Matthew Postgate, "Programming culture in the 2nd-generation attention economy," presentation at CHI Workshop on Entertainment media at home - looking at the social aspects. (2006).

^[38] Blackwell, “First steps in programming: A rationale for Attention Investment models”

^[39] Alan F. Blackwell, "What is programming?" in Proceedings of the Psychology of Programming Interest Group (PPIG) (2002), 204-218.

^[40] Alan F. Blackwell and Thomas R.G. Green, "Investment of Attention as an Analytic Approach to Cognitive Dimensions," in Proceedings of the Psychology of Programming Interest Group (PPIG) (1999), 24-35.

^[41] Green and Blackwell, "Ironies of Abstraction"

^[42] Alan F. Blackwell, "See what you need: Helping end-users to build abstractions," Journal of Visual Languages and Computing 12, no. 5 (2001): 475-499.

^[43] Alan F. Blackwell, Lee Wilson, Charles Boulton, and John Knell. Creating value across boundaries: Maximising the return from interdisciplinary innovation. NESTA Research Report CVAB/48, May 2010.

^[44] Alan F. Blackwell, "Design research and academic disciplines," Design Research Quarterly 3, no. 4 (2008): 3-8.

^[45] Alan F. Blackwell, Lee Wilson, Alice Street, Charles Boulton, and John Knell. Radical innovation: crossing knowledge boundaries with interdisciplinary teams. University of Cambridge Computer Laboratory Technical Report UCAM-CL-TR-760, 2009.

^[46] Alan F. Blackwell, "HCI as an Inter-Discipline," in Proceedings of the 33rd Annual ACM CHI Conference on Human Factors in Computing Systems, (2015), 503-516.

^[47] Alan F. Blackwell, "Filling the big hole in HCI research," Interactions 22, no. 6 (2015): 37-41.

^[48] Alan F. Blackwell, Mark Blythe, and Jofish Kaye, "Undisciplined disciples: everything you always wanted to know about ethnomethodology but were afraid to ask Yoda," Personal and Ubiquitous Computing 21 (2017): 571-592.

^[49] Alan F. Blackwell, Claudia M. Eckert, Louis L. Bucciarelli, and Chris F. Earl, "Witnesses to design: a phenomenology of comparative design," Design Issues 25, no. 1 (2009): 36-47.    

^[50] Claudia Eckert, Alan F. Blackwell, Martin Stacey, Christopher Earl, and Luke Church, "Sketching across design domains: Roles and formalities," Artificial Intelligence for Engineering Design, Analysis and Manufacturing 26, no. 3 (2012): 245-266.