Last week I presented my paper Design Methods to Investigate User Experiences of Artificial Intelligence at the 2018 AAAI Spring Symposium on the User Experiences of Artificial Intelligence. The picture above was taken by a Google Clips camera, which was also one of the presented papers. Below are my slides, which contain supplementary images to my paper on the three design methods I have engaged with relative to the UX of AI. Another blog post will soon follow with reflections on other papers presented.
Design Methods to Investigate User Experiences of Artificial Intelligence
Central to three design methods I’ll be speaking about is the following question: How might Artificial Intelligence support a more meaningful choreography between explicit and implicit interactions? And this question is motivated by a growing interest in technologies that preempt our needs – that are contextual, anticipatory, and personalized – which has catalyzed a shift from explicit to implicit interactions, the latter being systems in which actions are not actively guided or chosen by users but instead come from inference driven activity.
Within this notion of a choreography we draw upon Ju and Leifer’s characterization of explicit and implicit interactions – explicit interactions being those that demand our immediate attention for direct engagement and manipulation, and implicit interactions as those that rely on peripheral information to seamlessly behave in the background until appropriately shifted into attention. And though such choreographies that are supported by AI obviously do exist, many of the resulting user experiences are far from seamless, sure, and secure.
Thus, to investigate what a more meaningful choreography might be, or the UX of AI, we’ve been deploying design methods to understand and conceptualize implicit systems driven by AI to both question their developing form and also investigate how both UX designers and end users might participate in the creation of these choreographies. Our design-led approach falls under Research through Design, in which both our design process and the artifacts themselves (whether objects, systems, or interactions) are ways to acquire new knowledge.
Our research goals can be broken down in three potential contributions:
• Surveying and challenging existing user interactions with ubiquitous and smart technology to expose design opportunities.
• Understanding Machine Learning as an actual limited part of systems that can be approached and shaped by designers and users.
• Unpacking the social implications of implicit interactions across information, interfaces, and infrastructures.
And for each of these in the remainder of my presentation, I’ll detail the progress of three design research methods that correspond to each of the potential contributions, and draw attention to initial reflections as they are still very much in progress.
1. A Design Workbook on Implicit Interaction
The first method has been the creation of a Design Workbook on Implicit Interaction to collaboratively unpack definitions and implications of implicit interaction which exploring opportunities for intelligent system action. A design workbook, as defined by Gaver, is a collection of design concepts, proposals, and related material that creates a design space in which participants can engage with or expand upon design ideas, issues, and investigations. While design workbooks can be beneficial for designers working alone or in design teams, its recognition that complex design emerge slowly often through the synthesis of tacit relationships between an array of concepts, affords its position as a boundary object for multidisciplinary teams and in particular communicating the intellectual rigor of design. As our project work is comprised of multiple academic disciplines, our workbook serves as a design space in which intentions, objects and aspirations can be communicated and aligned. Ultimately attempting to dispel both black boxes – that of ML and of design.
At the time of writing this paper, our workbook was composed of five sections:
Implicit: Meanings, Definitions, Terms – a collection of words from meetings, workshops, and emails that have been used to describe implicit interaction. This section has been particularly important in challenging disciplinary assumptions and mental models.
Examples: Interactions, Services, Systems – a visual collection of projects that both inspire and provoke, but more importantly affording concrete examples to reference during project activities.
Domains: Situations, Contexts, Opportunities – another visual collection, yet of problem spaces, complex challenges and interesting areas that prompt ideation and foreground an alignment in meaningful real-world applications.
Technology: Data, Activations, Inferences – a list of existing and aspirational data streams and sources that has been a key starting point in latter engagements with data as a design material.
Concepts: Projects, Abstracts, Briefs – a working portfolio of completed and potential projects from speculative academic abstracts to utilitarian ideas to disturbing provocations.
One of these provocations is the designing and prototyping pee-ometer, or a proposed wearable devices that predicts when a user has to pee based on fine bodily movements. While we are not necessarily advocating that there should be pee-ometers, the possibility of such a device, working on a brief tied to an intimate and taboo topic such that of human waste, simultaneously reveals and exasperates social tensions, relational frictions and interactional loops with smart technology – while inviting those working on it to extend technical practices such as training, into the design space. And I want to highlight two things about this slide – the first being the slightly ambiguous image of an Arduino connected to a toilet paper roll because we have started working on the project both as an object of critique and genuine inquiry.
And a the most recent state of the workbook, which now focuses on the original fifth section – a collection of fictional and real paper abstracts (failed and successful!), and speculative and in-progress project briefs. These are arranged in chronological order for both a reflective account of our progression, but to also not suggest clustering or relationships between often disparate ideas, because the workbook can also be used as ideation cards in which new relationships can emerge.
2. A Workshop on Designing with Data
As from the perspective of AI – or again more specifically for us, a Machine Learning model – data is an absolute requirement, for nothing can be learned if no learning material is available – thus, our second approach has been to investigate through design workshops what diverse data sources might mean and how they can be used to think about implicit system action.
A growing body of research in the HCI Design research community has been investigating data as a design material, that is, a material that is approachable and shapeable by designers and possibly end users – from tangible interactions, across scale, and into the built environment.
Within our own work on implicit systems, data as a design material might be more specifically expressed as something that enables system action without that action being necessary well defined. Our design process began with self-data collection in which screenshots – a data source idea that stemmed from the tech section in our original workbook – from the authors’ computers were taken every minute over a six-week period of time. While we also wrote a program that utilized Google’s image recognition API to convert these screenshots to text, we decided in parallel to inquire into the conceptual properties and arrangements of the gathered data. Here we drew upon material experiences from Giaccardi and Karana to investigate the practices, or situated ways of doing, between people and data. And in our first workshop, we used a simple script to randomly generate five booklets of data from my own screenshot database – each booklet consisting of 20 screenshots from varying time intervals, and each booklet indiscriminately yet blindly shaped by myself as the data owner, designer, and workshop leader – in which a series of predetermined filters, distortions, and zoom lenses were applied. At the beginning of the workshop, the data booklets were handed out to each participant to familiarize with, and then prompt cards were given to extract and map inferences and reflections from the data (encompassing themes of ownership, contexts, emotions, aspirations, and ecosystems).
The workshop concluded with a speculative exercise in which participants were asked to imagine how different actors, from specific colleagues and technologies to more general personas and services, might misuse mapped inferences. Structured to design disruptions, the concluding step situated the experienced properties and performances of the data in external and consequential contexts.
Here are workshop participants working with the data, and a later analysis a week later during which we presented and reviewed themes.
Prior to the workshop, our application ideas and directions for the captured data centered on actions such as recommendations that could be based on text extracted from the screenshots. But through the materialization of data, we were able to open a design space regarding how this data could be used to present more complex representations of users in new and different ways, surfacing four primary themes:
Activity and inactivity hierarchies: Instead of imagining a screenshot as a two-dimensional visual space in which text can be scrapped to infer activity, extending the desktop metaphor into 3-dimensions to investigate layered windows of activity
Behavioral adjustments in response to data tracking: Both altering online activity and physical device arrangements to maintain, yet alter, the flow of data
Social traces of data sharing: ‘Invisible’ practices such as copying to the clipboard and more obvious indications of data sharing such as highlighting text or remnants of a copy/paste in email threads
Pacing of online rhythms and routines: Evidence of certain application usage revealed professional identity and aspirations of the data owner, such as submitting to CHI, and professional assumptions of the data reviewer, such as Photoshop a sign of procrastination
Thus, investigating data as a material to understand the strange and perhaps even hidden aspects of online and computer based activity has enabled us to reimagine new possibilities of how intelligent systems might approach data through activities centered on how humans make sense of data.
Drawing upon the themes of activity hierarchies and data sharing, the first of two very early concepts that emerged from this workshop is a shared activity printer and copy-past indicator. Envisioned to be situated within an open office space, each worker would have the device pictured attached to their computer. The screen refreshes when anyone in the office copies data from one program to another, not revealing what was copied, but between which programs and the size of the data based on the amount of characters. This concept plays upon professional assumptions and potential speculations that the information might ignite. In addition, the yellow thread of paper prints whenever the computer own has been actively using the same computer program as another co-worker, rethinking what it means to work together.
A second concept explores behavioral adjustments in response to data tracking. Instead of stopping the flow of data to obscure or alter resulting information, how might selected data sets by magnified? Prompting questions regarding the creation of a private space by emphasizing something else.
3. A Prototype on Asymmetrical Interactions
In an effort to unpack the social implications of implicit interactions, our third approach has been the prototyping of a simple informational infrastructure, or a custom Internet of Things application, to understand and design counter-strategies for asymmetrical interactions of data-driven systems in use.
Prototyping is an established, interdisciplinary method for multiple purposes including, but not limited to, understanding an intended experience. While prototypes can also reveal potential implications of proposed products, services and systems, it is less clear how designers might engage with the underlying informational infrastructures of data-driven devices and applications, such as those support by AI, to not only expose but also transform their functioning.
This engagement by designers to materialize, or open up an infrastructure for either design or user intervention, is of particular interest to our work regarding conflicts of agency and concerns for privacy. Therefore, in addition to design explorations into new application areas, an ongoing prototype in which we are investigating the materialization of an everyday data-driven infrastructures is the autobiographical design probe Leaky Objects.
Leaky Objects is a project that was prompted by my own observed change in communication patterns between my partner and myself upon our relocation from London to Stockholm for my PhD position, which resulted in him initially unemployed in Stockholm with a contrasting flexibility in environment and routines to my consistent work schedule. This variability shifted the nature of my daily thoughts about him from specific inquiries into known events to general curiosities about possible happenings and overall well-being. For example, in London, I would inquire about a deadline status, presentation result or colleague interaction. Whereas initially in Stockholm, I would wonder if he was awake, at home or around town. The former often elicited a lengthy dialogue, while the latter only necessitates a brief response. With this change in communication content and flow, i began to wonder how might our method of communication correspondingly be adapted? Instead of directly communicating with each other, could we indirectly communicate with objects about each other?
So in our small studio, we body stormed in the space what the placement of simple sensors might suggest or imply that he (or myself) is doing at home – intentionally designing for ambiguity and speculation.
Following the deployment of simple sensors, I set up a custom web application in which the status of these sensors could be requested from an Arduino. As this inevitable created a very obvious power imbalance, or asymmetry in interaction, I incorporated a mechanism to reveal when sensor information is accessed. For example as one sensor is a photocell attached to a floor lamp that checks the status of the light, a custom power-switch was appropriated into an awareness indicator, causing the light to flicker when its status is remotely requested.
While the prototype introduced the concept of leaky objects, a playful reimagining of the computer science notion leaky abstraction, to describe the phenomenon in which shared objects leak implicit information that results in unintentional communication, it additionally surfaces the potential for further investigations into strategies of obfuscation as the inherently unfinished and messy nature of a prototype creates an opening for the design of further interactions, appropriations, and hacking.
While we have used the prototyping of a simple informational infrastructure as a design method to investigate the potential social implications of implicit interactions in data-driven systems, we also hope to engage interaction designers in discussions on potential strategies of approaching the complex challenges of asymmetry in concerns of agency and privacy. As we continue to engage with more complex and layered data streams that afford Artificial Intelligence and Machine Learning techniques to support implicit interactions, we plan to continue an increased engagement in prototyping as a method for the design of meaningful and responsible user and system interactions.
In summary, in the first method, we have explored the creation of a design workbook to map varied conceptual approaches and definitions of implicit interaction. In the second method, a workshop on designing with data was employed to explore and understand how data can be used in novel ways. Lastly, in the third method we have developed a simple system that rethinks a technical notion (“leaky abstractions”) to explore new types of system behavior.
