Category: ABC

During the vision keynote at Microsoft Build 2018, Satya Nadella introduced the Microsoft 365 platform, a new technology which aims to improve the integration between different Microsoft products and devices (and even other vendors). As great as that is, the underlying motivation is much more interesting and reads like the introduction to a typical paper on activity-centric computing—a line of research originating in the 80s (referred to in my earlier writing as ‘activity-based computing’) which argues for an alternative to the current file-, window-, and application-dominated user interface paradigm.

During a single day you’re using multiple devices, you’re at multiple locations, working with multiple people, and interacting using multiple senses. That is the world we already live in. We need an operating system, we need a platform, that abstracts the hardware at that level, that creates an app model at that level. Single devices will remain important, but this ‘meta orchestration’ is what we need to do. We need to ‘up level’ even our concept of what an operating system is. And that is what Microsoft 365 does. – Satya Nadella, Microsoft Build 2018

This ‘meta orchestration’ is what research prototypes in activity-centric computing typically support by introducing ‘computational activities’ which can be managed by the user or inferred by the system, can be suspended and resumed, can roam across devices, and can be shared with others.

Activity-centric computing addresses deep-rooted information management problems in traditional application-centric computing by providing a unifying computational model for human goal-oriented ‘activity’, cutting across system boundaries. – Upcoming review on activity-centric computing.

Certainly, Microsoft is aware about activity-centric computing and has even contributed to it in the past (e.g., Scalable Fabric and Project Colletta). Even more, Satya starts the keynote by quoting Weiser who first introduced the term ubiquitous computing, a related and overlapping field of research.

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. – Mark Weiser, The Computer for the 21st Century

But, what is really exciting about this announcement is that this is the first time a major software company seems to embrace (and publicly announce) this as the future of computing. In the past, a research group at Apple Computer (including Donald Norman) has toyed with the idea, IBM has a suite of collaboration tools supporting activities, and some features (such as virtual desktops) have been slowly incorporated into most operating systems. But for activity-centric computing to truly succeed it needs to be integrated into the very fabric of computing. In research, most prototypes have largely been inhibited by having to build on top of the very infrastructure they set out to change. As I conclude in my PhD thesis on Task and Interruption Management in Activity-Centric Computing:

Norman and Verganti compare incremental innovation with radical innovation through the use of a hill-climbing analogy: “Incremental innovation attempts to reach the highest point on the current hill. Radical innovation seeks the highest hill”:

… [activity-centric computing] envisions a radically new computing paradigm incompatible with the current antiquated desktop metaphor for office work. Building activity-centric computing systems is a continuous uphill battle against an industry which is only trying to achieve local maxima. In contrast, activity-centric computing pursues radical innovation, aiming for one unifying global maximum.

With the introduction of Microsoft 365, Microsoft is taking its first steps towards a new infrastructure which could potentially host this new computing paradigm and its entailing principles.

The world’s applications going forward need a ubiquitous computing fabric from the cloud to the edge. They need a new app model that is distributed, event-driven, and serverless. – Satya Nadella, Microsoft Build 2018

This is reflected in some of the other announcements, such as live code sharing between Visual Studio and Visual Code (which in research we call ‘activity sharing’), application-less event handlers hosted using Azure functions and adaptive cards which can host content within other applications (decoupling of functionality and applications), but perhaps most clearly by how ‘computational activities’ will be represented in Windows 10 going forward.

Timeline and sets in Windows 10

In the technology keynote on day 2 of the Build conference, Joe Belfiore discussed how the recently introduced ‘timeline’ feature in Windows 10 interplays with Microsoft Graph, a ‘cloud-backed data store’ to store organizational and personal data.

With timeline, the basic idea is that the things that you do on your PC, or other devices, are available in a single click from the taskbar. You can scroll back through time, see everything you were working on, and just click to resume. Now, the key idea here though is that timeline is based on the Microsoft Graph and therefore it enables cross-device experiences [including iPhone browsing history]. – Joe Belfiore, Microsoft Build 2018

This history of the user’s interactions across devices are not in themselves ‘computational activities’. Computational activities require the ability for the user to aggregate disparate resources (managed by separate applications), such as web pages, Office documents, images, etc. This, Microsoft has decided to implement as what they call ‘sets’. In 2012 I discussed why traditional tabbed windows (managed by a single application) are inherently broken, and suggested how an implementation similar to ‘sets’ might lead to more scalable window management. With sets, all work related to a single ‘activity’ can be grouped under a single window using multiple tabs (regardless of the application used). More importantly, this ‘activity’ window can be closed without losing work and retrieved at a later moment in time when ready to resume work (which in research we call ‘activity suspend and resume’). Furthermore, this can even be done on a different device than where the work was first initiated (which in research we call ‘activity roaming’).

In my own research, I implemented similar features on top of Windows 7 in a system called ‘Laevo’, but in addition also looked at how activity planning and the handling of to-do items (traditionally supported by electronic calendars) can be integrated with such ‘computational activities’ or ‘sets’; why duplicate the effort of managing items in a calendar when they are already represented as activities?

 

Most of the time spent implementing such novel new systems is on ‘hacking’ the operating system and applications you intend to support to become ‘activity-aware’, and building a distributed ‘activity model’ which different devices can use to show and resume activities. With Microsoft Graph, Microsoft is building the necessary distributed infrastructure to start building such ‘activity models’ and is encouraging application developers to integrate with it; an essential step towards moving activity-centric computing from ‘the lab’ into production.

It thus seems Microsoft is finally implementing activity-centric computing as it was first envisioned in 1986 by Yoshiro Miyata and Donald A. Norman, targeting radical innovation as opposed to incremental innovation. There is still a long way to go towards incorporating other aspects of the original conceptualization of personal computing (as demoed by Alan Kay in the video below), but this is definitely a step in the right direction!

Tabbed Document Interfaces (TDIs) have become mainstream nowadays. Did you ever stop and wonder why? Which problem has been solved by introducing them? In this post I will shortly take you through their history, explain why I feel tabs don’t solve the fundamental problem, and why Activity-Based Computing (ABC) might be a more appropriate solution.

History

Tabs aren’t as recent as most people think. They gained mainstream success once tabbed browsing was introduced (and no, Opera wasn’t the first web browser to do so), but early environments already supported them. Smalltalk (from Xerox PARC) was a programming language, but its windowed GUI is what eventually inspired windowing environments for personal computers.

Although not exactly tabs as we know them today, I found it important to show what most likely has been the inspiration for them. Since the tabs are always positioned at the top left of the window, you can imagine quickly running out of horizontal screen space if you were to outline them next to each other. In this sense you can argue these aren’t tabs at all, but just plain windows. This changed when UniPress’s Gosling Emacs moved the tab to the right hand side of the window. Since tabs take up a lot less space vertically, outlining them underneath each other doesn’t take up as much space as outlining them horizontally.

Don Hopkins presents having more than 10 windows open under HyperTIES, which is an Emacs based authoring tool. The tabs might not behave as they do today, but they solve a similar problem: allowing to have many documents open simultaneously and easily switching between them.

The problem

Having many documents open and switching between them is the task of a window manager, not specific applications. The definition of a window manager according to dictionary.com is:

A part of a window system which arranges windows on a screen. It is responsible for moving and resizing windows, and other such functions common to all applications.

When an application lets the operating system’s window manager handle all its individual windows separately, it has a single document interface (SDI). This approach quickly resulted in cluttered taskbars, which is why multiple document interfaces (MDIs) and tabbed document interfaces (TDIs) became popular. Limitations of the window manager forced applications to address problems which should have been addressed by window managers in the first place. Consequently, many applications are reinventing the wheel.

Remember that taskbar at the bottom of your screen (usually)? Remember what it used to do before it got all modernized and fancy new features were added? When you’d maximize all of your windows you basically had a tabbed interface with the tabs at the bottom. To exemplify I turned off some of the features in my current Windows 7 setup in order to make it resemble what it looked like in Windows 95, as also visualized in the website I have open.

All TDIs and MDIs are doing is moving the problem to a different location. They hide the clutter, only for it to reappear once the application has to manage many open documents itself. This can be useful when the open documents are related to each other, as is often the case in an integrated development environment (IDE), but this definitely isn’t the case for every application. Ironically web browsers are a prime example of applications which often manage unrelated open ‘documents’.

Imagine a desktop where every application decides for itself where to place the ‘close’ button or how to resize it’s window. The exact same thing is currently going on with document management.

Existing solutions

The problem should be handled by the window manager. Applications don’t need to re-implement the same functionality over and over again. This is in line with a principle in software development called ‘Don’t Repeat Yourself’ (DRY) which I hold in high regard. Additionally the same look and feel can be provided across applications. Some steps have been taken towards this end, of which I’ll discuss a few here.

The windows 7 taskbar allows you to group windows together based on the application they belong to. In the screenshot below explorer windows are grouped together in one icon on the taskbar. Usually you know in which application you have a certain document open, so you can easily find it by performing just one extra step.

The problem I see here is the same as mentioned earlier: this simply moves the problem to a different location, and again, documents open in one application aren’t necessarily related to each other contextually. Additionally, even when the taskbar isn’t full, you still need to go through this extra step. (Note: you can configure to only group windows when the taskbar is full.)

Some modern window managers allow you to group windows together manually.

• KDE’s Plasma Desktop supports tabs, but this is just one possible solution.
• You can group windows together in Compiz in many interesting visually pleasing ways.
• Virtual desktops allow you to group windows related to each other together on separate desktops.
• Scalable Fabric allows grouping windows together and seeing zoomed out versions of groups you aren’t focusing on at the moment on the sides.
• Bumptop (now acquired by Google) allows you to organize files and windows in 3D space.
• … and the list goes on.

Allowing you to manually organize windows at least gives you the flexibility to create your own context as you see fit, but unfortunately again these approaches don’t scale well.

… which of course begs the question, do they have to scale that well?

I prefer turning this question on its head and looking at it from an opportunistic angle.

Scaling window management

What if window management would be more scalable?

First, some food for thought. Yes, more questions …

• What is your main reason for closing a window?
• Why do you organize files?
• What is your main reason for not deleting a file?
• In what way do you organize windows?
• How does file management relate to window management?

A lot of effort goes into setting up the working environment for a particular activity. Wouldn’t it be nice if you were able to ‘store’ this setup and retrieve (or even reuse) it at a later time, just as you are able to store (and reuse) files?

Perhaps the real problem isn’t how we access windows, but how we store them. In order to store and organize these different configurations of windows I proposed using a time line in my thesis.

This is inspired by Activity-Based Computing (ABC) which I wrote about previously, which states that activity should be the central computational unit instead of files and applications. In the context of window management activity can be seen as all the windows needed to achieve a certain goal. It’s up to you to decide what makes up a relevant context, organizing work as you like.

Taking this to the extreme: assume closing a window would be the same as deleting a file. Would you actually ever have to know about the underlying file system again? Window management and file management could become one and the same thing.

As I’ve mentioned before, I’ve been quite busy, hence the lack of posts over the past few months. A big portion of my time went into developing a prototype user interface for my thesis titled “Managing Activities in Time and Space in an Activity-Based Computing System”. No worries if you don’t have a clue what that means, as I’ll try to do my best to explain the concept in at least 80 pages less than my thesis. In future posts I will disclose the actual prototype and will work towards a stable beta release. Until then – for the curious – you can already access the open source project, but I still need to add instructions on how to compile it in the readme.

UPDATE: Since writing this post, I’ve released an installer for the Alpha version of this prototype, called Laevo.

It came to me as a surprise, that the very same idea I had for my thesis was already being discussed in the 80s, in the early days of Human-Computer Interaction (HCI) over at Xerox PARC. It came to me as a bigger surprise that even though several research departments have been researching the subject over a course of more than 30 years (including a research group over at Apple Computer), the concept still hasn’t caught on in mainstream computing. Activity-Based Computing (ABC) is a new computing paradigm which attempts to address some of the problems with the current paradigm.

Applications: what a terrible term. What a terrible concept. Applications have little to do with the tasks that people are attempting to accomplish. Look. We don’t do word processing; we write letters, or memos, or reports, or notes to ourselves. – Donald A. Norman

Computers are inherently application and document oriented. This is all good and well for simple tasks, such as using a text editor to write a certain document, but is that still a realistic scenario? Computers nowadays make up a big portion of our daily lives. We don’t just write a paper; we look up information on the internet, translate words, use a dictionary, email with contacts for further information, import images, keep track of the deadline when the work needs to be finished, share a draft version, wait for feedback, and later have to reopen our entire working context again to make amendments. A lot of tasks users need to carry out require the use of several applications and documents, simultaneously. Furthermore, we don’t work on just one task at a time. Interruptions can cause us to suspend work on one task and open up other documents/applications required to continue work on another.

Gaaah! I believe this portrays the problem pretty well. Most likely your desktop never looks this bad, but that’s because you tend to close windows you believe you no longer need, exactly to avoid this kind of situation. But at what cost? Think back for a second how often the following thought went through your head: “If only I hadn’t closed that before.”. We lose a lot of time re-initializing a working context which we already set up before, be it one hour, one day, or a month ago. But that’s not all. Computing has become ubiquitous. We don’t just use one device, but several. We don’t keep documents to ourselves either, we share them. Managing applications and documents has become a task in it’s own right; a repetitive, distracting, possibly unnecessary task. I hate unnecessary repetition.

Proponents of Activity-Based Computing feel a lot of this overhead can be eliminated by using activity as the basic computational unit, instead of the file or application. Activity, as defined in activity theory, is seen as “a system of human “doing” whereby a subject works on an object in order to obtain a desired outcome.” This takes into account the entire setting in which the activity takes place; the tools used, the people you communicate and share resources with, the rules governing the community and how work on the activity is divided between people. Activity being a computational unit implies you can open, close, save and manage activities, not too dissimilar as to how you use files now. “The end-user is directly supported by computational activities which can be initiated, suspended, stored, resumed, and shared on any computing device in the infrastructure at any point in time, handed over to other persons, or shared among several persons.”

Humans perform plenty of activities, and thus within existing ABC systems the question arised how to manage them. The concept I explored in my thesis was to provide an overview of activities in time and space, showing when work was done on them and their inner relationships. I argue it is a worthwhile path to explore using a calendar not only to plan your activities, but also to use them. My prototype works towards this integration of activities as a computational unit managed on a calendar. Since the start of this month I started work as a research assistant at the IT University of Copenhagen where I will be researching how ABC can be used by software developers, specifically in software companies with distributed teams, following agile software development. Exciting times are ahead!