CHAPTER

•· The wheel is an extension of the foot, the book is an extension

of the eye, clothing, an ex tension of the skin, electric circuitry '' an extens ion of the centra l nervous system.

CHAPTER OUTLINE 10.1 Introduction

10.2 Keyboards and Keypads

10.3 Pointing Devices

10.4 Displays

Marshall McLuhan and Quentin Fiore

The Medium Is the Message, 1967

337

338 Chapter 10 Devices

10.1 Introduction

Input and output devices represent the physical medium through which users oper­ ate computers. Along with improvements in computer processor speeds and storage capabilities in the past 50 years, their physical form factor and basic functionality have also changed dramatically. Only two decades ago, the standard computer plat­ form was the desktop or laptop personal computer equipped with a screen, a mouse, and a keyboard, but mobile devices have revolutionized the face of computing to the point that many people do not realize that their ever-present smartphones, tablets, or portable MP3 players are, indeed, powerful computers. Computing has reached a point where it is deeply woven into the very fabric of our everyday existence (Dourish and Bell, 2011). With easily more than 5 billion mobile devices in existence, more than 25% of them "smart" and able to access the internet, compared to some 800 million personal computers, it is clear that mobile computing is the universal computing platform for the world (Baudisch and Holz, 2010). What's more, unlike the previous generation of personal computers, this new pervasiveness of mobile computing is no longer restricted to industrialized parts of the world but is quickly becoming an integral arld iI1tegrated aspec t of life even in rural, poor, and underde­ ve loped regions (Pew Research Center, 2014). In fact, many regional efforts, such as Data Wind's $35 Aakash tablet designed for the Indian market (Fig. 10.1) as well as the Uhuru Tab by Rig Communications Limited in Ghana, are making sigruficant inroads toward making advanced computing available to everyone.

The veritable explosion of new and exciting computing technology has increased the importa nce of interaction design so as to accommodate such a wide diversity of input and output modalities. To keep up with this rapid pace of change, successful designers are increasingly employiI1g micro-HCI and macro­ HCI theories to transcend the specific capabilities and characteristics of individual devices. Some of these theories involve micro -scale ideas on consistency, respon ­ siveness, discoverability, hierarchies, mformation archi tecture, and feedback as well as macro-scale ones on contex t, social setting s, emo tions, learning, and per­ sonalization. Refer to Chapter 3 for a discussion on micro-HC I and macro-HCI.

Despite increased complexity, devices also represent the part of computer tech­ nology that perhaps has the largest capacity for game-changmg irmovation for user interfaces. Indeed, the hype of such innovations often concerns the user-interface aspects of a new device. The Apple iPhone and iPad changed smartphones and tab­ let computing overnight when they were introduced, mainly on account of their

See also:

Chapter 7, Direct M anipulat ion and lmm ersive Environments

Chapter 8, Fluid Navigation

~ourc 1enl

FIGURE 10.1 Indian IT minister Kapil Sibal announcing the Aakash, a $35 tablet for the Indian market, in 2011.

10.1 Introduction 339

smooth and natural user experi­ ence. The Nintendo Wiimote and the Xbox Kinect introduced ges­ tural and full -body interaction, respecti vely, to the living rooms of millions of people around the world. Finally, the Oculus Rift and the Microsoft HoloLens are bring­ ing vir tual and augmented reality to life for everyone (see Chapter 7).

Given such diversity and scope, this chap ter gives a brief introduc­ tion to the most important fami­ lies of input and output devices. The chapter first reviews text

en try (Section 10.2), including keyboards and keypads as well as their layout, physical design, and accessibility adaptations. It also discusses text entry tech­ niques for mobile de vices . Pointing (Section 10.3) is another common interac ­ tion for interfaces, and the section below reviews ideas for ho,-v to make pointing mor e efficien t, more accurate, and more accessible. Section 10.4 presents both traditi onal as well as novel display technologies, focusing on particularities of large and small disp lays as well as wearable computing

FIGURE 10.2 Many baby monitors use video cameras to provide a real -time feed of the baby's activities on a remote display device. There are also some wearable baby monitors that incorporate advanced features such as a baby's heart rate, respiration patterns, and skin temperature, which parents can track using their smartphone.

340 Chapter 10 Devices

and shape -chan ging disp lays. Examples of possible solutions for users with disabilities are distributed throughout the chapter.

1 0.2 Keyboards and Keypads

Text entry is one of the most common input tasks, and the primary mode of text entry is still the keyboard (Fig. 10.3). Despite having received much criticism over the years, the keyboard is very successful and still represents the most effi­ cient text-entry mechanism. Billions of people use keyboards; although the rate for beginners is genera lly less than one keystroke per second and the rate for average office workers is five keystrokes per second (approximately 50 words per minute), some users achieve speeds of up to 15 keystrokes per second (approximately 150 words per minute). Contemporary keyboards generally per­ mit only one keypress at a time, although dual keypresses are used for capita ls (Shift plus a letter) and special functions (Ctrl or Alt plus a letter).

More rapid data ent ry can be accomplished if several keys can be pressed simul taneously (i.e., chording). An inspiration might be the piano keyboard, an impressi ve data-entry device that permit s severa l finger presses at once and is responsive to different pressures and durations. Similarly, chord keyboards use multiple keypresses that represent several characters or entire words. While this requires trainiI1g ai1d continued use, such chorded keyboards can allow for text entry outside the sta11dard office environment, such as one-han ded or eyes-free typing on small mobile devices or for wearable computing. In the courtroom,

•

-·

I

FIGURE 10.3 An Apple MacBook Air laptop with a QWERTY keyboard (left) showing the inverted T movement keys at the bottom right and function keys across the top. A multi-touch trackpad supports pointing. On the right, a detail photograph of a Lenovo laptop keyboard shows a pointing stick (also called a trackpoint) mounted between the G and H keys on the keyboard.

10.2 Keyboards and Keypads 341

such devices are called stenotypes and allow court reporters to rapidly enter the full text of spoken arguments at rates of up to 300 words per minute. Hoovever, this feat requires months of training and frequent use to retain the complex pat­ terns of chord presses.

10.2.1 Keyboard layouts The Smithsonian Institution's National Museum of American History in Washington , DC, has a remarkab le exhibit on the development of the type­ writ er. During the middle of the nineteenth century, hundreds of attemp ts were made to build typewriters, with a stunning ,,ariety of positions for the paper, mechanisms for producing characters, and layouts for the keys. By the 1870s, Christopher Latham Sholes's design was becoming dominant – it had a good mechanical design and a clever placement of the letters that slowed down the users enough that key jamming was infrequent. This so-called QWERTY layout puts frequently used letter pairs far apart, thereby increasing finger travel distances.

Sholes's success led to such widespread standardization that, more than a century later, almost all keyboards use the QWERTY layout or one of its varia­ tions deve loped for other languages. The development of electronic keyboards eliminated the mechanica l prob lems of typewriters and led many twentieth­ century inventors to propose alternative layouts to reduce finger travel dis­ tances. The Dvorak layout could increase the typing rate of expert typists from about 150 words per minute to more than 200 words per minute and even reduce errors. Its failure to gain acceptance is an interesting examp le of how even docu ­ mented improvements can be impossible to disseminate because the perceived benefit of change does not outwe igh the effort required to learn a new, nonstan­ dard interface.

A third keyboard layout of some interest is tl1e ABCDE style, which has the 26 letters of the English alphabet laid out in alphabetical order. The rationale here is that non-typists will find it easier to locate the keys. A few data-entry terminals for numeric and alphabetic codes still use this style, though studies have shown no advan tage for the ABCDE style; users with little QWERTY expe­ rience are eager to acquire this expertise and often resent having to use the ABCDE layout.

Number pads are a further sour ce of controversy. Telephones have the 1- 2- 3 keys on the top row, but calcula tors place the 7-8- 9 keys on the top row. Studies have shown a slight advantage for the telephone layout, but most computer keyboards use the calculator layout.

Some researchers have recognized tha t the wrist and hand placement required for standard keyboards is awkward and have proposed more ergo­ nomic keyboards. Various geometries have been tried with split and tilted keyboards, but empirical verification of benefits in typing speed, accuracy, or reduced repetitive stra in injury is elusive.

342 Chapter 10 Devices

10.2.2 Accessible text entry While people with motor impairments often can still use regular keyboards, albeit very slowly, several approaches to aid such users exist. Early solu tions were based on large menus of fixed choices, but methods currently used in practice include adaptive keyboards, where keys are lowered instead of raised to aid acquisition, as well as on-screen keyboards accessed using alternative input devices like head pointers or oversized trackbal ls. All such text-entry methods can be improved sig­ nificantly by incorporating dictionary-based auto-completion as well as automatic error correction (Kane et al., 2008). In contrast, visua lly impaired users represent a particular challenge for text entry. Perklnput (Azenkot et al., 2012) and BrailleTouch (Southern et al., 2012) both provide nonvisual input methods for one-handed or two-handed Braille typing on multi-touch smartphone displays.

Some technique s go beyond the traditional keyboard. Dasher predicts proba­ ble characters and words as users make their selections in a continuous 2-D stream of choices and has been adapted to brain-computer interfaces (BCI), where people use their brain alone to input text (Wills and MacKay, 2006). Also, orbiTouch's Keyless Keyboard replaces the keys with two inver ted bowls, on top of which the user's hands rest comfortably (Fig. 10.4). A combination of small hand movements and small finger presses on the two bowls selects letters or con­ trols the cursor. No finger or wrist movement is needed, which might be I-1elpful to users with carpal tunnel syndrome or arthritis.

Finally, yet another approach reconsiders the use of a keyboard entirely. One idea is to rely on pointing devices such as mice, touchpads, or eye-trackers for data entry. Another builds on wearab le devices, such as a wris tband or ring form factor, to e11ter text (Ye et al., 2014). Common among many accessible text­ entry methods, particular ly for mobile settings, is the increasing use of speech input for this purpose (Chapter 9).

FIGURE 10.4

orbiTouch Keyless Keyboard with integrated mouse functionality (http://orbi tou ch .org /). The orbiTouch requires sma ll finger presses and no actual hand motion to operate yet supports high-performance typing and point ing.

10.2 Keyboards and Keypads 343

10.2.3 Keys Keyboards keys have been refined carefully and tested thorot1ghJy in research lab­ oratories and the marketplace. The keys tend to have slightly concave surfaces for good contact with fingertips and a matte finish to reduce both reflective glare and the chance of finger slips. Keypresses require a 40- to 125-gram force and a dis­ placement of 1 to 4 millimeters, which enables rapid typing with low error rates while providing sujtable feedback to users. An important element in key design is the profile of force displacement. When the key has been depressed far enough to send a signal, the key gives way and emits a very light click. llis tactile and audi­ ble feedback is extremely important in touch typing; hence, membrane keyboards that use a nonmoving surface are difficult to use for extensive touch typing. How­ ever, such keyboards are durable and therefore acceptable for challenging environ­ ments such as fast-food restaurants, factory floors, or amusement parks.

Certain keys, such as the space bar, Enter key, Shift key, or Ctrl key, should be larger than others to allow easy, reliable access. Other keys, such as Caps Lock and Num Lock, should have a clear indication of their state, such as by physical locking in a lowered position or by an embedded light. Large-print keyboards are available for vision-impaired users. The placement of the cursor-movement keys (up, down, left, and right) is important in facilitating rapid and error-free use. The popular and compact inverted-T arrangement of arrow keys (Fig. 10.3) allows users to place their middle three fingers in a way that reduces hand and finger movement. The cross arrangement is a good choice for novice users. Some large keyboards reuse the peripheral eight keys on the numerical keypad (all keys except the central 5 key) to simplify diagonal movements . For such key­ boards, the Num Lock key is used to toggle between keypad and arrow mode. In some applications, such as games, where users spend hours using the move­ ment keys, designers reassign letter keys as cursor-movement keys to minimize finger motion between the movement keys and other action keys. The WASD keys are often used for this pt1rpose . Finally, the au to-repeat feature, where rep­ etition occurs automatically with continued depression, may improve perfor­ mance, but control of the repetition rate must be provided to accommodate user preferences (thi s is particularly impo rtan t for very young users, older adult users, and users with motor impairm ents).

10.2.4 Mobile text entry As computers morph into new form factors – such as tables , tablet s, and phones-as well as become universally usable for a broader population of users from different backgrounds, nationalities, and capabilities, text entry is also changing beyond the traditional keyboard. Most older or low-cost mobile devices provide only a numeric keypad. Entering text using keypads requires multiple taps, where users hit a number key multiple times to cycle through

344 Chapter 10 Devices

several letters assigned to that key. Using the same key for consecutive letters requires the user to pause between letters. Predictive techniques, such as T9® by Tegic Communications, use dictionary-based disambiguation to speed up text entry and are often preferred for writing longer texts. Similarly, LetterWise uses the probabilities of prefixes and facilitate s the entry of non-dictionary words, such as a proper nouns, abbreviations, or slang. After training, users were able to type 20 words per minute with LetterWise compared with 15 words per min­ ute wi th multi-tap (MacKenzie et al., 2001).

While the current generation of sma rtphones, heralded by the initial release of the Apple iPhone (Fig. 10.6) in 2007, tends to eschew physical keyboards in favor of soft keyboards, some still use a traditional QWERTY keyboard. Fig. 10.5 shows two such mobile devices. Physical keyboards are still preferred by many mobile users who need to enter large amoU11ts of text usiI1g tl1eir phones, such as to manage e-mail on the go . With practice, users can reach speeds of 60 words per minute when using both thumbs with those mechanical keyboards or more when the device auto-corrects "off-by-one" erro rs where the user accidentally presses a key adjacent to the one intended (Clawson et al., 2008).

Nevertheless, the proliferation of touchscreen technology means that physi ­ cal mobile keyboards are increasingly being replaced by virtual, or so-called "sof t," keyboards, where the keyboard is merely a visual representation on the tou chscree n (Dunlop and Masters, 2008). Projection keyboards, where the physical world is appropriated (Harrison, 2010) to disp lay an image of the key­ board, are based on the same principle. The benefit of soft keyboards is that they can be dynamically relabeled, such as for a new charac ter set or layout, as well as rescaled and rotat ed to fit the physical display and device orientation

~ ··~~– – – · . • ! ! • . I ' ~ 'm ID m I

#

.f " 8 -.. • • l tj • 12) fJ lfl ID 0

, ' ~

+I I •• * 5 " ~ st "

, ..• , 2 3 4 5 • 7 ' a · 9 0 a., 0 ' IV . E • T y " u 0 p • ., A . 5 D F c; H J K " L , … • ~ .. • – z •. X" C · V , -~· 8 – N • .. ~ •

FIGURE 10.5 A Blackberry 010 (http://www.b lackberry.com) shown here on the left with a small physical QWERTY keyboard; users typically type with one finger or with both thumbs. On the right, a larger keyboard uses the longer dimension of a LG Cosmos 2 device and can be sl id back into the device when not needed ( http://www. LG .com/).

10.2 Keyboards and Keypads 345

oogsitter'?

– tter? P.9.g§t …… .

Hey!

Are you available

t

q w e r

F GURE 10.6 Soft keyboard on an Apple iPhone with the Shift technique (Voge l and Baudisch, 2007). Shift displaces a pressed or hovered key to a position above the user's finger to reduce t he so-called "fat finger" prob lem, whe re the finger occludes the touch target. The iPhone keyboard also increases precision by allowing repositioning and then activates on lift-off .

(Fig. 10.6). However, because soft keyboards lack the tangible and tactile feedback of a physical keyboard, they are difficult to use for eyes-free operation and typi­ cally yield only modest performance, around 20 to 30 words per minute. Still, one study demonstrated that providing tactile feedback using the phone's vibration motor could improve typing speed (Hoggan et aL, 2008), and another study found that expert typists can reach an average of 59 wpm and 90% key accuracy withou t even seeing a visual representation of the keyboard (Findlater et al., 2011).

Several methods exist to improve text entry on touchscreens. Just like keypad ­ based text-entry methods can make use of dictionary-based or predictive text-entry algorithms, current touchscreen text-entry methods commonly sugg est possible word completions given an input string. More advanced techniques use language models to predict the word the user is trying to write based on the current sentence; such language models are incorporated in new Apple iOS and Android mobile operating systems. Similar ly, Swype and ShapeWriter (Zhai and Kristensson, 2003)

346 Chapter 10 Devices

enable typing by tracing letters using a single touch gesture v.rithout the need to lift the user's finger, resolving conflicts using a language model. Finally, while origi­ nally designed for pen-based interfaces, the Shift technique has been adopted to mitigate the "fat finger" problem in text entry, for example, by the Apple iPhone, where the user's own finger occludes the key being pressed (Fig. 10.6).

Another text entry method is simply to write by hand on a touch-sensitive surface, typically with a stylus, but character recognition remains error-prone. Contextual clues and stroke speed plus direction can enhance recognition rates, but successful gestural data-entry methods are based on simplified and more eas­ ily recognizable character sets, such as the unistrokes used by Graffiti® for Palm OS devices. Another promising method is to allow shorthand gesturing on a key­ board instead of tapping on a touchscreen keyboard, using shapes that match the tapping patterns. Lor1g-term studies confirm that it is possible to achieve good text-entry performance with this technique (Kristensson and Denby, 2009).

For some languages, such as Japanese or Chinese, handwriting recognition has the potential to dramatically increase the number of potential users. On the other hand, users with disabilities, older adults, and young children may not have the necessary fine -motor control to use such interfaces on tiny touch -sensi ­ tive surfaces. For them, innovations such as EdgeWrite (Wobbrock et al., 2003) might be helpful. EdgeWrite relies on the use of a physical border to frame the drawing area and uses a modified character set that can be recognized by iden­ tifying the series of corners being hit instead of the pattern of the pen stroke, resulting in higher accuracy for all users compared with Graffiti. The Edge Write character set has also been used successfully with trackballs or eye-trackers to address the needs of users with disabilities (Wobbrock et al., 2008).

Finally, the pro liferation of so-called smartwatches (Fig. 10.23) has led to new research efforts on making text entry practica l even on such devices. Of course, the challenge with smartwatches is that their effective display and input area is approximately on the order of 1 inch (about 2 to 3 on). Zoom Board (Oney et al., 2013) is one approach that uses iterative zooming to make impossibly tiny keys usable on a small display; other methods exist or are likely forthcoming.

10.3 Pointing Devices

The new generation of touch displays invites users to tap, drag, and pinch the images on the screen directly. Furthermore, with complex information displays such as those found in computer -assisted design tools, drawing tools, or air­ traffic-contro l systems, it is often convenient for the user to point at and select items. This direct-manipulation approach (see Chapter 7) is attractive because the users can avoid having to learn commands, reduce the chance of typographic errors on a keyboard, and keep their attention on the display. The resu lts are

10.3 Pointing Dev ices 347

often faster performance, fewer errors, easier learning, and higher satisfaction. Pointing devices are also important for small devices and large wall displays that make keyboard interaction less practical.

The diversity of tasks, the variety of devices, and the strategies for using them create a rich design space (Hinckley and Wigdor, 2011). There are many ways to categorize pointing devices, such as physical device attributes (rotation or linear movement), number degrees of freedom (horizontal, vertical, yaw, pitch, etc.), and positioning (relative or absolute). The description below focuses on tasks and degree of directness as organizing dimensions.

10.3. 1 Pointing tasks and control modes Pointing devices are useful for seven types of interaction tasks:

l. Select. Choosing from a set of items. This technique is used for traditional menu selection, tl1e identification of objects of interest, or marking an object in a slide deck.

2. Position. Choosing a point in a one-, two-, three -, or h ighe r-dimensiona l space. Positioning may be used to place shapes in a drawing, to place a new window, or to relocate a block of text in a figure.

3. Orient. Choose a direction in a two-, three-, or higher-dimensional space. The direction may rotate a symbol on the screen, indica te a direction of motion, or control the operation of a device, such as a robot arm.

4. Path. Define a series of positioning and orientation operations. The path may be realized as a curving line in a drawing program, a character to be recognized, or the instructions for a cloili-cutting or other type of machine.

5. Quantify. Specify a numeric value. The quantify task is usually a one­ dimensional selection of integer or real values to set parameters, such as the page number in a docwnent, the velocity of a vehicle, or ilie music playback volume.

6. Gesture. Perform an action by executing a pre-defined motion. Examples of gestures include dwel ling on an object to bring up a context menu, swiping to the left (or right) to turn a page forward (or backward), and pinching (or separating) your fingers to zoom out (or in).

7. Text. Enter, move, and edit text iI12-D space. The pointing device indicates ilie location of an insertion, deletion, or change; see Section 10.2 for details on text­ entry devices. More elaborate text tasks include centering, setting margins and font sizes, highlighting (boldface or underscore), and page layout.

vlhile all of iliese tasks can be performed using a keyboard by specifying directions, coordinates, and distances using a command language, this is an indirect and inefficient method that requires training. In ilie past, the keyboard '"'as used for au of these purposes, but now most users employ pointing devices to perform the tasks more rapidly and w iili fewer errors; expert users can

348 Chapter 10 Devi ces

furthe r impr ove performance by using keyboard short cuts for tasks that are invoked frequently (e.g., Ctrl-C followed by Ctrl-V to copy and pas te).

Pointing devices can be grouped into those that offer direct control on the screen surface, such as the touchscreen or sty lus, and those that offer indirect control away from the screen surf ace, such as the mou se, trackball, joys tick, graphics tablet, or touchpad. Within each category are many variations, and nove l designs emerge frequently (Box 10.1).

DOX 10.1 Pointing devices.

Direct control devices (easy to learn and use, but hand may obscure display)

• Touchscreen (sing le- and mul t i-touch)

• Stylus (passive and acti ve )

Indirect control devices (take time to learn)

• Mouse

• Trackball

• Joystick

• Point ing stick (trackpoint )

• Touchpad

• Graphics tablet

Novel devices and strategies (for spec ial purposes )

• Bimanual input

• Eye-trackers

• Sensors (accelerometer , gyroscopes, depth cameras )

• 3-D trackers

• Data g loves

• Haptic feedback

• Foot contro ls

• Tangible user interfaces

• Digital paper

Criteria for success

• Speed and accuracy

• Efficacy for task

• Learning time

• Cost and reliabi lity

• Size and weight

10.3 Pointing Dev ices 349

Another way to think about pointing de,,ices is whether they use absolute or relative input. Touchsc reens, graphics tablets, and eye-trackers also use an input model where the input (motor) space is directly mapped to the output (visual) space. This is called absolute input, since one point in motor space corresponds to one point in visua l space . Relative input, on the other hand, deals with transla­ tions (and rotations) from a current position and includes devices such as the mouse, joystick, and trackball. While this distinction is not used in the below discussio11, the reader may want to think about each device in absolute versus relative terms as well.

10.3.2 Direct-control pointing devices Touchscreens are the canonical direct control pointing devices and allow users to interact directly with the visual content of the screen by touching it with their fin­ gers. Because of their natural affordance, i.e. their form inviting appropriate action, touch-enabled screens are often integrated into applications directed at novice users in which the keyboard can be eliminated and touch is the main interface mechanism.

Early touchscreen imp lementations had problems wi th imprecise pointing, as the software accepted the touch immediately (the land-on strategy), denying users the opportunity to verify the correctness of the selected spot. These early designs were based on physical pressure, impact, or interruption of a grid of infrared beams. High-precision designs dramatically improved touchscreens. The resistive, capacitive, or surface-acoustic-wave hardware often provides up to 1600 x 1600 pixel resolution, and the so-called lift-off strategy enabled users to point at a single pix