A Better iPad Pro and Apple Pencil User Experience

[ YouTube ] Steve Jobs, back in 2007 during the introduction of the iPhone:

Who wants a stylus? […] Nobody wants a stylus.

Steve Jobs again, in 2010 while introducing the iPad:

If you see a stylus, they blew it.

At first glance, it seems Steve Jobs doesn’t like the stylus. For good reason: remember Microsoft Windows XP Tablet Edition? I do. Windows XP, as well as all of the previous versions of Windows, was designed from the ground up with the assumption that you would be using a keyboard and mouse (or some derivative) to interact with its graphical user interface (GUI). With the Tablet Edition, Microsoft added the ability to use a stylus, while keeping almost all of the GUI the same. Yes, you could write and draw on the screen with a stylus, but the way you click, or tap, menu items, dialog boxes, etc. were identical. The stylus was thought of as just another input device, in addition to the keyboard and mouse.

In the mobile space, Palm did a similar thing with its Treo: add a stylus as an additional input device to the keyboard and d-pad. The main modus operandi for interfacing with the Palm operating system was the keyboard and d-pad. The stylus was a third option. UI elements remained the same and so did the user experience.

I remember using a Motorola Q running Windows Mobile 6.5, with nested menus and all; it was terrible, because I needed a mouse to more effectively maneuver around a desktop UI that was shrunk down to smartphone size.

Steve Jobs didn’t like the experience either. His thoughts on using the stylus this way on a desktop UI was spot on: nobody wanted it because nobody liked it. What he had in mind — a mobile UI designed around the use of our fingers — saved us from a lot of frustration.

Continuing with this logic, if Steve Jobs was alive today, he would not be against the idea of a stylus for writing and drawing, and not as an additional input device for maneuvering around the UI. I think he would be enthusiastically for it.

The Smartphone, Tablet & Stylus

In 2011 Samsung introduced the Galaxy Note, a 5.3-inch smartphone with a stylus. The stylus wasn’t like anything else on the smartphone market. The Palm Treo used resistive touch technology; you pressed down on the screen with the stylus (finger too) to depress the top level of wires and touch the bottom layer of wires. Resistive touch isn’t very accurate and wears out after a while.

The original iPhone as well as almost all subsequent smartphones and tablets to date use capacitive touch technology, a solid-state technology that technically does not wear out. A stylus used with a capacitive touch screen works just like a finger does; there are many cheap styli you can get for your iPhone and iPad. Dirt cheap, yes, but there are some significant limitations: no pressure sensitivity, no hardware-based palm rejection, no hardware-based eraser, etc. Smarter software can offset some of these limitations, and a Bluetooth stylus helps to alleviate a bit more, but you’re reminded from time to time that these limitations still exist. Palm rejection not always working being the most frustrating.

I wanted to write and draw on my iPad 3 so when the Pencil by 53 was introduced I got it. I liked it, when it worked. Palm rejection worked most of the time, but when it didn’t it was frustrating especially when I was in the middle of drawing something elaborate. I had to put the Pencil down, put two of my fingers and rotate them counter-clockwise to undo. That didn’t work all the time either! Despite the excellent feel of real wood, a soft but not too soft rubber tip, and an eraser, the times when it didn’t work frustrated me enough not to want to use it much.

iPad Pro and Apple Pencil

The new iPad Pro and Apple Pencil takes this technology — capacitive touch screen and Bluetooth stylus — up a notch. Apple:

A new Multi‑Touch subsystem in iPad Pro enables the striking capabilities and pixel‑perfect precision of Apple Pencil. Using incredibly fine‑tuned pressure and tilt sensors, Apple Pencil instantly recognizes when you are pressing harder or shifting its angle. So you can vary line weight, create subtle shading, and produce a wide range of artistic effects — just like with a conventional pencil. But Apple Pencil does far more. It can be a fountain pen, a watercolor brush, and even a calligrapher’s nib.

One more from Apple:

The lightning-fast responsiveness of Apple Pencil separates it from other creative tools. That’s because its latency — the tiny delay between when you begin drawing and the time it appears on the screen — has been reduced to an almost imperceptible level.

iPad Pro knows whether you’re using your finger or Apple Pencil. When iPad Pro senses Apple Pencil, the subsystem scans its signal at an astounding 240 times per second, giving it twice the data points it normally collects with your finger. This data, combined with Apple‑designed software, means that there’s only milliseconds between the image you have in your mind and the one you see on the display.

Within its slender case are intricate and accurate pressure sensors, capable of measuring a range of forces. The carefully positioned sensor determines precisely how hard the tip of Apple Pencil is being pressed down. Press harder to draw thicker lines. Or use a gentle touch for wispy hairlines. The variety of creative effects is virtually limitless. Which means you are as well.

Two tilt sensors built within the tip of Apple Pencil calculate the exact orientation and angle of your hand. As you naturally write or draw, the relative positions of each of these sensors can be detected by the Multi-Touch display. So you can create shading effects simply by tilting Apple Pencil the way you would a charcoal or conventional pencil.

I took the iPad Pro and Apple Pencil for a spin at a local Apple Store and the two felt good together. A took a couple of photos of what I was able to do with the two: one and the other one. A friend asked me what my experience was like and I responded that Apple seems to have designed the iPad Pro and Apple Pencil for artists. For comparison I went across to the Microsoft store and played with the Surface Book and I felt as though Microsoft built the Surface Book for business people. The iPad Pro and Apple Pencil combo is not laggy at all, but there was still a very slight lag. I don’t know exactly how many milliseconds, but the lag was there. Was it a bad experience? No, it was significantly better than any pre-iPad Pro and Bluetooth stylus combination.

The Best Stylus Experience, So Far

But what’s better is Samsung’s solution used in its Note series of smartphones and tablets. When I use the stylus with my Galaxy Note 4 there is no perceptible lag. Sure the stylus stinks, the apps are mediocre, and the overall UX isn’t all that great, but in terms of lag it’s the best there is. And there’s good reason for it: Samsung went to the best digitizer maker in the world and made a deal. And that brings me to Wacom.

Samsung uses Wacom’s digitizer technology. When it comes to the professional stylus market Japan-based Wacom is the de facto standard; Wacom owns about 85% of the world’s digitizer market.

[ Know Your Mobile ] Paul Briden:

“We had a similar mindset, a similar vision,” added Masahiko. He said that both Samsung and Wacom viewed the use of styluses in mobile devices as focusing on creativity and content production as opposed to the content consumption typical of touch-based devices without styluses.

That’s Masahiko Yamada, president of Wacom.

“You cannot write with a finger,” said Masahiko, “Touch does great things for control such as zoom and navigation, but when you try to write something, touch doesn’t work.”

Masahiko also considers Samsung to have been uniquely positioned to develop the Note concept: “Samsung’s the only one with enough clout to commit to the product,” he said.

Both Samsung and Wacom saw the opportunity for an active digitizer on smartphones and tablets. They saw correctly. Writing and drawing with a finger — unless you’re finger-painting — doesn’t work very well. What works best is an active digitizer. EMR or Electro-Magnetic Resonance technology was originally developed by Wacom. What’s so special about EMR? The stylus doesn’t need to be powered: batteries are not required. This little feature impacts the experience in many ways. Because batteries are not required, a complicated printed circuit board is not necessary. You don’t need to worry about the stylus not working because it is running out of power. The stylus can be simple, light, and inexpensive. Because of simplicity there is less to break and can last a long time. The stylus can be light. And if you lose your stylus it won’t cost $99 to replace it.

Samsung’s Galaxy Note incorporates a Wacom-technology sensor board in the screen to detect the movement of your stylus. This particular screen generates a magnetic field, which then induces a small amount of energy from the resonant circuit in the stylus. That small amount of energy generated in the stylus returns a magnetic signal to the screen. And that’s how movement of the stylus — including angle, speed, and pressure — is detected. Design professionals have been using Wacom for a long time, and for good reason: there is no perceptible lag between the screen and the stylus. And the system never — and I mean 100% never — confuses your palm for your stylus. There’s one more thing that’s really cool: you can hover over hyperlinks with your stylus and enjoy all the hover animations that a lot of designers incorporate into websites. If you’d like to read more about EMR, hop on over to Wacom’s EMR page. Oh, and Samsung owns 5% of Wacom. I don’t know if Apple can license Wacom technology, but I would be surprised.

A Better iPad Pro & Apple Pencil Experience

If Apple were to incorporate a Wacom-like EMR-based active digitizer in a later version of the iPad Pro and Apple Pencil, the experience would be a huge upgrade. Apple could remove a lot of circuitry, including the accelerometer and gyroscope, to make a much simpler Pencil. The much improved and simpler Apple Pencil would:

  • be considerably less expensive,
  • let you hover over hyperlinks,
  • never run out of battery (and you’ll never have to recharge it),
  • be a lot more rugged (less things to break),
  • let you flip the Pencil over and erase like you would a regular pencil with an eraser.

This future is not likely though, because Wacom owns EMR, and Samsung owns 5% of Wacom. I guess Apple could outright buy Wacom. That’s certainly possible with all the money Apple has in the bank. But I’d like to introduce you to a different possible future for Apple: TRAIS, and its Wacom-like digitizer technology that does not infringe on Wacom patents.

South Korea-based TRAIS was founded in 2000 and is headquartered in Ansan. TRAIS developed what the company calls T-Digitizer, which sports all the benefits of Wacom’s EMR technology: hardware-based complete palm rejection, imperceptible lag, simple circuitry (relative to a capacitive touch plus Bluetooth stylus like the Apple Pencil), tilt and pressure detection. Another benefit over Wacom for LCD manufacturers is the ability to completely integrate the T-Digitizer into the touchscreen. It’s also lighter than Wacom’s solution and consumes a whole lot less power. I think Apple might like that.

Full disclosure: I have been contracted by a friend to help folks get to know about TRAIS technologies, especial folks at Apple and other companies that want to incorporate a better stylus experience on smartphones and tablets. If you’re interested, contact me: