Neat, but I'd imagine that this represents a steep learning curve for many. Say that I borrow a friend's car that has this. How do I figure out how to change the temperature? Touching five fingers close together is easy but not intuitive.
Try tapping in the very bottom right corner with one finger. There's an invisible button that shows the fake Driving Directions UI. Then tap on that with 5 fingers at once to make the hints appear. They are just static graphics too though.
I agree. There are many good interface ideas here, but cars interfaces need to reach people of all sorts. ie. people with handicaps, disabilities, etc.
Great ideas though. It's great to see someone working on this! Nice job.
There is going to be a lot of hate in here about how impractical this is, but I just wanted to say nice job on the clean design and presentation. I watched through all of it.
Touchscreens are not only dangerous to drivers, but fairly unusable unless you can make judgements based on touch alone. I've used this[0] system before and it satisfies both of those requirements. Can it be optimized further? Possibly. But I don't beleive touch screen is the future (until we get driverless).
This is hands down far, far better than what's out there already. A lot of thought went into this and it shows.
Could there be a few options to enter other modes, though? What I mean is that right now there are only 8 adjustments possible. That's better than the 4 I thought it was limited to, but still nowhere near enough for modern cars.
I would estimate that on a modern car you need to be able to adjust something like 100 parameters. Only a few of them frequently, but definitely more than 8 in total.
One of the thoughts I had was that using a (say 90 degree?) twist motion along with a number of fingers could bring up sub-menus. That way if you want to adjust other radio parameters like fade, balance, equalizer, etc you would put two fingers on (for volume) and twist.
Maybe you'd arrange the different controls in a loop so that every twist indexes to the next and there's a ~10 second timeout before you head back to the default. So if you want to adjust all the "volume" parameters you put two fingers and twist right once for balance and adjust, twist right again to get fade, twist right again to get highs, etc. Make a mistake? Twist left to go back and re-adjust.
If you want to get out of one of the sub-menus you're in prior to the ~10 second time-out for default, another gesture? Maybe a double tap? I don't know on that one, really.
> I would estimate that on a modern car you need to be able to adjust something like 100 parameters.
Not the author, but the way I see it this is specifically an interface for adjusting things eyes-free when driving. In an actual implementation, there would be some sort of physical switch between the in-drive control (demoed here) and a high-density "at rest" operation for passengers or more specific tasks (e.g. configuring the GPS).
> One of the thoughts I had was that using a (say 90 degree?) twist motion along with a number of fingers could bring up sub-menus.
The problem with this is there'd likely be a rotational component to movement, and a movement component to rotations, discriminating becomes harder and the chances of false positive (and thus frustration) increase. Only using a single axis is actually a smart move as far as I'm concerned.
> The problem with this is there'd likely be a rotational component to movement, and a movement component to rotations, discriminating becomes harder and the chances of false positive (and thus frustration) increase.
I agree that you might end up rotating your fingers SOME while moving them up or down. But I'm not sure that I agree that you would move them 90 degrees completely unaware. In order to move your thumb and index finger from vertical aligned to horizontally aligned you're going to require some (maybe a lot) of wrist motion. It's hard to make that wrist motion happen accidentally.
You might end up setting it to accept a range between 70 and 120 degrees to qualify for a "go to the next setting" motion and you can provide some kind of feedback (vibratory or audible or both) that you have indeed twisted far enough to trigger the change.
When designing pie menus, there are both physical (articulatory) and mental (cognitive) factors that you should consider when choosing the number of items and their directions.
Gordon Kurtenbach and Bill Buxton did an experiment that varied the number of menu items and measured the selection time and error rate. They expected the selection time to be monotonically increasing as they added more items. To their surprise, that was almost true, except for the transition from seven to eight items. It was quicker to select from an eight item menu, then from a seven item menu!
That, I believe, was because of the effect of the cognitive bottleneck of associating the items with which direction to move, not the physical difficulty of moving in those directions. While the slices of the seven item menus were wider and had more area than the slices of the eight item menus, and Fitts' Law would predict that seven item menus were faster than eight item menus, Fitts' Law does not take into account the time it takes for users to map their intended command to the direction of that command, and how the mental framework in which users remember and relate the directions to each other affects selection time.
Eight item menus have all of their items in well known directions, each of which is associated with a very familiar concept, which come in nice pairs, and the pairs come in convenient orthogonal groups, like up/down, left/right, vertical/horizontal/diagonal, like compass directions.
Twelve item menus like a clock face also work well, especially for circular sets of items like hours, months, zodiac signs, etc. But the effect going from 11 to 12 is not as dramatic as from 7 to 8. But still the 12 item menu has a nice familiar aesthetically pleasing cognitive framework (including opposite and orthogonal pairs, first tier vertical/horizontal axes and second tier in-betweens) that you can often exploit, depending on the content.
Three item menus are still slightly faster than four item menus, because the effect of the proportional difference in target area overwhelms the difference in the number of items, and two of three triangular directions are well known concepts that are pretty easy to remember, compared to six of seven directions.
An eight item menu optimally exploits that effect, while a seven item is unfortunately sub-optimal with mostly difficult to remember directions. There's no word or concept in the English language for six out of seven of those directions -- they're all just kind of slanted differently, similar to but not quite like the well known compass directions, and there are no nice symmetries, opposite or orthogonal groupings to exploit, by arranging complementary items in opposite directions, independent pairs along orthogonal axes, etc.
So if you're designing a pie menu with seven items (or even eleven), it's better to just throw another item in to bring it up to eight (or twelve)! I gave an example in the DDJ article of a "seven days a week" menu with an additional "today" item thrown in at the bottom to bring it up to eight, with the weekend and today in the lower part of the menu, and Wednesday at the top. See, I don't even have to link to a picture or enumerate every item for you to easily visualize and remember it!
I would build this switch with two IR sensors on each side of the console. If the arm and hand touching the screen is approaching from the passenger side, many things are possible. If the car is in motion and the arm is approaching from the driver's side, many things are locked out.
> Not the author, but the way I see it this is specifically an interface for adjusting things eyes-free when driving. In an actual implementation, there would be some sort of physical switch between the in-drive control (demoed here) and a high-density "at rest" operation for passengers or more specific tasks (e.g. configuring the GPS).
I've found that pie menus with four different directions are quite reliable (see the description I posted here of the ConnectedTV palm app, which supported buttons with four different stroke directions plus tap). Eight different directions work well with a mouse or big touch screen, in situations where you can afford to give the screen some visual attention and the consequences of making a mistake aren't deadly, but I believe four directions are reliable enough for automotive applications, as long as you require enough motion to get enough "leverage" and give decent feedback.
At some point you would probably be better off having physical button. A huge touch screen on which you need to draw weird gestures you had to memorize does not strike me as very user friendly.
That said that is neat implementation and it very elegant but I would like to have that in the living room rather than the car.
A possible evolution I guess would be to have different screen you would flick through that would group 2 or 3 command together with a big visual cue on the background.
It's really not. It's a huge step back from even the interfaces he presented.
You exhibit the symptoms of somebody just introduced to multitouch and gesture recognition and trying to "explore the possibilities." The reality of gesture control is that they make for horrible user experiences. You are encouraged to try it on your own if you aren't convinced. Adobe, Apple, and a number of other companies did a lot of user study into this.
Double tap, twists, etc. All those gestures suck and fail for different reasons sadly.
Actual buttons are better than fake buttons on a touch screen. What he dreamed up is in my opinion better than fake buttons on a touch screen. Real buttons might be better than his interface.
I got interested in multi-touch something like 6 years ago when I saw this: http://vimeo.com/6712657 So I don't think that I'm someone "just introduced to multitouch" at all. But thanks for trying to subtly call me a naive idiot!
So far the best people are able to do is two finger scrolling and pinch zoom. It's kinda sad.
> What he dreamed up is in my opinion better than fake buttons on a touch screen.
No you are "new to multitouch" in the sense that you haven't tried to implement multitouch yourself, conducted a user study on multitouch learning curves and intuitiveness, or used a real life application exhibiting the multitouch you claim to understand for a prolonged length of time.
Seeing one video qualifies you perfectly for being "just introduced to multitouch."
It's hands down better than other touchscreen-based controls. It's still quite a ways behind the old technology -- i.e., physical buttons.
I drive a 1998 car and can operate every single control without looking at the console, except audio tone controls and choosing a specific FM station not in my presets. Granted, navigation is missing since it's an older car -- better voice controls on the phone could make up the gap.
Perhaps I'm just not envisioning it correctly but this seems to replace looking at the screen to touch a button with looking at the screen to figure out how far apart my fingers are, or how many fingers have touched the screen after hitting a bump. I'm not yet convinced it's a great improvement, though an interesting idea. I'd like to see a demo in a more realistic environment, i.e. in a car on a non-perfect road rather than on a table.
Most people can tell how far apart their fingers are via proprioception. It may take a little while to know where the boundaries are, but it's easy to determine if your fingers are close or far.
Feeling how many fingers are touching the screen is also easy with the normal sense of touch. Add in a little hysteresis to handle bumps (you could even integrate info from the accelerometer) and I bet that even bumpy conditions will be significantly improved with this system.
As someone who has instructed parents and other non-techies on how the number of fingers and how to move those fingers on a multi-touch display, I don't consider it remotely trivial for 'most people'.
I generally agree. I just bought a car and one of the reasons that I skipped the nav system is that I didn't want a touch screen for things that work better as knobs.
That being said there is a balance to strike in today's smart-device world. Having a touch screen can save space and reduce control clutter when trying to cram bluetooth, gps, hd radio, playlist, and other controls onto the dash. They allow the car to only show me what I need when I need it.
Looking at most touch screen dashes, such as those in the article, show that they fail at this, of course.
In a conference room at Sun in the 1980's, I saw a touch screen control panel on the speaker's podium for raising and lowering the lights, projector screen, controlling the projector, etc. It has a small solenoid with a rubber mallet at the tip which would go "thunk" against the back of the glass to provide haptic feedback for the virtual buttons. Pretty low tech, cheap, simple but effective.
What if you put one finger down to activate the interface, and tap with another finger (while still holding the first one on the display) to cycle through the menu?
This requires at least two fingers, but would allow for control of more settings, and would’t require you to remember how far apart your fingers need to be to get access to a control. You could have the car dictate the currently selected / active setting as you cycled through.
a touch screen is essentially one big button. It hides away all the complexities of a physical interface.
The problem with this is people tend to add a lot of extras. I wrote[0] that the internet radio thing cannot be done in the Tesla without a touchscreen.
FWIW, this idea is slightly better than the one in my Tesla screen complaint, but still not any better.
I like this attempt to rethink the UI for vehicles and I think you are on to something. The absence of any cues I guess is still a problem and the finger-distance based approach could be a problem for some.
How about selecting what you want to control (Temp, Channel, Air etc) on the first pass followed by adjustment of the selected control on the second?
Everything he shows here is accomplished already by a fairly nice set of buttons on my steering wheel. Each button has a unique shape, so finding them without ever looking down is very easy, and my hands never leave the wheel.
What I find hard about in-car UIs are things like entering and editing navigation directions or selecting music to listen to given a 32GB USB stick full of songs.
There's always some horribly designed onscreen keyboard or "iDrive" type knob controller, both of which are quite horrible experiences. To make matters worse, the UI is laggy and unresponsive, and in an effort to make things "easier" onscreen controls frequently dim & disable when not actionable or when the car is moving vs. stationary.
1) Why was a single touch interaction not demonstrated? Each control requires 2+ fingers. Maybe that can bring up a help menu or other information?
2) There is no demonstration of boolean switches and button controls. Things such as turning on a defroster, or enabling pairing with a bluetooth device. I'd be curious to see how those would play into the ux flow.
I'm not sure if this is intended, but I'd imagine single-finger controls would fall back to the buttons layout. It would be pretty impractical to have this as the only way to control the infotainment system, since there'd be way more than 8 parameters.
I really like this. My only issue is that the prototype ignores single finger, which to me feels like it should be the most important control. The other issue with number of fingers based gestures is, well, people who are missing fingers. I also think using both the X & Y position could be really useful. Why not tune the radio by dragging up and down and adjust the volume by move right and left, or other combined gestures?
I kinda feel like the default screen for this thing would be a map, so a single finger would just scroll the map around. And with one finger you can't find a diameter so there's no second option with wider spacing for a single finger.
I really like the idea of an always-on GPS that can intercept commands for other stuff (radio, HVAC, etc) and forward them to the appropriate app/microcontroller/hardware.
One possible explanation is that these are intended as additional controls and the single finger interaction is still reserved for a more button based approach.
This may be quite hard to learn for a while. Especially the less often used options. I'd be interested to give it a go though.
For now the best car interface I've used was voice recognition. It usually works great, because it's not a general purpose text dictation. It only needs to recognise less than 100 words and it's not that hard to do. For example I'm not a native English speaker, and I can't get google voice input to capture most of the sentences correctly - it's simply not usable for me at the moment. But I'm quite happy to use voice commands in a Prius - I don't think I've ever used more than 4 or 5 of them, always perfectly recognised.
Voice seems to be the best and safest user interaction for vehicles. If the voice software knew the car was in motion or at a difficult intersection perhaps it could change it's tone and sense of urgency. It does make me think if there will be VI(voice interaction) designers in the future.
I find voice controls extremely distracting, mentally. It takes your focus off the road just as much if not more, I believe. Plus, I always have to repeat myself a bunch because I believe they are programed to recognize a man's voice.
Interesting concept but I am concerned about using multi-touch to select devices. First off it's hard to remember, and secondly, if I am missing a finger, I can't set the air-flow rate in my car? Also the differentiation between close and spread fingers is going to be tricky for a lot of people to remember.
Edit: Another thought... the UI for many funtions needs to provide visual feed back... say for what system I am selecting or what station I am tuning. The text is quite small which would cause even more of a distraction to try to read. Yes I can begin the selection process without a visual cue, but I still need to look to see what I'm selecting. Not sure I see a large improvement in driver distraction there.
Slick for users like us? Definitely. Intuitive for the average Joe out there? I don't think it would fly. Too much to remember and too much dexterity required. My Mom could never use this. At best I see this as an alternate UI the driver could select over something more conventional when they feel comfortable with it.
Kudos for thinking outside the box though, there are definite UI gems here that could be leveraged.
I personally wouldn't scrap an entire design when < 1% of the population can't use it.
Intuition isn't important. The designer mentioned that this is a trade off, and is intended for regular users who have muscle memory of their controls. You can always add a tutorial or "help" mode by pressing a button in the bottom right or whatever, anyways.
The beauty about touch screens as input mechanisms is that they can obviously show an infinite number of interfaces. I absolutely agree that there's a right time for the right interface. My proposal is just one possible way to use a touch screen for input.
It's not just people missing fingers. Anyone wearing gloves, or with dirty hands, will have issues operating a multitouch screen, in a different way than people with phones might.
BMW's iDrive was/is considered a mess because of the single knob-do-everything approach, but it had potential for what could had been. This seems like it took what bmw had, implemented it as a virtual knob and removed the main problem (idrive's submenu system). I like this, great way to think about the problem and present a potential, easy to understand solution. This approach also allows the touchscreen to be put in a more natural position like closer to the arm rest. I'd just make it a taste more visual. I bet the haptic feedback that some android keyboards offer when keys are pressed would be a great addition.
idrive is way better now than it was initially... they added some buttons. I use it every day and it's very easy. if bmw tried to replace it with some touch screen gizmo I would push the car off a cliff.
The latest iteration is pretty much spot on. Everybody unfairly discredited the first version - the technology hadn't matured. Much like the iPhone 2G (dismissed by the majority of non-techies)! However, not everyone can afford a brand new BMW, whereas pretty much everyone I know has an iPhone/Android, so their opinion was never re-evaluated.
The latest iteration is virtually perfected. Rocker buttons on top, and with a known number of clicks I can control 80% of all the main functions without taking my eyes off of the road at all. For the other 20%, it takes a quick glance to see the state of the screen and then I know "okay, now just two clicks down and I'm there".
The upcoming version (currently on the only the latest model to be released) has a small circular touchpad on-top of the wheel so you can draw letters without having to twist the knob. Useful for entering Postcodes into Nav, for instance.
I recall that both Audi and Mercedes have since copied the iDrive paradigm.
This is way better than what they're building today, but honestly, I think a panel of PHYSICAL buttons, knobs and switches is even better.
The iPhone-style interaction -- a flat screen of glass where you manipulate "controls" with touch sensors -- has one major advantage: it allows an INFINITE number of different controls. (Every app is its own set of controls.) But to do this it relies heavily on continual interactive feedback with your vision.
When driving, you can't do that continual feedback. Perhaps someone clever can figure out how to do interactive feedback with audio (you can use that sense somewhat without taking your attention from the road). But otherwise, you can really only manage as many motions as your fingers can "memorize" (muscle memory mostly). If you're doing that, why not stick with switches and knobs, where your tactile sense can help out.
Something in between would be ideal for me. A center console with a touch screen for configuring settings when you're not driving, but physical buttons, switches and knobs on the steering wheel for when you're driving.
My Model S has these same controls embedded in the steering wheel. Volume, music control, and climate controls are all there.
That's actually where I find I spend most of my time interacting with the car. I usually only use the touchscreen for more complex things like manipulating the map. For complex inputs like choosing a song or entering my destination, I use the voice controls.
I don't think the solution is one interface. It's multiple interfaces that function best in certain contexts, with some overlap. Look at Google Glass for an example of this.
I'm actually a big fan of the Hyundai Veloster's touch screen. It provides all the functionality you'd expect, without the convoluted menu systems of other cars.
The biggest oversight seems to be the utter lack of discoverability. Can you imagine trying to figure out of this works on a rental car ? or explaining to your grand-father how this works ?
The designer gave a lot of thought to the problems with current touchscreen UI in cars, and kudos to him for doing so.
However, it's not the solution yet, in my opinion.
First, all the methods of interaction are invisible. There are no cues to tell the user what two fingers vs. four fingers will do, or what the difference is between a close-fingered gesture and a wide gesture. At best, you'd have to standardize the meanings across all cars (something that rarely happens in autos, even today there's no standard for where the wiper controls go, for instance), and even then, people would have to memorize those invisible gestures before they could use their car.
Secondly, accidental inputs would happen with some frequency. People have a wide range of hand sizes, motor control skills, even number of fingers. Suppose I've got two of my fingers in a cast? I can use an existing car's dashboard just fine with that temporary impairment, but not this.
Finally, it's addressing a problem that already has a solution—physical, dedicated input devices. Humans have spatial memory, we learn where things are, and learn to reach for them without looking or even thinking. Your muscle memory tells you how to reach the wiper stalk or the gear shift, which is why it's sometimes disconcerting to get into a car with the gears on the steering column if you normally shift in the center console. We need in-car systems that let the physical feedback, affordances, and reliable location of buttons, knobs, and switches interact with display systems that are designed for the attention a driver requires. You should be able to keep your eyes on the road and still adjust the air conditioning—something the designer in this video recognizes—but that shouldn't require learning invisible gestures that are prone to user error.
You had the same gut reaction as me but if you look at the beginning before he started using it there was a legend of how many fingers map to each icon (no room for text as well?).
Beyond spatial memory what's good about this is that you don't have to look anywhere while still adjusting (some speech feedback about what mode you've selected would be good). The lack of sight is the most compelling aspect -- but I can guarantee you my parents would likely never remember the mappings and always have to look at the legend. Even then they'll probably be confused.
By "cues," I mean visual affordances that persist, not instructions that are shown before using it. It's a violation of one of the fundamental rules of usability, "Recognition Rather than Recall."
Look at a traditional dashboard. The air conditioning knob is a different shape and size from the stereo's volume knob, and located in a different place (ideally). Those things make it easy for the the driver to distinguish one from the other, and find them while keeping their eyes on the road.
There are other cues that usually exist to help as well, such as a blue-to-red graphic around the temperature control, the volume control being next to other audio features like the radio display, etc.
In an environment where the user has to keep a 3,500 lb steel machine safely controlled while traveling 100 feet per second, the rules of usability become incredibly important.
"The air conditioning knob is a different shape and size from the stereo's volume knob, and located in a different place (ideally)."
You've nailed it sir. Shape coding[1]. If it's good enough for F-14 fighter pilots, it's good enough for my Toyota Camry.
I echo the sentiment of others. The OPs heart is in the right place here. Mimicking sight free displays on a touch screen is laughably terrible. Take this picture [2] from OPs post. Why are the media buttons ellipsoids? They're just smaller targets to hit compared to squares!
In academia, we've been researching this problem, eyes-free mobile interaction, for a very long time. Particularly in the context of interaction while walking.
There are numerous approaches (e.g., Flower menus [3], utilizing pressure [4], utilizing motion [5], the list goes on). The big takeaway is that it seems like more successful systems result from the multimodal approach.
OP, you've done some good work here, but consider incorporating other sensors. Voice? A camera for in-air gesture recognition? A biometric sensor that gives reasonable defaults automatically based on learned use of the driver? Touch is only one part of the problem if we move away from shape coding, and we will very likely need to utilize multimodal input if we hope for success...
...or, you know, we could just stick to that whole buttons and knobs thing.
I think you got it right. My proposal was definitely not meant to be the end of the conversation. Much rather the beginning.
I'm sure there are huge opportunities in exploring a more diverse set of input mechanisms (and their combination).
In this case, I'm simply trying to make a point that touch screens can be more control panels with buttons and sliders on them.
Memorably differentiating the possible commands is the reason I designed the layout of the pie menus in SimCity the way I did.
SimCity has a tool pallet of various building and editing tools, which are related to each other in various ways, and have different costs, functions and properties. I tried to arrange them into a set of submenus that made those relationships and differences more obvious and easier to learn. I also arranged the static tool palette on the screen to reflect the layout of the pie menus and submenus, to serve as a kind of map to the pie menus (which foreshadowed the spatial map design I explored with Method of Loci, described in another posting).
The top level menu includes: magic marker and eraser (for drawing on the map), roads and rails, power lines and a bulldozer, and submenus for common zones and bigger buildings. The zone submenu includes residential, commercial and industrial zones, police and fire stations, and a query tool. The building submenu includes a stadium, park and seaport, coal and nuclear power plants, and an airport.
As you can see, the size of the icon reflects the cost of the tool. The borders of the icons are color coded to reflect the cursor you see on the map that shows where the tool will affect. (That was useful for multi player mode, so you could see which tools the other players had selected, so the tool palette served as a legend for the cursors on the screen.) And the tool palette looks kind of like a totem pole, which is vertically asymmetrical, differentiating different kinds of tools and making them look unique, and horizontally symmetrical, reflecting pairs of similar tools and making it look aesthetically pleasing.
If all the tools were the same sized squares arranged in a regular grid, it would be much harder to differentiate them and quickly select the one you want. Instead, they're arranged more like a bouquet of unique flowers that each have their own special features that make them easily recognizable and memorable, while telling you something about themselves.
Designing user interfaces this way is a artistic balancing act, highly dependent on the set of commands, and requiring a lot of iteration, testing and measurement, as well as willingness to explore and experiment with many different alternatives. It is not easy, and there is never a best solution, or even always a good one. It's not something you can expect end-users to be able to do with their own custom built menus. But it's worth the effort to try, and I think it's also worth the effort to train designers and even users to promote a literacy in interface design.
One idea I've had is to develop a game called "PieCraft", that has user-editable pie menus that are first-order in-game player craftable artifacts. World of Warcraft and its rich ecosystem of user interface extensions supports a wide range of player customizable user interfaces, because there are so many different commands, spells, buffs, weapons, armor, etc, and each role of each character of each player requires a totally different set of them in different situations. The more casual MMPORG "Glitch" had bags that users could arrange in hierarchies and put their items inside, which were surfaced on the user interface as buttons they could press and objects they could drag in and out of the world and other bags. How well you arrange your items in WoW and Glitch had a significant effect on gameplay.
PieCraft could take this further, to train players in user interface design literacy (much like "Code Hero" aims to train players in computer programming). You could find empty or pre-populated pie menus in the world, pick them up and make them your own, and edit them by moving i...
Don Hopkins! I've read the pie menus paper more than once in my academic career, and it took this post for me to make the connection between The Sims (which I played in my youth), pie menus, and you.
So, since I have the opportunity to fanboy out a second: thank you for your contributions, both to gaming and human-computer interaction.
Thank you! I still ask Ben Shneiderman to review stuff that I write, and value his feedback -- I've learned a lot from him, and he really cares a lot for users and is great at explaining stuff. He's the one who insisted on calling it HCI instead of CHI, to put Humans before Computers. I got to talk with Mark Weiser about The Sims before he passed away, and he thought it would be popular, but he never got to play it, unfortunately. He certainly had a clear vision about where ubiquitous computing was heading, and would be fascinated and excited about what's possible now, so I try to be that way myself on his behalf.
Ben Shneiderman toured my laboratory a couple of months back. He actually took the time to sit down and use and discuss some of the UI stuff I was developing as part of my PhD research.
I'm sure he doesn't remember it, but those 15 minutes were extremely beneficial to me. He's a super humble, brilliant scientist and designer. I have the utmost respect for him.
Yes, he's a really nice personal guy who listens well and is great at writing and talking. He loves to wave his hands around and gesture as he talks, so it's really fun to watch his talks on fast forward, and guess what he's talking about! ;)
I'll bet he remembers, and tells people about your look and feel that he saw and felt!
I'll also add that there are way too many ways of interactions that must necessarily be learned with a substantial time investment. Time might not seem a big deal when dealing with cars since you keep one at least for a couple years, but the mental effort required to learn all those gestures from scratch is so high that it'd definitely put off most users and result in a frustrating experience for a long stretch of time.
Maybe the designer could attack the standards problem by writing this for embedded android with the idea that android will fill the car ui space soon (assuming the _embedded_ android docs improve).
The UI can provide user feedback with static electricity, and the current could increase with the speed of the car.
As for the final point, a ten key resting within hands reach would work fine. You could memorize the number sequence for different commands, and get audio feedback. Talk about a proven UI, basically the telephone. Baby boomers ought to be comfortable with it.
From my experience working at TomTom, which uses touch screen as well as voice commands and feedback, and has purchased automotive oriented companies and integrated their products and employees into their own, I've learned that the automotive industry moves at a glacial pace compared to the computer industry.
There are both good and bad reasons for this. But the important point is that automotive interfaces are designed to be built into cars at the factory, and used for many years without requiring any updates. Even if updates are possible, most people don't get them. So it's a completely different mindset and approach and time scale than people from the Silicon Valley dot-com startup industry have.
Another factor is that the devices built into cars have a very long design and production lead time, and by the time the car comes out, the built in hardware and software is already quite obsolete compared to the smartphone the driver probably owns.
Factoring the problem out of the car to run in a smartphone or tablet itself also has its own frustrating problems, because when you're designing an automotive computer system, you can't predict what kind of technology and standards will be available or popular by the time it ships.
It's a very difficult problem space, and the stakes are extremely high, not just financially, but also because cars are weapons of mass destruction that kill more people and destroy more property than all terrorists combined could possibly dream of.
"Factoring the problem out of the car to run in a smartphone or tablet itself also has its own frustrating problems, because when you're designing an automotive computer system, you can't predict what kind of technology and standards will be available or popular by the time it ships."
Would you mind to elaborate on this statement?
To me it seems like moving control over some functions within the car, for example music or calling, might make more sense to do on the phone specifically because of the phone's fast upgrade cycle and tie-ins to one's personal information cloud.
They prefer to invent their own protocols and busses for the different systems of the car to communicate with each other, that aren't compatible with phones or even protocols like http.
For example, TomTom had a big emotional investment in some 1990's-style proprietary remote procedure call protocol that one of the automotive companies they bought had developed. It had its own interface definition language and rpc stub compiler, which seemed revolutionary in 1988 when some clever student came up with the same idea to make it easier to write C wrappers for SunRPC protocols so you didn't have to write them by hand.
More rational heads were pushing to just use modern off-the-shelf technologies like http/rest/json/etc, but the jobs of some people in some department depended on them being perceived as having been doing productive work on their own proprietary message bus for the past three years or so, and that was what they delivered, so that was what they were damn well going to use.
Smartphones are a huge threat to companies that make their own little boxes that are hard to convince people who have smartphones to buy, so they weren't exactly enthusiastic about embracing and supporting smartphones and tablets.
When a company that operates at glacial automotive development speeds has been working on their own proprietary solution to a problem that modern technology has made trivial to solve in a standard way, they're often reluctant to throw out the proprietary solution they've developed, and just use standard off-the-shelf protocols that would enable third party developers to plug into their systems and make their expensive products superfluous.
You may think it's a great idea for your car to simply be running a web server on a TCP/IP network that you can just connect to with bluetooth or wifi, but that's a terrifying concept to companies that are trying to lock you into systems they developed years or decades ago...
Even after they finally bit the bullet and decided to use WebKit in the TomTom device to implement the user interface, they still insisted on plugging their silly RPC protocol into the web browser via an old school NSAPI plug-in adaptor to talk to their fancy proprietary protocol library, instead of just talking http between everything. They wouldn't listen to reason, or technological arguments, because the political arguments had already been made and the decisions had been set in stone that they were going to use their proprietary technology for a long time. And no way were they ever going to offer third party developers access to their silly RPC protocol, which they were clinging to because they actually perceived open standards as a threat, not a blessing. People's jobs depended on it!
They knew and gave lip service to the idea that they had to think "out of the box" to survive against the onslaught of google and cheap android phones, but hell if they were going to go through that particular door, like trying to force a cat out a window it refuses to go through.
You may recall how TomTom for WinCE used to have an SDK that let you hook into the TomTom Navigator app in a few unsatisfying ways -- it was extremely sub-optimal: you would write files out into a directory and stick your thumb up your ass until the app noticed the file, read it, did something, and wrote another file with a reply, that you had to keep polling for. Instead of developing that into a real API for integrating with TomTom Navigator, they just kind of took it out back and suffocated it with a pillow -- a step in the wrong direction if you ask me! It would have been so straightforward to simply expose an http service, but that dog won't hunt in a company like that.
I'd heard of the TomTom SDK, but never even explored the option for quite a few of the concerns you managed to affirm from first hand experience.
I very much agree on your point about platform lock-in worries associated with automotive OEMs. Though Google via the Open Auto Alliance is making headway to push forward a more standard platform, it seems likely that OEMs will still wage territorial wars to lock app devs out of the data garden. The OEM caution is not without merit, they're the most likely legal target if someone drives into a wall because of poor usability from some buggy 3rd party Music app on their platform.
Given all of these potential in-car platform issues, how do you feel about creating an interface on the smartphone that makes accessing apps safer?
I feel its good concept and addresses current problem of UI in cars. Still, its not a permanent solution as User has to physically touch the screen which is still a distraction while driving. I feel permanent solution would be through a Voice activated controls where you have 360 Degree freedom for yourself. Till then this idea is very much usable for majority of People,Leaving few corner cases as you mentioned("People have a wide range of hand sizes, motor control skills, even number of fingers").
I don't really understand why a car's sound system or climate control can't be effectively controlled with physical knobs and buttons. This touchscreen fad will go away, just like the digital speedometer fad of old.
It's supposed to reduce costs and material use (if everything uses touch screens, they're cheaper to mass produce) and the dashboard design will be simplified, like how digital electrical systems replaced the miles of wires in analog-wired cars.
But yeah, right now it's stupid - I can't even use my phone well outside when it's a bit cold and my fingers are dry...
A touch (on a physical object) can make a change in half a second, requires no thinking, and can be done without looking.
Voice control (even if it works reliably, which it does not today) takes several seconds and requires thought in composition and execution.
Touch on a flat screen may be quick, but requires looking. But it's probably not quick, because you'll probably have to change modes, which will require thinking and several seconds of manipulation.
Just put some damn hard buttons on the dash. I know why manufacturers love the screens: they require no tooling, provide lots of real estate for all the doohickeys you want to cram in, and they're hip (Ooh! Like and iPad(TM)!). But they don't belong in cars. Even good ones like Tesla's are bad.
Years ago I said I wouldn't buy a car with screen controls. Now it looks like I just won't be able to buy a new car.
I repeat: Just put some damn hard buttons on the dash.
You're making a bunch of good points. All of them show that there's still a lot to try, learn and discover in this space.
And keep in mind: I was thinking of this as being an additional mode to whatever the more "standard" interface would be. You could always have your standard AC controls (on a tactile-feedback-less screen though) but this quick input mode could be invoked when you know what you're doing.
I don't agree that he is making good points. He is making points that are of the current mindset. I can see myself and others like me easily transition to that type of interaction until the next generation of customers, aka, kids grows up knowing and feeling comfortable interacting in that or a similarly smart manner.
I do think that the gestures could be broken out into additional interactions, e.g., gestures towards the top do a different thing than the same gestures at the bottom or even have four quadrants that initiate different interactions.
I would think that it would require a type of introduction and practice mode that progressively teaches how to interact and also allows a demo mode like display that can be engaged for those who are not versed in that type of interaction. If not being intuitive was really an impediment there would be no keyboard shortcuts. If you can't take a second to memorize some keyboard shortcuts that will quadruple your effectiveness and efficiency then you cannot be helped.
This is an interesting (and beautiful) implementation, good to see another fighting the good fight against distracted driving. I suspect though that you may not have tested this during actual driving situations. The amount of concentration required to maintain 2 fingers or more on a flat, glassy surface while driving is immense and would result in mistaken inputs in all but the most placid of driving conditions.
Pie menus frame this kind of interaction as pop-up menus, which provide a "self revealing gestural user interface". The menu pops up and leads you through the possible selections. That feedback trains you to rehearse the gestures. Soon you begin to make the gesture without looking at the menu, then wait for the menu feedback to confirm you're doing the right thing. Finally you gain enough skill and confidence through "muscle memory" to make the gestures quickly without even looking at the screen or requiring any feedback.
It's best if the menus can provide real time in-world feedback, like applying the effect of the interaction immediately as the menu is tracking. That makes it feel more like immersive "direct manipulation" than indirect "menu selection". It's important that pie menus support "reselection", which makes them much more forgiving and differentiates them from traditional blind gesture recognition, so you at any time during tracking you can change the selection to any item you desire.
Pie menus completely saturate the entire possible gesture space with usable and accessible commands: there is no such thing as a syntax error, and you can always correct any gesture to select what you want, no matter how bad it started out, or cancel the menu, by moving around to the desired item or back to the center to cancel.
Handwriting and gesture recognition does not have this property, and it can be quite frustrating because you can't correct or cancel mistakes, and dangerous because mistakes can be misinterpreted as the wrong command. Most gestures are syntax errors. Blind gesture recognition doesn't have a good way to prompt and train you with the possible gestures, which only cover a tiny fraction of the possible gesture space. All the rest of the space of possible gestures is wasted, and interpreted as a syntax error (or worse, misinterpreted as the wrong gesture), instead of enabling the user to correct mistakes and reselect different gestures.
Even "fuzzy matching" of gestures trades off gestural precision with making it even harder to cancel or correct a gesture, without accidentally being misinterpreted as the wrong gesture. That's not the kind of an interface you would want to use in a mission critical application such as a car or airplane.
Another way to reframe the gestural, self revealing and reselectable qualities of pie menus is as navigation through a map, as opposed to climbing up a hierarchical menu tree. Instead of laboriously climbing up a tree of submenus, you simply navigate around a map of "sibmenus" -- sibling menus that you can easily move back and forth between by moving in opposite directions.
This demo of an iPhone app I developed called "iLoci" demonstrates the idea, enabling users to create their own memorable maps of "locations" instead of "menus", which they can edit by dragging around, that are related to each other by two-way reversible links. It exploits the "Method of Loci," an ancient memorization technique from the time before iPhones when people had to use their own brains to remember things, in order to leverage your spatial navigation memory and make it easy to learn your way around. http://vimeo.com/2419009
https://en.wikipedia.org/wiki/Method_of_loci
"The Method of loci (plural of Latin locus for place or location), also called the memory palace, is a mnemonic device introduced in ancient Roman and Greek rhetorical treatises (in the anonymous Rhetorica ad Herennium, Cicero's De Oratore, and Quintilian's Institutio Oratoria). In basic terms, it is a method of memory enhancement which uses visualization to organize and recall information. Many memory contest champions claim to use this technique to recall faces, digits, and lists of words. These champion...
Some people have already mentioned that you may want more than 8 controls. Perhaps a solution for this would be to have a max of 8 controls but the user could define which ones are mapped. Perhaps this interface can be a layer on top of a more traditional interface. That way the driver could customize the controls to fit his needs.
162 comments
[ 5.4 ms ] story [ 237 ms ] threadI don't know how to invoke it on the demo/prototype though.
Great ideas though. It's great to see someone working on this! Nice job.
I wish more car touch-screens were capacitive.
[0]- http://www.bmw.com/com/en/newvehicles/x/x5/2013/showroom/dri...
Could there be a few options to enter other modes, though? What I mean is that right now there are only 8 adjustments possible. That's better than the 4 I thought it was limited to, but still nowhere near enough for modern cars.
I would estimate that on a modern car you need to be able to adjust something like 100 parameters. Only a few of them frequently, but definitely more than 8 in total.
One of the thoughts I had was that using a (say 90 degree?) twist motion along with a number of fingers could bring up sub-menus. That way if you want to adjust other radio parameters like fade, balance, equalizer, etc you would put two fingers on (for volume) and twist.
Maybe you'd arrange the different controls in a loop so that every twist indexes to the next and there's a ~10 second timeout before you head back to the default. So if you want to adjust all the "volume" parameters you put two fingers and twist right once for balance and adjust, twist right again to get fade, twist right again to get highs, etc. Make a mistake? Twist left to go back and re-adjust.
If you want to get out of one of the sub-menus you're in prior to the ~10 second time-out for default, another gesture? Maybe a double tap? I don't know on that one, really.
Not the author, but the way I see it this is specifically an interface for adjusting things eyes-free when driving. In an actual implementation, there would be some sort of physical switch between the in-drive control (demoed here) and a high-density "at rest" operation for passengers or more specific tasks (e.g. configuring the GPS).
> One of the thoughts I had was that using a (say 90 degree?) twist motion along with a number of fingers could bring up sub-menus.
The problem with this is there'd likely be a rotational component to movement, and a movement component to rotations, discriminating becomes harder and the chances of false positive (and thus frustration) increase. Only using a single axis is actually a smart move as far as I'm concerned.
I agree that you might end up rotating your fingers SOME while moving them up or down. But I'm not sure that I agree that you would move them 90 degrees completely unaware. In order to move your thumb and index finger from vertical aligned to horizontally aligned you're going to require some (maybe a lot) of wrist motion. It's hard to make that wrist motion happen accidentally.
You might end up setting it to accept a range between 70 and 120 degrees to qualify for a "go to the next setting" motion and you can provide some kind of feedback (vibratory or audible or both) that you have indeed twisted far enough to trigger the change.
Gordon Kurtenbach and Bill Buxton did an experiment that varied the number of menu items and measured the selection time and error rate. They expected the selection time to be monotonically increasing as they added more items. To their surprise, that was almost true, except for the transition from seven to eight items. It was quicker to select from an eight item menu, then from a seven item menu!
That, I believe, was because of the effect of the cognitive bottleneck of associating the items with which direction to move, not the physical difficulty of moving in those directions. While the slices of the seven item menus were wider and had more area than the slices of the eight item menus, and Fitts' Law would predict that seven item menus were faster than eight item menus, Fitts' Law does not take into account the time it takes for users to map their intended command to the direction of that command, and how the mental framework in which users remember and relate the directions to each other affects selection time.
Eight item menus have all of their items in well known directions, each of which is associated with a very familiar concept, which come in nice pairs, and the pairs come in convenient orthogonal groups, like up/down, left/right, vertical/horizontal/diagonal, like compass directions.
Twelve item menus like a clock face also work well, especially for circular sets of items like hours, months, zodiac signs, etc. But the effect going from 11 to 12 is not as dramatic as from 7 to 8. But still the 12 item menu has a nice familiar aesthetically pleasing cognitive framework (including opposite and orthogonal pairs, first tier vertical/horizontal axes and second tier in-betweens) that you can often exploit, depending on the content.
Three item menus are still slightly faster than four item menus, because the effect of the proportional difference in target area overwhelms the difference in the number of items, and two of three triangular directions are well known concepts that are pretty easy to remember, compared to six of seven directions.
An eight item menu optimally exploits that effect, while a seven item is unfortunately sub-optimal with mostly difficult to remember directions. There's no word or concept in the English language for six out of seven of those directions -- they're all just kind of slanted differently, similar to but not quite like the well known compass directions, and there are no nice symmetries, opposite or orthogonal groupings to exploit, by arranging complementary items in opposite directions, independent pairs along orthogonal axes, etc.
So if you're designing a pie menu with seven items (or even eleven), it's better to just throw another item in to bring it up to eight (or twelve)! I gave an example in the DDJ article of a "seven days a week" menu with an additional "today" item thrown in at the bottom to bring it up to eight, with the weekend and today in the lower part of the menu, and Wednesday at the top. See, I don't even have to link to a picture or enumerate every item for you to easily visualize and remember it!
Yes!
That said that is neat implementation and it very elegant but I would like to have that in the living room rather than the car.
A possible evolution I guess would be to have different screen you would flick through that would group 2 or 3 command together with a big visual cue on the background.
You exhibit the symptoms of somebody just introduced to multitouch and gesture recognition and trying to "explore the possibilities." The reality of gesture control is that they make for horrible user experiences. You are encouraged to try it on your own if you aren't convinced. Adobe, Apple, and a number of other companies did a lot of user study into this.
Double tap, twists, etc. All those gestures suck and fail for different reasons sadly.
I got interested in multi-touch something like 6 years ago when I saw this: http://vimeo.com/6712657 So I don't think that I'm someone "just introduced to multitouch" at all. But thanks for trying to subtly call me a naive idiot!
So far the best people are able to do is two finger scrolling and pinch zoom. It's kinda sad.
No you are "new to multitouch" in the sense that you haven't tried to implement multitouch yourself, conducted a user study on multitouch learning curves and intuitiveness, or used a real life application exhibiting the multitouch you claim to understand for a prolonged length of time.
Seeing one video qualifies you perfectly for being "just introduced to multitouch."
I drive a 1998 car and can operate every single control without looking at the console, except audio tone controls and choosing a specific FM station not in my presets. Granted, navigation is missing since it's an older car -- better voice controls on the phone could make up the gap.
Feeling how many fingers are touching the screen is also easy with the normal sense of touch. Add in a little hysteresis to handle bumps (you could even integrate info from the accelerometer) and I bet that even bumpy conditions will be significantly improved with this system.
Same reason why my phone has a physical keyboard.
I generally agree. I just bought a car and one of the reasons that I skipped the nav system is that I didn't want a touch screen for things that work better as knobs.
That being said there is a balance to strike in today's smart-device world. Having a touch screen can save space and reduce control clutter when trying to cram bluetooth, gps, hd radio, playlist, and other controls onto the dash. They allow the car to only show me what I need when I need it.
Looking at most touch screen dashes, such as those in the article, show that they fail at this, of course.
Using different numbers of fingers to control different things is interesting- how would people with missing fingers use this?
What if you put one finger down to activate the interface, and tap with another finger (while still holding the first one on the display) to cycle through the menu?
This requires at least two fingers, but would allow for control of more settings, and would’t require you to remember how far apart your fingers need to be to get access to a control. You could have the car dictate the currently selected / active setting as you cycled through.
The problem with this is people tend to add a lot of extras. I wrote[0] that the internet radio thing cannot be done in the Tesla without a touchscreen.
FWIW, this idea is slightly better than the one in my Tesla screen complaint, but still not any better.
[0]http://blog.chewxy.com/2014/02/04/the-switchless-switch/
How about selecting what you want to control (Temp, Channel, Air etc) on the first pass followed by adjustment of the selected control on the second?
But how would you do this selection, and know unambiguously that you've selected the right control without looking at the display?
What I find hard about in-car UIs are things like entering and editing navigation directions or selecting music to listen to given a 32GB USB stick full of songs.
There's always some horribly designed onscreen keyboard or "iDrive" type knob controller, both of which are quite horrible experiences. To make matters worse, the UI is laggy and unresponsive, and in an effort to make things "easier" onscreen controls frequently dim & disable when not actionable or when the car is moving vs. stationary.
1) Why was a single touch interaction not demonstrated? Each control requires 2+ fingers. Maybe that can bring up a help menu or other information?
2) There is no demonstration of boolean switches and button controls. Things such as turning on a defroster, or enabling pairing with a bluetooth device. I'd be curious to see how those would play into the ux flow.
I really like the idea of an always-on GPS that can intercept commands for other stuff (radio, HVAC, etc) and forward them to the appropriate app/microcontroller/hardware.
For now the best car interface I've used was voice recognition. It usually works great, because it's not a general purpose text dictation. It only needs to recognise less than 100 words and it's not that hard to do. For example I'm not a native English speaker, and I can't get google voice input to capture most of the sentences correctly - it's simply not usable for me at the moment. But I'm quite happy to use voice commands in a Prius - I don't think I've ever used more than 4 or 5 of them, always perfectly recognised.
Edit: Another thought... the UI for many funtions needs to provide visual feed back... say for what system I am selecting or what station I am tuning. The text is quite small which would cause even more of a distraction to try to read. Yes I can begin the selection process without a visual cue, but I still need to look to see what I'm selecting. Not sure I see a large improvement in driver distraction there.
Slick for users like us? Definitely. Intuitive for the average Joe out there? I don't think it would fly. Too much to remember and too much dexterity required. My Mom could never use this. At best I see this as an alternate UI the driver could select over something more conventional when they feel comfortable with it.
Kudos for thinking outside the box though, there are definite UI gems here that could be leveraged.
Intuition isn't important. The designer mentioned that this is a trade off, and is intended for regular users who have muscle memory of their controls. You can always add a tutorial or "help" mode by pressing a button in the bottom right or whatever, anyways.
Do they get 20% less things they can control?
I ask in all seriousness because I know several people who have lost one or more fingers.
It seems to me that something like Google or Apple voice commands are what is really needed, assuming on wheel controls don't suffice.
The latest iteration is virtually perfected. Rocker buttons on top, and with a known number of clicks I can control 80% of all the main functions without taking my eyes off of the road at all. For the other 20%, it takes a quick glance to see the state of the screen and then I know "okay, now just two clicks down and I'm there".
The upcoming version (currently on the only the latest model to be released) has a small circular touchpad on-top of the wheel so you can draw letters without having to twist the knob. Useful for entering Postcodes into Nav, for instance.
I recall that both Audi and Mercedes have since copied the iDrive paradigm.
The iPhone-style interaction -- a flat screen of glass where you manipulate "controls" with touch sensors -- has one major advantage: it allows an INFINITE number of different controls. (Every app is its own set of controls.) But to do this it relies heavily on continual interactive feedback with your vision.
When driving, you can't do that continual feedback. Perhaps someone clever can figure out how to do interactive feedback with audio (you can use that sense somewhat without taking your attention from the road). But otherwise, you can really only manage as many motions as your fingers can "memorize" (muscle memory mostly). If you're doing that, why not stick with switches and knobs, where your tactile sense can help out.
My Model S has these same controls embedded in the steering wheel. Volume, music control, and climate controls are all there.
That's actually where I find I spend most of my time interacting with the car. I usually only use the touchscreen for more complex things like manipulating the map. For complex inputs like choosing a song or entering my destination, I use the voice controls.
I don't think the solution is one interface. It's multiple interfaces that function best in certain contexts, with some overlap. Look at Google Glass for an example of this.
Teach them in schools? Remember kids when confronted with a blank screen 3 fingers is audio source!
http://gaspuol.files.wordpress.com/2012/11/2013-hyundai-velo...
http://www.extremetech.com/wp-content/uploads/2011/09/MY12_F...
Granted, this UI looks great, although has a few interaction oversights: what if I lost a couple fingers in an industrial accident and only have 3?
However, it's not the solution yet, in my opinion.
First, all the methods of interaction are invisible. There are no cues to tell the user what two fingers vs. four fingers will do, or what the difference is between a close-fingered gesture and a wide gesture. At best, you'd have to standardize the meanings across all cars (something that rarely happens in autos, even today there's no standard for where the wiper controls go, for instance), and even then, people would have to memorize those invisible gestures before they could use their car.
Secondly, accidental inputs would happen with some frequency. People have a wide range of hand sizes, motor control skills, even number of fingers. Suppose I've got two of my fingers in a cast? I can use an existing car's dashboard just fine with that temporary impairment, but not this.
Finally, it's addressing a problem that already has a solution—physical, dedicated input devices. Humans have spatial memory, we learn where things are, and learn to reach for them without looking or even thinking. Your muscle memory tells you how to reach the wiper stalk or the gear shift, which is why it's sometimes disconcerting to get into a car with the gears on the steering column if you normally shift in the center console. We need in-car systems that let the physical feedback, affordances, and reliable location of buttons, knobs, and switches interact with display systems that are designed for the attention a driver requires. You should be able to keep your eyes on the road and still adjust the air conditioning—something the designer in this video recognizes—but that shouldn't require learning invisible gestures that are prone to user error.
Beyond spatial memory what's good about this is that you don't have to look anywhere while still adjusting (some speech feedback about what mode you've selected would be good). The lack of sight is the most compelling aspect -- but I can guarantee you my parents would likely never remember the mappings and always have to look at the legend. Even then they'll probably be confused.
Look at a traditional dashboard. The air conditioning knob is a different shape and size from the stereo's volume knob, and located in a different place (ideally). Those things make it easy for the the driver to distinguish one from the other, and find them while keeping their eyes on the road.
There are other cues that usually exist to help as well, such as a blue-to-red graphic around the temperature control, the volume control being next to other audio features like the radio display, etc.
In an environment where the user has to keep a 3,500 lb steel machine safely controlled while traveling 100 feet per second, the rules of usability become incredibly important.
You've nailed it sir. Shape coding[1]. If it's good enough for F-14 fighter pilots, it's good enough for my Toyota Camry.
I echo the sentiment of others. The OPs heart is in the right place here. Mimicking sight free displays on a touch screen is laughably terrible. Take this picture [2] from OPs post. Why are the media buttons ellipsoids? They're just smaller targets to hit compared to squares!
In academia, we've been researching this problem, eyes-free mobile interaction, for a very long time. Particularly in the context of interaction while walking.
There are numerous approaches (e.g., Flower menus [3], utilizing pressure [4], utilizing motion [5], the list goes on). The big takeaway is that it seems like more successful systems result from the multimodal approach.
OP, you've done some good work here, but consider incorporating other sensors. Voice? A camera for in-air gesture recognition? A biometric sensor that gives reasonable defaults automatically based on learned use of the driver? Touch is only one part of the problem if we move away from shape coding, and we will very likely need to utilize multimodal input if we hope for success...
...or, you know, we could just stick to that whole buttons and knobs thing.
-----
[1] http://en.wikipedia.org/wiki/Alphonse_Chapanis
[2] http://matthaeuskrenn.com/new-car-ui/images/statusquo.jpg
[3] http://www.youtube.com/watch?v=QRSASiEBw5k
[4] http://www.mobilevce.com/newsite/sites/default/files/infosto...
[5] http://katrinwolf.info/wp-content/uploads/2013/05/MobileHCI2...
In this case, I'm simply trying to make a point that touch screens can be more control panels with buttons and sliders on them.
SimCity has a tool pallet of various building and editing tools, which are related to each other in various ways, and have different costs, functions and properties. I tried to arrange them into a set of submenus that made those relationships and differences more obvious and easier to learn. I also arranged the static tool palette on the screen to reflect the layout of the pie menus and submenus, to serve as a kind of map to the pie menus (which foreshadowed the spatial map design I explored with Method of Loci, described in another posting).
Here is a screen dump of the multi player version of SimCity running on a Sun workstation on TCL/Tk/X11: http://www.donhopkins.com/home/catalog/simcity/SimCity-Sun.g...
The top level menu includes: magic marker and eraser (for drawing on the map), roads and rails, power lines and a bulldozer, and submenus for common zones and bigger buildings. The zone submenu includes residential, commercial and industrial zones, police and fire stations, and a query tool. The building submenu includes a stadium, park and seaport, coal and nuclear power plants, and an airport.
As you can see, the size of the icon reflects the cost of the tool. The borders of the icons are color coded to reflect the cursor you see on the map that shows where the tool will affect. (That was useful for multi player mode, so you could see which tools the other players had selected, so the tool palette served as a legend for the cursors on the screen.) And the tool palette looks kind of like a totem pole, which is vertically asymmetrical, differentiating different kinds of tools and making them look unique, and horizontally symmetrical, reflecting pairs of similar tools and making it look aesthetically pleasing.
If all the tools were the same sized squares arranged in a regular grid, it would be much harder to differentiate them and quickly select the one you want. Instead, they're arranged more like a bouquet of unique flowers that each have their own special features that make them easily recognizable and memorable, while telling you something about themselves.
Designing user interfaces this way is a artistic balancing act, highly dependent on the set of commands, and requiring a lot of iteration, testing and measurement, as well as willingness to explore and experiment with many different alternatives. It is not easy, and there is never a best solution, or even always a good one. It's not something you can expect end-users to be able to do with their own custom built menus. But it's worth the effort to try, and I think it's also worth the effort to train designers and even users to promote a literacy in interface design.
One idea I've had is to develop a game called "PieCraft", that has user-editable pie menus that are first-order in-game player craftable artifacts. World of Warcraft and its rich ecosystem of user interface extensions supports a wide range of player customizable user interfaces, because there are so many different commands, spells, buffs, weapons, armor, etc, and each role of each character of each player requires a totally different set of them in different situations. The more casual MMPORG "Glitch" had bags that users could arrange in hierarchies and put their items inside, which were surfaced on the user interface as buttons they could press and objects they could drag in and out of the world and other bags. How well you arrange your items in WoW and Glitch had a significant effect on gameplay.
PieCraft could take this further, to train players in user interface design literacy (much like "Code Hero" aims to train players in computer programming). You could find empty or pre-populated pie menus in the world, pick them up and make them your own, and edit them by moving i...
So, since I have the opportunity to fanboy out a second: thank you for your contributions, both to gaming and human-computer interaction.
I'm sure he doesn't remember it, but those 15 minutes were extremely beneficial to me. He's a super humble, brilliant scientist and designer. I have the utmost respect for him.
I'll bet he remembers, and tells people about your look and feel that he saw and felt!
The UI can provide user feedback with static electricity, and the current could increase with the speed of the car.
As for the final point, a ten key resting within hands reach would work fine. You could memorize the number sequence for different commands, and get audio feedback. Talk about a proven UI, basically the telephone. Baby boomers ought to be comfortable with it.
There are both good and bad reasons for this. But the important point is that automotive interfaces are designed to be built into cars at the factory, and used for many years without requiring any updates. Even if updates are possible, most people don't get them. So it's a completely different mindset and approach and time scale than people from the Silicon Valley dot-com startup industry have.
Another factor is that the devices built into cars have a very long design and production lead time, and by the time the car comes out, the built in hardware and software is already quite obsolete compared to the smartphone the driver probably owns.
Factoring the problem out of the car to run in a smartphone or tablet itself also has its own frustrating problems, because when you're designing an automotive computer system, you can't predict what kind of technology and standards will be available or popular by the time it ships.
It's a very difficult problem space, and the stakes are extremely high, not just financially, but also because cars are weapons of mass destruction that kill more people and destroy more property than all terrorists combined could possibly dream of.
Would you mind to elaborate on this statement?
To me it seems like moving control over some functions within the car, for example music or calling, might make more sense to do on the phone specifically because of the phone's fast upgrade cycle and tie-ins to one's personal information cloud.
For example, TomTom had a big emotional investment in some 1990's-style proprietary remote procedure call protocol that one of the automotive companies they bought had developed. It had its own interface definition language and rpc stub compiler, which seemed revolutionary in 1988 when some clever student came up with the same idea to make it easier to write C wrappers for SunRPC protocols so you didn't have to write them by hand.
More rational heads were pushing to just use modern off-the-shelf technologies like http/rest/json/etc, but the jobs of some people in some department depended on them being perceived as having been doing productive work on their own proprietary message bus for the past three years or so, and that was what they delivered, so that was what they were damn well going to use.
Smartphones are a huge threat to companies that make their own little boxes that are hard to convince people who have smartphones to buy, so they weren't exactly enthusiastic about embracing and supporting smartphones and tablets.
When a company that operates at glacial automotive development speeds has been working on their own proprietary solution to a problem that modern technology has made trivial to solve in a standard way, they're often reluctant to throw out the proprietary solution they've developed, and just use standard off-the-shelf protocols that would enable third party developers to plug into their systems and make their expensive products superfluous.
You may think it's a great idea for your car to simply be running a web server on a TCP/IP network that you can just connect to with bluetooth or wifi, but that's a terrifying concept to companies that are trying to lock you into systems they developed years or decades ago...
Even after they finally bit the bullet and decided to use WebKit in the TomTom device to implement the user interface, they still insisted on plugging their silly RPC protocol into the web browser via an old school NSAPI plug-in adaptor to talk to their fancy proprietary protocol library, instead of just talking http between everything. They wouldn't listen to reason, or technological arguments, because the political arguments had already been made and the decisions had been set in stone that they were going to use their proprietary technology for a long time. And no way were they ever going to offer third party developers access to their silly RPC protocol, which they were clinging to because they actually perceived open standards as a threat, not a blessing. People's jobs depended on it!
They knew and gave lip service to the idea that they had to think "out of the box" to survive against the onslaught of google and cheap android phones, but hell if they were going to go through that particular door, like trying to force a cat out a window it refuses to go through.
You may recall how TomTom for WinCE used to have an SDK that let you hook into the TomTom Navigator app in a few unsatisfying ways -- it was extremely sub-optimal: you would write files out into a directory and stick your thumb up your ass until the app noticed the file, read it, did something, and wrote another file with a reply, that you had to keep polling for. Instead of developing that into a real API for integrating with TomTom Navigator, they just kind of took it out back and suffocated it with a pillow -- a step in the wrong direction if you ask me! It would have been so straightforward to simply expose an http service, but that dog won't hunt in a company like that.
I very much agree on your point about platform lock-in worries associated with automotive OEMs. Though Google via the Open Auto Alliance is making headway to push forward a more standard platform, it seems likely that OEMs will still wage territorial wars to lock app devs out of the data garden. The OEM caution is not without merit, they're the most likely legal target if someone drives into a wall because of poor usability from some buggy 3rd party Music app on their platform.
Given all of these potential in-car platform issues, how do you feel about creating an interface on the smartphone that makes accessing apps safer?
But yeah, right now it's stupid - I can't even use my phone well outside when it's a bit cold and my fingers are dry...
Voice control (even if it works reliably, which it does not today) takes several seconds and requires thought in composition and execution.
Touch on a flat screen may be quick, but requires looking. But it's probably not quick, because you'll probably have to change modes, which will require thinking and several seconds of manipulation.
Just put some damn hard buttons on the dash. I know why manufacturers love the screens: they require no tooling, provide lots of real estate for all the doohickeys you want to cram in, and they're hip (Ooh! Like and iPad(TM)!). But they don't belong in cars. Even good ones like Tesla's are bad.
Years ago I said I wouldn't buy a car with screen controls. Now it looks like I just won't be able to buy a new car.
I repeat: Just put some damn hard buttons on the dash.
And keep in mind: I was thinking of this as being an additional mode to whatever the more "standard" interface would be. You could always have your standard AC controls (on a tactile-feedback-less screen though) but this quick input mode could be invoked when you know what you're doing.
I do think that the gestures could be broken out into additional interactions, e.g., gestures towards the top do a different thing than the same gestures at the bottom or even have four quadrants that initiate different interactions.
I would think that it would require a type of introduction and practice mode that progressively teaches how to interact and also allows a demo mode like display that can be engaged for those who are not versed in that type of interaction. If not being intuitive was really an impediment there would be no keyboard shortcuts. If you can't take a second to memorize some keyboard shortcuts that will quadruple your effectiveness and efficiency then you cannot be helped.
One suggestion is to look at the Cognitive, Visual, and Manual Load framework common in automotive UI design. http://www.esurance.com/safety/3-types-of-distracted-driving
This UI concept has a low visual, but very high manual and cognitive load.
It's best if the menus can provide real time in-world feedback, like applying the effect of the interaction immediately as the menu is tracking. That makes it feel more like immersive "direct manipulation" than indirect "menu selection". It's important that pie menus support "reselection", which makes them much more forgiving and differentiates them from traditional blind gesture recognition, so you at any time during tracking you can change the selection to any item you desire.
Pie menus completely saturate the entire possible gesture space with usable and accessible commands: there is no such thing as a syntax error, and you can always correct any gesture to select what you want, no matter how bad it started out, or cancel the menu, by moving around to the desired item or back to the center to cancel.
Handwriting and gesture recognition does not have this property, and it can be quite frustrating because you can't correct or cancel mistakes, and dangerous because mistakes can be misinterpreted as the wrong command. Most gestures are syntax errors. Blind gesture recognition doesn't have a good way to prompt and train you with the possible gestures, which only cover a tiny fraction of the possible gesture space. All the rest of the space of possible gestures is wasted, and interpreted as a syntax error (or worse, misinterpreted as the wrong gesture), instead of enabling the user to correct mistakes and reselect different gestures.
Even "fuzzy matching" of gestures trades off gestural precision with making it even harder to cancel or correct a gesture, without accidentally being misinterpreted as the wrong gesture. That's not the kind of an interface you would want to use in a mission critical application such as a car or airplane.
Another way to reframe the gestural, self revealing and reselectable qualities of pie menus is as navigation through a map, as opposed to climbing up a hierarchical menu tree. Instead of laboriously climbing up a tree of submenus, you simply navigate around a map of "sibmenus" -- sibling menus that you can easily move back and forth between by moving in opposite directions.
This demo of an iPhone app I developed called "iLoci" demonstrates the idea, enabling users to create their own memorable maps of "locations" instead of "menus", which they can edit by dragging around, that are related to each other by two-way reversible links. It exploits the "Method of Loci," an ancient memorization technique from the time before iPhones when people had to use their own brains to remember things, in order to leverage your spatial navigation memory and make it easy to learn your way around. http://vimeo.com/2419009
https://en.wikipedia.org/wiki/Method_of_loci "The Method of loci (plural of Latin locus for place or location), also called the memory palace, is a mnemonic device introduced in ancient Roman and Greek rhetorical treatises (in the anonymous Rhetorica ad Herennium, Cicero's De Oratore, and Quintilian's Institutio Oratoria). In basic terms, it is a method of memory enhancement which uses visualization to organize and recall information. Many memory contest champions claim to use this technique to recall faces, digits, and lists of words. These champion...
http://en.wikipedia.org/wiki/Windows_Embedded_Automotive