Fingertip skin stretch feedback (shear feedback) for communicating direction cues
Single Contactor Skin Stretch Feedback
In this research, we are applying skin stretch feedback (also referred to as shear feedback) to the fingertip and using these stimuli to communicate direction cues. This concept is portrayed in the below image, where the skin is subtly stretched by a plate or other skin contactor that is moved in the plane of the skin. This functionality has been built into portable devices which can in turn be embedded within a multitude of other products (e.g., a cell phone, steering wheel, or game controller).
 |  |  |
Multi-Contactor Skin Stretch Feedback
We have also shown that use of multiple moving skin stretch contactors can be used to communicate to a user. In the past we have used multiple skin stretch contactors in gaming controllers, where each contactor was located in the middle of a thumbstick on a game controller. However, more recently we have discovered that the use of multiple contactors placed opposite each other (back-to-back) can be used to create torque-like (or rotary) sensations. That is, when moving the back-to-back contactors in the same direction this produces a translational cue, but when moved in opposite directions, this produces a rotational cue as portrayed in the images below. This same feedback mechanism can also be built into the handle of a device to give translational and rotational cues to a person’s hand. This approach was used in developing our lab’s Motion-based Game Controllers, which are described below. For more information on this research, see our publications listed towards the bottom of this page or the Motion-based Game Controllers described below.
 |  |  |
Use of two actuated skin stretch feedback contactors placed back to back to communicate translational and rotational cues to a user.
Applications
Shear feedback could be used to provide direction cues for navigation or attention cuing. In particular, this technology could be used to interface with a GPS navigation system in a car as a substitute or complement to current spoken directions (also providing a non-visual interface for those with hearing impairments), guide those with vision impairments, or augment handheld electronics by providing tactile cues that could, for example, provide a pulse each time a song on a music player’s song list scrolls past. Images and a video clip portraying these applications are shown below.
 |  |  |
Shear feedback placed on a steering wheel rim for providing navigation cues
 |  |  |
| Shear feedback to guide those with vision impairments | Shear feedback for enhancing human interfaces in handheld electronics | A multi-modal handheld device with embedded shear feedback |
Movie showing the concept and applications of fingertip skin stretch feedback (also referred to as shear feedback). The device has applications for use in providing in-car navigation cues to the fingertips, guidance for the blind, or for enhancing handheld devices. (Link to YouTube site showing the above movie). A version of this video without captions can be viewed here.
Gaming
Our lab has designed multiple generations of game controllers that incorporate embedded skin stretch feedback. In the past, we have developed a game controller with skin stretch feedback embedded into the center of each thumbstick of a conventional console-style game controller (see “Console-style game controller” below).
Motion-based Game Controller
| We have adapted our lab’s skin stretch feedback technology to be integrated into the handle of a device and can directly provide tactile feedback to a user’s hand. Our technology can be used to convey motion and force information. It does this through the motion of sliding contactor bars that are built into the handle of the device. Translational motions and forces can be portrayed along the length of the handle by moving the bars in unison in the corresponding direction; whereas moving opposing slider bars in opposite directions creates the feeling of the device’s handle wrenching within the user’s grasp. Because of its ability to recreate the skin sensations of actually holding an object, this type of feedback creates an engaging experience that brings gaming interactions to life, and we have hence named this feedback technology, “Reactive Grip” Skin Stretch Feedback. |  |
This technology is compatible with motion-based controller interfaces, such as the Nintendo Wii, Sony Move, or Microsoft Kinect, and could also make a formidable pairing with a head mounted display for unprecedented levels of immersion and realism in a virtual reality gaming environment. The current game controller prototype uses a Razer Hydra as its motion tracking system, but our technology is not tracker dependent. For updates on the commercialization of this technology, check out the Tactical Haptics facebook and company pages.
Movie showing the concept and some game scenarios for the use of “Reactive Grip” Skin Stretch Feedback. The device can portray the reaction force at the handle of a device and is shown with users in a shooting gallery, sword fighting, and swinging a medieval flail. It could also be used to portray the forces in fly flishing, guide the swing of a golfer, or even embedded in the handle of a walking cane to guide the blind. (Link to YouTube site showing the above movie).
Console-style game controller
We have also designed a game controller with skin stretch feedback embedded into the center of each thumbstick of a conventional console-style game controller. Hence, our game controller allows a gamer to receive tactile feedback through the same interface that he/she makes game inputs. Providing this new mode of touch feedback has the potential to enhance the gaming experience.
An initial prototype was constructed which allows two handle configurations to ensure that skin stretch direction cues could still be properly interpreted with a traditional angled handle game controller design. A fully integrated game controller with vibration and skin stretch feedback has now also been designed and prototyped.
Several game prototypes have been developed to highlight the use of skin stretch feedback in our fully integrated game controller. We have also developed feedback modes that enhance immersion and realism for gaming scenarios such as collision, recoil from a gun, the feeling of being pushed by ocean waves, or crawling prone in a first person shooter game.
 |  |
| Initial prototype shown with two handle configurations | Integrated game controller with vibrotactile feedback and skin stretch feedback embedded into the center of each thumbstick |
Movie showing the integrated game controller with vibration feedback and embedded skin stretch feedback (also referred to as shear feedback). The thumbsticks are still used for input just as on conventional game controllers, but skin stretch tactors, located in the middle of the thumbsticks, can also be used to provide tactile feedback to the user. The video provides an overview of the functionality of the controller, some haptic effects that can be used to enhance immersion within the gaming environment, and an example fishing game which utilizes skin stretch feedback to represent the feeling of being adrift and forces experienced while fighting a fish. (Link to YouTube site showing the above movie). A version of this video with fewer captions can be viewed here.
Link to Vimeo movie showing the design and initial characterization of a video game controller prototype with tactile skin stretch displays embedded into its thumb joysticks. With this prototype, we verify that people are capable of the same high accuracy rates for identifying direction cues as was observed previously. Modern game controllers typically have users angle their thumbs diagonally inwards to reach the joysticks, however, this may affect the cognition of skin stretch cues. No significant reduction in performance due to mental rotation of stimuli delivered in the angled thumb configuration was shown. Furthermore, the angled configuration appears to have ergonomic advantages.
Movie showing a game that was developed for the Imagine Cup competition. This game uses our lab’s game controller that has embedded skin stretch and vibration feedback. The game was designed to provide a more interactive experience for people with hearing impairments. The skin stretch and vibration feedback is programmed to correspond to the game’s music soundtrack, allowing user’s to “feel” the music. This video also documents a visit to show off the device and newly developed game and get feedback from a group of high school students who are hearing impaired. (Link to YouTube site showing the above movie).
Perceptual Studies
Initial perception studies have been conducted and show the promise of this type of feedback. These studies have found several factors to be important for communicating direction via skin stretch at the fingertip. Among the most important factors are the speed and displacement of skin stretch applied to the fingertip. The below images show the interface used in our initial perception experiments and the results of these experiments. Further experiments and their results can be found in the articles below.
 |  | |
| The tactor in contact with the finger. The thimble and thimble mount are shown translucent so that the finger and tactor can be seen. The thimble is free to move up and down, but constrained in the plane of tactor motion. | Experimental results, combining data from all stimulus directions. Subjects attempted to identify the direction of skin-stretch stimuli at a range of stimulus speeds and displacements. Identification accuracy rates and corresponding 95% confidence intervals are show in the grid squares. |
Forearm-mounted skin stretch feedback
While the fingertips excel at accurately discerning direction cues, it is often desirable to leave the fingers unencumbered for other tasks. This desire has motivated an effort to explore other locations to which direction cues can be applied. Studies of human sensitivity indicate that the hand or forearm may be a good location for skin stretch cues. Direction perception tests were conducted on 4 hand/arm locations to determine acuity and suitability of these locations. Based on this information, a portable device was designed that applies direction cues to the forearm. See below publication by [Caswell et al. 2012].
 |  |  |
| The four tested hand/arm locations | Direction identification accuracy on hand/forearm | Forearm-mounted skin stretch feedback prototype |
Further Details
Several US and International patents are currently pending on this technology. Please contact Dr. Provancher if you have interests in collaborating/partnering or in licensing this technology (wil_at_mech.utah.edu). Further information on this research can be found in
Related Publications & Media Coverage
Publications
- Guinan, AL, Hornbaker, NC, Montandon, MN, Doxon, AJ, and Provancher WR, “Back-to-Back Skin Stretch Feedback for Communicating Five Degree-of-Freedom Direction Cues,” In Proc. of the 2013 World Haptics Conference, Daejeon, Korea, April 14–17, 2013, pp. 6.
- Gwilliam, LT, Doxon, AJ, and Provancher WR, “Haptic Matching of Directional Force and Skin Stretch Feedback Cues,” In Proc. of the 2013 World Haptics Conference, Daejeon, Korea, April 14–17, 2013, pp. 6.
- Guinan AL, Caswell NA, Drews FA, and Provancher WR, “A Video Game Controller with Skin Stretch Haptic Feedback.” Proc. of the 2013 IEEE International Conference on Consumer Electronics, Las Vegas, NV, USA, January 10–13, 2013, pp. 2.
- Quek ZF, Schorr SB, Nisky I, Okamura AM, and Provancher WR, Sensory Augmentation of Stiffness using Fingerpad Skin Stretch. Proc. of the 2013 World Haptics Conference, Daejeon, Korea, April 14–17, 2013, pp. 6.
- Schorr SB, Quek ZF, Romano RY, Nisky I, Provancher WR, and Okamura AM. Sensory Substitution via Cutaneous Skin Stretch Feedback. IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany, May 6–10, 2013, pp. 6.
- Montandon, MN and Provancher WR, “A Smart Phone Peripheral with Bi-Manual Skin Stretch Haptic Feedback and User Input.” Proc. of the 2013 IEEE International Conference on Consumer Electronics, Las Vegas, NV, USA, January 10–13, 2013, pp. 2.
- Guinan, A.L., Montandon, M.N., Caswell, N.A., and Provancher, W.R., “Skin Stretch Feedback for Gaming Environments,” presented at the 2012 IEEE Symposium on Haptic Audio-Visual Environments and Games (HAVE 2012), Munich, Germany, Oct.8–9, 2012
- Guinan, A.L., Koslover, R.L., Caswell, N.A., and Provancher, W.R., “Bi-manual Skin Stretch Feedback Embedded within a Game Controller,” presented at the 2012 IEEE Haptics Symposium, Vancouver, BC, Canada, March 4–7, 2012, pp. 7.
- Gleeson, B.T. and Provancher, W.R., “Mental Rotation of Directional Tactile Stimuli,” presented at the IEEE Haptics Symposium, Vancouver, BC, Canada, March 4–7, 2012, pp. 6.
- Caswell, N.A., Yardley, R.T., Montandon, M.N., and Provancher, W.R., “Design of a Forearm-Mounted Directional Skin Stretch Device,” presented at the IEEE Haptics Symposium, Vancouver, BC, Canada, March 4–7, 2012, pp. 6.
- Koslover, R.L., Gleeson, B.T., de Bever, J.T., and Provancher, W. R., “Mobile Navigation using Haptic, Audio, and Visual Direction Cues with a Handheld Test Platform,” IEEE Transactions on Haptics, 5(1), Jan.-March 2012, pp. 33–38.
- Massimiliano Solazzi, Antonio Frisoli, William R. Provancher, Massimo Bergamasco, “Design of an SMA Actuated 2-DoF Tactile Device for Displaying Tangential Skin Displacement,” 2011 World Haptics Conference, Istanbul, Turkey, June 22–24, 2011.
- Medeiros-Ward, N., Cooper, J. M., Doxon, A. J., Strayer, D. L., and Provancher, W. R., “Bypassing the Bottleneck: The Advantage of Fingertip Shear Feedback for Navigational Cues,” Proc. of the Annual Meeting of the Human Factors and Ergonomics Society,Sept, 2010, pp. 5.
- Gleeson, B.T., Horschel, S.K., and Provancher, W. R., “Perception of Direction for Applied Tangential Skin Displacement: Effects of Speed, Displacement and Repetition,” IEEE Transactions on Haptics - Selected for Special Issue: World Haptics Spotlight , Vol. 3(3), Jul-Sept, 2010, pp. 12.
- Gleeson, B.T., Horschel, S.K., and Provancher, W. R., “Design of a Fingertip-Mounted Tactile Display with Tangential Skin Displacement Feedback,” IEEE Transactions on Haptics, Vol. 3(4), Oct-Dec, 2010, pp. 6 (in press).
- Gleeson, B.T., Stewart, C.A., and Provancher, W. R., “Improved Tactile Shear Feedback: Tactor Design and an Aperture-Based Restraint,” IEEE Transactions on Haptics, pp. 12 (in press).
- Provancher, W. R., Stewart, C. A., Montandon, M., and Greer, A., “Empowering Consumer Products with Embedded Shear Feedback,” In Proc. of the IEEE Haptics Symposium, Waltham, Mass., USA, March, 2010.
- Gleeson, B.T., Horschel, S.K., and Provancher, W. R., “Communication of Direction through Lateral Skin Stretch at the Fingertip,” In Proc. of the World Haptics Conference, Salt Lake City, Utah, USA, March, 2009, pp. 172–177. (Best Paper Award)
- Horschel, S.K., Gleeson, B.T., and Provancher, W. R., “Fingertip Shear Tactile Display,” World Haptics Conference, Salt Lake City, Utah, USA, March, 2009, pp. 611–612.
Media Coverage
- see (**news stories with video are marked with red label**)
- Skin Stretch Game Controller
- Thumb-stretching controller pitched to console makers: A video games controller which pulls and stretches the skin of its users’ thumbs has been unveiled in Canada, by British Broadcasting Corp., March 5, 2012
- University of Utah professor invents new video game controller, By Vince Horiuchi, The Salt Lake Tribune, Mar 05, 2012.
- This Video Game Controller Pushes Back on Your Thumbs [VIDEO], by Sam Laird and video broadcast by Lauren Gores, Mashable, March 07, 2012. (**News story with video**)
- Game Controller Talks To You By Stretching Your Skin Making It: A novel system gives these controllers surreal haptic feedback, instead of just rumbling, by Mark Wilson, Fast Company — Co-Design, March 9, 2012.
- New Game Controller Stretches Thumbs to Mimic Action On Screen, by Leslie Horn, PC Magazine, March 6, 2012.
- Haptics Symposium 2012 – Game controller featured as part of Haptics Symposium 2012 coverage by Briana McIvor on Electric Playground. The segment was broadcast nationally across Canada on Citytv, G4tv, & FX Canada on March 19th. (**News story with video**)
- Experimental controller has ‘thumbstick within thumbstick’ for blistering sensations, by Sharif Sakr posted on Engadget, Mar 6th, 2012
- Pesquisadores desenvolvem controle que aumenta sensações dos jogadores, by Thais de Luna for the Brazilian newspaper Correio Braziliense, April 2, 2012
- Prototype Controller Ditches Rumble For Tactile Feedback, by Don Hatfield, MTV.com, Posted 3/6/12.
- Prototype control pad offers generational leap in tactile feedback for games, by James Holloway, GizMag, March 5, 2012
- Thumbscrews promise skin-stretching gaming controller: More complex than the Vita, by Matthew Finnegan in London, March 5, 2012.
- U. engineers design new controller that could change gaming, by Joey Ferguson, Deseret News, Tuesday, March 6 2012.
- Skin Stretch Game Controller, KSTU-SLC (FOX) - Salt Lake City, UT, FOX 13 News at Nine, March 6, 2012. (**News story with video**)
- Haptic thumbsticks add pull and stretch feedback to game controllers, by: Matthew Humphries, on Geek.com, Mar. 6, 2012.
- Skin-stretching game controller pitched for next Xbox, By Luke Westaway of CNET-UK, on March 6, 2012,
- Engineers develop skin-stretching controller, by Gareth Halfacree on Bit Gamer, March 6, 2012
- Next Generation Haptic Feedback Game Controller Concept, by Julian Horsey on Geeky Gadgets, Tuesday 6th March 2012.
- ‘Skin stretch’ controller adds new kind of feedback to games, by Andrew Webster on The Verge, March 5, 2012.
- Thumb stretching technology adds new dimension to game controllers, by Oliver, Idealo News, on 6 March, 2012
- Your Next Controller May Tug at Your Thumbs, by Jim, Overclockers Club, March 5, 2012.
- Video game controller tugs on your thumbs, by Jacob Aron, New Scientist, March 7, 2012
- Prototype controller adds haptic feedback to analog sticks, by Geoff Gasior, on The Tech Report, March 6, 2012
- New Touch Feedback Technology Pitched to Console Developers, by Stuart Bedford, What Culture, March 6, 2012.
- Game Controllers Level Up With Thumb-Tractors, by Smriti Jha on Crazy Engineers, March 7, 2012
- University of Utah student newsgroup story on Skin Stretch Game Controller (see from 4:10 to 7:52 of the news show), University of Utah, Newsbreak Show, 3/21/2012 (Newsbreak site link) (**News story with video**)
- A New Direction for Game Controllers — Prototypes Tug at Thumb Tips to Enhance Video Gaming, University of Utah Press Release, March 5, 2012.
- Skin Stretch Cues for Navigation while Driving
- Saving Distracted Drivers, Discoveries and Breakthroughs Inside Science, Ivanhoe Broadcasting, Dec. 2010. (**News story with video**)
- University Of Utah Researching High-Tech Tactile Navigation, Fox 13 (SLC, UT) News, September 27th, 2010 (Story on shear feedback for in-car navigation).
- Need Directions? Pull My Finger, on Discovery News, Oct. 7, 2010.
- Touch-based navigation may move the blind, on Medill Reports Chicago, Oct. 6, 2010.
- Let Your Fingers Do the Driving If you don’t hear directions, you can feel them, on US News and World Report, Sept. 28, 2010
- Could a Touch-Based Navigation System Make Driving Safer? Devices that push fingertips left and right more effective than GPS for distracted drivers, researchers find, Buisness Week, Sept. 28, 2010.
- Experimental new sat nav tugs drivers’ fingertips, on Wired UK, Oct. 8, 2010.
- Could a Touch-Based Navigation System Make Driving Safer?, on MSN Heath & Fitness, Sept. 28, 2010.
- Shear feedback GPS navigation tells your fingers where to go, you just have to follow, on Engadget.com, Oct. 11, 2010.
- Le GPS et le doigt de l’automobiliste, on LeMonde.fr science blog, Oct. 10, 2010
- Feeling directions: Let your fingers do the driving, R&D Magazine, Sept. 27, 2010
- Let Your Fingers Do the Driving, on Product Design & Development, Sept. 27, 2010
- A tug on the finger may make driving easier, safer, Baltimore Sun, Sept. 27, 2010.
- Navi-Zukunft: Fahren mit Fingerspitzengefühl: Taktile Information am Steuer macht Navigationssysteme sicherer, Pressetext, Austria, Sept. 27, 2010.
- Researchers develop touch device for safer driving, French Tribune, Sept. 28, 2010.
- Let Your Fingers Do The Driving, on NSF Science 360, Sept. 28, 2010.
- Hearing impaired? Steering wheel can guide you, CNET News, Sept. 27, 2010.
- Touch-enabled steering wheel gives sat-nav guidance by pulling skin, CNET UK, Sept. 28, 2010
- Touch-based directional devices let users feel directions, gizmag.com, Sept. 27, 2010.
- Driving Safely Gets All Touchy-Feely, on Inventor Spot, Sept. 27, 2010.
- Let Your Fingers Do the Driving: If You Don’t Hear Directions, You Can Feel Them, Science Daily, Sept. 27, 2010.
- Let Your Fingers Do The Driving, on redorbit.com, Sept. 27, 2010.
- Let your fingers do the driving: If you don’t hear directions, you can feel them, on PhysOrg.com, Sept. 27, 2010.
- Researchers Consider New Way for Drivers to Safely Get Directions, EHS Today, Sept. 28, 2010.
- Let your fingers do the driving, on ECNmag.com, Sept. 27, 2010.
- Let your fingers do the driving, on Medical Daily, Sept. 27, 2010.
- A Device Mounted on the Steering Wheel to Guide Your Fingers While Driving, on Med. India, Sept. 28, 2010.
- Let your fingers do the driving, India Times, India, Sept. 27, 2010.
- Tactile Navigation Lets You Feel GPS Directions, on switched.com, Sept. 28, 2010.
- Tactile Navigation System Lets Drivers Feel Out Their Routes Without Distractions, on Popsci.com, Sept. 28, 2010.
- Let your fingers do the driving, eurekalert.org, Sept. 26, 2010
- Utah engineers develop navigating system using touch technology, Video on Fox 13 (SLC, UT) News, 5:41 PM MDT, October 3, 2010 (Story on shear feedback for in-car navigation).
- Let Your Fingers Do the Driving: If You Don’t Hear Directions, You Can Feel Them, University of Utah Press Release, Sept. 26, 2010.
This work is supported by the National Science Foundation under grant numbers IIS-0746914, IIS-0904456, and DGE-0654414. It was also supported by a University of Utah Research Foundation Technology Commercialization Project. 
