Fingertip skin stretch feedback (shear feedback) for communicating direction cues

Single Contactor Skin Stretch Feedback

In this research, we are applying skin stretch feedback (also referred to as shear feedback) to the fingertip and using these stimuli to communicate direction cues. This concept is portrayed in the below image, where the skin is subtly stretched by a plate or other skin contactor that is moved in the plane of the skin. This functionality has been built into portable devices which can in turn be embedded within a multitude of other products (e.g., a cell phone, steering wheel, or game controller).

Multi-Contactor Skin Stretch Feedback

We have also shown that use of multiple moving skin stretch contactors can be used to communicate to a user. In the past we have used multiple skin stretch contactors in gaming controllers, where each contactor was located in the middle of a thumbstick on a game controller. However, more recently we have discovered that the use of multiple contactors placed opposite each other (back-to-back) can be used to create torque-like (or rotary) sensations. That is, when moving the back-to-back contactors in the same direction this produces a translational cue, but when moved in opposite directions, this produces a rotational cue as portrayed in the images below. This same feedback mechanism can also be built into the handle of a device to give translational and rotational cues to a person’s hand. This approach was used in developing our lab’s Motion-based Game Controllers, which are described below. For more information on this research, see our publications listed towards the bottom of this page or the Motion-based Game Controllers described below.

Use of two actuated shear feedback contactors placed back to back to communicate translational and rotational cues to a user.

 


Movie showing the concept and some interaction scenarios for the use of “Reactive Grip” Shear Feedback. The device can portray the reaction forces and torques within the user’s precision grasp, between their index finger and thumb. We currently have demos of interacting with a virtual mass-spring system, opening a door, or guiding the user’s hand motions. We are currently pursuing applications in virtual reality training, robotic surgery, and upper limb rehabilitation. (Link to YouTube site showing the above movie).

 

Applications

Shear feedback could be used to provide direction cues for navigation or attention cuing. In particular, this technology could be used to interface with a GPS navigation system in a car as a substitute or complement to current spoken directions (also providing a non-visual interface for those with hearing impairments), guide those with vision impairments, or augment handheld electronics by providing tactile cues that could, for example, provide a pulse each time a song on a music player’s song list scrolls past. Images and a video clip portraying these applications are shown below.

Shear feedback placed on a steering wheel rim for providing navigation cues

Shear feedback to guide those with vision impairmentsShear feedback for enhancing human interfaces in handheld electronicsA multi-modal handheld device with embedded shear feedback
 


Movie showing the concept and applications of fingertip skin stretch feedback (also referred to as shear feedback). The device has applications for use in providing in-car navigation cues to the fingertips, guidance for the blind, or for enhancing handheld devices. (Link to YouTube site showing the above movie). A version of this video without captions can be viewed here.

Gaming

Our lab has designed multiple generations of game controllers that incorporate embedded skin stretch feedback. In the past, we have developed a game controller with skin stretch feedback embedded into the center of each thumbstick of a conventional console-style game controller (see “Console-style game controller” below).

Motion-based Game Controller
We have adapted our lab’s skin stretch feedback technology to be integrated into the handle of a device and can directly provide tactile feedback to a user’s hand. Our technology can be used to convey motion and force information. It does this through the motion of sliding contactor bars that are built into the handle of the device. Translational motions and forces can be portrayed along the length of the handle by moving the bars in unison in the corresponding direction; whereas moving opposing slider bars in opposite directions creates the feeling of the device’s handle wrenching within the user’s grasp. Because of its ability to recreate the skin sensations of actually holding an object, this type of feedback creates an engaging experience that brings gaming interactions to life, and we have hence named this feedback technology, “Reactive Grip” Skin Stretch Feedback.

This technology is compatible with motion-based controller interfaces, such as the Nintendo Wii, Sony Move, or Microsoft Kinect, and could also make a formidable pairing with a head mounted display for unprecedented levels of immersion and realism in a virtual reality gaming environment. The current game controller prototype uses a Razer Hydra as its motion tracking system, but our technology is not tracker dependent. For updates on the commercialization of this technology, check out the Tactical Haptics facebook and company pages.

 


Movie showing the concept and some game scenarios for the use of “Reactive Grip” Skin Stretch Feedback. The device can portray the reaction force at the handle of a device and is shown with users in a shooting gallery, sword fighting, and swinging a medieval flail. It could also be used to portray the forces in fly flishing, guide the swing of a golfer, or even embedded in the handle of a walking cane to guide the blind. (Link to YouTube site showing the above movie).

Console-style game controller

We have also designed a game controller with skin stretch feedback embedded into the center of each thumbstick of a conventional console-style game controller. Hence, our game controller allows a gamer to receive tactile feedback through the same interface that he/she makes game inputs. Providing this new mode of touch feedback has the potential to enhance the gaming experience.

An initial prototype was constructed which allows two handle configurations to ensure that skin stretch direction cues could still be properly interpreted with a traditional angled handle game controller design. A fully integrated game controller with vibration and skin stretch feedback has now also been designed and prototyped.

Several game prototypes have been developed to highlight the use of skin stretch feedback in our fully integrated game controller. We have also developed feedback modes that enhance immersion and realism for gaming scenarios such as collision, recoil from a gun, the feeling of being pushed by ocean waves, or crawling prone in a first person shooter game.

Initial prototype shown with two handle configurationsIntegrated game controller with vibrotactile feedback and skin stretch feedback embedded into the center of each thumbstick
 


Movie showing the integrated game controller with vibration feedback and embedded skin stretch feedback (also referred to as shear feedback). The thumbsticks are still used for input just as on conventional game controllers, but skin stretch tactors, located in the middle of the thumbsticks, can also be used to provide tactile feedback to the user. The video provides an overview of the functionality of the controller, some haptic effects that can be used to enhance immersion within the gaming environment, and an example fishing game which utilizes skin stretch feedback to represent the feeling of being adrift and forces experienced while fighting a fish. (Link to YouTube site showing the above movie). A version of this video with fewer captions can be viewed here.


Link to Vimeo movie showing the design and initial characterization of a video game controller prototype with tactile skin stretch displays embedded into its thumb joysticks. With this prototype, we verify that people are capable of the same high accuracy rates for identifying direction cues as was observed previously. Modern game controllers typically have users angle their thumbs diagonally inwards to reach the joysticks, however, this may affect the cognition of skin stretch cues. No significant reduction in performance due to mental rotation of stimuli delivered in the angled thumb configuration was shown. Furthermore, the angled configuration appears to have ergonomic advantages.


Movie showing a game that was developed for the Imagine Cup competition. This game uses our lab’s game controller that has embedded skin stretch and vibration feedback. The game was designed to provide a more interactive experience for people with hearing impairments. The skin stretch and vibration feedback is programmed to correspond to the game’s music soundtrack, allowing user’s to “feel” the music. This video also documents a visit to show off the device and newly developed game and get feedback from a group of high school students who are hearing impaired. (Link to YouTube site showing the above movie).

 

Perceptual Studies

Initial perception studies have been conducted and show the promise of this type of feedback. These studies have found several factors to be important for communicating direction via skin stretch at the fingertip. Among the most important factors are the speed and displacement of skin stretch applied to the fingertip. The below images show the interface used in our initial perception experiments and the results of these experiments. Further experiments and their results can be found in the articles below.

 
The tactor in contact with the finger. The thimble and thimble mount are shown translucent so that the finger and tactor can be seen. The thimble is free to move up and down, but constrained in the plane of tactor motion. Experimental results, combining data from all stimulus directions. Subjects attempted to identify the direction of skin-stretch stimuli at a range of stimulus speeds and displacements. Identification accuracy rates and corresponding 95% confidence intervals are show in the grid squares.

Forearm-mounted skin stretch feedback

While the fingertips excel at accurately discerning direction cues, it is often desirable to leave the fingers unencumbered for other tasks. This desire has motivated an effort to explore other locations to which direction cues can be applied. Studies of human sensitivity indicate that the hand or forearm may be a good location for skin stretch cues. Direction perception tests were conducted on 4 hand/arm locations to determine acuity and suitability of these locations. Based on this information, a portable device was designed that applies direction cues to the forearm. See below publication by [Caswell et al. 2012].

The four tested hand/arm locationsDirection identification accuracy on hand/forearmForearm-mounted skin stretch feedback prototype
 

Further Details

Several US and International patents are currently pending on this technology. Please contact Dr. Provancher if you have interests in collaborating/partnering or in licensing this technology (wil_at_mech.utah.edu). Further information on this research can be found in

Related technologies are being commercialized through Tactical Haptics LLC

Related Publications & Media Coverage


Publications

Media Coverage

see (**news stories with video are marked with red label**)

This work is supported by the National Science Foundation under grant numbers IIS-0746914, IIS-0904456, and DGE-0654414. It was also supported by a University of Utah Research Foundation Technology Commercialization Project.


HEML Home

Research

People

Publications

Resources

Contact Us