• MSN
  • Hotmail
  • More
    • Autos
    • My MSN
    • Video
    • Careers & Jobs
    • Personals
    • Weather
    • Delish
    • Quotes
    • White Pages
    • Games
    • Real Estate
    • Wonderwall
    • Horoscopes
    • Shopping
    • Yellow Pages
    • Local Edition
    • Traffic
    • Feedback
    • Maps & Directions
    • Travel
    • Full MSN Index
  • Bing
  • NBCNews.com
  • TODAY
  • Nightly News
  • Rock Center
  • Meet the Press
  • Dateline
  • msnbc
  • Breaking News
  • Newsvine
  • Home
  • US
  • World
  • Politics
  • Business
  • Sports
  • Entertainment
  • Health
  • Tech
  • Science
  • Travel
  • Local
  • Weather
Advertise | AdChoices
  • Recommended: Pizza printouts? NASA funds project to make space meals with 3-D printer
  • Recommended: Months after death, Sally Ride wins honors from White House and NASA
  • Recommended: Dolphins persuade Navy trainers to dredge up 130-year-old torpedo
  • Recommended: Why sign up for a one-way Mars trip? Three applicants explain the appeal

Quantum fluctuations in science, space and society, from quarks to Hubble and Mars. Served up by Alan Boyle, NBC News Digital science editor. E-mail Alan, or connect via Facebook, Twitter or Google+.

  • ↓ About this blog
  • ↓ Archives
    • Icons Email E-mail updates
    • Icons Twitter Follow on Twitter
    • Icons Feed Subscribe to RSS
  • 18
    Feb
    2012
    6:30pm, EST

    Device turns gestures into song

    Researchers at the University of British Columbia demonstrate a gesture-controlled artificial speech system that's good enough to sing.

    By Alan Boyle, Science Editor, NBC News

    Follow @b0yle



    Researchers have created a system that converts hand gestures into speech, and yes, into song as well. Although the system isn't yet ready for a shot at "American Idol," its name — Digital Ventriloquized Actor, or DiVA — gives you an idea where the technology is going.


    "It is a singing synthesizer," said Sidney Fels, director of the University of British Columbia's Media and Graphics Interdisciplinary Center, or MAGIC. Fels explained how DiVA does its thing today in Vancouver at the annual meeting of the American Association for the Advancement of Science.

    With the gestures of the right hand, DiVA's operator controls the pitch and the character of the sounds. Closed-hand gestures produce consonants. Open-hand gestures produce vowels. Meanwhile, the left hand is hooked up with finger contacts to create stop sounds like and buh. "We designed a gestural space that mimics the vocal tract," Fels explained.

    The result is eerie: In the video above, you'll see a singer accompanying herself with the DiVA's voice. (I'm not ready to put it on my playlist just yet.) And in a series of videos, DiVA operator Sageev Oore synthetically sings the alphabet song and recites Dr. Seuss' "Green Eggs and Ham" verse as if he were playing two characters. (Which is kind of like Gollum talking to himself in the "Lord of the Rings" movies.)

    If DiVA goes commercial, it could provide a new way for people with speech disabilities to make themselves heard. But why go to all that trouble when there are other speech synthesizers out there, including the electronic voice made famous by physicist Stephen Hawking?

    "The problem with that is, you won't be able to sing. You won't be able to be expressive," Fels said.

    One of the intended applications for the technology is to create new types of singing musical instruments that can be played in real time. Fels said there have been five compositions written for DiVA so far, played by musicians trained to use the device. "It takes about 100 hours for a performer to learn how to speak and use the system," Fels said in a news release.

    The gloves, the volume-control foot pedals, the magnetic-sensor system and other components that bring DiVA to life can get rather unwieldy. "It's a backpack full of equipment," Fels told journalists. "I wouldn't walk around the restaurant and order sushi with it." But Fels and his MAGIC team are developing a version that can be operated with a computer tablet.

    That hints at what may be more important applications in the longer run. The DiVA project got started as a way to teach people how to control a complex system with gestures and give them auditory feedback to let them know when they're doing the gestures right.

    Follow @CosmicLog

    "Other possible applications for this discovery are interfaces to make certain tasks easier, such as controlling cranes or other heavy machinery," Fels said. It's also conceivable that gesture-based training might offer an alternative way to learn and practice foreign languages, particularly Asian dialects that depend on precise tonal control.

    Gesture-controlled input devices ranging from Nintendo's Wii and Microsoft's Kinect have already revolutionized the gaming industry. Will DiVA, or other devices like it, open up a whole new frontier for the field? Does the future belong to gestures? Feel free to weigh in with your comments below.

    More about gesture-controlled devices:

    • Game device adapted for a robot's touch
    • Control the cosmos with your fingers
    • Hack lets game device read sign language
    • Will your next smartphone recognize gestures?

    More from the AAAS meeting in Vancouver:

    • Researchers working to build a better leaf
    • Answers ahead for physics' deepest mysteries
    • Scientists revive sounds of Stonehenge and other sacred spaces
    • Gas-drilling gaffes aren't unique to fracking, study says 

    Since I mentioned Kinect, I should note that msnbc.com is a joint venture involving Microsoft as well as NBC Universal.

    Alan Boyle is science editor at msnbc.com. Connect with the Cosmic Log community by "liking" the log's Facebook page, following @b0yle on Twitter or adding the Cosmic Log Google+ page to your circles. You can also check out "The Case for Pluto," my book about the controversial dwarf planet and the search for new worlds.

    12 comments

    You can think of this as a theremin that talks ... which is more complex than your run-of-the-mill theremin. That being said, even a run-of-the-mill theremin is pretty cool. In fact, the effect is used in the Cosmic Log theme song, written by yours truly and performed by "rocker scientist" James Eml …

    Show more
    Explore related topics: music, speech, science, featured, aaas

Browse

  • featured,
  • science,
  • space,
  • images,
  • nasa,
  • innovation,
  • cosmic-log,
  • video,
  • john-roach,
  • tech-science,
  • mars,
  • new-space,
  • daily-dose,
  • technology,
  • energy,
  • participation,
  • environment,
  • whimsy,
  • holiday-calendar,
  • planets,
  • on-the-fringe,
  • archaeology,
  • physics,
  • spacex,
  • curiosity,
  • moon,
  • books,
  • msl,
  • politics,
  • aurora,
  • hubble,
  • sun,
  • robot,
  • religion,
  • japan,
  • 3-d,
  • genetics,
  • iss,
  • movies,
  • astrobiology,
  • saturn,
  • automotive,
  • updated,
  • evolution,
  • shuttle
Also
Advertise | AdChoices

Alan Boyle, Science Editor, NBC News

Science editor at msnbc.com, author of "The Case for Pluto," winner of the National Academies Communication Award for Cosmic Log in 2008. Alan Boyle covers the physical sciences, anthropology, technological innovation and space science and exploration for msnbc.com. Check out Cosmic Log's archives by following the links below, and see Boyle's full biography at http://bit.ly/boyle-bio

Alan Boyle, Science Editor, NBC News Blogroll

  • Bad Astronomy
  • CollectSpace
  • Cosmic Variance
  • Curmudgeons Corner
  • Discovery News
  • The Daily Grail
  • EarthSky
  • GeekPress
  • Habitable Zone
  • HobbySpace Log
  • LiveScience
  • The Loom
  • NASA Watch
  • NASA Spaceflight
  • Out of the Cradle
  • SciDev.net
  • Science Blog
  • ScienceBlogs
  • Science Quest
  • SciAm Observations
  • Seed Magazine
  • Slashdot Science
  • Space.com
  • Spaceflight Now
  • Space Fellowship
  • The Space Review
  • Transterrestrial Musings
  • Universe Today
  • Unmanned Spaceflight
  • Phenomena
  • Planetary Society Blog
  • Science News
  • Popular Mechanics
  • Popular Science
  • Science Insider
  • NASAEngineer.com
  • EurekAlert
  • Nature: The Great Beyond
  • Space Daily
  • Space Politics
The Case for Pluto
Alan Boyle's first book tells the story of Pluto's ups and downs as well as the discoveries of other dwarf planets in our own solar system and even more alien worlds beyond. Buy "The Case for Pluto" ...

Archives

  • 2013
    • May (33)
    • April (55)
    • March (53)
    • February (44)
    • January (45)
  • 2012
    • December (67)
    • November (12)
    • October (39)
    • September (43)
    • August (62)
    • July (45)
    • June (51)
    • May (46)
    • April (40)
    • March (56)
    • February (63)
    • January (66)
  • 2011
    • December (89)
    • November (73)
    • October (62)
    • September (67)
    • August (61)
    • July (70)
    • June (82)
    • May (86)
    • April (69)
    • March (94)
    • February (67)
    • January (82)
  • 2010
    • December (118)
    • November (62)
    • October (82)
    • September (63)
    • August (62)
    • July (54)
    • June (83)
    • May (51)
    • April (31)
    • March (35)
    • February (36)
    • January (35)
  • 2009
    • December (42)
    • November (34)
    • October (35)
    • September (40)
    • August (32)
    • July (38)
    • June (45)
    • May (37)
    • April (42)
    • March (38)
    • February (37)
    • January (35)
  • 2008
    • December (33)
    • November (31)
    • October (42)
    • September (48)
    • August (35)
    • July (37)
    • June (42)
    • May (43)
    • April (40)
    • March (39)
    • February (42)
    • January (42)
  • 2007
    • December (29)
    • November (40)
    • October (57)
    • September (35)
    • August (47)
    • July (38)
    • June (44)
    • May (44)
    • April (43)
    • March (40)
    • February (41)
    • January (47)
  • 2006
    • December (45)
    • November (49)
    • October (39)
    • September (50)
    • August (58)
    • July (45)
    • June (56)
    • May (8)

Most Commented

  • Why sign up for a one-way Mars trip? Three applicants explain the appeal (326)
  • Virgin birth or hanky-panky? Anteater mom sparks a scientific debate (91)
  • Curse or coincidence? Scientists study Tornado Alley's past and future (118)
  • Buggy hordes of cicadas sighted in Virginia ... but New York? Not yet (77)
  • Dolphins persuade Navy trainers to dredge up 130-year-old torpedo (44)
  • Months after death, Sally Ride wins honors from White House and NASA (63)
  • Scientists respond to planet hunter's plight with pointers – and poetry (28)

Other blogs

  • The Body Odd
  • Red Tape Chronicles
  • PhotoBlog
  • US News
  • Open Channel

NBCNews.com top stories

3147,10
© 2013 NBCNews.com
  • Science on NBCNews.com
  • About us
  • Contact
  • Help
  • Site map
  • Careers
  • Closed captioning
  • Terms & Conditions
  • Privacy policy
  • Advertise