Movie Clips Reconstructed From Brain Waves

Your favorite spot for news, entertainment, video games, TV, movies, books, your mom, etc.

Moderator: Tsuki

Post Reply
User avatar
aine
ANGERME
Posts: 3843
Joined: Sun Sep 17, 2006 4:08 pm

Movie Clips Reconstructed From Brain Waves

Post by aine »

Movie Clips Reconstructed From Brain Waves



Researchers at UC Berkeley have reconstructed video clips based only on the brain activity of the people who watched them. To the left is the original clip. To the right is an average of 100 similar clips from YouTube, picked out by a computer program that matched the clips to the brain activity triggered by the original.



Image



Welcome to the future: Scientists can now peer inside the brain and reconstruct videos of what a person has seen, based only on their brain activity.



The reconstructed videos could be seen as a primitive — and somewhat blurry — form of mind reading, though researchers are decades from being able to decode anything as personal as memories or thoughts, if such a thing is even possible. Currently, the mind-reading technique requires powerful magnets, hours of time and millions of seconds of YouTube videos.



But in the long term, similar methods could be used to communicate with stroke patients or coma patients living in a "locked-in" state, said study researcher Jack Gallant, a neuroscientist at the University of California, Berkeley.



"The idea is that they would be able to visualize a movie of what they want to talk about, and you would be able to decode that," Gallant told LiveScience.



Decoding the brain



Gallant's team has decoded the brain before. In 2008, the researchers reported that they'd developed a computer model that takes in brain activity data from functional magnetic resonance imaging (fMRI), compares it to a library of photos, and spits out the photo that the person was most likely looking at when the brain activity measurements were taken. That technique was accurate at picking the right photo nine out of 10 times.



But reconstructing video instead of still images is much tougher, Gallant said. That's because fMRI doesn't measure the activity of brain cells directly; it measures blood flow to active areas of the brain. This blood flow happens much more slowly than the zippy communication of the billions of neurons in the brain. [Inside the Brain: A Journey Through Time]



So Gallant and postdoctoral researcher Shinji Nishimoto built a computer program to bridge that gap. Part of the program was a model of thousands of virtual neurons. The other half was a model of how the activity of neurons affects the blood flow to active regions of the brain. Using this virtual bridge, the researchers were able to translate information from the slow blood flow into the speedy language of neuron activity.



Movie night … for science



Next came the fun part: Three volunteers, all neuroscientists on the project, watched hours of video clips while inside an fMRI machine. Outside volunteers weren't used because of the amount of time and effort involved, and because the neuroscientists were highly motivated to focus on the videos, ensuring better brain images.



Using the brain-imaging data, Gallant and his colleagues built a "dictionary" that linked brain activity patterns to individual video clips — much like their 2008 study did with pictures. This brain-movie translator was able to identify the movie that produced a given brain signal 95 percent of the time, plus or minus one second in the clip, when given 400 seconds of clips to choose from. Even when the computer model was given 1 million seconds of clips, it picked the right second more than 75 percent of the time.



With this accurate brain-to-movie-clip dictionary in hand, the researchers then introduced a new level of challenge. They gave the computer model 18 million seconds of new clips, all randomly downloaded from YouTube videos. None of the experiment participants had ever seen these clips.



The researchers then ran the participants' brain activity through the model, commanding it to pick the clips most likely to trigger each second of activity. The result was a from-scratch reconstruction of the person's visual experience of the movie. In other words, if the participants had seen a clip that showed Steve Martin sitting on the right side of the screen, the program could look at their brain activity and pick the YouTube clip that looked most like Martin sitting on the right side of the screen.



You can see the video clips here and here. In the first clip, the original video is on the left, while an average of the top 100 clips that were closest based on brain activity is on the right. (Averages were necessary, and also the reason for the blur, Gallant said, because even 18 million seconds of YouTube videos doesn’t come close to capturing all of the visual variety in the original clips.) The second segment of the video shows the original clip at the top and reconstructions below. The far-left column is average reconstructions, while the remaining columns are individual videos picked out by the program as being closest to the original.



(more in the source)


Coming up next - a dream-reading machine full of redtube clips. <img src='http://mm-bbs.org/public/style_emoticon ... ohyeah.gif' class='bbc_emoticon' alt=':D/' />
Last edited by aine on Fri Sep 23, 2011 12:52 pm, edited 1 time in total.
Image

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
#whereisyui #ゆいちゃん
~breathe your last and fall into my eternal embrace~
User avatar
Ap2000
つんく♂
Posts: 9525
Joined: Sat Oct 21, 2006 11:43 am

Re: Movie Clips Reconstructed From Brain Waves

Post by Ap2000 »

It's still only comparing the stuff though and not "rendering" it directly.



But they can use those visualizations in Horror movies. lol
User avatar
sadude
ANGERME
Posts: 1757
Joined: Sun Sep 17, 2006 4:03 pm

Re: Movie Clips Reconstructed From Brain Waves

Post by sadude »

That's pretty awesome. This'll be a nice project for neuroscience to work on until they come up with something to measure other than blood flow in the brain.
User avatar
Melon
ANGERME
Posts: 4394
Joined: Sun Sep 17, 2006 6:40 pm

Re: Movie Clips Reconstructed From Brain Waves

Post by Melon »

That is super awesome. I want to record my crazy dreams at night! Make it happen SCIENCE!
Image
User avatar
Raven
Kenshuusei
Posts: 13
Joined: Wed Sep 28, 2011 4:54 am

Re: Movie Clips Reconstructed From Brain Waves

Post by Raven »

coool <img src='http://mm-bbs.org/public/style_emoticon ... nceman.gif' class='bbc_emoticon' alt=':danceman:' />
[font="Lucida Console]KAMEI ERI - ATOMIC WARHEAD OF FOOLISHNESS [/font]
User avatar
Ap2000
つんく♂
Posts: 9525
Joined: Sat Oct 21, 2006 11:43 am

Re: Movie Clips Reconstructed From Brain Waves

Post by Ap2000 »

[quote name='luvKamei' timestamp='1317455960' post='109393']

coool <img src='http://mm-bbs.org/public/style_emoticon ... nceman.gif' class='bbc_emoticon' alt=':danceman:' />

[/quote]



ARE YOU SERIOUS !?



Another person to put on my ignore list.
User avatar
Raven
Kenshuusei
Posts: 13
Joined: Wed Sep 28, 2011 4:54 am

Re: Movie Clips Reconstructed From Brain Waves

Post by Raven »

[quote name='Ap2000' timestamp='1317476217' post='109411']

[quote name='luvKamei' timestamp='1317455960' post='109393']

coool <img src='http://mm-bbs.org/public/style_emoticon ... nceman.gif' class='bbc_emoticon' alt=':danceman:' />

[/quote]



ARE YOU SERIOUS !?



Another person to put on my ignore list.

[/quote]



Huh? add to ignore list? Lol? I just find this pretty cool, thats all. So what are you unhappy about? <img src='http://mm-bbs.org/public/style_emoticon ... hatthe.gif' class='bbc_emoticon' alt=':whatthe:' />
Last edited by Raven on Tue Oct 04, 2011 3:43 am, edited 1 time in total.
[font="Lucida Console]KAMEI ERI - ATOMIC WARHEAD OF FOOLISHNESS [/font]
User avatar
Haru
ANGERME
Posts: 3798
Joined: Fri Oct 09, 2009 7:18 am

Re: Movie Clips Reconstructed From Brain Waves

Post by Haru »

I don't think you've been introduced to the rules.

And if you have, you need to be reacquainted.
[11:13:35 PM] Liam: DO NOT LET JAPANESE IDOLS GET IN THE WAY OF THE *AMERICAN* DREAM
Post Reply