postdocs, Dr. Shinji Nishimoto, how they found volunteers who were willing to lie still for hours on end with only fragments of video footage to occupy the time. He said the people in the room, the grad students and postdocs, volunteered to be guinea pigs for their own research.
As the subject watches the movies, the MRI machine creates a 3-D image of the blood flow within the brain. The MRI image looks like a vast collection of thirty thousand dots, or voxels. Each voxel represents a pinpoint of neural energy, and the color of the dot corresponds to the intensity of the signal and blood flow. Red dots represent points of large neural activity, while blue dots represent points of less activity. (The final image looks very much like thousands of Christmas lights in the shape of the brain. Immediately you can see that the brain is concentrating most of its mental energy in the visual cortex, which is located at the back of the brain, while watching these videos.)
Gallant’s MRI machine is so powerful it can identify two to three hundreddistinct regions of the brain and, on average, can take snapshots that have one hundred dots per region of the brain. (One goal for future generations of MRI technology is to provide an even sharper resolution by increasing the number of dots per region of the brain.)
At first, this 3-D collection of colored dots looks like gibberish. But after years of research, Dr. Gallant and his colleagues have developed a mathematical formula that begins to find relationships between certain features of a picture (edges, textures, intensity, etc.) and the MRI voxels. For example, if you look at a boundary, you’ll notice it’s a region separating lighter and darker areas, and hence the edge generates a certain pattern of voxels. By having subject after subject view such a large library of movie clips, this mathematical formula is refined, allowing the computer to analyze how all sorts of images are converted into MRI voxels. Eventually the scientists were able to ascertain a direct correlation between certain MRI patterns of voxels and features within each picture.
At this point, the subject is then shown another movie trailer. The computer analyzes the voxels generated during this viewing and re-creates a rough approximation of the original image. (The computer selects images from one hundred movie clips that most closely resemble the one that the subject just saw and then merges images to create a close approximation.) In this way, the computer is able to create a fuzzy video of the visual imagery going through your mind. Dr. Gallant’s mathematical formula is so versatile that it can take a collection of MRI voxels and convert it into a picture, or it can do the reverse, taking a picture and then converting it to MRI voxels.
I had a chance to view the video created by Dr. Gallant’s group, and it was very impressive. Watching it was like viewing a movie with faces, animals, street scenes, and buildings through dark glasses. Although you could not see the details within each face or animal, you could clearly identify the kind of object you were seeing.
Not only can this program decode what you are looking at, it can also decode imaginary images circulating in your head. Let’s say you are asked to think of the Mona Lisa . We know from MRI scans that even though you’re not viewing the painting with your eyes, the visual cortex of your brain will light up. Dr. Gallant’s program then scans your brain while you are thinking of the Mona Lisa and flips through its data files of pictures, trying to find the closest match. In one experiment I saw, the computer selected a picture of the actress Salma Hayek as the closest approximation to the Mona Lisa . Ofcourse, the average person can easily recognize hundreds of faces, but the fact that the computer analyzed an image within a person’s brain and then picked out this picture from millions of random pictures at its disposal is still impressive.
The
As the subject watches the movies, the MRI machine creates a 3-D image of the blood flow within the brain. The MRI image looks like a vast collection of thirty thousand dots, or voxels. Each voxel represents a pinpoint of neural energy, and the color of the dot corresponds to the intensity of the signal and blood flow. Red dots represent points of large neural activity, while blue dots represent points of less activity. (The final image looks very much like thousands of Christmas lights in the shape of the brain. Immediately you can see that the brain is concentrating most of its mental energy in the visual cortex, which is located at the back of the brain, while watching these videos.)
Gallant’s MRI machine is so powerful it can identify two to three hundreddistinct regions of the brain and, on average, can take snapshots that have one hundred dots per region of the brain. (One goal for future generations of MRI technology is to provide an even sharper resolution by increasing the number of dots per region of the brain.)
At first, this 3-D collection of colored dots looks like gibberish. But after years of research, Dr. Gallant and his colleagues have developed a mathematical formula that begins to find relationships between certain features of a picture (edges, textures, intensity, etc.) and the MRI voxels. For example, if you look at a boundary, you’ll notice it’s a region separating lighter and darker areas, and hence the edge generates a certain pattern of voxels. By having subject after subject view such a large library of movie clips, this mathematical formula is refined, allowing the computer to analyze how all sorts of images are converted into MRI voxels. Eventually the scientists were able to ascertain a direct correlation between certain MRI patterns of voxels and features within each picture.
At this point, the subject is then shown another movie trailer. The computer analyzes the voxels generated during this viewing and re-creates a rough approximation of the original image. (The computer selects images from one hundred movie clips that most closely resemble the one that the subject just saw and then merges images to create a close approximation.) In this way, the computer is able to create a fuzzy video of the visual imagery going through your mind. Dr. Gallant’s mathematical formula is so versatile that it can take a collection of MRI voxels and convert it into a picture, or it can do the reverse, taking a picture and then converting it to MRI voxels.
I had a chance to view the video created by Dr. Gallant’s group, and it was very impressive. Watching it was like viewing a movie with faces, animals, street scenes, and buildings through dark glasses. Although you could not see the details within each face or animal, you could clearly identify the kind of object you were seeing.
Not only can this program decode what you are looking at, it can also decode imaginary images circulating in your head. Let’s say you are asked to think of the Mona Lisa . We know from MRI scans that even though you’re not viewing the painting with your eyes, the visual cortex of your brain will light up. Dr. Gallant’s program then scans your brain while you are thinking of the Mona Lisa and flips through its data files of pictures, trying to find the closest match. In one experiment I saw, the computer selected a picture of the actress Salma Hayek as the closest approximation to the Mona Lisa . Ofcourse, the average person can easily recognize hundreds of faces, but the fact that the computer analyzed an image within a person’s brain and then picked out this picture from millions of random pictures at its disposal is still impressive.
The
Similar Books
