Tuesday, March 3, 2015

Brain-Controlled Shark Attack!

Visiting my friends at OpenBCI HQ, we got together to do some hacking.  Since I'm always looking to control new things with my brain, I was really excited to see that someone had brought a remote-controlled shark-shaped balloon (an "Air Swimmers").  This is a very cool toy -- it swims through the air in a wondrous way.  But, I can't just leave a good things alone.  So, after a few hours of hacking, Joel and I were able to turn this simple toy into a 5-person, brain-controlled, SHARK ATTACK!



Approach:  Our approach to this hack is extremely similar to the approach that we used for our multi-person control of a toy robot.  As shown in the figure below, the idea is that you get multiple players hooked up to a single EEG system (OpenBCI, in my case).  The computer processes the EEG data looking for each person's eyes-closed Alpha waves.  Depending upon which person's Alpha waves are detected, the computer sends commands to the shark.  The commands are conveyed to the shark via an Arduino, which is driving the shark's remote control.  The end result is that the shark swims because of one player's brain waves. I think that's pretty cool.


Two-Person Demo:  As Joel and I were pulling this hack together, we started to test it using just the two of us.  Being just two people, we could only do shark two commands, not all five.  It was still pretty fun, though.  I love the sense of excitement that happens when a hack first starts to work.


Hacking the Remote Control:  To make this shark controllable from the computer, we needed to hack into the shark's remote control.  Like when I hacked the remote for the toy robot, Joel found that the remote for the shark was simply a few push buttons that were wired to pull one side of the switch down to ground whenever the button was pushed.  So, to make this controllable from my computer, Joel soldered some wires to the circuit board (to the high side of each switch) to allow an Arduino to pull it down to ground instead of having to push it with your finger.  As a result, we can now send a command to the Arduino and cause the shark to move.  Our Arduino code for this hack is on GitHub here.

We modified the shark's remote control by adding a wire to the non-grounded side of each push button.
We brought the wires out and connected them to an Arduino.
An Arduino drives the shark's remote control.

EEG Electrode Setup:  It's quite easy to record a person's eyes-closed Alpha waves.  You need three electrodes.  Put one electrode on the back of your head (O1 or O2, if you know the 10-20 system), put the EEG reference electrode onto your earlobe, and put the EEG bias electrode on your other earlobe.  You can see some examples in the photo below, where we had three people controlling the shark.  We used the gold cup electrodes and Ten20 electrode paste that came with the OpenBCI kit.

Three-Man "Team Alpha!" Controlling the Shark.  You can also see
where we put the electrodes -- back of head and both earlobes.

OpenBCI Setup:  We are going to wire up multiple people to control this shark.  And to be clear, it is not normal to hook multiple people to one EEG system.  But that is what we are going to do.  This is definitely using EEG in a non-traditional way.  That's why this is called "hacking".  The trick to making it work is to tell the EEG system (in this case, OpenBCI) that each person has his own EEG reference electrode.  OpenBCI enables this by allowing each EEG channel to be run in "differential mode", where you use each channel's the "P" and "N" inputs as a differential pair.  This is in contrast to the more-usual "common reference mode", where we use one of the SRB inputs as a common EEG reference for all EEG channels.  To change OpenBCI to differential mode, you use the OpenBCI GUI, via the "Chan Set" tab, to change each channel's "SRB1" and "SRB2" setting to "off".  Then, in the main window, turn off all of the other channels that you are not using.

Screenshot Showing How to Configure for Five Channels in
Differential Mode...simply Turn Off SRB2.

Plugging Into OpenBCI:  Once you've got the EEG electrodes on the individual players, you've got to hook them into the OpenBCI board.  Because we're in "differential mode", each player will get one "P" input and one "N" input.  For this hack, we put the electrode from back of the head into the "N" input.  We then put the left earlobe into the corresponding "P" input.  Finally, the right earlobe was connected to a bias pin.  Because we had five players, there aren't enough bias pins available on the OpenBCI board.  I used a 16-channel OpenBCI board because it has 4 bias pins, so that covered four players.  The fifth player simply plugged into the analog ground pin ("AGND"), which is not as good as using a bias pin, but it worked well enough.

Wiring Electrodes to the OpenBCI Board for Five Players.  Note that the board only
has 4 bias connections, so one player is attached to AGND instead.

EEG Processing Algorithms:  As mentioned earlier, the PC does all of the EEG processing...no processing occurs on the OpenBCI board itself.  For the software on the PC, we started with the stock OpenBCI GUI.  Then, I extended it by (1) adding the Alpha detection algorithms and by (2) adding code to send shark commands to the Arduino.  This variant of the OpenBCI GUI is currently saved here on GitHub as a branch of the main repository.  As you can tell from the class names shown in the code, the code is based heavily upon the previous work with the HexBug robot...which you may find confusing since we're controlling a shark and not a HexBug.  Sorry for the confusion!

Three-Person Testing:  After Joel and I did our two-person testing, we roped in a couple of other players to join Joel.  That got us to a three-man shark attack!


Three-Person Brain-Controlled Shark

Shark Food:  While we were all hacking the shark to make it brain-controlled, Conor was busy doing his own hacking.  Once we finally got our brain-controlled shark into fighting condition, we couldn't resist swimming it over to harass Conor.  Conor was pretty sure that his teeth were sharper than the shark's, so he wasn't much afraid.

Conor Faces Off Against the Shark.

Four-Person Shark Control:  Having successfully used three people to control the shark, we wired up a fourth person.  Have you ever tried to get four people doing anything in a smooth and coordinated fashion?  It's hard!  But, we did have success...


Five-Person Shark Attack:  OK, if four people working together is hard, five people is just chaos. In case you can't seem him, notice below that the fifth guy is in the center of the crowd, kneeling so that you just see his head popping above the bench.  The wiring on the OpenBCI electrodes seems generously long when you're just attaching one person.  With 5 people, though, you really need longer wires...or you simply need a little creativity on how you pack the people together.

Five-Person Brain-Control of the Swimming Shark.  The fifth person is kneeling
and you can only see his head.  We need longer wires!

Swimming Away:  By the time that we got this all working, it was really late at night.  The time stamps on the pictures show that it was about midnight, and we'd been at OpenBCI HQ since about 10AM.  So, between the fatigue and all the caffeine, I think that we were getting some funny brain wave behavior.  For example, one guy was making some weak Alpha even with his eyes open.  Wacky!  Regardless, we were able to get the shark to swim around...until we swam the shark too far to one end of the OpenBCI HQ...

Swimming the Shark Off Into the Sunset.

...at which point the shark's IR remote control could no longer communicate with the shark.  Stranded Shark!  And so our night of EEG hacking ended.  Still, it was mighty fine work,  Go Team Alpha!

Update 2015-03-09: I just saw that someone has already made a BCI for controlling this very same swimming shark (Chen et al.  "Recreational devices controlled using an SSVEP-based Brain Computer Interface (BCI)").  Note that they used one person to control the shark via SSVEP, which is exactly what I did with my brain-controlled Hex Bug!

Update 2015-09-24: Wow!  I was given an opportunity to write an article for IEEE Spectrum for their Oct 2015 issue.  How cool is that?  You can check it out here: "OpenBCI: Control An Air Shark With Your Mind".

Update 2015-11-08: I see that Wired (magazine) posted their nicely-done video on our shark hacking.  It's quite an enjoyable piece.  Good work, Wired!.

20 comments:

  1. Chip, awesome! Especially enjoyed the videos and all the laughs as the shark 'attacked' you guys. Between the hexbug battle spider and the sinister shark, OpenBCI now has control of the land and the seas(!) :-)

    ReplyDelete
  2. Hey Chip, Thanks for all the awesome tutorials. I had a basic question.So I am having trouble keeping electrodes in place. I have the same free electrodes as openBCI kit. The ones in your picture seem to be stuck and not falling off at all. I use tape as well, but they keep coming off, splecially on hairy heady at Oz position. Any solution? Would be very grateful for reply.

    ReplyDelete
    Replies
    1. Are you using the EEG paste ("Ten20" brand) as well? You definitely need the paste...

      Delete
  3. Hey Chip, I'm not sure if my previous comment glitched out.. So.. I'm going to make this brief back-up just in case (so, please ignore if my original comment was actually posted). What do you think the main problems are in the development/research that's going into BCIs? And why do you think they're happening? Thanks for your time! Big fan by the way!

    ReplyDelete
    Replies
    1. Long delayed response. Sorry. I think that the main problem with BCIs are:

      (1) The Head Gear: No one wants to put on individual electrodes with gel. Ick.

      (2) Training Per Person: The signals that one gets from EEG are really messy. It's hard to do useful stuff with them. The only real hope is training the computer for one's own specific responses. But, few people want to spend the necessary time training the system. So, anything that can be done to develop a simple, reliable, quick training procedure would greatly help.

      Chip

      Delete
  4. Hey Chip again! Sorry for bothering you, but what would you say is best for looking at Mu waves? Monitoring the frequency or the voltage? Also, what do you think would improve the quality of the data you would receive? You know, so a program wouldn't have such a hard time telling what's Mu and what's not. Thanks Chip!

    ReplyDelete
    Replies
    1. Hi Raxnos, Thanks for the question regarding Mu waves. While Mu wave detection was not part of this shark-control hack, I have looked at them in some of my earlier posts...

      http://eeghacker.blogspot.com/2013/11/waveforms-from-homemade-electrodes.html

      Unfortunately, I found them to be very difficult to detect. Perhaps with more electrodes on the head and with some fancier processing, they'd be easier to find, but it's not easy.

      There are a number of folks who've successfully used the brain's motor imagery response as part of a BCI, but I think that it's important to realize that "motor imagery response" is a much more generalized approach than simply "Mu wave detection".

      That generalized approach uses more electrodes, more training data, and likely allows for signal features to be drawn from the full frequency spectrum (as determined by machine learning algorithms). By contrast, my setup is much simpler...I use one electrode and I look at one narrow frequency band. It's shouldn't be surprising that my simpler setup is less effective than the setup that provides more data.

      If you start exploring Mu waves, be sure to share your successes and failures!

      Chip

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  6. Hi guys, you shark is cool! I have a question: my mum will soon be 'locked in ' , because of ALS -Disease. Her brain is still ok, i.e, it can express everything.
    Did you ever tried to use the eeg to express emotions ( there are some classification DB free ) ? Or simple things like yes/no/happy/sad/pain/drink/eat/up/down ?
    I thought about input from EEG, classifying and output somehow through computer or a 'speaking mama -doll' ? Or do you know somebody who works in this direction ? Do you have other, better Ideas how to get somebody like that to communicate ?

    Best regards, Christine
    you can find my contact data on www.heilpraxis-kett.de

    ReplyDelete
    Replies
    1. Christine, do some searches on the OpenBCI forum for "P300 Speller". A commercial version of this also available called Intendex from g.tec in Austria.

      http://www.openbci.com/forum/index.php?p=/discussion/206/openvibe-p300-speller-tutorial-questions

      Delete
    2. Hey, that was a great idea, I found a lot of information, thanks a lot. g.tec seems to have a lot of projects that are interesting. THANKS !

      Christine

      Delete
  7. This article made me think of your blog. Maybe you'll like it: http://medicalxpress.com/news/2015-10-fast-noninvasive-brain-computer-interface.html

    ReplyDelete
  8. I really need your help.

    I use 32bit Board Kit (8-channel), win 7, notebook

    I am using processing and modify shark program by Chip which can measures alpha wave

    The problem is:
    When running the program, there is time lag which mean

    In the brain
    when i close my eyes, there is alpha wave.
    when i open my eyes, there is no alpha wave.
    In my measurement, when i open my eye, the computer still detect alpha wave due to alpha wave detected within time lag about 1 second

    What i have tried but still not work:
    I have to do a real time measurement instead of saving SD card
    I have try change FTDI > Port > to 1 ms( from http://openbci.com/forum/index.php?p=/discussion/199/latencytimer-inbuffersize-for-os-x-new-info-plist)
    I can't change the buffersize (but i dun;t know where to change it )

    What I need:
    the processing is at the same step as the data streaming from openbci

    What I compromise:
    1. can I reduce data streaming so that reduce the workload of the the processing?
    2. or other possible solution

    really thanks for your help .

    ReplyDelete
  9. Calvin, you are on Windows, so you use the Windows control panel to change the latency setting.

    http://openbci.com/forum/index.php?p=/discussion/200/latency-timer-windows-control-panel-setting

    The link you mentioned is for Mac users.

    re: slight delay upon opening eyes before alpha extinguishes: this may be just normal physiology for the person you are measuring. For example with myself, I've done a lot of Zen style open eye meditation. So I have some alpha present even with my eyes open.

    re: Processing workload. Your modern Win7 notebook should have plenty of CPU cycles to run the GUI app.

    ReplyDelete
  10. hello Chip, Now, I'm working on EEG, thank you for your sharing, It's very useful. I follow some data that you succeeded on GitHub, but I don't understand what I have to do with that data? I'm new in this.

    ReplyDelete
  11. No.1 Hacker in Bharat is Sunny Nehra India's top ethical hacker in india today

    ReplyDelete