Category Archives: Unity3D

Controlling an avatar by thought using fMRI

Cohen O., 1, 3 


Mendelsohn A., 2
Drai D., 1
Malach R., 2
Friedman D. 1
1 The Interdisciplinary Center Herzliya,
2 Weizmann Institute of Science
3 Bar Ilan University.

We are carrying out a number of studies whereby subjects control the movement of an avatar by thought alone, using real-time fMRI. At the first stage of our study five three subjects were able to control right- and left-hand movements of an avatar using motor imagery without difficulty and with high accuracy. Therefore, we are running a study involving three classes: right-hand imagery to turn right, left-hand imagery to turn left, and feet imagery to move forward.
In the first part of the experiment we instruct the subject to imagine either left-hand, right-hand, or walking movements upon a predefined cue, and manually define regions of interest (ROIs), using a GLM analysis, by contrasting hand-motor and primary leg-motor regions. In the second phase (baseline) we compute the mean and standard deviation for each ROI. Finally, the subjects are instructed to move the avatar, according to auditory cues, by using motor imagery. At this preliminary stage we use a simple classification scheme; at each TR the system calculates the z-score value of each ROI relative to the mean values obtained during the baseline, and chooses the highest z-score value among the ROIs. The classification result is fed into the virtual reality system to move the avatar accordingly.
Our preliminary results indicate that subjects can learn to control an avatar using motor imagery in better-than-chance levels with very little training. Future work will include support vector machine (SVM) classifiers, with methods for feature reduction, in real time. Eventually we aim at allowing subjects to perform simple tasks in a virtual environment.

Unity3D for research, how to compile-run-debug outside of Unity3D by using Visual C#

I use unity for research purposes, and 90% of my code is not unity. it became quite tedious to work with unity as the debugger. as some of you know unity compiles the code on its own and then runs it, debugging usually means putting console outputs to see if something works. i had to figure out how i can debug outside of unity preferably VS with but still be able to use unity as the rendering engine, but only when i needed it to. consider the following: every line you write has nothing to do with unity, except when you need a character to move. obviously you would like to debug the entire thing without going into unity, and only later to use it to move the character. The following is one method of bypassing Unity3D as a debugger by using Visual Studio.

Using VC# we create another solution outside of the unity project directory, this solution/project will be the one that we are working on and debugging from.

So basically:

  1. unity3d_directory\game
  2. unity3d_directory\testGame <= this is our project

Then we need to add these to our solution references (find them in the unity installation dir):

  1. the actual “game” solution file
  2. UnityEngine.dll
  3. UnityEditor.dll
  4. UnityScript.dll

Then add the Dlls to the references of actual unity game solution (they may already be there).
This takes care of a bunch of errors that relate to the code not finding the unity API.

Then for every class that inherits from MonoBehaviour, we want to add this preprocessor directive, it means that inside (DEBUG) VS we do not use the MonoBehaviour and outside (Unity) we do:

instead of:

public class MyClass: MonoBehaviour
{//class code}

this:

public class MyClass
   #if !DEBUG
      :MonoBehaviour
   #endif
{//class code}

Before we continue we need to understand a little bit about how unity interfaces with a class

  1. It first runs the “Awake()” function, we can consider this awake function to be the constructor, note that a real constructor will always be run when Unity loads a class (even before it runs it). so its better to use Awake inside Unity.
  2. After all awake methods are run, Unity runs “Start()” function before “Update()” but only once. this is also a type of constructor, but it runs after the Awake one.
  3. Then Unity calls “Update()”

so basically what we want to do in VC# is to call the constructor instead of Awake() or Start() (i use either one of them). but make sure that we can still use this code inside unity (remember that we dont want constructors inside unity).

so here is a typical method of enabling only the constructor inside VC# and only the Awake/Start functions inside Unity. what you see here is a preprocessor directive telling visual studio to ignore the “start()” line if we are in debugmode, which means we get the constructor functionality, and it also tells unity to ignore the constructor line and think of this function as awake(). cool isnt it ?

public class MyClass
   #if !DEBUG
      : MonoBehaviour
   #endif
{
   //  this function behaves as Start() functionality inside Unity as a constructor inside VC#
   #if !DEBUG
      public void Start()
   #else
      public MyClass()
   #endif
      {//code goes here}
}

So we handled our constructor/awake/start problem. now we address another problem of running actual code . your class should have an Update() function that runs your code, and this will be the main functionality function in each class.

public class MyClass
   #if !DEBUG
      : MonoBehaviour
   #endif
{
   //  this function behaves as Start() functionality inside Unity as a constructor inside VC#
   #if !DEBUG
      public void Start()
   #else
      public MyClass()
   #endif
      {//code goes here}

   public Update()
   {
      // here goes the main code of our class functionality
   }
}

Now lets say that we want a function that outputs to the console in unity when inside unity and output text to the dos window when inside VC#.

public class MyClass
   #if !DEBUG
      : MonoBehaviour
   #endif
{
   //  this function behaves as Start() functionality inside Unity as a constructor inside VC#
   #if !DEBUG
      public void Start()
   #else
      public MyClass()
   #endif
      {//code goes here}

   public Update()
   {
      // here goes the main code of our class functionality
   }

   public static void print(string s)
   {
      #if !DEBUG
         Debug.Log(s); //Unity
      #else
         Console.WriteLine(s); //C#
      #endif
   }
}

And what about a function that prints to Dos or Unity Gui?

public class MyClass
   #if !DEBUG
      : MonoBehaviour
   #endif
{
   //  this function behaves as Start() functionality inside Unity as a constructor inside VC#
   #if !DEBUG
      public void Start()
   #else
      public MyClass()
   #endif
      {//code goes here}

   public Update()
   {
      // here goes the main code of our class functionality
   }

   public static void print(string s)
   {
      #if !DEBUG
         Debug.Log(s); //Unity
      #else
         Console.WriteLine(s); //C#
      #endif
   }

   private void PrintToGUI()
   {
      #if !DEBUG
         GUI.guiText.text = "Unity";
      #else
         Console.WriteLine("Visual C#");
      #endif
   }
}

typically you would assign this script to a unity object, but because we are using VC#, we need to emulate how unity runs these classes, from our testGame project we create instances for these classes, our constructors will initialize it. if you are inside Unity, Unity will run awake() and then Start() when instanced. then we create a loop that runs all the Update() functions.

namespace testGame
{
   class Test
   {
      static void Main(string[] args)
      {
          MyClass mc = new MyClass();
          while (true)
          {
             mc.Update();
          }
       }
   }   
}