import sys
import numpy as np
import cv2 as cv
USB = 1
try:
cam = cv.VideoCapture(USB)
except:
print("Cannot open camera on USB = " + str(USB) + "Try the values 0, 1, 2")
sys.exit
print("Q exits the program")
while True:
# Capture frame-by-frame
ret, frame = cam.read()
# if frame is read correctly ret is True
if not ret:
print("Can't receive frame (stream end?). Exiting ...")
break
# Our operations on the frame come here
gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)
# Display the resulting frame
cv.imshow('frame', gray)
if cv.waitKey(1) == ord('q'):
break
# When everything done, release the capture
cam.release()
cv.destroyAllWindows()The YET manual
Making YET
Making YET0
Headrests and supports
Installing the software
YET is currently programmed in Python and that makes the installation a little harder than you would expect.
The reason for that is that Python is an interpreted language, where the Software written in Python arrives at the users computer as source code and is run by a Python interpreter. In compiled languages, such as C, Java or Rust, the compiler translates the human-readable source code into binary code, which can run directly on the machine (i.e. the CPU in your computer). Unfortunately, the Python interpreter is a very complex software in its own right. This section will guide you through the steps to heave a fully-fledged Python environment on your computer, which will be able to run YET and many other Python programs.
Getting YET from Github
The YET code resides in my Github repository schmettow/YET. If you already know your way around Github, all you have to do is clone the YET repository to your computer.
If you are new to Github, you can download the repository as a zip file (<> Code -> Download Zip). Make sure to really unpack the Zip file, before you work with the YET code. It won’t work when you just enter the Zip file and try running it from there I
n both cases, you will end up with a folder called YET or YET-main on your computer. This folder contains all the files needed to run YET. For a shortcut: yeta/1/yeta_1.py is the main program.
Beginner’s choice: Thonny
If you are new to programming, I recommend to start with Thonny, which is a beginner-friendly Python editor. It is available for Windows, MacOS and Linux. You can download it from https://thonny.org/. The installation is straightforward and Thonny come with its own python environment, so you don’t have to install anything else.
Except: YET makes use of several Python libraries, which are not included in the standard Thonny installation. You have to install them manually. To do so, open Thonny and click on the “Tools” menu and select “Manage Packages”. In the window that opens, type the name of the package into the search field and hit Return. Then click on the “Install” button.
Repeat this for the following packages:
- scikit-learn
- pygame
- pandas
- opencv-camera
You can close the package manager window afterwards.
Advanced Python Users
Testing your environment and trouble shooting
YET makes use of several libraries, namely:
- Pygame for the graphical interface
- OpenCV for capturing video streams and eye detection
- Scikit-Learn for the machine-learning part
- Pandas for data handling
Scikit-Learn and Pandas are usually easy to install, but OpenCV and Pygames sometimes make trouble. Here is what you can do to verify your installation.
Testing the OpenCV installation
Now you can test whether CV was installed correctly, by copy-and-pasting the following code into a new Python file and hitting the Run button. If the program fails to open the camera, try changing line 3 to USB = 0 or USB = 2, until you see the video stream.
If CV2 is installed correctly, you should see yourself via your webcam or YET. If you still see error messages, that the USB device could not be found, this can have several reasons:
- It really only works, when at least one camera is connected via USB. On Notebooks, this usually is teh Webcam (USB0)
- The camera device is blocked by another application, such as video chat applications or OSB Studio. Close all other applications that use the webcam and try again.
- Since we are using ultra-cheap cameras for YET, maybe you got a bad sample, where the connectors arent’t good. Try connecting YET via USB and selecting it as a camera in your favorite video chat app. If that doesn’t work either, you probably got a broken sample.
Testing the Pygame installation
Pygame is used to create the user interface of YET. You can test the Pygame installation by pasting the following code into your programming editor:
import sys
import pygame
from pygame.locals import *
from pygame.compat import unichr_, unicode_
##### VARIABLES #####
# Colors
col_black = (0, 0, 0)
col_gray = (120, 120, 120)
col_white = (255, 255, 255)
WIN_SIZE = (500, 500)
pygame.init()
pygame.display.set_mode(WIN_SIZE)
pygame.display.set_caption("Eye game with Pygame")
WINDOW = pygame.display.get_surface()
WINDOW.fill(col_white)
print("Canvas size is (" + str(WIN_SIZE[0]) + "," + str(WIN_SIZE[1]) + ")")
print("(0,0) is the upper left corner")
while True:
WINDOW.fill(col_gray)
for event in pygame.event.get():
# IT
if event.type == QUIT:
pygame.quit()
# sys.exit()
# presentitionals
pygame.draw.circle(WINDOW, col_white, (250, 250), 200)
pygame.draw.circle(WINDOW, col_gray, (250, 250), 150)
pygame.draw.circle(WINDOW, col_black, (250, 250), 80)
pygame.display.update()If you see three circles, your Pygame installation is working
Using YET
An eye tracker is a device which records an image of the eyeball and uses it to calculate the position of the eyeball. Since digital cameras connected to computers are not a big deal, the magic of every eye tracker truly lies in how an eyeball picture (eye frame) obtained from the camera is converted into two numbers, x and y, to indicate which direction the participant is gazing.
Most eye trackers need training before they can produce coordinates for a particular person. This is often called calibration and usually involves that the participant is asked to look at some targets on the screen. The eye frame is recorded at these coordinates and used as a training sample. If you think this sounds a lot like a machine learning application, you are right. Section ?@sec-quad-bright will explain how Yet, or more specifically the Yeti14 engine, is doing the translation from eye frames to coordinates. It is more simple than you might think.
Yeta1 is a Python program that allows you to run eye tracking experiments using pictures in a slide show. It comes with two calibration routines. During the initial training the user is asked to look at nine target dots on the screen, one-by-one. During quick calibration, the participant is asked to look at a single dot at the center of the screen. During the experiment the quick calibration is used to frequently (but quickly) correct the translation for any movements of the head or the camera. This way, Yet can operate accurately over longer periods, without the need of head tracking or head support. However, it is still recommended to use a stable head mount (e.g. a headphone, see section ) and a simple head rest, at least.
An eye tracker would be useless if you don’t know what the user sees. Some eye trackers solve this problem by using a second head-mounted camera that points in forward direction. Yeta1 solves the problem by controlling what the user sees on the screen. With Yeta1 we can create experiments by calibrating the eye tracker for a computer screen and then showing a sequence pictures on the same screen.
The YET eye tracker system is designed as a a lean machine. You give it a bunch of pictures and Yet records the x,y coordinates while the participant is viewing them. There are no fancy dashboards or monitors. This part of the eye tracking research workflow is left to a data analysis tool, which was designed as a dynamic report in R/Quarto. This report reads all the recorded data and produces a whole set of useful measures, statistics and plots. After all, x,y coordinates are not very useful, per se. The dynamic report creates relevant measures, such as fixations, dwelling time and distance traveled. In addition, the Yeta1 dynamic report also supports area-of-interest coding.
The following section runs you through the design of experiment with Yet, by example of a real study.
Running an experiments with Yeta1
You should have obtained Yeta1 as a package, which contains:
yeta_1.py: the Python code of Yeta1.
yeta_1.Rmd: the dynamic report
a directory Stimuli, which contains:
a set of picture files
A Csv table Stimuli.csv, listing and describing the picture files.
a directory Data, which may contain Csv files. Here, Yeta_1 will drop the data.
a directory yeti14, which contains additional Python files.
If you want to see how Yeta_1 works, simply strap on your Yet0, open yeta_1.py in your favorite editor and run it.
The Uncanny Valley
The Uncanny Valley is a strange effect observed in human emotional responses to faces. While, generally, more human-like faces are preferred, faces that are human-like, but yet distinguishable, create feelings of erie. The UV22 experiment uses eye tracking to examine patterns of visual attention when viewing faces of Great Apes, including Homo Sapiens’ and their ancestors. Every face comes in two versions: with human-like eyes (white sclera) and with ape-like eyes (dark sclera). The research question is, whether a mismatch between eyes (ape or human) and skull (ape or human) is responsible for the Uncanny Valley effect.
And here is the quick start for running your own experiment:
- Replace the picture files with your own material
- Adjust the Stimuli.csv accordingly.
- Start yeta_1.py to run the experiment. Per run, Yeta_1 will create a new CSV file with a unique name in directory Data.
- If you need area-of-interest coding, you have to create your own AOI.csv table.
- After running one or multiple runs of the experiment, the collected results can be analyzed using the dynamic report yeta_1.rmd. For this purpose, open the Rmd file in Rstudio and knit it. The resulting report is designed to be self-explaining.
Basic configuration
Yeta1 does not have a user interface for the experimenter. In order to configure Yeta1, you will have to open the code (yeta_1.py) and adjust it to your needs. It is not complicated, because you only have to set the right values for a few parameters. The following table explains all configuration parameters, Yeta1 uses.
| Variable | Explanation |
|---|---|
| USB | USB device number. On systems equipped with a webcam, this usually is 1 (webcam is zero) |
| EXP_ID | Name of your experiment |
| EXPERIMENTER | Name or identifier of the experimenter. |
| STIM_PATH | Directory where the pictures reside. Default this is to collect stimuli in a directory Stimuli, which must be in the same folder as yeta_1.py |
| STIM_INFO | Directory where the stimulus description table is stored. Default this is Stimuli/Stimuli.csv |
| RESULT_DIR | Directory, where results are stored. Default is Data/ |
| PART_ID | Participant identifier. Default is a timestamp. |
| RESULT_FILE | File where the results of one run are stored. Defaults to Data/yeta1_Experiment_Experimenter_Part.csv |
| SCREEN_W | Width of the window |
| SCREEN_H | Height of the window |
| SLIDE_TIME | time in seconds between slides |
Most of these parameters have default values, that should work well, if you leave the directory structure of Yeta_1 untouched. A parameter that you may have to change is
USB = 1
This denotes that the second USB camera, that is connected to the system is used. Why the second? Because most systems already have a webcam built-in or external connected, which counts as the first device. If you have not yet acquired or made YET0, you can as well use the webcam for a first try. But it won’t get you very far, which was the initial problem that lead to the YET project.
The parameters that you should change are:
EXP_ID will appear as an identifier for the experiment Yeta_1 was used for. It is used to create the filenames, but also appears in the results table. Say, your experiment takes place in 2023 and is about the Stroop task, then you could create a experiment identifier by changing the line
EXP_ID = "UV22"
to
EXP_ID = "Strp23"
EXPERIMENTER will appear as an identifier for the experimenter. This is useful, when the data collection is carried out by a team, where every member has a different setup (e.g. screen size). If every experimenter sets the parameter to their own, say initials, the data analysis can make use of this by controlling for experimenter differences. If your name happened to be Zaphot Beeblebrox, then you could set the experimenter ID to:
EXERIMENTER = "ZB"
SCREEN_SIZE contains two valuzes, which are the width and the height of the Yeta_1 window. Both are measured in pixel and should not exceed the real screen height and width. The default of 800 by 800 should work on practically all systems, but is not optimal if you have a much better resolution. Note that these values should be set to a slightly smaller value than the actual screen size, to account for window decorations. If your computer has a the screen dimensions 1920 x 1080 (HD), the following values should work well:
SCREEN_SIZE = (1800, 1200)
Finally, the parameter SLIDES_TIME can be adjusted to increase or decrease the presentation time per stimulus in seconds. One reason to change this to a longer value if your stimuli contain text to be read, for example, when you are evaluating websites. A reason to set it to a smaller value is that when developing your experiment you will typically do many test runs. By setting the presentation time to half one second, this process gets much faster.
SLIDE_TIME = 0.5
Up to this point, you have given Yeta_1 some meta data and adjusted it to your screen. If you run Yeta_1 one more time, it should run the same experiment, but with adjusted window title and identifier columns in the results files. In the next section I will explain how to set up a new set of stimuli for Yet.
Preparing Stimuli
Working with eye tracking data
Limitations of YET0
- strong light, short episodes
- lack of head tracking