In 1968, Ivan Sutherland and his students developed the first head-mounted display (HMD). This new see-through device made it possible to overlay rudimentary virtual objects such as triangles or cubes on the surrounding environment of the user (Sutherland 1968). The digital objects could change in perspective according to the user’s head movements, which were tracked using a heavy mechanical system. It was the first prototype of an augmented reality display. Half a century later, commercially available virtual reality (VR) headsets allow users to project very detailed virtual objects and HMDs have become everyday devices for exploring virtual environments (VEs).

The key factors that have made this rapid development possible have been the astonishing technological advances in displays, sensors, motion detection algorithms, computer hardware and batteries, as well as an increasingly better understanding of our visual perception. By continuously adapting technological innovations to newly discovered perceptual principles, VEs can now be explored intuitively, using natural gestures and movements.

Today, we may have even reached a point where comparing the virtual with the real world can not only help us to create more realistic virtual experiences, but also expand our knowledge of how we perceive and behave in our environment in general.

The first part of this thesis deals with the evaluation of the basic requirements that must be met in order for HMDs to be used for the presentation of stimuli in scientific experiments in psychophysics. The introduction first presents the basic functions and technologies of current HMDs. It then provides a general overview of the anatomy of the eye and the different types of eye movements we use to perceive the world around us, which also shape our perception in VR. To accurately measure the effect of stimuli in psychophysical experiments, it is also important to know exactly which eye movements are performed at what point during the presentation. Therefore, a brief introduction to the various eye tracking methods available follows. Particular emphasis was put on the latency requirements for the presentation of gaze-dependent stimuli, which are used in many paradigms of perception research. This is followed by study [I], in which we measured the latency of eye tracking in HMDs to assess whether gaze-contingent paradigms can be presented in VR. Next, an overview of how to determine three-dimensional fixation points and classify eye movements in VEs is provided.

In the following chapters, eye movements during various activities in VR are analysed. This initially serves to expand our knowledge of human perception, which could subsequently be used to improve VR applications. First, a scientific overview of typical eye movements during actions is provided. Here, particular emphasis was put on eye movements that adapt in order to solve specific tasks, as well as eye movements during locomotion. Next is study [II], in which we recorded gaze, movement and orientation data in three different typical locomotion tasks. We then used this data to predict future waypoints. Here, we were particularly interested in whether eye tracking data is capable of improving prediction accuracy.

The following chapter deals with the extent to which our eye and head movements are guided by visual stimuli. It provides an introduction to the basic principles of visual search, the visual pathway and the visual cortex, and then explains how the shift of attention is possible at the neural level in the brain. In study [III], we recorded and analysed eye and head movements during visual search tasks in VR.

The final chapter concludes the thesis with a general discussion of our findings. It presents different approaches for extending our results and addresses the question of the extent to which our eye movements are typically controlled by bottom-up and top-down processes. Then, we discuss how the current limitations of VR as a method in psychophysics can be addressed in research. Finally, the main conclusions of the thesis are drawn.

Head-mounted Displays

To present virtual objects, an HMD obviously needs some sort of display. To achieve stereo vision, two separate two-dimensional projections of a simulated object or scene are usually displayed. All objects in both images are slightly offset based on the position of the eyes (Hibbard, Dam, and Scarfe 2020). The brain can then extract three-dimensional information from this offset, just as it does from the retinal projections of the real world (Scarfe and Glennerster 2019; Wheatstone 1838). The first HMD prototypes were equipped with two CRTs as screens that presented two different images, one for each eye.

Today, two types of screens are currently available in HMDs: LCDs and OLED displays. Compared to their CRT predecessors, both technologies enable lighter and thinner VR headsets with screens that require less power and therefore also generate less heat. In both types of screen, the image is generated using RGB sub-pixels. In LCDs, these consist of a backlight source that emits light, which is then polarised and passed through colour filters and a liquid crystal matrix (Chen et al. 2018). Depending on the strength of an electric current applied to the liquid crystal layer, the crystals align themselves and become less or more translucent. As a result, sub-pixels become darker or lighter. Due to viscosity, it can take up to 10 ms for an LCD to change from black to white. In early LCDs, the backlight light source was a cold cathode fluorescent lamp. In today’s displays, a LED matrix is typically used, which consumes less power and enables splitting up the display surface in multiple dimming zones. As a result, black areas of the image appear darker and the overall contrast of the image is increased.

OLED displays have self-illuminating RGB subpixels, which consist of an LED matrix with self-emitting diodes and no backlight. This makes it possible to design even thinner and lighter displays, in which individual pixels are simply switched off to display black. It also lowers the overall power consumption and makes black to white response times of well under 1 ms possible. Yet current OLED displays have problems with brightness and stability (Liu et al. 2020): With continuous use, self-emitting LEDs become less efficient and therefore reduce their light output. This can lead to burnt-in images, especially if monochrome red, green, or blue stimuli are displayed at the same position using the same pixels over a long period of time.

To delay LED-wear, the maximum brightness in OLED displays is often reduced below the maximum brightness of the diodes by using PWM. During active PWM, the individual LEDs constantly switch between on- and off-states (Baker 2022). The longer each on-state (duty cycle) lasts, the higher the image brightness. For instance, a 90% duty cycle creates a bright image while a 10% duty cycle creates a dark image.

If the switching occurs at a frequency above the critical flicker fusion frequency of approximately 50 to 90 Hz, we generally do not perceive the image as flickering (Mankowska et al. 2021). Up to 500 Hz, however, we can still perceive flicker (with slight variations for different colours) when we are asked to detect it on modern displays (J. Davis, Hsieh, and Lee 2015). When PWM was first introduced in smartphone displays, frequencies of around 240 Hz were used, which some users found very unpleasant. Some can even get headaches from it, which might be related to migraineurs having slightly lower critical flicker frequencies (Kowacs et al. 2005, 2004).

Interestingly, the off-states, which are also sometimes described as artificially added black frames, have another useful property for displaying moving images: when we follow a moving rigid object with our eyes, we follow it with a continuous eye movement. By briefly inserting a black frame between two images, each stable image is visible for a shorter amount of time. Similar to the effect of a shorter exposure time in a camera, the perceived movement on the screen with black frames inserted appears sharper. This technique was also used in film projectors, which often used shutters with three blades at 24 frames per second to increase the flicker frequency to 72 Hz and achieve the same sharpening effect (Hammock 2015).

Unlike film projectors, LCDs and OLED displays do not automatically produce off-states between individual frames. Instead, the rendered content follows a ‘sample-and-hold’ characteristic (Marsh 2017). This means that a moving virtual object jumps forward frame by frame on a screen. Between jumps, the image remains static. This sample-and-hold image representation does not reflect the continuous movement in the real world. Each frame of the rendered content is only valid for a brief moment. In reality, at any subsequent point in time, the objects in the scene have moved to different locations. Sample-and-hold rendering presents another problem: When our eyes move across a static frame with a continuous eye movement, the content is perceived as blurry. Especially in VR, shifts of the whole FOV caused by head movements would then result in significant motion blur. Therefore, inserting artificial black images instead of presenting one frame until the next update has become a frequently used technique in HMDs. In this context, the length of each individual on-state of a frame is also referred to as the persistence of the display. In LCDs, synchronised backlight flashing is used to insert black frames. In OLED displays, all LEDs can be turned on and off directly to achieve the same effect.

Lenses

To distribute the image to the eyes of the user, a lens collects the light from the display and maps it onto the FOV (Bang et al. 2021). For this purpose, three different types of lens designs are currently used in HMDs: Fresnel lenses, aspheric lenses and pancake lenses (see Figure 1). All three VR lenses have positive optical power and increase the angular size. Therefore, they make the close display look large and spread the image across the FOV.

Fresnel lenses consist of many prism-like, ring-shaped sections. With each section and from the inside to the outside, the angle at which the emerging light is diffracted increases (A. Davis and Kühnlenz 2007). Fresnel lenses can be made of plastic and are therefore comparatively light. However, with the designs currently in use, the edges of each lens section are clearly visible. In addition, a sharp image can only be seen when the user’s eyes are directly opposite the flat surface in the centre of the lens. This makes the area through which an image can be perceived sharply (sweet spot) comparably small. Moreover, light, especially from outer parts of the lens, can stray into other, darker areas of the image (Haitao et al. 2022). Sometimes the resulting visual effect is referred to as a ghost image. Additionally, Fresnel lenses introduce distortion to the transmitted image, accompanied by chromatic aberrations, often resulting in blurring and colourful aberrations on high-contrast edges. To present an undistorted image, the displayed content is preprocessed by software (Anthes et al. 2016; Nikonorov et al. 2015).

Aspherical lenses have a continuously ground surface and can be made of plastic or glass, giving the transmitted image no visible internal edges. Moreover, aspheric lenses have a larger sweet spot. On the one hand, this means the image does not become blurry when the HMD slips slightly on the user’s head. On the other hand, distortions introduced by the lens still vary depending on the exact pupil position. Thus, such slippage and even small positional movements caused by an eye movement can lead to local distortions. Sometimes these effects, also known as the pupil swim effect, cause people to feel uncomfortable. To compensate them with software, very accurate and low latency eye tracking is needed (Rui et al. 2023).

In pancake lenses, the incoming light is reflected back and forth once within the lens using a half-mirror effect. As the optical path is therefore longer than the physical distance, the lens can be positioned closer to the display, which enables overall smaller HMDs (Bang et al. 2021). A side effect of the half-mirror, however, is that a part of the incoming light is reflected towards the display, resulting in a darker outgoing image. Although this can be compensated for by increasing the brightness of the screen, this in turn has other side effects, such as greater wear and tear on light sources of the displays as well as higher power consumption and greater heat generation. Like aspherical lenses, pancake lenses have a large sweet spot and can also cause pupil swim effects (Jia et al. 2023). Stray light caused by multiple surface reflections and imperfect polarisation control within the lens can also create ghost images (Luo et al. 2024).

Movement Tracking in Virtual Reality

To enable a VE that can not only be displayed but also be explored using movements such as head turning and walking, a VR headset needs a range of sensors that measure the user’s behaviour and transfers it into the virtual space. Modern HMDs are equipped with an IMU that usually includes an accelerometer, a gyroscope and a magnetometer to accurately track head rotation (Anthes et al. 2016).

The position can be tracked using external devices or cameras within the HMD, which is also known as inside-out tracking (Niehorster, Li, and Lappe 2017; Anthes et al. 2016). The latter is typically based on visual information from multiple front cameras that are on the outside of the headset (see Figure 2) to gather data on the current surroundings and can deliver even more accurate position tracking than common external tracking methods based on infrared lighthouses (Holzwarth et al. 2021). Both types of systems can also be used to track devices, such as controllers or body trackers that can be attached to joints such as knees or feet (Anthes et al. 2016). Using a multi-stage process to estimate hand pose and finger angles, inside-out tracking systems can also be used to track the hands without additional devices in about 45 ms (Abdlkarim et al. 2024). This allows users to interact with virtual objects, for example, by pointing and grasping to select and single out targets. Moreover, the camera-based tracking can be complemented with a depth sensor, to achieve a more detailed map of the surrounding objects.

On top of a depth sensor, additional sensors aimed at the user’s face, can be used to track subtle facial expressions, and together with live audio recordings from microphones, this information can be used to create ever more realistic virtual avatars that enable immersive conversations with other users. In addition, computer-generated three-dimensional avatars are now being developed that can even interact directly with the user via speech (Virk 2025).

In summary, today’s HMDs enable full body tracking while also being small and light. In addition to improved displays and lenses, all necessary sensors, computing and graphics processors can now be integrated into a single head-mounted device. As a result, VEs can now be rendered directly on the wearable device, enabling the exploration of VEs without restrictions.

Eye Movements

After an image is displayed on an HMD, it is passed through the lens and reaches the user. When the eyelids are open, light enters through the cornea, pupil and the lens of the eye (see Figure 3). The perceived brightness of the image is not only influenced by the display brightness alone: The diameter of the pupil also plays a role. When looking at bright light or increasingly brighter areas, the iris contracts, causing the pupil to narrow so that less light may enter the eye (Ferree, Rand, and Harris 1933). In dim light, the pupil dilates to increase the incoming photons. Therefore, to a certain extent, our eyes adapt to the given input (which also means that a dark image can be perceived as brighter after an adaptation period).

To see objects at varying distances sharply, the cornea and lens focus the incoming light beam directly onto the retina through contraction of the ciliary eye muscles. With increased age, the eye lens becomes less flexible (presbyopia), which makes it impossible to focus on near objects (Mordi and Ciuffreda 1998). In case of myopia, distant objects always appear blurred because the light rays reflected from these objects are focused in front of the retina, even when the ciliary muscles are relaxed (Baird et al. 2020).

When the incoming photons reach the retina, around 92 million rod- and approximately 4.6 million cone photoreceptors convert the incident light into action potentials, which are then propagated to the entire visual system (Curcio et al. 1990). Human eyes have three types of cones and one type of rod: the different cone photoreceptors each react optimally to certain wavelengths of light (red, blue and green), whereas the rods are not able to distinguish colours, but are much more sensitive to light and are therefore more suitable for night vision (Prasad and Galetta 2011).

The density of the cells varies across the FOV: The area with the highest density of cone cells (15,000 cells/degree²), and thus the highest possible visual acuity of 150 cycles per degree, is the fovea centralis in the centre of the visual field (Tuten and Harmening 2021; Miller et al. 1996). This means that we can see up to 150 black and white lines spread over 1° of visual angle in the centre of our FOV. Therefore, a minimum resolution of 300 pixels per degree of visual angle is required to display objects with foveal acuity. Currently available HMDs offer resolutions below 50 pixels per degree. However, this is still sufficient to display many small details. Acuity defined as standard vision (20/20) when using a Snellen chart is 30 cycles per degree, or 60 pixels per degree (Caltrider, Gupta, and Tripathy 2023). Towards the periphery, cone density and visual acuity rapidly decreases (Hirsch and Curcio 1989). Thus, we tend to bring objects of interest to the centre of our FOV by moving our eyes.

Fixating Objects

While we inspect an object with foveal accuracy, our eyes are in a comparatively still fixation state. In general, we fixate for about 90% of the total viewing time (A. T. Duchowski 2007). In free viewing, typical fixations have durations of between 0.1 and 0.4 s (Gajewski et al. 2005). The duration of a fixation can vary greatly, from a few hundredths to a few tenths of a second. In certain real-world situations they can also last several seconds (Benjamin W. Tatler et al. 2011; Hayhoe et al. 2003; Michael F. Land, Mennie, and Rusted 1999). In other experiments with memory tasks, the fixation time depended on the time required to obtain the necessary visual information for the current action (Benjamin W. Tatler et al. 2011; Hayhoe et al. 2003; Michael F. Land, Mennie, and Rusted 1999; Hayhoe, Bensinger, and Ballard 1998; Droll et al. 2005).

When we fixate on an object in the far distance, the horizontal angles of the left and right eye are almost parallel. In a situation in which the fixated object comes closer to us, our eyes turn continuously in the direction of the centre of the FOV. Our eyes therefore turn in opposite directions towards each other simultaneously. This movement is called vergence and is usually accompanied by a lens adjustment of the eye by the ciliary muscles, to keep the now closer object in focus. Both movements automatically happen at the same time through the accommodation-convergence reflex. While the visual offset between both eyes can be simulated by presenting stereo images, the screens of current HMDs are displaying images at a fixed focal length. This means that regardless of how far away a virtual object appears (which determines the convergence of the eyes), our eyes must always adjust to the fixed distance of the screen. As a result, unlike in reality, the entire field of view appears to have the same depth of field. Therefore, in VR the usual relationship between accommodation and convergence, to which we are accustomed, is disrupted. This phenomenon is called VAC.

Fixational Eye Movements

Our eyeballs can rotate on three rotation-axes using six different muscles: the medial and lateral recti for lateral movements along the yaw axis, the superior and inferior recti for upward and downward movements on the pitch axis and the superior and inferior obliques for twist movements along the roll axis (Davson 1990). In addition, the eyes can translate in the eye socket, which happens when closing the eyelid (Davson 1990; Kirchner, Watson, and Lappe 2022).

When examining fixations carefully and with very high spatial and temporal resolution, it becomes clear that the eyes are actually moving on a very small scale during fixations (Alexander and Martinez-Conde 2019). Fixational eye movements contribute to fixation stability and allow us to inspect different small details within an object with maximum foveal accuracy (Poletti, Listorti, and Rucci 2013; Rolfs 2009). These movements are categorised in three different types: tremor, ocular drift and microsaccades. Tremor is the smallest type of movement with amplitudes of 4/1000° and frequencies that normally range between 30 and 100 Hz (Alexander and Martinez-Conde 2019; Ditchburn and Ginsborg 1953; Martinez-Conde, Macknik, and Hubel 2004). Tremors are likely caused by the firing of motor neurons or the balancing process of the antagonistic eye muscles (Alexander and Martinez-Conde 2019; Spauschus et al. 1999; Ezenman, Hallett, and Frecker 1985). Ocular drift moves the eyes minimally and slowly in frequently changing directions (Rucci and Poletti 2015). Ocular drift’s amplitude is usually less than 45 arcmin and its speed is typically around 50 arcmin/s.

Microsaccades are involuntary, short, jagged eye movements that occur once or twice a second during fixation (Martinez-Conde, Otero-Millan, and Macknik 2013; Rolfs 2009). Some definitions of microsaccades include eye movements with an amplitude up to 1° or even 2° (Rolfs 2009). However, the term originally referred to movements with an amplitude of less than 30 arcmin that stabilise an image within the foveola, a tiny area of the retina where cones are the most densely packed (Rucci and Poletti 2015).

Saccades

To move our gaze from one fixation point to the next, we use saccades with much higher amplitudes. These saccades often reach peak speeds of over 300°/s, making them the fastest eye movements we can perform. Our most common eye movement pattern can be described as a saccade and fixation strategy, in which we regularly switch between looking at one object and making a quick, jerky eye movement to another location (Michael F. Land, Mennie, and Rusted 1999).

In general, saccade amplitude can vary widely, depending on the ongoing situation (Michael F. Land and Tatler 2009): When reading, small saccades are sufficient, whereas when preparing a meal, most fixations fall on task-relevant objects. Thus, the amplitudes of the saccades depend on the arrangement of the environment (Schütz, Braun, and Gegenfurtner 2011; Michael F. Land, Mennie, and Rusted 1999; Hayhoe et al. 2003). Even objects at a distance of 50° can normally be reached within one eye movement (Michael F. Land 2009; Benjamin W. Tatler et al. 2011). Up to a certain amplitude range, the peak velocity increases linearly with increasing amplitude, which is also referred to as the saccadic main sequence (Smit, Van Gisbergen, and Cools 1987; Gibaldi and Sabatini 2021; Bahill, Clark, and Stark 1975). Beyond this range, peak velocity saturates.

Saccadic targeting is not perfectly accurate. When looking closely at the saccade landing points of saccades directed at small peripheral objects, it can often be observed that saccades slightly undershoot their target location. Typically, this is then corrected by a second short saccade so that the fovea is then focused precisely on the new fixation object (Benjamin W. Tatler et al. 2011).

Saccadic Suppression

During saccades the retinal image is moving. Surprisingly, however, our visual perception is never interrupted by smeared phases during saccades (as compared to a moving camera with a slow shutter speed). Instead, we only perceive a continuous, clear stream of the objects we fixate. While a saccade takes place, visual input is suppressed (Ditchburn 1955; Wallach and Lewis 1966). Thus, a light flash that is only presented during a saccade cannot be seen (Bridgeman, Hendry, and Stark 1975; Latour 1962; B. L. Zuber and Stark 1966). The pupil also does not react to intrasaccadic brightness changes (B. L. Zuber, Stark, and Lorber 1966). Interestingly, saccadic suppression usually occurs a few tenths of a second before the saccade onset reaches its maximum, right at the start of the saccade and disappears gradually a few tenths of a second after it ends (Volkmann 1986; Diamond, Ross, and Morrone 2000). This time course changes depending on various factors, such as luminance characteristics of the background and the presented stimuli, the current light adaptation of the eye, characteristics of the stimulus, the position of the stimulus on the retina and the amplitude of the saccade (Volkmann 1986; Stevenson et al. 1986).

In voluntary saccades, saccadic suppression is stronger than in reactive saccades (Gremmler and Lappe 2017). Nevertheless, in most situations, we are able to recognise a previously perceived object after a voluntary saccade that has then shifted to a different position in the FOV. This allows us, for example, to select one of many apples on a tree, move our gaze, and immediately recognise the same apple again, even though it is now in a completely different position relative to the retina.

Thus, our mental representation of our environment is not tied to the object’s absolute position in space or to the available visual information during the movement (Irwin 1991). Instead, it seems we build our mental representation on assumptions that are constructed from visual information after the saccade:

For example, we do not notice when an object is displaced during a saccade if the displacement is less than a third of the amplitude of the saccade (Bridgeman, Hendry, and Stark 1975). Interestingly the same effect can be observed when the target is already displaced just before the saccade is initiated.

In other words, if a fitting postsaccadic target is found just after the saccade at a location not too far away from the object’s previous location, we assume that our environment is stationary (Deubel, Schneider, and Bridgeman 1996).

If no suitable target is found in the required spatio-temporal window, we seem to recalibrate the mental representation of our environment based on other information such as extraretinal signals (Deubel, Schneider, and Bridgeman 1996).

This recalibration leads to effects such as perceived spatial compression (Burr et al. 2010) This was the case in an experiment in which participants first saw a bar and a ruler. During a saccade, the bar was moved while the ruler disappeared. After the saccade, the ruler reappeared in the same place. The participants perceived the bar as immobile and assumed that the ruler had been moved after the saccade (Matsumiya and Uchikawa 2003).

In general, the compression effect occurs if visual references are available immediately after, rather than before or during, the saccade (Lappe, Awater, and Krekelberg 2000). It therefore seems to be mainly based on visual information that is available after the saccade.

A side effect of spatial compression can be that the number of elements perceived in the environment is reduced (Morrone, Ross, and Burr 2005; Ross, Morrone, and Burr 1997). This is because their perceived positions overlap due to the compression. Furthermore, the compression effect is not only limited to space, but can also influence time perception: During a saccade, the perception of time intervals can also be compressed. This can shorten the perceived duration by about half and even reverse the perceived temporal sequence of successive stimuli (Morrone, Ross, and Burr 2005; Burr et al. 2010).

Blinks

Another obvious example of suppressed visual perception is blinking. During blinking, the eyelid closes, the eyeballs rotate downwards and slightly translate towards the skull (Kirchner, Watson, and Lappe 2022). Although blinking, like saccades, occurs regularly and is easy to observe from the outside, we do not normally perceive the interruptions to our stream of visual perception. Instead, again it seems as if our visual system skips parts of our input. Suppression of visual input during blinking appears to be associated with an inhibitory signal from the brain triggered by efferent discharge during eyelid closure (Volkmann et al. 1982). After 100-150 ms, the eye usually returns to its previous position and the eyelid opens again (Volkmann 1986). The recovery of the suppression of the visual input during the blink occurs gradually over a period of 100-200 ms after the onset of the blink and is therefore somewhat faster compared to saccades (Volkmann 1986). Interestingly, there does not appear to be a hard-wired physiological mechanism that ensures visual stimuli are suppressed while we keep our eyes closed, as we can still perceive visual stimuli even with our eyelids shut. Instead, suppression seems to be mainly related to the closing of the lid. Although most light is blocked when the eyes are closed, it is thus possible to detect differences in brightness (Bierman, Figueiro, and Rea 2011; Sakai 2023). This works especially well with light sources with long wavelengths and can be tested easily when being outdoors by moving the head towards or away from the sun with the eyes closed.

In general, blinks can be clustered in different subtypes. During reflexive blinking, the eyelid is closed to protect the eye or to moisten it with tears (Evinger 1995). This blink reflex is, for example, triggered when air is blown into the eye or an object is quickly moving towards the face. Typically, we make 10–15 spontaneous blinks per minute, which is more than necessary to just prevent a dry cornea. Thus, we seem to also blink in situations without the need to protect the eyes. Moreover, our blink rate varies with different tasks (Karson et al. 1981; Doughty 2001). A consciously performed closing of one or both eyelids, is called a voluntary blink or wink. While we blink, we can also perform other eye movements. For example, it is possible to perform saccades during a blink (Kirchner et al. 2022; Rottach et al. 1998). Although the kinematics of these saccades are significantly altered, their parallel execution could be useful in reducing the time during which only very limited visual stimuli are available.

Gaze

To look at objects more closely, we move not only our eyes but also our head. In this thesis, combined eye-head movements will be referred to as gaze movements.

Horizontally, eye amplitudes can reach values around 50°, whereas head rotation amplitudes can even reach 175° (J. K. Wu Tiffany C. AND Tsotsos 2025). Thus, gaze shifts are useful to reach objects beyond our FOV (Thier and Ilg 2005; Foulsham 2015; Einhäuser et al. 2007).

In this case a coordinated head movement extends the saccade amplitude. However, a large proportion of eye and head movements are performed in opposite directions (Einhäuser et al. 2007; J. K. Wu Tiffany C. AND Tsotsos 2025).

This is, for example, the case, when we fixate a stationary object while moving our head. Then, vestibular movement signals that can detect head movements are transmitted to the eye movement control circuits (Cullen and Roy 2004).

This signal is the basis for the VOR, through which our head movements are compensated by counter-rotating eye movements with low latency, so that a more stable retinal image is generated (Fetter 2007). However, the image cannot be perfectly stabilised across the entire FOV because the eye position also shifts due to head movement, as the eyes are not positioned in the centre of the head’s axis of rotation (Harris 1994). Therefore, this translational shift must either be ignored or stabilisation must be optimised for a specific object of interest.

To coordinate gaze movements accurately, the neural circuits for head and eye movement are linked closely: A case report on a patient with eye muscle paralysis even showed that when eye movements are not possible, saccade-like head movements are performed instead (Gilchrist, Brown, and Findlay 1997).

During head movements, the speed of eye drift increases threefold, suggesting that drift also helps to compensate for head movements (Rucci and Poletti 2015; Skavenski et al. 1979; Aytekin, Victor, and Rucci 2014). The properties of the microsaccades, however, do not change, so it appears that they capture fine spatial details of objects independently of head movements (Jolly et al. 2023).

Gaze Stabilisation

When we try to fixate objects that are moving through the environment, we move our eyes to compensate for their motion.

Such ocular following responses are evoked with very short latencies by sudden visible motion (Miladinović et al. 2024). The following continuous slow eye movement that tracks a moving target is called smooth pursuit (Haarmeier and Thier 1999; Haarmeier et al. 1997; Drewes, Feder, and Einhäuser 2021). To reach a moving target, our eyes can first adjust their velocity to the objects’ speed and direction, then a saccade can bring the eyes to the target’s position while the smooth pursuit continues (Lisberger 2015; Rashbass 1961).

During pursuit, the eyes can smoothly compensate for speeds of up to 30°/s. At higher speeds, smooth pursuit movements are usually accompanied by catch-up saccades in motion direction.

Interestingly, tracking an object using pursuit eye movements requires a centrally-derived motion percept of the target (Steinbach 1976). This can be based on visual signal of the object’s motion, such as the visible edge of the tracked object. We can also follow acoustic or sensory signals, such as when our eyes follow our own hand movements or a sound in the dark (Hashiba et al. 1996; Berryhill, Chiu, and Hughes 2006; Rashbass 1961; Lisberger 2015). If perception of the object movement fails, we can no longer follow it smoothly with our eyes. Instead, the object is tracked through a series of saccades and smooth pursuit phases in an attempt to follow it, while the target appears to jump in space (Koerfer, Watson, and Lappe 2024).

When we perceive motion anywhere across our FOV, for example when looking out of the window of a train, a similar reflex called optokinetic nystagmus is triggered. Initially, we tend to again initiate a smooth eye movement, whose dynamic is similar to pursuit (Magnusson, Pyykkö, and and 1986). Next, however, our eyes do a fast eye movement in the opposite direction of the visible motion, where the cycle restarts and another smooth movement in motion direction begins (Drewes, Feder, and Einhäuser 2021). As a result, we perceive the object clearly, although everything in our FOV is in motion.

Eye Tracking

Eye movements are an essential part of our visual system. When we observe another person’s eyes, we can easily detect whether a saccade or blink is occurring. Although we intuitively have a rough idea of where the other person is looking, it is difficult to analyse the rapidly changing movements more precisely based on manual observation alone. Consequently, the first attempts at automated, objective measurement of eye movements were made over a century ago and have been supplemented over the decades by numerous further developments and new methods of eye and gaze tracking (Hutton 2019).

Initially, both Huey and Delabarre used a mechanical device consisting of a contact lens with rods attached to it, which marked the ongoing eye movements on a kymograph cylinder (Hutton 2019; Huey 1898; Delabarre 1898). With this method, slow eye movements could be recorded, although the weight of the measurement equipment probably added noticeable strain on the eye muscles and therefore influenced the measured eye movements to some extent. The weight could be reduced further, when Robinson introduced contact lenses with wire coils inside (Robinson 1963). Their movement could be measured using an electromagnetic field surrounding the head of the subject. To this day, this method offers a comparatively high degree of precision, with approximately 5 arc seconds over a limited range of approximately 5° (Young and Sheena 1975a).

Although wearing comfort has improved significantly and today’s lenses no longer require anaesthetic drugs such as cocaine to be applied to the cornea before insertion, wearing a comparatively thick lens can still be unpleasant for the subject and insertion requires practice (A. T. Duchowski 2007).

A less invasive approach is electrooculography (EOG). This eye tracking method makes use of the standing electric potential between the anterior positive and posterior negative pole of the eye. Using electrodes placed around the eye, one can determine the eyeball orientation with respect to the head position, although it lacks some spatial accuracy, especially on the vertical axis (A. T. Duchowski 2007). This data can be collected at very high sampling rates. Unfortunately, the signal often drifts slightly over a short amount of time since the electric potential of the pigmented epithelium also changes based on the amount of light that enters the eye. Therefore, although accurately measuring the eye direction with EOG requires very frequent recalibration, the method is well suited to detect saccades and blinks with low latency (Bolte and Lappe 2015; Kirchner et al. 2022).

The first predecessor of today’s most common eye tracking method was the photochronograph by Dodge and Cline (1901) (Dodge and Cline 1901; Hutton 2019). Instead of attaching measuring equipment directly on to or close to the eye, this method measured movements remotely. First, a light source was directed at the user’s eyes. Since eyes are not perfectly spherical, the reflection of light on the cornea changes depending on the direction of gaze. The reflected light was then exposed to a film plate that was later analysed. Therefore, this method was only suitable for visualizing the accumulated eye movements over a certain period of time. In revised devices, horizontal and vertical components of the eye reflection could be separated and exposed with higher temporal resolution on photographic tape or film (Judd, McAllister, and Steele 1905; Buswell 1935; Macele and Mueggenburg 2024).

Later, Yarbus (1967) developed contact lenses that were attached to the eye with suction cups and to which either eye-fixed visual stimuli or tiny mirrors were attached, which also made it possible to record eye movements on film (Hutton 2019; Benjamin W. Tatler et al. 2010).

Eventually, Yarbus (1967)’ approach led to a number of different remote eye trackers, all of which used different types of light reflections from the eye to determine the direction of gaze. To allow the subject to still see a stimulus while using such an eye tracker, the remote light source was soon replaced by infrared light. Infrared limbal eye trackers exploit the fact that the white sclera of the eye reflects more light than the iris, using measurements from a high-frequency infrared sensor as a signal for the direction of gaze on the horizontal or vertical axis (Hutton 2019).

On closer inspection, not just one, but four visible reflections of incoming light (also known as Purkinje reflections) can be observed in the eye: these come from the anterior and posterior surfaces of the cornea and the lens of the eye. Dual Purkinje eye trackers measure the disparity between the first reflection from the cornea and the fourth reflection from the eye lens (Cornsweet and Crane 1973; Richardson and Spivey 2008; Hutton 2019). In the first Purkinje eye trackers, this measurement was done mechanically: Adjustable mirrors were aligned in such a way that the two reflections overlapped. This calibration procedure allowed accurate (up to 1 min of arc) and fast eye direction estimations and was therefore suited to measure microsaccades, if the head was stabilised carefully.

As soon as camera equipment enabled real-time video recordings of eye movements, VOG became the most popular method of estimating gaze direction based on light reflections of the eye (Richardson and Spivey 2008; Young and Sheena 1975b; Merchant, Morrissette, and Porterfield 1974). The most common technique in VOG is dark pupil tracking. As in limbal eye trackers, this approach makes use of the fact that different features of the eyes reflect different amounts of light. In this case, since the pupil reflects even less light than the iris, the pupil can be detected as a dark small ellipse in each frame of the recording. This ellipse, can then be compared with the first corneal reflection that is induced using an infrared emitter. The distance between the two can be calibrated to estimate the gaze direction. In some devices, multiple infrared light sources and therefore multiple first order Purkinje reflections are used (see Figure 4). Moreover, the dual Purkinje eye tracker has been further developed with the help of camera technology: Instead of analogue mechanical parts, a digital dual Purkinje eye tracker uses an optical setup (with no moving components) and a digital imaging module (R.-J. Wu et al. 2023). Again, to track eye movements meticulously, the offset of the first and fourth Purkinje reflections are used, now obtained from a digital image.

Thanks to technological advances in cameras, processing hardware and batteries, VOG devices can be so lightweight that they can be integrated into wearable devices to track eye movements in natural and virtual environments. However, apart from contact lenses with wire coils, the presented tracking methods only measure the eye movements relative to the head. To track the gaze relative to a person’s surroundings, the orientation and position of the head must also be taken into account. In the case of wearable eye trackers, this can be done using an outward-facing camera, from whose recordings of the FOV the tracked gaze direction can later be visualised. When researching human behaviour in VEs, the data from the position tracking system, the IMU and the eye tracker can be combined to obtain a gaze direction signal. This signal can be both recorded and also used to make the VE react to the user’s eye movements.

Gaze-contingent Displays

Even before Sutherland built his first HMD prototype, he envisioned the capabilities of the ultimate display of the future (Sutherland 1965). Interestingly, despite the limited technology of the 1960s, his vision included a display that was capable of eye tracking. In his essay, Sutherland (1965) laid out an experiment in which the device would display a triangle whose edges would round off as soon as a user fixated them. He hoped that this approach would contribute to a better understanding of the mechanisms of human vision. To implement this idea, the eye tracking signal must be available in near real time so that visual stimuli can be updated in a timely manner. Such so-called GCDs were developed a few years later in the 1970s, initially for reading research (McConkie and Rayner 1975; Rayner 2014; Andrew T. Duchowski, Cournia, and Murphy 2004). However, these GCDs were not attached to the head, but worked with remote eye trackers and screens positioned on desks. Later, the same principle was applied again with the aim of improving system performance when displaying computationally intensive stimuli (Murphy and Duchowski 2002). In foveated rendering, for example, the level of detail or resolution of visual content in the periphery is reduced. It is therefore important that eye movements with a large amplitude in particular are detected quickly so that the image can be updated accordingly, giving users a detailed, high-resolution image in the centre of their FOV (Patney et al. 2016; Andrew T. Duchowski, Cournia, and Murphy 2004; O’Sullivan, Dingliana, and Howlett 2003).

In addition, GCDs met the expectation that they could deepen our understanding of our visual system. They enabled experiments on the time span of the perceptual window, which showed that we process up to 15 letters in reading and about 5° in searching (Bertera and Rayner 2000; McConkie and Rayner 1975). With the help of GCDs, it became possible to measure how vision degrades from the fovea to the periphery. It was found that this can vary depending on image characteristics such as spatial and temporal resolution, colour, luminance and contrast (Andrew T. Duchowski, Cournia, and Murphy 2004; Loschky and Wolverton 2007). Further experiments with desktop-based GCDs showed that contrast sensitivity for natural stimuli is significantly reduced compared to static images (Dorr and Bex 2011), gaze-contingent contrast influences colour perception (Mauderer, Flatla, and Nacenta 2016) and high cognitive load leads to a loss of accuracy evenly distributed over the eccentricities of the retina (Gaspar et al. 2016).

GCDs were also used in retinal visual impairments simulations that mimicked macular degeneration due to ageing (Geisler and Perry 2002; Vinnikov, Allison, and Swierad 2008). With the help of such applications, circumvention strategies of the visually impaired could be better understood and environments suitable for the disabled could be evaluated more easily. In addition, personally experiencing impairment simulations could help to raise awareness and increase empathy among the public. Finally, gaze-contingent HMDs could compensate the pupil swim effect and improve visible cues for the perception of depth in VR (Konrad, Angelopoulos, and Wetzstein 2020; Rui et al. 2023).

An ideal gaze-contingent HMD would have a refresh rate above the fusion flicker perception threshold and an eye tracking latency below the frame to frame interval. This would require very fast and expensive sensors that consume a lot of energy, heat up quickly and cannot be operated for long periods on battery power. However, because we use saccades to move our eyes from one fixation to another and vision is suppressed, during and shortly after each saccade, it is possible to present gaze-contingent, world-fixated stimuli using hardware with higher end-to-end latencies.

The maximum possible latency of the GCD at which the user cannot see the adjustment of the stimuli also depends on the saccadic main sequence. This is because the phase of maximum suppression and the recovery of vision take longer with saccades with higher amplitudes. (Volkmann 1986; Stevenson et al. 1986). Thus, a voluntary saccade with an amplitude of 8° results in suppression times around 40 ms after saccade onset, whereas a voluntary saccade with an amplitude of 16° results in suppression times around 80 ms after onset (Stevenson et al. 1986).

This, however, does not mean that larger saccades are always helpful when setting up a gaze-contingent experiment. For example, in foveated rendering, objects in the periphery are displayed with a lower resolution, which is why a degradation factor must be implemented that adjusts the resolution of a stimulus based on the distance to the centre of the FOV. Since saccades with longer amplitudes land further away from the original fixation, gaze-dependent manipulations such as low resolution may appear more obvious to the user after a saccade, if updated even shortly after the saccadic suppression period, while small saccades may still fall in a higher resolution area where the manipulation is less likely to be detected even if the visual content is updated too late (Loschky and Wolverton 2007). Therefore, the end-to-end latency from an eye movement to the change in display content, the expected saccade amplitude and the gradient of the manipulation must be carefully considered when setting up gaze-contingent manipulations that should not be perceived.

Requirements for Gaze-contingent HMDs

Since some currently available HMDs are equipped with eye trackers, it may be possible to use them as gaze-contingent HMDs. In contrast to desktop-based gaze-dependent setups, these could enable stimuli to be presented at specific positions in the FOV, independent of both eye movements and head movements, which would allow the presentation of gaze-contingent stimuli in VEs.

In order to assess whether this is possible, it is first necessary to estimate the minimum latency requirements for GCDs. When using a remote eye tracker and a desktop screen to create a gaze-contingent setup, no differences in eye movements were observed between an artificial latency of 5 ms and 15 ms (Loschky and McConkie 2000). At an added latency of 45 ms, however, Loschky and McConkie (2000) observed slightly longer fixation durations, although even then, the performance in a search task was not altered.

In a detection task of gaze-contingent peripheral blur, performance did not significantly decrease until a latency of 60 ms was reached (Loschky and Wolverton 2007). Although these observations do not lead to a clear maximum latency for a GCD, they can give an idea of the required latency range which is between 15 and 60 ms.

Displays in currently available HMDs have a frame rate of about 90 Hz (11.1 ms). However, to implement a VR-GCD, not only the frame-to-frame interval, but the overall end-to-end latency from a gaze shift to an image that is displayed on the HMD needs to be considered. In that time frame, multiple processing steps need to be done. First, motions from all available tracking sensors need to be recorded. Then, the tracked signal needs to be used to estimate the current state of, head and eyes with accuracy and precision. Next, the VE needs to be updated based on this estimate. Finally, the updated VE needs to be rendered to a stereo-image that is then displayed on the HMD.

The latency between a change in position and a corresponding change on the display can be determined externally by simultaneously recording the start of the HMD’s position change and the display with a high-speed camera. For head tracking, current HMDs have motion to photon latencies between 21 and 42 ms, which is below the required latency range for GCDs (Warburton et al. 2023; Niehorster, Li, and Lappe 2017).

Because the HMD occludes the eyes, a different method is required to measure the latency of VR eye trackers. We present this method in the following chapter. The development of HMDs has come a long way since 1968. Prototypes have become commercially available products that are used for research. At the same time, by continuously innovating new eye tracking methods, we have increased our knowledge about our visual perception and how it is influenced by the way we move our eyes. In order to investigate the potential use of VR with eye tracking in psychophysical research, for example to present strictly controlled visual stimuli while simultaneously measuring head, eye and walking movements, we must be able to estimate the latency of the tracking sensors. In the following study, we address this need by measuring the time required by currently available HMDs to register an actual saccade, record the corresponding eye tracking data and finally display a change on the screen depending on the recorded data.

References

• ← Previous
• Next →