Many eye movement patterns can be mapped to different task demands. Some task-related gaze patterns, such as turning, are very similar in different scenes and seem to be part of our natural motor programme for the task. Thus, it is possible to predict future actions based on our eye movements to a certain extent.

However, this does not apply to all eye movements and does not fully explain how we decide which objects to inspect next. How do we, for example, decide which object to look at first when there are multiple task-relevant options available? Where does our gaze wander when we look around without any particular intention?

Some of these gaze movements can be explained by general eye movement patterns: For example, when looking at images we seem to have a bias for making horizontal eye movements (Foulsham, Kingstone, and Underwood 2008). This seems to also be the case, while we are searching for something (Gilchrist and Harvey 2006).

Furthermore, we seem to choose locations close to our current fixation as our next fixations when we look around in a natural environment (Tatler et al. 2011; Tatler, Baddeley, and Vincent 2006; Gajewski et al. 2005). One possible explanation for this pattern is that we adjust our fixation targets based on the available information and thus on the basis of the distribution of the resolution of the retina (Najemnik and Geisler 2008).

Even though these biases can explain some of our eye movements, they are not sufficient to fully explain where we intuitively look and which objects are particularly eye-catching. In some situations, this seems to happen almost automatically. When a visual stimulus suddenly appears in our FOV or starts to move, we tend to look at it (Irwin et al. 2000; Theeuwes et al. 1998, 1999; Tatler et al. 2011).

Similarly, visible optical flow intuitively triggers saccades and subsequent smooth pursuit movements towards the focus of expansion (Chow et al. 2021). To prepare such a reactive saccade towards an appearing object, we only need between 150 and 200 ms (Smit, Van Gisbergen, and Cools 1987). In contrast, the preparation time for a voluntary saccade towards a memorised or continuously visible target while viewing a scene is likely to be somewhat longer. However, the exact preparation time is difficult to quantify, as we do not know the exact moment when the decision to move the eyes is made (Gremmler and Lappe 2017). If we consciously alter a reactive saccade, for example, by making an eye movement to a memorised location in the opposite direction of an appearing stimulus (anti-saccade), our saccade latency increases even further, reaching values between 300 and 500 ms (Smit, Van Gisbergen, and Cools 1987).

Whenever we make a saccade towards an interesting object, our visual system must somehow select a future fixation point and plan an eye movement towards it, even before it can be seen with the high resolution of the fovea. One theory describing this process is the principle of attentional shift. According to this principle, before each eye movement, our focus first shifts to the location of the next fixation, even before our eyes move. Objects at the location of attention are then processed in a targeted manner. Visual attention could therefore be interpreted as a process that highlights part of our sensory impressions in order to make better use of the information available at that location.

In so-called cueing experiments, participants have to press a button as soon as they perceive a visual target that could either be at an expected position (cued) or somewhere else. If the target appears at a location that was previously indicated by a cue so that the participants could shift their attention before answering, the reaction time is shortened (Posner, Snyder, and Davidson 1980). In the first experiments of this kind, participants were instructed not to move their eyes. In later experiments, in which participants were asked to fixate on the target when selecting it, it turned out that cueing also had an influence on the timing of the onset of saccades towards the target (Shepherd, Findlay, and Hockey 1986). In other words, if our attention is already focused on a location that will later be the target of a saccade, the eye movement can be performed more quickly. First, we seem to make a voluntary shift of attention towards the cue. As instructed, our eyes follow this shift to the same location, once the target appears. In contrast, if we first receive an attention cue on the right side, while a saccade target subsequently appears on the left side, our saccade preparation time increases. In this case, we seem to make a reactive shift of attention towards the non-target followed by a second voluntary shift of attention towards the target in preparation for the saccade that follows. This additional shift of attention to a different location just before the saccade increases the latency.

Interestingly, this also means that shifts in attention can occur even when we direct our gaze to another point (covert attention) and when no saccade is ever executed towards the point of focus. Thus, it is possible that our attention has already shifted while our eyes are still fixating the previous target. While we focus our attention on a new object, it is also possible that, at least for a short period of time, two objects are simultaneously in the focus of our attention (Coral Gabbay and Lamy 2019). Our eye movements therefore do not always reveal where we are focusing our attention and lag slightly behind any ongoing shifts in attention. Accordingly, the terms ’fixation’ and ’focus of attention’ should not be used interchangeably in the context of attention research.

When we focus our attention on a specific location, we can better and more rapidly process targets at that position (Boynton 2005; Ede, Lange, and Maris 2012; Posner, Snyder, and Davidson 1980). However, there is also plenty of evidence that our representation of the visual world is built from a combination of the content we are currently paying attention to and a non-selective global view of what is visible in our FOV (Jeremy M. Wolfe et al. 2011). For example, when looking at a forest, we are able to perceive and process the scene as a whole without looking at the details of any individual tree using the so-called non-selective pathway of our visual system (Greene and Oliva 2009). However, if we want to focus our attention on a tree to inspect individual branches or leaves, it would require us to use the selective pathway (Evans and Treisman 2005). As a result, shifting attention is an important mechanism to process the details of objects in our FOV.

This in turn leads us to a series of new questions: why do we focus our attention on certain stimuli and not others? What are the characteristics of objects that we intuitively focus our attention on? And finally: how are these processes implemented in our brains?

Visual Search and Salience

In many situations, even before we enter a room, we already have a preconception of what we want to do there and which objects we want to interact with. To do this, it is important to be able to locate these target objects in the FOV. The psychophysical paradigm of visual search provides some insights in the basic functions of this process.

Visual search was originally developed to investigate the capacity limits of the attention system (Treisman and Gelade 1980). During these experiments, participants are asked to search for a target object among a set of distractors as quickly as possible. Usually, a search ends either immediately because the search target pops out quickly to the participant, or after a longer search period.

One explanation for this pattern could be that objects in the FOV are either processed serially one after the other, or in parallel.

In their theory of guided search, Jeremy M. Wolfe and Horowitz (2017) assume that the type of search (parallel or serial) depends on the salience of the target. They postulate that salience increases with the visible difference between target and distractors and also with the homogeneity of distractors. In other words, if a target is particularly unique and stands out from very similar distractors, it pops out and attracts our attention. Thus, we can solve this search task in parallel. The most important property dimensions in which targets and distractors can be unique or similar are referred to as the basic (or guiding) features. These special features are capable of drawing attention in a way that other factors cannot (Jeremy M. Wolfe and Horowitz 2017; Higuchi et al. 2019): The list of guiding features includes colour, sudden appearance, motion onset and target shape. Other factors that can create such a pop-out effect in search tasks are object size, stereoscopic depth and tilt, optic flow, luminance, line termination and curvature.

Indeed, individual targets that differ from the distractors in one of their basic characteristics, usually catch our attention immediately (Treisman and Gelade 1980; Treisman and Sato 1990; Jeremy M. Wolfe and Horowitz 2017) and can be found without or within only a single saccade (Binello, Mannan, and Ruddock 1995). Moreover, these stimuli seem to automatically draw our attention towards them since we intuitively execute saccades towards their location when we detect them in a search display (Hulleman and Olivers 2017).

If no salient target is available because the distractions do not have at least one clear basic guiding feature that differs from the target itself, individual stimuli or groups of stimuli are examined one after the other. In these conjunction searches, the length of the search, during which we typically perform multiple saccades and fixations, is influenced by the time required to examine and reject each distractor (Jeremy M. Wolfe and Horowitz 2017).

The Visual Pathway

To understand how attention shifting, target perception and, to a certain extent, eye movement planning work, it is also necessary to examine the neural structures that perform these processing steps. This section provides an overview of the various elements of the visual pathway in our brain and how these structures are linked to some of the behavioural findings presented in the previous sections.

After light is detected by the photoreceptors, the biochemical signal, which from then on represents our visual input, is transmitted first via bipolar and then via ganglion cells, while horizontal and amacrine cells form lateral connections between the retinal layers (Prasad and Galetta 2011). To maintain high spatial acuity, the input from the central areas of the FOV is passed on to individual bipolar cells, while in the periphery one bipolar cell has a receptive field that combines several photoreceptors (Prasad and Galetta 2011). Next, visual information is carried from the eyes towards the brain via the optic nerve which is formed by the axons of the ganglion cells (Prasad and Galetta 2011). At the optic chiasm, the information from both eyes partially crosses sides, so that from here on a combined image from the perception of both eyes is available.

About 10 % of retinal axons from the optic nerve end in the tectum of the midbrain and in the hypothalamus (where they most likely contribute to the regulation of our circadian rhythm (Joukal 2017)) the pretectal nuclei and the SC. Together with input from other sensory organs, such as the vestibular and somatosensory systems and the visual cortex, these connections are important for various ocular functions. For example, they regulate the pupillary reflex and the VOR (De Moraes 2013; Joukal 2017). Of these structures, the SC in particular has been shown to be a likely candidate for coordinating our head and eye movements and may even be involved in the preparation of saccades. Indeed, a number of experiments with monkeys suggest that the brain cells in the SC contain a representation of the FOV and are involved in planning and executing our eye movements (Bollimunta, Bogadhi, and Krauzlis 2018; Groner and Groner 1989; Krauzlis, Lovejoy, and Zénon 2013). Already in (1972), Goldberg and Wurtz (1972) observed an increase in the activity of individual cells in the SC as soon as the eyes of a monkey began to move. Artificial lesions of the same area led to an increase in saccade latency of 150-300 ms when a saccade was directed to the receptive field of the lesioned brain cells. At the same time, saccade velocity decreased slightly after the lesion (R. H. Wurtz and Goldberg 1972). Interestingly, however, targeting accuracy remained unchanged. In addition, further experiments showed that when a monkey performed hand movements instead of eye movements towards a target, the firing rate of cells in the SC did not increase. Accordingly, the previously observed cell discharges were linked to the preparation of saccades, but not attention shifting (Groner and Groner 1989; R. H. Wurtz and Mohler 1976).

Further down our visual pathway, the majority of all retinal axons (90 %) form the optic tract and send signals to the LGN of the thalamus (Joukal 2017).

Although neighbouring cells can inhibit each other, the visual pathway still appears to run mainly unidirectionally from the eyes towards higher-level areas of the brain. Thus, up to this point, the activity of the cells involved is mainly driven by the incoming light. In the LGN itself, however, the retinal axons account for only 5-10 % of the synapses and most of them originate from upward-downward modulating connections from other brain areas such as the reticular nucleus of the thalamus, the pulvinar nucleus and the visual cortex (De Moraes 2013). Retinal signals in the LGN travel to the visual cortex via three pathways. The parvocellular pathway that includes the information for high spatial acuity and red-green colour vision, the magnocellular pathway that holds the signal for achromatic visual sensitivity and motion vision and a third pathway via heterogeneous koniocellular tracts (Solomon 2021). We still know very little about the koniocellular pathways, but the lower sampling density suggests that they could provide only a rough sketch of the retinal image, which could still be used to regulate the other pathways and perhaps emphasise particularly important areas of the retinal image (Solomon 2021).

Processing in the Visual Cortex

From the LGN visual information is transmitted to the two occipital lobes, where the more retinotopic areas are arranged in parallel mirror-symmetrical bands (Grill-Spector and Malach 2004).

Current research suggests that the organisation of visual information in the cortex follows two different basic principles: hierarchical processing and functional specialisation (Grill-Spector and Malach 2004).

Thus, a particular brain function, such as recognizing a particular shape, is typically achieved through a stepwise process. The information is first reflected locally and retinotopically in the brain and then transformed into more abstract and even multimodal representations through a sequence of processes (DeYoe and Van Essen 1988; Grill-Spector and Malach 2004).

Therefore, many of the early cortical areas of our visual system are defined by their retinotopic maps, whereas others are defined by their function, like their preference for certain classes of visual images such as faces or scenes (Winawer and Witthoft 2015; Kanwisher 2010; Wandell and Winawer 2011).

In general, the average size of the receptive fields in the retinotopic regions is smallest in V1. From V2 to V3, V3a and V4, it then increases continuously (Grill-Spector and Malach 2004; Press et al. 2001; Smith et al. 2001; Kastner et al. 2001).

Even in the visual cortex, the representation of the centre of the FOV is magnified. The connections of 1 mm² around the fovea (representing the central 10° or about 2 % of the total FOV) represent about 60 % of cells in V1 (De Moraes 2013).

Interestingly, while V1-V3 have representations of the full FOV, the receptive fields of cells in V4 are mostly located within 3-4° of the fovea, which suggests that this area is specialised for functions that depend heavily on foveal and para-foveal vision (Winawer and Witthoft 2015).

Anatomically, the foveal representations in V1, V2, V3 and V4 are very close to each other at the occipital pole, which suggests they are part of one common map cluster (Winawer and Witthoft 2015).

The visual cortex is classified not only on the basis of its anatomical structure but also on the basis of its functions.

Broadly speaking, the visual pathway in the brain is often divided into two pathways (Joukal 2017):

The ventral ‘what’ pathway extends from the occipital lobe downwards to the temporal lobe and is responsible for functions related to the recognition of object information such as shape, contrast and colour.

The dorsal ‘where’ pathway extends from the occipital lobe upwards to the parietal lobe and is responsible for functions such as spatial features and movement.

Grill-Spector and Malach (2004) investigated hierarchical processing based on retinotopy, motion sensitivity and object selectivity: They found that early retinotopic areas V1, V2 and V3 showed a high degree of retinotopy, but only low specificity for stimulus motion and form.

Thus, in principle, V1 and V2 are active in every type of visual task and activation in V1 increases, for example when the contrast of a visual stimulus increases.

In V1 itself, there are three different systems that fulfil separate functions (Joukal 2017). The first system is formed by three cortical columns and is specific for binocular vision and is thus fundamental for depth perception. The second system consists of cells with identical retinal positions and preferred orientation axes and enables the perception of movement. The third system is responsible for the perception of colours and shapes.

As we move from one area to the next, features to which the neurons respond become more complex. Some functions are also represented in several areas: For example, in both macaque monkeys and the human brain, there is a relationship between the areas V1 /V2 and higher areas including V4 for colour processing and the areas V1 /V2 and V5, MST and V3a for motion perception (Zeki et al. 1991; Grill-Spector and Malach 2004; Winawer and Witthoft 2015; Brewer et al. 2005). In general, intermediate visual areas such as V3a and V4 show a lower degree of retinotopy and, to some extent, stronger responses to objects and moving low-contrast gratings (Grill-Spector and Malach 2004).

In subsequent studies, V4 neurons were associated with a range of other functions (Roe et al. 2012), such as selectivity for binocular disparity (Hinkle and Connor 2001), disparity-defined shapes in random-dot stereograms (Hegdé and Van Essen 2005) and three-dimensional orientation of bars (Hinkle and Connor 2002). A similar area was found in humans (Hansen, Kay, and Gallant 2007) and could also be associated with the encoding of shapes (Dumoulin and Hess 2007), surfaces (Bouvier, Cardinal, and Engel 2008) and object-selectivity (Konen and Kastner 2008).

For higher-order areas further down the processing stream, Grill-Spector and Malach (2004) found that retinotopy continues to decrease, while a higher degree of specialisation develops. Neurons in V5 showed a strong preference for moving over stationary stimuli, while having no object selectivity. Neurons in LO respond more strongly to objects compared to scenes and textures, but show little response to moving compared to stationary gratings (Grill-Spector and Malach 2004).

Neural Basis of Attention

With regard to the results of the visual search, we can now ask ourselves which areas of our brain recognise the features of, for example, a salient search target. It seems plausible that neurons that enable basic feature recognition rely on visual input, but not necessarily multimodal input. The fact that we are able to use these features in a parallel search to quickly identify the location of the target anywhere in our FOV suggests that these features are represented in a brain area that is retinotopically organised. In addition, many guiding features require at least the detection of contrast, colour and motion. These considerations suggest that the neurons that can control our attention may be located in one of the intermediate areas of the visual cortex.

In line with these thoughts, Bichot, Rossi, and Desimone (2005) conducted experiments in which macaque monkeys searched for a target defined by colour and shape, and found interesting neurons in V4. Throughout the search phase, the neurons showed enhanced responses as soon as a stimulus in their receptive field matched a feature of the target. Thus, this detection mechanism worked in parallel across the entire visual field display, even before the target was localised. This could potentially represent the parallel processing of all stimuli in our FOV while we search. The neural response of these attention neurons encodes position and specific target features. Signals are sent from V4 to other areas, such as the inferotemporal and frontal cortex. This signal could therefore ultimately direct spatial attention to a potential target and trigger an eye movement to the same location. This mechanism could explain how, for example, when attention is directed to a stimulus with lower contrast, the influence of this stimulus can outweigh the influence of a competing stimulus with higher contrast (Reynolds and Desimone 2003).

Interestingly, Bichot, Rossi, and Desimone (2005) also found that the same neurons responded more strongly to potential targets selected for saccades or foveation, which could potentially represent shifts of attention during serial search when we move our eyes from one potential target to the next. This is consistent with results by Theeuwes, Van der Burg, and Belopolsky (2008) which suggest that detecting the presence of a coloured target is always associated with a shift of spatial attention to the location of the object (Theeuwes 2010; Theeuwes, Van der Burg, and Belopolsky 2008).

Therefore, V4 is likely an important area for mediating attentional effects that recognises or is indirectly involved in detecting important features and then uses them for attentional mechanisms in a spatially and functionally specific manner (Roe et al. 2012). Interestingly, this list of selective features that have been found in V4 could potentially also explain some of the results in visual search. In particular, it might shed light on why some unique features of a given target create a pop-out effect while other features lack the capability to make a target salient.

The neural activity found in V4 in monkeys could explain findings of conjunction searches in which the target does not pop-out and a serial search is necessary. In these searches, the distractors, which shared some but not all characteristics of the target, led to neuronal activity similar to that of the actual target (Bichot, Rossi, and Desimone 2005). The positions that the animals then worked through serially corresponded to the activity map resulting from the receptive fields of these neurons. In other words, the serial search was the result of several target candidates based on a basic feature-guided parallel pre-selection. In summary, the neurons found in V4 seem to enable an important part of our attention processes and have a receptive field that encompasses all potential stimuli of a search display in the FOV. Thus, one could conclude that our attention system also operates in a retinal coordinate system.

However, when we search for something in the real world, our target is often not yet visible in our FOV. Instead, we often use a combination of eye and head movements to inspect different parts of our environment. Therefore, we might ask ourselves what guides our head movements? How does the attentional system guide our eye movements while we move our head to extend our FOV? From behavioural observations, we already know that both types of movements are closely linked and that we can intuitively plan ahead and perform our comparatively slow head movements while inspecting our surroundings. Studies on walking have shown that we can gather knowledge about the structure of our environment and even about other people in it, and use this knowledge to adjust our gaze distribution accordingly. However, how exactly our attention system deals with head movements and how these influence our eyes during a search is still an open question.

In Study [III], we used an HMD with eye tracking capabilities to closely monitor eye and head movements while participants performed a visual search in a controlled and simplified VE.

References

• ← Previous
• Next →