Initial result from the Robotic Implementation of Object-Based Visual Attention workgroup at Capocaccia Neuromorphic Workshop 2011.

A robotic "eye" makes saccades to the shapes (triangles or squares). The robot only uses shape information to distinguish between the different targets.

A reward is given for making a saccade to a triangle and the system learns to associate the reward with the triangular features (these are 6 x 6 pixel patches from the image).

The key feature of this model is that the scene is segmented at a very early stage of processing. Feature saliencies which belong to the same segment compete against the feature saliencies of different segments.

Top left: Image recorded from the camera at each step
Top right: Visual saliency map
Bottom: Graph of the cumulative number of rewards over time for 100 saccades

Object-based attention in a visual search task. A camera, mounted on a pan-tilt unit, searches for a rewarding digit. The video is presented in three epochs: in the first epoch no reward is available, in the second a reward is given for a foveation to the digit 2 and in the thrid epoch a reward is given for a foveation to the digit 6.

A working memory with a feature-based topology is used to bias competition between locations on the visual saliency map towards rewarding features.

Upper: image captured by the camera.
Bottom left: neural activity in the feature-based working memory layer.
Bottom right: location of the current saccadic target in the feature space.

Relevant publication: Brohan, Cope, Gurney & Dudek, 2010

Earlier version of the object-based visual attention model, recorded at Capocaccia Neuromorphic Workshop (2010). The laser beam, which is invisible to the robot, shows the location of the fovea. Saccades are preferentially directed towards the digit 6.

Relevant publication: Brohan, Cope, Gurney & Dudek, 2010

Recognition of paper digits on a neutral background. This is the basic recognition algorithm which was later used in the digit-based visual attention and search model.

Left: input video stream.
Right: identities and locations of recognised digits.

Relevant publication: Brohan, Cope, Gurney & Dudek, 2010

Feature-based visual search and attention. A virtual eye makes a series of saccades to different colour cues and an internal representation of the colour cues is developed on a self-organising map.

A reward is available for making a saccade to a particular colour and reward expectation is represented in a working memory with the same topology as the self-organising map. As the system learns the rewarding colour, an increasing fraction of the self-organising map represents features in this region, allowing fine distinctions to be made between rewarding and unrewarding shades of the target colour.

If the target colour changes then the plasticity of the map increases, allowing focus of representation to shift to another (the newly rewarding) region of the input space.

Top left: the entire visual world. The grey area is the region which is currently visible.
Bottom left: the working memory representation of rewarding features. This has the same topology as the self-organising map.
Bottom centre: the self-organising representation of the different colour cues.
Top right: the cumulative number of foveations to each colour.
Bottom right: the cumulative number of rewards received.

Relevant publication: Brohan, Gurney & Dudek, 2010