A gesture of hope


How gesture based technology is transforming the lives of his pupils with PMLD and SLD

I am the ICT Coordinator at a school for pupils with severe learning difficulties (SLD) and profound and multiple learning difficulties (PMLD). We have 126 pupils on the roll aged three to19. We have been focussing on increasing independent access to the curriculum for our most profoundly affected pupils and one important element of this work has been utilising gesture based technology.

What is gesture based technology?

Gesture based technology is increasingly becoming part of our everyday lives at home through the use of smart phones, tablets and gaming devices. It has also been around for a while in schools, mainly as touch screens and light beam technology. The basic principle is that it uses gesture as the main input method, rather than a mouse, switch or keyboard. This gesture can be touch (swiping, pinching or pressing a screen), physical gesture (moving hands, arms, fingers, the head or even the eyes) or speech and sound. The emphasis is on a direct interaction between the self and the device, with little or nothing in-between.

The attraction of this should be obvious to those working with pupils with SEN, as users just need to move, touch or make a sound to have a direct effect on their environment. Depending on the equipment used, this gestural input can be anything from the wiggle of a finger to a shout or a kick of the legs. We are well aware that all of the pupils we teach are different; they have their own individual physical capabilities, their own interests and their own cognitive abilities. Therefore, through using a wide variety of gesture based technologies, we can harness these capabilities to give each pupil what we all want them to have – independence and as much control over their environment and their lives as possible.

Our practice has involved a mix of utilising existing technologies that can be bought off the shelf and some creative ingenuity, all focused on the individual pupils and their needs. The school leads a gesture based technology Professional Learning Community (PLC) group. In Wales, PLCs are teacher-led groups that meet and research a particular focused topic. The PLC is a collaboration of 11 special schools throughout the UK and Ireland, independent SEN advisors and interactive designers. Our intention has been to make widespread use throughout the school of these emerging gestural technologies.


I have written on the advantages of tablets for a previous article in SEN Magazine (SEN62, January/February 2013). To summarise, the tablet is beneficial for a whole host of reasons:

  • they are portable and can be used in any position or place that the pupil needs them
  • they have touch-responsive screens
  • photo and video cameras are generally included as standard
  • they can be tilted to create effects
  • spoken word or sounds can be used for interaction
  • they can convert touch or sound input into a visual or audio response.

Tablet use is embedded throughout the school with each class and specialist having one. In 2011, we led another PLC with other special schools in South Wales which focused on using tablets with pupils with SLD and PMLD. Individual pupils were identified in the school who could benefit from their use and 14 case studies were compiled from teachers, the music specialist and the communication and sensory support teams. The case studies mainly involved hard-to-reach pupils who had behavioural or physical issues with engagement.

Tablets in action
J is a pupil with cerebral palsy who has limited movement of his elbows and head. He is a very accomplished switch user and uses switches to access music tracks, computer programs and sound clips. He loves high pitched sounds and harp noises so, for his free time in school, he accesses a tablet either by stroking it with his cheek to create the sounds he likes or by using apps that use the camera to detect his motion and convert that into sound. This has the advantage of enabling him to create sounds in a more fluid way than with a single switch press; he can either move quickly or slowly, with large or small movements, to create a variety of sounds for himself.

Eyegaze offers new opportunities for pupils with virtually no independent movement.Interactive floor projectors

This technology was introduced for two specific purposes. It allows pupils with very poor physical mobility to interact meaningfully and independently with their immediate environment and it also gives slower movers the motivation to be more active. The main users of this are those with severe physical disabilities such as cerebral palsy. The system uses infra-red motion cameras and a projector to create an interactive area on the floor.

Projectors in action
One of our most responsive floor projector users is a seven-year-old pupil with severe cerebral palsy working at a P3i level. He has limited movement of his arms and legs and cannot stand or sit unaided, but when out of his chair he can roll around on the floor. He absolutely loves the floor projector and it really motivates him to move his arms, legs and head as when he does, stars, lights and flowers emerge from every movement. In this way, he can also pop balloons, squish insects, splat ice creams and move graphics. When he is using the floor projector, his activity levels increase dramatically as he has a compelling reason to move. The most important thing is that he is doing it himself with minimal staff intervention or help.

Depth sensing cameras

Depth sensing cameras normally use both a normal camera and an infrared camera to track a large three dimensional area. They can recognise people and movement in this area and feed this information back into the computer. A typical set up uses a PC, a projector and a depth sensing camera to enable pupils to interact with programs by moving part or all of their bodies. As well as using programs created specifically for pupils with SEN, the school has been adapting programs designed originally for nightclubs and DJs or as installation art. Programmers of these types of equipment have, by and large, been very supportive and interested to see their creations being used with pupils with SEN.

The depth sensing camera converts gross motor movement into a visual or musical response and has been used mainly with our severely autistic pupils. These pupils are typically hard to engage, have multiple sensory-processing issues and poor awareness of self and how they can influence the outside world. Cause and effect is instantaneous with this technology, as any movement will cause a reaction on screen and these reactions then invite further exploration. All the programs we use are non-task directed and this is important as we are not teaching specific skills but encouraging play, creativity and interaction – deficits of which are at the heart of autism. They work well with all pupils who have movement and good vision.

Depth sensing cameras in action
As part of a post-graduate action research project, a 14-year-old pupil with severe autistic spectrum disorder working at around a P5 level was studied using this technology. He had very limited interests and his behaviour was deteriorating as he became more withdrawn and hard to engage. Through four weeks of sessions, the pupil made significant improvements in his levels of movement, interaction and engagement. Usually he was rooted to the spot or used short, specific movements, but after a few sessions he began to walk around, wave his arms and displayed much more fluidity of movement. The pupil, who has immediate and delayed echolalia, also independently named the colours on the projected screen twice as he changed them during the study. Such independent expressive language had not previously been observed in the pupil in the five years since he entered the school setting. Perhaps most importantly, he had found something that he really enjoyed doing which increased his wellbeing and involvement in learning.


This technology, which has been used traditionally for alternative and augmentative communication (AAC) devices, can also be used as an open ended engagement tool. The eyegaze device, which fits under the PC screen, tracks the eyes and enables the user to control the mouse cursor. By using it, pupils can create lovely sensory effects, or choose from options just by looking at a screen. Eyegaze technology is offering a window into the minds of a lot of our pupils and, as the only physical movement it requires is by the eyes, we are currently trialling it with a large cohort of our pupils with PMLD.

Eyegaze in action
One of our eyegaze users is a seven-year-old pupil working at a P2 level. He has lissencephaly, is tube fed and only has controlled movement of one thumb. When using eyegaze he totally fixates on the screen and really enjoys the sensory interaction programs that create sounds and graphics as he looks around the screen. He will track specific targets across the screen very well and he can also use the “dwell” function, which is where the user looks at a single place for a short amount of time to emulate a mouse click. He is accessing interactive content, such as switch based programs and animal scenes, and the next step with him is to introduce early choice making activities using icons. It is evidence enough that until we try technology such as this we do not always fully appreciate the cognitive abilities of our pupils with severe physical disabilities.

Our use of gesture based technology is constantly developing. The more we use it, the more we learn about how the pupils interact with it and how they can use it. In many ways, there is no real precedent to gesture based technology; there is no textbook for its use, so it is a case of giving it a go, seeing what happens and building on the positives. The common theme with all of this technology is that it enables some pupils to do things they never would have been able to do without it. The most crucial aspect of this is that they are doing these things themselves.

Further information

Anthony Rhys is the ICT Coordinator and on the senior leadership team at Trinity Fields School and Resource Centre, Wales:

To share its expertise, the Gesture Based Technology PLC runs three websites on using some of the technology described above with pupils with SLD and PMLD:

Anthony Rees
Author: Anthony Rees

assistive technology Trinity Fields School and Resource Centre, Wales

+ posts

assistive technology
Trinity Fields School and Resource Centre, Wales



Please enter your comment!
Please enter your name here