WO1986003863A1

WO1986003863A1 - Improvements relating to eye-gaze-direction controlled apparatus

Info

Publication number: WO1986003863A1
Application number: PCT/AU1985/000324
Authority: WO
Inventors: Andrew Richard Downing
Original assignee: The University Of Adelaide
Priority date: 1984-12-24
Filing date: 1985-12-24
Publication date: 1986-07-03
Also published as: JPS62501802A; AU5237786A; GB2179147A; GB8620269D0

Abstract

Eye-gaze-direction controlled apparatus including eye-gaze-direction determining means to detect a direction of gaze of any eye of a user, which means are also adapted to determine a relative position of the direction of eye gaze with respect to a target direction (8), said means further adapted to mesure and record a plurality of eye-gaze-directions (a-h) each measure being taken between sequentially selected intervals of time and to provide an ''activate'' or ''identify for subsequent action'' signal upon at least a preselected proportion of a preselected number of sequentially measured directions being detected which are within a preselected range of positions from the target direction bounded by boundaries (9, 10, 11, 12).

Description

IMPROVEMENTS RELATING TO EYE-GAZE-DIRECTION CONTROLLED APPARATUS

This invention relates to eye-gaze-direction controlled apparatus and to a method of controlling such apparatus.

5. It is known to have eye-gaze-direction determining means which include a substantially point reference source of light and detection means which are adapted to detect the reflections of the reference light from the retina and cornea or from other parts of the eye of 10. the user and then use this to measure and ascertain a relative viewing direction of the eye of the user.

A discussion of one such technique is given by Merchant et al in the IEEE transactions on Bϊomedical 15. Engineering volume BME-21, No. 4, July 1974.

A further description of such apparatus is given in Australian patent 544490 in the name of International Business Machines Corporation, and in United States 20. patent 4109145 in the name of Carl P. Graf.

The problem to which this invention is directed is in respect of machines of the type previously described.

25. A difficulty associated with such a technique, however, is to determine with reasonable certainty at what and when a user has aligned his or her sight on a target, or at least in the target direction, and ultimately intends that such alignment effect a result,

30. such as identifying the target itself with respect to which some action could occur or even having the target identified as such to the user as being coincident with his or her line of sight. It has been previously proposed that if the measured eye-gaze-direction remain accurately aligned with respect to a target for a pre-determined period of time, which time is sufficient to avoid accidental 5. activation, then this would provide a solution to this problem.

While in practice, this system provides a solution, the difficulty is to pack as large a number of possible 10. targets into a given screen area, so that there are as many options as possible available to the user. A further difficulty is that the time taken to effect an activation may be unnecessarily prolonged.

15. The aim is to therefore achieve as many options as is possible within a viewing area, while, at the same time providing a user with as rapid and consistent a response as is practicable, taking into account the need to minimise the chance of error through the

20. unintentional selection of another target.

The object of this invention therefore is to propose an arrangement and method which will assist in increasing accuracy and speed of selection.

25. The invention accordingly resides in an eye-gaze- direction controlled apparatus including eye-gaze- direction determining means to detect a direction of gaze of an eye of a user, which means are also adapted to determine a relative position of the direction

30. of eye gaze with respect to a target direction, said means further adapted to measure and record a plurality of eye gaze directions, each measure being taken between sequentially selected intervals of time and adapted to provide an "activate" or "identify" signal upon at

35. least a preselected proportion of a preselected number of sequentially detected directions being detected which are within a preselected range of positions from the target direction.

5. By using the technique of determining whether a selected proportion of measured readings are within a selected range, this has been able to provide for a quick and quite accurate initial presumption of a target location.

10. A person will not ordinarily be able to maintain a highly consistent eye-gaze-direction.

This problem arises both because of small involuntary eye movements that occur even when attempting 15. a steady fixation, and there is also an inherent noise level within any system that will make'Very narrow detection targets difficult to isolate.

In the normal continuous fixation technique, 20. any measured eye direction which extends out beyond a unique target range will cause the interpretation to recommence with subsequent delay. Signal averaging techniques such as low pass filtering in prior art applications are also known to cause difficulties in the 25. accurate and timely determination of a person's intended target.

<

It is very tiring to try to maintain such a steady gaze within a very small unique range of a target, and 30. hence, it is a significant advantage of this invention that such concentration which can achieve rapid and positive target fixing is not as necessary as systems hitherto proposed.

35 _m Such a feature is of high importance, considering the type of people that may wish to use this system, such as, the severely incapacitated or others who may be in a highly stressed condition so that there is advantage if the additional stress of extreme concentration can be avoided.

5.

In preference, the target and its range are displayed on a video display unit and the direction of eye gaze is determined with reference to the said display target on the video display unit. In a further 10. form the target forms to which this invention applies includes three dimensional targets of variable shape, and both moving and stationary targets.

In preference, at least one of the target areas 15. includes a control activation which will indeed change the menu or the results of activation of any of these so that the user not only can activate from the first visible screen, but can in fact change the visible screen from time to time so that a vast range of possibilities 20. becomes then open for controlling outside occurrences using nothing but an intended persistence of eye-gaze- direction with respect to a target area.

In preference, and where the user is capable of 25. performing this task, supplementary means of initiating an action, through the use of switchs, a keyboard or other transducers, and with respect to a target identifying eye, can be provided, as this can futher increase the number of options available, and the 30. speed and accurancy of reflection.

While the invention in its broadest sense is not limited to such an arrangement, it is most convenient in the eye-gaze-direction determining means to include a 35. substantially point reference source of light, and detection means, which comprise a vidicon television camera, which is adapted to detect the reflections of the reference light from the retina and cornea of the eye of the user. In a conventional television raster, the scan rate is such that a reading of eye direction can be achieved from each frame of picture information so that either 50 such readings can be made per second in one system or 60 5. such readings can be made per second in another.

The invention in a further form can be said to reside in a method of determining a required target by determination of an eye-gaze-direction which includes,

10. the steps of measuring and recording a plurality of eye-gaze-directions relative to a target direction, then effecting an "activate" signal only when at least a selected proportion of the immediate sequentially preceding preselected number of measurements are within a

15. selected range of the target.

In preference, the number of measured and recorded eye-gaze-directions used to determine a proportion to effect an "activate" signal is within the range of 3 to 20. 200, depending on the nature of the action which is to follow.

In preference, the number of measured and recorded eye-gaze-directions used to determine a proportion 25. to effect an "activate" signal is such that they are taken within the time range 3/50ths to 200/50ths of a second.

In preference the proportion of the number of 30. measured and recorded eye-gaze-directions which are within a selected range of the position of the target is within the range of 0.5 to 0.9, depending on the nature of the action which is to follow.

35_# In preference, the selected range of the position of the target should be equivalent to angles subtended to the user's eye of at least 1 degree in each direction. The number of measured and recorded eye-gaze- directions and the required proportion of that number, and the range of position of the target may be varied dynamically to suit the individual user, or the operating conditions of the eye gaze monitor to further optimise the speed and accuracy of selections. In a further form the user can modify one or more of these parameters, by separate control functions or by eye gaze selection.

10 The invention will be better understood when referred to a preferred embodiment which shall now be described with the assistance of drawings in which FIG. 1 shows in schematic manner the arrangement of the eye of a user and a target and means forming 15. part of the arrangement for determining the eye-gaze- direction.

FIG. 2 illustrates a target, a unique range boundary of such a target and expected typical detected 20. eye-gaze-direction locations relative to the target in the range and FIG, 3 is a flow chart for determining the aiming point of the eye-gaze-direction.

Referring to the drawings and especially to

25 FIG. 1 there is seen a representation of an eye of a user at 1, a substantially point source of light 2 and a vidicon television camera 3 detecting the relative position of the pupil 4 by virtue of light reflected by the retina 4a with respect to the reflection

30 of the source of the light 2 by the cornea 5 of the user 5. The substantially point source of light is provided by a filament 2 in front of a reflector 2a, the light from the filament 2 being directed through a filter 2b onto a mirror 2c and then onto a beam splitter 2d so as 5. to be directed into the direction of the eye 1.

The reflection is directed back to the beam splitter 2d through filter 2e and into the vidicon television camera 3.

10.

The filters 2b and 2e are each of an identical type and are selected to admit only a narrow band width of light in the near infra red region. The effect of this of course, is to reduce the possiblity of external light

15 interfering with the operation of the eyegaze-direction measurement.

The camera 3 in this embodiment is a monochrome television camera 625 line 50 cycles per second,

20, 2:1 interlace raster scan system, synchronized to signals derived from crystal oscillator locked column (pixel) and row counters used to detect the positions of the boundaries of the features of the eye seen by the camera.

25.

Such an arrangement provides that a reading and recording of an eye-gaze-direction can be made 50 times every second.

_3Q In the drawing, the target 7 is part of the screen of a video display unit which is divided into a number of separate and uniquely identified ranges each of which includes an appropriate target therewithin. By way of illustration the screen is divided into 49 different areas and in this embodiment, the unique range for each target, is the range shown by the dividing lines being therefore approximately l/49th of the -5. total area bounded by the rectangular boundaries shown.

In FIG. 2 there is shown the target 8 which is bounded by the shown rectangular lines 9, 10, 11 and 12.

10. There is firstly required an assessment of whether the line of sight of the eye 1 is directed at the target 8.

Shown in the drawing are eight sequential typical 15. measurements of eye-gaze-direction as they might be expected to intersect the target screen.

_.*-

Accordingly, (a) is within the unique target area range, (b) is just outside, (c) is within, (d) is within, 20. (e) is just outside, (f) is in and (g) and (h) are both in.

Of the 8 sequentially taken readings each of which is recorded, it will be seen that there is a proportion 25. of 0.75 that is 6 out of the 8 readings are within the ^* specified range so that within 8/50ths of a second, a determination can be made that that target is that being chosen.

30. In the embodiment, upon such a preliminary determination being made, there are means effective to then provide that a cursor is located beneath the target 8 as visual feedback to the user.

35_^ Thereafter, further measuring and counting continues using a significantly greater number of sequential measurements and recordings of eye-gaze-direction and in preference if very rapid determination is required, the selected number of sequential readings can be 20 with a proportion of 0.8 whereby to indicate after the cursor has indicated that the said target 8 has been that chosen, so that a further action can then be initiated.

5.

20 readings with a proportion of 0.8 is appropriate for an experienced user but as an illustration, for a less experienced user a larger number is more appropriate and a number as high as 70 can be used 10. as the sequential measurement with a proportion of 0.83.

A factor in the proportion is the degree of accuracy required, and the distance of the screen 15. and the size of the screen should be such that no greater accuracy is required than 1 degree in the vertical direction and 1 degree in the horizontal direction.

20. FIG. 3 is a flow chart for determining the aiming point of the eye-gaze-direction.

Step 13 after the computer has been appropriately set up includes the testing of information from the 25. eye-gaze-direction determining apparatus such information being head position, loss of eye location, blinks, or saccades, in addition to the eye gaze direction itself.

The next test at 14 is a check of head position 30. and supplying a warning if realignment is desirable.

15 then is a digital filtering of line of sight information using a last selected number of s mples.

35. In a further functional step at 16, then a calculation of the current target being looked at is made and if this is not a legitimate target a minus 1 level is set as the target position which is interpretahle as no position being determined. From 17 to 26, there exists an area for determining whether an action from within a highest priority action class "A" should be effected.

-. 17 is the test for continuous action or single event and if continuous action the flow chart continues to the left and if a single event, to the right.

The test at 18 is whether for any target, the

10, proporation of the immediately. preceding selected number of measured positions within the range uniquely ^• identified for that target is greater than the selected proportion of all readings wherever they have been located at.

If no for any target which a continuous action had

15, previously been initiated, the flow chart path continues to the left through a discontinue action sequence at 19.

--*^*

If yes, the flow chart path continues to the right -_jn through a test at 20 to ascertain whether the action associated with that target is enabled, an execute or continue action at 21 and then through an enable or disable other actions if appropriate step through 22.

- 23 provides the same proportion test as 18 for the current target and if the proportion is greater than that selected the path proceeds to 24 which tests if an action is enabled and if effected proceeds through 25 which is an execute action and set target for all previous measurements to minus 1.

30

If no, on any of the previous several instances, the path proceeds to further tests for other classes of lower priority actions (b) , (c) etc. in the manner decided above in slips 17 to 26 and sequentially

35, reverting through 27 to a re-commencement of the flow path. Once the technique described herein has enabled a person to identify or select a target, confirmation of the action intended for that target can occur immediately, or after a further time period using 5. similar criteria for the proportion of time for which fixation is to continue.

Alternatively, the person can use eye gaze to identify a target, and then use a supplementary means 10. such as a switch or key pad or any other distinctly separate device to cause some action to occur relative to that target.

In either case, but especially for eye gaze 15. control, it is useful to provide a second stronger, form of visual feed-back (such as the whole target area being highlighted rather than a small central portion while identificat-ion is in progress) to confirm that the action took place. Other forms of feedback, expecially auditory 20. feedback, may be used as well as or in place of the visual feedback.

The advantages of the invention thus now described are that a quicker fixation of a target can be determined

25. than has hitherto been the case and this can lead to more rapid effecting of an action using eye gaze control and this can be a result of being able to put many more possible targets on a given screen area for use by the user. Such closer packing of

30. targets than has hitherto been possible provides thereby a much greater range of options, the direct result of the advantages of this invention leading to both faster control and indeed, more reliable control.

Claims

THE CLAIMS DEFINING THIS INVENTION ARE AS FOLLOWS:

1. Eye-gaze-direction controlled apparatus including eye-gaze-direction determining means to detect a direction of gaze of an eye of a user, which means are also adapted to determine a relative position

5. of the direction of eye gaze with respect to a target direction, said means further adapted to measure and record a plurality of eye-gaze-directions each measure being taken between sequentially selected intervals of

10. time and to provide an "activate" or "identify for subsequent action" signal upon at least a preselected proportion of a preselected number of sequentially measured directions being detected which are within a preselected range of positions from the target

15. directions.

2. Eye-gaze-direction controlled apparatus as in claim 1 further characterised in that the eye-gaze- direction determining means includes a substantially point reference source of light, and detection means 5. adapted to detect the reflections of the reference light from one or more of the retina, cornea, sclara, iris or lens of the eye of the user.

3. Eye-gaze-direction controlled apparatus as in either of claims 1 or 2 further characterised in that there are one or more targets displayed on a video display unit and the direction of the eye gaze is 5. determined with reference to the said displayed targets

4. Eye-gaze-direction contolled apparatus as in any one of the preceding claims further characterised in that there are means adapted to effect an action subsequent to an "activate" signal being raised. 5. Eye-gaze-direction controlled apparatus as in any one of the preceding claims further characterised in that the "activate" signal is adapted to effect, on a video display unit, a "target located" identifying 5. signal.

6. Eye-gaze-direction controlled apparatus as in the immediately preceding claim further characterised in that the target located identifying signal is a cursor either superimposed over or located adjacent to 5. the target location.

7. Eye-gaze-direction controlled apparatus as in any one of the preceding claims further characterised in that upon an "activate" signal being raised, said means are adapted to further measure and record 5. the eye gaze direction for a further plurality of eye gaze directions, and to effect an "action" signal upon at least a preselected proportion of a selected number of sequentially detected directions being detected which are within a preselected range of 10. positions from the target direction.

8. Eye-gaze-direction controlled apparatus as in any one of the preceding claims further characterised in that the number of measured and recorded eye gaze directions used to determine a

5. proportion to effect an "activate" signal is within the range of 3 to 200.

9. Eye-gaze-direction controlled apparatus as in any one of the preceding claims further characterised in that the number of measured and recorded eye gaze directions used to determine a proportion to 5. effect an "activate" signal is such that they are taken within the range of time period 3/50 to 200/50 of a second. 10. Eye-gaze-direction controlled apparatus as in any one of the preceding claims further characterised in that the selected proportion of the selected number of measured and recorded eye gaze directions which are 5. within a selected range of the position of the target is within the range of 0.6 to 0.9.

11. Eye-gaze-direction controlled apparatus substantially as described in the specification with reference to and as illustrated by the accompanying drawings.

12. A method of determining a required target by determination of an eye gaze direction which includes the steps of measuring and recording a plurality of eye-gaze-directions relative to a target direction, 5. then effecting an "activate" signal only when at least a selected proportion of the immediatery sequentially preceding preselected number of measurements are within a selected range of the target.

13. A method of determining a required target by determination of an eye-gaze-direction substantially as described in the specification.