US5999895A - Sound operated menu method and apparatus - Google Patents

Sound operated menu method and apparatus Download PDF

Info

Publication number
US5999895A
US5999895A US08/506,152 US50615295A US5999895A US 5999895 A US5999895 A US 5999895A US 50615295 A US50615295 A US 50615295A US 5999895 A US5999895 A US 5999895A
Authority
US
United States
Prior art keywords
sound
sequences
display
operator
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US08/506,152
Inventor
Donald K. Forest
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to US08/506,152 priority Critical patent/US5999895A/en
Application granted granted Critical
Publication of US5999895A publication Critical patent/US5999895A/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/023Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
    • G06F3/0233Character input methods
    • G06F3/0237Character input methods using prediction or retrieval techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/023Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
    • G06F3/0233Character input methods
    • G06F3/0236Character input methods using selection techniques to select from displayed items
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482Interaction with lists of selectable items, e.g. menus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842Selection of displayed objects or displayed text elements
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B21/00Teaching, or communicating with, the blind, deaf or mute
    • G09B21/009Teaching or communicating with deaf persons

Definitions

  • the present invention relates generally to interactive display terminals and interactive display methods, and more particularly, to interactive display terminals and interactive display methods for use by persons temporarily or permanently lacking normal motor capabilities. It also relates to systems and methods for the assessment of the motor capabilities of persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for use in speech synthesis for persons having impaired speech. It also relates to systems and methods for the control of devices, including appliances, by persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for selecting one menu option from a menu. It further relates to systems and methods utilizing sound recognition for selecting a menu option from a menu. It further relates to data and order entry systems including, and data and order entry methods utilizing, an interactive display terminal.
  • CP Cerebral Palsy
  • Traumatic Brain Injury, Spinal Cord Injury, Muscular Dystrophy, Amyotrophic Lateral Sclerosis and Multiple Sclerosis can result in a reduced ability to voluntarily control or prevent the movement of parts of the body, including the head, limbs and digits, muscle stiffness, weakness, limited range of motion, abnormal posture, involuntary muscle tremors, involuntary muscle activity causing involuntary motion, impaired ability to voluntarily stop motion, impaired ability to coordinate muscle activity, and/or impaired ability to sense the position of a part of the body. Any one of these symptoms may impair an affected individual's fine motor control.
  • some individuals affected by a neuromuscular disorder may be able to exercise fine motor control with enormous effort, the struggle to do so often fatigues the individual, limiting the period of time the individual is capable or comfortable performing the fine motor control task.
  • Neuromuscular disorders are often systemic in effect, impairing an individual's ability to operate prosthetic devices, such as a wheelchair, and to perform the activities of daily life, such as speaking, walking and operating household appliances.
  • Speech is frequently affected since the mechanics of producing speech require coordination of many muscle groups--the muscles of the diaphragm which push air over the vocal cords, the muscles of the larynx, jaws, tongue and lips.
  • the inability to use or coordinate these muscle groups may result in impaired speech.
  • speech may be totally absent, present but impaired to the point of unintelligibility, or intelligible on the whole but with occasional unintelligible words.
  • the ability to walk is often affected since walking requires coordination and voluntary control of many muscle groups.
  • impaired fine motor control may prevent or impede an individuals from effectively operating household appliances or computer input devices.
  • NMD operators Devices which produce speech, control appliances and facilitate computer access for some persons having neuromuscular disorders
  • AAC Augmentative and Alternative Communication
  • NMD operators Devices which produce speech for individuals whose own speech is impaired, called Augmentative and Alternative Communication (“AAC") devices, allow the operator to select words or phrases by spelling the words, by specifying an abbreviation for the phrase or by selecting a sequence of symbols, and then speak the selected words or phrases using an electronic speech synthesizer.
  • AAC Augmentative and Alternative Communication
  • NMD operators are often unable to efficiently use a standard keyboard and mouse.
  • an NMD operator who is unable to stop the movement of a limb with precision, when attempting to use a keyboard or mouse, may move his arm toward the target key or move the cursor toward the target object on the display but overshoot the target. If he has involuntary tremors and cannot hold a limb still, then, when attempting to use a keyboard, he may hit keys adjacent to his target key. If he has involuntary motion moving left to right, then, when attempting to use a keyboard, he may have difficulty accessing an intended key on the right side of the keyboard.
  • a computer connected to a speech synthesizer enables an NMD operator with impaired speech to direct the computer to speak for him.
  • a user who is physically unable to operate a household appliance for example, a television, video cassette recorder, compact disc player, radio, alarm clock, telephone, light, thermostat, dimmer or power switch, may be able to control the appliance via a computer equipped with an interface he can control.
  • NMD operators may make use of the same general purpose computer application programs ("applications") as able-bodied users, including applications for word processing, database, computer-aided instruction, access to literature accessible via computer, spreadsheet, time management and computer utilities.
  • applications general purpose computer application programs
  • Adolescents with CP are obviously different from their peers. They are often surrounded by non-normative assistive technology, e.g. wheelchair or walker, special school bus equipped with a chair lift, stair lift, standing aid, AAC device, feeding apparatus, bath seat, toiletting apparatus, etc. In addition, they may drool, lacking the ability to coordinate lip closure with swallowing. Nonetheless they are adolescents and need peer approval to support them in their maturation from dependent children to independent adults.
  • Today, demonstrated facility with a computer is an emblem of intelligence among adolescents, so computer use provides adolescents the opportunity to prove their intelligence and thus potentially rewards NMD operators with both self-esteem and peer approval.
  • NMD operators vary widely in their motor capabilities. Even individuals having the same medical diagnosis may require completely different technologies for computer access. Many NMD operators are able to use an oversize keyboard, a device having a pressure-sensitive surface divided into squares, each square associated with a letter of the alphabet. The squares may be sized to match the operator's abilities, but typically each square is two inches on either side. NMD operators who are unable to efficiently use an oversize keyboard may use another conventional computer access solution, called an "on-screen keyboard", which, as illustrated in FIG. 1, is a picture of keyboard drawn on a computer display (1101).
  • an "on-screen keyboard" is a picture of keyboard drawn on a computer display (1101).
  • Switch operation includes, but is not limited to, each of the following: opening the switch, closing the switch, opening the switch multiple times within a predetermined period, and closing the switch multiple times within a predetermined period.
  • a program executing on the computer determines which letter the operator has selected and processes the letter or passes it to some other application program which processes the letter as if it came from a true keyboard.
  • Conventional pointing devices include a mouse, trackball, joystick (which may be integrated into a keyboard, e.g. TrackPoint II®), stylus and graphics tablet, lightpen, thumb wheel, touch screen, touch panel, head pointer, occulometer, intraoral pointer and eye tracker. They may be active, e.g. a lightpen that emits an infrared beam, or passive, e.g. an eye tracker that uses images of an individual's eyes to determine where his eyes are focusing.
  • Conventional switches include a button on the mouse, a switch in the tip of the stylus actuated by pressure or the release of pressure, a switch mounted on the user's wheelchair operated by a turn of the head to or the switch below a keyboard key.
  • Dwell time may be continuous or discontinuous depending upon the operator's motor capabilities. In continuous dwelling, if the operator moves the cursor from one key image to another region of the display, the time accumulated on the key image is discarded so that if the operator returns to that key image he must dwell on it for the full selection threshold to select it. Discontinuous dwelling, by contrast, compensates for involuntary tremors which pull the operator off the desired key image. Accumulated dwell time on a key image is remembered, so that on return to a key image, the operator need only dwell for a period equal to the difference between the selection threshold and the previously accumulated dwell time for that key image. Accumulated dwell time is reset to zero for all key images following the selection of any one key image. Conventional on-screen keyboards do not indicate to the operator the dwell time associated with any key image.
  • each key image may be associated with its own selection threshold period. In the latter case, keys associated with shorter selection threshold periods are easier to select than keys associated with longer selection threshold periods.
  • FIG. 2 illustrates an example of a combined display of an on-screen keyboard and a word processing application program.
  • the on-screen keyboard (0201) is shown on the lower portion of a display connected to a computer system (not shown) which also executes the word processing application program whose output (0203) appears on the upper portion of the display. Letters selected by the operator are input to the word processing application program.
  • FIG. 1 shows an on-screen keyboard containing 81 total keys including 26 alphabetic keys, 10 numeric keys, 12 function keys, 4 arrow keys and 29 special purpose keys. Drawing this many key images on a display restricts the size of each key image making each very difficult for many NMD operators to select.
  • quaternary keyboard provides for larger key images.
  • the alphabet is divided into four groups of letters, each displayed in one of the four quadrants (1302), (1304), (1306) and (1308) of the display, as shown in FIG. 3.
  • the operator selects one of the four groups by, for example, pointing to and dwelling on one quadrant of the display.
  • the selected group is then exploded into four subgroups, each displayed in one quadrant of the display, as shown in FIG. 4. Once more the operator selects one of the four.
  • the selected group is exploded into four letters and each letter displayed in one quadrant of the display, as shown in FIG. 5. The operator again selects one of the four. This letter is then input to an application program (not shown).
  • the quaternary keyboard illustrates the use of a menu hierarchy in computer access.
  • Each of the four groups of letters (1302), (1304), (1306) and (1308) is a menu option.
  • Each of these menu options is itself a menu which includes other menu options.
  • a menu hierarchy exists if at least one of a menu's menu options is itself a menu.
  • a menu accessed from another menu may be called a submenu, and the options of the submenu may be called submenu options.
  • a menu hierarchy is narrow and deep, many selections are required to make the desired choice.
  • a menu hierarchy is broad and shallow, each layer is composed of many menu options.
  • the quaternary keyboard greatly expands the size of a single key image and thus accommodates certain NMD operators with drift or involuntary tremors.
  • the cost of this adjustment is high Instead of selecting a letter with one pointing motion and dwelling for one selection threshold, the quaternary keyboard requires three pointing motions and dwelling for three selection thresholds.
  • the operator's productivity is dramatically reduced from the standard on-screen keyboard depicted in FIG. 1.
  • FIG. 6 illustrates a display combining a quaternary keyboard and output from two application programs.
  • pie menu is an opaque region on a display divided into selectable slices, each slice associated with a menu option.
  • the pie menu suffers some of the drawbacks discussed above, particularly that, while displayed the pie menu occupies more space than a linear menu and obscures much of the output of the operator's application program.
  • pie menus see Callahan, Jack et. al., "An Empirical Comparison of Pie vs. Linear Menus", Computer Science Technical Report Series, CS-TR-1919, University of Maryland, College Park, Md., September 1987.
  • FIG. 7 depicts a conventional joystick pattern device.
  • the device (1602) is connected to a joystick and to a computer.
  • the operator pushes the joystick to the top, bottom, left, right, top left corner, top right corner, lower left corner or lower right corner, closing one of eight switch contacts within the joystick housing. That switch position is then indicated on the display (1604).
  • a sequence of consecutive of switch closures encodes a letter or other programmed output that the device (1602) displays on an LCD display (1606) and sends to the connected computer, simulating keyboard input.
  • the conventional joystick pattern device is ill-suited for many NMD operators.
  • the involuntary tremors common some neuromuscular disorders may result in unintended switch closures.
  • the device does not provide an indication that the operator is moving a body member in an unintended direction until switch closure occurs. For example, an operator with CP who intends to move the joystick the right but actually moves it to the upper right receives no indication from the device, prior to switch closure, that he's not on target.
  • the device requires that the operator memorize the encoding of each letter or other output since there's no indication on the display (1606) which sequence encodes which letter.
  • the device provides no support for head pointing, although the head is often the best controlled part of an NMD operator's body.
  • NMD operators who cannot effectively use either a conventional keyboard or a pointing device but can reliably actuate a switch may use a computer access method called "scanning", which is subdivided by cursor control technique into three types of scanning: automatic, directed and step.
  • automatic scanning all the operators' options, for example, the letters of the alphabet, appear on either a static or dynamic display (depending upon the implementation), organized in rows and columns.
  • a cursor moves from one row to the next.
  • the operator wants he closes a switch.
  • the machine now moves the cursor from one letter to the next within the selected row until the operator closes the switch again.
  • the operator has now selected one letter.
  • the cursor moves at the frequency determined by the scanning interval, however, it moves only when the switch is closed.
  • the operator opens the switch while the cursor indicates the desired option.
  • step scanning the cursor moves with each switch activation.
  • AAC operators AAC device operators
  • operators who speak with AAC devices, particularly NMD operators whose motor deficits impair their ability to use a keyboard lag substantially in their conversations.
  • the slow pace of an AAC operator's word production disrupts normal verbal interaction.
  • Speaking persons, accustomed or not to the AAC operator's slow rate often lose patience in conversations with AAC operators. They may prematurely terminate the conversation, read the AAC device display in an attempt to guess at the AAC operator's intended utterance and so accelerate the interaction, lead the AAC operator, ask predominantly yes/no questions, change the topic of conversation with little input from the AAC operator and otherwise dominate the interaction.
  • the AAC operator often has difficulty participating as an equal partner in the conversation. He may be unable to change the topic, interject a humorous comment in a timely fashion or respond to a question before the speaking person changes the topic. Slow AAC operators may be perceived as mentally slow. Thus the quality of verbal interactions where one party uses an AAC device to speak depends significantly upon the AAC operator's rate of word production.
  • Letter or menu option selection time includes the time the operator requires (a) to comprehend the menu options displayed, (b) to move the pointer to the desired menu option on the display, and, in selection by dwell, (c) the selection threshold period, or, in selection by click, (c) the time required to operate the switch. Decreasing any one of these increases the operator's productivity, assuming all other steps in the selection process are unaffected.
  • Another consequence of personal interaction during conversation for an AAC device operator is that the operator needs a way to easily enable and disable the AAC device operator interface so that movement the operator makes during personal interaction, for example, nodding his head, is not interpreted by the AAC device.
  • neurogenic muscular disorders may impair the ability of an individual to sense the position of a body member.
  • An NMD operator thus relies more than his able-bodied peer on the location of a cursor or similar automated indication of body member position.
  • Conventional access methods which use a pointer do not provide additional feedback to the operator of the position of a body member.
  • Access methods which require the NMD operator to make the same movement for most selections such as single switch access, mouth sticking (the use of a small rod held in the mouth and used to depress keys on a keyboard) and head sticking (the use of a rod mounted on the head and used to depress keys on a keyboard), may result in repetitive motion injury, particularly after years of use.
  • OCR optical character recognition
  • OCR systems which attempt to recognize graphic symbols and words based on attributes for optical recognition purposes, for example, the appearance of graphic symbols, the ratio of dark space to light space within part or all of a graphic symbol, the ratio of dark space in one part of a graphic symbol to the dark space in another part of the graphic symbol, and the derivative of darkness over the scan of the graphic symbol.
  • OCR systems convert the contents of a typewritten document into a computer encoding of the same. OCR systems at times are unable to recognize a graphic symbol or word, or may err in selecting a graphic symbol or word from a plurality of candidates. Therefore, following optical character recognition, a human may proofread and correct the computer encoded document. The proofreader may indicate where an error or omission in the computer encoded document occurred and may select from a plurality of menu options, each representing a candidate graphic symbol or word.
  • One object of the invention is to facilitate computer access by a disabled operator.
  • a further object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to maintain a body member in a steady position.
  • Another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to stop motion.
  • Yet another object of the Perimeter Menu and Confinement aspects of the invention is to simultaneously display an application program window and a computer access menu which does not obstruct the application program window.
  • Another object of the Perimeter Menu and Confinement aspects of the invention is to allow an operator to enable and disable a menu.
  • a still further object of the Perimeter Menu and Confinement aspects of the invention is to synthesize speech for an operator having impaired speech and impaired fine motor control.
  • Still another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate device control for a disabled operator.
  • Another object of the Perimeter Menu and Confinement aspects of the invention is to reduce the cognitive demand of speech synthesis for the disabled.
  • a still further object of the Perimeter Menu and Confinement aspects of the invention is to enlarge the effective area of a selectable region without concomitantly reducing the area available for information display.
  • Another object of the Perimeter Menu and Confinement aspects of the invention is to speed data entry.
  • Yet another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate computer access for an operator having impaired ability to sense the position of a body member used for computer access.
  • Another object of the invention is to facilitate the correction of errors or completion of omissions in a computer encoded document produced, at least in part, by optical character recognition.
  • Conventional menu-driven data entry and order entry systems incorporating pointing at intended selections employ a two step selection procedure.
  • the operator indicates, with a pointer, his intended selection.
  • the system then provides feedback, for example, by highlighting the indicated selection, showing which selection the operator has indicated.
  • the operator selects the indicated selection, for example, by operating a switch.
  • One object of the invention is to facilitate the use of systems allowing selection by dwell.
  • Still another object of the invention is to facilitate device control by the disabled.
  • a further object of the invention is to increase the independence of the disabled.
  • Yet another object of the invention is to facilitate the use of a data entry or order entry system by an intermittent operator.
  • Another object of the invention is to facilitate ordering by someone seated in a vehicle.
  • the on-screen keyboard with dwell selectable key images is ill-suited for use by many NMD operators. Selection by dwell may fatigue NMD operators or may require greater fine motor control than they bring to this task. Operators with impaired ability to stop motion and those having involuntary tremors have difficulty maintaining the location indicated by a pointer on a key image for a period sufficient to distinguish intentional dwelling from unintentional dwelling. Consequently, some NMD operators who try to use on-screen keyboards often miss their target key images and/or accidentally select unintended key images. Following such an error, the operator must erase his erroneous selection by selecting the backspace or undo key. As the number of erroneous selections increases, the operator's productivity decreases markedly, since each error requires a correction in which there might be another error.
  • Conventional on-screen keyboards require the ability to select by dwell or by click and thus are limited to operators with these capabilities.
  • Conventional on-screen keyboards do not utilize the relatively intact motor capabilities of some NMD operators to compensate for impaired ability to select by dwell or by click or to speed up the slow process of selecting by dwell. For example, while an NMD operator may overshoot a key image, his directional control may be relatively intact.
  • Conventional on-screen keyboards do not exploit this capability.
  • Windows® Operating System assigns meaning to the cursor location.
  • Windows® interprets the action as manifesting an intent to choose that menu item.
  • the operator's path to that menu item, whether direct or circuitous, is irrelevant. Operators who can move toward a target accurately but cannot maintain the location indicated by a pointer on the target cannot effectively use standard Windows® applications through the conventional interface to these applications.
  • NMD operators cannot steady a pointer while operating a switch; the act of operating the switch triggers involuntary muscle activity pulling the cursor off target.
  • conventional selection by click is not practicable.
  • Conventional selection by dwell also requires greater fine motor control than many NMD operators bring to this task.
  • Operators with impaired ability to stop motion may overshoot their intended target.
  • Operators whose voluntary muscle activity is accompanied by some involuntary muscle activity affecting their directional control often cannot point accurately.
  • Operators with involuntary tremors often cannot maintain the location indicated by a pointer on a key image. Consequently, NMD operators who try to use on-screen keyboards often miss their target key images and accidentally select unintended key images.
  • Measures of an individual's fine motor control assist a physician or therapist in evaluating the effectiveness of a treatment program, including assistive technology, and in gauging the severity of a disability. Such measures help the physician or therapist in determining what treatment course to pursue and whether the severity of a certain disability justifies the risk of a particular treatment option, such as neurosurgery.
  • One object of the invention is to facilitate selection of an option from a menu.
  • Another object of the invention is to indicate to an operator moving a cursor toward an option in a menu displayed by a computer system, which option the computer system believes the operator is moving toward.
  • Another object of the invention is to display a menu on a display so that a large contiguous area on the display is not obstructed by the menu.
  • Still another object of the invention is to make use, in computer access, of relatively unimpaired directional control in persons having impaired fine motor control.
  • a further object of the invention is to indicate to an operator moving a cursor toward a dwell-selectable option in a menu, a changed selection threshold of the dwell-selectable option.
  • Yet another object of the invention is to more efficiently select an option from a menu on a display.
  • Another object of the invention is to speed up selection of an option from a menu by an operator having impaired ability to operate a switch while the operator simultaneously keeps a cursor location within a region on a display.
  • Another object of the invention is to help an operator with a disability control a pointer.
  • Yet another object of the invention is to speed data entry by an individual with a disability.
  • a still further object of the invention is to facilitate artificial speech generation by a person having impaired speech due to a neurogenic muscular disorder.
  • Yet another object of the invention is to facilitate device control by a person having a neurogenic muscular disorder.
  • Another object of the invention is to measure an individual's ability to move one of the individual's body members in a direct path from a starting position to an ending position.
  • Yet another object of the invention is to facilitate computer access for an individual who cannot stop movement cleanly.
  • One object of the invention is to facilitate computer access by an operator having impaired ability to maintain a body member in a steady position.
  • Another object of the invention is to facilitate the selection of a desired menu option by an operator having impaired fine motor control.
  • a further object of the invention is to synthesize speech for an operator having impaired speech and impaired motor control.
  • Yet another object of the invention is to use an operator's directional control in computer access.
  • a still further object of the invention is, in selecting a menu option from a menu of dwell-selectable menu options, to compensate for an operator's impaired ability to maintain a body member in a steady position by using the operator's relatively intact motor capability .
  • Alignment is also a problematic for NMD operators who use a pointer, such as a mouse, with which the operator indicates by a location on a surface, e.g. a desk top, which corresponds to a desired location on the display, and achieve alignment by removing the pointer from the surface, e.g. lifting the mouse, moving the mouse, then replacing it on the surface. Due to impaired fine motor control, many NMD operators cannot remove a pointer from the surface and replace it on the surface at a desired location without unintentional movement or extraordinary effort. For these operators, alignment cannot be effectively achieved through conventional means.
  • a pointer such as a mouse
  • misalignment interferes with accurate pointing and the process of correcting for misalignment may result in the selection of unintended key images.
  • One object of the invention is to allow an operator to align a pointer with a location on a surface.
  • Another object of the invention is to indicate to an operator a location on a surface with which he may align a pointer.
  • Still another object of the invention is to indicate to an operator when he may align a pointer with a location on a surface.
  • a further object of the invention is to allow an operator having impaired motor control to align a pointer with a cursor.
  • one of the elements determining the menu option selection time is the time the operator requires to comprehend the menu options displayed. This time may be reduced if the operator can limit the number of menu options he searches in looking for his desired menu option.
  • Conventional word prediction systems attempt to reduce this operator search time.
  • the operator of a conventional word prediction system may, for example, select the letter "p".
  • the system displays some number, say six, of the most frequently used words beginning with the letter "p".
  • these six words are displayed either in alphabetic order or in order of frequency of use. Assuming the operator does not see his desired word on the display, he selects another letter, say "r”. The system then displays the six most frequently used words beginning with the letters "pr".
  • Searching a displayed list of words in alphabetic order requires that the operator focus his attention on the selection task, as opposed to the information content of the conversation or other task the operator is engaged in. Further, determining whether a given word is alphabetically greater or lesser than a desired word takes substantial time, slowing the selection process.
  • An alphabetically ordered list is of limited use to an individual who has below normal spelling ability, a frequent problem among individuals with impaired speech. Ordering words by frequency of use often does not limit the number of words the operator must search. The word at the bottom of the displayed list, for example, the sixth most frequently used word beginning with the letters "pr" may be a very common word, even though it is less frequently used than the other five displayed words.
  • One object of the invention is to reduce the time an operator requires to comprehend displayed menu options.
  • Another object of the invention is to reduce operator search time.
  • Still another object of the invention is to limit the number of menu options an operator searches for in looking for his desired menu option.
  • a further object of the invention is to speed data entry.
  • Yet another object of the invention is to increase productivity in speech synthesis for an operator having impaired speech.
  • FIG. 8 Another approach is the conventional eye gaze system for a speech impaired individual, depicted in FIG. 8.
  • the system consists of a plexiglass frame (6352) having a centrally located aperture (6354).
  • the eye gaze system is positioned between the speech impaired individual and person with whom the speech impaired individual is communicating.
  • the clear square matches the aperture (6354). All squares are labeled with symbols representing items to be communicated. These labels are not shown in FIG. 8.
  • the person with whom the speech impaired individual is communicating observes the eyes of the speech impaired individual to determine the target of the speech impaired individual's eye gaze. To communicate an item, the speech impaired individual gazes first toward the one of the eight groups of five squares, indicating that he wants to communicate one of the symbols in that group, and gazes second toward one of the four corners and aperture (6354) matching the color of the square labeled with the item to be communicated in the previously indicated group.
  • FIG. 9 depicts a menu having three menu options, labeled "High”, “Medium” and “Low”, each displayed on a display (4807), each associated respectively with selectable regions (4801), (4803) and (4805), and each located adjacent the associated selectable region.
  • FIG. 10 depicts a menu having the same three menu options, each displayed on a display (4807), each associated respectively with selectable regions (4901), (4903) and (4905), and each intersecting the associated selectable region.
  • a menu option is selected by pointing to and clicking on the associated selectable region.
  • menu hierarchies in automated systems built from menus of the type shown in FIG. 9 or FIG. 10, require that the operator proceed sequentially through the steps of searching menu options, selecting one of them, and, assuming a menu option including a submenu was selected, searching the submenu options, and selecting one of them.
  • selection from menu hierarchies constitutes a substantial component of the operator's activities, the slowness of the selection process diminishes productivity.
  • Locating selectable regions or parts thereof outside the display allows the large areas outside the display to be used, a major advantage for operators having impaired fine motor control who are unable to maintain a pointer on a small selectable region while selecting by click or by dwell.
  • menu options are displayed on the display near the perimeter of display and near their associated selectable region, the operator has an indication of the location of each selectable region but may not be able to see all the displayed menu options in a glance. Because an operator usually searches a displayed menu for his intended menu option, placing the menu only near the perimeter of the display may increase menu search time, thus increasing menu option selection time.
  • One object of the invention is to indicate to an operator of a menu system having selectable regions outside the display, the menu option associated with each selectable region and the location of each selectable region.
  • Another object of the invention is to facilitate selection from a menu by an operator having impaired motor control.
  • Still another object of the invention is to speed selection of a menu option from a menu and of a submenu option from a menu hierarchy.
  • a further object of the invention is to speed speech synthesis for a person having impaired speech and impaired motor control.
  • Conventional speech recognition systems facilitate computer access for individuals unable to use a standard keyboard whose speech is relatively unimpaired, for example, an individual with quadriplegia, and hands-free computer access for able-bodied individuals.
  • the operator of such a speech recognition system reads a menu option out loud, for example, "open file", and the system, which includes sound receiving means, for example, a microphone coupled to a sound board having a Digital Signal Processor ("DSP"), receives the sound of the read menu option, digitizes the sound of the read menu option, and then provides the digitized sound to another component of the speech recognition system, sound matching means which includes an application program for matching the digitized sound to one of a plurality of sounds, each representing respectively the sound of a spoken menu option. The system determines which sound best matches the sound of the read menu option and selects the menu option associated with this best matched sound.
  • DSP Digital Signal Processor
  • One object of the invention is to facilitate selection from a menu, and, in particular, from a menu of homophones.
  • Another object of the invention is to facilitate speech synthesis and voice activated computer access by individuals with speech impairments.
  • Still another object of the invention is to speed data entry in Ideographic Languages.
  • ideographs as the graphic symbols in written languages is found in many parts of the world.
  • An ideograph as used herein, is a graphic symbol used to represent an object, an idea or a word, without expressing, as in a phonetic system, the specific sounds forming the verbal expression of the object, idea or word.
  • Ideographic languages include Chinese, Japanese and Korean.
  • a graphic symbol, as used herein, includes, but is not limited to, each of the following: a letter of an alphabet, a Japanese kana, and an ideograph.
  • specific reference will be made herein to a preferred embodiment of the system and method as it applies to the Chinese language.
  • Prior art methods for selecting Chinese ideographs make use of various ideograph classification systems known to Chinese speakers.
  • the operator first specifies a class of ideograph, based on a first characteristic common to many ideographs. Ideographs having that common characteristic are displayed and the operator selects from among them, either directly, by selecting an individual ideograph, or indirectly, by specifying a second common characteristic usually dependent upon the first characteristic, thus further limiting the displayed ideographs to those having both the first and second common characteristics.
  • the operator may continue to specify characteristics until he has specified a unique ideograph.
  • ideograph classification system is called the Pin Yin System.
  • This classification system uses the phonetic structure of the Chinese language. In spoken Chinese there are approximately 412 basic phonetic units, each having a monosyllabic sound, for example, "nee", "how” and "ma”. Four intonations can potentially be applied to each phonetic unit, resulting in approximately 1280 distinct sounds. With 10,000 ideographs in current use, each represented by one of approximately 1280 distinct sounds, it is evident that many Chinese ideographs are homophones, i.e. have the same sound. Over 80% of Chinese ideographs have homophones. The Pin Yin System uses this limited number of phonetic units as the basis for its classification. Ideographs which are homophones are classified together; the common characteristic of the Pin Yin System is the distinct sound.
  • the operator specifies a distinct sound using a keyboard labeled with symbols representing the Latin alphabet (Pin Yin method) or Chinese phonetic units (Zhu Yin method).
  • the first key operation or sequence of key operations specifies the phonetic unit.
  • the second key operation specifies the intonation. In general, less than 15 ideographs have this sound, though in some cases there are many more homophones. These are displayed and the user selects from among them. In such cases, the operator, depending upon the system, may page through matching ideographs or specify another common characteristic to further limit the number of ideographs displayed. A common characteristic which may be used at this stage exploits another feature of the Chinese language.
  • a second common characteristic may limit matching ideograph pairs to a number sufficiently small for the operator to efficiently search and select from, or may uniquely specify an ideograph pair.
  • Another common characteristic the operator may specify to limit the number of matching ideographs is a meaning or meaning class to which one or more sequences of one or more ideographs belong.
  • An ideograph block is a sequence of four ideographs which together has its own meaning which may or may not be related to that of the constituent ideographs.
  • the operator specified a distinct sound for the second of two ideographs of an ideograph pair, so may the operator specify a distinct sound for the second, third and/or fourth ideograph of an ideograph block, to limit the number of matching ideograph blocks.
  • Ideographs are built from a set of 214 components, called radicals. Different radicals, perhaps placed within different locations within an ideograph, are combined to create an ideograph.
  • the operator specifies one or more radicals appearing in the ideograph he wishes to enter. He may, for example, use a keyboard having at least 214 keys, each corresponding to a radical, or may actuate a sequence of keys, the sequence corresponding to a radical.
  • Other common characteristics the operator may specify to limit the number of matching ideographs include a phonetic unit, the first brush stroke, and the last brush stroke used to draw the ideograph.
  • Another conventional ideograph classification system makes use of a classification of parts of ideographs.
  • the operator specifies the classification of the four corners of the ideograph he wishes to enter.
  • Other common characteristics the operator may specify to further limit the number of matching ideographs include the number of horizontal strokes used to draw the ideograph, and the classification of a certain part of the ideograph above the lower right corner.
  • Yet another conventional ideograph classification system makes use of a classification ideographs based on the basic strokes from which each ideograph is built.
  • each ideograph being composed of between 1 and 33 such strokes.
  • Ideographs may be classified by a small number of basic strokes, preferably according to strict rules regarding the order of stroke entry.
  • the operator specifies only the first and last basic strokes of the desired ideograph, then selects from a display of all ideographs sharing this first-last basic stroke combination.
  • Kana which includes hiragana and katakana.
  • ideographs are frequently combined with kana.
  • Kana may be may specified phonetically, for example, to designate the hiragana pronounced "ko" an operator of a Japanese word processing system may type "k” and then "o" on a Latin alphabetic keyboard or may type a single key associated with this hiragana.
  • Kana has multiple uses in a Japanese word processing system. Kana may represent itself, since kana may stand alone in Japanese text.
  • kana may be used to specify Japanese ideographs, either by specifying the radicals which compose Japanese ideographs or by specifying the pronunciation of Japanese ideographs.
  • a sequence of phonetic units specified by kana may represent that sequence of kana, a single Japanese ideograph, multiple Japanese ideographs, or a combination of one or more Japanese ideographs and one or more kana.
  • a single Japanese ideograph may have multiple pronunciations, including a Japanese pronunciation and a Chinese pronunciation, and may have multiple kana spellings.
  • Another drawback of many word processing systems for ideographic languages relates to the ease of copying a document. Ideally, the operator concentrates on the document to be copied, only occasionally scanning text he has input. For those word processing systems that display ideographs on a display for the operator's selection, the operator must frequently shift his gaze from the document to the display and back again. The operator cannot concentrate on both the document and the display simultaneously.
  • Ideographs also include the symbols of symbol sets used for communication by individuals who have hearing, speech or language impairments, for teaching literacy skills to those lacking them, including pre-literate children and individuals with intellectual disabilities, and for international written communication.
  • symbol sets include, but are not limited to, each of the following: Picture Communication Symbols, Rebus, Picsym, Pictogram Ideogram Communication Symbols, Yerkish, Blissymbolics and depictions of the signs of a manual sign language. Examples of symbols of the Picture Communication Symbols, Rebus, Picsyms, and Blissymbolics symbol sets are shown in FIG. 11, Pictogram Ideogram Communication Symbols in FIGS.
  • FIGS. 13(a)-13(j) Picture Communication Symbols, Rebus, Picsyms, Pictogram Ideogram Communication Symbols, Yerkish, and Blissymbolics are each described in Beukelman, David R. & Mirenda, Pat, Augmentative and Alternative Communication. Management of Severe Communication Disorders in Children and Adults, Paul H. Brookes Publishing Co., 1992, pp. 22-29.
  • Individuals who have not acquired or who have lost their literacy skills may use symbolic symbol sets in learning to read. If the individual lacks fine motor control, for example, due to cerebral palsy, the individual's disability may inhibit the acquisition of literacy skills by, for example, inhibiting repetition of an exercise by the individual, by limiting the individual's ability to participate in the classroom, or by making skill assessment by a teacher difficult so that the teacher may incorrectly believe that remediation is necessary or that a particular skill has been mastered. If the individual also has impaired speech, literacy acquisition is more difficult still.
  • One object of the invention is to display a menu of sequences of one or more ideographs on a display so that a large contiguous area on the display is not obstructed by the menu.
  • Another object of the invention is to facilitate ideograph entry in word processing systems for the Chinese, Japanese and Korean languages.
  • Still another object of the invention is to speed selection of sequences of graphics including one or more ideographs.
  • Yet another object of the invention is to allow an operator of a word processing system for an ideographic language to select a sequence of one or more ideographs without lifting either hand from the keyboard.
  • a further object of the invention is to indicate to an operator the progress toward selection of a dwell-selectable sequence of one or more ideographic characters.
  • a still further object of the invention is to synthesize speech for an operator having impaired speech.
  • a method of selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language includes several steps.
  • the first step is displaying on a display the plurality of sequences.
  • Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with one of a plurality of displayed sound indicators.
  • Each of the sound indicators respectively indicates one of a plurality of indicated sounds.
  • Each of the indicated sounds differs from each other indicated sound.
  • Another step is matching a received sound to any one of the indicated sounds.
  • Yet another step is selecting the sequence associated with the matched indicated sound.
  • a method of selecting a sequence of one or more graphic symbols in a homophone-rich language from a plurality of sequences of one or more graphic symbols in the language includes several steps.
  • One step is displaying the plurality of sequences.
  • Each of the plurality of sequences is associated respectively with a sound which is not a phonetic representation of the associated sequence.
  • Each of the plurality of sounds differs from each other sound in the plurality of sounds.
  • Another step is receiving a sound signal.
  • Still another step is matching the received sound signal to any one of the plurality of sounds.
  • Yet another step is selecting the sequence associated with the matched sound.
  • a method of editing a first sequence of two or more words in an ideographic language includes several steps.
  • One step is displaying on a display the first sequence.
  • the first sequence includes a plurality of second sequences of one or more graphic symbols in the language.
  • Each of the second sequences is associated respectively with a sound indicator.
  • Each of the sound indicators respectively indicates a sound.
  • Each of the indicated sounds differs from each of the other indicated sounds; and (b) the sound indicators.
  • Another step is receiving a sound.
  • Still another step is matching the received sound to any one of the indicated sounds.
  • a further step is replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.
  • an apparatus for use with a general purpose computer system including a display, for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols.
  • One or more sequences of the plurality of sequences includes an ideograph in an ideographic language.
  • the apparatus includes a computer readable medium, and a program, stored on the medium and executable by the general purpose computer system.
  • the program is operative to display the plurality of sequences on the display. Each of the sequences is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound.
  • Each of the plurality of indicated sounds differs from each of the other indicated sounds.
  • a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator.
  • the program is further operative to match a received sound to the sound indicated by the particular sound indicator, and to select the sequence associated with the particular sound indicator.
  • an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language includes a display on which may be displayed the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator.
  • the apparatus further includes a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.
  • an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language includes means for displaying the plurality of sequences.
  • Each of the plurality of sequences has a common characteristic in the ideographic language and each associated is respectively with a displayed sound indicator.
  • Each of the sound indicators respectively indicates a sound.
  • Each of the plurality of indicated sounds differs from each of the other indicated sounds.
  • a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator.
  • the apparatus further includes means for: (1) matching a sound to the sound indicated by the particular sound indicator; and (2) selecting the sequence associated with the particular sound indicator.
  • the device controller includes a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language.
  • Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator.
  • Each of the sound indicators respectively indicates a sound.
  • Each of the plurality of indicated sounds differs from each of the other indicated sounds.
  • a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator.
  • the device controller further includes a signal generating device, coupled to a controlled device, for: (1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and (2) generating a device control signal corresponding to the function of the controlled device.
  • a signal generating device coupled to a controlled device, for: (1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and (2) generating a device control signal corresponding to the function of the controlled device.
  • FIG. 1 is an illustration of a display showing a conventional on-screen keyboard.
  • FIG. 2 is an illustration of a display showing a conventional on-screen keyboard and output from a word processing application program.
  • FIGS. 3, 4 and 5 are each illustrations of the display of each step of letter selection using a conventional quaternary on-screen keyboard.
  • FIG. 6 is an illustration of a display showing a conventional quaternary on-screen keyboard and output from two application programs.
  • FIG. 7 is an illustration of a conventional device implementing joystick patterns.
  • FIG. 8 is an illustration of a display of a conventional eye gaze system.
  • FIGS. 9 and 10 are each illustrations of a display showing a conventional menu.
  • FIG. 11 depicts examples of symbols from the Picture Communication Symbols, Rebus, PicSym and Blissymbols symbol sets.
  • FIGS. 12(a)-FIG. 12(d) depict examples of symbols from the Pictogram Ideogram Communication Symbols symbol set.
  • FIGS. 13(a)-FIG. 13(j) depict examples of symbols from the Yerkish symbol set.
  • FIG. 14 is a block diagram of a computer which may be utilized in accordance with the present invention.
  • FIG. 15 is a block diagram of a speech synthesis system which may be utilized in accordance with the present invention.
  • FIG. 16 is an illustration of software components of an apparatus in accordance with an embodiment of the Perimeter Menu aspect of the invention.
  • FIGS. 17 and 18 are each illustrations of a display and structures in accordance with an embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 19 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 20 is an illustration of a display and structures in accordance with still another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 21 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 22 is an illustration of a display and structures in accordance with yet another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 23 is an illustration of an apparatus in accordance with a further embodiment of the Perimeter Menu aspect of the invention.
  • FIGS. 24 and 25 are each illustrations of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
  • FIGS. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 26 depicts a front view of the apparatus.
  • FIG. 27 depicts a cut away view from the top of the apparatus.
  • FIG. 28 is a top view of a headrest in accordance with an embodiment of the Perimeter Menu aspect invention.
  • FIGS. 29 and 30 illustrate the state table aPocketFsm in accordance with the preferred embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 31 is an illustration of a display and structures in accordance with the preferred embodiment of the Confinement aspect of the invention.
  • FIG. 32 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
  • FIG. 33 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
  • FIG. 34 is an illustration of an apparatus in accordance with the Dwell aspect of the invention and with the Path Directness aspect of the invention.
  • FIG. 35 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
  • FIG. 36 is an illustration of still another apparatus in accordance with the Dwell aspect of the invention.
  • FIG. 37 is an illustration of yet another apparatus in accordance with the Dwell aspect of the invention.
  • FIG. 38 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
  • FIG. 39 is an illustration of a display and structures in accordance with the preferred embodiment of the Path Directness aspect of the invention.
  • FIG. 40 is an illustration of a display and structures in accordance with an embodiment of the Path Directness aspect of the invention.
  • FIG. 41 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
  • FIG. 42 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
  • FIG. 43 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
  • FIGS. 44, 45 and 46 are each illustrations of a display and structures in accordance with the preferred embodiment of the Intersection aspect of the invention.
  • FIGS. 47 and 48 are each illustrations of a display and structures in accordance with another embodiment of the Intersection aspect of the invention.
  • FIGS. 49, 50 and 51 are each illustrations of a display and structures in accordance with the preferred embodiment of the Alignment aspect of the invention.
  • FIG. 52 is an illustration of a display and structures in accordance with the preferred embodiment of the Location Indication and the Length Order aspects of the invention.
  • FIG. 53 is an illustration of a display and structures in accordance with an embodiment of the Location Indication aspect of the invention.
  • FIGS. 54 and 55 are each illustrations of a display and structures in accordance with another embodiment of the Location Indication aspect of the invention.
  • FIG. 56 is an illustration of a display and structures in accordance with a still further embodiment of the Location Indication aspect of the invention.
  • FIG. 57 is an illustration of a display and structures in accordance with a further embodiment of the Location Indication aspect of the invention.
  • FIGS. 58 and 59 are illustrations of a display and structures in accordance with the preferred embodiment of the Sound Match aspect of the invention.
  • FIG. 60 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
  • FIG. 61 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
  • FIG. 62 is a block diagram of a speech recognition system which may be utilized in accordance with the Sound Match aspect of the invention.
  • FIG. 63 is an illustration of software components of an apparatus in accordance with an embodiment of the Sound Match aspect of the invention.
  • FIG. 64 is an illustration of a display and structures in accordance with the preferred embodiment of the Ideographic Language aspect of the invention.
  • FIG. 65 is an illustration of a display and structures in accordance with another embodiment of the Ideographic Language aspect of the invention.
  • FIG. 66 is an illustration of a display and structures in accordance with still another embodiment of the Ideographic Language aspect of the invention.
  • FIG. 67 is an illustration of a display and structures in accordance with yet another embodiment of the Ideographic Language aspect of the invention.
  • FIGS. 15 depicts a block diagram of the hardware components of the prototype (2214), including a conventional general purpose computer system (2218), an optional pointer (2202), an optional printer (2220) and a speech synthesizer (2206).
  • the general purpose computer system (2218) includes a conventional computer system (2116), a storage unit (2208), a keyboard (2210), and a diskette drive (2216).
  • FIG. 14 depicts a block diagram of the conventional computer system (2116), including a processing unit (2102) and a display (2112).
  • the processing unit (2102) includes a processor (2104), a memory (2106) and control circuitry (2108).
  • the prototype employs the Toshiba T6400DXC general purpose computer system manufactured by Toshiba Corporation, Kawasaki, Japan.
  • the T6400DXC is preferably substituted with the IBM ThinkPad 755C computer system, part number 9545F0C, manufactured by IBM Corporation, Armonk, N.Y., USA., because the former requires a 110VAC power source while the latter is powered by an integral battery.
  • An integral battery allows an NMD operator to use the system when a 100VAC source is not available or when attaching to a 110VAC power source is inconvenient.
  • the prototype further includes a head mounted pointer communicating via an infrared link with the computer system so that there are no cables tethering the operator to the computer system. Any cable between the operator and the computer system would have to be connected, probably by an attendant since the operator may lack the fine motor skills required to make such a connection.
  • Preferably computer access can be accomplished independently by the operator.
  • the choice of a pointing device is primarily dictated by the particular capabilities of the operator. Usually the best pointing device for a particular operator is the one drawing on that operator's best motor control. For example, if an operator's foot control is superior to his head control, a pointing device using his foot is preferably to a head pointer.
  • the prototype employs the Remote Headmaster® manufactured by the Prentke Romich Company, Wooster, Ohio, USA.
  • the combination of the HeadMaster® Plus, part number HM-1P, HeadMaster® Plus Remote Adapter, part number HM-RA, and HeadMaster® Plus Laptop Adapter, part number HM-LA, all available from the Prentke Romich Company is preferable because the headset is more comfortable and the HeadMaster® Plus ultrasonic transmitter mounts more easily on a laptop computer system than the Remote Headmaster ultrasonic transmitter.
  • the preferred embodiment further includes a battery powered printer, the MobileWriter®, part number 730879, manufactured by Mannessmann Tally Corporation, Kent, Wash., USA and a speech synthesizer, the Multivoice Speech Synthesizer, part number MV2-SS, manufactured by The Institute on Applied Technology, Children's Hospital, Boston, Mass., USA.
  • the pointer (2202) is a device which provides data concerning the relative or absolute position of the operator or any body member of the operator.
  • the display (2112) and pointer (2202) together provide for the interactive nature of the general purpose computer system (2218) in that, in accord with the various aspects of the invention, the interpretation that the processor (2104) gives to a certain pointer action made by the operator depends, in the majority of situations, upon what is being displayed to the operator at that time.
  • the prototype (2214) shown in FIG. 15 further includes a keyboard (2210), which functions to provide input from an able-bodied operator to the general purpose computer system (2218).
  • the keyboard (2210) is useful for configuration, diagnostic and backup purposes, functions which are performed relatively infrequently and usually require an able-bodied person for ancillary activities, for example, loading backup media into the general purpose computer system.
  • the prototype (2214) also optionally includes a printer (2220) which functions to provide hard copy output of data developed or stored in the general purpose computer system, and a speech synthesizer (2206), which functions to provide speech output for utterances and words composed using or retrieved from the general purpose computer system (2218).
  • the couplings between the devices depicted in FIG. 15 may be made by any means which permits the orderly and timely exchange of data across the interface.
  • the interfaces between the pointer (2202) and the general purpose computer system (2218) and between the general purpose computer system (2218) and the speech synthesizer (2206) conform to the Electronic Industries Association RS-232 interface specification.
  • the interface between the general purpose computer system (2218) and the printer (2220) conform to the Centronix 50 pin parallel interface specification.
  • the software component of the prototype are stored in memory (2106) and executed on the processing unit (2102).
  • the software component of the prototype, depicted in FIG. 16, include a software driver (1202), an operating system (1204), an optional database program (1210), and the prototype access program code and data, hereinafter collectively referred to as the "access program" (1206).
  • one or more application programs (1208) may also execute on the processing unit (2102) and accept control and data from the access program (1206) via the operating system (1204).
  • the software driver (1202) of the prototype is the Logitech Mouse Driver included with Windows® version 3.1.
  • the operating system (1204) of the prototype is Windows® version 3.1 in combination with MS-DOS® version 6.2.
  • the operating system is referred to simply as "Windows®", available from Microsoft Corporation, Redmond, Wash., USA.
  • the optional database program (1210) is described in the detailed description of the Length Order aspect of the invention.
  • the prototype access program (1206) is described in detail below.
  • the software components of the prototype are stored in memory (2106).
  • portions of these programs may be transferred as needed between memory (2106) and the storage unit (2206) or between memory (2106) and a diskette in the diskette drive (2216) depicted in FIG. 15.
  • the basic function of the storage unit (2206) and the diskette drive (2216) is to store programs and data that are employed by the general purpose computer system (2218) and which may readily be transferred to the memory (2106) when needed.
  • the processing unit of the general purpose computer system may be substituted with a microprocessor coupled to custom electronics for performing the functions of the various aspects of the invention, or the color display of the prototype may be substituted with a monochrome display.
  • a selectable region is a region, delimited with respect to a display or a surface, and associated with a menu option which may be selected, usually by a selection event.
  • a subregion is a selectable region that is included within another selectable region. Thus a subregion is, by itself, a selectable region. Assuming that a certain selectable region includes subregions A and B, dwell time on subregions A and B may be combined, for example, by summing, so that dwelling on either subregion A or B or a combination of both for the selection threshold period selects the menu option associated with the selectable region.
  • FIG. 17 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) and eight selectable regions.
  • Each of the eight selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion located outside the display (2112) and adjacent the visible subregion.
  • the selectable region at 11 o'clock in FIG. 17 labeled with menu option "vort ⁇ space>x" consists of invisible subregion (0104) and visible subregion (0106), and within this description of the Perimeter Menu aspect of the invention is referred to as selectable region (0104/0106).
  • selectable region (0104/0106
  • selectable region (0104/0106) are (0108/0110), (0112/0114), (0116/0118), (0120/0122), (0124/0126), (0128/0130) and (0132/0134).
  • Each subregion may be sized to suit the operator's preferences and abilities.
  • Each selectable region is associated respectively with a menu option. In FIG.
  • selectable region (0104/0106) is associated with menu option vort ⁇ space>x, selectable region (0108/0110) with menu option "sumac", selectable region (0112/0114) with menu option "wizen'", selectable region (0116/0118) with the menu option undo indicated by an icon on visible subregion (0118) representing an undo function, selectable region (0120/0122) with menu option "words”, selectable region (0124/0126) with menu option "talk”, selectable region (0128/0130) with menu option “ldhbfk” and selectable region (0132/0134) with menu option "ypgqj,".
  • the eight visible subregions circumscribe region (0150) on the display.
  • Selectable regions may be delimited by data indicative of one or more boundaries of the selectable region. Equivalently, the delimit means may be detectors operative to determine when the location indicated by the movement related signal has crossed one of those boundaries or intersects a selectable region. A partially delimited region or subregion is one which is unbounded on at least one side.
  • the operator moves a pointer (2202 in FIG. 15) coupled to the general purpose computer system (2218 in FIG. 15) to indicate a location on the selectable region, including either subregion, associated with the desired menu option and maintains the indicated location on the selectable region for the selection threshold period.
  • the period of time required for selection may vary responsive to the proximity of the indicated location to the location of a cursor on the display or to the proximity of the indicated location to a point within the intersected selectable region. Dwell time may be continuous, discontinuous or dynamic (described below) for either or both subregions of the selectable region.
  • a dwell event includes, but is not limited to, each of the following: (a) the durations of one or more periods of intersection of locations indicated by a movement related signal, a body member or a cursor (including any part of the cursor) and a selectable region equalling or exceeding a predetermined period; (b) a first quantity responsive to the durations of the periods referred to in (a) equalling or exceeding a predetermined quantity; (c) dwell event (a) or (b) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the intersected selectable region; and (d) dwell event (a) or (b) wherein the period of intersection required for selection of a selectable region increases in response to a non-intersection or a period of non-intersection of locations indicated by the movement related signal, the body member or the cursor and the selectable region ("dynamic dwell event").
  • non-intersection or a period of non-intersection in determining the duration of a period of intersection required for selection is called dynamic dwell.
  • an intersected selectable region Associated with each type of dwell event is an intersected selectable region. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor which triggers the dwell event by causing the period or the first quantity to equal or exceed the predetermined period or the predetermined quantity, respectively.
  • Selection may also be in response to a selection event.
  • a selection event includes, but is not limited to: (a) a dwell event; (b) a switch operation at or near the time of an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; (c) an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; and (d) selection event (c) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the selectable region it previously intersected.
  • an intersected selectable region is associated with each type of selection event. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor.
  • the fact that a selection event has occurred may be indicated to the operator, for example, visually by changing the cursor appearance or location, by changing location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font or fill of an object on the surface, or by displaying a graphic or a point distinguishable from its immediate surroundings on a surface or removing a graphic or point distinguishable from its immediate surroundings from a surface; auditively by generating a sound or changing the pitch or volume of an extant sound; tactilely by changing the surface or temperature of a contact area or the pressure exerted by a contact area; or by other means.
  • the hue of the visible subregion of the selected selectable region is changed from green to magenta.
  • a cursor includes a temporary marking on a display which emphasizes to an operator, in an optical manner, a momentarily important location or object.
  • body member means any part of the body including, but not limited to, each of the following: the shoulder, arm, elbow, wrist, hand, finger, thumb, leg, knee, ankle, foot, toe, hip, trunk, neck, tongue, lip, eye and head.
  • the received movement related signal includes, but is not limited to, a signal indicative of movement or from which movement can be derived, such as a plurality of relative or absolute positions or a difference between two relative or absolute positions.
  • Movement related signal receiving means includes, but is not limited to, each of the following: (a) pointer interface circuitry found in a general purpose computer system; (b) one or more detectors operative to detect movement of a pointer; and (c) one or more detectors operative to detect movement of a body member of an operator.
  • the movement related signal receiving includes electronic circuitry in the general purpose computer system (2218) operative to receive the movement related signal generated in part by the movement of the pointer (2202).
  • subregions are displayed on the display (2112).
  • other means for displaying may be substituted for the means used in the prototype, for example, a projector for projecting an image, a surface having a static display thereon, or other suitable means.
  • each of six selectable regions is now associated with a submenu option of the selected menu option "vort ⁇ space>x".
  • Selectable region (0104/0106) is now associated with submenu option " ⁇ space>", selectable region (0108/0110) with submenu option "o”, selectable region (0112/0114) with submenu option "t”, selectable region (0124/0126) with submenu option "x”, selectable region (0128/0130) with submenu option "v”, and selectable region (0132/0134) with submenu option "r”.
  • Selectable regions (0116/0118) and (0120/0122) remains associated with the same menu options with which each was associated in FIG. 17. The operator may now select one of these submenu options.
  • the selected character may be input to an apparatus coupled to the general purpose computer system (2218 in FIG. 15), or input to an application program (1208) executing on the general purpose computer system (2218) coupled to the display (2112).
  • Inputting, as used herein, includes, but is not limited to, generating or passing signals representative of the selected menu option along a path toward the destination apparatus or program.
  • the computer program displays at least some of its output in the circumscribed region (0150).
  • an operator may, with a single selection indicate one of eight menu options, with two selections indicate one of up to 64 different menu options, with three selections indicate one of up to 256 menu options, etc.
  • Each of these menu options may represent a sequence of one or more characters, a sequence of one or more data or control inputs to an application program (1208), or a control function for one or more devices or speech synthesizers coupled to the general purpose computer system (2218).
  • a character includes a space, a control character as defined by the American National Standards Institute (ANSI) or the American Standard Code for Information Exchange (ASCII), and a letter from one of the linguistics, Bulgarian, Amharic, Arabic, Armenian, Assamese, Assyrian, Avar, Azerbaijani, Balinese, Bamara, Bantu, Bashkir, Basque, Bengali, Birhari, Bulgarian, Buluba-Lulua, Burmese, Buryat, Byelorussian, Caddoan, Catalan, Chechen, Chikaranga, Chippewa, Choctaw, Church Slavik, Chuvash, Coptic, Cree, Croatian, Cyrillic, Czech, Dakota, Danish, Dari, Devanagari, Dutch, Dzongkha, English, Eskimo, Esperanto, Estonian, Ewe, Farsi, Fijian, Weg, Finnish, Flemish, French, Fulani, Gaelic, Galician, Georgian
  • each of a character, ideograph, control input and control function includes a computer encoding of the same.
  • a device includes, but is not limited to, each of a wheelchair, a household appliance, an appliance for use in an office, a workstation, a robot, and a computer peripheral.
  • the operator may, for example, increase the volume of an external speech synthesizer, or turn a wheelchair to the left.
  • the selectable regions organized as described above help an NMD operator make the menu selection he intends.
  • an NMD operator intends to move a pointer (2202) that is indicating point (0154) to indicate point (0156), a location in subregion (0130), but who is unable to quickly stop motion, so that the location indicated by the pointer (2202) moves from point (0154) past point (0156) to point (0158). Because point (0158) lies within the same selectable region (0128/0130) as the overshot subregion (0130), dwelling at point (0158) operates to select the intended selectable region (0128/0130).
  • Invisible subregions in accordance with the Perimeter Menu aspect of the invention, may extend outward from the edge of the display (2112) to infinity. In such an embodiment, dwelling at point (0162) would operate to select selectable region (0128/0130).
  • invisible subregions extend a finite distance from the edge of the display (2112). In such an embodiment, dwelling at point (0162) would not operate to select selectable region (0128/0130).
  • the sizes of the invisible subregions shown in FIGS. 17 and 18 are illustrative only. Preferably the size of each invisible subregion is large enough to encompass overshoot but small enough to avoid unintentional selections when the NMD operator turns to see someone or something.
  • the prototype utilizes the area outside the display to facilitate menu selection by a disabled operator. If an operator has impaired ability to maintain a steady position, he can point to a relatively large invisible subregion outside the display which is more forgiving of the operator's involuntary motion than the relatively small selectable regions on the display in conventional on-screen keyboards. Thus, the effective area of a selectable region is expanded beyond the region's visible subregion shown on a display. If an operator has impaired ability to stop motion he may, starting from the center of the screen, point to any selectable region. His impaired ability to stop will not impair his ability to select his intended target, assuming his directional control is relatively unimpaired, since, in the prototype, each selectable region is unbounded on its side furthest from the center of the display.
  • the Perimeter Menu aspect of the invention is preferably implemented on a general purpose computer system. If the general purpose computer system is coupled to a speech synthesizer and the menu hierarchy allows the selection of letters and/or words, an operator having impaired speech may speak using the speech synthesizer. If a word processing or data entry application program is run on the general purpose computer system, the operator may enter words or data, respectively, for input to the application program. If the general purpose computer system is coupled to a devices capable of executing commands and the menu hierarchy allows the selection of commands, a disabled operator may select and issue commands to control these devices.
  • the prototype places less cognitive demand on the operator than a scanning system for selecting options from a menu.
  • FIG. 19 illustrates a display in accordance with an alternative embodiment of the Perimeter Menu aspect of the invention having eight selectable regions circumscribing a central region (6950) on the display.
  • FIG. 20 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having twenty selectable regions circumscribing a central region (0806) on the display.
  • FIG. 21 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having four selectable regions (0508), (0506), (0504) and (0502) circumscribing a central region (0510) on the display.
  • FIG. 22 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having four selectable regions (4008), (4006), (4004) and (4002) with no space between them other than a circumscribed region (4010) on the display.
  • FIG. 23 illustrates an apparatus in accordance with of another embodiment of the Perimeter Menu aspect of the invention.
  • region (3510) is located on interior display (3514) which is circumscribed by peripheral display (3512).
  • Selectable regions (3508), (3535), (3504) and (3502) are located on the peripheral display (3512).
  • FIGS. 24 and 25 are each illustrations of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 24 depicts ten selectable regions (6502), (6504), (6506), (6508), (6510), (6512), (6514), (6514), (6516), (6518), and (6520).
  • Each selectable region is located on the display (2112) adjacent the edge of the display and associated respectively with a menu option.
  • the menu options are shown on their associated selectable region.
  • the ten selectable region circumscribe region (6550) on the display.
  • the display changes to that shown in FIG.
  • FIGS. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter Menu aspect of the invention.
  • FIG. 26 depicts a front view of the apparatus;
  • FIG. 27 a cut away view from the top of the apparatus.
  • FIG. 26 depicts detector area (0348 in FIG. 27) on, in or below which are located a plurality of selectable regions (0304), (0306), (0308), (0310), (0312), (0314), (0316), (0318), (0320), (0322), (0324), (0326), (0328), (0330), (0332), (0334), (0336), (0338), (0340) and (0342).
  • Adjacent the detector area (0348 in FIG. 27) is a berm (0350 in both FIG. 26 and 27) for confining a body member of the operator or a pointer controlled by the operator to the detector area (0348 in FIG. 27).
  • FIG. 28 depicts a headrest for an operator using his head to indicate a location on a display.
  • FIG. 28 shows an irregularity (0703) on the surface (0701) of the headrest.
  • the irregularity is in physical contact with the operator and tactilely indicates to him the position of his head.
  • the tactile indication means may be concave, convex or both or may differ from the surface in temperature. For individuals having impaired ability to sense the position of a body member, e.g. the operator's head, the tactile input thus provided to the operator improves the operator's ability to sense the position of his head.
  • a state table is a tool for processing sequential inputs and is most easily understood by analogy. Imagine yourself in a room having a ticket window and three exits, each regulated by a turnstile. You collect a ticket at the ticket window which, when inserted into the appropriate turnstile, allows passage to a connecting room. The turnstile keeps the ticket. Any given ticket operates only one turnstile in a room, though different tickets may operate the same turnstile. You begin in a certain room, collect a ticket, insert it into the appropriate turnstile and pass to a connecting room, where you perform certain tasks associated with the new room. Then you collect another ticket from the ticket window in that room, insert the ticket into the appropriate turnstile in that room, pass to a connecting room, perform certain tasks associated with the new room, and so on.
  • the state table used in the prototype is depicted in FIGS. 29 and 30.
  • the rooms are states represented by the rows of the state table; the tickets are events represented by the columns of the state table.
  • Each entry in the state table represents a passage from one state to another.
  • the tasks performed upon entry into a room correspond to the processing performed by the processor (2104) on entry to a new state ("state processing").
  • state processing may cause a slight lightening of the color of a selectable region. Reentry into that state five times may successively lighten a selectable region five times.
  • each selectable region is directly controlled by one associated state machine.
  • Each state machine directly controls only its associated selectable region.
  • Each state machine includes data uniquely associated with its associated selectable region, the shared state table shown in FIGS. 29 and 30, and the shared code for state processing, described below.
  • This embodiment means that the state machine associated with selectable region (1614) may be in state ST -- SELECTED while the state machine associated with selectable region (1602) is in state ST -- INITIAL.
  • these separate states are reflected only in separate values for data uniquely associated with each state machine.
  • Each state machine has a unique index.
  • the state table used in the prototype defines 18 states, composed of states zero through seventeen shown in FIGS. 29 and 30.
  • State 1 (ST -- ERROR -- STATE in FIG. 29) is not used. Preferably, it is omitted.
  • the differences between the several state machines for example, the state of a particular state machine at any given time, are confined to data structures associated with that state machine.
  • the state machines share the same code and the same state table. In other words, returning to the ticket and turnstile analogy, there are multiple travelers each with his own ticket (event) and his own baggage (data) moving from room to room in the same labyrinth. Each traveler's actions in each room usually affect only his own baggage.
  • External events are generated outside the state table, for example, by the operator moving the pointer (2202) or by a timer expiring.
  • Pointer movement may generate an event indicating that the operator has moved the cursor across the selectable region boundary from without the selectable region to within it. This event causes a transition from one state to another ("drives" a state machine to a new state). For example, assuming a state machine is in state ST -- CREST -- TIDE, row 6 in the state table shown in FIG. 29, when an event EV -- CROSS -- OUT, column 4 of the state table, occurs. At the intersection of row 6 and column 4 is a 7.
  • event EV -- CROSS -- OUT drives the state machine from state ST -- CREST -- TIDE to state ST -- SELECTED.
  • the computer performs the state processing associated with the new state.
  • Internal events are generated during state processing to handle circumstances where a first state transition is made due to an external event and the processing associated with the new state determines that a second state transition is necessary.
  • the first, external, event has already been used so a second, internal, event is generated by the state.
  • the cursor may sit without moving on a selectable region for a considerable period of time. It is desirable to detect this condition, move the cursor to the center of the screen so the operator can easily find the cursor, and reset all selectable regions to their initial color. Detection is accomplished with a timer.
  • state ST -- IDLE which centers the cursor. From there it is desirable to transition to state ST -- RESET which, among other processing, initializes selectable region color.
  • state ST -- IDLE is driven by an internal event, generated by state ST -- IDLE state processing.
  • each state machine may be equivalently represented as an instantiation of a selectable region class for processing inputs affecting a particular selectable region.
  • processors may include processors, electronic circuitry, state tables or code for state processing used for processing fewer than all the selectable regions or used to process certain selectable regions at one time and other selectable regions at other times.
  • each state machine processes events independently of all other state machines, though a state machine may send an event to another state machine. For example, when the operator selects a selectable region, the associated state machine sends the event EV -- RESET to all other state machines so that all selectable regions revert to their respective initial colors.
  • a single operator action may result in the issuance of different events to different state machines. For example, when the operator moves the cursor from without a selectable region to within it, EV -- DWELL is sent to the newly indicated state machine. EV -- MOVEMENT is sent to all other state machines.
  • each event used in the prototype and an example of the use of each event.
  • For all uses of each event refer to the state table shown in FIGS. 29 and 30 which determines what state transition occurs from every state on occurrence of a given event.
  • the event EV -- RESET is an internal event which drives a state machine to its initial state. For example, when the operator has not moved the pointer (2202) for a predetermined period of time, event EV -- RESET is sent to all other selectable regions.
  • the event EV -- DECAY is an external event which indicates that the cursor hotspot does not intersect the selectable region associated with the state machine.
  • EV -- DECAY is sent to a state machine periodically when the operator has positioned the cursor hotspot on a selectable region other than the selectable region associated with that state machine.
  • the event EV -- DWELL is an external event which indicates that the cursor hotspot intersects the selectable region associated with the state machine.
  • EV -- DWELL is sent to a state machine periodically when the operator has positioned the cursor hotspot on the associated selectable region.
  • the event EV -- CROSS -- OUT is an external event which indicates that the cursor hotspot has moved from a location intersecting the selectable region associated with the state machine to a location not intersecting the selectable region. After the operator selects a selectable region, he must move the cursor hotspot out of the selectable region, generating EV -- CROSS -- OUT, before he can again select that selectable region.
  • the event EV -- STEP -- UP is an internal event which indicates that a selectable region's selection threshold has been satisfied.
  • the event EV -- MOVEMENT is an external event which indicates that the cursor hotspot has moved. If the cursor hotspot intersects a selectable region without moving for a predetermined period of time, a timeout occurs, causing all state machines to transition to the reset state. EV -- MOVEMENT drives the state machine out of the reset state.
  • the event EV -- IDLE -- TIMEOUT is an external event which indicates that the cursor hotspot has intersected a selectable region without moving for a predetermined period of time. EV -- IDLE -- TIMEOUT causes the state machine to move the cursor hotspot to the center of the display.
  • EV -- CEILING is an external event which indicates that the cursor hotspot intersects a selectable region and the color of the selectable region equals the selectable region color ceiling. If the locking feature is enabled, EV -- CEILING drives the state machine to the begin lock state where it displays the lock icon.
  • the null event, EV -- NULL is a multi-purpose internal event used in a variety of situations to drive a state machine to another state. For example, after a timeout has been detected, EV -- IDLE -- TIMEOUT is generated and sent to the appropriate state machine driving it to the idle state, the receiving state machine sends itself EV -- NULL in order to drive itself to the reset state.
  • EV -- NULL allows states to be simpler and the reset state to be reused.
  • the prototype uses eight partially delimited selectable regions.
  • the portion of each selectable region shown on the display is referred to as the visible subregion of the selectable region.
  • the portion of each selectable region outside the display is referred to as the invisible subregion of the selectable region.
  • the access program (1206) of the prototype only reads hotspot cursor locations within the Windows® cursor clipping rectangle, even though the operator may in fact be pointing to a location outside the Windows® cursor clipping rectangle, e.g. within an invisible subregion.
  • the access program (1206) does not distinguish between two locations indicated by the operator, the first at a first location on the edge of the Windows® cursor clipping rectangle and the second outside the Windows® cursor clipping rectangle whose location is reported by the software driver (1202) to be the first location.
  • the software driver (1202) in the prototype reports the cursor hotspot location to be the closest point within the visible subregion (0106). Consequently, in the prototype, all invisible subregions are unbounded on their side furthest from and parallel to the edge of the display.
  • point (0162) in FIG. 17 lies within selectable region (0128/0130) since the rightmost side of selectable region (0128/0130) is unbounded.
  • pPocket ⁇ State indicates a particular item of data within the set of data associated with the state machine, in this case, the variable "State”.
  • the Windows® cursor clipping rectangle is set so that most of the arrow cursor is always visible on the display.
  • the cursor indicating on the display (2112) the location indicated by the movement related signal receiving means, is positioned at the center of the display.
  • aLabel an array of data structures defining the menu and submenu options and the menu hierarchy, is initialized.
  • one of the elements of aLabel defines menu option "vort ⁇ space>x".
  • This element includes fields which determine that this menu option is displayed horizontally starting at certain (x,y) coordinates, that on selection certain actions are to be taken, for example, outputting text to a speech synthesizer, and that on selection certain submenu options, in this example, "v”, “o”, “r”, “t”, “ ⁇ space>” and "x”, and related data are to be associated with certain state machines.
  • this association is accomplished by modifying pLabel in the set of data of the associated state machine to point to the aLabel element corresponding to the menu option to be associated with that state machine.
  • At least one window is created in the circumscribed region (0150) for the display of selected letters.
  • the state table is initialized to the values shown in FIGS. 29 and 30.
  • EV -- RESET is sent to each state machine by a procedure call of the form PocketFsm (pPocket, EV -- RESET).
  • pPocket indicates the state machine associated with selectable region (0104/0106).
  • fInternalEvent is set to TRUE and consequently control passes into the while loop.
  • fInternalEvent is now set to FALSE.
  • the current state, pPocket ⁇ State is stored in pPocket ⁇ PreviousState. Now a state transition is made.
  • the current state, ST -- INITIAL, having a value of 2, and the current event, EV -- RESET, having a value of 1, are used as row and column indices respectively into the state table aPocketFsm shown in FIGS. 29 and 30, to determine the value of the new state of the state machine, in this example, the state machine associated with selectable region (0104/0106).
  • aPocketFsm[2][1] equals 3.
  • the new state of the state machine is 3, the value of ST -- RESET.
  • Control passes, via the switch statement, to the ST -- RESET case and ST -- RESET state processing is performed.
  • the time of selectable region selection is set to the current time, the state machine's fInvert flag is set to FALSE and the value of the state machine's color variable, pPocket ⁇ Color, is compared to the state machine's initial color, pPocket ⁇ InitialColor.
  • pPocket ⁇ Color was initialized to zero, which is not the value of pPocket ⁇ InitialColor. Consequently, pPocket ⁇ Color is set to pPocket ⁇ InitialColor and the flag pPocket ⁇ fPaint is set to TRUE.
  • control passes through the bottom of the switch statement and the value of pPocket ⁇ fPaint is tested. Since it is TRUE, the client area rectangle is invalidated.
  • Windows® 1204 responds to invalidating the client rectangle by sending the access program (1206) a WM -- PAINT message.
  • the access program (1206) On receipt of a WM -- PAINT message, the access program (1206) redraws all selectable regions and any menu option located thereon for each state machines having pPocket ⁇ fPaint equal to TRUE. Thus, somewhat indirectly, visible subregion (0106) is drawn on the display (2112). Then control returns to Windows® (1204).
  • the other seven state machines are similarly initialized so that each state machine transitions from ST -- INITIAL to ST -- RESET and draws its respective visible subregion and any menu option located thereon on the display (2112).
  • the display shown in FIG. 17 now appears on the display (2112) of the computer system (2116).
  • a periodic timer called the cursor polling timer.
  • This timer provides the access program (1206) with a WM -- TIMER message at frequent intervals, in the prototype every system clock tick which occurs approximately every 54 milliseconds.
  • the access program (1206) calls the procedure CreateEvent to determine the appropriate event for each state machine and to complete the event data structure addressed by pEvent accordingly, then repetitively calls the procedure PocketFsm for each state machine, passing the unique indicator for the state machine and the appropriate event for that state machine.
  • the cursor polling timer is more frequent so that color changes to visible subregions are smaller and more frequent, giving a smoother appearance to color change.
  • the operator is moving the cursor from its initial location in the center of the display toward selectable region (0104/0106), but since only 54 milliseconds have elapsed, the cursor hotspot has moved only slightly in that direction.
  • the procedure CreateEvent determines that the cursor hotspot does not lie within any selectable region and that the cursor hotspot has not crossed out of a selectable region in the past 54 milliseconds. Therefore, the procedure CreateEvent determines that each state machine should receive EV -- MOVEMENT.
  • the procedure PocketFsm is called with the indicator for the state machine associated with selectable region (0104/0106) and with EV -- MOVEMENT.
  • the event EV -- MOVEMENT drives this state machine from its current state, ST -- RESET, to ST -- EBB -- TIDE.
  • the pseudo-code for ST -- EBB -- TIDE in procedure PocketFsm is a break statement, indicating that no state specific action is taken at this time, other than the transition to ST -- EBB -- TIDE. Control returns to Windows® (1204).
  • the ST -- DECAY state sets fInternalEvent to TRUE, and, in this case, sets Event to EV -- NULL.
  • fInternalEvent condition is true and another state transition occurs, using the value of pPocket ⁇ State which was set by ST -- DECAY state processing to ST -- EBB -- TIDE, the previous state.
  • the new state is found at aPocketFsm[ST -- EBB -- TIDE][EV -- NULL], which equals ST -- EBB -- TIDE.
  • This state transition is unlike an ordinary state transition because the starting state is set by ST -- DECAY. All transitions from ST -- DECAY share this distinction.
  • the state machine executes the code for the new state, ST -- EBB -- TIDE, which is simply a break statement.
  • the procedure PocketFsm determines that fpaint is FALSE and exits. Control returns to Windows® (1204).
  • fInternalEvent is set to TRUE and Event is set to EV -- NULL, resulting in another state transition to aPocketFsm[ST -- ENTRY][EV -- NULL], which equals ST -- LOW -- TIDE.
  • the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV -- MOVEMENT, which drives it from ST -- LOW -- TIDE to ST -- DWELL. Stepping through the pseudo-code for ST -- DWELL, the ST -- DWELL state sets pPocket ⁇ State to the value stored in pPocket ⁇ PreviousState, increments pPocket ⁇ Color by pPocket ⁇ Increment, but not above the value of pPocket ⁇ Ceiling, sets pPocket ⁇ fPaint to TRUE, sets fInternalEvent to TRUE, and, in this case, sets Event to EV -- NULL.
  • the access program (1204) checks the value of fpaint for each state machine, and if TRUE, sets fpaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine. After redrawing, control returns to Windows® (1204).
  • the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST -- LOW -- TIDE to ST -- DWELL to ST -- LOW -- TIDE, driven by the cursor polling timer. With each transition to ST -- DWELL, visible subregion (0106) is brightened a bit. The polling timer interval is short enough and the increment to pPocket ⁇ Color is small enough that visible subregion (0106) appears to gradually brighten although in fact it progresses rapidly through a series of discrete brightness levels. With each iteration through ST -- DWELL, pPocket ⁇ Color in incremented.
  • Event is set to EV -- STEP -- UP, driving a transition to aPocketFsm[ST -- LOW -- TIDE][EV -- STEP -- UP], which equals ST -- SELECTED.
  • the state processing for ST -- SELECTED provides the operator with an audible indication that a selection has just been made, takes the action appropriate upon selection of this selectable region, including selecting the menu option associated with the selected selectable region.
  • the access program (1206) does not generate output to another program or device at this time.
  • ST -- SELECTED next sets pPocket ⁇ fInvert to TRUE, and, if appropriate, changes the menu options associated with various selectable regions.
  • selectable region (0104/0106) is now associated with menu option " ⁇ space>", selectable region (0108/0110) with menu option "o”, selectable region (0112/0114) with menu option "t”, selectable region (0124/0126) with menu option "x”, selectable region (0128/0130) with menu option "v”, and selectable region (0132/0134) with menu option "r".
  • Selectable regions (0116/0118) and (0120/0122) remains associated with the same menu options with which they were associated in FIG. 17.
  • fpaint is set to TRUE for the state machines associated with the selectable regions having changed menu options.
  • ST -- SELECTED next sets Event to EV -- NULL and fInternalEvent to TRUE.
  • the state machine now makes the transition to aPocketFsm[ST -- SELECTED -- TIDE][EV -- NULL], which equals ST -- CREST -- TIDE.
  • the pseudo-code for ST -- CREST -- TIDE is only a break statement. Control returns to Windows® (1204).
  • the setting of fInvert to TRUE causes the visible subregion (0106) to be drawn in the color complementary to the value then indicated by pPocket ⁇ Color.
  • the setting of fpaint to TRUE for all selectable regions associated with changed menu options causes those selectable regions and the menu options thereon to be redrawn.
  • the display shown in FIG. 18 now appears on the display (2112) of the computer system (2116).
  • the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST -- CREST -- TIDE to ST -- DWELL to ST -- CREST -- TIDE, driven by the cursor polling timer.
  • ST -- DWELL With each transition to ST -- DWELL, visible subregion (0106) is brightened a bit, though it is now magenta, the complement of green.
  • the polling timer interval is short enough and the increment to pPocket ⁇ Color is small enough that visible subregion (0106) appears to gradually brighten although in fact it progresses rapidly through a series of discrete brightness levels.
  • pPocket ⁇ Color With each iteration through ST -- DWELL, pPocket ⁇ Color in incremented. Assuming that the operator maintains the cursor hotspot on selectable region (0104/0106), pPocket ⁇ Color will eventually equal pPocket ⁇ Ceiling, a variable set at initialization time and not changed thereafter.
  • CreateEvent generates the EV -- CROSS -- OUT for the state machine associated with selectable region (0104/0106), driving the state machine to aPocketFsm[ST -- CREST -- TIDE][EV -- CROSS -- OUT], which equals ST -- SELECT -- AND -- OUT.
  • the state processing for ST -- SELECT -- AND -- OUT sends EV -- RESET and then EV -- MOVEMENT to all state machines, driving each of them from their current state to ST -- RESET and then to ST -- EBB -- TIDE.
  • the prototype continues to sample cursor location at 54 millisecond intervals, determine the appropriate event for each state machine and send that event each state machine, causing state transitions in each state machine according to the state table shown in FIGS. 29 and 30. Assuming that the operator next selects selectable region (0108/0110), the state processing in ST -- SELECTED displays the selection, the letter "o", in the circumscribed region (0150), and associates the menu options shown in FIG. 17 with their respective selectable regions. The display shown in FIG. 17 appears on the display (2112) of the computer system.(2116).
  • FIG. 31 20 selectable regions, e.g. (1704), (1708) and (1714), are depicted on display (2112). Each of the selectable regions is located on the display (2112) adjacent an edge of the display, and the selectable regions together circumscribe a region (1702) on the display.
  • the top and left edges of the Windows® cursor clipping rectangle (1750) lie on the top and left edges, respectively, of the display (2112).
  • the bottom and right edges of the Windows® cursor clipping rectangle (1750) lie on the display parallel to and slightly indented from the bottom and right edges, respectively, of the display.
  • a confining polygon is delimited on the display.
  • the boundary of the confining polygon follows the top edge of the Windows® cursor clipping rectangle to the left until it reaches region (1732), where the boundary follows the side of region (1732) down, to the left, and back up to the top edge of the Windows® cursor clipping rectangle.
  • the boundary continues left along the top edge of the Windows® cursor clipping rectangle until it reaches region (1734), where the boundary follows the side of region (1734) down, to the left, and back up to the top edge of the Windows® cursor clipping rectangle.
  • the boundary continues left along the top edge of the Windows® cursor clipping rectangle to the upper left corner and then turns down along the left edge of the Windows® cursor clipping rectangle until it reaches region (1736), where the boundary follows the side of region (1736) to the right, down, and to the left to the left edge of the Windows® cursor clipping rectangle.
  • the boundary of the confining polygon continues in this fashion around to the upper right corner of the Windows® cursor clipping rectangle.
  • the confining polygon thus includes all the area of the Windows® cursor clipping rectangle except for the regions (1732), (1734), (1736), (1738), (1740), (1742), (1744), and (1746).
  • An operator controlling a pointer indicating successive locations with respect to the display and attempting to select a target selectable region may overshoot the target so that some of the successive locations lie outside the Windows® cursor clipping rectangle.
  • the cursor (1724) is confined to the confining polygon.
  • the preferred embodiment is responsive to an intersection of the cursor hotspot and any one selectable region so that an overshot selectable region may be selected by click or by dwell without moving the location presently indicated by the pointer to a location in the overshot selectable region.
  • the preferred embodiment of the Confinement aspect of the invention allows an operator with impaired ability to stop motion to maintain the cursor more easily on a selectable region, and so select the intended selectable region, than do conventional user interface systems.
  • a selectable region having a single side abutting a confining polygon prevents cursor movement only beyond the abutting side.
  • NMD operators who drift may drift in more than one direction. Assume that a certain NMD operator tends to drift both up and to the left and that he is attempting to select selectable region (1704). If he moves the cursor into that selectable region his upward drift will be confined; the drift will not move the cursor beyond the confining polygon. However, the cursor will move to the left, since, in the preferred embodiment of the Confinement aspect of the invention, movement in this direction is not affected by the confining polygon, and consequently the cursor may move into selectable region (1708), the selectable region to the left of selectable region (1704).
  • NMD operators having this type of drift may be assisted in selecting by confining corners. For example, such an operator, attempting to select selectable region (1704) could move the cursor to location (1706) in selectable region (1704). As the operator drifts to the left, he can compensate by moving the pointer to the right. Assuming the operator lacks fine motor control, he may overcompensate and indicate a location to the right of the Windows(cursor clipping rectangle (1750). However, since the cursor is confined to the confining polygon, the cursor remains in the intended selectable region.
  • Confining corners facilitate the selection process for some NMD operators.
  • the preferred embodiment of the Confinement aspect of the invention creates a corner or virtual corner in each selectable region.
  • a virtual corner is a corner of a selectable region formed by the intersection of two sides of a selectable region both of which abut a confining polygon.
  • corner (1710) in selectable region (1708) abuts the confining polygon both along the top edge of the Windows® cursor clipping rectangle and along the right side of region (1732). If an NMD operator drifts from selectable region (1708) to the left into region (1732), the cursor remains in selectable region (1708). Thus, drift to the left does not move the cursor out of selectable region (1732).
  • An operator trying to select selectable region (1708) may overcompensate for drift to the right by moving the pointer to indicate a location in region (1732).
  • FIG. 32 illustrates a display in accordance with an alternative embodiment of the Confinement aspect of the invention.
  • FIG. 32 depicts 16 selectable regions, e.g. selectable region (0602), on a display (2112), the selectable regions together at least partially circumscribing region (0660) on the display. Circumscribed region (0660) intersects four selectable regions, (0604), (0608), (0610), and (0612).
  • Selectable region (0608) includes virtual corner (0652).
  • the confining polygon includes all the area of the Windows® cursor clipping rectangle except for regions (0632), (0634), (0636), (0638), (0640), (0642), (0644), and (0646).
  • FIG. 33 illustrates a display and structures in accordance with an another embodiment of the Confinement aspect of the invention.
  • FIG. 33 depicts 20 selectable regions, e.g. selectable region (0918), on a display (2112), the selectable regions together at least partially circumscribing region (0902) on the display.
  • the confining polygon includes all the area of the Windows® cursor clipping rectangle except for regions (0906), (0908), (0910), and (0904).
  • the Confinement aspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention.
  • the modifications required are: (1) define 20 state machines and 20 respectively associated selectable regions located as shown in FIG.
  • CreateEvent determines whether the current hotspot cursor location intersects the confining polygon, and, if not, set the Windows® hotspot cursor location to the previous hotspot cursor location and transfers control to the code at the beginning of the CreateEvent procedure which gets the current cursor hotspot location from Windows®. If the current hotspot cursor location intersects the confining polygon, CreateEvent takes the same action as in the access program (1206).
  • the preferred embodiment of the Dwell aspect of the invention will now be described in detail from a functional and implementation perspective.
  • the prototype implements dynamic dwell.
  • the effect of dwelling on a selectable region and its implementation have been described in the detailed description of the Perimeter Menu aspect of the invention.
  • Now the effect and implementation of moving the cursor hotspot off a selectable region will be described.
  • Stepping through the pseudo-code for ST -- DECAY the ST -- DECAY state sets pPocket ⁇ State to the value stored in pPocket ⁇ PreviousState, decrements pPocket ⁇ Color by pPocket ⁇ Decrement, but not below the value of pPocket ⁇ InitialColor, sets pPocket ⁇ fPaint to TRUE, sets fInternalEvent to TRUE, and, in this case, sets Event to EV -- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs.
  • the new state is found at aPocketFsm[ST -- LOW -- TIDE][EV -- NULL], which equals ST -- LOW -- TIDE.
  • ST -- LOW -- TIDE This path is unlike an ordinary state transition because the starting state is set by ST -- DECAY. All transitions from ST -- DECAY share this distinction.
  • the state machine executes the code for the new state, ST -- LOW -- TIDE, which is simply a break statement.
  • the procedure PocketFsm determines that fPaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM -- PAINT message to the access program (1206).
  • the access program (1204) checks the value of fPaint for each state machine, and if TRUE, sets fpaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine. After redrawing, control returns to Windows® (1204).
  • the procedure Create Event will determine that not EV -- MOVEMENT, but EV -- DECAY, should be sent to the state machine associated with selectable region (0104/0106).
  • EV -- MOVEMENT drives the state machine associated with selectable region (0104/0106) to ST -- DECAY.
  • ST -- DECAY The same state processing as described above for ST -- DECAY takes place, including the transition back to ST -- LOW -- TIDE.
  • the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST -- LOW -- TIDE to ST -- DECAY to ST -- LOW -- TIDE, driven by the cursor polling timer.
  • visible subregion (0106) is darkened a bit.
  • the polling timer interval is short enough and the decrement to pPocket ⁇ Color is small enough that visible subregion (0106) appears to gradually darken although in fact it progresses rapidly through a series of discrete brightness levels.
  • pPocket ⁇ Color With each iteration through ST -- DECAY, pPocket ⁇ Color in decremented until pPocket ⁇ Color reaches pPocket ⁇ InitialColor.
  • the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV -- DECAY, continuing the repetitively cycling through from ST -- LOW -- TIDE to ST -- DECAY to ST -- LOW -- TIDE, driven by the cursor polling timer. Driven by the same polling timer, the procedure CreateEvent determines that the state machine associated with selectable region (0132/0134) should receive EV -- DWELL, driving this state machine through the state transitions described in the detailed description of the Perimeter Menu aspect of the invention.
  • the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV -- DWELL, driving this state machine through the state transitions described in the detailed description of the Perimeter Menu aspect of the invention.
  • the selectable regions are shown on a display (2112).
  • the selectable regions may appear on a static display, or they may be projected on a surface.
  • the brightness of a visible subregion at any time indicates the progress of the selection of the selectable region including the visible subregion.
  • a brightness close to the initial brightness indicates that a relatively long period of dwelling on this selectable region is required for selection.
  • a brightness close to the brightness just prior to selection indicates that relatively short period of dwelling on this selectable region is required for selection.
  • means for indicating an intersection of the location indicated by the movement related signal and a selectable region, or the duration of a period of such an intersection includes, but is not limited to, a change in cursor appearance or location, a change in location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font or fill of an object on the surface, a display of or change in a graphic on the surface or the removal of a graphic from the surface, a generation of a sound or a change in the pitch or volume of an extant sound, a change in the temperature or surface of a contact area, the pressure exerted by a contact area, or frequency of contact by a contact area, or other suitable means. Any of these indications may be continuous or frequent.
  • dwell is implemented in the prototype using a data value, specifically pPocket ⁇ Color, it may be implemented using a signal, for example, voltage or current, varying in response to the intersection and subsequent non-intersection of a location indicated by a pointer and a selectable region.
  • a selectable region may include a detector and coupled electronics or electrical circuitry operative to increase the voltage level of a capacitor. Once elevated, the voltage level may decrease over time. Upon reaching a predetermined threshold, the voltage level may trigger selection.
  • the prototype allows an operator to make selections by dwell more efficiently than in conventional systems.
  • the brightness of a visible subregion indicates the dwell time required for selection.
  • a practiced operator may accurately estimate when he may plan his next pointer movement, when he may begin moving the pointer and may determine when a bit more exertion will select a selectable region and when it will not.
  • a disabled operator who is fatigued by computer access and can maintain a pointer in a steady position for only brief periods, may optimize his energy expenditure, for example, exerting himself to maintain the cursor on a certain selectable region only when doing so will quickly select the selectable region.
  • the prototype may increase the independence of a disable individual by allowing him to control devices such as a TV, thermostat and other household appliances.
  • the Perimeter Menu aspect of the invention may be implemented on a general purpose computer system. If the general purpose computer system is coupled to a devices capable of executing commands and the menu hierarchy allows the selection of commands, a disabled operator may select and issue commands to control these devices.
  • FIG. 34 illustrates a display in accordance with of an embodiment of the Dwell aspect of the invention having 20 selectable regions (3101), (3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119), (3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143), and (3145) circumscribing a surface (3151) having an indicating region (3147) thereon.
  • FIG. 35 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 20 selectable regions circumscribing a surface, each selectable region associated respectively with an indicating region adjacent its associated selectable region.
  • Selectable region (3101) is associated with indicating region (3201), selectable region (3103) with indicating region (3203), etc.
  • FIG. 36 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 20 detectors circumscribing an aperture (3350), each detector associated respectively with an indicator intersecting its associated detector, e.g. detector (3301) is associated with indicator (3303).
  • FIG. 37 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 24 selectable regions arranged in a grid of four rows and six columns, each selectable region associated respectively with an indicating region intersecting its associated selectable region.
  • selectable region (3402) is associated with indicating region (3401).
  • FIG. 38 illustrates a display in accordance with of an embodiment of the Dwell aspect of the invention having a plurality of detectors, (2501), (2503), (2505), etc., arranged in a grid.
  • Each detector is associated respectively with an indicator intersecting its associated detector., e.g. detector (2501) is associated with indicator (2502).
  • Each detector is also associated respectively with an order entry item which may be selected by dwell.
  • detector (2501) is associated with a hamburger.
  • Pointer (2512) (not drawn to scale) emits energy detectable by the detectors.
  • Pointer (2512) may be housed in stationary housing (2514) (not drawn to scale).
  • Means for determining the difference between two data items or signals and means for totaling two or more data items or signals may each include a processing unit programmed to calculate this difference or total.
  • the difference or total may be determined by electronic, mechanical, optical, or other suitable means.
  • the operator points pointer (2512) at an order entry item.
  • the item brightens responsive to the signals falling on the associated detector, indicating to the operator which order entry item he is dwelling on and his dwell time on that order entry item. When his dwell time equals or exceeds the selection threshold, the order entry item is selected.
  • the operation of such a system is intuitive and may be learned or relearned by pointing and dwelling. No surface is required, unlike a standard mouse.
  • an operator seated in the driver's seat of a vehicle is probably right-handed and may be making selections with his left hand, the hand closest to the window in a left-hand drive vehicle. Pointing and maintaining a pointer in a steady position requires less coordination than pointing and clicking.
  • the Path Directness aspect of the invention includes several aspects, hereinafter “subaspects”, called Facilitated Dwell, Direction and Intersection, Direction, Appraisal and Drift Attenuation.
  • Subaspects called Facilitated Dwell, Direction and Intersection, Direction, Appraisal and Drift Attenuation.
  • the preferred embodiment and certain alternative embodiments of each of these subaspects will now be described.
  • the duration of the dwell period required for selection varies with the directness of the cursor's path to that selectable region.
  • selection threshold period the duration of the dwell period required for selection
  • the duration of the dwell period required for selection varies with the directness of the cursor's path to that selectable region.
  • Each selectable region is associated respectively with a selection threshold period.
  • Each cursor location in a cursor path may slightly decrease one or more the selection threshold periods, except for the selection threshold period associated with the previously selected selectable region, selectable region (2318) in this example.
  • the effect of a cursor location depends, in the preferred embodiment, on whether that cursor location intersects one of the corridors. If it does, as for example location (2324) intersects corridor (2308), then the selection threshold period associated with the selectable region to which the intersected corridor leads, (2348) in this example, is decreased, preferably to a limit of approximately 20% of the initial value of the selection threshold period so that some period of intersection of the cursor and the intended selectable region is still required for selection.
  • a changed selection threshold period is preferably indicated by a change in the brightness of the selectable region associated with the changed selection threshold period.
  • the selectable region associated with the corridor brightens, indicating both the target selectable region the system believes the cursor is headed toward and the changed selection threshold period.
  • the selectable region associated with the previously intersected corridor darkens in accord with the dynamic dwell aspect of the invention, indicating both that the system no longer believes the cursor is headed toward that selectable region and the changed selection threshold period. Reducing the selection threshold period facilitates selection of dwell-selectable regions without unduly increasing the likelihood of erroneous selections, since cursor locations within a corridor evidence the operator's intention to select the selectable region associated with the corridor.
  • corridors are hidden from view, but they may be may be shown on the display or shown only at certain times or under certain conditions.
  • Corridors may have fixed boundaries, depending on which selectable region has been selected, or their boundaries may be determined when a starting location, for example, cursor location (2316) in FIG. 39, is known.
  • Corridor shape, size, number and position about the associated selectable region may vary, as illustrated by the alterative embodiments shown in FIGS. 40 and 41.
  • a cursor location intersecting two or more corridors may be defined to be in a cursor path toward zero, one or more selectable regions associated with the intersected corridors.
  • intersection of a cursor location and a corridor is but one means of identifying which one of a plurality of selectable regions is most nearly along a cursor path.
  • a cursor path may be indicated by an intersection of a cursor location and a predetermined region, e.g. a corridor, by a cursor location and a movement related signal from which may be derived a second location, or by two or more successive cursor locations.
  • successive locations include a plurality of locations distributed in time. Successive location may be, but need not be, consecutive.
  • an intention to select a particular selectable region may be inferred, for example, by extrapolation, and the selection thresholds associated with either or both the intended or unintended selectable regions modified accordingly.
  • successive cursor locations are periodically stored in a ring buffer and the magnitude of the angle between two line segments, the first between the oldest cursor location in the ring buffer and a predetermined point in the selectable region, and the second between the oldest cursor location in the ring buffer and the current cursor location, is determined.
  • the selectable region associated with the smallest of these angles may be considered to be the selectable region most nearly along the cursor path indicated by the first line segment.
  • An alternative means of identifying which one of a plurality of selectable region is most nearly along a cursor path is to determine the ratio of the number cursor locations indicating a selectable region to the total cursor locations in the cursor path.
  • FIG. 42 illustrates still another alternative means for identifying which one of a plurality of selectable regions is most nearly along a cursor path.
  • FIG. 42 shows line (2410) from starting cursor location (2406) to ending cursor location (2408) just within selectable region (2404).
  • Line (2410) is the most direct path between these two points.
  • the actual path traveled by the cursor between these two points is path (2412).
  • the ending cursor location (2408) is known before the selection threshold period associated with selectable region (2404) is modified. Identification may be made by measuring or approximating the area within region (2414) bounded by line (2410) and cursor path (2412). The smaller the area, the more direct the cursor path.
  • identification may be made by storing a sampling among successive cursor locations along a cursor path and, in response to the cursor intersecting a selectable region, the selectable region most nearly along the cursor path may be identified at one or more sampled points along the cursor path.
  • the identification may be based upon a plurality of cursor locations or upon a single cursor location and a movement related signal.
  • identification may be made by measuring and comparing the number of times a cursor path diverges from a predetermined path toward the intersected selectable region and/or the degree of divergence of a cursor path from a predetermined path toward the intersected selectable region.
  • selectable regions (3101), (3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119), (3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143) and (3145) circumscribe area (3151).
  • Area (3151) and optionally the selectable regions include detectors for sensing radiant energy emitted from a pointer (2202) coupled to a body member of the operator.
  • a computer coupled to the detectors determines which selectable region is most nearly along the path indicated by the body member of the operator.
  • the embodiment may facilitate the selection of one of the selectable regions by reducing a selection threshold, may select a selectable region upon intersection of the point indicated by the pointer (2202) and a selectable region, or may select a particular selectable region in advance of intersection of the point indicated by the pointer (2202) and the particular selectable region.
  • a move direction of a body member of an operator may be determined in any way that a cursor path may be determined, including sampling among data indicative of position of the body member.
  • data indicative of body member positions may serve the same function as cursor locations in indicating a path toward a selectable region.
  • Position indicating means used, for example, in indicating a position of the body member, includes each of the means for indicating that a selection event has occurred.
  • an embodiment may include a plurality of selection thresholds, each associated respectively with a selectable region. One or more of the selection thresholds may be increased when the direction of cursor movement does not indicate a path toward the associated selectable region.
  • Means for indicating which one of the plurality of selectable regions is most nearly along the cursor path includes each of the means for indicating an intersection of the location indicated by the movement related signal and a selectable region.
  • the selection threshold period is completely satisfied in response to a cursor path to a particular selectable region, so that when the cursor intersects the particular selectable region, that region is selected.
  • the selection threshold period is completely satisfied in response to a measure of directness of a cursor path to a particular selectable region equalling or exceeding a predetermined measure of directness. In circumstances where the measured directness is less than the predetermined measure, a dwell period is required for selection of the particular selectable region.
  • a selectable region is selected in response to a cursor path to that selectable region, in advance of an intersection of the cursor and that selectable region.
  • the selectable region is selected in response to a measure of directness of a cursor path to a particular selectable region equalling or exceeding a predetermined measure of directness. In circumstances where the measured directness is less than the predetermined measure, a dwell period is required for selection of the particular selectable region.
  • the directness of a cursor path to a selectable region is measured.
  • the means for measuring the directness of a cursor path includes each of the means for identifying which one of a plurality of selectable regions is most nearly along a cursor path.
  • the particular means for identifying which one of a plurality of selectable regions is most nearly along a cursor path which best correlates with an operator's intended target selectable region may be identified.
  • the Facilitated Dwell, Direction and Intersection, and Direction subaspects of the Path Directness aspect of the invention utilize that capability for computer access. Specifically, the ability of an operator to move a cursor in a direct path toward a selectable region is used to facilitate selection of that selectable region. The selectable region is selected more quickly than in conventional systems utilizing selection by dwell, increasing operator productivity.
  • a cursor path toward a selectable region is often unambiguous, since usually there is only one selectable region along a cursor path, and a large rectangular area on the display is available for the output of an application program and is not obstructed by the menu.
  • the operator may receive an indication of which selectable region the system believes the operator is moving the cursor toward. The operator may adjust the cursor path in response to this feedback and thus move the pointer more accurately.
  • the operator may receive an indication of the dwell time required to select the selectable region most nearly along the cursor path as the required dwell time changes in response to the cursor path.
  • FIG. 43 depicts the upper right corner of a display (2112) having two selectable regions (0434) and (0430) thereon.
  • a movement related signal indicates the path shown from point (0402) to point (0404) ("first segment") at a relatively high velocity and from point (0404) to point (0408) ("second segment") at a relatively low velocity.
  • the path of the first segment is relatively direct, the path of the second segment, relatively meandering.
  • the cursor preferably tracks the exact path indicated by the movement related signal.
  • the movement of the cursor is attenuated, preferably so that the cursor does not leave selectable region (0434) until it is selected.
  • drift that is, unintentional movement, indicated by the movement related signal following intentional movement is distinguished from the intentional movement, and cursor movement responsive to the drift is attenuated relative to cursor movement responsive to intentional movement.
  • unintentional movements of NMD operators are filtered so that the cursor is displayed closer to the location intended by the operator and drifting of the cursor into a nearby, but unintended, selectable region, is avoided, resulting in fewer errors due to unintended selections.
  • selectable region (2346) in accord with the preferred embodiment of the Facilitated Dwell subaspect of the invention will now be described with reference to FIG. 39, assuming that the prototype has been modified as described, that the operator has just selected selectable region (2318), that all state machines are in state ST -- RESET, and that the operator moves the cursor hotspot from a location inside selectable region (2318) to location (2316), a location just outside selectable region (2318).
  • procedure PocketFsm is called with the indicator for the state machine associated with selectable region (2318) and with event EV -- CROSS -- OUT.
  • Event EV -- CROSS -- OUT drives this state machine from its current state, ST -- RESET, to ST -- EBB -- TIDE.
  • the pseudo-code for ST -- EBB -- TIDE in procedure PocketFsm is a break statement, indicating that no state specific action is taken at this time, other than the transition to ST -- EBB -- TIDE. Control returns to Windows® (1204).
  • All state machines receive EV -- MOVEMENT and are driven from ST -- EBB -- TIDE to ST -- DECAY to ST -- EBB -- TIDE.
  • the ST -- CORRIDOR state sets pPocket ⁇ State to the value stored in pPocket ⁇ PreviousState, increments pPocket ⁇ Color by pPocket ⁇ CorridorIncrement, but not above pPocket ⁇ InitialColor plus 80% of the difference between pPocket ⁇ CrestTide and pPocket ⁇ InitialColor, sets pPocket ⁇ fPaint to TRUE, sets fInternalEvent to TRUE, and sets Event to EV -- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs.
  • the new state is found at aPocketFsm[ST -- EBB -- TIDE][EV -- NULL], which equals ST -- EBB -- TIDE.
  • ST -- EBB -- TIDE This path is unlike an ordinary state transition because the starting state is set by ST -- CORRIDOR. All transitions from ST -- CORRIDOR share this distinction.
  • the state machine executes the code for the new state, ST -- EBB -- TIDE, which is simply a break statement.
  • the procedure PocketFsm determines that fpaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM -- PAINT message to the access program (1206).
  • the access program (1204) checks the value of fpaint for each state machine, and if TRUE, sets fPaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine.
  • the incremented value of pPocket ⁇ Color results in a slight brightening of selectable region (2348) and reduces the difference between pPocket ⁇ Color and pPocket ⁇ CrestTide, corresponding to the dwell period required to select the associated selectable region. After redrawing, control returns to Windows® (1204).
  • This scenario is repeated at 54 millisecond intervals while the cursor hotspot travels along path (2322) to location (2326), a location intersecting corridor (2308).
  • the procedure CreateEvent determines that all state machines should receive EV -- MOVEMENT, driving each of them from their current state to ST -- DECAY and back to their current state.
  • ST -- DECAY state processing darkens the selectable region associated with the state machine, but not below a predetermined brightness represented by the variable InitialColor.
  • selectable region (2348) darkens when the cursor hotspot no longer intersects corridor (2308).
  • selectable region (2346) From location (2328) to location (2330), the procedure CreateEvent determines that the state machine associated with selectable region (2346) should received EV -- CORRIDOR and all other state machines EV -- MOVEMENT. Consequently, selectable region (2346) gradually brightens up to a ceiling represented by pPocket ⁇ InitialColor plus 80% of the difference between pPocket ⁇ CrestTide and pPocket ⁇ InitialColor. The duration of dwell time required for selection of selectable region (2346) is thus reduced to approximately 20% of the dwell period required without Facilitated Dwell.
  • the Direction and Intersection subaspect of the invention is implemented by making the changes to the prototype described for the Facilitated Dwell subaspect, except that, in incrementing pPocket ⁇ Color in ST -- CORRIDOR, the upper limit for pPocket ⁇ Color in ST -- CORRIDOR state processing is pPocket ⁇ CrestTide minus pPocket ⁇ Increment. Assuming these changes, a selectable region whose associated Color variable is at this upper limit is selected during processing of the WM -- TIMER message immediately following the intersection of the cursor hotspot and the selectable region.
  • the Direction subaspect of the invention is implemented by making the changes to the prototype described for the Direction and Intersection subaspect of the invention, except that (1) the corridors are narrow, as illustrated in FIG. 40; and (2) the pseudo-code fir state processing in ST -- CORRIDOR is as follows:
  • State ST -- CORRIDOR may now generate the internal event EV -- STEP -- UP, as state ST -- DWELL does in the detailed description of the Perimeter Menu aspect of the invention.
  • a state machine in state ST -- EBB -- TIDE having a Color variable equal to or exceeding the CrestTide variable will transition via PocketFsm[ST -- EBB -- TIDE][EV -- STEP -- UP] to ST -- SELECTED, and perform the ST -- SELECTED state processing described in the detailed description of the Perimeter Menu aspect of the invention.
  • the Appraisal subaspect of the invention is implemented by making the changes to the prototype described for the Facilitated Dwell subaspect, except that (1) the, corridors are visible on the display (2112), (2) one of the selectable regions is designated to be the target selectable region and this is indicated to the operator, (3) cursor locations are stored in memory (2106), and (4) following an intersection of the cursor hotspot and the target or selection of a selectable region other than the target, path directness is measured in accordance with the stored cursor locations.
  • the Drift Attenuation aspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention, modified as described above in the description of the implementation of the preferred embodiment of the Facilitated Dwell subaspect of the invention and further modified as follows: (1) add two booleans to the data set associated with each state machine, one called fDirectPath, the other fAttenuateDrift, and initialize each of them in all state machines to FALSE; (2) in both ST -- DECAY and ST -- CORRIDOR, set both pPocket ⁇ fDirectPath and pPocket ⁇ fAttenuateDrift to FALSE; (3) append to ST -- ENTRY state processing corresponding to the following pseudo-code:
  • FIG. 44 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) to which is coupled a pointer (2202). Eight regions are delimited with respect to the display (2112) and together circumscribe region (2650) on the display (2112). Four of these regions, (2602), (2612), (2622), and (2632), are entirely on the display (2112).
  • Each of the other four regions respectively includes both a visible subregion ((2606), (2616), (2626), and (2636)) on the display (2112) and an invisible subregion ((2604), (2614), (2624), and (2634)) adjacent and outside the display (2112).
  • the pointer is indicating location (2656) on the display (2112) and that the operator moves the pointer so that the location indicated by the pointer first intersects one of the regions at location (2652) in region (2612).
  • FIG. 45 depicts a selectable region consisting of the union of invisible subregion (2714) and visible subregion (2716), hereinafter referred to as selectable region (2714/2716).
  • Selectable region (2714/2712) is associated with menu option "sumac".
  • the operator may select menu option "sumac” by dwelling on any part of the selectable region for the selection threshold period. Assuming, for example, that instead of moving the pointer from a position indicating location (2656) to location (2652), the operator instead moves the pointer so that the location indicated by the pointer first enters one of the regions at location (2654) in region (2614/2616). Upon this intersection, the display changes to that shown in FIG. 46.
  • FIG. 46 depicts a selectable region consisting of the union of invisible subregion (2814) and visible subregion (2816), hereinafter referred to as selectable region (2814/2816).
  • Selectable region (2814/2816) is associated with menu option "vort ⁇ space>x". In the preferred embodiment, the operator may select menu option "vort ⁇ space>x" by dwelling on any part of the selectable region for the selection threshold period.
  • each of the other seven regions shown in FIG. 44 is associated with a selectable region and each selectable region is associated with a menu option.
  • the menu options shown in FIG. 44 in addition to “sumac” and “vort ⁇ space>x" are “wizen”, “backspace”, “words”, “talk”, ldhbfk” and “ypgqj,”.
  • the operator may select the menu option associated with any one of the selectable regions by moving the location indicated by the pointer from circumscribed region (2650) into the region associated with the selectable region and then dwelling on the selectable region for the selection threshold period.
  • Each selectable region preferably includes all the area of its associated region.
  • An operator having impaired ability to maintain a body member in a steady position but who can control the point at which the location indicated by a body member enters a region may, in accord with the Intersection aspect of the invention, use his relatively unimpaired motor capability to selectively enlarge a selectable region or determine which of two or more selectable regions will occupy a predetermined area, thus making it easier for him to select.
  • the general purpose computer system of the preferred embodiment is coupled to a speech synthesizer and the menu options are letters or words, an operator with impaired speech may select or spell words and speak them.
  • FIGS. 47 and 48 illustrate the upper right corner of a display in accordance with an alternative embodiment of the Intersection aspect of the invention.
  • FIG. 47 depicts; the upper right corner of a display (2112) of a computer system (2116) having thereon two regions, (1404) and (1402), each associated respectively with a selectable region, each selectable region associated respectively with menu options "bdfhkl" and " ⁇ space>cmnrst".
  • the selectable region associated with region (1402) includes all of region (1402).
  • the selectable region associated with region (1404) includes all of region (1404) plus area (1406) between region (1404) and the right edge of the display (2112).
  • Dwelling in area (1406) operates to select the menu option " ⁇ space>cmnrst". If instead the location to first intersect the union of regions (1402) and (1404) intersects region (1404), the display changes to that shown in FIG. 48. Dwelling in area (1406) then operates to select the menu option "bdfhkl".
  • the Intersection aspect is preferably implemented by modifications to the access program (1206) described above in the detailed description of the Perimeter Menu aspect of the invention.
  • the modifications are: (1) add a row to the state table aPocketFsm at row index 18 for a new state, ST -- EXIT and initialize the value of aPocketFsm[ST -- EXIT][EV -- NULL] to 4, the value of ST -- LOW -- TIDE; (2) at initialization time, change aPocketFsm[ST -- LOW -- TIDE][EV -- CROSS -- OUT] to 18, the value of ST -- EXIT; (3) at initialization time for each state machine, (a) if the visible subregion associated with the state machine does not abut the edge of the display, set fInterior to TRUE, otherwise set fInterior to FALSE; (b) initialize iAdjacentPocket to the index of the state
  • iAdjacentPocket in the data set associated with the state machine associated with region (2614/2616) is initialized to 3 and iAdjacentPocket in the data set associated with the state machine associated with region (2612) is initialized to 2; (c) initialize the second array of points to define the boundaries of the region associated with the selectable region associated with each state machine, for example, region (2612), and initialize the first array of points to define the boundaries of the visible subregion of the selectable region associated with the state machine, for example, visible subregion (2712); (4) remove creation of Windows® regions corresponding to visible subregions from the access program (1206) initialization; (5) append to ST -- ENTRY state processing corresponding to the following pseudo-code:
  • fInternalEvent is set to TRUE, Event is set to EV -- NULL, the Windows® region corresponding to region (2612), ie. the Windows® region having the handle pPocket ⁇ hRegion, is deleted and a new Windows® region having the boundaries indicated in the first array of points, ie. visible subregion (2712) in FIG. 45, is created and the handle stored in pPocket ⁇ hRegion.
  • pPocket ⁇ fPaint is set to TRUE, EV -- RESET is sent to state machines associated with regions other than (2612) and (2614/2616).
  • pPocket-fPaint is TRUE so the client rectangle is invalidated, resulting in the redrawing of Windows® regions associated with state machines having fpaint equal to TRUE so that the display (2112) appears as shown in FIG. 45. Control returns to Windows® (1204).
  • the procedure CreateEvent sends the event EV -- CROSS -- OUT to the state machine associate with that selectable region, driving it to ST -- EXIT.
  • the state processing for ST -- EXIT causes the display (2112) to change to that shown in FIG. 44.
  • FIGS. 49, 50, and 51 depict the upper right corner of a display (2112) having two visible subregions on the display. In these Figures, no subregions outside the display (2112) are shown. Alignment is achieved in several steps and requires operator interaction with the apparatus.
  • FIG. 49 depicts the upper right corner of the display (2112), that an operator, fitted with a head pointer, desires to keep the cursor on the display directly in his line of sight and that the location indicated by the pointer is presently 15 degrees to the right of the location of the arrow cursor (1802).
  • the apparatus responds to this extended dwell period by changing the display to that shown in FIG. 49.
  • the arrow cursor is removed from the display and the lock icon (1902) is displayed in a predetermined location of the intersected selectable region ("lockspot") on the display.
  • the lock icon remains on the lockspot for a predetermined period ("the lock period”); it does not move responsive to the operator's head movement.
  • the lock icon is displayed, the operator turns his head, bringing his line of sight into alignment with the lockspot.
  • the apparatus changes the display to that shown in FIG. 51.
  • the lock icon is erased and the arrow cursor appears in the lockspot, which is where the operator is now looking.
  • the arrow cursor moves in response to the operator's head movement.
  • An operator who loses alignment between location indicated by his pointer and the cursor may thus initiate an alignment sequence, and then, by moving his head or other body member when the prototype indicates he should do so by displaying the lock icon, regain alignment. That the lock icon is displayed indicates to the operator that he can align his head or other body member. The position of the lock icon indicates to the operator the location on the display with which he should align his head or other body member. This is the location where the arrow cursor will appear at the expiration of the lock period.
  • the Alignment aspect will now be described in detail from an implementation perspective as implemented in the prototype.
  • the Alignment aspect of the invention is implemented an integral part of the state machine described in the detailed description of the Perimeter Menu aspect of the invention, using the same state machines and initialization, with the exception that aPocketFsm[ST -- CREST -- TIDE][EV -- CEILING] is changed from 9, the value of ST -- DWELL, to 14, the value of ST -- BEGIN -- LOCK.
  • the operator initiates the alignment process by moving the cursor so that the cursor hotspot intersects a selectable region.
  • FIG. 49 shows the arrow cursor (1802) intersecting selectable region (1834).
  • This operator action causes the state machine associated with the intersected selectable region to reach state ST -- CREST -- TIDE.
  • the procedure CreateEvent creates event EV -- CEILING, which drives the state machine from state ST -- CREST -- TIDE to state ST -- BEGIN -- LOCK.
  • the procedure CreateEvent may generate an event EV -- DECAY, EV -- DWELL, EV -- CROSS -- OUT, EV -- MOVEMENT or EV -- CEILING. Each of these events drives the state machine from state ST -- BEGIN -- LOCK to state ST -- LOCK.
  • the global Event is set to EV -- NULL and fInternalEvent is set to TRUE so that another state transition occurs immediately.
  • This transition drives the state machine from state ST -- LOCK to state ST -- END -- LOCK.
  • the lock icon is erased, Windows® (1204) is directed to display the arrow cursor (2002) at the lockspot, as shown in FIG. 51, the fpaint flag associated with the intersected selectable region is set to TRUE so that the selectable region will be drawn, restoring the background behind the erased lock icon.
  • CreateEvent generates EV -- DWELL, which drives the state machine to ST -- DISCARD.
  • ST -- DISCARD state processing sets State to PreviousState, returning the state machine to ST -- END -- LOCK and in effect, discarding the last event.
  • the procedure CreateEvent generates an event EV -- CROSS -- OUT which drives the state machine from state ST -- END -- LOCK to state ST -- SELECT -- AND -- OUT.
  • the access program (1206) performs the stale processing for state ST -- SELECT -- AND -- OUT and subsequent states as previously described in the description of the Perimeter Menu aspect of the invention.
  • the operator may initiate alignment by depressing any character key on the keyboard.
  • the access program (1206) removes the arrow cursor from the display and displays the lock icon at a predetermined location on the display.
  • the predetermined location is the center of the display (2112). After a predetermined period, two seconds in the prototype, the lock cursor icon is erased and the arrow cursor displayed at the predetermined location.
  • the cursor is automatically centered if the cursor hotspot does not move for two minutes. Lack of movement of the cursor hotspot is detected in the procedure CreateEvent, which generates the event EV -- IDLE -- TIMEOUT for all state machines.
  • the state processing of each state machine on receipt of EV -- IDLE -- TIMEOUT depends upon its current state. State machines in states ST -- INITIAL and ST -- RESET stay in those states. State machines in all other states in which an external event can be received are driven to ST -- IDLE. Referring to the pseudo-code in PocketFsm, ST -- IDLE state processing moves the cursor to the center of the display, sets fInternalEvent to TRUE and sets Event to EV -- NULL. PocketFsm[ST -- IDLE][NULL] equals 3, the value of ST -- RESET.
  • the access program (1206) performs the state processing for the ST -- RESET and subsequent states as described in the description of the Perimeter aspect of the invention.
  • the Length Order aspect of the invention will now be described in detail from a functional and implementation perspective using an example depicted in FIG. 52 and described in the detailed description of the Location Indication aspect of the invention.
  • the string "s" is passed from the access program (1206 in FIG. 16) via the Windows® Dynamic Data Exchange ("DDE") interface to a database program (1210 in FIG. 16), preferably the FoxPro for Windows® version 2.6 program, available from Microsoft Corporation, Redmond, Wash., USA.
  • the FoxPro program looks up the record having the index "s" in a database composed of 26 letters, one for each letter of the alphabet.
  • Each record includes one field for a letter of the alphabet and 12 fields containing the 12 most frequently used English words beginning with that letter.
  • the 12 words in the record are ordered primarily by length, determined by the number of letters in each word, and secondarily by alphabetic order.
  • the FoxPro program returns these 12 words to the access program (1206) and these are displayed on the display (2112) as named menu options.
  • the twelve named menu options, ordered as described and depicted in FIG. 52, are "so", “say”, “she”, “said”, “show”, “some”, “such”, “state”, “school”, “social”, “service”, and "student".
  • root word only the root form of inflected forms of words which can be created through available suffixes (root word) may be displayed as named menu options, so that the limited number of available menu options in combination with the apparatus' capability to add suffixes offers a large number of inflected forms.
  • Words which may be named menu options in the same menu may be ordered by any suitable method.
  • ordering is done by an ordering program operating on a corpus of text or speech including text or speech produced by individuals whose age, sex, geographic location and disability are the same as or similar to that of the operator.
  • the ordering program determines the frequency of use of root words in the corpus, selects the twelve most common root words, beginning with every possible combination of one, two and three letters, and stores them in three FoxPro databases for one, two and three letter word beginnings respectively, the words in each record ordered as described above. Ordering the words prior to a request minimizes the delay between the operator's selection of a letter or letters and the display of the named menu options.
  • the ordering program also creates a database of records for root words beginning with four or more letters.
  • Each record includes the words and its frequency of use in the corpus.
  • the access program (1206) requests via DDE that the FoxPro program (1210) look up words starting with four or more letters in the database for words beginning with four or more letters, select the 12 most frequently used words matching the selected letters, order them as described above, and return them to the access program (1206).
  • An operator searching named menu options for a desired menu option may start his search in the area on the display most likely to contain the desired menu option. Upon comparison of the length of the desired menu option to a displayed menu option, the operator may determine whether to continue his search from the displayed menu option toward the front of the list or toward the rear of the list, or to jump to another displayed menu option in the list Further, he may make this determination more quickly than if the displayed menu options were sorted conventionally.
  • the reduced menu option search time increases the operator's productivity with respect to conventional menu interfaces.
  • FIG. 52 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) having thereon eight selectable regions, each associated respectively with a menu option.
  • Each of the eight selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion (not shown) located outside the display (2112) and adjacent the visible subregion.
  • Selectable region (4301) is associated with menu option “so", selectable region (4303) with menu option “say”, selectable region (4305) with menu option “she”, selectable region (4307) with menu option “suffixes”, selectable region (4309) with menu option “words”, selectable region (4311) with menu option “some”, selectable region (4313) with menu option “show” and selectable region (4315) with menu option “said”. Also shown on the display (2112) are 12 indicating regions, each associated respectively with a menu option.
  • Indicating region (4351) is associated with menu option “so”, indicating region (4353) with menu option “say”, indicating region (4355) with menu option “she”, indicating region (4361) with menu option “some”, indicating region (4363) with menu option “show”, indicating region (4365) with menu option “said”, indicating region (4371) is associated with menu option “such", indicating region (4373) with menu option “state”, indicating region (4375) with menu option "school”, indicating region (4381) with menu option "student”, indicating region (4383) with menu option "service” and indicating region (4385) with menu option “social”.
  • Each indicating region is located on the display (2112) in the region (4350) circumscribed by the selectable regions.
  • the location of each of the indicating regions (4351), (4352), (4355), (4361), (4363) and (4365) indicates the location of each of the selectable regions associated with the menu option associated with the indicating region.
  • These selectable regions are (4301), (4303), (4305), (4311), (4313) and (4315).
  • selection of menu option "words” causes selectable region (4301) to be associated with menu option “such” instead of menu option “so", selectable region (4303) to be associated with menu option “state” instead of menu option “say”, selectable region (4305) to be associated with menu option "school” instead of menu option “she”, selectable region (4311) to be associated with menu option “student” instead of menu option “some”, selectable region (4313) to be associated with menu option "service” instead of menu option “show”, and selectable region (4315) to be associated with menu option “social” instead of menu option "said”.
  • the menu option newly associated with each selectable region is displayed on that selectable region (not shown).
  • indicating regions (4371), (4373), (4375), (4381), (4383) and (4385) each indicate the location of each of the selectable regions associated with the menu option associated with the indicating region.
  • the indicating regions and the menu options displayed thereon in FIG. 52 are disproportionately large relative to the rest of FIG. 52. They are approximately 1.5 times their proportionate size. They are represented as shown in FIG. 52 in compliance with Patent Cooperation Treaty Rules requiring a minimum size for letters in figures.
  • the displayed menu options displayed on the selectable regions cannot be seen in a glance by many operators.
  • the displayed menu options can be seen in a glance, facilitating searching of the menu by the operator for his intended menu option. Since the location of each selectable region is indicated by the location of the associated indicating region, the operator may point to the intended selectable region without searching it out or pausing to verify that the menu option associated with a selectable region is the menu option he desires. The frequent operator of such a menu interface may habituate the process of selecting an intended menu option so he can focus his attention on another task while selecting the option, for example, planning his next interaction with the menu interface.
  • FIG. 53 depicts a display and structures in accord with an another embodiment of the Location Indication aspect of the invention.
  • Selectable regions (4604), (4608), (4612), (4616), (4620), (4624), (4628) and (4632) are located adjacent the display (2112) and associated respectively with menu options "so", “say”, “she”, “suffixes", “said”, “show”, “some” and “words".
  • Each selectable region is unbounded on the side furthest from and parallel to the edge of the display.
  • Indicating regions (4351), (4353), (4355), (4357), (4367), (4361), (4363) and (4365) are associated respectively with menu options “so”, “say”, “she”, “suffixes”, “said”, “show”, “some” and “words”.
  • the location of an indicating region indicates the location of the selectable region associated with the menu option associated with the indicating region.
  • the large size of the selectable regions outside the display facilitate selection by individuals with impaired fine motor control while the indicating regions indicate the location of each associated selectable region.
  • FIG. 54 depicts a display (2112) having thereon eight selectable regions, each associated respectively with a menu option.
  • selectable region (3911) is associated with menu option “aeiou”, selectable region (3910) with menu option “gqjyvwxz”, selectable region (3908) with menu option “setup”, selectable region (3907) with menu option undo, selectable region (3905) with menu option “control”, selectable region (3904) with menu option "words", selectable region (3902) with menu option “nsrm ⁇ space>tcp” and selectable region (3901) with menu option "bdfhkl”.
  • Menu option “aeiou” is associated with a submenu which includes submenu options “a”, “e”, “i”, “o” and “u”, displayed on indicating regions (3930), (3928), (3926), (3924) and (3922) respectively.
  • the background pattern of indicating region (3930) matches the background pattern of selectable region (3908), indicating that submenu option “a” will be associated with selectable region (3908) following selection of menu option "aeiou”.
  • the background pattern of indicating region (3928) matches the background pattern of selectable region (3910), indicating that submenu option "e” will be associated with selectable region (3910) following selection of menu option "aeiou”.
  • the indication may be made by the size, shape, hue, brightness, contrast, dithering, fill, blinking, hatching or pattern of the indicating region or any part thereof, including either of the foreground and background of the indicating region.
  • selectable region (3911) When the operator selects selectable region (3911), the display (2112) changes to that shown in FIG. 55. Now, selectable region (3910) is associated with submenu option “e”, selectable region (3908) with submenu option “a”, selectable region (3904) with submenu option “u”, selectable region (3902) with submenu option “o” and selectable region (3901) with submenu option "i”.
  • the appearance of an indicating region indicates the location of the selectable regions associated with the submenu option associated with the indicating region.
  • FIGS. 54 and 55 illustrates how the Location Indication aspect of the invention speeds selection of a submenu option of a menu hierarchy.
  • the operator by observing the background pattern of the submenu option within the displayed menu option, may determine which selectable region he should next dwell on. The operator may make this determination prior to selection of the menu option and need not wait for the submenu options to be displayed on their associated selectable regions.
  • the operator may select or spell out words more quickly than with conventional automated menu hierarchy. Assuming the display (2112) is part of computer system, these words may be input to an application program, and, if the computer system is coupled to a speech synthesizer, these words may be spoken.
  • FIG. 56 depicts another apparatus in accord with the Location Indication aspect of the invention. An example of the operation of this apparatus will now be described.
  • FIG. 56 depicts display area (4770) adjacent to which are located selectable regions (4701), (4703), (4705), (4707), (4709), (4711), (4713) and (4715).
  • selectable regions 4701, (4703), (4705), (4707), (4709), (4711), (4713) and (4715).
  • the plurality of selectable regions is associated with a different set of eight menu options such that, for a one second period, each selectable region is associated respectively with one menu option of the associated set of menu options.
  • On the display area are 24 indicators, each associated respectively with one of the menu options.
  • Each indicator indicates when a selectable region is associated with the menu option associated with the indicator.
  • each of the indicators (4721), (4723), (4725), (4727), (4729), (4731), (4733) and (4735) is associated respectively with a menu option which may be in turn associated respectively, for example, during a first one second period, with one of the selectable regions.
  • the operator may select a desired one of the 24 menu options by selecting the associated selectable region during the period when the desired menu option is associated with the associated selectable region.
  • the apparatus shown in FIG. 56 may require two successive selection events, the first selection event to select a set of eight menu options or to select a set of three menu options, the set being associated with one selectable region, and the second selection event to select one of the selected set.
  • the first selection event may select the set of menu options associated with indicators (4773), (4753) and (4733), and the second selection event may select one menu option from this set.
  • FIG. 57 Still another apparatus in accord with the Location Indication aspect of the invention is shown in FIG. 57.
  • FIG. 57 Still another apparatus in accord with the Location Indication aspect of the invention is shown in FIG. 57.
  • 57 depicts display area (4770) outside of which are located selectable regions (5001), (5003), (5005), (5007), (5009), (5011), (5013), (5015), (5021), (5023), (5025), (5027), (5029), (5031), (5033) and (5035).
  • the location of each selectable region is indicated by an indicating region on the display area, (5041), (5043), (5045), (5047), (5049), (5051), (5053), (5055), (5061), (5063), (5065), (5067), (5069), (5071), (5073) and (5075), respectively.
  • the Location Indication aspect of the invention is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention.
  • the modifications required for the Location Indication aspect of the invention are: (1) at initialization time: (a) create twelve child edit windows, each corresponding to one of the indicating regions shown in FIG.
  • each of the class "edit” each having the style (WS -- CHILD
  • the six menu options displayed in indicating regions (4371), (4374), (4375), (4381), (4383) and (4385) are present in the aLabel array when FIG. 52 is shown, but are not associated with selectable regions until the operator selects menu option "words".
  • FIG. 58 shows a display (2112) of a computer system (2116) having thereon six regions or sound indicators, (5801), (5803), (5805), (5807), (5809), and (5811), each associated with a menu option, "ypgqj,”, “ldhbfk”, “wizen”, “words”, “vort x", and “sumac”, respectively.
  • Each menu option is displayed on its associated sound indicator.
  • Each sound indicator has a distinct hue. For example, sound indicator (5801) may be green, (5803) white, (5805) blue, (5807) red, (5809) orange and (5811) grey.
  • Each of these sound indicators indicates a sound the operator is able to consistently produce, for example, the vowel sound e as it sounds in green, i as it sounds in white, u as it sounds in blue, e as it sounds in red, o as it sounds in orange and e it sounds in grey.
  • the menu option associated with the orange sound indicator (5809) is selected and the display is changed to that shown in FIG. 59.
  • submenu options space, "x”, "r”, “t”, “v” and “o” are now associated with each of the sound indicators (5901), (5903), (5905), (5907), (5909), and (5911) respectively.
  • submenu option space is selected and a space is input to the application program (6107) whose output appears in region (5850).
  • FIG. 60 illustrates a display and structures in accordance with an alternative embodiment of the Sound Match aspect of the invention.
  • FIG. 60 depicts a display (2112) of a computer system (2116). Adjacent the display (2112) is a static display (6252) having thereon eight sound indicators (6204), (6208), (6212), (6216), (6220), (6224), (6228), and (6232). Each of the eight sound indicators is a symbol in a phonetic character set graphically representing a sound.
  • the sound indicators may include a picture of an object, for example, a tree, a house, a boy, or a map of a country, or may include a shape, for example, a square, a circle, or a triangle, or may include a hue indicator, a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch indicator, or a change in volume indicator.
  • Eight menu options “vort ⁇ space>x”, “sumac”, “wizen”, undo, "words", “talk”, “ldhbfk” and “ypgqj,”, are displayed on the display (2112), each adjacent and associated respectively with one of the sound indicators, the eight menu options together circumscribing region (6250) on the display. The operator may select any one of the menu options by producing the sound indicated by the sound indicator associated with the menu option. In response to a menu selection, submenu options of the selected menu option may be displayed and associated respectively with a sound indicator.
  • An operator with impaired speech but who is able to consistently produce a relatively small number of sounds distinguishable by conventional speech recognition means may, in accord with the Sound Match aspect of the invention, select from among the small number of menu options by using the sounds he can produce.
  • the menu option may represent inputs to an application program, and, if the computer system is coupled to a speech synthesizer, the menu option may represent words to be spoken.
  • FIG. 61 illustrates a display and structures in accordance with an alternative embodiment of the Sound Match aspect of the invention.
  • FIG. 61 depicts a display (2112) of a computer system (2116) having thereon six square sound indicators, (6401), (6403), (6405), (6411), (6413), and (6415) arranged in two columns of three sound indicators. Each sound indicator abuts a sound indicator in the other column, the sound indicator located above it, if any, and the sound indicator located below it, if any.
  • each of the sound indicators shown in FIG. 61 is associated respectively with a menu option, each of the plurality of menu options having a common characteristic.
  • the menu options are homophones and the common characteristic is a phonetic unit and intonation but may be a phonetic unit alone, an intonation alone, a stroke used to draw an ideograph, a number of horizontal strokes, a number of vertical strokes, a number of total strokes, a stroke order, a radical, a part of speech, an ideograph, a kana, a diacritic, a classification of a part of an ideograph or other characteristic of a class of ideographs.
  • Each of the sound indicators has a distinct hue. The operator may select any one of the displayed menu options, according to the Sound Match aspect of the invention, by speaking the sound associated with the sound indicator associated with the desired menu option.
  • the Sound Match aspect of the invention thus allows an operator to select from an option from a menu, using speech recognition means, whether or not the menu options are homophones.
  • the operator does not need to use his hands to make this selection and so may keep both his hands on the home row of the keyboard, in preparation for entering the next common characteristic, or, if specifying the common characteristic by voice, may select a menu option without interrupting the manual activity he's engaged in.
  • FIG. 62 depicts a speech recognition system (6001) including a computer system (2116) and keyboard (2210), as earlier described, a sound board (6007) and a microphone (6009).
  • the conventional computer system (2116) preferably includes an 80486 CPU running at 33 MHz or faster, and is provided with Dragon Dictate Power Edition software, available from Dragon Systems, Inc., Newton, Mass., USA.
  • the sound board is the Audio Capture Playback Adaptor and the microphone is the Shur Headset microphone, both available from Dragon Systems, Inc.
  • FIG. 62 depicts a speech recognition system (6001) including a computer system (2116) and keyboard (2210), as earlier described, a sound board (6007) and a microphone (6009).
  • the conventional computer system (2116) preferably includes an 80486 CPU running at 33 MHz or faster, and is provided with Dragon Dictate Power Edition software, available from Dragon Systems, Inc., Newton, Mass., USA.
  • the sound board is the Audio Capture Playback Adaptor and the microphone is the Shur
  • the Dragon Application (6103) is configured to match a sample of each of six sounds distinguishable by the Dragon Application (6103) which the operator can consistently produce. Using the example described above, these are the vowel sounds produced by the operator of e as it sounds in green, i as it sounds in white, u as it sounds in blue, e as it sounds in red, o as it sounds in orange and e it sounds in grey.
  • the speech recognition access program (6105) is preferably a Windows® application program developed using, in part, the Voice Tools Software Development Kit available from Dragon Systems, Inc.
  • the speech recognition access program (6105) defines an array of data structures defining the menu and submenu options and the menu hierarchy. For example, one of the elements of this array determines that, on selection, certain actions are to be taken, for example, inputting text to an application program, and that certain submenu options and related data are to be associated with certain child edit windows.
  • the speech recognition access program (6105) initializes callback procedures using the Dragon Application's Application Program Interface to receive notification from the Dragon Application when a sound has been matched. Also at initialization time, the main window of the speech recognition access program (6105) is preferably sized to just encompass the sound indicators shown in FIG. 58. Also at initialization time, the speech recognition access program (6105) creates six child edit controls, each corresponding to one of the sound indicators shown in FIG. 58. Each of the child edit controls has the background color described above and a text color of black or white depending upon which provides better contrast against the background color of the child edit control, and each is located on the display (2112) as shown in FIG. 58.
  • the menu options of the initial menu are each associated respectively with one of the child edit controls.
  • the speech recognition access program (6105) upon notification from the Dragon Application that a sound has been received and a match attempted, sequentially searches the list of identifiers matched to the sound by the Dragon Application (6103), starting with the best match, until it finds an identifier corresponding to any one of the six sounds distinguishable by the Dragon Application (6103). Assuming the operator says o as it sounds in orange, the Dragon Application (6103) provides to the speech recognition access program (6105) a list of matches including, before any other identifier corresponding to any one of the six sounds distinguishable by the Dragon Application (6103), the identifier associated with child edit control (5809).
  • This child edit control is currently associated with menu option "vort x", the matched menu option.
  • the speech recognition access program (6105) sets the text of each of the child edit controls to one of the submenu options associated with the matched menu option.
  • submenu options space, "x", “r”, “t”, “v” and “o” are associated with each of the child edit controls respectively and the text of the associated child edit control is set to the submenu option.
  • submenu option space is the matched submenu option and a space is input to the application program (6107) whose output appears in region (5850).
  • the application program (6107) is an application program capable of executing a WM -- SIZE command so that the speech recognition access program (6105) may size the windows of the application program (6107) to fit neatly in region (5850) and is capable of executing WM -- CHAR messages so that the speech recognition access program (6105) may input characters to the application program (6107).
  • FIG. 64 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) and 12 selectable regions.
  • Each of the 12 selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion located outside the display (2112) and adjacent the visible subregion.
  • the selectable region in the upper left corner both above and below the top of the display (2112) in FIG. 64 consists of invisible subregion (3604) and visible subregion (3606), and hereinafter is referred to as selectable region (3604/3606).
  • selectable region 3604/3606
  • selectable region (3604/3606) are (3608/36,10), (3612/3614), (3616/3618), (3620/3622), (3624/3626), (3644/3646), (3648/3650), (3652/3654), (3656/3658), (3660/3662) and (3664/3666).
  • each of the visible subregions is adjacent an edge of the display (2112).
  • the selectable regions together circumscribe region (3680) in the center of the display.
  • Also shown on the display (2112) within region (3680) in FIG. 64 are ten square indicating regions arranged in two columns of five indicating regions.
  • each indicating region abuts an indicating region in the other column, the indicating region located above it, if any, and the indicating region located below it, if any.
  • each indicating region indicates by its location the location of a respectively associated selectable region, in accord with the Location Indication aspect of the invention.
  • the uppermost indicating region in the left column of indicating regions (3605) is associated with the uppermost selectable region (3604/3606) on the left side of the display.
  • Indicating region (3609) located immediately below indicating region (3605) is associated with selectable region (3608/3610) located immediately below selectable region (3604/3606).
  • each of the plurality of the selectable regions shown in FIG. 64 is associated with one of a plurality of menu options, each of the plurality of menu options having a common characteristic.
  • the common characteristic is a phonetic unit and intonation but may be a phonetic unit alone, an intonation alone, a stroke used to draw an ideograph, a number of horizontal strokes, a number of vertical strokes, a number of total strokes, a stroke order, a radical, a part of speech, an ideograph, a kana, a diacritic, a classification of a part of an ideograph or other characteristic of a class of ideographs.
  • the ten menu options shown in FIG. 64 are Chinese ideographs each starting with the distinct sound "fu". Alternatively, the menu options may be sequences of graphic symbols including one or more kanji. In FIG.
  • each of the selectable regions associated with a menu option starting with the distinct sound "fu" is associated with a menu option not previously displayed and the newly associated menu option replaces the old menu option on the display.
  • a cross hair cursor (3686) is displayed in the circumscribed region (3680). Assuming the operator desires to select the menu option associated with selectable region (3608/3610), he turns his head to the left and the cross hair cursor (3686) moves to the left, responsive to the head movement, until the cross hair cursor hotspot intersects selectable region (3608/3610) and he maintains the location of the cross hair cursor hotspot on that selectable region for the selection threshold period.
  • the menu option associated with selectable region (3608/3610) is selected and added to text (3684) displayed in the circumscribed region (3680), the general purpose computer system (2218) emits an audible beep indicating that selection has occurred and the displayed menu options, both on the indicating regions and the visible subregions, are removed from the display.
  • selection is made in accord with the Facilitated Dwell subaspect of the Dwell aspect of the invention, described above.
  • the operator receives an indication of the progress of his selection by a change in appearance of the indicating region associated with the intersected selectable region.
  • selection may be by intersection of a location indicated by the at least part of a cursor and a selectable region alone, by such an intersection accompanied by a switch operation, for example, a depression of a space bar on the keyboard, or by other suitable means.
  • the operator sees the entire menu in the compact indicating regions and may discover the location of the selectable region associated with each menu option without having to visually scan all the visible subregions.
  • the plurality of indicating regions may be moved to a different location on the display to avoid obstructing the area of the display showing most recently added graphic symbols or the area of the display where graphic symbols will soon be added.
  • an operator may select from among many sequences of one or more ideographs without lifting either hand from the keyboard, thus speeding entry of single ideographs or sequences of ideographs in word processing systems for the Chinese, Japanese and Korean languages.
  • the selectable regions are located adjacent the edge of the display, a large rectangular region remains available on the display for the output of an application program. Further, if the selectable regions are located entirely outside the display, the indicating regions obstruct only a relatively small portion of the circumscribed region (3680), permitting the display of a sequences of ideographs for selection simultaneous with the display of previously selected ideographs, neither obstructing the operator's view of the other.
  • the general purpose computer system is coupled to a speech synthesizer and the ideographs are symbols of a symbol set, for example, the Blissymbolics Symbol Set
  • an illiterate operator for example, a school child having impaired speech, may select symbols associated with words and those words may be spoken via the speech synthesizer.
  • FIG. 65 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention.
  • 18 selectable regions (3704/3706), (3708/3710), (3712/3714), (3716/3718), (3720/3722), (3724/3726), (3728/3730), (3732/3734), (3736/3738), (3740/3742), (3744/3746), (3748/3750), (3752/3754), (3756/3758), (3760/3762), (3764/3766), (3768/3770), (3772/3774) and (3776/3778) circumscribe region (3780) on the display (2112).
  • the visible subregions of four of the 18 selectable regions abut each of the top, left and right edges of the display (2112).
  • the visible subregions of six of the 18 selectable regions abut the bottom edge of the display (2112).
  • the 18 indicating regions are located within the top half of region (3780). 12 of the 18 indicating regions are arranged in two columns of six indicating regions each. The column on the left is close to the visible subregions abutting the left edge of the display. The column on the right is close to the visible subregions abutting the right edge of the display. Two of the 18 indicating regions are located between the top indicating regions in each of the left and right columns. The remaining four of the 18 indicating regions are located between the bottom indicating regions in each of the left and right columns.
  • Each of the four indicating regions in the top row of indicating regions indicates the location of a respectively associated selectable region along the top edge of the display.
  • Each of the six indicating regions in the bottom row of indicating regions indicates the location of a respectively associated selectable region along the bottom edge of the display.
  • Each of the middle four indicating regions in each of the left and right columns of indicating regions indicates the location of a respectively associated selectable region along the left and right edge of the display.
  • the 16 menu options shown in FIG. 65 are Chinese ideographs each of which includes the radical kou, a radical having the shape of a rectangle. Each ideograph is displayed on the visible subregion of the associated selectable region and on the indicating region associated with the selectable region.
  • the remaining two of the 18 selectable regions, (3724/3726) and (3744/3746) are associated with menu options for undo and for displaying more menu options, respectively.
  • the indicating regions (3782) in FIG. 65 are disproportionately large relative to the rest of FIG. 65. They are approximately 1.5 times their proportionate size. They are represented as shown in FIG. 65 in compliance with Patent Cooperation Treaty Rules requiring a minimum size for letters in figures.
  • FIG. 66 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention.
  • FIG. 66 depicts a display (2112) of a computer system (2116) having thereon ten square indicating regions, (3801), (3303), (3805), (3807), (3809), (3811), (3813), (3815), (3817), and (3819), arranged in two columns of five indicating regions.
  • Each indicating region abuts an indicating region in the other column, the indicating region located above it, if any, and the indicating region located below it, if any.
  • Each indicating region indicates the location of a respectively associated selectable region (not shown) outside the display.
  • FIG. 67 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention.
  • FIG. 67 depicts a display (2112) of a computer system (2116) having thereon eight selectable regions (6701), (6703), (6705), (6707), (6709), (6711), (6713), and (6715), each located on the display (2112) adjacent the edge of the display (2112) and associated respectively with a menu option. Together the eight selectable region circumscribe region (6750) on the display. In FIG. 67 each of eight menu options is displayed on its associated selectable region.
  • the sequences are ordered by length, the shorter sequences on selectable regions on the left side of the display (2112) ordered by length from the top to the bottom of the display (2112), the longer sequences along the right side of the display (2112) ordered by length from the top to the bottom of the display (2112).
  • the remaining two of the eight selectable regions, (6709) and (6711) are associated with menu options for displaying the previous and the next display of menu options, respectively.
  • the menu option associated with a selectable region may be selected by a selection event.
  • the Ideographic Language aspect of the invention is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention, modified as described in the descriptions of the Facilitated Dwell subaspect and the Location Indication aspects of the invention.
  • the modifications required for the Ideographic Language aspect of the invention are: (1) install on the general purpose computer system a font for the ideographic language of the embodiment; (2) at initialization time: (a) set the text of all labels for display on the selectable regions to null; (b) do not enable the cursor polling timer; and (c) hide the Windows® (1204) system cursor; (3) include in the main window processing procedure of the access program (1206) code to process WM -- CHAR messages and, when a sequence of one or more WM -- CHAR messages indicates a common characteristic: (a) lookup sequences or representations of sequences of one or more ideographs (hereinafter "sequences") having the common characteristic; (b) copy ten of the sequences to the labels; (c) set the Windows® (1204) system cursor to the cross hair cursor, set the cursor location to a predetermined location near an indicating region and show the Windows® (1204) system cursor; (d) set fpaint to TRUE for every state machine; (

Abstract

The apparatus and method of the invention relate to data entry and menu selection. Applications include: (a) data entry for ideographic languages, including Chinese, Japanese and Korean; (b) fast food ordering; (c) correction of documents generated by optical character recognition; and (d) computer access and speech synthesis by persons temporarily or permanently lacking normal motor capabilities. In a preferred embodiment, each option of a menu is associated respectively with a selectable region displayed adjacent an edge of a display, forming a perimeter menu and leaving a region in the center of the perimeter menu for the output of an application program. Selectable regions may be on the display, outside the display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold period or by a cursor path toward the selectable region, or by a combination thereof Remaining dwell time required to select a selectable region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored indicating regions and selection of an ideograph may be made by speaking the name of the associated color.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This is a continuation-in-part application of pending prior international application Number PCT/US95/03591 filed Mar, 27, 1995, International Publication Date Oct, 3, 1996, International Publication No. WO 96/30822, entitled "Method of and Apparatus for Data Entry" which designated the United States, as amended Apr 25, 1995 and May 26, 1995.
FIELD OF THE INVENTION
The present invention relates generally to interactive display terminals and interactive display methods, and more particularly, to interactive display terminals and interactive display methods for use by persons temporarily or permanently lacking normal motor capabilities. It also relates to systems and methods for the assessment of the motor capabilities of persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for use in speech synthesis for persons having impaired speech. It also relates to systems and methods for the control of devices, including appliances, by persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for selecting one menu option from a menu. It further relates to systems and methods utilizing sound recognition for selecting a menu option from a menu. It further relates to data and order entry systems including, and data and order entry methods utilizing, an interactive display terminal. It also relates to interactive display terminals and interactive display methods for displaying and selecting ideographic characters, such as are used in the Chinese, Japanese and Korean languages. It also relates to interactive display terminals and interactive display methods for producing an indication of progress toward and/or away from selection of a menu option.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the World Intellectual Property Organization patent file or records, but otherwise reserves all copyright rights whatsoever.
BACKGROUND ART
Many persons suffer from various neurogenic muscular disorders, such as Cerebral Palsy ("CP"), Traumatic Brain Injury, Spinal Cord Injury, Muscular Dystrophy, Amyotrophic Lateral Sclerosis and Multiple Sclerosis. These conditions can result in a reduced ability to voluntarily control or prevent the movement of parts of the body, including the head, limbs and digits, muscle stiffness, weakness, limited range of motion, abnormal posture, involuntary muscle tremors, involuntary muscle activity causing involuntary motion, impaired ability to voluntarily stop motion, impaired ability to coordinate muscle activity, and/or impaired ability to sense the position of a part of the body. Any one of these symptoms may impair an affected individual's fine motor control. Moreover, while some individuals affected by a neuromuscular disorder may be able to exercise fine motor control with enormous effort, the struggle to do so often fatigues the individual, limiting the period of time the individual is capable or comfortable performing the fine motor control task.
Neuromuscular disorders are often systemic in effect, impairing an individual's ability to operate prosthetic devices, such as a wheelchair, and to perform the activities of daily life, such as speaking, walking and operating household appliances. Speech is frequently affected since the mechanics of producing speech require coordination of many muscle groups--the muscles of the diaphragm which push air over the vocal cords, the muscles of the larynx, jaws, tongue and lips. The inability to use or coordinate these muscle groups may result in impaired speech. Depending upon the degree of impairment, speech may be totally absent, present but impaired to the point of unintelligibility, or intelligible on the whole but with occasional unintelligible words. The ability to walk is often affected since walking requires coordination and voluntary control of many muscle groups. Furthermore, impaired fine motor control may prevent or impede an individuals from effectively operating household appliances or computer input devices.
Devices are available which produce speech, control appliances and facilitate computer access for some persons having neuromuscular disorders ("NMD operators"). Devices which produce speech for individuals whose own speech is impaired, called Augmentative and Alternative Communication ("AAC") devices, allow the operator to select words or phrases by spelling the words, by specifying an abbreviation for the phrase or by selecting a sequence of symbols, and then speak the selected words or phrases using an electronic speech synthesizer. However, due to the systemic nature of neuromuscular disorders, NMD operators are often unable to efficiently use a standard keyboard and mouse. For example, an NMD operator who is unable to stop the movement of a limb with precision, when attempting to use a keyboard or mouse, may move his arm toward the target key or move the cursor toward the target object on the display but overshoot the target. If he has involuntary tremors and cannot hold a limb still, then, when attempting to use a keyboard, he may hit keys adjacent to his target key. If he has involuntary motion moving left to right, then, when attempting to use a keyboard, he may have difficulty accessing an intended key on the right side of the keyboard.
The benefits of interfacing an NMD operator to a general purpose computer so that he may control the computer and devices attached to it ("computer access") are both numerous, because many of the problems faced by the disabled are susceptible to a computer-driven solution, and profound, because of the psychological deprivation occasioned by a severe physical disability. The benefits potentially obtained through computer access for individuals affected by neuromuscular disorders include:
a. Speech synthesis. A computer connected to a speech synthesizer enables an NMD operator with impaired speech to direct the computer to speak for him.
b. Device control. A user who is physically unable to operate a household appliance, for example, a television, video cassette recorder, compact disc player, radio, alarm clock, telephone, light, thermostat, dimmer or power switch, may be able to control the appliance via a computer equipped with an interface he can control.
c. Access to general purpose computer applications. NMD operators may make use of the same general purpose computer application programs ("applications") as able-bodied users, including applications for word processing, database, computer-aided instruction, access to literature accessible via computer, spreadsheet, time management and computer utilities.
d. Enhanced self-esteem and peer approval. Adolescents with CP are obviously different from their peers. They are often surrounded by non-normative assistive technology, e.g. wheelchair or walker, special school bus equipped with a chair lift, stair lift, standing aid, AAC device, feeding apparatus, bath seat, toiletting apparatus, etc. In addition, they may drool, lacking the ability to coordinate lip closure with swallowing. Nonetheless they are adolescents and need peer approval to support them in their maturation from dependent children to independent adults. Today, demonstrated facility with a computer is an emblem of intelligence among adolescents, so computer use provides adolescents the opportunity to prove their intelligence and thus potentially rewards NMD operators with both self-esteem and peer approval.
e. Privacy. Some severely disabled school-aged children require nearly constant physical assistance to transfer them to and from bed, to feed them, help them with toiletting and personal hygiene, etc. Because they are constantly attended, all their mistakes in class or when doing homework are known to their attendant, often a family member. They do not have the opportunity to make mistakes in private. Computer use, if it can be done without assistance, affords the NMD operator the opportunity to avoid the embarrassment of showing their failings to their attendant.
f. Expanded personal interaction. Some severely disabled individuals, e.g. quadriplegics, are essentially incarcerated by their disability. They interact with their family or their caretakers, depending upon whether they live at home or in an institution. Their circle of friends is often very small. Using a computer and a modem, they can expand their circle of friends to include the tens of thousands of people who periodically connect to worldwide electronic networks to trade information on topics of mutual interest. Moreover, the interaction via present computer networks is mostly textual; there is no voice or visual interaction between users. Since messages are customarily composed and read off-line to minimize connect time charges, no one even knows how long it took the sender to enter the text. Electronic networks thus afford the disabled user an opportunity to relate to others on an equal footing, not as a disabled person to his able-bodied peers, something many NMD operators dearly want to do but were never able to.
NMD operators vary widely in their motor capabilities. Even individuals having the same medical diagnosis may require completely different technologies for computer access. Many NMD operators are able to use an oversize keyboard, a device having a pressure-sensitive surface divided into squares, each square associated with a letter of the alphabet. The squares may be sized to match the operator's abilities, but typically each square is two inches on either side. NMD operators who are unable to efficiently use an oversize keyboard may use another conventional computer access solution, called an "on-screen keyboard", which, as illustrated in FIG. 1, is a picture of keyboard drawn on a computer display (1101). The operator selects a letter by pointing to that letter's key image on the display with a pointing device ("pointer"), then indicating that he has reached his target either by operating a switch, a process called selection by click, or by maintaining the location indicated by the pointer ("dwelling") on the key image for a predetermined period of time (the "selection threshold"), a process called selection by dwell. Switch operation includes, but is not limited to, each of the following: opening the switch, closing the switch, opening the switch multiple times within a predetermined period, and closing the switch multiple times within a predetermined period. A program executing on the computer determines which letter the operator has selected and processes the letter or passes it to some other application program which processes the letter as if it came from a true keyboard.
Conventional pointing devices include a mouse, trackball, joystick (which may be integrated into a keyboard, e.g. TrackPoint II®), stylus and graphics tablet, lightpen, thumb wheel, touch screen, touch panel, head pointer, occulometer, intraoral pointer and eye tracker. They may be active, e.g. a lightpen that emits an infrared beam, or passive, e.g. an eye tracker that uses images of an individual's eyes to determine where his eyes are focusing. Conventional switches include a button on the mouse, a switch in the tip of the stylus actuated by pressure or the release of pressure, a switch mounted on the user's wheelchair operated by a turn of the head to or the switch below a keyboard key.
Dwell time may be continuous or discontinuous depending upon the operator's motor capabilities. In continuous dwelling, if the operator moves the cursor from one key image to another region of the display, the time accumulated on the key image is discarded so that if the operator returns to that key image he must dwell on it for the full selection threshold to select it. Discontinuous dwelling, by contrast, compensates for involuntary tremors which pull the operator off the desired key image. Accumulated dwell time on a key image is remembered, so that on return to a key image, the operator need only dwell for a period equal to the difference between the selection threshold and the previously accumulated dwell time for that key image. Accumulated dwell time is reset to zero for all key images following the selection of any one key image. Conventional on-screen keyboards do not indicate to the operator the dwell time associated with any key image.
There may be a single selection threshold period for all key images or each key image may be associated with its own selection threshold period. In the latter case, keys associated with shorter selection threshold periods are easier to select than keys associated with longer selection threshold periods.
As was mentioned earlier, computer access permits an NMD operator to run a variety of applications. One such application is speech synthesis. In a computer-based speech synthesis system, a computer system displaying an on-screen keyboard is connected to a speech synthesizer. The operator spells the desired word or words using the on-screen keyboard. These are then spoken by the speech synthesizer. Another application of the on-screen keyboard is word processing. FIG. 2 illustrates an example of a combined display of an on-screen keyboard and a word processing application program. The on-screen keyboard (0201) is shown on the lower portion of a display connected to a computer system (not shown) which also executes the word processing application program whose output (0203) appears on the upper portion of the display. Letters selected by the operator are input to the word processing application program.
Due to impaired fine motor control, many NMD operators have difficulty selecting a key image by click or by dwell and this difficulty increases as the size of the key image decreases. FIG. 1 shows an on-screen keyboard containing 81 total keys including 26 alphabetic keys, 10 numeric keys, 12 function keys, 4 arrow keys and 29 special purpose keys. Drawing this many key images on a display restricts the size of each key image making each very difficult for many NMD operators to select.
When a display is shared between application program output (0203) and an on-screen keyboard (0201), as is the display shown in FIG. 2, the size of each key image must be reduced from its size in FIG. 1 to allow space for the application program output. Thus, as more display space is allotted to application program output, the key images become more difficult for an NMD operator to select.
Many NMD operators have difficulty using the conventional dwell selectable on-screen keyboard because they cannot maintain a steady pointer position. The body member with which they control the pointer may move slightly ("drift") when they want it to remain still. One approach to this problem is a variation of the on-screen keyboard, depicted in FIGS. 3, 4 and 5 and called a quaternary on-screen keyboard ("quaternary keyboard"). The quaternary keyboard provides for larger key images. The alphabet is divided into four groups of letters, each displayed in one of the four quadrants (1302), (1304), (1306) and (1308) of the display, as shown in FIG. 3. The operator selects one of the four groups by, for example, pointing to and dwelling on one quadrant of the display. The selected group is then exploded into four subgroups, each displayed in one quadrant of the display, as shown in FIG. 4. Once more the operator selects one of the four. The selected group is exploded into four letters and each letter displayed in one quadrant of the display, as shown in FIG. 5. The operator again selects one of the four. This letter is then input to an application program (not shown).
The quaternary keyboard illustrates the use of a menu hierarchy in computer access. Each of the four groups of letters (1302), (1304), (1306) and (1308) is a menu option. Each of these menu options is itself a menu which includes other menu options. A menu hierarchy exists if at least one of a menu's menu options is itself a menu. Hereinafter, a menu accessed from another menu may be called a submenu, and the options of the submenu may be called submenu options. If a menu hierarchy is narrow and deep, many selections are required to make the desired choice. If a menu hierarchy is broad and shallow, each layer is composed of many menu options.
The quaternary keyboard greatly expands the size of a single key image and thus accommodates certain NMD operators with drift or involuntary tremors. The cost of this adjustment is high Instead of selecting a letter with one pointing motion and dwelling for one selection threshold, the quaternary keyboard requires three pointing motions and dwelling for three selection thresholds. Thus the operator's productivity is dramatically reduced from the standard on-screen keyboard depicted in FIG. 1.
The computer access advantage gained from the quaternary keyboard is greatest when the quaternary keyboard occupies the entire display. In this configuration the size of each of the four active display regions is maximized, making them easier to hit and dwell on for the operator. However, this configuration allows no room on the display for the output of the application program being run by the operator, the reason he is sitting at the computer in the first place. This does not prevent the on-screen keyboard from passing letters to the application program since an application program need not be visible to be active, but it does prevent the operator from seeing what the application program has to show him. The more of the application program output that is displayed, the smaller the on-screen keyboard, the smaller each active region of the on-screen keyboard and the more difficult access becomes. FIG. 6 illustrates a display combining a quaternary keyboard and output from two application programs.
Another conventional structure for selecting of a menu option from a menu is a pie menu A pie menu is an opaque region on a display divided into selectable slices, each slice associated with a menu option. The pie menu suffers some of the drawbacks discussed above, particularly that, while displayed the pie menu occupies more space than a linear menu and obscures much of the output of the operator's application program. For illustrations and a discussion of pie menus, see Callahan, Jack et. al., "An Empirical Comparison of Pie vs. Linear Menus", Computer Science Technical Report Series, CS-TR-1919, University of Maryland, College Park, Md., September 1987.
NMD operators who cannot effectively use a conventional keyboard or a pointing device may use a computer access method called "joystick patterns". FIG. 7 depicts a conventional joystick pattern device. The device (1602) is connected to a joystick and to a computer. The operator pushes the joystick to the top, bottom, left, right, top left corner, top right corner, lower left corner or lower right corner, closing one of eight switch contacts within the joystick housing. That switch position is then indicated on the display (1604). A sequence of consecutive of switch closures encodes a letter or other programmed output that the device (1602) displays on an LCD display (1606) and sends to the connected computer, simulating keyboard input.
The conventional joystick pattern device is ill-suited for many NMD operators. The involuntary tremors common some neuromuscular disorders may result in unintended switch closures. In addition, the device does not provide an indication that the operator is moving a body member in an unintended direction until switch closure occurs. For example, an operator with CP who intends to move the joystick the right but actually moves it to the upper right receives no indication from the device, prior to switch closure, that he's not on target. Moreover, the device requires that the operator memorize the encoding of each letter or other output since there's no indication on the display (1606) which sequence encodes which letter. Further, the device provides no support for head pointing, although the head is often the best controlled part of an NMD operator's body.
NMD operators who cannot effectively use either a conventional keyboard or a pointing device but can reliably actuate a switch may use a computer access method called "scanning", which is subdivided by cursor control technique into three types of scanning: automatic, directed and step. In automatic scanning all the operators' options, for example, the letters of the alphabet, appear on either a static or dynamic display (depending upon the implementation), organized in rows and columns. At the scanning interval, usually about one second, a cursor moves from one row to the next. When the cursor indicates the row containing the letter the operator wants, he closes a switch. The machine now moves the cursor from one letter to the next within the selected row until the operator closes the switch again. The operator has now selected one letter. In directed scanning, like automatic scanning, the cursor moves at the frequency determined by the scanning interval, however, it moves only when the switch is closed. To select an option, such as a row or a letter in a row, the operator opens the switch while the cursor indicates the desired option. In step scanning, the cursor moves with each switch activation.
As one can well imagine, writing a sentence via any of these scanning techniques is an extremely slow process, since selecting a single letter may take many seconds.
Problems of computer access cascade and affect the quality of verbal interactions between AAC device operators ("AAC operators") and others. People speak much faster than they type. Not surprisingly, operators who speak with AAC devices, particularly NMD operators whose motor deficits impair their ability to use a keyboard, lag substantially in their conversations. The slow pace of an AAC operator's word production disrupts normal verbal interaction. Speaking persons, accustomed or not to the AAC operator's slow rate, often lose patience in conversations with AAC operators. They may prematurely terminate the conversation, read the AAC device display in an attempt to guess at the AAC operator's intended utterance and so accelerate the interaction, lead the AAC operator, ask predominantly yes/no questions, change the topic of conversation with little input from the AAC operator and otherwise dominate the interaction. The AAC operator often has difficulty participating as an equal partner in the conversation. He may be unable to change the topic, interject a humorous comment in a timely fashion or respond to a question before the speaking person changes the topic. Slow AAC operators may be perceived as mentally slow. Thus the quality of verbal interactions where one party uses an AAC device to speak depends significantly upon the AAC operator's rate of word production.
Increasing an operator's letter or menu option selection rate proportionately increases his word production rate and increases the operator's productivity in data entry generally. Letter or menu option selection time includes the time the operator requires (a) to comprehend the menu options displayed, (b) to move the pointer to the desired menu option on the display, and, in selection by dwell, (c) the selection threshold period, or, in selection by click, (c) the time required to operate the switch. Decreasing any one of these increases the operator's productivity, assuming all other steps in the selection process are unaffected.
Personal interactions are composed of more than speech alone. People in conversation gesture to one another, use facial expressions, change the object of their gaze and make non-speech utterances (e.g. "hmmm-mmmm") to bid for a turn to speak, to grant such a bid made by the other party, to request to continue speaking and to acknowledge, accept or dispute what has been said. Ideally, the production of speech from an AAC devices does not distract the AAC operator from the personal interaction and subject matter of the conversation. This is possible if the operator habituates the AAC device access technique and menu structure, producing speech without focusing on each step of the process, much as automobile drivers habituate mechanical tasks, such as changing gears and switching between foot pedals, and focus their attention on pedestrians or traffic lights while operating their vehicle.
Another consequence of personal interaction during conversation for an AAC device operator is that the operator needs a way to easily enable and disable the AAC device operator interface so that movement the operator makes during personal interaction, for example, nodding his head, is not interpreted by the AAC device.
As noted previously, neurogenic muscular disorders may impair the ability of an individual to sense the position of a body member. An NMD operator thus relies more than his able-bodied peer on the location of a cursor or similar automated indication of body member position. Conventional access methods which use a pointer do not provide additional feedback to the operator of the position of a body member.
Access methods which require the NMD operator to make the same movement for most selections, such as single switch access, mouth sticking (the use of a small rod held in the mouth and used to depress keys on a keyboard) and head sticking (the use of a rod mounted on the head and used to depress keys on a keyboard), may result in repetitive motion injury, particularly after years of use.
The need for quick selection from a menu also arises from the use of optical character recognition ("OCR") systems which attempt to recognize graphic symbols and words based on attributes for optical recognition purposes, for example, the appearance of graphic symbols, the ratio of dark space to light space within part or all of a graphic symbol, the ratio of dark space in one part of a graphic symbol to the dark space in another part of the graphic symbol, and the derivative of darkness over the scan of the graphic symbol. OCR systems convert the contents of a typewritten document into a computer encoding of the same. OCR systems at times are unable to recognize a graphic symbol or word, or may err in selecting a graphic symbol or word from a plurality of candidates. Therefore, following optical character recognition, a human may proofread and correct the computer encoded document. The proofreader may indicate where an error or omission in the computer encoded document occurred and may select from a plurality of menu options, each representing a candidate graphic symbol or word.
There are several aspects of the invention, each addressing one or more of the problems described above and/or one or more problems specifically addressed by that aspect of the invention. The objects, disclosure and description of each aspect is separately described below under one of the headings: (A) Perimeter Menu, (B) Confinement, (C) Dwell, (D) Path Directness, (E) Intersection, (F) Alignment, (G) Length Order, (H) Location Indication, (I) Sound Match, and (J) Ideographic Languages,. Where there is background art applicable to an aspect in addition to that described above, the additional background art is described below.
A & B. Perimeter Menu and Confinement
One object of the invention is to facilitate computer access by a disabled operator.
A further object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to maintain a body member in a steady position.
Another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to stop motion.
Yet another object of the Perimeter Menu and Confinement aspects of the invention is to simultaneously display an application program window and a computer access menu which does not obstruct the application program window.
Another object of the Perimeter Menu and Confinement aspects of the invention is to allow an operator to enable and disable a menu.
A still further object of the Perimeter Menu and Confinement aspects of the invention is to synthesize speech for an operator having impaired speech and impaired fine motor control.
Still another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate device control for a disabled operator.
Another object of the Perimeter Menu and Confinement aspects of the invention is to reduce the cognitive demand of speech synthesis for the disabled.
A still further object of the Perimeter Menu and Confinement aspects of the invention is to enlarge the effective area of a selectable region without concomitantly reducing the area available for information display.
Another object of the Perimeter Menu and Confinement aspects of the invention is to speed data entry.
Yet another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate computer access for an operator having impaired ability to sense the position of a body member used for computer access.
Another object of the invention is to facilitate the correction of errors or completion of omissions in a computer encoded document produced, at least in part, by optical character recognition.
C. Dwell
Conventional systems allowing selection by dwell do not provide an indication to the operator of either how much dwell time has been accumulated for any selectable region or how much more dwell time is required to select a selectable region. Consequently, an operator of a conventional system who is dwelling on a intended selectable region has no indication, other than his estimation from prior use of the system, that he has nearly made his selection and can plan his next movement to the next selectable region or that he has very nearly made his selection and can begin moving to the next selectable region. Furthermore, an operator of a conventional system who is dwelling on an unintended selectable region, has no indication, other than his estimation from prior use of the system, of how close he is to making an unintended selection and thus how important it is to act quickly. Conventional systems using discontinuous dwell give no indication of the accumulated dwell time associated with a selectable region either when the operator dwells on that region or when the operator ceases dwelling on that region. Some disabled users can dwell relatively easily on their intended targets for short periods of time, but have difficulty dwelling for long periods. If such an operator knows that only a little more dwell time is needed he may be able to satisfy the dwell time required for selection, without preparing himself to dwell for an extended period.
Conventional menu-driven data entry and order entry systems incorporating pointing at intended selections employ a two step selection procedure. In the first step the operator indicates, with a pointer, his intended selection. The system then provides feedback, for example, by highlighting the indicated selection, showing which selection the operator has indicated. In the second step, the operator selects the indicated selection, for example, by operating a switch. Thus, conventional data entry and order entry systems are ill-suited to circumstances where the operator cannot easily operate a switch while maintaining the pointer on the intended selection.
While the two step procedure is not complicated, many operators require some training to learn it, and, if they are infrequent users of the system, these operators may require refresher training. Simplifying the procedure further would lessen the need for initial and refresher training.
One object of the invention is to facilitate the use of systems allowing selection by dwell.
Still another object of the invention is to facilitate device control by the disabled.
A further object of the invention is to increase the independence of the disabled.
Yet another object of the invention is to facilitate the use of a data entry or order entry system by an intermittent operator.
Another object of the invention is to facilitate ordering by someone seated in a vehicle.
D. Path Directness
The on-screen keyboard with dwell selectable key images is ill-suited for use by many NMD operators. Selection by dwell may fatigue NMD operators or may require greater fine motor control than they bring to this task. Operators with impaired ability to stop motion and those having involuntary tremors have difficulty maintaining the location indicated by a pointer on a key image for a period sufficient to distinguish intentional dwelling from unintentional dwelling. Consequently, some NMD operators who try to use on-screen keyboards often miss their target key images and/or accidentally select unintended key images. Following such an error, the operator must erase his erroneous selection by selecting the backspace or undo key. As the number of erroneous selections increases, the operator's productivity decreases markedly, since each error requires a correction in which there might be another error.
Conventional on-screen keyboards require the ability to select by dwell or by click and thus are limited to operators with these capabilities. Conventional on-screen keyboards do not utilize the relatively intact motor capabilities of some NMD operators to compensate for impaired ability to select by dwell or by click or to speed up the slow process of selecting by dwell. For example, while an NMD operator may overshoot a key image, his directional control may be relatively intact. Conventional on-screen keyboards do not exploit this capability.
The dominant computer operating system for graphic applications on general purpose computer systems today is the Windows® Operating System. Windows® assigns meaning to the cursor location. When the operator moves the cursor on top of a menu item and clicks, Windows® interprets the action as manifesting an intent to choose that menu item. The operator's path to that menu item, whether direct or circuitous, is irrelevant. Operators who can move toward a target accurately but cannot maintain the location indicated by a pointer on the target cannot effectively use standard Windows® applications through the conventional interface to these applications.
Often NMD operators cannot steady a pointer while operating a switch; the act of operating the switch triggers involuntary muscle activity pulling the cursor off target. For these operators, conventional selection by click is not practicable. Conventional selection by dwell also requires greater fine motor control than many NMD operators bring to this task. Operators with impaired ability to stop motion may overshoot their intended target. Operators whose voluntary muscle activity is accompanied by some involuntary muscle activity affecting their directional control often cannot point accurately. Operators with involuntary tremors often cannot maintain the location indicated by a pointer on a key image. Consequently, NMD operators who try to use on-screen keyboards often miss their target key images and accidentally select unintended key images. Following such an error, the operator must erase his erroneous selection by selecting the backspace or undo key. As the number of erroneous selections increases, the operator's productivity decreases markedly, since each error requires a correction in which there might be another error.
Measures of an individual's fine motor control assist a physician or therapist in evaluating the effectiveness of a treatment program, including assistive technology, and in gauging the severity of a disability. Such measures help the physician or therapist in determining what treatment course to pursue and whether the severity of a certain disability justifies the risk of a particular treatment option, such as neurosurgery.
One object of the invention is to facilitate selection of an option from a menu.
Another object of the invention is to indicate to an operator moving a cursor toward an option in a menu displayed by a computer system, which option the computer system believes the operator is moving toward.
Another object of the invention is to display a menu on a display so that a large contiguous area on the display is not obstructed by the menu.
Still another object of the invention is to make use, in computer access, of relatively unimpaired directional control in persons having impaired fine motor control.
A further object of the invention is to indicate to an operator moving a cursor toward a dwell-selectable option in a menu, a changed selection threshold of the dwell-selectable option.
Yet another object of the invention is to more efficiently select an option from a menu on a display.
Another object of the invention is to speed up selection of an option from a menu by an operator having impaired ability to operate a switch while the operator simultaneously keeps a cursor location within a region on a display.
Another object of the invention is to help an operator with a disability control a pointer.
Yet another object of the invention is to speed data entry by an individual with a disability.
A still further object of the invention is to facilitate artificial speech generation by a person having impaired speech due to a neurogenic muscular disorder.
Yet another object of the invention is to facilitate device control by a person having a neurogenic muscular disorder.
Another object of the invention is to measure an individual's ability to move one of the individual's body members in a direct path from a starting position to an ending position.
Yet another object of the invention is to facilitate computer access for an individual who cannot stop movement cleanly.
E. Intersection
One object of the invention is to facilitate computer access by an operator having impaired ability to maintain a body member in a steady position.
Another object of the invention is to facilitate the selection of a desired menu option by an operator having impaired fine motor control.
A further object of the invention is to synthesize speech for an operator having impaired speech and impaired motor control.
Yet another object of the invention is to use an operator's directional control in computer access.
A still further object of the invention is, in selecting a menu option from a menu of dwell-selectable menu options, to compensate for an operator's impaired ability to maintain a body member in a steady position by using the operator's relatively intact motor capability .
F. Alignment
Conventional on-screen keyboards do not compensate for NMD operators' inability to stop motion. Suppose the operator has been fitted with a head pointing device so that his head motion moves the cursor and that he's using the quaternary keyboard shown in FIGS. 3, 4 and 5. Assume further that, as he attempts to point to the quadrant containing the "j" key image (1308) in FIG. 3, he is unable to stop on that quadrant and continues turning 20 more degrees to the left. There are two known ways of responding to this situation: (1) the cursor may continue to track the operator's motion and disappear from the display, leaving no indication to the operator of the location of the cursor and consequently causing some operator disorientation, or (2) the cursor may "stick", i.e. remain confined to the display, at, for example, point (1312). Conventional on-screen keyboards respond in this way. The operator's line of sight is now 20 degrees to the left of the cursor location. After the dwell period, the quadrant (1308) is selected and FIG. 4 is displayed. The cursor hasn't moved. It is now at point (1320). Assume that again the operator attempts to point to the quadrant containing the "j" key image (1324) in FIG. 4. As the operator turns his head to the right, the cursor immediately moves with him. Thus, the operator's line of sight remains 20 degrees to the left of the cursor location as the cursor moves to the right across the display. The operator must watch the cursor out of his right eye. The problem is aggravated if either the operator cannot cleanly stop or if he drifts as he dwells. Assume that while attempting to dwell on quadrant (1324) the operator drifts 25 degrees past the bottom of the screen. His line of sight is now 25 degrees below and 20 degrees to the left of the cursor. To correct this misalignment in the conventional quaternary keyboard, the operator must turn his head to the right, "stick" the cursor against the right edge of the display, and continue turning 20 degrees until he has the cursor in his line of sight. Then he must lift his head until he sticks the cursor against the top edge and continue lifting 25 degrees more. Alternatively, in this scenario, the operator could stick the cursor in the upper right corner of the display and simultaneously rotate his head up and to the right until he brought the cursor into his line of sight.
Alignment is also a problematic for NMD operators who use a pointer, such as a mouse, with which the operator indicates by a location on a surface, e.g. a desk top, which corresponds to a desired location on the display, and achieve alignment by removing the pointer from the surface, e.g. lifting the mouse, moving the mouse, then replacing it on the surface. Due to impaired fine motor control, many NMD operators cannot remove a pointer from the surface and replace it on the surface at a desired location without unintentional movement or extraordinary effort. For these operators, alignment cannot be effectively achieved through conventional means.
In summary, misalignment interferes with accurate pointing and the process of correcting for misalignment may result in the selection of unintended key images.
One object of the invention is to allow an operator to align a pointer with a location on a surface.
Another object of the invention is to indicate to an operator a location on a surface with which he may align a pointer.
Still another object of the invention is to indicate to an operator when he may align a pointer with a location on a surface.
A further object of the invention is to allow an operator having impaired motor control to align a pointer with a cursor.
G. Length Order
As noted previously, one of the elements determining the menu option selection time is the time the operator requires to comprehend the menu options displayed. This time may be reduced if the operator can limit the number of menu options he searches in looking for his desired menu option.
Conventional word prediction systems attempt to reduce this operator search time. The operator of a conventional word prediction system may, for example, select the letter "p". The system displays some number, say six, of the most frequently used words beginning with the letter "p". Conventionally these six words are displayed either in alphabetic order or in order of frequency of use. Assuming the operator does not see his desired word on the display, he selects another letter, say "r". The system then displays the six most frequently used words beginning with the letters "pr".
Searching a displayed list of words in alphabetic order requires that the operator focus his attention on the selection task, as opposed to the information content of the conversation or other task the operator is engaged in. Further, determining whether a given word is alphabetically greater or lesser than a desired word takes substantial time, slowing the selection process. An alphabetically ordered list is of limited use to an individual who has below normal spelling ability, a frequent problem among individuals with impaired speech. Ordering words by frequency of use often does not limit the number of words the operator must search. The word at the bottom of the displayed list, for example, the sixth most frequently used word beginning with the letters "pr" may be a very common word, even though it is less frequently used than the other five displayed words.
One object of the invention is to reduce the time an operator requires to comprehend displayed menu options.
Another object of the invention is to reduce operator search time.
Still another object of the invention is to limit the number of menu options an operator searches for in looking for his desired menu option.
A further object of the invention is to speed data entry.
Yet another object of the invention is to increase productivity in speech synthesis for an operator having impaired speech.
H. Location Indication
The difficulties experienced by NMD operators in pointing to relatively small selectable regions have already been described. One approach to these difficulties is to enlarge the on-screen selectable region, illustrated by the quaternary expansion on-screen keyboard already described. Another approach is the conventional eye gaze system for a speech impaired individual, depicted in FIG. 8. The system consists of a plexiglass frame (6352) having a centrally located aperture (6354). The eye gaze system is positioned between the speech impaired individual and person with whom the speech impaired individual is communicating. There are eight groups of five squares each on the plexiglass frame. Each square within each group of five squares is color coded, e.g. red, blue, green, yellow and clear, matching the color on each of the four corners of the plexiglass frame. The clear square matches the aperture (6354). All squares are labeled with symbols representing items to be communicated. These labels are not shown in FIG. 8. The person with whom the speech impaired individual is communicating observes the eyes of the speech impaired individual to determine the target of the speech impaired individual's eye gaze. To communicate an item, the speech impaired individual gazes first toward the one of the eight groups of five squares, indicating that he wants to communicate one of the symbols in that group, and gazes second toward one of the four corners and aperture (6354) matching the color of the square labeled with the item to be communicated in the previously indicated group.
Two types of selectable regions are conventionally used in a point and click menu interface in a graphical user interface. The first, shown in FIG. 9, depicts a menu having three menu options, labeled "High", "Medium" and "Low", each displayed on a display (4807), each associated respectively with selectable regions (4801), (4803) and (4805), and each located adjacent the associated selectable region. FIG. 10 depicts a menu having the same three menu options, each displayed on a display (4807), each associated respectively with selectable regions (4901), (4903) and (4905), and each intersecting the associated selectable region. In both these conventional menus, a menu option is selected by pointing to and clicking on the associated selectable region.
Conventional menu hierarchies in automated systems, built from menus of the type shown in FIG. 9 or FIG. 10, require that the operator proceed sequentially through the steps of searching menu options, selecting one of them, and, assuming a menu option including a submenu was selected, searching the submenu options, and selecting one of them. Where selection from menu hierarchies constitutes a substantial component of the operator's activities, the slowness of the selection process diminishes productivity.
Locating selectable regions or parts thereof outside the display, in accordance with the Perimeter Menu aspect of the invention, allows the large areas outside the display to be used, a major advantage for operators having impaired fine motor control who are unable to maintain a pointer on a small selectable region while selecting by click or by dwell. However, if menu options are displayed on the display near the perimeter of display and near their associated selectable region, the operator has an indication of the location of each selectable region but may not be able to see all the displayed menu options in a glance. Because an operator usually searches a displayed menu for his intended menu option, placing the menu only near the perimeter of the display may increase menu search time, thus increasing menu option selection time.
One object of the invention is to indicate to an operator of a menu system having selectable regions outside the display, the menu option associated with each selectable region and the location of each selectable region.
Another object of the invention is to facilitate selection from a menu by an operator having impaired motor control.
Still another object of the invention is to speed selection of a menu option from a menu and of a submenu option from a menu hierarchy.
A further object of the invention is to speed speech synthesis for a person having impaired speech and impaired motor control.
I. Sound Match
Conventional speech recognition systems facilitate computer access for individuals unable to use a standard keyboard whose speech is relatively unimpaired, for example, an individual with quadriplegia, and hands-free computer access for able-bodied individuals. The operator of such a speech recognition system reads a menu option out loud, for example, "open file", and the system, which includes sound receiving means, for example, a microphone coupled to a sound board having a Digital Signal Processor ("DSP"), receives the sound of the read menu option, digitizes the sound of the read menu option, and then provides the digitized sound to another component of the speech recognition system, sound matching means which includes an application program for matching the digitized sound to one of a plurality of sounds, each representing respectively the sound of a spoken menu option. The system determines which sound best matches the sound of the read menu option and selects the menu option associated with this best matched sound.
Individuals whose speech is impaired are often unable to effectively use conventional speech recognition systems because they often cannot produce a large distinct variety of sounds characteristic of phonetic languages. For example, such an individual may produce similar sounds for the two consonants "t" and "d" so that these sound are indistinguishable to a conventional speech recognition system, or such an individual may not be able to consistently produce sounds distinguishable by a speech recognition system, resulting in false matches. Other symptoms of impaired speech, for example, similarities among certain phonemes and impaired ability to start or stop sound production appropriately, may substantially limit the variety of sounds distinguishable to a conventional speech recognition system an individual may consistently produce.
Conventional speech recognition systems provide limited capabilities in languages rich in homophones, for example, Chinese, because in such languages, a distinct sound is often insufficient to specify a word, as is described in the Background Art section of the Ideographic Languages aspect of the invention. The problem may be briefly illustrated by an example. Suppose a Chinese data entry operator using a conventional speech recognition system speaks the phonetic unit "fu" with a particular intonation. This distinct sound may well have over 15 homophones. Although the operator could use the keyboard to select one of these 15 homophones, this defeats the purpose of speech recognition, which is to facilitate hands-free computer access.
One object of the invention is to facilitate selection from a menu, and, in particular, from a menu of homophones.
Another object of the invention is to facilitate speech synthesis and voice activated computer access by individuals with speech impairments.
Still another object of the invention is to speed data entry in Ideographic Languages.
J. Ideographic Language
The use of ideographs as the graphic symbols in written languages is found in many parts of the world. An ideograph, as used herein, is a graphic symbol used to represent an object, an idea or a word, without expressing, as in a phonetic system, the specific sounds forming the verbal expression of the object, idea or word. Ideographic languages include Chinese, Japanese and Korean. A graphic symbol, as used herein, includes, but is not limited to, each of the following: a letter of an alphabet, a Japanese kana, and an ideograph. For purposes of illustrating the concepts of the present invention specific reference will be made herein to a preferred embodiment of the system and method as it applies to the Chinese language.
In modern Chinese, a repertoire of between 2500 and 3000 ideographs is necessary to achieve normal business adequacy in reading and writing, while the language itself has approximately 50,000 ideographs that have been identified historically, with about 10,000 ideographs in current use. The conventional keyboard, with approximately 100 keys, is designed for languages with phonetic scripts, such languages having a small set of graphic symbols, i.e. letters. If such a keyboard were to be used in a corresponding manner for the direct input of Chinese ideographs, it would require many thousands of keys since, unlike western phonetic languages, Chinese has many thousands of ideographs. Selection of an ideograph from such a keyboard would require the operator to search a great many keys for the desired key, and thus be impracticably slow.
Prior art methods for selecting Chinese ideographs make use of various ideograph classification systems known to Chinese speakers. The operator first specifies a class of ideograph, based on a first characteristic common to many ideographs. Ideographs having that common characteristic are displayed and the operator selects from among them, either directly, by selecting an individual ideograph, or indirectly, by specifying a second common characteristic usually dependent upon the first characteristic, thus further limiting the displayed ideographs to those having both the first and second common characteristics. In some prior art methods, the operator may continue to specify characteristics until he has specified a unique ideograph.
One ideograph classification system is called the Pin Yin System. This classification system uses the phonetic structure of the Chinese language. In spoken Chinese there are approximately 412 basic phonetic units, each having a monosyllabic sound, for example, "nee", "how" and "ma". Four intonations can potentially be applied to each phonetic unit, resulting in approximately 1280 distinct sounds. With 10,000 ideographs in current use, each represented by one of approximately 1280 distinct sounds, it is evident that many Chinese ideographs are homophones, i.e. have the same sound. Over 80% of Chinese ideographs have homophones. The Pin Yin System uses this limited number of phonetic units as the basis for its classification. Ideographs which are homophones are classified together; the common characteristic of the Pin Yin System is the distinct sound.
According to the Pin Yin and Zhu Yin coding methods, known in the prior art, the operator specifies a distinct sound using a keyboard labeled with symbols representing the Latin alphabet (Pin Yin method) or Chinese phonetic units (Zhu Yin method). The first key operation or sequence of key operations specifies the phonetic unit. The second key operation specifies the intonation. In general, less than 15 ideographs have this sound, though in some cases there are many more homophones. These are displayed and the user selects from among them. In such cases, the operator, depending upon the system, may page through matching ideographs or specify another common characteristic to further limit the number of ideographs displayed. A common characteristic which may be used at this stage exploits another feature of the Chinese language. The majority of Chinese words are expressed by a combination of two ideographs, the meaning of the paired ideographs has its own meaning which may or may not be related to that of the constituent ideographs. Assuming the operator has specified a first distinct sound matching 40 ideographs, he may specify a second distinct sound which alone may match, for example, 20 ideographs, but there may be only two ideograph pairs having the specified first and second distinct sounds in that order. Thus, a second common characteristic may limit matching ideograph pairs to a number sufficiently small for the operator to efficiently search and select from, or may uniquely specify an ideograph pair. Another common characteristic the operator may specify to limit the number of matching ideographs is a meaning or meaning class to which one or more sequences of one or more ideographs belong.
Yet another feature of the Chinese language which may be exploited to limit the number of matching sequences of ideographs is the ideograph block. An ideograph block is a sequence of four ideographs which together has its own meaning which may or may not be related to that of the constituent ideographs. As above, where the operator specified a distinct sound for the second of two ideographs of an ideograph pair, so may the operator specify a distinct sound for the second, third and/or fourth ideograph of an ideograph block, to limit the number of matching ideograph blocks.
Another conventional ideograph classification system makes use of a classification of parts of ideographs. Ideographs are built from a set of 214 components, called radicals. Different radicals, perhaps placed within different locations within an ideograph, are combined to create an ideograph. According to the Chan Jie coding method, known in the prior art, the operator specifies one or more radicals appearing in the ideograph he wishes to enter. He may, for example, use a keyboard having at least 214 keys, each corresponding to a radical, or may actuate a sequence of keys, the sequence corresponding to a radical. Other common characteristics the operator may specify to limit the number of matching ideographs include a phonetic unit, the first brush stroke, and the last brush stroke used to draw the ideograph.
Another conventional ideograph classification system makes use of a classification of parts of ideographs. According to the Four Corner coding method, known in the prior art, the operator specifies the classification of the four corners of the ideograph he wishes to enter. Other common characteristics the operator may specify to further limit the number of matching ideographs include the number of horizontal strokes used to draw the ideograph, and the classification of a certain part of the ideograph above the lower right corner.
Yet another conventional ideograph classification system makes use of a classification ideographs based on the basic strokes from which each ideograph is built. In Chinese, there are a limited number of basic strokes, each ideograph being composed of between 1 and 33 such strokes. Ideographs may be classified by a small number of basic strokes, preferably according to strict rules regarding the order of stroke entry. In one conventional application of this coding method, the operator specifies only the first and last basic strokes of the desired ideograph, then selects from a display of all ideographs sharing this first-last basic stroke combination.
Japanese is somewhat more complicated than Chinese. In addition to ideographs, the Japanese language uses graphic symbols called kana, which includes hiragana and katakana. In written Japanese, ideographs are frequently combined with kana. Kana may be may specified phonetically, for example, to designate the hiragana pronounced "ko" an operator of a Japanese word processing system may type "k" and then "o" on a Latin alphabetic keyboard or may type a single key associated with this hiragana. Kana has multiple uses in a Japanese word processing system. Kana may represent itself, since kana may stand alone in Japanese text. Alternatively, kana may be used to specify Japanese ideographs, either by specifying the radicals which compose Japanese ideographs or by specifying the pronunciation of Japanese ideographs. A sequence of phonetic units specified by kana may represent that sequence of kana, a single Japanese ideograph, multiple Japanese ideographs, or a combination of one or more Japanese ideographs and one or more kana. In addition, a single Japanese ideograph may have multiple pronunciations, including a Japanese pronunciation and a Chinese pronunciation, and may have multiple kana spellings.
Conventional word processing systems for ideographic languages suffer from certain deficiencies. First, in systems where the operator specifies common characteristics until he has uniquely specified an ideograph, the operator must be extensively trained in the particular classification system. Depending upon the system, the operator may need to know, for example, how may horizontal brush strokes are required to draw a desired ideograph, or each of the 214 radicals and the encoding of each of them on a keyboard having less than 214 keys. Second, in systems where the operator uses both hands on the keyboard to specify a common characteristic, then selects from among ideographs, ideograph pairs or ideograph blocks by operating a function key or by pointing to one of the options with a mouse or other hand operated pointer and then operating a switch, the operator lifts one of his hands from the keyboard, makes the selection and then moves his hand back to the keyboard to specify another common characteristic. This sequence occurs often and contributes to the slow average rate of word entry (approximately 20 words per minute) for Chinese relative to alphabetic languages. Another problem in these systems is that the display of ideographs for selection may obscure part of the image of the previously entered ideographs or other information on the display.
Another drawback of many word processing systems for ideographic languages relates to the ease of copying a document. Ideally, the operator concentrates on the document to be copied, only occasionally scanning text he has input. For those word processing systems that display ideographs on a display for the operator's selection, the operator must frequently shift his gaze from the document to the display and back again. The operator cannot concentrate on both the document and the display simultaneously.
Ideographs, as used herein, also include the symbols of symbol sets used for communication by individuals who have hearing, speech or language impairments, for teaching literacy skills to those lacking them, including pre-literate children and individuals with intellectual disabilities, and for international written communication. These symbol sets include, but are not limited to, each of the following: Picture Communication Symbols, Rebus, Picsym, Pictogram Ideogram Communication Symbols, Yerkish, Blissymbolics and depictions of the signs of a manual sign language. Examples of symbols of the Picture Communication Symbols, Rebus, Picsyms, and Blissymbolics symbol sets are shown in FIG. 11, Pictogram Ideogram Communication Symbols in FIGS. 12(a)-12(d) and Yerkish in FIGS. 13(a)-13(j). Picture Communication Symbols, Rebus, Picsyms, Pictogram Ideogram Communication Symbols, Yerkish, and Blissymbolics are each described in Beukelman, David R. & Mirenda, Pat, Augmentative and Alternative Communication. Management of Severe Communication Disorders in Children and Adults, Paul H. Brookes Publishing Co., 1992, pp. 22-29.
Individuals who have not acquired or who have lost their literacy skills may use symbolic symbol sets in learning to read. If the individual lacks fine motor control, for example, due to cerebral palsy, the individual's disability may inhibit the acquisition of literacy skills by, for example, inhibiting repetition of an exercise by the individual, by limiting the individual's ability to participate in the classroom, or by making skill assessment by a teacher difficult so that the teacher may incorrectly believe that remediation is necessary or that a particular skill has been mastered. If the individual also has impaired speech, literacy acquisition is more difficult still.
Conventional literacy training systems for individuals who are unable to use a standard keyboard or mouse may use switch access, often in combination with scanning. As already described, scanning is an extremely slow process. Moreover, as the number of symbols in the symbol set increases, the time required to select a symbol also increases. Of the symbol sets mentioned above, Picture Communication Symbols contains approximately 1800 symbols, Rebus contains approximately 800 symbols, Picsyms contains approximately 1800 symbols, Pictogram Ideogram Communication Symbols contains approximately 400 symbols and Blissymbolics contains approximately 1400 symbols. When using a system with a static display, the operator may expend considerable time and effort finding the desired symbol; when using a system with a dynamic display, the operator may expend considerable time effort memorizing and recalling where a particular symbol is located within a hierarchy of symbols. This time and effort generally does not contribute to the acquisition of literacy skills.
One object of the invention is to display a menu of sequences of one or more ideographs on a display so that a large contiguous area on the display is not obstructed by the menu.
Another object of the invention is to facilitate ideograph entry in word processing systems for the Chinese, Japanese and Korean languages.
Still another object of the invention is to speed selection of sequences of graphics including one or more ideographs.
Yet another object of the invention is to allow an operator of a word processing system for an ideographic language to select a sequence of one or more ideographs without lifting either hand from the keyboard.
A further object of the invention is to indicate to an operator the progress toward selection of a dwell-selectable sequence of one or more ideographic characters.
A still further object of the invention is to synthesize speech for an operator having impaired speech.
Additional objects, advantages and novel features of the invention will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or may be learned by practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.
SUMMARY OF THE INVENTION
According to the present invention, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The method includes several steps. The first step is displaying on a display the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with one of a plurality of displayed sound indicators. Each of the sound indicators respectively indicates one of a plurality of indicated sounds. Each of the indicated sounds differs from each other indicated sound. Another step is matching a received sound to any one of the indicated sounds. Yet another step is selecting the sequence associated with the matched indicated sound.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of selecting a sequence of one or more graphic symbols in a homophone-rich language from a plurality of sequences of one or more graphic symbols in the language. The method includes several steps. One step is displaying the plurality of sequences. Each of the plurality of sequences is associated respectively with a sound which is not a phonetic representation of the associated sequence. Each of the plurality of sounds differs from each other sound in the plurality of sounds. Another step is receiving a sound signal. Still another step is matching the received sound signal to any one of the plurality of sounds. Yet another step is selecting the sequence associated with the matched sound.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of editing a first sequence of two or more words in an ideographic language. The method includes several steps. One step is displaying on a display the first sequence. The first sequence includes a plurality of second sequences of one or more graphic symbols in the language. Each of the second sequences is associated respectively with a sound indicator. Each of the sound indicators respectively indicates a sound. Each of the indicated sounds differs from each of the other indicated sounds; and (b) the sound indicators. Another step is receiving a sound. Still another step is matching the received sound to any one of the indicated sounds. A further step is replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus, for use with a general purpose computer system including a display, for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols. One or more sequences of the plurality of sequences includes an ideograph in an ideographic language. The apparatus includes a computer readable medium, and a program, stored on the medium and executable by the general purpose computer system. The program is operative to display the plurality of sequences on the display. Each of the sequences is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The program is further operative to match a received sound to the sound indicated by the particular sound indicator, and to select the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The apparatus includes a display on which may be displayed the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The apparatus further includes a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The apparatus includes means for displaying the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each associated is respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The apparatus further includes means for: (1) matching a sound to the sound indicated by the particular sound indicator; and (2) selecting the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a device controller. The device controller includes a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The device controller further includes a signal generating device, coupled to a controlled device, for: (1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and (2) generating a device control signal corresponding to the function of the controlled device.
Still other objects and advantages of the present invention will become readily apparent to those skilled in this art from the following detailed descriptions, wherein I have shown and described the preferred embodiment of each aspect of the invention, simply by way of illustration of the best mode contemplated by me of carrying out each aspect of my invention. As will be realized, each aspect of the invention is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is an illustration of a display showing a conventional on-screen keyboard.
FIG. 2 is an illustration of a display showing a conventional on-screen keyboard and output from a word processing application program.
FIGS. 3, 4 and 5 are each illustrations of the display of each step of letter selection using a conventional quaternary on-screen keyboard.
FIG. 6 is an illustration of a display showing a conventional quaternary on-screen keyboard and output from two application programs.
FIG. 7 is an illustration of a conventional device implementing joystick patterns.
FIG. 8 is an illustration of a display of a conventional eye gaze system.
FIGS. 9 and 10 are each illustrations of a display showing a conventional menu.
FIG. 11 depicts examples of symbols from the Picture Communication Symbols, Rebus, PicSym and Blissymbols symbol sets.
FIGS. 12(a)-FIG. 12(d) depict examples of symbols from the Pictogram Ideogram Communication Symbols symbol set.
FIGS. 13(a)-FIG. 13(j) depict examples of symbols from the Yerkish symbol set.
FIG. 14 is a block diagram of a computer which may be utilized in accordance with the present invention.
FIG. 15 is a block diagram of a speech synthesis system which may be utilized in accordance with the present invention.
FIG. 16 is an illustration of software components of an apparatus in accordance with an embodiment of the Perimeter Menu aspect of the invention.
FIGS. 17 and 18 are each illustrations of a display and structures in accordance with an embodiment of the Perimeter Menu aspect of the invention.
FIG. 19 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIG. 20 is an illustration of a display and structures in accordance with still another embodiment of the Perimeter Menu aspect of the invention.
FIG. 21 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIG. 22 is an illustration of a display and structures in accordance with yet another embodiment of the Perimeter Menu aspect of the invention.
FIG. 23 is an illustration of an apparatus in accordance with a further embodiment of the Perimeter Menu aspect of the invention.
FIGS. 24 and 25 are each illustrations of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIGS. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter Menu aspect of the invention. FIG. 26 depicts a front view of the apparatus. FIG. 27 depicts a cut away view from the top of the apparatus.
FIG. 28 is a top view of a headrest in accordance with an embodiment of the Perimeter Menu aspect invention.
FIGS. 29 and 30 illustrate the state table aPocketFsm in accordance with the preferred embodiment of the Perimeter Menu aspect of the invention.
FIG. 31 is an illustration of a display and structures in accordance with the preferred embodiment of the Confinement aspect of the invention.
FIG. 32 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
FIG. 33 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
FIG. 34 is an illustration of an apparatus in accordance with the Dwell aspect of the invention and with the Path Directness aspect of the invention.
FIG. 35 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
FIG. 36 is an illustration of still another apparatus in accordance with the Dwell aspect of the invention.
FIG. 37 is an illustration of yet another apparatus in accordance with the Dwell aspect of the invention.
FIG. 38 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
FIG. 39 is an illustration of a display and structures in accordance with the preferred embodiment of the Path Directness aspect of the invention.
FIG. 40 is an illustration of a display and structures in accordance with an embodiment of the Path Directness aspect of the invention.
FIG. 41 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIG. 42 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIG. 43 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIGS. 44, 45 and 46 are each illustrations of a display and structures in accordance with the preferred embodiment of the Intersection aspect of the invention.
FIGS. 47 and 48 are each illustrations of a display and structures in accordance with another embodiment of the Intersection aspect of the invention.
FIGS. 49, 50 and 51 are each illustrations of a display and structures in accordance with the preferred embodiment of the Alignment aspect of the invention.
FIG. 52 is an illustration of a display and structures in accordance with the preferred embodiment of the Location Indication and the Length Order aspects of the invention.
FIG. 53 is an illustration of a display and structures in accordance with an embodiment of the Location Indication aspect of the invention.
FIGS. 54 and 55 are each illustrations of a display and structures in accordance with another embodiment of the Location Indication aspect of the invention.
FIG. 56 is an illustration of a display and structures in accordance with a still further embodiment of the Location Indication aspect of the invention.
FIG. 57 is an illustration of a display and structures in accordance with a further embodiment of the Location Indication aspect of the invention.
FIGS. 58 and 59 are illustrations of a display and structures in accordance with the preferred embodiment of the Sound Match aspect of the invention.
FIG. 60 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
FIG. 61 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
FIG. 62 is a block diagram of a speech recognition system which may be utilized in accordance with the Sound Match aspect of the invention.
FIG. 63 is an illustration of software components of an apparatus in accordance with an embodiment of the Sound Match aspect of the invention.
FIG. 64 is an illustration of a display and structures in accordance with the preferred embodiment of the Ideographic Language aspect of the invention.
FIG. 65 is an illustration of a display and structures in accordance with another embodiment of the Ideographic Language aspect of the invention.
FIG. 66 is an illustration of a display and structures in accordance with still another embodiment of the Ideographic Language aspect of the invention.
FIG. 67 is an illustration of a display and structures in accordance with yet another embodiment of the Ideographic Language aspect of the invention.
BEST MODE FOR CARRYING OUT THE INVENTION
The hardware and software operating environment of the preferred embodiment of all aspects of the invention will now be described with reference to a particular embodiment of the invention, hereinafter "prototype". The prototype of the invention illustrates the best mode of practicing each aspect of the invention known by me except where a preferred mode is described.
FIGS. 15 depicts a block diagram of the hardware components of the prototype (2214), including a conventional general purpose computer system (2218), an optional pointer (2202), an optional printer (2220) and a speech synthesizer (2206). The general purpose computer system (2218) includes a conventional computer system (2116), a storage unit (2208), a keyboard (2210), and a diskette drive (2216). FIG. 14 depicts a block diagram of the conventional computer system (2116), including a processing unit (2102) and a display (2112). The processing unit (2102) includes a processor (2104), a memory (2106) and control circuitry (2108). The prototype employs the Toshiba T6400DXC general purpose computer system manufactured by Toshiba Corporation, Kawasaki, Japan. However, the T6400DXC is preferably substituted with the IBM ThinkPad 755C computer system, part number 9545F0C, manufactured by IBM Corporation, Armonk, N.Y., USA., because the former requires a 110VAC power source while the latter is powered by an integral battery. An integral battery allows an NMD operator to use the system when a 100VAC source is not available or when attaching to a 110VAC power source is inconvenient. The prototype further includes a head mounted pointer communicating via an infrared link with the computer system so that there are no cables tethering the operator to the computer system. Any cable between the operator and the computer system would have to be connected, probably by an attendant since the operator may lack the fine motor skills required to make such a connection. Preferably computer access can be accomplished independently by the operator. The choice of a pointing device is primarily dictated by the particular capabilities of the operator. Usually the best pointing device for a particular operator is the one drawing on that operator's best motor control. For example, if an operator's foot control is superior to his head control, a pointing device using his foot is preferably to a head pointer.
The prototype employs the Remote Headmaster® manufactured by the Prentke Romich Company, Wooster, Ohio, USA. However, the combination of the HeadMaster® Plus, part number HM-1P, HeadMaster® Plus Remote Adapter, part number HM-RA, and HeadMaster® Plus Laptop Adapter, part number HM-LA, all available from the Prentke Romich Company, is preferable because the headset is more comfortable and the HeadMaster® Plus ultrasonic transmitter mounts more easily on a laptop computer system than the Remote Headmaster ultrasonic transmitter.
The preferred embodiment further includes a battery powered printer, the MobileWriter®, part number 730879, manufactured by Mannessmann Tally Corporation, Kent, Wash., USA and a speech synthesizer, the Multivoice Speech Synthesizer, part number MV2-SS, manufactured by The Institute on Applied Technology, Children's Hospital, Boston, Mass., USA.
The pointer (2202) is a device which provides data concerning the relative or absolute position of the operator or any body member of the operator. The display (2112) and pointer (2202) together provide for the interactive nature of the general purpose computer system (2218) in that, in accord with the various aspects of the invention, the interpretation that the processor (2104) gives to a certain pointer action made by the operator depends, in the majority of situations, upon what is being displayed to the operator at that time.
The prototype (2214) shown in FIG. 15 further includes a keyboard (2210), which functions to provide input from an able-bodied operator to the general purpose computer system (2218). The keyboard (2210) is useful for configuration, diagnostic and backup purposes, functions which are performed relatively infrequently and usually require an able-bodied person for ancillary activities, for example, loading backup media into the general purpose computer system. The prototype (2214) also optionally includes a printer (2220) which functions to provide hard copy output of data developed or stored in the general purpose computer system, and a speech synthesizer (2206), which functions to provide speech output for utterances and words composed using or retrieved from the general purpose computer system (2218).
The couplings between the devices depicted in FIG. 15 may be made by any means which permits the orderly and timely exchange of data across the interface. In the preferred embodiment, the interfaces between the pointer (2202) and the general purpose computer system (2218) and between the general purpose computer system (2218) and the speech synthesizer (2206) conform to the Electronic Industries Association RS-232 interface specification. The interface between the general purpose computer system (2218) and the printer (2220) conform to the Centronix 50 pin parallel interface specification.
The software component of the prototype are stored in memory (2106) and executed on the processing unit (2102). The software component of the prototype, depicted in FIG. 16, include a software driver (1202), an operating system (1204), an optional database program (1210), and the prototype access program code and data, hereinafter collectively referred to as the "access program" (1206). In the preferred embodiment, one or more application programs (1208) may also execute on the processing unit (2102) and accept control and data from the access program (1206) via the operating system (1204). The software driver (1202) of the prototype is the Logitech Mouse Driver included with Windows® version 3.1. The operating system (1204) of the prototype is Windows® version 3.1 in combination with MS-DOS® version 6.2. Hereinafter, the operating system is referred to simply as "Windows®", available from Microsoft Corporation, Redmond, Wash., USA.
The optional database program (1210) is described in the detailed description of the Length Order aspect of the invention. The prototype access program (1206) is described in detail below.
As stated earlier, the software components of the prototype are stored in memory (2106). Depending on the capacity of memory (2106) and the size of the application programs, portions of these programs may be transferred as needed between memory (2106) and the storage unit (2206) or between memory (2106) and a diskette in the diskette drive (2216) depicted in FIG. 15. The basic function of the storage unit (2206) and the diskette drive (2216) is to store programs and data that are employed by the general purpose computer system (2218) and which may readily be transferred to the memory (2106) when needed.
It is to be understood that components others than those used in the prototype may be utilized in accordance with the invention. It is only necessary that the substitute component or components have the capacity to carry out the functions described. For example, the processing unit of the general purpose computer system may be substituted with a microprocessor coupled to custom electronics for performing the functions of the various aspects of the invention, or the color display of the prototype may be substituted with a monochrome display.
A. Perimeter Menu
The preferred embodiment of the Perimeter Menu aspect of the invention will now be described in detail from a functional perspective using an example. This description refers to selectable regions which include one or more subregions. A selectable region is a region, delimited with respect to a display or a surface, and associated with a menu option which may be selected, usually by a selection event. A subregion is a selectable region that is included within another selectable region. Thus a subregion is, by itself, a selectable region. Assuming that a certain selectable region includes subregions A and B, dwell time on subregions A and B may be combined, for example, by summing, so that dwelling on either subregion A or B or a combination of both for the selection threshold period selects the menu option associated with the selectable region. Reference will now be made to FIGS. 17 and 18 which depict an example of the preferred embodiment of the Perimeter Menu aspect of the invention. FIG. 17 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) and eight selectable regions. Each of the eight selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion located outside the display (2112) and adjacent the visible subregion. For example, the selectable region at 11 o'clock in FIG. 17 labeled with menu option "vort<space>x" consists of invisible subregion (0104) and visible subregion (0106), and within this description of the Perimeter Menu aspect of the invention is referred to as selectable region (0104/0106). The other selectable regions shown in FIG. 17, proceeding counter clockwise from selectable region (0104/0106) are (0108/0110), (0112/0114), (0116/0118), (0120/0122), (0124/0126), (0128/0130) and (0132/0134). Each subregion may be sized to suit the operator's preferences and abilities. Each selectable region is associated respectively with a menu option. In FIG. 17, selectable region (0104/0106) is associated with menu option vort<space>x, selectable region (0108/0110) with menu option "sumac", selectable region (0112/0114) with menu option "wizen'", selectable region (0116/0118) with the menu option undo indicated by an icon on visible subregion (0118) representing an undo function, selectable region (0120/0122) with menu option "words", selectable region (0124/0126) with menu option "talk", selectable region (0128/0130) with menu option "ldhbfk" and selectable region (0132/0134) with menu option "ypgqj,". Together, the eight visible subregions circumscribe region (0150) on the display.
Selectable regions may be delimited by data indicative of one or more boundaries of the selectable region. Equivalently, the delimit means may be detectors operative to determine when the location indicated by the movement related signal has crossed one of those boundaries or intersects a selectable region. A partially delimited region or subregion is one which is unbounded on at least one side.
Resuming, now, with the example, to select a menu option associated with a selectable region the operator moves a pointer (2202 in FIG. 15) coupled to the general purpose computer system (2218 in FIG. 15) to indicate a location on the selectable region, including either subregion, associated with the desired menu option and maintains the indicated location on the selectable region for the selection threshold period. The period of time required for selection may vary responsive to the proximity of the indicated location to the location of a cursor on the display or to the proximity of the indicated location to a point within the intersected selectable region. Dwell time may be continuous, discontinuous or dynamic (described below) for either or both subregions of the selectable region.
Selection in the above example is in response to a dwell event. A dwell event includes, but is not limited to, each of the following: (a) the durations of one or more periods of intersection of locations indicated by a movement related signal, a body member or a cursor (including any part of the cursor) and a selectable region equalling or exceeding a predetermined period; (b) a first quantity responsive to the durations of the periods referred to in (a) equalling or exceeding a predetermined quantity; (c) dwell event (a) or (b) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the intersected selectable region; and (d) dwell event (a) or (b) wherein the period of intersection required for selection of a selectable region increases in response to a non-intersection or a period of non-intersection of locations indicated by the movement related signal, the body member or the cursor and the selectable region ("dynamic dwell event"). The use of non-intersection or a period of non-intersection in determining the duration of a period of intersection required for selection is called dynamic dwell. Associated with each type of dwell event is an intersected selectable region. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor which triggers the dwell event by causing the period or the first quantity to equal or exceed the predetermined period or the predetermined quantity, respectively.
Selection may also be in response to a selection event. A selection event includes, but is not limited to: (a) a dwell event; (b) a switch operation at or near the time of an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; (c) an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; and (d) selection event (c) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the selectable region it previously intersected. Associated with each type of selection event is an intersected selectable region. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor. The fact that a selection event has occurred may be indicated to the operator, for example, visually by changing the cursor appearance or location, by changing location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font or fill of an object on the surface, or by displaying a graphic or a point distinguishable from its immediate surroundings on a surface or removing a graphic or point distinguishable from its immediate surroundings from a surface; auditively by generating a sound or changing the pitch or volume of an extant sound; tactilely by changing the surface or temperature of a contact area or the pressure exerted by a contact area; or by other means. In the prototype, following selection, the hue of the visible subregion of the selected selectable region is changed from green to magenta.
As used herein, a cursor includes a temporary marking on a display which emphasizes to an operator, in an optical manner, a momentarily important location or object. As used herein, body member means any part of the body including, but not limited to, each of the following: the shoulder, arm, elbow, wrist, hand, finger, thumb, leg, knee, ankle, foot, toe, hip, trunk, neck, tongue, lip, eye and head. The received movement related signal includes, but is not limited to, a signal indicative of movement or from which movement can be derived, such as a plurality of relative or absolute positions or a difference between two relative or absolute positions. Movement related signal receiving means includes, but is not limited to, each of the following: (a) pointer interface circuitry found in a general purpose computer system; (b) one or more detectors operative to detect movement of a pointer; and (c) one or more detectors operative to detect movement of a body member of an operator. In the prototype, the movement related signal receiving includes electronic circuitry in the general purpose computer system (2218) operative to receive the movement related signal generated in part by the movement of the pointer (2202).
In the prototype subregions are displayed on the display (2112). However, other means for displaying may be substituted for the means used in the prototype, for example, a projector for projecting an image, a surface having a static display thereon, or other suitable means.
Resuming, now, with the example of FIG. 17, and assuming that the operator has selected menu option "vort<space>x", the display is changed to that shown in FIG. 18. In FIG. 18, each of six selectable regions is now associated with a submenu option of the selected menu option "vort<space>x". Selectable region (0104/0106) is now associated with submenu option "<space>", selectable region (0108/0110) with submenu option "o", selectable region (0112/0114) with submenu option "t", selectable region (0124/0126) with submenu option "x", selectable region (0128/0130) with submenu option "v", and selectable region (0132/0134) with submenu option "r". Selectable regions (0116/0118) and (0120/0122) remains associated with the same menu options with which each was associated in FIG. 17. The operator may now select one of these submenu options.
Assuming that the selected submenu option is one of v, o, r, t, x and space, the selected character, or a corresponding computer encoding of that character, may be input to an apparatus coupled to the general purpose computer system (2218 in FIG. 15), or input to an application program (1208) executing on the general purpose computer system (2218) coupled to the display (2112). Inputting, as used herein, includes, but is not limited to, generating or passing signals representative of the selected menu option along a path toward the destination apparatus or program. Preferably, the computer program displays at least some of its output in the circumscribed region (0150).
Given a display having eight selectable regions, an operator may, with a single selection indicate one of eight menu options, with two selections indicate one of up to 64 different menu options, with three selections indicate one of up to 256 menu options, etc. Each of these menu options may represent a sequence of one or more characters, a sequence of one or more data or control inputs to an application program (1208), or a control function for one or more devices or speech synthesizers coupled to the general purpose computer system (2218). As used herein, a character includes a space, a control character as defined by the American National Standards Institute (ANSI) or the American Standard Code for Information Exchange (ASCII), and a letter from one of the Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Assyrian, Avar, Azerbaijani, Balinese, Bamara, Bantu, Bashkir, Basque, Bengali, Birhari, Bulgarian, Buluba-Lulua, Burmese, Buryat, Byelorussian, Caddoan, Catalan, Chechen, Chikaranga, Chippewa, Choctaw, Church Slavik, Chuvash, Coptic, Cree, Croatian, Cyrillic, Czech, Dakota, Danish, Dari, Devanagari, Dutch, Dzongkha, English, Eskimo, Esperanto, Estonian, Ewe, Farsi, Fijian, Filipino, Finnish, Flemish, French, Fulani, Gaelic, Galician, Georgian, German, Greek, Gujarati, Gurmakhi, Harari, Hausa, Hawaiian, Hebrew, Hindi, Hiragana, Ibo, Icelandic, Indonesian, Irish, Irogquoian, Italian, Kabardian, Kalmyk, Kannada, Kanuri, Kashmiri, Katakana, Kazakh, Khasi, Khmer, Kirghiz, Kishmiri, Komi, Kongo, Kurdish, Lao, Latin, Latvian, Lithuanian, Lu-Guanda, Macedonian, Magahi, Maithili, Makua, Malagasy, Malay, Malayalam, Maltese, Mandingo, Manipuri, Marathi, Masai, Mizo, Moldavian, Mongolian, Munda, Naga, Navaho, Nyanja, Nepalese, Norwegian, Oriya, Oromo, Ossetian, Pashto, Polish, Portugese, Punjabi, Rajasthani, Rhaeto-Romanic, Rumanian, Russian, Samoan, Sangs, Serbian, Serbo-Croatian, Sinhalese, Sinhi, Sioux, Slovak, Slovenia, Spanish, Sundanese, Swahili, Swedish, Syriac, Tadzhik, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkish, Turkmen, Udmurt, Uighur, Ukranian, Umbundu, Urdu, Uzbek, Vietnamese, Visayan, Welsh, Yakut, Yoruba and phonetic alphabets. As used herein, each of a character, ideograph, control input and control function includes a computer encoding of the same. As used herein, a device includes, but is not limited to, each of a wheelchair, a household appliance, an appliance for use in an office, a workstation, a robot, and a computer peripheral. Thus, by selecting from a menu, the operator may, for example, increase the volume of an external speech synthesizer, or turn a wheelchair to the left.
The selectable regions organized as described above help an NMD operator make the menu selection he intends. Referring to FIG. 17, suppose, for example, an NMD operator intends to move a pointer (2202) that is indicating point (0154) to indicate point (0156), a location in subregion (0130), but who is unable to quickly stop motion, so that the location indicated by the pointer (2202) moves from point (0154) past point (0156) to point (0158). Because point (0158) lies within the same selectable region (0128/0130) as the overshot subregion (0130), dwelling at point (0158) operates to select the intended selectable region (0128/0130). Invisible subregions, in accordance with the Perimeter Menu aspect of the invention, may extend outward from the edge of the display (2112) to infinity. In such an embodiment, dwelling at point (0162) would operate to select selectable region (0128/0130). Preferably, invisible subregions extend a finite distance from the edge of the display (2112). In such an embodiment, dwelling at point (0162) would not operate to select selectable region (0128/0130). The sizes of the invisible subregions shown in FIGS. 17 and 18 are illustrative only. Preferably the size of each invisible subregion is large enough to encompass overshoot but small enough to avoid unintentional selections when the NMD operator turns to see someone or something.
The prototype utilizes the area outside the display to facilitate menu selection by a disabled operator. If an operator has impaired ability to maintain a steady position, he can point to a relatively large invisible subregion outside the display which is more forgiving of the operator's involuntary motion than the relatively small selectable regions on the display in conventional on-screen keyboards. Thus, the effective area of a selectable region is expanded beyond the region's visible subregion shown on a display. If an operator has impaired ability to stop motion he may, starting from the center of the screen, point to any selectable region. His impaired ability to stop will not impair his ability to select his intended target, assuming his directional control is relatively unimpaired, since, in the prototype, each selectable region is unbounded on its side furthest from the center of the display. Although the selectable regions of the prototype are large, only the visible subregion of each selectable region uses space on the display. Thus a large rectangular region remains available on the display for the output of an application program. The Perimeter Menu aspect of the invention is preferably implemented on a general purpose computer system. If the general purpose computer system is coupled to a speech synthesizer and the menu hierarchy allows the selection of letters and/or words, an operator having impaired speech may speak using the speech synthesizer. If a word processing or data entry application program is run on the general purpose computer system, the operator may enter words or data, respectively, for input to the application program. If the general purpose computer system is coupled to a devices capable of executing commands and the menu hierarchy allows the selection of commands, a disabled operator may select and issue commands to control these devices.
Since the cognitive demand for scanning is greater than that for direct selection, the prototype places less cognitive demand on the operator than a scanning system for selecting options from a menu.
FIG. 19 illustrates a display in accordance with an alternative embodiment of the Perimeter Menu aspect of the invention having eight selectable regions circumscribing a central region (6950) on the display.
FIG. 20 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having twenty selectable regions circumscribing a central region (0806) on the display.
FIG. 21 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having four selectable regions (0508), (0506), (0504) and (0502) circumscribing a central region (0510) on the display.
FIG. 22 illustrates a display in accordance with an embodiment of the Perimeter Menu aspect of the invention having four selectable regions (4008), (4006), (4004) and (4002) with no space between them other than a circumscribed region (4010) on the display.
FIG. 23 illustrates an apparatus in accordance with of another embodiment of the Perimeter Menu aspect of the invention. In FIG. 23, region (3510) is located on interior display (3514) which is circumscribed by peripheral display (3512). Selectable regions (3508), (3535), (3504) and (3502) are located on the peripheral display (3512).
FIGS. 24 and 25 are each illustrations of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention. FIG. 24 depicts ten selectable regions (6502), (6504), (6506), (6508), (6510), (6512), (6514), (6514), (6516), (6518), and (6520). Each selectable region is located on the display (2112) adjacent the edge of the display and associated respectively with a menu option. In FIG. 24, the menu options are shown on their associated selectable region. Together the ten selectable region circumscribe region (6550) on the display. In response to only an intersection of a location indicated by a movement related signal and selectable region (6506), the display changes to that shown in FIG. 25, on which are located ten selectable regions each located on the display (2112) adjacent the edge of the display, nine of the ten selectable regions associated respectively with a submenu option. Selectable region (6608) is not associated with a submenu option. Any submenu option may be selected by a selection event.
FIGS. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter Menu aspect of the invention. FIG. 26 depicts a front view of the apparatus; FIG. 27 a cut away view from the top of the apparatus. FIG. 26 depicts detector area (0348 in FIG. 27) on, in or below which are located a plurality of selectable regions (0304), (0306), (0308), (0310), (0312), (0314), (0316), (0318), (0320), (0322), (0324), (0326), (0328), (0330), (0332), (0334), (0336), (0338), (0340) and (0342). Adjacent the detector area (0348 in FIG. 27) is a berm (0350 in both FIG. 26 and 27) for confining a body member of the operator or a pointer controlled by the operator to the detector area (0348 in FIG. 27).
Still another apparatus in accordance with the Perimeter Menu aspect of the invention is illustrated in FIG. 28 which depicts a headrest for an operator using his head to indicate a location on a display. FIG. 28 shows an irregularity (0703) on the surface (0701) of the headrest. The irregularity is in physical contact with the operator and tactilely indicates to him the position of his head. The tactile indication means may be concave, convex or both or may differ from the surface in temperature. For individuals having impaired ability to sense the position of a body member, e.g. the operator's head, the tactile input thus provided to the operator improves the operator's ability to sense the position of his head.
The prototype of the invention will now be described in detail and where the preferred mode of practicing the invention differs from the prototype, the preferred mode is described. The description is broken into several parts:
1. A brief overview of how a state table works.
2. A description of the operation of the events, state table and state processing used in the prototype.
3. A general description of each event and one example of the use of that event.
4. An example of state machine processing in the operation of the prototype.
The prototype implements the Perimeter Menu aspect of the invention as a state table. A state table is a tool for processing sequential inputs and is most easily understood by analogy. Imagine yourself in a room having a ticket window and three exits, each regulated by a turnstile. You collect a ticket at the ticket window which, when inserted into the appropriate turnstile, allows passage to a connecting room. The turnstile keeps the ticket. Any given ticket operates only one turnstile in a room, though different tickets may operate the same turnstile. You begin in a certain room, collect a ticket, insert it into the appropriate turnstile and pass to a connecting room, where you perform certain tasks associated with the new room. Then you collect another ticket from the ticket window in that room, insert the ticket into the appropriate turnstile in that room, pass to a connecting room, perform certain tasks associated with the new room, and so on.
The state table used in the prototype is depicted in FIGS. 29 and 30. Moving from the analogy above, the rooms are states represented by the rows of the state table; the tickets are events represented by the columns of the state table. Each entry in the state table represents a passage from one state to another. The tasks performed upon entry into a room correspond to the processing performed by the processor (2104) on entry to a new state ("state processing"). For example, state processing may cause a slight lightening of the color of a selectable region. Reentry into that state five times may successively lighten a selectable region five times.
In the prototype, each selectable region is directly controlled by one associated state machine. Each state machine directly controls only its associated selectable region. Each state machine includes data uniquely associated with its associated selectable region, the shared state table shown in FIGS. 29 and 30, and the shared code for state processing, described below. This embodiment means that the state machine associated with selectable region (1614) may be in state ST-- SELECTED while the state machine associated with selectable region (1602) is in state ST-- INITIAL. In the prototype, these separate states are reflected only in separate values for data uniquely associated with each state machine. Each state machine has a unique index.
The state table used in the prototype defines 18 states, composed of states zero through seventeen shown in FIGS. 29 and 30. State 1 (ST-- ERROR-- STATE in FIG. 29) is not used. Preferably, it is omitted. In the prototype, the differences between the several state machines, for example, the state of a particular state machine at any given time, are confined to data structures associated with that state machine. The state machines share the same code and the same state table. In other words, returning to the ticket and turnstile analogy, there are multiple travelers each with his own ticket (event) and his own baggage (data) moving from room to room in the same labyrinth. Each traveler's actions in each room usually affect only his own baggage.
In the prototype, there are two types of sequential inputs processed by the state table: external events and internal events. External events are generated outside the state table, for example, by the operator moving the pointer (2202) or by a timer expiring. Pointer movement may generate an event indicating that the operator has moved the cursor across the selectable region boundary from without the selectable region to within it. This event causes a transition from one state to another ("drives" a state machine to a new state). For example, assuming a state machine is in state ST-- CREST-- TIDE, row 6 in the state table shown in FIG. 29, when an event EV-- CROSS-- OUT, column 4 of the state table, occurs. At the intersection of row 6 and column 4 is a 7. This represents the new state, row 7 of the state table, state ST-- SELECTED. Thus event EV-- CROSS-- OUT drives the state machine from state ST-- CREST-- TIDE to state ST-- SELECTED. On entry to the new state, the computer performs the state processing associated with the new state.
Internal events are generated during state processing to handle circumstances where a first state transition is made due to an external event and the processing associated with the new state determines that a second state transition is necessary. The first, external, event has already been used so a second, internal, event is generated by the state. For example, if the operator has a prolonged muscle spasm, common among individuals with CP, or loses his grip on a hand held pointing device, the cursor may sit without moving on a selectable region for a considerable period of time. It is desirable to detect this condition, move the cursor to the center of the screen so the operator can easily find the cursor, and reset all selectable regions to their initial color. Detection is accomplished with a timer. On timer expiration, a state machine transitions to state ST-- IDLE which centers the cursor. From there it is desirable to transition to state ST-- RESET which, among other processing, initializes selectable region color. The transition from state ST-- IDLE to state ST-- RESET is driven by an internal event, generated by state ST-- IDLE state processing.
Although the preferred embodiment uses only one state to perform certain state processing, that state processing may be equivalently performed in multiple states. Likewise, the state processing of multiple states of the described embodiment may be equivalently performed in a single state.
Although the prototype uses a state table to control the flow of program execution and to select one of a plurality of selectable regions, the same function may be equivalently performed using object oriented software architecture, if-then-else statements or a combination of these. In particular, in accord with object oriented software architecture, each state machine may be equivalently represented as an instantiation of a selectable region class for processing inputs affecting a particular selectable region.
Although the described embodiment uses a single processor, state table and code for state processing for all the selectable regions, each of these may be duplicated. Alternate embodiments may include processors, electronic circuitry, state tables or code for state processing used for processing fewer than all the selectable regions or used to process certain selectable regions at one time and other selectable regions at other times.
In the prototype, each state machine processes events independently of all other state machines, though a state machine may send an event to another state machine. For example, when the operator selects a selectable region, the associated state machine sends the event EV-- RESET to all other state machines so that all selectable regions revert to their respective initial colors.
A single operator action may result in the issuance of different events to different state machines. For example, when the operator moves the cursor from without a selectable region to within it, EV-- DWELL is sent to the newly indicated state machine. EV-- MOVEMENT is sent to all other state machines.
Following is a general description of each event used in the prototype and an example of the use of each event. For all uses of each event, refer to the state table shown in FIGS. 29 and 30 which determines what state transition occurs from every state on occurrence of a given event.
The event EV-- RESET is an internal event which drives a state machine to its initial state. For example, when the operator has not moved the pointer (2202) for a predetermined period of time, event EV-- RESET is sent to all other selectable regions.
The event EV-- DECAY is an external event which indicates that the cursor hotspot does not intersect the selectable region associated with the state machine. EV-- DECAY is sent to a state machine periodically when the operator has positioned the cursor hotspot on a selectable region other than the selectable region associated with that state machine.
The event EV-- DWELL is an external event which indicates that the cursor hotspot intersects the selectable region associated with the state machine. EV-- DWELL is sent to a state machine periodically when the operator has positioned the cursor hotspot on the associated selectable region.
The event EV-- CROSS-- OUT is an external event which indicates that the cursor hotspot has moved from a location intersecting the selectable region associated with the state machine to a location not intersecting the selectable region. After the operator selects a selectable region, he must move the cursor hotspot out of the selectable region, generating EV-- CROSS-- OUT, before he can again select that selectable region.
The event EV-- STEP-- UP is an internal event which indicates that a selectable region's selection threshold has been satisfied.
The event EV-- MOVEMENT is an external event which indicates that the cursor hotspot has moved. If the cursor hotspot intersects a selectable region without moving for a predetermined period of time, a timeout occurs, causing all state machines to transition to the reset state. EV-- MOVEMENT drives the state machine out of the reset state.
The event EV-- IDLE-- TIMEOUT is an external event which indicates that the cursor hotspot has intersected a selectable region without moving for a predetermined period of time. EV-- IDLE-- TIMEOUT causes the state machine to move the cursor hotspot to the center of the display.
The event EV-- CEILING is an external event which indicates that the cursor hotspot intersects a selectable region and the color of the selectable region equals the selectable region color ceiling. If the locking feature is enabled, EV-- CEILING drives the state machine to the begin lock state where it displays the lock icon.
The null event, EV-- NULL, is a multi-purpose internal event used in a variety of situations to drive a state machine to another state. For example, after a timeout has been detected, EV-- IDLE-- TIMEOUT is generated and sent to the appropriate state machine driving it to the idle state, the receiving state machine sends itself EV-- NULL in order to drive itself to the reset state. The use of EV-- NULL here allows states to be simpler and the reset state to be reused.
The prototype uses eight partially delimited selectable regions. In the description below, the portion of each selectable region shown on the display is referred to as the visible subregion of the selectable region. The portion of each selectable region outside the display is referred to as the invisible subregion of the selectable region. Because the software driver (1202) confines the cursor hotspot to the Windows® cursor clipping rectangle, a rectangle on the display slightly smaller in area than the display, the access program (1206) of the prototype only reads hotspot cursor locations within the Windows® cursor clipping rectangle, even though the operator may in fact be pointing to a location outside the Windows® cursor clipping rectangle, e.g. within an invisible subregion. Thus the access program (1206) does not distinguish between two locations indicated by the operator, the first at a first location on the edge of the Windows® cursor clipping rectangle and the second outside the Windows® cursor clipping rectangle whose location is reported by the software driver (1202) to be the first location. For example, assuming that the operator moves the location indicated by the pointer (2202) to a location within invisible subregion (0104 in FIG. 17), the software driver (1202) in the prototype reports the cursor hotspot location to be the closest point within the visible subregion (0106). Consequently, in the prototype, all invisible subregions are unbounded on their side furthest from and parallel to the edge of the display. Thus, in the prototype, point (0162) in FIG. 17 lies within selectable region (0128/0130) since the rightmost side of selectable region (0128/0130) is unbounded.
The operation of the prototype will now be described using, as an example, the selection of a menu option associated with selectable region (0104/0106). First described are notation conventions used in the description, then initialization in the prototype, and then the example. The description refers to the procedures PocketFsm and CreateEvent. These procedures are listed in pseudo-code in Appendix I.
Notational conventions used in the description below:
1. pPocket→indicates a set of data associated with a particular state machine, in this example, the state machine associated with selectable region (0104/0106).
2. pPocket→State indicates a particular item of data within the set of data associated with the state machine, in this case, the variable "State".
During initialization:
1. The Windows® cursor clipping rectangle is set so that most of the arrow cursor is always visible on the display.
2. The cursor, indicating on the display (2112) the location indicated by the movement related signal receiving means, is positioned at the center of the display.
3. All data associated with each state machine are initialized. The following variables are included in the set of data associated with each state machine. There are as many independent copies of these variables as there are state machines. For each state machine the following variables are initialized as indicated.
__________________________________________________________________________
variable  initial value    meaning                                        
__________________________________________________________________________
fInvert   FALSE            if TRUE, display the color complementary to    
                           that                                           
                           indicated by this selectable region's Color    
                           variable                                       
State     ST.sub.-- INITIAL                                               
                           state of the state machine                     
PreviousState                                                             
          0                previous state of the state machine            
Color     0                RGB encoding of visible subregion color        
fPaint    FALSE            if TRUE, paint visible subregion on receipt    
                           of                                             
                           WM.sub.-- PAINT message                        
InitialColor                                                              
          RGB (0,32,0)     initial value corresponds to a dark green      
pLabel    initial menu option                                             
                           indicates an element within the                
          for the state machine's                                         
                           aLabel array, described beIow                  
          selectable region                                               
Decrement RGB (0,1,0)      value subtracted from Color on EV.sub.--       
                           DECAY                                          
Increment RGB (0,4,0)      value added to Color on EV.sub.-- DWELL        
Ceiling   RGB (0,255,0)    initial value corresponds to a very light,     
                           bright green                                   
CrestTide RGB (0,143,0)    initial value corresponds to a                 
                           light green                                    
Lockspot.x                                                                
          a point within the                                              
                           x coordinate of location for the               
          selectable region                                               
                           display ofthe lock icon                        
          located two thirds of                                           
          the length of the                                               
          selectable region from                                          
          the closest corner of                                           
          the display                                                     
Lockspot.y                                                                
          a point within the                                              
                           y coordinate of location for the               
          selectable region                                               
                           display of the lock icon                       
          located two thirds of                                           
          the length of the                                               
          selectable region from                                          
          the closest corner of                                           
          the display                                                     
fInterior TRUE for interior                                               
                           if TRUE, this is an interior region            
          regions, otherwise                                              
                           of adjacent regions                            
          FALSE; used for                                                 
          the Intersection                                                
          aspect of the invention                                         
iAdjacentPocket                                                           
          index of state machine associated                               
          with the adjacent region; used for                              
          the Intersection                                                
          aspect of the invention                                         
first array of points                                                     
          boundaries of the                                               
          visible subregion                                               
second array of points                                                    
          boundaries of the                                               
          region; used for the Intersection                               
          aspect of the invention                                         
hRegion   handle to Windows ® region                                  
          corresponding to the visible subregion;                         
          used for the Intersection                                       
          aspect of the invention                                         
__________________________________________________________________________
4. aLabel, an array of data structures defining the menu and submenu options and the menu hierarchy, is initialized. For example, one of the elements of aLabel defines menu option "vort<space>x". This element includes fields which determine that this menu option is displayed horizontally starting at certain (x,y) coordinates, that on selection certain actions are to be taken, for example, outputting text to a speech synthesizer, and that on selection certain submenu options, in this example, "v", "o", "r", "t", "<space>" and "x", and related data are to be associated with certain state machines. In the prototype, this association is accomplished by modifying pLabel in the set of data of the associated state machine to point to the aLabel element corresponding to the menu option to be associated with that state machine.
5. At least one window is created in the circumscribed region (0150) for the display of selected letters.
6. The state table is initialized to the values shown in FIGS. 29 and 30.
7. EV-- RESET is sent to each state machine by a procedure call of the form PocketFsm (pPocket, EV-- RESET). As an example, assume pPocket indicates the state machine associated with selectable region (0104/0106). Proceeding through the pseudo-code for the procedure PocketFsm listed in Appendix I, fInternalEvent is set to TRUE and consequently control passes into the while loop. fInternalEvent is now set to FALSE. The current state, pPocket→State, is stored in pPocket→PreviousState. Now a state transition is made. The current state, ST-- INITIAL, having a value of 2, and the current event, EV-- RESET, having a value of 1, are used as row and column indices respectively into the state table aPocketFsm shown in FIGS. 29 and 30, to determine the value of the new state of the state machine, in this example, the state machine associated with selectable region (0104/0106). aPocketFsm[2][1] equals 3. Thus the new state of the state machine is 3, the value of ST-- RESET. Control passes, via the switch statement, to the ST-- RESET case and ST-- RESET state processing is performed. The time of selectable region selection is set to the current time, the state machine's fInvert flag is set to FALSE and the value of the state machine's color variable, pPocket→Color, is compared to the state machine's initial color, pPocket→InitialColor. pPocket→Color was initialized to zero, which is not the value of pPocket→InitialColor. Consequently, pPocket→Color is set to pPocket→InitialColor and the flag pPocket→fPaint is set to TRUE. Upon reaching the break statement, control passes through the bottom of the switch statement and the value of pPocket→fPaint is tested. Since it is TRUE, the client area rectangle is invalidated. In the prototype, Windows® (1204) responds to invalidating the client rectangle by sending the access program (1206) a WM-- PAINT message. On receipt of a WM-- PAINT message, the access program (1206) redraws all selectable regions and any menu option located thereon for each state machines having pPocket→fPaint equal to TRUE. Thus, somewhat indirectly, visible subregion (0106) is drawn on the display (2112). Then control returns to Windows® (1204). The other seven state machines, each associated respectively with a selectable region, are similarly initialized so that each state machine transitions from ST-- INITIAL to ST-- RESET and draws its respective visible subregion and any menu option located thereon on the display (2112). The display shown in FIG. 17 now appears on the display (2112) of the computer system (2116).
8. A periodic timer, called the cursor polling timer, is set. This timer provides the access program (1206) with a WM-- TIMER message at frequent intervals, in the prototype every system clock tick which occurs approximately every 54 milliseconds. Following access program (1206) initialization, most state transitions are made on expiration of the cursor polling timer. The access program (1206) calls the procedure CreateEvent to determine the appropriate event for each state machine and to complete the event data structure addressed by pEvent accordingly, then repetitively calls the procedure PocketFsm for each state machine, passing the unique indicator for the state machine and the appropriate event for that state machine. Preferably, the cursor polling timer is more frequent so that color changes to visible subregions are smaller and more frequent, giving a smoother appearance to color change.
An example of the selection of selectable region (0104/0106) in accord with the Perimeter Menu aspect of the invention will now be described. Following initialization, assume that the operator now begins to move the pointer (2202). Every 54 milliseconds the cursor polling timer expires, causing Windows® (1204) to send a WM-- TIMER message to the access program (1206). Following receipt of WM-- TIMER, the access program (1206) calls the procedure CreateEvent. The procedure CreateEvent, among other functions, determines whether the current cursor hotspot location lies within any selectable region. In this example, the operator is moving the cursor from its initial location in the center of the display toward selectable region (0104/0106), but since only 54 milliseconds have elapsed, the cursor hotspot has moved only slightly in that direction. The procedure CreateEvent determines that the cursor hotspot does not lie within any selectable region and that the cursor hotspot has not crossed out of a selectable region in the past 54 milliseconds. Therefore, the procedure CreateEvent determines that each state machine should receive EV-- MOVEMENT. The procedure PocketFsm is called with the indicator for the state machine associated with selectable region (0104/0106) and with EV-- MOVEMENT. The event EV-- MOVEMENT drives this state machine from its current state, ST-- RESET, to ST-- EBB-- TIDE. The pseudo-code for ST-- EBB-- TIDE in procedure PocketFsm is a break statement, indicating that no state specific action is taken at this time, other than the transition to ST-- EBB-- TIDE. Control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (0104/0106) receives EV-- MOVEMENT, all other state machines each receive EV-- MOVEMENT and each makes the same transition from ST-- RESET to ST-- EBB-- TIDE.
Another 54 milliseconds elapses and again procedure PocketFsm is called, sending EV-- MOVEMENT to the state machine associated with selectable region (0104/0106) and driving state machine from ST-- EBB-- TIDE to ST-- DECAY. Stepping through the pseudo-code for ST-- DECAY, the ST-- DECAY state sets pPocket→State to the value stored in pPocket→PreviousState and decrements pPocket→Color but not below the value of pPocket→InitialColor. pPocket→Color determines the color and brightness of visible subregion (0106). Decrementing pPocket→Color results in a darkening of visible subregion (0106). Resuming with the pseudo-code for ST-- DECAY state processing, the ST-- DECAY state sets fInternalEvent to TRUE, and, in this case, sets Event to EV-- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs, using the value of pPocket→State which was set by ST-- DECAY state processing to ST-- EBB-- TIDE, the previous state. The new state is found at aPocketFsm[ST-- EBB-- TIDE][EV-- NULL], which equals ST-- EBB-- TIDE. This state transition is unlike an ordinary state transition because the starting state is set by ST-- DECAY. All transitions from ST-- DECAY share this distinction. The state machine executes the code for the new state, ST-- EBB-- TIDE, which is simply a break statement. The procedure PocketFsm determines that fpaint is FALSE and exits. Control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (0104/0106) receives EV-- MOVEMENT, all other state machines each receive EV-- MOVEMENT and each makes the same transitions from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE.
Every 54 milliseconds this scenario is repeated for each state machine until the operator moves the cursor hotspot to a point within selectable region (0104/0106). At this time the procedure CreateEvent determines that there is an active selectable region, specifically selectable region (0104/0106), and that consequently EV-- DWELL should be sent to the associated state machine. The procedure PocketFsm is called with the indicator for the state machine associated with selectable region (0104/0106) and the event EV-- DWELL. EV-- DWELL drives this state machine from ST-- EBB-- TIDE to ST-- ENTRY. Following the pseudo-code for ST-- ENTRY state processing shown in PocketFsm pseudo-code, fInternalEvent is set to TRUE and Event is set to EV-- NULL, resulting in another state transition to aPocketFsm[ST-- ENTRY][EV-- NULL], which equals ST-- LOW-- TIDE.
The pseudo-code for ST-- LOW-- TIDE is only a break statement, so there is no state specific action for ST-- LOW-- TIDE other than entry into this state. fpaint is FALSE so the selectable region is not redrawn. Control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (0104/0106) receives EV-- DWELL, all other state machines each receive EV-- MOVEMENT and each makes the same transitions from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE.
Another 54 milliseconds elapses. The procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV-- MOVEMENT, which drives it from ST-- LOW-- TIDE to ST-- DWELL. Stepping through the pseudo-code for ST-- DWELL, the ST-- DWELL state sets pPocket→State to the value stored in pPocket→PreviousState, increments pPocket→Color by pPocket→Increment, but not above the value of pPocket→Ceiling, sets pPocket→fPaint to TRUE, sets fInternalEvent to TRUE, and, in this case, sets Event to EV-- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs. The new state is found at aPocketFsm[ST-- LOW-- TIDE][EV-- NULL], which equals ST-- LOW-- TIDE. This path is unlike an ordinary state transition because the starting state is set by ST-- DWELL. All transitions from ST-- DWELL share this distinction. The state machine executes the code for the new state, ST-- LOW-- TIDE, which is simply a break statement. The procedure PocketFsm determines that fpaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM-- PAINT message to the access program (1206). On receipt of WM-- PAINT, the access program (1204) checks the value of fpaint for each state machine, and if TRUE, sets fpaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine. After redrawing, control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (0104/0106) receives EV-- DWELL, all other state machines each receive EV-- MOVEMENT and each makes the state transitions from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE.
Assuming the operator maintains the cursor hotspot in the selectable region (0104/0106), the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST-- LOW-- TIDE to ST-- DWELL to ST-- LOW-- TIDE, driven by the cursor polling timer. With each transition to ST-- DWELL, visible subregion (0106) is brightened a bit. The polling timer interval is short enough and the increment to pPocket→Color is small enough that visible subregion (0106) appears to gradually brighten although in fact it progresses rapidly through a series of discrete brightness levels. With each iteration through ST-- DWELL, pPocket→Color in incremented.
Assuming that the operator maintains the cursor hotspot on selectable region (0104/0106), pPocket→Color will eventually equal or exceed pPocket→CrestTide, a variable set at initialization time and not changed thereafter. At this time, Event is set to EV-- STEP-- UP, driving a transition to aPocketFsm[ST-- LOW-- TIDE][EV-- STEP-- UP], which equals ST-- SELECTED. The state processing for ST-- SELECTED provides the operator with an audible indication that a selection has just been made, takes the action appropriate upon selection of this selectable region, including selecting the menu option associated with the selected selectable region. In this example, the access program (1206) does not generate output to another program or device at this time. ST-- SELECTED next sets pPocket→fInvert to TRUE, and, if appropriate, changes the menu options associated with various selectable regions. In this example, selectable region (0104/0106) is now associated with menu option "<space>", selectable region (0108/0110) with menu option "o", selectable region (0112/0114) with menu option "t", selectable region (0124/0126) with menu option "x", selectable region (0128/0130) with menu option "v", and selectable region (0132/0134) with menu option "r". Selectable regions (0116/0118) and (0120/0122) remains associated with the same menu options with which they were associated in FIG. 17. fpaint is set to TRUE for the state machines associated with the selectable regions having changed menu options. ST-- SELECTED next sets Event to EV-- NULL and fInternalEvent to TRUE. The state machine now makes the transition to aPocketFsm[ST-- SELECTED-- TIDE][EV-- NULL], which equals ST-- CREST-- TIDE. There is no state specific action for ST-- CREST-- TIDE other than entry into this state. The pseudo-code for ST-- CREST-- TIDE is only a break statement. Control returns to Windows® (1204).
When the access program (1206) later receives the WM-- PAINT message resulting from the invalidating of the client rectangle, the setting of fInvert to TRUE causes the visible subregion (0106) to be drawn in the color complementary to the value then indicated by pPocket→Color. The setting of fpaint to TRUE for all selectable regions associated with changed menu options causes those selectable regions and the menu options thereon to be redrawn. The display shown in FIG. 18 now appears on the display (2112) of the computer system (2116).
Assuming the operator maintains the cursor hotspot in the selectable region (0104/0106), the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST-- CREST-- TIDE to ST-- DWELL to ST-- CREST-- TIDE, driven by the cursor polling timer. With each transition to ST-- DWELL, visible subregion (0106) is brightened a bit, though it is now magenta, the complement of green. The polling timer interval is short enough and the increment to pPocket→Color is small enough that visible subregion (0106) appears to gradually brighten although in fact it progresses rapidly through a series of discrete brightness levels. With each iteration through ST-- DWELL, pPocket→Color in incremented. Assuming that the operator maintains the cursor hotspot on selectable region (0104/0106), pPocket→Color will eventually equal pPocket→Ceiling, a variable set at initialization time and not changed thereafter.
Assuming that the operator now moves the cursor so that the hotspot is located outside selectable region (0104/0106), CreateEvent generates the EV-- CROSS-- OUT for the state machine associated with selectable region (0104/0106), driving the state machine to aPocketFsm[ST-- CREST-- TIDE][EV-- CROSS-- OUT], which equals ST-- SELECT-- AND-- OUT. The state processing for ST-- SELECT-- AND-- OUT sends EV-- RESET and then EV-- MOVEMENT to all state machines, driving each of them from their current state to ST-- RESET and then to ST-- EBB-- TIDE. For each state machine, the transition ST-- RESET and then to ST-- EBB-- TIDE results in actions previously described for those states, except that, unlike before, the value of pPocket→Color for the state machine associated with selectable region (0104/0106) does not equal the value of pPocket→InitialColor for the state machine associated with selectable region (0104/0106). Consequently, pPocket→Color is set to pPocket→InitialColor and fpaint is set to TRUE, subsequently causing visible subregion (0106) to be redrawn in its initial color. Control returns to Windows® (1204).
The prototype continues to sample cursor location at 54 millisecond intervals, determine the appropriate event for each state machine and send that event each state machine, causing state transitions in each state machine according to the state table shown in FIGS. 29 and 30. Assuming that the operator next selects selectable region (0108/0110), the state processing in ST-- SELECTED displays the selection, the letter "o", in the circumscribed region (0150), and associates the menu options shown in FIG. 17 with their respective selectable regions. The display shown in FIG. 17 appears on the display (2112) of the computer system.(2116).
B. Confinement
The preferred embodiment of the Confinement aspect of the invention will now be described in detail from a functional perspective using an example illustrated in FIG. 31. In FIG. 31, 20 selectable regions, e.g. (1704), (1708) and (1714), are depicted on display (2112). Each of the selectable regions is located on the display (2112) adjacent an edge of the display, and the selectable regions together circumscribe a region (1702) on the display. The top and left edges of the Windows® cursor clipping rectangle (1750) lie on the top and left edges, respectively, of the display (2112). The bottom and right edges of the Windows® cursor clipping rectangle (1750) lie on the display parallel to and slightly indented from the bottom and right edges, respectively, of the display. A confining polygon is delimited on the display. The boundary of the confining polygon, starting from the upper right corner of the Windows® cursor clipping rectangle, follows the top edge of the Windows® cursor clipping rectangle to the left until it reaches region (1732), where the boundary follows the side of region (1732) down, to the left, and back up to the top edge of the Windows® cursor clipping rectangle. The boundary continues left along the top edge of the Windows® cursor clipping rectangle until it reaches region (1734), where the boundary follows the side of region (1734) down, to the left, and back up to the top edge of the Windows® cursor clipping rectangle. The boundary continues left along the top edge of the Windows® cursor clipping rectangle to the upper left corner and then turns down along the left edge of the Windows® cursor clipping rectangle until it reaches region (1736), where the boundary follows the side of region (1736) to the right, down, and to the left to the left edge of the Windows® cursor clipping rectangle. The boundary of the confining polygon continues in this fashion around to the upper right corner of the Windows® cursor clipping rectangle. The confining polygon thus includes all the area of the Windows® cursor clipping rectangle except for the regions (1732), (1734), (1736), (1738), (1740), (1742), (1744), and (1746). An operator controlling a pointer indicating successive locations with respect to the display and attempting to select a target selectable region may overshoot the target so that some of the successive locations lie outside the Windows® cursor clipping rectangle. In the preferred embodiment of the Confinement aspect of the invention, the cursor (1724) is confined to the confining polygon. The preferred embodiment is responsive to an intersection of the cursor hotspot and any one selectable region so that an overshot selectable region may be selected by click or by dwell without moving the location presently indicated by the pointer to a location in the overshot selectable region. Thus, the preferred embodiment of the Confinement aspect of the invention allows an operator with impaired ability to stop motion to maintain the cursor more easily on a selectable region, and so select the intended selectable region, than do conventional user interface systems.
A selectable region having a single side abutting a confining polygon prevents cursor movement only beyond the abutting side. However, NMD operators who drift may drift in more than one direction. Assume that a certain NMD operator tends to drift both up and to the left and that he is attempting to select selectable region (1704). If he moves the cursor into that selectable region his upward drift will be confined; the drift will not move the cursor beyond the confining polygon. However, the cursor will move to the left, since, in the preferred embodiment of the Confinement aspect of the invention, movement in this direction is not affected by the confining polygon, and consequently the cursor may move into selectable region (1708), the selectable region to the left of selectable region (1704). NMD operators having this type of drift may be assisted in selecting by confining corners. For example, such an operator, attempting to select selectable region (1704) could move the cursor to location (1706) in selectable region (1704). As the operator drifts to the left, he can compensate by moving the pointer to the right. Assuming the operator lacks fine motor control, he may overcompensate and indicate a location to the right of the Windows(cursor clipping rectangle (1750). However, since the cursor is confined to the confining polygon, the cursor remains in the intended selectable region.
Confining corners facilitate the selection process for some NMD operators. The preferred embodiment of the Confinement aspect of the invention creates a corner or virtual corner in each selectable region. A virtual corner is a corner of a selectable region formed by the intersection of two sides of a selectable region both of which abut a confining polygon. For example, corner (1710) in selectable region (1708) abuts the confining polygon both along the top edge of the Windows® cursor clipping rectangle and along the right side of region (1732). If an NMD operator drifts from selectable region (1708) to the left into region (1732), the cursor remains in selectable region (1708). Thus, drift to the left does not move the cursor out of selectable region (1732). An operator trying to select selectable region (1708) may overcompensate for drift to the right by moving the pointer to indicate a location in region (1732).
FIG. 32 illustrates a display in accordance with an alternative embodiment of the Confinement aspect of the invention. FIG. 32 depicts 16 selectable regions, e.g. selectable region (0602), on a display (2112), the selectable regions together at least partially circumscribing region (0660) on the display. Circumscribed region (0660) intersects four selectable regions, (0604), (0608), (0610), and (0612). Selectable region (0608) includes virtual corner (0652). The confining polygon includes all the area of the Windows® cursor clipping rectangle except for regions (0632), (0634), (0636), (0638), (0640), (0642), (0644), and (0646).
FIG. 33 illustrates a display and structures in accordance with an another embodiment of the Confinement aspect of the invention. FIG. 33 depicts 20 selectable regions, e.g. selectable region (0918), on a display (2112), the selectable regions together at least partially circumscribing region (0902) on the display. The confining polygon includes all the area of the Windows® cursor clipping rectangle except for regions (0906), (0908), (0910), and (0904).
The preferred embodiment of the Confinement aspect of the invention will now be described in detail from an implementation perspective. Preferably, the Confinement aspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention. The modifications required are: (1) define 20 state machines and 20 respectively associated selectable regions located as shown in FIG. 31; (2) at initialization time, create a Windows® region corresponding to the confining polygon described above in the functional description of the preferred embodiment of the Confinement aspect of the invention; and (3) modify the procedure CreateEvent so that before generating an EV-- CROSS-- OUT event, CreateEvent determines whether the current hotspot cursor location intersects the confining polygon, and, if not, set the Windows® hotspot cursor location to the previous hotspot cursor location and transfers control to the code at the beginning of the CreateEvent procedure which gets the current cursor hotspot location from Windows®. If the current hotspot cursor location intersects the confining polygon, CreateEvent takes the same action as in the access program (1206).
C. Dwell
The preferred embodiment of the Dwell aspect of the invention will now be described in detail from a functional and implementation perspective. The prototype implements dynamic dwell. The effect of dwelling on a selectable region and its implementation have been described in the detailed description of the Perimeter Menu aspect of the invention. Now the effect and implementation of moving the cursor hotspot off a selectable region will be described.
Referring now to FIG. 17, assuming that the state machine associated with the selectable region (0104/0106) is in state ST-- LOW-- TIDE, that pPocket→Color has been incremented above its initial value, that the cursor hotspot intersected selectable region (0104/0106) al the last expiration of the cursor polling timer, and that the operator initiates movement of the cursor hotspot to the right from selectable region (0104/0106) toward selectable region (0132/0134) so that, when the cursor polling timer next expires, the cursor hotspot is located in area (0150) between visible subregions (0106) and (0134), the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV-- CROSS-- OUT, which drives it from ST-- LOW-- TIDE to ST-- LOW-- TIDE. The state processing for ST-- LOW-- TIDE has already been described in the detailed description of the Perimeter Menu aspect of the invention. Assuming that the operator continues to move the cursor hotspot toward selectable region (0132/0134), on the next expiration of the cursor polling timer, the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV-- MOVEMENT, which drives it from ST-- LOW-- TIDE to ST-- DECAY. Stepping through the pseudo-code for ST-- DECAY, the ST-- DECAY state sets pPocket→State to the value stored in pPocket→PreviousState, decrements pPocket→Color by pPocket→Decrement, but not below the value of pPocket→InitialColor, sets pPocket→fPaint to TRUE, sets fInternalEvent to TRUE, and, in this case, sets Event to EV-- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs. The new state is found at aPocketFsm[ST-- LOW-- TIDE][EV-- NULL], which equals ST-- LOW-- TIDE. This path is unlike an ordinary state transition because the starting state is set by ST-- DECAY. All transitions from ST-- DECAY share this distinction. The state machine executes the code for the new state, ST-- LOW-- TIDE, which is simply a break statement. The procedure PocketFsm determines that fPaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM-- PAINT message to the access program (1206). On receipt of WM-- PAINT, the access program (1204) checks the value of fPaint for each state machine, and if TRUE, sets fpaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine. After redrawing, control returns to Windows® (1204).
If the movement of the cursor hotspot pause between successive samplings of its location, the procedure Create Event will determine that not EV-- MOVEMENT, but EV-- DECAY, should be sent to the state machine associated with selectable region (0104/0106). Like EV-- DECAY, EV-- MOVEMENT drives the state machine associated with selectable region (0104/0106) to ST-- DECAY. The same state processing as described above for ST-- DECAY takes place, including the transition back to ST-- LOW-- TIDE. Assuming the operator maintains the cursor hotspot in area (0150), the state machine associated with selectable region (0104/0106) cycles repetitively through the state transitions from ST-- LOW-- TIDE to ST-- DECAY to ST-- LOW-- TIDE, driven by the cursor polling timer. With each transition to ST-- DECAY, visible subregion (0106) is darkened a bit. The polling timer interval is short enough and the decrement to pPocket→Color is small enough that visible subregion (0106) appears to gradually darken although in fact it progresses rapidly through a series of discrete brightness levels. With each iteration through ST-- DECAY, pPocket→Color in decremented until pPocket→Color reaches pPocket→InitialColor.
When the cursor hotspot reaches selectable region (0132/0134), the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV-- DECAY, continuing the repetitively cycling through from ST-- LOW-- TIDE to ST-- DECAY to ST-- LOW-- TIDE, driven by the cursor polling timer. Driven by the same polling timer, the procedure CreateEvent determines that the state machine associated with selectable region (0132/0134) should receive EV-- DWELL, driving this state machine through the state transitions described in the detailed description of the Perimeter Menu aspect of the invention.
If the operator moves the cursor hotspot from selectable region (0132/0134) back to selectable region (0104/0106), the procedure CreateEvent determines that the state machine associated with selectable region (0104/0106) should receive EV-- DWELL, driving this state machine through the state transitions described in the detailed description of the Perimeter Menu aspect of the invention.
In the prototype, the selectable regions are shown on a display (2112). Alternatively, the selectable regions may appear on a static display, or they may be projected on a surface.
In the prototype, the brightness of a visible subregion at any time indicates the progress of the selection of the selectable region including the visible subregion. A brightness close to the initial brightness indicates that a relatively long period of dwelling on this selectable region is required for selection. A brightness close to the brightness just prior to selection indicates that relatively short period of dwelling on this selectable region is required for selection. However, means for indicating an intersection of the location indicated by the movement related signal and a selectable region, or the duration of a period of such an intersection includes, but is not limited to, a change in cursor appearance or location, a change in location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font or fill of an object on the surface, a display of or change in a graphic on the surface or the removal of a graphic from the surface, a generation of a sound or a change in the pitch or volume of an extant sound, a change in the temperature or surface of a contact area, the pressure exerted by a contact area, or frequency of contact by a contact area, or other suitable means. Any of these indications may be continuous or frequent.
Although dwell is implemented in the prototype using a data value, specifically pPocket→Color, it may be implemented using a signal, for example, voltage or current, varying in response to the intersection and subsequent non-intersection of a location indicated by a pointer and a selectable region. For example, a selectable region may include a detector and coupled electronics or electrical circuitry operative to increase the voltage level of a capacitor. Once elevated, the voltage level may decrease over time. Upon reaching a predetermined threshold, the voltage level may trigger selection.
The prototype allows an operator to make selections by dwell more efficiently than in conventional systems. In the prototype, the brightness of a visible subregion indicates the dwell time required for selection. A practiced operator may accurately estimate when he may plan his next pointer movement, when he may begin moving the pointer and may determine when a bit more exertion will select a selectable region and when it will not. Thus a disabled operator who is fatigued by computer access and can maintain a pointer in a steady position for only brief periods, may optimize his energy expenditure, for example, exerting himself to maintain the cursor on a certain selectable region only when doing so will quickly select the selectable region.
The prototype may increase the independence of a disable individual by allowing him to control devices such as a TV, thermostat and other household appliances. As stated earlier, the Perimeter Menu aspect of the invention may be implemented on a general purpose computer system. If the general purpose computer system is coupled to a devices capable of executing commands and the menu hierarchy allows the selection of commands, a disabled operator may select and issue commands to control these devices.
FIG. 34 illustrates a display in accordance with of an embodiment of the Dwell aspect of the invention having 20 selectable regions (3101), (3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119), (3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143), and (3145) circumscribing a surface (3151) having an indicating region (3147) thereon.
FIG. 35 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 20 selectable regions circumscribing a surface, each selectable region associated respectively with an indicating region adjacent its associated selectable region. Selectable region (3101) is associated with indicating region (3201), selectable region (3103) with indicating region (3203), etc.
FIG. 36 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 20 detectors circumscribing an aperture (3350), each detector associated respectively with an indicator intersecting its associated detector, e.g. detector (3301) is associated with indicator (3303).
FIG. 37 illustrates an apparatus in accordance with of an embodiment of the Dwell aspect of the invention having 24 selectable regions arranged in a grid of four rows and six columns, each selectable region associated respectively with an indicating region intersecting its associated selectable region. For example, selectable region (3402) is associated with indicating region (3401).
FIG. 38 illustrates a display in accordance with of an embodiment of the Dwell aspect of the invention having a plurality of detectors, (2501), (2503), (2505), etc., arranged in a grid. Each detector is associated respectively with an indicator intersecting its associated detector., e.g. detector (2501) is associated with indicator (2502). Each detector is also associated respectively with an order entry item which may be selected by dwell. For example, detector (2501) is associated with a hamburger. Pointer (2512) (not drawn to scale) emits energy detectable by the detectors. Pointer (2512) may be housed in stationary housing (2514) (not drawn to scale).
Means for determining the difference between two data items or signals and means for totaling two or more data items or signals may each include a processing unit programmed to calculate this difference or total. Alternatively, the difference or total may be determined by electronic, mechanical, optical, or other suitable means.
The two step procedure described earlier for operating conventional menu-driven data entry and order entry systems incorporating pointing at intended selections may be simplified to a single step in accordance with the Dwell aspect of the invention. Referring now to FIG. 38, the operator points pointer (2512) at an order entry item. The item brightens responsive to the signals falling on the associated detector, indicating to the operator which order entry item he is dwelling on and his dwell time on that order entry item. When his dwell time equals or exceeds the selection threshold, the order entry item is selected. The operation of such a system is intuitive and may be learned or relearned by pointing and dwelling. No surface is required, unlike a standard mouse. Additionally, an operator seated in the driver's seat of a vehicle is probably right-handed and may be making selections with his left hand, the hand closest to the window in a left-hand drive vehicle. Pointing and maintaining a pointer in a steady position requires less coordination than pointing and clicking.
D. Path Directness
The Path Directness aspect of the invention includes several aspects, hereinafter "subaspects", called Facilitated Dwell, Direction and Intersection, Direction, Appraisal and Drift Attenuation. The preferred embodiment and certain alternative embodiments of each of these subaspects will now be described.
According to the Facilitated Dwell subaspect of the invention, the duration of the dwell period required for selection ("selection threshold period") of a menu option associated with a selectable region varies with the directness of the cursor's path to that selectable region. The preferred embodiment of the Facilitated Dwell subaspect of the invention will now be described in detail from a functional perspective using an example illustrated in FIG. 39. In FIG. 39 on display (2112) are defined corridors (2302), (2304), (2306), (2308), (2310), (2312) and (2314) lying between the previously selected selectable region (2318) and, respectively, selectable regions (2342), (2344), (2346), (2348), (2350), (2352) and (2354). Each selectable region is associated respectively with a selection threshold period. Each cursor location in a cursor path, for example cursor path (2322), may slightly decrease one or more the selection threshold periods, except for the selection threshold period associated with the previously selected selectable region, selectable region (2318) in this example. The effect of a cursor location depends, in the preferred embodiment, on whether that cursor location intersects one of the corridors. If it does, as for example location (2324) intersects corridor (2308), then the selection threshold period associated with the selectable region to which the intersected corridor leads, (2348) in this example, is decreased, preferably to a limit of approximately 20% of the initial value of the selection threshold period so that some period of intersection of the cursor and the intended selectable region is still required for selection. A changed selection threshold period is preferably indicated by a change in the brightness of the selectable region associated with the changed selection threshold period. Thus, when the operator moves the cursor within a corridor, the selectable region associated with the corridor brightens, indicating both the target selectable region the system believes the cursor is headed toward and the changed selection threshold period. When the operator moves the cursor outside a corridor the cursor had previously intersected, the selectable region associated with the previously intersected corridor darkens in accord with the dynamic dwell aspect of the invention, indicating both that the system no longer believes the cursor is headed toward that selectable region and the changed selection threshold period. Reducing the selection threshold period facilitates selection of dwell-selectable regions without unduly increasing the likelihood of erroneous selections, since cursor locations within a corridor evidence the operator's intention to select the selectable region associated with the corridor.
Preferably, corridors are hidden from view, but they may be may be shown on the display or shown only at certain times or under certain conditions. Corridors may have fixed boundaries, depending on which selectable region has been selected, or their boundaries may be determined when a starting location, for example, cursor location (2316) in FIG. 39, is known. Corridor shape, size, number and position about the associated selectable region may vary, as illustrated by the alterative embodiments shown in FIGS. 40 and 41. Where corridors overlap, a cursor location intersecting two or more corridors may be defined to be in a cursor path toward zero, one or more selectable regions associated with the intersected corridors.
The intersection of a cursor location and a corridor is but one means of identifying which one of a plurality of selectable regions is most nearly along a cursor path. A cursor path may be indicated by an intersection of a cursor location and a predetermined region, e.g. a corridor, by a cursor location and a movement related signal from which may be derived a second location, or by two or more successive cursor locations. As used herein, successive locations include a plurality of locations distributed in time. Successive location may be, but need not be, consecutive. Given a location and a movement related signal or two locations, an intention to select a particular selectable region may be inferred, for example, by extrapolation, and the selection thresholds associated with either or both the intended or unintended selectable regions modified accordingly. As an example, assume successive cursor locations are periodically stored in a ring buffer and the magnitude of the angle between two line segments, the first between the oldest cursor location in the ring buffer and a predetermined point in the selectable region, and the second between the oldest cursor location in the ring buffer and the current cursor location, is determined. The selectable region associated with the smallest of these angles may be considered to be the selectable region most nearly along the cursor path indicated by the first line segment.
An alternative means of identifying which one of a plurality of selectable region is most nearly along a cursor path is to determine the ratio of the number cursor locations indicating a selectable region to the total cursor locations in the cursor path.
FIG. 42 illustrates still another alternative means for identifying which one of a plurality of selectable regions is most nearly along a cursor path. FIG. 42 shows line (2410) from starting cursor location (2406) to ending cursor location (2408) just within selectable region (2404). Line (2410) is the most direct path between these two points. In this example, the actual path traveled by the cursor between these two points is path (2412). The ending cursor location (2408) is known before the selection threshold period associated with selectable region (2404) is modified. Identification may be made by measuring or approximating the area within region (2414) bounded by line (2410) and cursor path (2412). The smaller the area, the more direct the cursor path. Alternatively, identification may be made by storing a sampling among successive cursor locations along a cursor path and, in response to the cursor intersecting a selectable region, the selectable region most nearly along the cursor path may be identified at one or more sampled points along the cursor path. The identification may be based upon a plurality of cursor locations or upon a single cursor location and a movement related signal. Alternatively, identification may be made by measuring and comparing the number of times a cursor path diverges from a predetermined path toward the intersected selectable region and/or the degree of divergence of a cursor path from a predetermined path toward the intersected selectable region.
Another apparatus in accord with the Path Directness aspect of the invention is illustrated in FIG. 34. In this Figure, selectable regions (3101), (3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119), (3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143) and (3145) circumscribe area (3151). Area (3151) and optionally the selectable regions include detectors for sensing radiant energy emitted from a pointer (2202) coupled to a body member of the operator. A computer coupled to the detectors determines which selectable region is most nearly along the path indicated by the body member of the operator. Responsive to the indicated path, the embodiment may facilitate the selection of one of the selectable regions by reducing a selection threshold, may select a selectable region upon intersection of the point indicated by the pointer (2202) and a selectable region, or may select a particular selectable region in advance of intersection of the point indicated by the pointer (2202) and the particular selectable region.
A move direction of a body member of an operator may be determined in any way that a cursor path may be determined, including sampling among data indicative of position of the body member. In determining move direction of a body member, data indicative of body member positions may serve the same function as cursor locations in indicating a path toward a selectable region. Position indicating means, used, for example, in indicating a position of the body member, includes each of the means for indicating that a selection event has occurred.
As an alternative to decreasing a selection threshold period, an embodiment may include a plurality of selection thresholds, each associated respectively with a selectable region. One or more of the selection thresholds may be increased when the direction of cursor movement does not indicate a path toward the associated selectable region.
Means for indicating which one of the plurality of selectable regions is most nearly along the cursor path includes each of the means for indicating an intersection of the location indicated by the movement related signal and a selectable region.
According to the Direction and Intersection subaspect of the invention, the selection threshold period is completely satisfied in response to a cursor path to a particular selectable region, so that when the cursor intersects the particular selectable region, that region is selected. Preferably, the selection threshold period is completely satisfied in response to a measure of directness of a cursor path to a particular selectable region equalling or exceeding a predetermined measure of directness. In circumstances where the measured directness is less than the predetermined measure, a dwell period is required for selection of the particular selectable region.
According to the Direction subaspect of the invention, a selectable region is selected in response to a cursor path to that selectable region, in advance of an intersection of the cursor and that selectable region. Preferably, the selectable region is selected in response to a measure of directness of a cursor path to a particular selectable region equalling or exceeding a predetermined measure of directness. In circumstances where the measured directness is less than the predetermined measure, a dwell period is required for selection of the particular selectable region.
According to the Appraisal subaspect of the invention, the directness of a cursor path to a selectable region is measured. Preferably, the means for measuring the directness of a cursor path includes each of the means for identifying which one of a plurality of selectable regions is most nearly along a cursor path. Thus, the particular means for identifying which one of a plurality of selectable regions is most nearly along a cursor path which best correlates with an operator's intended target selectable region may be identified.
As earlier described, some NMD operators have relatively unimpaired directional control, despite having other movement disorders. The Facilitated Dwell, Direction and Intersection, and Direction subaspects of the Path Directness aspect of the invention utilize that capability for computer access. Specifically, the ability of an operator to move a cursor in a direct path toward a selectable region is used to facilitate selection of that selectable region. The selectable region is selected more quickly than in conventional systems utilizing selection by dwell, increasing operator productivity. In addition, when selectable regions are located in accordance with the Perimeter Menu aspect of the invention, a cursor path toward a selectable region is often unambiguous, since usually there is only one selectable region along a cursor path, and a large rectangular area on the display is available for the output of an application program and is not obstructed by the menu. In certain embodiments in accordance with the Facilitated Dwell, Direction and Intersection, and Direction subaspects of the Path Directness aspect of the invention, the operator may receive an indication of which selectable region the system believes the operator is moving the cursor toward. The operator may adjust the cursor path in response to this feedback and thus move the pointer more accurately. Additionally, in embodiments in accordance with both the Facilitated Dwell subaspect of the Path Directness aspect of the invention and the Dwell aspect of the invention, the operator may receive an indication of the dwell time required to select the selectable region most nearly along the cursor path as the required dwell time changes in response to the cursor path.
The preferred embodiment of the Drift Attenuation subaspect of the invention will now be described in detail from a functional perspective using an example illustrated in FIG. 43. FIG. 43 depicts the upper right corner of a display (2112) having two selectable regions (0434) and (0430) thereon. Assuming, for purposes of this example, that a movement related signal indicates the path shown from point (0402) to point (0404) ("first segment") at a relatively high velocity and from point (0404) to point (0408) ("second segment") at a relatively low velocity. The path of the first segment is relatively direct, the path of the second segment, relatively meandering. During receipt of the movement related signal for the first segment, the cursor preferably tracks the exact path indicated by the movement related signal. During receipt of the movement related signal for the second segment, the movement of the cursor is attenuated, preferably so that the cursor does not leave selectable region (0434) until it is selected.
Many NMD operators are unable to cleanly stop movement of a body member, resulting in a relatively slow or meandering path being indicated by the movement related signal. According to the Drift Attenuation subaspect of the invention, drift, that is, unintentional movement, indicated by the movement related signal following intentional movement is distinguished from the intentional movement, and cursor movement responsive to the drift is attenuated relative to cursor movement responsive to intentional movement. Thus unintentional movements of NMD operators are filtered so that the cursor is displayed closer to the location intended by the operator and drifting of the cursor into a nearby, but unintended, selectable region, is avoided, resulting in fewer errors due to unintended selections.
The preferred embodiment of each subaspect of the Path Directness aspect of the invention will now be described in detail from an implementation perspective. Preferably, the Facilitated Dwell subaspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention. The modifications required are: (1) Additional state processing should be added to ST-- SELECTED to create a Windows® region in the shape of a corridor starting a predetermined distance from the selected selectable region to each of the other selectable regions; (2) The event data structure should be expanded to accommodate an event for a selectable region along a cursor path; (3) The procedure CreateEvent should be changed so that, before setting an event to EV-- DECAY or EV-- MOVEMENT, a test is made for the intersection of the cursor hotspot and each corridor, and, on finding such an intersection, a new event, EV-- CORRIDOR, is stored in the event data structure indicated by pEvent for the selectable region associated with the intersected corridor; (4) A column should be added to the state table so that each state which in the prototype may receive either EV-- MOVEMENT or EV-- DECAY will on receipt of EV-- CORRIDOR drive that state machine to ST-- CORRIDOR; (5) An additional variable, CorridorIncrement, preferably having an initial value one half the value of Increment, should be added the set of data associated with each state machine; and (6) A new state, ST-- CORRIDOR, should be added to the procedure PocketFsm. The pseudo-code for state processing in ST-- CORRIDOR follows:
______________________________________                                    
case ST.sub.-- CORRIDOR:                                                  
             /* set state to previous state */                            
             /* in preparation for the next */                            
             /* state transition */                                       
set pPocket->State to pPocket->PreviousState                              
incrernent pPocket->Color by                                              
pPocket->CorridorIncrement, but not above                                 
pPocket->InitialColor plus 80% of the difference                          
between pPocket->CrestTide and                                            
pPocket->InitiaiColor                                                     
if PPocket->Color was changed                                             
 set pPocket->fPaint to TRUE                                              
set fInternalEvent to TRUE                                                
set Event to EV.sub.-- NULL                                               
break                                                                     
______________________________________                                    
An example of the selection of selectable region (2346) in accord with the preferred embodiment of the Facilitated Dwell subaspect of the invention will now be described with reference to FIG. 39, assuming that the prototype has been modified as described, that the operator has just selected selectable region (2318), that all state machines are in state ST-- RESET, and that the operator moves the cursor hotspot from a location inside selectable region (2318) to location (2316), a location just outside selectable region (2318). On receipt of the next WM-- TIMER message, procedure PocketFsm is called with the indicator for the state machine associated with selectable region (2318) and with event EV-- CROSS-- OUT. Event EV-- CROSS-- OUT drives this state machine from its current state, ST-- RESET, to ST-- EBB-- TIDE. The pseudo-code for ST-- EBB-- TIDE in procedure PocketFsm is a break statement, indicating that no state specific action is taken at this time, other than the transition to ST-- EBB-- TIDE. Control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (2318) receives EV-- CROSS-- OUT all other state machines each receive EV-- MOVEMENT and each makes the transition from ST-- RESET to ST-- EBB-- TIDE.
Assume that the operator moves the cursor hotspot along path (2322) toward location (2332), a location outside all corridors, and then the next WM-- TIMER message is received. All state machines receive EV-- MOVEMENT and are driven from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE.
The state processing associated with these states has been described in the description of the Perimeter Menu aspect of the invention.
Assuming that the operator moves the cursor hotspot along path (2322), the transition from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE is repeated every 54 milliseconds for each state machine until the operator moves the cursor hotspot beyond location (2332). At this time the procedure CreateEvent determines that the cursor hotspot intersects corridor (2308) and that consequently EV-- CORRIDOR should be sent to the state machine associated with selectable region (2348). EV-- CORRIDOR drives this state machine from ST-- EBB-- TIDE to ST-- CORRIDOR. Stepping through the pseudo-code for ST-- CORRIDOR, the ST-- CORRIDOR state sets pPocket→State to the value stored in pPocket→PreviousState, increments pPocket→Color by pPocket→CorridorIncrement, but not above pPocket→InitialColor plus 80% of the difference between pPocket→CrestTide and pPocket→InitialColor, sets pPocket→fPaint to TRUE, sets fInternalEvent to TRUE, and sets Event to EV-- NULL. Following the break statement, the while fInternalEvent condition is true and another state transition occurs. The new state is found at aPocketFsm[ST-- EBB-- TIDE][EV-- NULL], which equals ST-- EBB-- TIDE. This path is unlike an ordinary state transition because the starting state is set by ST-- CORRIDOR. All transitions from ST-- CORRIDOR share this distinction. The state machine executes the code for the new state, ST-- EBB-- TIDE, which is simply a break statement. The procedure PocketFsm determines that fpaint is TRUE, invalidates the client rectangle and exits. As a result of invalidating the client rectangle, Windows® (1204) sends a WM-- PAINT message to the access program (1206). On receipt of WM-- PAINT, the access program (1204) checks the value of fpaint for each state machine, and if TRUE, sets fPaint to FALSE and redraws the visible subregion of the selectable region associated with that state machine and any menu option located thereon. The color of the redrawn visible subregion is determined by the value of the Color variable for that state machine. The incremented value of pPocket→Color results in a slight brightening of selectable region (2348) and reduces the difference between pPocket→Color and pPocket→CrestTide, corresponding to the dwell period required to select the associated selectable region. After redrawing, control returns to Windows® (1204).
Shortly before or shortly after the state machine associated with selectable region (2348) receives EV-- CORRIDOR, all other state machines each receive EV-- MOVEMENT and each makes the state transitions from ST-- EBB-- TIDE to ST-- DECAY to ST-- EBB-- TIDE.
This scenario is repeated at 54 millisecond intervals while the cursor hotspot travels along path (2322) to location (2326), a location intersecting corridor (2308). Between this location (2326) and location (2328), a location intersecting corridor (2306), along path (2322), the procedure CreateEvent determines that all state machines should receive EV-- MOVEMENT, driving each of them from their current state to ST-- DECAY and back to their current state. As described in the example in the detailed description of the Perimeter Menu aspect of the invention, ST-- DECAY state processing darkens the selectable region associated with the state machine, but not below a predetermined brightness represented by the variable InitialColor. Thus selectable region (2348) darkens when the cursor hotspot no longer intersects corridor (2308). From location (2328) to location (2330), the procedure CreateEvent determines that the state machine associated with selectable region (2346) should received EV-- CORRIDOR and all other state machines EV-- MOVEMENT. Consequently, selectable region (2346) gradually brightens up to a ceiling represented by pPocket→InitialColor plus 80% of the difference between pPocket→CrestTide and pPocket→InitialColor. The duration of dwell time required for selection of selectable region (2346) is thus reduced to approximately 20% of the dwell period required without Facilitated Dwell.
Preferably, the Direction and Intersection subaspect of the invention is implemented by making the changes to the prototype described for the Facilitated Dwell subaspect, except that, in incrementing pPocket→Color in ST-- CORRIDOR, the upper limit for pPocket→Color in ST-- CORRIDOR state processing is pPocket→CrestTide minus pPocket→Increment. Assuming these changes, a selectable region whose associated Color variable is at this upper limit is selected during processing of the WM-- TIMER message immediately following the intersection of the cursor hotspot and the selectable region.
Preferably, the Direction subaspect of the invention is implemented by making the changes to the prototype described for the Direction and Intersection subaspect of the invention, except that (1) the corridors are narrow, as illustrated in FIG. 40; and (2) the pseudo-code fir state processing in ST-- CORRIDOR is as follows:
______________________________________                                    
caseST.sub.-- CORRIDOR:                                                   
             /* set state to previous state */                            
             /* in preparation for the next */                            
             /* state transition */                                       
set pPocket->State to pPocket->PreviousState                              
increment pPocket->Color by                                               
pPocket->CorridorIncrement but not above                                  
pPocket ->Ceiling                                                         
if pPocket->Color was changed                                             
 set pPocket->fPaint to TRUE                                              
set fInternalEvent to TRUE                                                
if pPocket->Color was changed from a value below                          
pPocket->CrestTide to a value greater than or                             
equal to                                                                  
pPocket->CrestTide                                                        
 set Event to EV.sub.-- STEP.sub.-- UP                                    
else                                                                      
 set Event to EV.sub.-- NULL                                              
break                                                                     
______________________________________                                    
State ST-- CORRIDOR may now generate the internal event EV-- STEP-- UP, as state ST-- DWELL does in the detailed description of the Perimeter Menu aspect of the invention. A state machine in state ST-- EBB-- TIDE having a Color variable equal to or exceeding the CrestTide variable, will transition via PocketFsm[ST-- EBB-- TIDE][EV-- STEP-- UP] to ST-- SELECTED, and perform the ST-- SELECTED state processing described in the detailed description of the Perimeter Menu aspect of the invention.
Preferably, the Appraisal subaspect of the invention is implemented by making the changes to the prototype described for the Facilitated Dwell subaspect, except that (1) the, corridors are visible on the display (2112), (2) one of the selectable regions is designated to be the target selectable region and this is indicated to the operator, (3) cursor locations are stored in memory (2106), and (4) following an intersection of the cursor hotspot and the target or selection of a selectable region other than the target, path directness is measured in accordance with the stored cursor locations.
The preferred embodiment of the Drift Attenuation aspect of the invention will now be described in detail from an implementation perspective. Preferably, the Drift Attenuation aspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention, modified as described above in the description of the implementation of the preferred embodiment of the Facilitated Dwell subaspect of the invention and further modified as follows: (1) add two booleans to the data set associated with each state machine, one called fDirectPath, the other fAttenuateDrift, and initialize each of them in all state machines to FALSE; (2) in both ST-- DECAY and ST-- CORRIDOR, set both pPocket→fDirectPath and pPocket→fAttenuateDrift to FALSE; (3) append to ST-- ENTRY state processing corresponding to the following pseudo-code:
______________________________________                                    
if pPocket->Color equals or exceeds                                       
(pPocket->InitialColor + (0.5 * (pPocket->CrestTide -                     
pPocket->InitialColor))                                                   
 set pPocket->fDirectPath to TRUE                                         
______________________________________                                    
(4) append to ST-- DWELL state processing corresponding to the following pseudo-code:
______________________________________                                    
if the present cursor position intersects the edge of the                 
Windows ® cursor clipping rectangle and                               
pPocket->fDirectPath is TRUE                                              
 set pPocket->fAttentuateDrift to TRUE                                    
if pPocket->fAttentuateDrift is TRUE                                      
 store the difference in each of the x and y                              
 coordinates ("delta") between consecutive cursor                         
 locations in a circular buffer accommodating the                         
 last ten deltas, overwriting the oldest delta with                       
 the newest delta                                                         
 if ten deltas have been accumulated                                      
  calculate the average acceleration indicated by                         
  the last ten deltas                                                     
  if the average acceleration is negative                                 
   set the cursor at the location one half the                            
   distance between the current cursor location                           
   and the previous cursor location                                       
   display the cursor at this new cursor                                  
   location                                                               
______________________________________                                    
E. Intersection
The preferred embodiment of the Intersection aspect of the invention will now be described in detail from a functional perspective using an example depicted in FIGS. 44, 45, 46. FIG. 44 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) to which is coupled a pointer (2202). Eight regions are delimited with respect to the display (2112) and together circumscribe region (2650) on the display (2112). Four of these regions, (2602), (2612), (2622), and (2632), are entirely on the display (2112). Each of the other four regions respectively includes both a visible subregion ((2606), (2616), (2626), and (2636)) on the display (2112) and an invisible subregion ((2604), (2614), (2624), and (2634)) adjacent and outside the display (2112). Assume, for example, that the pointer is indicating location (2656) on the display (2112) and that the operator moves the pointer so that the location indicated by the pointer first intersects one of the regions at location (2652) in region (2612). Upon this intersection, the display changes to that shown in FIG. 45. FIG. 45 depicts a selectable region consisting of the union of invisible subregion (2714) and visible subregion (2716), hereinafter referred to as selectable region (2714/2716). Selectable region (2714/2712) is associated with menu option "sumac". In the preferred embodiment, the operator may select menu option "sumac" by dwelling on any part of the selectable region for the selection threshold period. Assuming, for example, that instead of moving the pointer from a position indicating location (2656) to location (2652), the operator instead moves the pointer so that the location indicated by the pointer first enters one of the regions at location (2654) in region (2614/2616). Upon this intersection, the display changes to that shown in FIG. 46. FIG. 46 depicts a selectable region consisting of the union of invisible subregion (2814) and visible subregion (2816), hereinafter referred to as selectable region (2814/2816). Selectable region (2814/2816) is associated with menu option "vort<space>x". In the preferred embodiment, the operator may select menu option "vort<space>x" by dwelling on any part of the selectable region for the selection threshold period.
In the preferred embodiment, each of the other seven regions shown in FIG. 44 is associated with a selectable region and each selectable region is associated with a menu option. The menu options shown in FIG. 44, in addition to "sumac" and "vort<space>x" are "wizen", "backspace", "words", "talk", ldhbfk" and "ypgqj,". The operator may select the menu option associated with any one of the selectable regions by moving the location indicated by the pointer from circumscribed region (2650) into the region associated with the selectable region and then dwelling on the selectable region for the selection threshold period. Each selectable region preferably includes all the area of its associated region.
An operator having impaired ability to maintain a body member in a steady position but who can control the point at which the location indicated by a body member enters a region may, in accord with the Intersection aspect of the invention, use his relatively unimpaired motor capability to selectively enlarge a selectable region or determine which of two or more selectable regions will occupy a predetermined area, thus making it easier for him to select. If the general purpose computer system of the preferred embodiment is coupled to a speech synthesizer and the menu options are letters or words, an operator with impaired speech may select or spell words and speak them.
FIGS. 47 and 48 illustrate the upper right corner of a display in accordance with an alternative embodiment of the Intersection aspect of the invention. FIG. 47 depicts; the upper right corner of a display (2112) of a computer system (2116) having thereon two regions, (1404) and (1402), each associated respectively with a selectable region, each selectable region associated respectively with menu options "bdfhkl" and "<space>cmnrst". The selectable region associated with region (1402) includes all of region (1402). The selectable region associated with region (1404) includes all of region (1404) plus area (1406) between region (1404) and the right edge of the display (2112). Assuming the operator uses a pointer to indicate a location on the display and that the location to first intersect the union of regions (1402) and (1404) intersects region (1402), the display remains as shown in FIG. 47. Dwelling in area (1406) operates to select the menu option "<space>cmnrst". If instead the location to first intersect the union of regions (1402) and (1404) intersects region (1404), the display changes to that shown in FIG. 48. Dwelling in area (1406) then operates to select the menu option "bdfhkl".
The preferred embodiment of the Intersection aspect will now be described in detail from an implementation perspective. The Intersection aspect is preferably implemented by modifications to the access program (1206) described above in the detailed description of the Perimeter Menu aspect of the invention. The modifications are: (1) add a row to the state table aPocketFsm at row index 18 for a new state, ST-- EXIT and initialize the value of aPocketFsm[ST-- EXIT][EV-- NULL] to 4, the value of ST-- LOW-- TIDE; (2) at initialization time, change aPocketFsm[ST-- LOW-- TIDE][EV-- CROSS-- OUT] to 18, the value of ST-- EXIT; (3) at initialization time for each state machine, (a) if the visible subregion associated with the state machine does not abut the edge of the display, set fInterior to TRUE, otherwise set fInterior to FALSE; (b) initialize iAdjacentPocket to the index of the state machine associated with the adjacent region. For example, assuming the index of the state machine associated with region (2614/2616) in FIG. 44 is 2 and the index of the state machine associated with region (2612) in FIG. 44 is 3, iAdjacentPocket in the data set associated with the state machine associated with region (2614/2616) is initialized to 3 and iAdjacentPocket in the data set associated with the state machine associated with region (2612) is initialized to 2; (c) initialize the second array of points to define the boundaries of the region associated with the selectable region associated with each state machine, for example, region (2612), and initialize the first array of points to define the boundaries of the visible subregion of the selectable region associated with the state machine, for example, visible subregion (2712); (4) remove creation of Windows® regions corresponding to visible subregions from the access program (1206) initialization; (5) append to ST-- ENTRY state processing corresponding to the following pseudo-code:
______________________________________                                    
if pPocket->fInterior equals TRUE                                         
 if pPocket->hRegion is not NULL                                          
  delete the Windows ® region having the handle                       
  pPocket->hRegion                                                        
 create a Windows ® region having the boundaries                      
 defined by the second array of points associated                         
 with this state machine and set pPocket->hRegion to                      
 the handle to this region                                                
 set pPocket->fPaint to TRUE                                              
send EV.sub.-- RESET to all state machines except this one and            
the state machine having the index                                        
pPocket->iAdjacentPocket                                                  
set pAdjacentPocket to point to the data set associated                   
with the state machine having the index                                   
pPocket->iAdjacentPocket                                                  
delete the Windows ® region having the handle                         
pAdjacentPocket->hRegion                                                  
set pAdjacentPocket->hRegion to NULL                                      
set pAdjacentPocket->fPaint to TRUE                                       
______________________________________                                    
(4) append to ST-- RESET state processing corresponding to the following pseudo-code:
______________________________________                                    
if pPocket->hRegion is not NULL                                           
 delete the Windows ® region having the handle                        
 pPocket- >hRegion                                                        
create a Windows ® region having the boundaries defined by            
the second array of points associated with this state                     
machine and set pPocket->hRegion to the handle to this                    
region                                                                    
______________________________________                                    
An example of the selection of selectable region in accord with the Intersection aspect of the invention will now be described with reference to FIGS. 44 and 45. Following initialization, assume that the operator moves the pointer (2202) from a position indicating location (2656) toward location (2652) in FIG. 44. At the next expiration of the cursor polling timer, the procedure CreateEvent sends an EV-- MOVEMENT event to the state machine associated with the selectable region associated with the menu option "sumac". The event EV-- MOVEMENT drives this state machine from its current state, ST-- RESET, to ST-- EBB-- TIDE. The pseudo-code for ST-- EBB-- TIDE in procedure PocketFsm is a break statement, indicating that no state specific action is taken at this time, other than the transition to ST-- EBB-- TIDE. Control returns to Windows® (1204). At the expiration of the cursor polling timer after the location indicated by the pointer reaches location (2652), the procedure CreateEvent sends event EV-- DWELL to the state machine associated with the selectable region associated with the menu option "sumac". EV-- DWELL drives this state machine from ST-- EBB-- TIDE to ST-- ENTRY. Following the pseudo-code for ST-- ENTRY state processing shown above and in the Procedure PocketFsm, fInternalEvent is set to TRUE, Event is set to EV-- NULL, the Windows® region corresponding to region (2612), ie. the Windows® region having the handle pPocket→hRegion, is deleted and a new Windows® region having the boundaries indicated in the first array of points, ie. visible subregion (2712) in FIG. 45, is created and the handle stored in pPocket→hRegion. pPocket→fPaint is set to TRUE, EV-- RESET is sent to state machines associated with regions other than (2612) and (2614/2616). Data in the data set associated with the adjacent region (2614/2616) is now modified. The Windows® region corresponding to region (2614/2616) is deleted, hRegion for the state machine associated with that region is set to NULL and fpaint associated with that state machine is set to TRUE. The event EV-- NULL drives the state machine associated with selectable region (2714/2712) to ST-- LOW-- TIDE. The pseudo-code for ST-- LOW-- TIDE is only a break statement, so there is no state specific action for ST-- LOW-- TIDE other than entry into this state. pPocket-fPaint is TRUE so the client rectangle is invalidated, resulting in the redrawing of Windows® regions associated with state machines having fpaint equal to TRUE so that the display (2112) appears as shown in FIG. 45. Control returns to Windows® (1204).
Dwelling at any location within selectable region (2714/2712) now cause the procedure CreateEvent to send EV-- DWELL to the state machine associated with this selectable region. Selection of this selectable region proceeds as described above in the detailed description of the Perimeter Menu aspect of the invention.
Assuming the operator moves the location indicated by the pointer out of selectable region (2714/2712) prior to selection, the procedure CreateEvent sends the event EV-- CROSS-- OUT to the state machine associate with that selectable region, driving it to ST-- EXIT. The state processing for ST-- EXIT causes the display (2112) to change to that shown in FIG. 44.
F. Alignment
The embodiment of the Alignment aspect of the invention as implemented in the prototype will now be described in detail from a functional and implementation perspective using an example depicted in FIGS. 49, 50, and 51. Each of these Figures depicts the upper right corner of a display (2112) having two visible subregions on the display. In these Figures, no subregions outside the display (2112) are shown. Alignment is achieved in several steps and requires operator interaction with the apparatus. For purposes of this example, assume that FIG. 49 depicts the upper right corner of the display (2112), that an operator, fitted with a head pointer, desires to keep the cursor on the display directly in his line of sight and that the location indicated by the pointer is presently 15 degrees to the right of the location of the arrow cursor (1802). The operator now dwells on selectable region (1834) for an predetermined period ("the lock threshold") which preferably is significantly greater than the selection threshold period. The apparatus responds to this extended dwell period by changing the display to that shown in FIG. 49. The arrow cursor is removed from the display and the lock icon (1902) is displayed in a predetermined location of the intersected selectable region ("lockspot") on the display. The lock icon remains on the lockspot for a predetermined period ("the lock period"); it does not move responsive to the operator's head movement. While the lock icon is displayed, the operator turns his head, bringing his line of sight into alignment with the lockspot. At the expiration of the lock period, the apparatus changes the display to that shown in FIG. 51. The lock icon is erased and the arrow cursor appears in the lockspot, which is where the operator is now looking. The arrow cursor moves in response to the operator's head movement.
An operator who loses alignment between location indicated by his pointer and the cursor may thus initiate an alignment sequence, and then, by moving his head or other body member when the prototype indicates he should do so by displaying the lock icon, regain alignment. That the lock icon is displayed indicates to the operator that he can align his head or other body member. The position of the lock icon indicates to the operator the location on the display with which he should align his head or other body member. This is the location where the arrow cursor will appear at the expiration of the lock period.
The Alignment aspect will now be described in detail from an implementation perspective as implemented in the prototype. The Alignment aspect of the invention is implemented an integral part of the state machine described in the detailed description of the Perimeter Menu aspect of the invention, using the same state machines and initialization, with the exception that aPocketFsm[ST-- CREST-- TIDE][EV-- CEILING] is changed from 9, the value of ST-- DWELL, to 14, the value of ST-- BEGIN-- LOCK. The operator initiates the alignment process by moving the cursor so that the cursor hotspot intersects a selectable region. As an example, FIG. 49 shows the arrow cursor (1802) intersecting selectable region (1834). The operator then dwells on the selectable region for the lock threshold, preferably at least one second greater than the selection threshold period. This operator action causes the state machine associated with the intersected selectable region to reach state ST-- CREST-- TIDE. On receipt of the first WM-- TIMER message following entry into state ST-- CREST-- TIDE, the procedure CreateEvent creates event EV-- CEILING, which drives the state machine from state ST-- CREST-- TIDE to state ST-- BEGIN-- LOCK. The state processing within state ST-- BEGIN-- LOCK beeps, sets the system cursor location to the selectable region's lockspot, sets the system cursor to null erasing the arrow cursor, displays a the lock icon (1902 in FIG. 50) on the selectable region's lockspot and initializes the global variable iLockCursor to the number of expirations of the cursor polling timer corresponding to the period of time the cursor will be locked (the "lock period") configured by the operator, preferably two seconds, and control is returned to Windows® (1204). Following the next expiration of the cursor polling timer, the procedure CreateEvent may generate an event EV-- DECAY, EV-- DWELL, EV-- CROSS-- OUT, EV-- MOVEMENT or EV-- CEILING. Each of these events drives the state machine from state ST-- BEGIN-- LOCK to state ST-- LOCK. Within state ST LOCK the system cursor is moved to the selectable region's lockspot and iLockCursor is decremented. Then it is determined whether iLockCursor equals zero. If not, control returns to Windows® (1204) for another iteration through ST-- LOCK. During the lock period, the system cursor is moved to the lockspot upon every expiration of the cursor polling timer, thus inhibiting movement of the cursor from the lockspot so that, during the lock period, the operator may move the location indicated by the pointer while the lock icon remains on or very close to the selectable region's lockspot. When, eventually, iLockCursor is decremented to zero, the global Event is set to EV-- NULL and fInternalEvent is set to TRUE so that another state transition occurs immediately. This transition drives the state machine from state ST-- LOCK to state ST-- END-- LOCK. Within state ST-- END-- LOCK, the lock icon is erased, Windows® (1204) is directed to display the arrow cursor (2002) at the lockspot, as shown in FIG. 51, the fpaint flag associated with the intersected selectable region is set to TRUE so that the selectable region will be drawn, restoring the background behind the erased lock icon. Assuming the cursor hotspot remains on the selectable region for another 54 milliseconds, CreateEvent generates EV-- DWELL, which drives the state machine to ST-- DISCARD. Referring to the pseudo-code in PocketFsm, ST-- DISCARD state processing sets State to PreviousState, returning the state machine to ST-- END-- LOCK and in effect, discarding the last event. Following the expiration of the cursor polling timer after the operator moves the cursor out of the selectable region it presently intersects, the procedure CreateEvent generates an event EV-- CROSS-- OUT which drives the state machine from state ST-- END-- LOCK to state ST-- SELECT-- AND-- OUT. The access program (1206) performs the stale processing for state ST-- SELECT-- AND-- OUT and subsequent states as previously described in the description of the Perimeter Menu aspect of the invention.
In the prototype, the operator may initiate alignment by depressing any character key on the keyboard. On receipt of a WM-- CHAR message from Windows® (1204), the access program (1206) removes the arrow cursor from the display and displays the lock icon at a predetermined location on the display. In the prototype, the predetermined location is the center of the display (2112). After a predetermined period, two seconds in the prototype, the lock cursor icon is erased and the arrow cursor displayed at the predetermined location.
In the prototype, the cursor is automatically centered if the cursor hotspot does not move for two minutes. Lack of movement of the cursor hotspot is detected in the procedure CreateEvent, which generates the event EV-- IDLE-- TIMEOUT for all state machines. The state processing of each state machine on receipt of EV-- IDLE-- TIMEOUT depends upon its current state. State machines in states ST-- INITIAL and ST-- RESET stay in those states. State machines in all other states in which an external event can be received are driven to ST-- IDLE. Referring to the pseudo-code in PocketFsm, ST-- IDLE state processing moves the cursor to the center of the display, sets fInternalEvent to TRUE and sets Event to EV-- NULL. PocketFsm[ST-- IDLE][NULL] equals 3, the value of ST-- RESET. The access program (1206) performs the state processing for the ST-- RESET and subsequent states as described in the description of the Perimeter aspect of the invention.
G. Length Order
The preferred embodiment of the Length Order aspect of the invention will now be described in detail from a functional and implementation perspective using an example depicted in FIG. 52 and described in the detailed description of the Location Indication aspect of the invention. Assume, for purposes of this example, that the operator has previously selected the letter "s". In the preferred embodiment, the string "s" is passed from the access program (1206 in FIG. 16) via the Windows® Dynamic Data Exchange ("DDE") interface to a database program (1210 in FIG. 16), preferably the FoxPro for Windows® version 2.6 program, available from Microsoft Corporation, Redmond, Wash., USA. The FoxPro program looks up the record having the index "s" in a database composed of 26 letters, one for each letter of the alphabet. Each record includes one field for a letter of the alphabet and 12 fields containing the 12 most frequently used English words beginning with that letter. The 12 words in the record are ordered primarily by length, determined by the number of letters in each word, and secondarily by alphabetic order. The FoxPro program returns these 12 words to the access program (1206) and these are displayed on the display (2112) as named menu options. The twelve named menu options, ordered as described and depicted in FIG. 52, are "so", "say", "she", "said", "show", "some", "such", "state", "school", "social", "service", and "student". Preferably, only the root form of inflected forms of words which can be created through available suffixes ("root word") may be displayed as named menu options, so that the limited number of available menu options in combination with the apparatus' capability to add suffixes offers a large number of inflected forms.
Words which may be named menu options in the same menu may be ordered by any suitable method. Preferably ordering is done by an ordering program operating on a corpus of text or speech including text or speech produced by individuals whose age, sex, geographic location and disability are the same as or similar to that of the operator. The ordering program determines the frequency of use of root words in the corpus, selects the twelve most common root words, beginning with every possible combination of one, two and three letters, and stores them in three FoxPro databases for one, two and three letter word beginnings respectively, the words in each record ordered as described above. Ordering the words prior to a request minimizes the delay between the operator's selection of a letter or letters and the display of the named menu options. Preferably, the ordering program also creates a database of records for root words beginning with four or more letters. Each record includes the words and its frequency of use in the corpus. When the operator selects four or more letters consecutively, the access program (1206) requests via DDE that the FoxPro program (1210) look up words starting with four or more letters in the database for words beginning with four or more letters, select the 12 most frequently used words matching the selected letters, order them as described above, and return them to the access program (1206).
An operator searching named menu options for a desired menu option may start his search in the area on the display most likely to contain the desired menu option. Upon comparison of the length of the desired menu option to a displayed menu option, the operator may determine whether to continue his search from the displayed menu option toward the front of the list or toward the rear of the list, or to jump to another displayed menu option in the list Further, he may make this determination more quickly than if the displayed menu options were sorted conventionally. The reduced menu option search time increases the operator's productivity with respect to conventional menu interfaces.
H. Location Indication
The preferred embodiment of the Location Indication aspect of the invention will now be described in detail from a functional perspective using an example depicted in FIG. 52. FIG. 52 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) having thereon eight selectable regions, each associated respectively with a menu option. Each of the eight selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion (not shown) located outside the display (2112) and adjacent the visible subregion. Selectable region (4301) is associated with menu option "so", selectable region (4303) with menu option "say", selectable region (4305) with menu option "she", selectable region (4307) with menu option "suffixes", selectable region (4309) with menu option "words", selectable region (4311) with menu option "some", selectable region (4313) with menu option "show" and selectable region (4315) with menu option "said". Also shown on the display (2112) are 12 indicating regions, each associated respectively with a menu option. Indicating region (4351) is associated with menu option "so", indicating region (4353) with menu option "say", indicating region (4355) with menu option "she", indicating region (4361) with menu option "some", indicating region (4363) with menu option "show", indicating region (4365) with menu option "said", indicating region (4371) is associated with menu option "such", indicating region (4373) with menu option "state", indicating region (4375) with menu option "school", indicating region (4381) with menu option "student", indicating region (4383) with menu option "service" and indicating region (4385) with menu option "social". Each indicating region is located on the display (2112) in the region (4350) circumscribed by the selectable regions. In accord with the Location Indication aspect of the invention, the location of each of the indicating regions (4351), (4352), (4355), (4361), (4363) and (4365) indicates the location of each of the selectable regions associated with the menu option associated with the indicating region. These selectable regions, respectively, are (4301), (4303), (4305), (4311), (4313) and (4315).
In the preferred embodiment, selection of menu option "words" causes selectable region (4301) to be associated with menu option "such" instead of menu option "so", selectable region (4303) to be associated with menu option "state" instead of menu option "say", selectable region (4305) to be associated with menu option "school" instead of menu option "she", selectable region (4311) to be associated with menu option "student" instead of menu option "some", selectable region (4313) to be associated with menu option "service" instead of menu option "show", and selectable region (4315) to be associated with menu option "social" instead of menu option "said". In the preferred embodiment, following the selection of menu option "words", the menu option newly associated with each selectable region is displayed on that selectable region (not shown). Following the selection of menu option "words", indicating regions (4371), (4373), (4375), (4381), (4383) and (4385) each indicate the location of each of the selectable regions associated with the menu option associated with the indicating region.
The indicating regions and the menu options displayed thereon in FIG. 52 are disproportionately large relative to the rest of FIG. 52. They are approximately 1.5 times their proportionate size. They are represented as shown in FIG. 52 in compliance with Patent Cooperation Treaty Rules requiring a minimum size for letters in figures.
In the preferred embodiment of the Perimeter Menu aspect of the invention, all the menu options displayed on the selectable regions cannot be seen in a glance by many operators. However, when the displayed menu options are displayed on the indicating regions, for example, as shown in FIG. 52, the displayed menu options can be seen in a glance, facilitating searching of the menu by the operator for his intended menu option. Since the location of each selectable region is indicated by the location of the associated indicating region, the operator may point to the intended selectable region without searching it out or pausing to verify that the menu option associated with a selectable region is the menu option he desires. The frequent operator of such a menu interface may habituate the process of selecting an intended menu option so he can focus his attention on another task while selecting the option, for example, planning his next interaction with the menu interface.
FIG. 53 depicts a display and structures in accord with an another embodiment of the Location Indication aspect of the invention. Selectable regions (4604), (4608), (4612), (4616), (4620), (4624), (4628) and (4632) are located adjacent the display (2112) and associated respectively with menu options "so", "say", "she", "suffixes", "said", "show", "some" and "words". Each selectable region is unbounded on the side furthest from and parallel to the edge of the display. Indicating regions (4351), (4353), (4355), (4357), (4367), (4361), (4363) and (4365) are associated respectively with menu options "so", "say", "she", "suffixes", "said", "show", "some" and "words". In accord with the Location Indication aspect of the invention, the location of an indicating region indicates the location of the selectable region associated with the menu option associated with the indicating region.
The large size of the selectable regions outside the display, for example, as shown in FIG. 53, facilitate selection by individuals with impaired fine motor control while the indicating regions indicate the location of each associated selectable region.
Yet another embodiment in accord with the Location Indication aspect of the invention is shown in FIGS. 54 and 55. FIG. 54 depicts a display (2112) having thereon eight selectable regions, each associated respectively with a menu option. As shown in the Figure, selectable region (3911) is associated with menu option "aeiou", selectable region (3910) with menu option "gqjyvwxz", selectable region (3908) with menu option "setup", selectable region (3907) with menu option undo, selectable region (3905) with menu option "control", selectable region (3904) with menu option "words", selectable region (3902) with menu option "nsrm<space>tcp" and selectable region (3901) with menu option "bdfhkl". Menu option "aeiou" is associated with a submenu which includes submenu options "a", "e", "i", "o" and "u", displayed on indicating regions (3930), (3928), (3926), (3924) and (3922) respectively. The background pattern of indicating region (3930) matches the background pattern of selectable region (3908), indicating that submenu option "a" will be associated with selectable region (3908) following selection of menu option "aeiou". Similarly, the background pattern of indicating region (3928) matches the background pattern of selectable region (3910), indicating that submenu option "e" will be associated with selectable region (3910) following selection of menu option "aeiou". Alternatively, the indication may be made by the size, shape, hue, brightness, contrast, dithering, fill, blinking, hatching or pattern of the indicating region or any part thereof, including either of the foreground and background of the indicating region.
When the operator selects selectable region (3911), the display (2112) changes to that shown in FIG. 55. Now, selectable region (3910) is associated with submenu option "e", selectable region (3908) with submenu option "a", selectable region (3904) with submenu option "u", selectable region (3902) with submenu option "o" and selectable region (3901) with submenu option "i". In accord with the Location Indication aspect of the invention, the appearance of an indicating region indicates the location of the selectable regions associated with the submenu option associated with the indicating region.
The embodiment shown in FIGS. 54 and 55 illustrates how the Location Indication aspect of the invention speeds selection of a submenu option of a menu hierarchy. The operator, by observing the background pattern of the submenu option within the displayed menu option, may determine which selectable region he should next dwell on. The operator may make this determination prior to selection of the menu option and need not wait for the submenu options to be displayed on their associated selectable regions. In accordance with the Location Indication aspect of the invention, the operator may select or spell out words more quickly than with conventional automated menu hierarchy. Assuming the display (2112) is part of computer system, these words may be input to an application program, and, if the computer system is coupled to a speech synthesizer, these words may be spoken.
FIG. 56 depicts another apparatus in accord with the Location Indication aspect of the invention. An example of the operation of this apparatus will now be described. FIG. 56 depicts display area (4770) adjacent to which are located selectable regions (4701), (4703), (4705), (4707), (4709), (4711), (4713) and (4715). Not shown in the Figure are three sets of eight menu options. At different times during the operation of the apparatus shown in FIG. 56, for example at one second intervals, the plurality of selectable regions is associated with a different set of eight menu options such that, for a one second period, each selectable region is associated respectively with one menu option of the associated set of menu options. On the display area are 24 indicators, each associated respectively with one of the menu options. Each indicator indicates when a selectable region is associated with the menu option associated with the indicator. Thus, each of the indicators (4721), (4723), (4725), (4727), (4729), (4731), (4733) and (4735) is associated respectively with a menu option which may be in turn associated respectively, for example, during a first one second period, with one of the selectable regions. The operator may select a desired one of the 24 menu options by selecting the associated selectable region during the period when the desired menu option is associated with the associated selectable region.
Alternatively, the apparatus shown in FIG. 56 may require two successive selection events, the first selection event to select a set of eight menu options or to select a set of three menu options, the set being associated with one selectable region, and the second selection event to select one of the selected set. For example, the first selection event may select the set of menu options associated with indicators (4773), (4753) and (4733), and the second selection event may select one menu option from this set. Still another apparatus in accord with the Location Indication aspect of the invention is shown in FIG. 57. FIG. 57 depicts display area (4770) outside of which are located selectable regions (5001), (5003), (5005), (5007), (5009), (5011), (5013), (5015), (5021), (5023), (5025), (5027), (5029), (5031), (5033) and (5035). The location of each selectable region is indicated by an indicating region on the display area, (5041), (5043), (5045), (5047), (5049), (5051), (5053), (5055), (5061), (5063), (5065), (5067), (5069), (5071), (5073) and (5075), respectively.
The Location Indication aspect of the invention will now be described from an implementation perspective with reference to FIG. 52. Preferably, the Location Indication aspect is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention. The modifications required for the Location Indication aspect of the invention are: (1) at initialization time: (a) create twelve child edit windows, each corresponding to one of the indicating regions shown in FIG. 52, each of the class "edit", each having the style (WS-- CHILD | WS-- BORDER | WS-- DISABLED | ES-- MULTILINE | AlignmentStyle) where "|" represents a logical OR operation and where AlignmentStyle equals ES-- RIGHT for child windows having right justified text and ES-- LEFT for child windows having left justified text, as shown in FIG. 52; (b) move the child edit windows so they are located in or near the center of the display; (c) display the child edit windows; and (d) initialize an array which maps each child edit window, for example, by an index between 0 and 11 inclusive, to an element of the aLabel array; and (2) modify the code implementing ST-- SELECTED state processing, so that after changing the menu options associated with various selectable regions (by changing the pLabel element of the data set associated with each of the state machines to point to the menu option in the aLabel array to be associated with that state machine), the array mapping each child window to an element of the aLabel array is used to access the text of the menu option and the text of each child edit window is set accordingly. In the preferred embodiment, the six menu options displayed in indicating regions (4371), (4374), (4375), (4381), (4383) and (4385) are present in the aLabel array when FIG. 52 is shown, but are not associated with selectable regions until the operator selects menu option "words".
I. Sound Match
The preferred embodiment of the Sound Match aspect of the invention will now be described in detail from a functional perspective using an example depicted in FIG. 58. FIG. 58 shows a display (2112) of a computer system (2116) having thereon six regions or sound indicators, (5801), (5803), (5805), (5807), (5809), and (5811), each associated with a menu option, "ypgqj,", "ldhbfk", "wizen", "words", "vort x", and "sumac", respectively. Each menu option is displayed on its associated sound indicator. Each sound indicator has a distinct hue. For example, sound indicator (5801) may be green, (5803) white, (5805) blue, (5807) red, (5809) orange and (5811) grey. Each of these sound indicators indicates a sound the operator is able to consistently produce, for example, the vowel sound e as it sounds in green, i as it sounds in white, u as it sounds in blue, e as it sounds in red, o as it sounds in orange and e it sounds in grey. Assuming the operator says o as it sounds in orange, "vort x", the menu option associated with the orange sound indicator (5809) is selected and the display is changed to that shown in FIG. 59. In the preferred embodiment, submenu options space, "x", "r", "t", "v" and "o" are now associated with each of the sound indicators (5901), (5903), (5905), (5907), (5909), and (5911) respectively. Assuming the operator says e as in green, submenu option space is selected and a space is input to the application program (6107) whose output appears in region (5850).
FIG. 60 illustrates a display and structures in accordance with an alternative embodiment of the Sound Match aspect of the invention. FIG. 60 depicts a display (2112) of a computer system (2116). Adjacent the display (2112) is a static display (6252) having thereon eight sound indicators (6204), (6208), (6212), (6216), (6220), (6224), (6228), and (6232). Each of the eight sound indicators is a symbol in a phonetic character set graphically representing a sound. Alternatively, the sound indicators may include a picture of an object, for example, a tree, a house, a boy, or a map of a country, or may include a shape, for example, a square, a circle, or a triangle, or may include a hue indicator, a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch indicator, or a change in volume indicator. Eight menu options, "vort<space>x", "sumac", "wizen", undo, "words", "talk", "ldhbfk" and "ypgqj,", are displayed on the display (2112), each adjacent and associated respectively with one of the sound indicators, the eight menu options together circumscribing region (6250) on the display. The operator may select any one of the menu options by producing the sound indicated by the sound indicator associated with the menu option. In response to a menu selection, submenu options of the selected menu option may be displayed and associated respectively with a sound indicator.
An operator with impaired speech but who is able to consistently produce a relatively small number of sounds distinguishable by conventional speech recognition means may, in accord with the Sound Match aspect of the invention, select from among the small number of menu options by using the sounds he can produce. Assuming the display (2112) is part of computer system, the menu option may represent inputs to an application program, and, if the computer system is coupled to a speech synthesizer, the menu option may represent words to be spoken.
FIG. 61 illustrates a display and structures in accordance with an alternative embodiment of the Sound Match aspect of the invention. FIG. 61 depicts a display (2112) of a computer system (2116) having thereon six square sound indicators, (6401), (6403), (6405), (6411), (6413), and (6415) arranged in two columns of three sound indicators. Each sound indicator abuts a sound indicator in the other column, the sound indicator located above it, if any, and the sound indicator located below it, if any. Assume that the operator is entering Chinese, that he uses a keyboard to enter a phonetic unit and intonation according to the Pin Yin coding method for the Chinese language, and that he enters the distinct sound "fu", each of the sound indicators shown in FIG. 61 is associated respectively with a menu option, each of the plurality of menu options having a common characteristic. In this example, the menu options are homophones and the common characteristic is a phonetic unit and intonation but may be a phonetic unit alone, an intonation alone, a stroke used to draw an ideograph, a number of horizontal strokes, a number of vertical strokes, a number of total strokes, a stroke order, a radical, a part of speech, an ideograph, a kana, a diacritic, a classification of a part of an ideograph or other characteristic of a class of ideographs. Each of the sound indicators has a distinct hue. The operator may select any one of the displayed menu options, according to the Sound Match aspect of the invention, by speaking the sound associated with the sound indicator associated with the desired menu option. For example, assume the operator says, "blue" or a translation thereof, preferably a Chinese translation in this example. The ideograph located on the blue sound indicator is selected. In this example, this ideograph is input to a word processing program and appears on the display (2112).
The Sound Match aspect of the invention thus allows an operator to select from an option from a menu, using speech recognition means, whether or not the menu options are homophones. The operator does not need to use his hands to make this selection and so may keep both his hands on the home row of the keyboard, in preparation for entering the next common characteristic, or, if specifying the common characteristic by voice, may select a menu option without interrupting the manual activity he's engaged in.
The preferred embodiment of the Sound Match aspect of the invention will now be described in detail from an implementation perspective, beginning with the hardware and software operating environment which will now be described with reference to FIG. 62 which depicts a speech recognition system (6001) including a computer system (2116) and keyboard (2210), as earlier described, a sound board (6007) and a microphone (6009). The conventional computer system (2116) preferably includes an 80486 CPU running at 33 MHz or faster, and is provided with Dragon Dictate Power Edition software, available from Dragon Systems, Inc., Newton, Mass., USA. Preferably, the sound board is the Audio Capture Playback Adaptor and the microphone is the Shur Headset microphone, both available from Dragon Systems, Inc. FIG. 63 depicts the software environment of the preferred embodiment, which includes the earlier described Windows® version 3.1 operating system (1204), an optional Windows® application program (6107), the Windows® Dragon software driver (6101 ) included in the Dragon Dictate Power Edition software, the Dragon Application (6103) included in the Dragon Dictate Power Edition software, and a speech recognition access program (6105). The Dragon Application (6103) is configured to match a sample of each of six sounds distinguishable by the Dragon Application (6103) which the operator can consistently produce. Using the example described above, these are the vowel sounds produced by the operator of e as it sounds in green, i as it sounds in white, u as it sounds in blue, e as it sounds in red, o as it sounds in orange and e it sounds in grey. Each of these sounds is associated respectively with an identifier, for example, a number or a sequence of one or more characters. The speech recognition access program (6105) is preferably a Windows® application program developed using, in part, the Voice Tools Software Development Kit available from Dragon Systems, Inc. At initialization time, the speech recognition access program (6105) defines an array of data structures defining the menu and submenu options and the menu hierarchy. For example, one of the elements of this array determines that, on selection, certain actions are to be taken, for example, inputting text to an application program, and that certain submenu options and related data are to be associated with certain child edit windows. Also at initialization time, the speech recognition access program (6105) initializes callback procedures using the Dragon Application's Application Program Interface to receive notification from the Dragon Application when a sound has been matched. Also at initialization time, the main window of the speech recognition access program (6105) is preferably sized to just encompass the sound indicators shown in FIG. 58. Also at initialization time, the speech recognition access program (6105) creates six child edit controls, each corresponding to one of the sound indicators shown in FIG. 58. Each of the child edit controls has the background color described above and a text color of black or white depending upon which provides better contrast against the background color of the child edit control, and each is located on the display (2112) as shown in FIG. 58. Also at initialization time, the menu options of the initial menu are each associated respectively with one of the child edit controls. After initialization, the speech recognition access program (6105), upon notification from the Dragon Application that a sound has been received and a match attempted, sequentially searches the list of identifiers matched to the sound by the Dragon Application (6103), starting with the best match, until it finds an identifier corresponding to any one of the six sounds distinguishable by the Dragon Application (6103). Assuming the operator says o as it sounds in orange, the Dragon Application (6103) provides to the speech recognition access program (6105) a list of matches including, before any other identifier corresponding to any one of the six sounds distinguishable by the Dragon Application (6103), the identifier associated with child edit control (5809). This child edit control is currently associated with menu option "vort x", the matched menu option. The speech recognition access program (6105) then sets the text of each of the child edit controls to one of the submenu options associated with the matched menu option. In the example above, submenu options space, "x", "r", "t", "v" and "o" are associated with each of the child edit controls respectively and the text of the associated child edit control is set to the submenu option. Assuming the operator says e as in green, submenu option space is the matched submenu option and a space is input to the application program (6107) whose output appears in region (5850). Preferably the application program (6107) is an application program capable of executing a WM-- SIZE command so that the speech recognition access program (6105) may size the windows of the application program (6107) to fit neatly in region (5850) and is capable of executing WM-- CHAR messages so that the speech recognition access program (6105) may input characters to the application program (6107).
J. Ideographic Language
The preferred embodiment of the Ideographic Language aspect of the invention will now be described in detail from a functional perspective using an example depicted in FIG. 64. FIG. 64 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) and 12 selectable regions. Each of the 12 selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion located outside the display (2112) and adjacent the visible subregion. For example, the selectable region in the upper left corner both above and below the top of the display (2112) in FIG. 64 consists of invisible subregion (3604) and visible subregion (3606), and hereinafter is referred to as selectable region (3604/3606). The other selectable regions shown in FIG. 64, proceeding counter clockwise from selectable region (3604/3606) are (3608/36,10), (3612/3614), (3616/3618), (3620/3622), (3624/3626), (3644/3646), (3648/3650), (3652/3654), (3656/3658), (3660/3662) and (3664/3666). In FIG. 64 each of the visible subregions is adjacent an edge of the display (2112). The selectable regions together circumscribe region (3680) in the center of the display. Also shown on the display (2112) within region (3680) in FIG. 64 are ten square indicating regions arranged in two columns of five indicating regions. Each indicating region abuts an indicating region in the other column, the indicating region located above it, if any, and the indicating region located below it, if any. In the preferred embodiment, each indicating region indicates by its location the location of a respectively associated selectable region, in accord with the Location Indication aspect of the invention. For example, the uppermost indicating region in the left column of indicating regions (3605) is associated with the uppermost selectable region (3604/3606) on the left side of the display. Indicating region (3609) located immediately below indicating region (3605) is associated with selectable region (3608/3610) located immediately below selectable region (3604/3606).
The operation of the example of the preferred embodiment of the Ideographic Language aspect of the invention will now be described. Assuming that the operator is fitted with a head pointer coupled to the general purpose computer system (2218), that he uses a keyboard to enter a phonetic unit and intonation according to the Pin Yin coding method for the Chinese language, and that he enters the distinct sound "fu", each of the plurality of the selectable regions shown in FIG. 64 is associated with one of a plurality of menu options, each of the plurality of menu options having a common characteristic. In this example, the common characteristic is a phonetic unit and intonation but may be a phonetic unit alone, an intonation alone, a stroke used to draw an ideograph, a number of horizontal strokes, a number of vertical strokes, a number of total strokes, a stroke order, a radical, a part of speech, an ideograph, a kana, a diacritic, a classification of a part of an ideograph or other characteristic of a class of ideographs. The ten menu options shown in FIG. 64 are Chinese ideographs each starting with the distinct sound "fu". Alternatively, the menu options may be sequences of graphic symbols including one or more kanji. In FIG. 64 the Chinese ideographs are each displayed on the visible subregion of the associated selectable region and on the indicating region associated with the selectable region. The remaining two of the 12 selectable regions, (3624/3626) and (3644/3646), are associated with menu options for undo and for displaying more menu options, respectively. In the preferred embodiment, in response to the selection of the menu option for displaying more menu options, each of the selectable regions associated with a menu option starting with the distinct sound "fu" is associated with a menu option not previously displayed and the newly associated menu option replaces the old menu option on the display.
Resuming now with the description of the example of the preferred embodiment, a cross hair cursor (3686) is displayed in the circumscribed region (3680). Assuming the operator desires to select the menu option associated with selectable region (3608/3610), he turns his head to the left and the cross hair cursor (3686) moves to the left, responsive to the head movement, until the cross hair cursor hotspot intersects selectable region (3608/3610) and he maintains the location of the cross hair cursor hotspot on that selectable region for the selection threshold period. The menu option associated with selectable region (3608/3610) is selected and added to text (3684) displayed in the circumscribed region (3680), the general purpose computer system (2218) emits an audible beep indicating that selection has occurred and the displayed menu options, both on the indicating regions and the visible subregions, are removed from the display.
In the preferred embodiment, selection is made in accord with the Facilitated Dwell subaspect of the Dwell aspect of the invention, described above. Thus, the operator receives an indication of the progress of his selection by a change in appearance of the indicating region associated with the intersected selectable region. Alternatively, selection may be by intersection of a location indicated by the at least part of a cursor and a selectable region alone, by such an intersection accompanied by a switch operation, for example, a depression of a space bar on the keyboard, or by other suitable means.
In accord with the Location Indication aspect of the invention, the operator sees the entire menu in the compact indicating regions and may discover the location of the selectable region associated with each menu option without having to visually scan all the visible subregions. In the preferred embodiment, the plurality of indicating regions may be moved to a different location on the display to avoid obstructing the area of the display showing most recently added graphic symbols or the area of the display where graphic symbols will soon be added. In accord with the Ideographic Language aspect of the invention, an operator may select from among many sequences of one or more ideographs without lifting either hand from the keyboard, thus speeding entry of single ideographs or sequences of ideographs in word processing systems for the Chinese, Japanese and Korean languages. Since, in the preferred embodiment, the selectable regions are located adjacent the edge of the display, a large rectangular region remains available on the display for the output of an application program. Further, if the selectable regions are located entirely outside the display, the indicating regions obstruct only a relatively small portion of the circumscribed region (3680), permitting the display of a sequences of ideographs for selection simultaneous with the display of previously selected ideographs, neither obstructing the operator's view of the other. If the general purpose computer system is coupled to a speech synthesizer and the ideographs are symbols of a symbol set, for example, the Blissymbolics Symbol Set, an illiterate operator, for example, a school child having impaired speech, may select symbols associated with words and those words may be spoken via the speech synthesizer.
FIG. 65 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention. In this Figure, 18 selectable regions (3704/3706), (3708/3710), (3712/3714), (3716/3718), (3720/3722), (3724/3726), (3728/3730), (3732/3734), (3736/3738), (3740/3742), (3744/3746), (3748/3750), (3752/3754), (3756/3758), (3760/3762), (3764/3766), (3768/3770), (3772/3774) and (3776/3778) circumscribe region (3780) on the display (2112). The visible subregions of four of the 18 selectable regions abut each of the top, left and right edges of the display (2112). The visible subregions of six of the 18 selectable regions abut the bottom edge of the display (2112). The 18 indicating regions are located within the top half of region (3780). 12 of the 18 indicating regions are arranged in two columns of six indicating regions each. The column on the left is close to the visible subregions abutting the left edge of the display. The column on the right is close to the visible subregions abutting the right edge of the display. Two of the 18 indicating regions are located between the top indicating regions in each of the left and right columns. The remaining four of the 18 indicating regions are located between the bottom indicating regions in each of the left and right columns. Each of the four indicating regions in the top row of indicating regions indicates the location of a respectively associated selectable region along the top edge of the display. Each of the six indicating regions in the bottom row of indicating regions indicates the location of a respectively associated selectable region along the bottom edge of the display. Each of the middle four indicating regions in each of the left and right columns of indicating regions indicates the location of a respectively associated selectable region along the left and right edge of the display. The 16 menu options shown in FIG. 65 are Chinese ideographs each of which includes the radical kou, a radical having the shape of a rectangle. Each ideograph is displayed on the visible subregion of the associated selectable region and on the indicating region associated with the selectable region. The remaining two of the 18 selectable regions, (3724/3726) and (3744/3746), are associated with menu options for undo and for displaying more menu options, respectively.
The indicating regions (3782) in FIG. 65 are disproportionately large relative to the rest of FIG. 65. They are approximately 1.5 times their proportionate size. They are represented as shown in FIG. 65 in compliance with Patent Cooperation Treaty Rules requiring a minimum size for letters in figures.
FIG. 66 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention. FIG. 66 depicts a display (2112) of a computer system (2116) having thereon ten square indicating regions, (3801), (3303), (3805), (3807), (3809), (3811), (3813), (3815), (3817), and (3819), arranged in two columns of five indicating regions. Each indicating region abuts an indicating region in the other column, the indicating region located above it, if any, and the indicating region located below it, if any. Each indicating region indicates the location of a respectively associated selectable region (not shown) outside the display. Assume that the operator is entering Chinese and uses a keyboard coupled to the computer system (2116) to enter the distinct sound "fu" . On each of the indicating regions is then displayed an ideograph having with the distinct sound "fu". The operator, using a pointer coupled to the computer system (2116), points to the selectable region associated with the indicating region on which is displayed the desired ideograph. The selectable region is selected in response to a selection event, and, in this example, the associated menu option is input to a word processing program and appears on the display (2112).
FIG. 67 illustrates a display and structures in accordance with an alternative embodiment of the Ideographic Language aspect of the invention. FIG. 67 depicts a display (2112) of a computer system (2116) having thereon eight selectable regions (6701), (6703), (6705), (6707), (6709), (6711), (6713), and (6715), each located on the display (2112) adjacent the edge of the display (2112) and associated respectively with a menu option. Together the eight selectable region circumscribe region (6750) on the display. In FIG. 67 each of eight menu options is displayed on its associated selectable region. Six of the eight menu options, associated with selectable regions (6703), (6705), (6707), (6701), (6715), and (6713) each represent a sequence of graphic symbols in the Japanese language. These sequences, in Romanized Japanese are respectively "hoka", "hoka ni", "hoka no", "hoka kara", "nanika hoka nomono", and "hoka demo nai ga". Each sequence includes the kanji "hoka" shown alone on selectable region (6703). The sequences are ordered by length, the shorter sequences on selectable regions on the left side of the display (2112) ordered by length from the top to the bottom of the display (2112), the longer sequences along the right side of the display (2112) ordered by length from the top to the bottom of the display (2112). The remaining two of the eight selectable regions, (6709) and (6711), are associated with menu options for displaying the previous and the next display of menu options, respectively. The menu option associated with a selectable region may be selected by a selection event.
The preferred embodiment of the Ideographic Language aspect of the invention will now be described in detail from an implementation perspective. Preferably, the Ideographic Language aspect of the invention is implemented by modifications to the access program (1206) described in the detailed description of the Perimeter Menu aspect of the invention, modified as described in the descriptions of the Facilitated Dwell subaspect and the Location Indication aspects of the invention. The modifications required for the Ideographic Language aspect of the invention are: (1) install on the general purpose computer system a font for the ideographic language of the embodiment; (2) at initialization time: (a) set the text of all labels for display on the selectable regions to null; (b) do not enable the cursor polling timer; and (c) hide the Windows® (1204) system cursor; (3) include in the main window processing procedure of the access program (1206) code to process WM-- CHAR messages and, when a sequence of one or more WM-- CHAR messages indicates a common characteristic: (a) lookup sequences or representations of sequences of one or more ideographs (hereinafter "sequences") having the common characteristic; (b) copy ten of the sequences to the labels; (c) set the Windows® (1204) system cursor to the cross hair cursor, set the cursor location to a predetermined location near an indicating region and show the Windows® (1204) system cursor; (d) set fpaint to TRUE for every state machine; (e) send EV-- RESET to every state machine; (f) display each sequence at the appropriate location in the indicating region; and (g) enable the cursor polling timer; and (4) add state processing to ST-- SELECTED to: (a) set the text of all labels for display to null; (b) hide the Windows® (1204) system cursor; (c) erase all sequences from the indicating region; (d) disable the cursor polling timer; (e) set fpaint to TRUE for every state machine; (f) send EV-- RESET to every state machine; and (g) insert the selected sequence into the work space.
In the above descriptions, there is shown and described only the preferred and certain alternate embodiments of each aspect of the invention, but, as aforementioned, it is to be understood that each aspect of the invention is capable of use in various other combinations and environments and is capable of changes or modifications within the scope of the inventive concepts as expressed herein.
______________________________________                                    
Appendix I                                                                
______________________________________                                    
Procedure: PocketFsm                                                      
/***********************************************************/             
/*                                     */                                 
/*  Procedure: PocketFsm               */                                 
/*                                     */                                 
/*  Description:                                                          
               State Machine for processing events to                     
                                       */                                 
/*             selectable regions.     */                                 
/*                                 */                                     
/*  Input Parameters:              */                                     
/*         pPocket  pointer to selectable region data                     
                                       */                                 
/*                  set                */                                 
/*         Event    event to process   */                                 
/*                                     */                                 
/***********************************************************/             
PocketFsm (pPocket, Event)                                                
 BOOL   fInternalEvent                                                    
                     /* declaration of local                              
                     boolean variable */                                  
 set fInternalEvent to TRUE                                               
 while (fInternalEvent is TRUE)                                           
 {                                                                        
  set fInternalEvent to FALSE                                             
  set pPocket->PreviousState to pPocket->State                            
  set pPocket->State to aPocketFsm[pPocket->State][Event]                 
  switch (pPocket->State)                                                 
  {                                                                       
   case ST.sub.-- ILLEGAL.sub.-- STATE:                                   
   case ST.sub.-- ERROR:                                                  
    turn off all timers                                                   
    log the error                                                         
    display an error message to the operator                              
    break                                                                 
   case ST.sub.-- INITIAL:                                                
    break                                                                 
   case ST.sub.-- RESET:                                                  
    set time of selection to the current time                             
    if pPocket->Color is not equal to                                     
    pPocket- >InitialColor                                                
     set pPocket->Color to pPocket->InitialColor                          
     set pPocket->fPaint to TRUE                                          
    set pPocket->fInvert to FALSE                                         
    break                                                                 
   case ST.sub.-- LOW.sub.-- TIDE:                                        
    break                                                                 
   case ST.sub.-- CREST.sub.-- TIDE:                                      
    break                                                                 
   case ST.sub.-- BEGIN.sub.-- LOCK:                                      
    emit an audible beep                                                  
    set the system cursor location to the visible                         
    subregion's lockspot                                                  
    set the system cursor to null                                         
    display the lock icon on the visible subregion's                      
    lockspot                                                              
               /* initialize counter for                                  
               locking */                                                 
    set iLockCursor to cLockCursor                                        
    break                                                                 
   case ST.sub.-- LOCK:                                                   
    set the system cursor location to the visible                         
    subregion's lockspot                                                  
               /* on lock counter                                         
               expiration, */                                             
               /* transfer to the next                                    
               state */                                                   
    decrement iLockCursor by 1                                            
    if iLockCursor equals 0                                               
     set Event to EV.sub.-- NULL                                          
     set fInternalEvent to TRUE                                           
    break                                                                 
   case ST.sub.-- END.sub.-- LOCK:                                        
    erase the lock icon with white                                        
    set the system cursor to the arrow cursor                             
    display the system cursor on the visible subregion'                   
    lockspot                                                              
    emit an audible beep                                                  
               /* paint selectable region so                              
               that **/                                                   
               /* the white erasure of the                                
               lock */                                                    
               /* cursor icon is                                          
               overwritten */                                             
    set pPocket->fPaint to TRUE                                           
    break                                                                 
   case ST.sub.-- SELECTED:                                               
    emit an audible beep                                                  
    take the action appropriate upon selection of this                    
    selectable region, for example, sending data to an                    
    application program, sending a control sequence to                    
    a device coupled to the computer, or displaying the                   
    selection                                                             
    set pPocket->fInvert to TRUE                                          
    if appropriate, change the menu options associated                    
    with various selectable regions and set fPaint to                     
    TRUE for the state machines associated with those                     
    selectable regions                                                    
    set Event to EV.sub.-- NULL                                           
    set fInternalEvent to TRUE                                            
    break                                                                 
   case ST.sub.-- SELECT.sub.-- AND.sub.-- OUT:                           
    send EV.sub.-- RESET then EV.sub.-- MOVEMENT to all state             
    machines, resetting them and indicating that the                      
    operator is moving the cursor                                         
    break                                                                 
   case ST DECAY:                                                         
               /* set state to previous                                   
               state */                                                   
               /* in preparation for the                                  
               next */                                                    
               /* state transition */                                     
    set pPocket->State to pPocket->PreviousState                          
    decrement pPocket->Color by pPocket->Decrement, but                   
    not below pPocket->InitialColor                                       
    if pPocket->Color was changed                                         
     set pPocket->fPaint to TRUE                                          
     set fInternalEvent to TRUE                                           
    if pPocket->Color was changed from a value greater                    
    than or equal to pPocket->CrestTide to a value below                  
    pPocket->CrestTide                                                    
     set Event to EV.sub.-- STEP.sub.-- DOWN                              
    else                                                                  
     set Event to EV.sub.-- NULL                                          
    break                                                                 
   case ST.sub.-- DWELL:                                                  
               /* set state to previous                                   
               state */                                                   
               /* in preparation for the                                  
               next */                                                    
               /* state transition */                                     
    set pPocket->State to pPocket->PreviousState                          
    increment pPocket->Color by pPocket->Increment, but                   
    not above pPocket->Ceiling                                            
    if pPocket->Color was changed                                         
     set pPocket->fPaint to TRUE                                          
     set fInternalEvent to TRUE                                           
    if pPocket->Color was changed from a value below                      
    pPocket->CrestTide to a value greater than or equal                   
    to pPocket->CrestTide                                                 
     set Event to EV.sub.-- STEP.sub.-- UP                                
    else                                                                  
     set Event to EV.sub.-- NULL                                          
    break                                                                 
   case ST.sub.-- IDLE:                                                   
    set the system cursor location to the center of the                   
    display                                                               
    set fInternalEvent to TRUE                                            
    set Event to EV.sub.-- NULL                                           
    break                                                                 
   case ST.sub.-- EBB.sub.-- TIDE:                                        
    break                                                                 
   case ST.sub.-- ENTRY:                                                  
    set fInternalEvent to TRUE                                            
    set Event to EV.sub.-- NULL                                           
    break                                                                 
   case ST.sub.-- DISCARD:                                                
               /* set state to previous state */                          
               /* for the next transition */                              
    set pPocket->State to pPocket->PreviousState                          
    break                                                                 
   case ST.sub.-- EXIT:                                                   
    set Event to EV.sub.-- NULL                                           
    set fInternalEvent to TRUE                                            
    if pPocket->fInterior equals TRUE                                     
     change the boundaries of the Windows ® region                    
     having the handle pPocket->hRegion to those                          
     defined by the second array of points associated                     
     with this state machine                                              
    set pPocket->fPaint to TRUE                                           
    send EV.sub.-- RESET to the state machine having the index            
    pPocket ->iAdjacentPocket                                             
    break                                                                 
   default:                                                               
    display error message                                                 
    break                                                                 
  }                                                                       
  /* end switch */                                                        
 }                                                                        
 /* end while */                                                          
 if pPocket->fPaint is equal to TRUE                                      
  invalidate the entire client area                                       
}                                                                         
/* end PocketFsm */                                                       
Procedure: CreateEvent                                                    
/***********************************************************/             
/*                                     */                                 
/*  Procedure:                                                            
              CreateEvent              */                                 
/*                                     */                                 
/*  Description:                                                          
              Examines movement of cursor and determines                  
                                       */                                 
/*            what events have occurred.                                  
                                       */                                 
/*                                     */                                 
/*  Input Parameters:              */                                     
/*        pEvent  pointer to an event data structure                      
                                       */                                 
/*                for output           */                                 
/*  Output:   A completed event data structure                            
                                       */                                 
/*            indicated by pEvent including:                              
                                       */                                 
/*            (1) an indication of whether the cursor hotspot             
                                       */                                 
/*              intersects a selectable region                            
                                       */                                 
/*            (2) the index of the intersected selectable                 
                                       */                                 
/*              region, if any         */                                 
/*            (3) the event for the intersected selectable                
                                       */                                 
/*              region, if any         */                                 
/*            (4) the event for all non-intersected                       
                                       */                                 
/*              selectable regions     */                                 
/*                                     */                                 
/*            Updated variables indicate:                                 
                                       */                                 
/*            (1) the previous location of the cursor hotspot             
                                       */                                 
/*            (2) the approximate time of the sampling of                 
                                       */e                                
/*              previous location of the cursor hotspot                   
                                       */                                 
/*                                     */                                 
/***********************************************************/             
CreateEvent (pEvent)                                                      
{                                                                         
 get current cursor hotspot location                                      
 if there has been cursor hotspot movement since the last cursor          
 location sampling                                                        
  set time of previous cursor movement to time latest received            
  WM.sub.-- TIMER message was generated                                   
  if cursor hotspot has crossed out of a selectable region                
   set event to EV.sub.-- CROSS.sub.-- OUT for the intersected            
selectable                                                                
   region and to EV.sub.-- MOVEMENT for all other selectable              
   regions                                                                
  else                                                                    
   if cursor hotspot intersects a selectable region                       
    if intersected selectable region's Color equals                       
    Ceiling                                                               
     set event to EV.sub.-- CEILING for the intersected                   
     selectable region and to EV.sub.-- DECAY for all other               
     selectable regions                                                   
    else                                                                  
     set event to EV.sub.-- DWELL for the intersected                     
     selectable region and to EV.sub.-- DECAY for all other               
     selectable regions                                                   
   else                                                                   
    set event to EV.sub.-- MOVEMENT for all selectable regions            
 else                                                                     
  if cursor hotspot intersects a selectable region                        
   if idle time exceeded                                                  
    set event to EV.sub.-- IDLE.sub.-- TIMEOUT for all selectable         
    regions                                                               
   else                                                                   
    if intersected selectable region's Color equals                       
    Ceiling                                                               
     set event to EV.sub.-- CEILING for the intersected                   
     selectable region and to EV.sub.-- DECAY for all other               
     selectable regions                                                   
    else                                                                  
     set event to EV.sub.-- DWELL for the intersected                     
     selectable region and to EV.sub.-- DECAY for all other               
     selectable regions                                                   
  else                                                                    
   set event to EV.sub.-- DECAY for all selectable regions                
 set the previous cursor hotspot location to the current hotspot          
 location                                                                 
}                                                                         
/* end CreateEvent */                                                     
______________________________________                                    

Claims (56)

I claim:
1. A method of selecting a sequence of one or more graphic symbols from a first plurality of sequences of one or more graphic symbols, one or more sequences of the first plurality of sequences including one or more ideographs in an ideographic language, said method comprising the steps of:
displaying on a display the first plurality of sequences, each of the first plurality of sequences having a first common characteristic in the ideographic language and each associated respectively with one of a first plurality of displayed sound indicators, each of the first sound indicators respectively indicating one of a first plurality of indicated sounds, each of the first indicated sounds differing from each other first indicated sound;
matching a first received sound to any one of the first indicated sounds; and
selecting the sequence associated with the matched first indicated sound.
2. The method of claim 1 wherein the first received sound is not a phonetic representation of the sequence associated with the matched first indicated sound.
3. The method of claim 1 wherein one of the first sound indicators is non-alphanumeric.
4. The method of claim 3 wherein the non-alphanumeric sound indicator is a hue.
5. The method of claim 3 wherein the non-alphanumeric indicator includes any one of a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch indicator, a change in volume indicator, a shape indicator, a color saturation indicator, a brightness indicator, a location indicator, and an object indicator.
6. The method of claim 1 further comprising the step of indicating the association of each of the first plurality of sequences with the associated sound indicator.
7. The method of claim 1 wherein each of the first plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.
8. The method of claim 1 wherein the first common characteristic includes a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation.
9. The method of claim 8 wherein the phonetic unit is specified by voice.
10. The method of claim 8 wherein each of the first sound indicators includes a hue of a region on the display each region having a hue different from the hue of all the regions of all other first sound indicators; wherein each region is either adjacent or nearly adjacent another of the regions; wherein the displaying step displays each sequence intersecting the region of the associated sound indicator; and wherein the first received sound is the name of the hue of the sound indicator to be matched.
11. The method of claim 10 further comprising the step of inputting the selected sequence to an application program.
12. The method of claim 8 wherein the phonetic unit is specified by one or more keystrokes.
13. The method of claim 8 wherein the phonetic unit is a phonetic unit in a first dialect of the ideographic language, and wherein the matching step includes matching the phonetic unit to a phonetic unit in a second dialect of the ideographic language.
14. The method of claim 1 wherein the first common characteristic includes any one of:
(a) an intonation;
(b) a distinct sound;
(c) a part of speech;
(d) a meaning; and
(e) a meaning class.
15. The method of claim 1 wherein the first common characteristic includes any one of:
(a) a stroke used to draw an ideograph;
(b) a number of horizontal strokes;
(c) a number of vertical strokes;
(d) a number of total strokes;
(e) a stroke order;
(f) a radical;
(g) an ideograph;
(h) a kana;
(i) a diacritic;
(j) a classification of a part of an ideograph;
(k) a root;
(l) a graphic symbol;
(m) an appearance of a graphic symbol; and
(n) an attribute for optical recognition purposes.
16. The method of claim 1 wherein the displaying step precedes the matching step.
17. The method of claim 1 wherein the first plurality of sound indicators together at least partially circumscribe a region on the display.
18. The method of claim 17 wherein each of the first sound indicators is outside the display.
19. The method of claim 1 wherein each of the first sound indicators is on the display and either adjacent or nearly adjacent another of the first sound indicators.
20. The method of claim 1 further comprising the steps of:
receiving a signal; and
responsive to the received signal, displaying on the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the first common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators respectively indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound.
21. The method of claim 20 wherein each sequence of the first plurality of sequences has a frequency of use greater than each sequence of the second plurality of sequences.
22. The method of claim 1 further comprising the steps of:
first storing a representation of the first common characteristics; and
second storing a representation of a second common characteristic.
23. The method of claim 22 wherein the first storing step and the second storing step precede the displaying step.
24. The method of claim 23 followed by the steps of:
displaying oil the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the second common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound;
matching a second received sound to any one of the second indicated sounds; and
selecting the sequence of the second plurality of sequences associated with the matched second indicated sound.
25. The method of claim 1 wherein one of the first sound indicators is numeric.
26. The method of claim 1 wherein the displaying step displays each sequence of the first plurality of sequences in an order responsive to a second characteristic of the sequence other than the first common characteristic, the second characteristic including any one of:
(a) a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation;
(b) an intonation;
(c) a distinct sound;
(d) a stroke used to draw an ideograph;
(e) a number of horizontal strokes;
(f) a number of vertical strokes;
(g) a number of total strokes;
(h) a stroke order;
(i) a radical;
(j) a part of speech;
(k) an ideograph;
(l) a kana;
(m) a diacritic;
(n) a classification of a part of an ideograph;
(o) a meaning;
(p) a meaning class;
(q) a root;
(r) a graphic symbol;
(s) an appearance of a graphic symbol;
(t) an attribute for optical recognition purposes;
(u) a frequency of use of each of the first plurality of sequences;
(v) a number of graphic symbols; and
(w) a frequency of use of each of a second plurality of sequences, of graphic symbols, each of the second plurality of sequences including one of the first plurality of sequences.
27. The method of claim 1 further comprising the step of inputting the selected sequence to an application program.
28. The method of claim 1 further comprising the steps of:
inputting a first sequence of one or more graphic symbols including one or more ideographs to an application program; and
replacing the first sequence with the selected sequence.
29. The method of claim 1 wherein the first common characteristic includes a first graphic symbol and the displaying step displays only a single instance of the first graphic symbol indicating that the first graphic symbol is included in each of the first plurality of sequences.
30. The method of claim 29 wherein the first graphic symbol is an ideograph.
31. The method of claim 29 wherein the displaying step displays the single instance of the first graphic symbol in a region of the display separate from the display of at least part of one of the first plurality of sequences or in a manner visibly distinct from the display of at least part of one of the first plurality of sequences.
32. The method of claim 1 wherein one of the first sound indicators includes at least part of a particular graphic symbol of the associated sequence.
33. The method of claim 32 wherein the first received sound includes any one of:
(a) a pronunciation of at least part of the particular graphic symbol;
(b) a name of at least part of the particular graphic symbol;
(c) a meaning of at least part of the particular graphic symbol;
(d) a meaning class of at least part of the particular graphic symbol;
(e) a part of speech of at least part of the particular graphic symbol;
(f) a classification of at least part of the particular graphic symbol;
(g) a stroke used to draw at least part of the particular graphic symbol;
(h) a number of horizontal strokes used to draw the particular graphic symbol;
(i) a number of vertical strokes used to draw the particular graphic symbol; and
(j) a number of total strokes used to draw the particular graphic symbol.
34. The method of claim 1 wherein the number of sequences in the first plurality of sequences is less than or equal to a predetermined number thereby increasing the likelihood of matching of the first received sound to the first indicated sound intended by a user.
35. For use in a language rich in homophones, a method of selecting a sequence of one or more graphic symbols in the language from a plurality of sequences of one or more graphic symbols in the language, said method comprising the steps of:
(a) displaying the plurality of sequences, each of the plurality of sequences associated respectively with a sound which is not a phonetic representation of the associated sequence, each of the plurality of sounds differing from each other sound in the plurality of sounds;
(b) receiving a sound signal;
(c) matching the received sound signal to any one of the plurality of sounds; and
(d) selecting the sequence associated with the matched sound.
36. The method of claim 35 wherein at least two of the sequences are homophones.
37. The method of claim 35 wherein each sequence represents a word in the language.
38. The method of claim 35 further comprising the steps of:
(e) displaying a plurality of sound indicators, each sound indicator associated respectively with one of the plurality of sounds; and
(f) indicating the association of each sequence to the sound indicator associated with the sound associated with the sequence.
39. A method of editing a first sequence of two or more words in an ideographic language, said method comprising the steps of:
displaying on a display:
(a) the first sequence, the first sequence including a plurality of second sequences of one or more graphic symbols in the language, each of the second sequences associated respectively with a sound indicator, each of the sound indicators respectively indicating a sound, each of the indicated sounds differing from each of the other indicated sounds; and
(b) the sound indicators;
receiving a sound;
matching the received sound to any one of the indicated sounds; and
replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.
40. For use with a general purpose computer system including a display, an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including an ideograph in an ideographic language, said apparatus comprising:
(a) a computer readable medium; and
(b) a program, stored on the medium and executable by the general purpose computer system, for:
(1) displaying the plurality of sequences on the display, each of the sequences associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator;
(2) matching a received sound to the sound indicated by the particular sound indicator; and
(3) selecting the sequence associated with the particular sound indicator.
41. The apparatus of claim 40 wherein each of the plurality of sequences has a common characteristic in the ideographic language.
42. The apparatus of claim 40 wherein the medium may be coupled to and uncoupled from the general purpose computer system.
43. The apparatus of claim 40 wherein the medium includes any one of:
(a) a random access memory;
(b) a magnetic store; and
(c) an optical store.
44. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:
(a) a display on which may be displayed the plurality of sequences, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and
(b) a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.
45. The apparatus of claim 44 wherein the common characteristic includes any one of the meaning classes:
(a) actions;
(b) amounts;
(c) animals;
(d) articles of clothing;
(e) bodily functions;
(f) buildings;
(g) business activities;
(h) cleaning activities;
(i) colors;
(j) communication activities;
(k) computer peripherals;
(l) days;
(m) devices used to maintain personal hygiene;
(n) directions;
(o) drinks;
(p) emergency conditions;
(q) emotions;
(r) financial activities;
(s) foods;
(t) government services;
(u) greetings;
(v) holidays;
(w) household appliances;
(x) illnesses;
(y) items of office equipment;
(z) jokes;
(aa) lengths;
(ab) locations, including locations frequented by a user of the apparatus;
(ac) meals;
(ad) means of transportation;
(ae) months;
(af) names;
(ag) numbers;
(ah) parts of the human body;
(ai) persons known to a user of the apparatus;
(aj) plants;
(ak) prosthetic device;
(al) recreational activities;
(am) rehabilitation activities;
(an) relative locations;
(ao) school activities;
(ap) shapes;
(aq) shopping activities;
(ar) sizes;
(as) smells;
(at) sports;
(au) tactile attributes;
(av) tastes;
(aw) telephone numbers;
(ax) temperatures;
(ay) times;
(az) topics of study;
(ba) utterances used as acknowledgements in conversation without conveying new substantive information;
(bb) utterances used to bid for a turn to speak in conversation;
(bc) weights; and
(bd) work activities.
46. The apparatus of claim 44 further comprising:
(a) a computer system including the display and a sound receiver for receiving the received sound; and
(b) a program, executable on the computer system;
and wherein the selection device is formed by the combination of the progam and the computer system.
47. The apparatus of claim 44 wherein the selection device includes a processor, a program executable oil the processor, and a medium readable by the processor; wherein the processor, during execution of the program, compares a representation of the received sound to a representation of each of the indicated sounds, the representation of each of the indicated sounds stored on the medium and read by the processor therefrom; and wherein the sound indicated by tile particular sound indicator is tile indicated sound whose representation is closest to the representation of the received sound.
48. The apparatus of claim 44 wherein the selection device includes a speech recognizer for comparing the received sound to the sound indicated by the particular sound indicator.
49. The apparatus of claim 40 wherein the program is transferred from the medium to the general purpose computer system via a telecommunications or data communications network.
50. The apparatus of claim 44 wherein the sound indicated by the particular sound indicator is either one of:
(a) a sound found in normal speech; and
(b) a sound not found in normal speech.
51. The apparatus of claim 44 wherein each of the plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.
52. The apparatus of claim 44 wherein the plurality of sound indicators together at least partially circumscribe a region on the display.
53. The apparatus of claim 52 wherein each of the sound indicators is outside the display.
54. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of tile plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:
(a) means for displaying tile plurality of sequences, each of tile plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with tile particular sound indicator: and
(b) means for:
(1) matching a sound to the sound indicated by the particular sound indicator; and
(2) selecting the sequence associated with the particular sound indicator.
55. A device controller comprising:
(a) a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and
(b) a signal generating device, coupled to a controlled device, for:
(1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and
(2) generating a device control signal corresponding to the function of the controlled device.
56. The apparatus of claim 55 wherein the controlled device includes any one of:
(a) a computer peripheral;
(b) a device capable of playing previously recorded sound;
(c) a device capable of playing previously recorded video;
(d) a household appliance;
(e) a lamp;
(f) a microprocessor;
(g) a motorized transport device including either one of a scooter and a wheelchair;
(h) a radio;
(i) a robot;
(j) a security system;
(k) a television;
(l) a thermostat;
(m) a voice output device;
(n) a workstation;
(o) an alarm; and
(p) an office appliance.
US08/506,152 1995-07-24 1995-07-24 Sound operated menu method and apparatus Expired - Lifetime US5999895A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US08/506,152 US5999895A (en) 1995-07-24 1995-07-24 Sound operated menu method and apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US08/506,152 US5999895A (en) 1995-07-24 1995-07-24 Sound operated menu method and apparatus

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1995/003591 Continuation-In-Part WO1996030822A1 (en) 1995-03-27 1995-03-27 Method of and apparatus for data entry

Publications (1)

Publication Number Publication Date
US5999895A true US5999895A (en) 1999-12-07

Family

ID=24013412

Family Applications (1)

Application Number Title Priority Date Filing Date
US08/506,152 Expired - Lifetime US5999895A (en) 1995-07-24 1995-07-24 Sound operated menu method and apparatus

Country Status (1)

Country Link
US (1) US5999895A (en)

Cited By (180)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6163758A (en) * 1998-06-05 2000-12-19 Tektronix, Inc. Detection of unusual waveforms
US6285367B1 (en) * 1998-05-26 2001-09-04 International Business Machines Corporation Method and apparatus for displaying and navigating a graph
US6393125B1 (en) 1997-07-11 2002-05-21 Zarlink Semiconductor Inc. Initializer for a confusion data generator
US6446056B1 (en) * 1999-09-10 2002-09-03 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US6514078B1 (en) * 2001-11-02 2003-02-04 Jacqueline E. Jones Electronic educational device
US6594649B2 (en) 1998-08-19 2003-07-15 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US6604091B2 (en) 1999-09-10 2003-08-05 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US20030197740A1 (en) * 2002-04-22 2003-10-23 Nokia Corporation System and method for navigating applications using a graphical user interface
WO2003090182A1 (en) * 2002-04-18 2003-10-30 Elia Life Technology Dynamic tactile and low vision fonts
US20030210272A1 (en) * 2002-05-09 2003-11-13 Gateway, Inc. System and method for providing a user interface
US6745053B1 (en) * 1998-12-17 2004-06-01 Nec Corporation Mobile communication terminal apparatus having character string editing function by use of speech recognition function
US20040161151A1 (en) * 2002-04-16 2004-08-19 Fujitsu Limited On-line handwritten character pattern recognizing and editing apparatus and method, and recording medium storing computer-executable program for realizing the method
US20040225504A1 (en) * 2003-05-09 2004-11-11 Junqua Jean-Claude Portable device for enhanced security and accessibility
US20040235530A1 (en) * 2003-05-23 2004-11-25 General Motors Corporation Context specific speaker adaptation user interface
US20040263480A1 (en) * 2003-06-24 2004-12-30 International Business Machines Corporation Method and system for providing integrated hot key configuration
US20050015254A1 (en) * 2003-07-18 2005-01-20 Apple Computer, Inc. Voice menu system
US20050027524A1 (en) * 2003-07-30 2005-02-03 Jianchao Wu System and method for disambiguating phonetic input
US20050027534A1 (en) * 2003-07-30 2005-02-03 Meurs Pim Van Phonetic and stroke input methods of Chinese characters and phrases
US20050091063A1 (en) * 2001-04-06 2005-04-28 Heinz-Jurgen Bergemann Program-controlled household appliance and method of operating the appliance
US7013258B1 (en) * 2001-03-07 2006-03-14 Lenovo (Singapore) Pte. Ltd. System and method for accelerating Chinese text input
US20060204137A1 (en) * 2005-03-07 2006-09-14 Hitachi, Ltd. Portable terminal and information-processing device, and system
US20070038456A1 (en) * 2005-08-12 2007-02-15 Delta Electronics, Inc. Text inputting device and method employing combination of associated character input method and automatic speech recognition method
US20070061139A1 (en) * 2005-09-14 2007-03-15 Delta Electronics, Inc. Interactive speech correcting method
US7292980B1 (en) * 1999-04-30 2007-11-06 Lucent Technologies Inc. Graphical user interface and method for modifying pronunciations in text-to-speech and speech recognition systems
US20080051197A1 (en) * 2006-08-15 2008-02-28 Mark Jawad Systems and methods for reducing jitter associated with a control device
US20080098331A1 (en) * 2005-09-16 2008-04-24 Gregory Novick Portable Multifunction Device with Soft Keyboards
US20080201662A1 (en) * 2007-02-13 2008-08-21 Harman Becker Automotive Systems Gmbh Methods for controlling a navigation system
US20080300885A1 (en) * 2007-05-30 2008-12-04 Chung-Hung Shih Speech communication system for patients having difficulty in speaking or writing
US20090099838A1 (en) * 2004-06-25 2009-04-16 Nokia Corporation Text messaging device
US7614008B2 (en) * 2004-07-30 2009-11-03 Apple Inc. Operation of a computer with touch screen interface
US20090327964A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Moving radial menus
US20090327955A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Selecting Menu Items
US20090327963A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Radial menu selection
US20100306702A1 (en) * 2009-05-29 2010-12-02 Peter Warner Radial Menus
US7899674B1 (en) * 2006-08-11 2011-03-01 The United States Of America As Represented By The Secretary Of The Navy GUI for the semantic normalization of natural language
US20110163973A1 (en) * 2010-01-06 2011-07-07 Bas Ording Device, Method, and Graphical User Interface for Accessing Alternative Keys
US20110173002A1 (en) * 2010-01-12 2011-07-14 Denso Corporation In-vehicle device and method for modifying display mode of icon indicated on the same
US8013840B1 (en) * 2000-04-06 2011-09-06 Microsoft Corporation User notification system with an illuminated computer input device
US20110320984A1 (en) * 2010-06-29 2011-12-29 Pourang Irani Selectable Parent and Submenu Object Display Method
US20120089400A1 (en) * 2010-10-06 2012-04-12 Caroline Gilles Henton Systems and methods for using homophone lexicons in english text-to-speech
US20120272144A1 (en) * 2011-04-20 2012-10-25 Microsoft Corporation Compact control menu for touch-enabled command execution
US8479122B2 (en) 2004-07-30 2013-07-02 Apple Inc. Gestures for touch sensitive input devices
US8494857B2 (en) 2009-01-06 2013-07-23 Regents Of The University Of Minnesota Automatic measurement of speech fluency
US20130253909A1 (en) * 2012-03-23 2013-09-26 Tata Consultancy Services Limited Second language acquisition system
US8560974B1 (en) * 2011-10-06 2013-10-15 Google Inc. Input method application for a touch-sensitive user interface
US8612856B2 (en) 2004-07-30 2013-12-17 Apple Inc. Proximity detector in handheld device
CN103997668A (en) * 2014-02-25 2014-08-20 华为技术有限公司 Method for displaying selection of mobile equipment, and terminal equipment
US8892446B2 (en) 2010-01-18 2014-11-18 Apple Inc. Service orchestration for intelligent automated assistant
US8898568B2 (en) 2008-09-09 2014-11-25 Apple Inc. Audio user interface
WO2015073880A1 (en) * 2013-11-15 2015-05-21 Kopin Corporation Head-tracking based selection technique for head mounted displays (hmd)
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US9244536B2 (en) 2007-01-05 2016-01-26 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US9292111B2 (en) 1998-01-26 2016-03-22 Apple Inc. Gesturing with a multipoint sensing device
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9383816B2 (en) 2013-11-15 2016-07-05 Kopin Corporation Text selection using HMD head-tracker and voice-command
USD763266S1 (en) * 2013-09-03 2016-08-09 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9576593B2 (en) 2012-03-15 2017-02-21 Regents Of The University Of Minnesota Automated verbal fluency assessment
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US20170068430A1 (en) * 2015-09-08 2017-03-09 Apple Inc. Device, Method, and Graphical User Interface for Providing Audiovisual Feedback
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
JP2018010512A (en) * 2016-07-14 2018-01-18 レノボ・シンガポール・プライベート・リミテッド Information processor, false input inhibiting method, and program
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9904360B2 (en) 2013-11-15 2018-02-27 Kopin Corporation Head tracking based gesture control techniques for head mounted displays
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10025501B2 (en) 2008-06-27 2018-07-17 Apple Inc. Touch screen device, method, and graphical user interface for inserting a character from an alternate keyboard
US20180225033A1 (en) * 2017-02-08 2018-08-09 Fuji Xerox Co., Ltd. Information processing apparatus and non-transitory computer readable medium
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US20180246569A1 (en) * 2017-02-27 2018-08-30 Fuji Xerox Co., Ltd. Information processing apparatus and method and non-transitory computer readable medium
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10073592B2 (en) 2015-06-18 2018-09-11 Apple Inc. Device, method, and graphical user interface for navigating media content
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US10209955B2 (en) 2013-11-15 2019-02-19 Kopin Corporation Automatic speech recognition (ASR) feedback for head mounted displays (HMD)
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10254953B2 (en) 2013-01-21 2019-04-09 Keypoint Technologies India Pvt. Ltd. Text input method using continuous trace across two or more clusters of candidate words to select two or more words to form a sequence, wherein the candidate words are arranged based on selection probabilities
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10283110B2 (en) 2009-07-02 2019-05-07 Apple Inc. Methods and apparatuses for automatic speech recognition
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US10338778B2 (en) * 2008-09-25 2019-07-02 Apple Inc. Collaboration system
US10354011B2 (en) 2016-06-09 2019-07-16 Apple Inc. Intelligent automated assistant in a home environment
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US10474333B2 (en) 2015-09-08 2019-11-12 Apple Inc. Devices, methods, and graphical user interfaces for moving a current focus using a touch-sensitive remote control
US10474355B2 (en) 2013-01-21 2019-11-12 Keypoint Technologies India Pvt. Ltd. Input pattern detection over virtual keyboard for candidate word identification
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10521466B2 (en) 2016-06-11 2019-12-31 Apple Inc. Data driven natural language event detection and classification
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US10568032B2 (en) 2007-04-03 2020-02-18 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US10607140B2 (en) 2010-01-25 2020-03-31 Newvaluexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10706373B2 (en) 2011-06-03 2020-07-07 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10733993B2 (en) 2016-06-10 2020-08-04 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US10791216B2 (en) 2013-08-06 2020-09-29 Apple Inc. Auto-activating smart responses based on activities from remote devices
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US10791176B2 (en) 2017-05-12 2020-09-29 Apple Inc. Synchronization and task delegation of a digital assistant
US10810274B2 (en) 2017-05-15 2020-10-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US11079933B2 (en) 2008-01-09 2021-08-03 Apple Inc. Method, device, and graphical user interface providing word recommendations for text input
US11210961B2 (en) 2018-03-12 2021-12-28 Neurological Rehabilitation Virtual Reality, LLC Systems and methods for neural pathways creation/reinforcement by neural detection with virtual feedback
US11256322B2 (en) * 2018-05-09 2022-02-22 Neurological Rehabilitation Virtual Reality, LLC Systems and methods for responsively adaptable virtual environments
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US11922006B2 (en) 2018-09-26 2024-03-05 Apple Inc. Media control for screensavers on an electronic device

Citations (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3541521A (en) * 1967-12-11 1970-11-17 Bunker Ramo Multiple ratio cursor control system
US4109145A (en) * 1974-05-20 1978-08-22 Honeywell Inc. Apparatus being controlled by movement of the eye
JPS5419618A (en) * 1977-07-15 1979-02-14 Hitachi Ltd Cursor control unit
US4181813A (en) * 1978-05-08 1980-01-01 John Marley System and method for speech recognition
US4291198A (en) * 1979-06-28 1981-09-22 Bell Telephone Laboratories, Incorporated General-purpose electronic telephone station set
US4310839A (en) * 1979-11-23 1982-01-12 Raytheon Company Interactive display system with touch data entry
JPS5759241A (en) * 1980-09-27 1982-04-09 Nobuhiko Sasaki Inputting method for kanji (chinese character)
JPS5773453A (en) * 1980-10-24 1982-05-08 Nec Corp Input device of character information
US4336421A (en) * 1980-04-08 1982-06-22 Threshold Technology, Inc. Apparatus and method for recognizing spoken words
US4345313A (en) * 1980-04-28 1982-08-17 Xerox Corporation Image processing method and apparatus having a digital airbrush for touch up
US4354184A (en) * 1979-09-28 1982-10-12 Siemens Aktiengesellschaft Display system for localizing regions in a mixed text & picture display
US4354185A (en) * 1979-09-28 1982-10-12 Siemens Aktiengesellschaft Display system for localizing regions in a mixed text and picture display
US4386346A (en) * 1981-03-27 1983-05-31 International Business Machines Corporation Cursor controller
US4393271A (en) * 1978-02-14 1983-07-12 Nippondenso Co., Ltd. Method for selectively displaying a plurality of information
GB2116341A (en) * 1982-02-25 1983-09-21 Bock James E Interactive chinese typewriter
JPS59149338A (en) * 1983-02-15 1984-08-27 Konishiroku Photo Ind Co Ltd Detector for abnormality of exposure lamp in electrophotographic copying device
US4531119A (en) * 1981-06-05 1985-07-23 Hitachi, Ltd. Method and apparatus for key-inputting Kanji
US4555193A (en) * 1983-10-31 1985-11-26 Stone Philip J Keyboards including color coding and one handed operation
US4559598A (en) * 1983-02-22 1985-12-17 Eric Goldwasser Method of creating text using a computer
US4586035A (en) * 1984-02-29 1986-04-29 International Business Machines Corporation Display terminal with a cursor responsive virtual distributed menu
US4587520A (en) * 1983-04-07 1986-05-06 Rca Corporation Cursor controlled page selection in a video display
US4591841A (en) * 1983-11-01 1986-05-27 Wisconsin Alumni Research Foundation Long range optical pointing for video screens
JPS61121185A (en) * 1984-11-19 1986-06-09 Canon Inc Input and output device for character graphic
US4595990A (en) * 1980-12-31 1986-06-17 International Business Machines Corporation Eye controlled information transfer
GB2173023A (en) * 1985-03-25 1986-10-01 Intech Systems Inc Method and apparatus for selecting and constructing language symbols and characters
US4638312A (en) * 1985-10-25 1987-01-20 Ncr Corporation Order entry system including an interactive menu display
US4648028A (en) * 1984-08-31 1987-03-03 General Electric Co. Color enhanced display for a numerical control system
US4651145A (en) * 1984-05-31 1987-03-17 Medical Research Institute Communication system for the disabled in which a display target is selected by encephalogram response
US4665402A (en) * 1982-05-18 1987-05-12 Litton Systems Canada, Ltd. Radar signal processing system
US4679951A (en) * 1979-11-06 1987-07-14 Cornell Research Foundation, Inc. Electronic keyboard system and method for reproducing selected symbolic language characters
US4720189A (en) * 1986-01-07 1988-01-19 Northern Telecom Limited Eye-position sensor
US4748502A (en) * 1986-08-18 1988-05-31 Sentient Systems Technology, Inc. Computer vision system based upon solid state image sensor
US4751503A (en) * 1984-12-24 1988-06-14 Xerox Corporation Image processing method with improved digital airbrush touch up
US4788649A (en) * 1985-01-22 1988-11-29 Shea Products, Inc. Portable vocalizing device
US4829576A (en) * 1986-10-21 1989-05-09 Dragon Systems, Inc. Voice recognition system
EP0324306A2 (en) * 1987-11-16 1989-07-19 International Business Machines Corporation Parallax error avoidance for a touch screen system
US4931783A (en) * 1988-07-26 1990-06-05 Apple Computer, Inc. Method and apparatus for removable menu window
US4979094A (en) * 1987-04-07 1990-12-18 Possum Controls Limited Control system
US4987411A (en) * 1987-07-02 1991-01-22 Kabushiki Kaisha Toshiba Pointing apparatus
US5012231A (en) * 1988-12-20 1991-04-30 Golemics, Inc. Method and apparatus for cursor motion having variable response
US5020107A (en) * 1989-12-04 1991-05-28 Motorola, Inc. Limited vocabulary speech recognition system
US5027109A (en) * 1989-06-08 1991-06-25 Donovan Paul M Apparatus and method for minimizing undesired cursor movement in a computer controlled display system
US5027406A (en) * 1988-12-06 1991-06-25 Dragon Systems, Inc. Method for interactive speech recognition and training
US5122951A (en) * 1989-07-14 1992-06-16 Sharp Kabushiki Kaisha Subject and word associating devices
US5146210A (en) * 1989-08-22 1992-09-08 Deutsche Itt Industries Gmbh Wireless remote control system for a television receiver
US5164900A (en) * 1983-11-14 1992-11-17 Colman Bernath Method and device for phonetically encoding Chinese textual data for data processing entry
US5175688A (en) * 1988-11-25 1992-12-29 Kabushiki Kaisha Okuma Tekkosho Graphic display method of machining state in numerical control grinding machine
US5177328A (en) * 1990-06-28 1993-01-05 Kabushiki Kaisha Toshiba Information processing apparatus
US5195179A (en) * 1986-01-29 1993-03-16 Hitachi, Ltd. Coordinate input apparatus
US5220361A (en) * 1991-06-05 1993-06-15 Allergan Humphrey Gaze tracking for field analyzer
US5223828A (en) * 1991-08-19 1993-06-29 International Business Machines Corporation Method and system for enabling a blind computer user to handle message boxes in a graphical user interface
US5233662A (en) * 1991-11-05 1993-08-03 Christensen Chris S Oral controller method and apparatus
DE9300231U1 (en) * 1993-01-11 1993-08-19 Hessabi Iradj counter
US5254983A (en) * 1991-02-05 1993-10-19 Hewlett-Packard Company Digitally synthesized gray scale for raster scan oscilloscope displays
US5257314A (en) * 1989-12-06 1993-10-26 Fujitsu Limited Voice recognition system having word frequency and intermediate result display features
US5266931A (en) * 1991-05-09 1993-11-30 Sony Corporation Apparatus and method for inputting data
US5285265A (en) * 1990-12-15 1994-02-08 Samsung Electronics, Co. Ltd. Display apparatus informing of programmed recording
US5287119A (en) * 1987-10-14 1994-02-15 Wang Laboratories, Inc. Computer input device using an orientation sensor
US5289168A (en) * 1990-01-23 1994-02-22 Crosfield Electronics Ltd. Image handling apparatus and controller for selecting display mode
US5300943A (en) * 1986-10-03 1994-04-05 Goldstar Electron Co., Ltd. Multiple display workstation with conductive surface overlay control
US5305435A (en) * 1990-07-17 1994-04-19 Hewlett-Packard Company Computer windows management system and method for simulating off-screen document storage and retrieval
JPH06149534A (en) * 1992-11-13 1994-05-27 Toshiba Corp Voice screen operation unit
US5329609A (en) * 1990-07-31 1994-07-12 Fujitsu Limited Recognition apparatus with function of displaying plural recognition candidates
US5387896A (en) * 1990-08-06 1995-02-07 Tektronix, Inc. Rasterscan display with adaptive decay
US5396264A (en) * 1994-01-03 1995-03-07 Motorola, Inc. Automatic menu item sequencing method
US5408582A (en) * 1990-07-30 1995-04-18 Colier; Ronald L. Method and apparatus adapted for an audibly-driven, handheld, keyless and mouseless computer for performing a user-centered natural computer language
US5428707A (en) * 1992-11-13 1995-06-27 Dragon Systems, Inc. Apparatus and methods for training speech recognition systems and their users and otherwise improving speech recognition performance
US5429513A (en) * 1994-02-10 1995-07-04 Diaz-Plaza; Ruth R. Interactive teaching apparatus and method for teaching graphemes, grapheme names, phonemes, and phonetics
US5454062A (en) * 1991-03-27 1995-09-26 Audio Navigation Systems, Inc. Method for recognizing spoken words
JPH07295784A (en) * 1994-04-25 1995-11-10 Hitachi Ltd Information processor by voice
US5586198A (en) * 1993-08-24 1996-12-17 Lakritz; David Method and apparatus for identifying characters in ideographic alphabet

Patent Citations (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3541521A (en) * 1967-12-11 1970-11-17 Bunker Ramo Multiple ratio cursor control system
US4109145A (en) * 1974-05-20 1978-08-22 Honeywell Inc. Apparatus being controlled by movement of the eye
JPS5419618A (en) * 1977-07-15 1979-02-14 Hitachi Ltd Cursor control unit
US4393271A (en) * 1978-02-14 1983-07-12 Nippondenso Co., Ltd. Method for selectively displaying a plurality of information
US4181813A (en) * 1978-05-08 1980-01-01 John Marley System and method for speech recognition
US4291198A (en) * 1979-06-28 1981-09-22 Bell Telephone Laboratories, Incorporated General-purpose electronic telephone station set
US4354184A (en) * 1979-09-28 1982-10-12 Siemens Aktiengesellschaft Display system for localizing regions in a mixed text & picture display
US4354185A (en) * 1979-09-28 1982-10-12 Siemens Aktiengesellschaft Display system for localizing regions in a mixed text and picture display
US4679951A (en) * 1979-11-06 1987-07-14 Cornell Research Foundation, Inc. Electronic keyboard system and method for reproducing selected symbolic language characters
US4310839A (en) * 1979-11-23 1982-01-12 Raytheon Company Interactive display system with touch data entry
US4336421A (en) * 1980-04-08 1982-06-22 Threshold Technology, Inc. Apparatus and method for recognizing spoken words
US4345313A (en) * 1980-04-28 1982-08-17 Xerox Corporation Image processing method and apparatus having a digital airbrush for touch up
JPS5759241A (en) * 1980-09-27 1982-04-09 Nobuhiko Sasaki Inputting method for kanji (chinese character)
JPS5773453A (en) * 1980-10-24 1982-05-08 Nec Corp Input device of character information
US4595990A (en) * 1980-12-31 1986-06-17 International Business Machines Corporation Eye controlled information transfer
US4386346A (en) * 1981-03-27 1983-05-31 International Business Machines Corporation Cursor controller
US4531119A (en) * 1981-06-05 1985-07-23 Hitachi, Ltd. Method and apparatus for key-inputting Kanji
GB2116341A (en) * 1982-02-25 1983-09-21 Bock James E Interactive chinese typewriter
US4665402A (en) * 1982-05-18 1987-05-12 Litton Systems Canada, Ltd. Radar signal processing system
JPS59149338A (en) * 1983-02-15 1984-08-27 Konishiroku Photo Ind Co Ltd Detector for abnormality of exposure lamp in electrophotographic copying device
US4559598A (en) * 1983-02-22 1985-12-17 Eric Goldwasser Method of creating text using a computer
US4587520A (en) * 1983-04-07 1986-05-06 Rca Corporation Cursor controlled page selection in a video display
US4555193A (en) * 1983-10-31 1985-11-26 Stone Philip J Keyboards including color coding and one handed operation
US4591841A (en) * 1983-11-01 1986-05-27 Wisconsin Alumni Research Foundation Long range optical pointing for video screens
US5164900A (en) * 1983-11-14 1992-11-17 Colman Bernath Method and device for phonetically encoding Chinese textual data for data processing entry
US4586035A (en) * 1984-02-29 1986-04-29 International Business Machines Corporation Display terminal with a cursor responsive virtual distributed menu
US4651145A (en) * 1984-05-31 1987-03-17 Medical Research Institute Communication system for the disabled in which a display target is selected by encephalogram response
US4648028A (en) * 1984-08-31 1987-03-03 General Electric Co. Color enhanced display for a numerical control system
JPS61121185A (en) * 1984-11-19 1986-06-09 Canon Inc Input and output device for character graphic
US4751503A (en) * 1984-12-24 1988-06-14 Xerox Corporation Image processing method with improved digital airbrush touch up
US4788649A (en) * 1985-01-22 1988-11-29 Shea Products, Inc. Portable vocalizing device
GB2173023A (en) * 1985-03-25 1986-10-01 Intech Systems Inc Method and apparatus for selecting and constructing language symbols and characters
US4638312A (en) * 1985-10-25 1987-01-20 Ncr Corporation Order entry system including an interactive menu display
US4720189A (en) * 1986-01-07 1988-01-19 Northern Telecom Limited Eye-position sensor
US5195179A (en) * 1986-01-29 1993-03-16 Hitachi, Ltd. Coordinate input apparatus
US4748502A (en) * 1986-08-18 1988-05-31 Sentient Systems Technology, Inc. Computer vision system based upon solid state image sensor
US5300943A (en) * 1986-10-03 1994-04-05 Goldstar Electron Co., Ltd. Multiple display workstation with conductive surface overlay control
US4829576A (en) * 1986-10-21 1989-05-09 Dragon Systems, Inc. Voice recognition system
US4979094A (en) * 1987-04-07 1990-12-18 Possum Controls Limited Control system
US4987411A (en) * 1987-07-02 1991-01-22 Kabushiki Kaisha Toshiba Pointing apparatus
US5287119A (en) * 1987-10-14 1994-02-15 Wang Laboratories, Inc. Computer input device using an orientation sensor
EP0324306A2 (en) * 1987-11-16 1989-07-19 International Business Machines Corporation Parallax error avoidance for a touch screen system
US4931783A (en) * 1988-07-26 1990-06-05 Apple Computer, Inc. Method and apparatus for removable menu window
US5175688A (en) * 1988-11-25 1992-12-29 Kabushiki Kaisha Okuma Tekkosho Graphic display method of machining state in numerical control grinding machine
US5027406A (en) * 1988-12-06 1991-06-25 Dragon Systems, Inc. Method for interactive speech recognition and training
US5012231A (en) * 1988-12-20 1991-04-30 Golemics, Inc. Method and apparatus for cursor motion having variable response
US5027109A (en) * 1989-06-08 1991-06-25 Donovan Paul M Apparatus and method for minimizing undesired cursor movement in a computer controlled display system
US5122951A (en) * 1989-07-14 1992-06-16 Sharp Kabushiki Kaisha Subject and word associating devices
US5146210A (en) * 1989-08-22 1992-09-08 Deutsche Itt Industries Gmbh Wireless remote control system for a television receiver
US5020107A (en) * 1989-12-04 1991-05-28 Motorola, Inc. Limited vocabulary speech recognition system
US5257314A (en) * 1989-12-06 1993-10-26 Fujitsu Limited Voice recognition system having word frequency and intermediate result display features
US5289168A (en) * 1990-01-23 1994-02-22 Crosfield Electronics Ltd. Image handling apparatus and controller for selecting display mode
US5177328A (en) * 1990-06-28 1993-01-05 Kabushiki Kaisha Toshiba Information processing apparatus
US5305435A (en) * 1990-07-17 1994-04-19 Hewlett-Packard Company Computer windows management system and method for simulating off-screen document storage and retrieval
US5408582A (en) * 1990-07-30 1995-04-18 Colier; Ronald L. Method and apparatus adapted for an audibly-driven, handheld, keyless and mouseless computer for performing a user-centered natural computer language
US5329609A (en) * 1990-07-31 1994-07-12 Fujitsu Limited Recognition apparatus with function of displaying plural recognition candidates
US5387896A (en) * 1990-08-06 1995-02-07 Tektronix, Inc. Rasterscan display with adaptive decay
US5285265A (en) * 1990-12-15 1994-02-08 Samsung Electronics, Co. Ltd. Display apparatus informing of programmed recording
US5254983A (en) * 1991-02-05 1993-10-19 Hewlett-Packard Company Digitally synthesized gray scale for raster scan oscilloscope displays
US5454062A (en) * 1991-03-27 1995-09-26 Audio Navigation Systems, Inc. Method for recognizing spoken words
US5266931A (en) * 1991-05-09 1993-11-30 Sony Corporation Apparatus and method for inputting data
US5220361A (en) * 1991-06-05 1993-06-15 Allergan Humphrey Gaze tracking for field analyzer
US5223828A (en) * 1991-08-19 1993-06-29 International Business Machines Corporation Method and system for enabling a blind computer user to handle message boxes in a graphical user interface
US5233662A (en) * 1991-11-05 1993-08-03 Christensen Chris S Oral controller method and apparatus
JPH06149534A (en) * 1992-11-13 1994-05-27 Toshiba Corp Voice screen operation unit
US5428707A (en) * 1992-11-13 1995-06-27 Dragon Systems, Inc. Apparatus and methods for training speech recognition systems and their users and otherwise improving speech recognition performance
DE9300231U1 (en) * 1993-01-11 1993-08-19 Hessabi Iradj counter
US5586198A (en) * 1993-08-24 1996-12-17 Lakritz; David Method and apparatus for identifying characters in ideographic alphabet
US5396264A (en) * 1994-01-03 1995-03-07 Motorola, Inc. Automatic menu item sequencing method
US5429513A (en) * 1994-02-10 1995-07-04 Diaz-Plaza; Ruth R. Interactive teaching apparatus and method for teaching graphemes, grapheme names, phonemes, and phonetics
JPH07295784A (en) * 1994-04-25 1995-11-10 Hitachi Ltd Information processor by voice

Non-Patent Citations (40)

* Cited by examiner, † Cited by third party
Title
"Discrete and continuous control using stable speech sounds", IBM Technical Disclosure Bulletin, vol. 33, No. 10a, Mar. 1991, New York, NY, XP 000110092, p. 378. See the entire document.
Barnett, C.J.; Dynche, M.P.; Golding, V.G.; Ng, Y.H.; Taylor, G.D.; "Speech Output Display Terminal," IBM Technical Disclosure Bulletin, vol. 26, No. 10A, Mar. 1984, pp. 4950-4951.
Barnett, C.J.; Dynche, M.P.; Golding, V.G.; Ng, Y.H.; Taylor, G.D.; Speech Output Display Terminal, IBM Technical Disclosure Bulletin, vol. 26, No. 10A, Mar. 1984, pp. 4950 4951. *
Becker, Joseph D., User Friendly Design for Japanese Typing, Xerox Office Systems Division Publication No. T8301, Xerox Corp., Palo Alto, CA, Aug. 1983. See pp. 14 & 15. *
Bryan, R.E.; Chang, F.C.; Hemple, B.C.; Iida, Y.; "Pre-selection highlighting," IBM Technical Disclosure Bulletin, vol. 32, No. 8b, Jan. 1990, New York, NY, XP 000082441. pp. 231 & 232.
Bryan, R.E.; Chang, F.C.; Hemple, B.C.; Iida, Y.; Pre selection highlighting, IBM Technical Disclosure Bulletin, vol. 32, No. 8b, Jan. 1990, New York, NY, XP 000082441. pp. 231 & 232. *
Callahan, Jack; Hopkins, Don; Weiser, Mark; Schneiderman, Ben, "An Empirical Comparison of Pie vs. Linear Menus," Computer Science Technical Report Series CS-TR-1919, College Park, MD: University of Maryland, Sep. 1987.
Callahan, Jack; Hopkins, Don; Weiser, Mark; Schneiderman, Ben, An Empirical Comparison of Pie vs. Linear Menus, Computer Science Technical Report Series CS TR 1919, College Park, MD: University of Maryland, Sep. 1987. *
Callahan, Jack; Hopkins, Don; Weiser, Mark; Schneiderman, Ben; "An Empirical Comparison of Pie vs. Linear Menus," Human Factors in Computing Systems: CHI '88, Conference Proceedings, pp. 95-100.
Callahan, Jack; Hopkins, Don; Weiser, Mark; Schneiderman, Ben; An Empirical Comparison of Pie vs. Linear Menus, Human Factors in Computing Systems: CHI 88, Conference Proceedings, pp. 95 100. *
Darci Too Product Description, WesTest Engineering Corp., Bountiful, UT: WesTest Engineering Corp., 1992. *
Discrete and continuous control using stable speech sounds , IBM Technical Disclosure Bulletin, vol. 33, No. 10a, Mar. 1991, New York, NY, XP 000110092, p. 378. See the entire document. *
DragonDictate, The Premier PC Dictation System, Dragon Systems, Inc., Newton, MA, 1994. p. 4. *
Drawing Gallery, In The Graphics Gallery Release 3.0 User Manual, Hewlett Packard Co., Boise, ID: 1988. *
Drawing Gallery, In The Graphics Gallery Release 3.0 User Manual, Hewlett-Packard Co., Boise, ID: 1988.
Fitts, P.M and Peterson, J.R., "Information capacity of discrete motor responses", Journal of Experimental Psychology, 1964, vol. 67, pp. 103-112.
Fitts, P.M and Peterson, J.R., Information capacity of discrete motor responses , Journal of Experimental Psychology, 1964, vol. 67, pp. 103 112. *
Fitts, P.M., "The information capacity of the human motor system in controlling the amplitude of movement", Journal of Experimental Psychology, 1954, vol. 47, pp. 381-391.
Fitts, P.M., The information capacity of the human motor system in controlling the amplitude of movement , Journal of Experimental Psychology, 1954, vol. 47, pp. 381 391. *
Golding, V.G.; Heneghan, M.J.; "Audio Response Terminal," IBM Technical Disclosure Bulletin, vol. 26, No. 10B, Mar. 1984, pp. 5633-5636.
Golding, V.G.; Heneghan, M.J.; Audio Response Terminal, IBM Technical Disclosure Bulletin, vol. 26, No. 10B, Mar. 1984, pp. 5633 5636. *
Goosens, Carol; Crain, Sharon Sapp, "Overview of Currently Used Eye-Gaze Communication Approaches," In Augmentative Communication Intervention Resources, 1986.
Goosens, Carol; Crain, Sharon Sapp, Overview of Currently Used Eye Gaze Communication Approaches, In Augmentative Communication Intervention Resources, 1986. *
Jacob, Robert J.K., What you look at is what you get, Computer, vol. 26, No. 7, Jul. 1993, Los Alamitos, CA. pp. 65 & 66. *
Johnson, Frank, "Communications system for speechless quadriplegics", Design Product News, Jan. 1994.
Johnson, Frank, Communications system for speechless quadriplegics , Design Product News, Jan. 1994. *
Lazzaro, Joseph J., "Computers for the Disabled," Byte, Jun. 1993, pp. 59-64.
Lazzaro, Joseph J., Computers for the Disabled, Byte , Jun. 1993, pp. 59 64. *
PCT International Search Report, International Application No. PCT/US95/03591, dated Mar. 28, 1996. *
Pointer Sytem 92/ 93 Catalog, Pointer Systems, Inc., Burlington, VT: Pointer Systems, Inc., 1992. *
Pointer Sytem '92/'93 Catalog, Pointer Systems, Inc., Burlington, VT: Pointer Systems, Inc., 1992.
U.S. Patent No. 4,149,716 filed Jan. 24, 1977, issued Apr. 17, 1979 to Scudder. *
U.S. Patent No. 4,586,035 filed Feb. 29, 1984, issued Apr. 29, 1986 to Baker. *
U.S. Patent No. 4,595,990 filed Mar. 31, 1983, issued Jun. 17, 1986 to Garwin et al. *
U.S. Patent No. 4,651,145 filed May 31, 1984, issued Mar. 17, 1987 to Sutter. *
U.S. Patent No. 5,223,828 filed Aug. 19, 1991, issued Jun. 29, 1993 to McKiel, Jr. *
WiViK2, Visual Keyboard for Windows 3.1 User s Guide , Prentke Romich Co., Wooster, OH: Prentke Romich Co., 1992. *
WiViK2, Visual Keyboard for Windows 3.1 User's Guide, Prentke Romich Co., Wooster, OH: Prentke Romich Co., 1992.
Yamada, Mitsuho; Fukada, Tadahiko, "Eye word processor (EWP) and peripheral controller for the ALS patient", I.E.E.E. Proceedings A Physical Scinece, Measurement & Instrumentation, Management & Education, Apr. 1987, vol. 134, No. 4, Part A, Stevenage & Herts, Great Britain, pp. 328-330. See the entire document.
Yamada, Mitsuho; Fukada, Tadahiko, Eye word processor (EWP) and peripheral controller for the ALS patient , I.E.E.E. Proceedings A Physical Scinece, Measurement & Instrumentation, Management & Education, Apr. 1987, vol. 134, No. 4, Part A, Stevenage & Herts, Great Britain, pp. 328 330. See the entire document. *

Cited By (281)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6393125B1 (en) 1997-07-11 2002-05-21 Zarlink Semiconductor Inc. Initializer for a confusion data generator
US6404888B1 (en) * 1997-07-11 2002-06-11 Zarlink Semiconductor Inc. Confusion data generator
US9239673B2 (en) 1998-01-26 2016-01-19 Apple Inc. Gesturing with a multipoint sensing device
US9292111B2 (en) 1998-01-26 2016-03-22 Apple Inc. Gesturing with a multipoint sensing device
US6285367B1 (en) * 1998-05-26 2001-09-04 International Business Machines Corporation Method and apparatus for displaying and navigating a graph
US6163758A (en) * 1998-06-05 2000-12-19 Tektronix, Inc. Detection of unusual waveforms
US6594649B2 (en) 1998-08-19 2003-07-15 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US6745053B1 (en) * 1998-12-17 2004-06-01 Nec Corporation Mobile communication terminal apparatus having character string editing function by use of speech recognition function
US7292980B1 (en) * 1999-04-30 2007-11-06 Lucent Technologies Inc. Graphical user interface and method for modifying pronunciations in text-to-speech and speech recognition systems
US6446056B1 (en) * 1999-09-10 2002-09-03 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US6604091B2 (en) 1999-09-10 2003-08-05 Yamaha Hatsudoki Kabushiki Kaisha Interactive artificial intelligence
US9646614B2 (en) 2000-03-16 2017-05-09 Apple Inc. Fast, language-independent method for user authentication by voice
US8629838B2 (en) 2000-04-06 2014-01-14 Microsoft Corporation User notification system with an illuminated computer input device
US8279177B2 (en) 2000-04-06 2012-10-02 Microsoft Corporation User notification system with an illuminated computer input device
US8013840B1 (en) * 2000-04-06 2011-09-06 Microsoft Corporation User notification system with an illuminated computer input device
US7013258B1 (en) * 2001-03-07 2006-03-14 Lenovo (Singapore) Pte. Ltd. System and method for accelerating Chinese text input
US7356459B2 (en) * 2001-04-06 2008-04-08 Bsh Bosch Und Siemens Hausgeraete Gmbh Program-controlled household appliance and method of operating the appliance
US20050091063A1 (en) * 2001-04-06 2005-04-28 Heinz-Jurgen Bergemann Program-controlled household appliance and method of operating the appliance
US6514078B1 (en) * 2001-11-02 2003-02-04 Jacqueline E. Jones Electronic educational device
US9606668B2 (en) 2002-02-07 2017-03-28 Apple Inc. Mode-based graphical user interfaces for touch sensitive input devices
US20040161151A1 (en) * 2002-04-16 2004-08-19 Fujitsu Limited On-line handwritten character pattern recognizing and editing apparatus and method, and recording medium storing computer-executable program for realizing the method
US20050106537A1 (en) * 2002-04-18 2005-05-19 Andrew Chepaitis Dynamic tactile and low vision fonts
WO2003090182A1 (en) * 2002-04-18 2003-10-30 Elia Life Technology Dynamic tactile and low vision fonts
US20090305199A1 (en) * 2002-04-18 2009-12-10 Andrew Chepaitis Dynamic tactile and low vision fonts
EP1497747A4 (en) * 2002-04-22 2005-07-20 Nokia Corp System and method for navigating computer programs using a graphical user interface
US7111788B2 (en) 2002-04-22 2006-09-26 Nokia Corporation System and method for navigating applications using a graphical user interface
US20030197740A1 (en) * 2002-04-22 2003-10-23 Nokia Corporation System and method for navigating applications using a graphical user interface
EP1497747A2 (en) * 2002-04-22 2005-01-19 Nokia Corporation System and method for navigating computer programs using a graphical user interface
US20030210272A1 (en) * 2002-05-09 2003-11-13 Gateway, Inc. System and method for providing a user interface
US20040225504A1 (en) * 2003-05-09 2004-11-11 Junqua Jean-Claude Portable device for enhanced security and accessibility
US7249025B2 (en) 2003-05-09 2007-07-24 Matsushita Electric Industrial Co., Ltd. Portable device for enhanced security and accessibility
US20040235530A1 (en) * 2003-05-23 2004-11-25 General Motors Corporation Context specific speaker adaptation user interface
US7986974B2 (en) * 2003-05-23 2011-07-26 General Motors Llc Context specific speaker adaptation user interface
US20040263480A1 (en) * 2003-06-24 2004-12-30 International Business Machines Corporation Method and system for providing integrated hot key configuration
US8020096B2 (en) * 2003-06-24 2011-09-13 International Business Machines Corporation Method and system for providing integrated hot key configuration
US20050015254A1 (en) * 2003-07-18 2005-01-20 Apple Computer, Inc. Voice menu system
US7757173B2 (en) * 2003-07-18 2010-07-13 Apple Inc. Voice menu system
US7395203B2 (en) * 2003-07-30 2008-07-01 Tegic Communications, Inc. System and method for disambiguating phonetic input
US20050027524A1 (en) * 2003-07-30 2005-02-03 Jianchao Wu System and method for disambiguating phonetic input
US20050027534A1 (en) * 2003-07-30 2005-02-03 Meurs Pim Van Phonetic and stroke input methods of Chinese characters and phrases
US9239677B2 (en) 2004-05-06 2016-01-19 Apple Inc. Operation of a computer with touch screen interface
US10338789B2 (en) 2004-05-06 2019-07-02 Apple Inc. Operation of a computer with touch screen interface
US20090099838A1 (en) * 2004-06-25 2009-04-16 Nokia Corporation Text messaging device
US7693552B2 (en) * 2004-06-25 2010-04-06 Nokia Corporation Text messaging device
US11036282B2 (en) 2004-07-30 2021-06-15 Apple Inc. Proximity detector in handheld device
US7614008B2 (en) * 2004-07-30 2009-11-03 Apple Inc. Operation of a computer with touch screen interface
US8612856B2 (en) 2004-07-30 2013-12-17 Apple Inc. Proximity detector in handheld device
US9348458B2 (en) 2004-07-30 2016-05-24 Apple Inc. Gestures for touch sensitive input devices
US8479122B2 (en) 2004-07-30 2013-07-02 Apple Inc. Gestures for touch sensitive input devices
US10042418B2 (en) 2004-07-30 2018-08-07 Apple Inc. Proximity detector in handheld device
US20060204137A1 (en) * 2005-03-07 2006-09-14 Hitachi, Ltd. Portable terminal and information-processing device, and system
US20070038456A1 (en) * 2005-08-12 2007-02-15 Delta Electronics, Inc. Text inputting device and method employing combination of associated character input method and automatic speech recognition method
US10318871B2 (en) 2005-09-08 2019-06-11 Apple Inc. Method and apparatus for building an intelligent automated assistant
US20070061139A1 (en) * 2005-09-14 2007-03-15 Delta Electronics, Inc. Interactive speech correcting method
US20080098331A1 (en) * 2005-09-16 2008-04-24 Gregory Novick Portable Multifunction Device with Soft Keyboards
US7899674B1 (en) * 2006-08-11 2011-03-01 The United States Of America As Represented By The Secretary Of The Navy GUI for the semantic normalization of natural language
US8096880B2 (en) * 2006-08-15 2012-01-17 Nintendo Co., Ltd. Systems and methods for reducing jitter associated with a control device
US20080051197A1 (en) * 2006-08-15 2008-02-28 Mark Jawad Systems and methods for reducing jitter associated with a control device
US9117447B2 (en) 2006-09-08 2015-08-25 Apple Inc. Using event alert text as input to an automated assistant
US8942986B2 (en) 2006-09-08 2015-01-27 Apple Inc. Determining user intent based on ontologies of domains
US8930191B2 (en) 2006-09-08 2015-01-06 Apple Inc. Paraphrasing of user requests and results by automated digital assistant
US11416141B2 (en) 2007-01-05 2022-08-16 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US10592100B2 (en) 2007-01-05 2020-03-17 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US11112968B2 (en) 2007-01-05 2021-09-07 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US9244536B2 (en) 2007-01-05 2016-01-26 Apple Inc. Method, system, and graphical user interface for providing word recommendations
US20080201662A1 (en) * 2007-02-13 2008-08-21 Harman Becker Automotive Systems Gmbh Methods for controlling a navigation system
US9140572B2 (en) * 2007-02-13 2015-09-22 Harman Becker Automotive Systems Gmbh Methods for controlling a navigation system
US10568032B2 (en) 2007-04-03 2020-02-18 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US20080300885A1 (en) * 2007-05-30 2008-12-04 Chung-Hung Shih Speech communication system for patients having difficulty in speaking or writing
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US10381016B2 (en) 2008-01-03 2019-08-13 Apple Inc. Methods and apparatus for altering audio output signals
US11474695B2 (en) 2008-01-09 2022-10-18 Apple Inc. Method, device, and graphical user interface providing word recommendations for text input
US11079933B2 (en) 2008-01-09 2021-08-03 Apple Inc. Method, device, and graphical user interface providing word recommendations for text input
US9626955B2 (en) 2008-04-05 2017-04-18 Apple Inc. Intelligent text-to-speech conversion
US9865248B2 (en) 2008-04-05 2018-01-09 Apple Inc. Intelligent text-to-speech conversion
US10430078B2 (en) 2008-06-27 2019-10-01 Apple Inc. Touch screen device, and graphical user interface for inserting a character from an alternate keyboard
US10025501B2 (en) 2008-06-27 2018-07-17 Apple Inc. Touch screen device, method, and graphical user interface for inserting a character from an alternate keyboard
US9459791B2 (en) 2008-06-28 2016-10-04 Apple Inc. Radial menu selection
US20090327963A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Radial menu selection
US8245156B2 (en) 2008-06-28 2012-08-14 Apple Inc. Radial menu selection
US20090327964A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Moving radial menus
US20090327955A1 (en) * 2008-06-28 2009-12-31 Mouilleseaux Jean-Pierre M Selecting Menu Items
US8826181B2 (en) 2008-06-28 2014-09-02 Apple Inc. Moving radial menus
US10108612B2 (en) 2008-07-31 2018-10-23 Apple Inc. Mobile device having human language translation capability with positional feedback
US9535906B2 (en) 2008-07-31 2017-01-03 Apple Inc. Mobile device having human language translation capability with positional feedback
US8898568B2 (en) 2008-09-09 2014-11-25 Apple Inc. Audio user interface
US10338778B2 (en) * 2008-09-25 2019-07-02 Apple Inc. Collaboration system
US9959870B2 (en) 2008-12-11 2018-05-01 Apple Inc. Speech recognition involving a mobile device
US8494857B2 (en) 2009-01-06 2013-07-23 Regents Of The University Of Minnesota Automatic measurement of speech fluency
US9230539B2 (en) 2009-01-06 2016-01-05 Regents Of The University Of Minnesota Automatic measurement of speech fluency
US20100306702A1 (en) * 2009-05-29 2010-12-02 Peter Warner Radial Menus
US8549432B2 (en) 2009-05-29 2013-10-01 Apple Inc. Radial menus
US9733796B2 (en) 2009-05-29 2017-08-15 Apple Inc. Radial menus
US11080012B2 (en) 2009-06-05 2021-08-03 Apple Inc. Interface for a virtual digital assistant
US10475446B2 (en) 2009-06-05 2019-11-12 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US10795541B2 (en) 2009-06-05 2020-10-06 Apple Inc. Intelligent organization of tasks items
US9858925B2 (en) 2009-06-05 2018-01-02 Apple Inc. Using context information to facilitate processing of commands in a virtual assistant
US10283110B2 (en) 2009-07-02 2019-05-07 Apple Inc. Methods and apparatuses for automatic speech recognition
US8806362B2 (en) 2010-01-06 2014-08-12 Apple Inc. Device, method, and graphical user interface for accessing alternate keys
US20110163973A1 (en) * 2010-01-06 2011-07-07 Bas Ording Device, Method, and Graphical User Interface for Accessing Alternative Keys
US8538756B2 (en) * 2010-01-12 2013-09-17 Denso Corporation In-vehicle device and method for modifying display mode of icon indicated on the same
US20110173002A1 (en) * 2010-01-12 2011-07-14 Denso Corporation In-vehicle device and method for modifying display mode of icon indicated on the same
US10496753B2 (en) 2010-01-18 2019-12-03 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10553209B2 (en) 2010-01-18 2020-02-04 Apple Inc. Systems and methods for hands-free notification summaries
US8892446B2 (en) 2010-01-18 2014-11-18 Apple Inc. Service orchestration for intelligent automated assistant
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US9548050B2 (en) 2010-01-18 2017-01-17 Apple Inc. Intelligent automated assistant
US8903716B2 (en) 2010-01-18 2014-12-02 Apple Inc. Personalized vocabulary for digital assistant
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US10679605B2 (en) 2010-01-18 2020-06-09 Apple Inc. Hands-free list-reading by intelligent automated assistant
US11423886B2 (en) 2010-01-18 2022-08-23 Apple Inc. Task flow identification based on user intent
US10706841B2 (en) 2010-01-18 2020-07-07 Apple Inc. Task flow identification based on user intent
US10705794B2 (en) 2010-01-18 2020-07-07 Apple Inc. Automatically adapting user interfaces for hands-free interaction
US10984327B2 (en) 2010-01-25 2021-04-20 New Valuexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US11410053B2 (en) 2010-01-25 2022-08-09 Newvaluexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US10984326B2 (en) 2010-01-25 2021-04-20 Newvaluexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US10607140B2 (en) 2010-01-25 2020-03-31 Newvaluexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US10607141B2 (en) 2010-01-25 2020-03-31 Newvaluexchange Ltd. Apparatuses, methods and systems for a digital conversation management platform
US9633660B2 (en) 2010-02-25 2017-04-25 Apple Inc. User profiling for voice input processing
US10049675B2 (en) 2010-02-25 2018-08-14 Apple Inc. User profiling for voice input processing
US9190062B2 (en) 2010-02-25 2015-11-17 Apple Inc. User profiling for voice input processing
US8423912B2 (en) * 2010-06-29 2013-04-16 Pourang Irani Selectable parent and submenu object display method with varied activation area shape
US20110320984A1 (en) * 2010-06-29 2011-12-29 Pourang Irani Selectable Parent and Submenu Object Display Method
US20120089400A1 (en) * 2010-10-06 2012-04-12 Caroline Gilles Henton Systems and methods for using homophone lexicons in english text-to-speech
US10762293B2 (en) 2010-12-22 2020-09-01 Apple Inc. Using parts-of-speech tagging and named entity recognition for spelling correction
US10102359B2 (en) 2011-03-21 2018-10-16 Apple Inc. Device access using voice authentication
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US20120272144A1 (en) * 2011-04-20 2012-10-25 Microsoft Corporation Compact control menu for touch-enabled command execution
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US11120372B2 (en) 2011-06-03 2021-09-14 Apple Inc. Performing actions associated with task items that represent tasks to perform
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
US10706373B2 (en) 2011-06-03 2020-07-07 Apple Inc. Performing actions associated with task items that represent tasks to perform
US9798393B2 (en) 2011-08-29 2017-10-24 Apple Inc. Text correction processing
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US8560974B1 (en) * 2011-10-06 2013-10-15 Google Inc. Input method application for a touch-sensitive user interface
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
US9483461B2 (en) 2012-03-06 2016-11-01 Apple Inc. Handling speech synthesis of content for multiple languages
US9576593B2 (en) 2012-03-15 2017-02-21 Regents Of The University Of Minnesota Automated verbal fluency assessment
US20130253909A1 (en) * 2012-03-23 2013-09-26 Tata Consultancy Services Limited Second language acquisition system
US9390085B2 (en) * 2012-03-23 2016-07-12 Tata Consultancy Sevices Limited Speech processing system and method for recognizing speech samples from a speaker with an oriyan accent when speaking english
US9953088B2 (en) 2012-05-14 2018-04-24 Apple Inc. Crowd sourcing information to fulfill user requests
US10079014B2 (en) 2012-06-08 2018-09-18 Apple Inc. Name recognition system
US9495129B2 (en) 2012-06-29 2016-11-15 Apple Inc. Device, method, and user interface for voice-activated navigation and browsing of a document
US9576574B2 (en) 2012-09-10 2017-02-21 Apple Inc. Context-sensitive handling of interruptions by intelligent digital assistant
US9971774B2 (en) 2012-09-19 2018-05-15 Apple Inc. Voice-based media searching
US10254953B2 (en) 2013-01-21 2019-04-09 Keypoint Technologies India Pvt. Ltd. Text input method using continuous trace across two or more clusters of candidate words to select two or more words to form a sequence, wherein the candidate words are arranged based on selection probabilities
US10474355B2 (en) 2013-01-21 2019-11-12 Keypoint Technologies India Pvt. Ltd. Input pattern detection over virtual keyboard for candidate word identification
US10199051B2 (en) 2013-02-07 2019-02-05 Apple Inc. Voice trigger for a digital assistant
US10978090B2 (en) 2013-02-07 2021-04-13 Apple Inc. Voice trigger for a digital assistant
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US11388291B2 (en) 2013-03-14 2022-07-12 Apple Inc. System and method for processing voicemail
US9368114B2 (en) 2013-03-14 2016-06-14 Apple Inc. Context-sensitive handling of interruptions
US9922642B2 (en) 2013-03-15 2018-03-20 Apple Inc. Training an at least partial voice command system
US9697822B1 (en) 2013-03-15 2017-07-04 Apple Inc. System and method for updating an adaptive speech recognition model
US9966060B2 (en) 2013-06-07 2018-05-08 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9620104B2 (en) 2013-06-07 2017-04-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9633674B2 (en) 2013-06-07 2017-04-25 Apple Inc. System and method for detecting errors in interactions with a voice-based digital assistant
US9582608B2 (en) 2013-06-07 2017-02-28 Apple Inc. Unified ranking with entropy-weighted information for phrase-based semantic auto-completion
US9966068B2 (en) 2013-06-08 2018-05-08 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10657961B2 (en) 2013-06-08 2020-05-19 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
US10185542B2 (en) 2013-06-09 2019-01-22 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
US9300784B2 (en) 2013-06-13 2016-03-29 Apple Inc. System and method for emergency calls initiated by voice command
US10791216B2 (en) 2013-08-06 2020-09-29 Apple Inc. Auto-activating smart responses based on activities from remote devices
USD763266S1 (en) * 2013-09-03 2016-08-09 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
US9500867B2 (en) 2013-11-15 2016-11-22 Kopin Corporation Head-tracking based selection technique for head mounted displays (HMD)
US9383816B2 (en) 2013-11-15 2016-07-05 Kopin Corporation Text selection using HMD head-tracker and voice-command
US10402162B2 (en) 2013-11-15 2019-09-03 Kopin Corporation Automatic speech recognition (ASR) feedback for head mounted displays (HMD)
WO2015073880A1 (en) * 2013-11-15 2015-05-21 Kopin Corporation Head-tracking based selection technique for head mounted displays (hmd)
US9904360B2 (en) 2013-11-15 2018-02-27 Kopin Corporation Head tracking based gesture control techniques for head mounted displays
US10209955B2 (en) 2013-11-15 2019-02-19 Kopin Corporation Automatic speech recognition (ASR) feedback for head mounted displays (HMD)
US20150241994A1 (en) * 2014-02-25 2015-08-27 Simon Ekstrand Display method of mobile device selection and terminal device
US9495019B2 (en) * 2014-02-25 2016-11-15 Huawei Technologies Co., Ltd. Display method of mobile device selection and terminal device
CN103997668A (en) * 2014-02-25 2014-08-20 华为技术有限公司 Method for displaying selection of mobile equipment, and terminal equipment
US9620105B2 (en) 2014-05-15 2017-04-11 Apple Inc. Analyzing audio input for efficient speech and music recognition
US10592095B2 (en) 2014-05-23 2020-03-17 Apple Inc. Instantaneous speaking of content on touch devices
US9502031B2 (en) 2014-05-27 2016-11-22 Apple Inc. Method for supporting dynamic grammars in WFST-based ASR
US9760559B2 (en) 2014-05-30 2017-09-12 Apple Inc. Predictive text input
US10289433B2 (en) 2014-05-30 2019-05-14 Apple Inc. Domain specific language for encoding assistant dialog
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US10078631B2 (en) 2014-05-30 2018-09-18 Apple Inc. Entropy-guided text prediction using combined word and character n-gram language models
US10497365B2 (en) 2014-05-30 2019-12-03 Apple Inc. Multi-command single utterance input method
US9842101B2 (en) 2014-05-30 2017-12-12 Apple Inc. Predictive conversion of language input
US9785630B2 (en) 2014-05-30 2017-10-10 Apple Inc. Text prediction using combined word N-gram and unigram language models
US9734193B2 (en) 2014-05-30 2017-08-15 Apple Inc. Determining domain salience ranking from ambiguous words in natural speech
US11133008B2 (en) 2014-05-30 2021-09-28 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10083690B2 (en) 2014-05-30 2018-09-25 Apple Inc. Better resolution when referencing to concepts
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
US11257504B2 (en) 2014-05-30 2022-02-22 Apple Inc. Intelligent assistant for home automation
US10169329B2 (en) 2014-05-30 2019-01-01 Apple Inc. Exemplar-based natural language processing
US9966065B2 (en) 2014-05-30 2018-05-08 Apple Inc. Multi-command single utterance input method
US10904611B2 (en) 2014-06-30 2021-01-26 Apple Inc. Intelligent automated assistant for TV user interactions
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US10659851B2 (en) 2014-06-30 2020-05-19 Apple Inc. Real-time digital assistant knowledge updates
US9668024B2 (en) 2014-06-30 2017-05-30 Apple Inc. Intelligent automated assistant for TV user interactions
US10446141B2 (en) 2014-08-28 2019-10-15 Apple Inc. Automatic speech recognition based on user feedback
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10431204B2 (en) 2014-09-11 2019-10-01 Apple Inc. Method and apparatus for discovering trending terms in speech requests
US10789041B2 (en) 2014-09-12 2020-09-29 Apple Inc. Dynamic thresholds for always listening speech trigger
US9986419B2 (en) 2014-09-30 2018-05-29 Apple Inc. Social reminders
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
US9886432B2 (en) 2014-09-30 2018-02-06 Apple Inc. Parsimonious handling of word inflection via categorical stem + suffix N-gram language models
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US9646609B2 (en) 2014-09-30 2017-05-09 Apple Inc. Caching apparatus for serving phonetic pronunciations
US10552013B2 (en) 2014-12-02 2020-02-04 Apple Inc. Data detection
US11556230B2 (en) 2014-12-02 2023-01-17 Apple Inc. Data detection
US9711141B2 (en) 2014-12-09 2017-07-18 Apple Inc. Disambiguating heteronyms in speech synthesis
US9865280B2 (en) 2015-03-06 2018-01-09 Apple Inc. Structured dictation using intelligent automated assistants
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US10311871B2 (en) 2015-03-08 2019-06-04 Apple Inc. Competing devices responding to voice triggers
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US11087759B2 (en) 2015-03-08 2021-08-10 Apple Inc. Virtual assistant activation
US9899019B2 (en) 2015-03-18 2018-02-20 Apple Inc. Systems and methods for structured stem and suffix language models
US9842105B2 (en) 2015-04-16 2017-12-12 Apple Inc. Parsimonious continuous-space phrase representations for natural language processing
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10127220B2 (en) 2015-06-04 2018-11-13 Apple Inc. Language identification from short strings
US10101822B2 (en) 2015-06-05 2018-10-16 Apple Inc. Language input correction
US10255907B2 (en) 2015-06-07 2019-04-09 Apple Inc. Automatic accent detection using acoustic models
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10186254B2 (en) 2015-06-07 2019-01-22 Apple Inc. Context-based endpoint detection
US10073592B2 (en) 2015-06-18 2018-09-11 Apple Inc. Device, method, and graphical user interface for navigating media content
US10572109B2 (en) 2015-06-18 2020-02-25 Apple Inc. Device, method, and graphical user interface for navigating media content
US10545635B2 (en) 2015-06-18 2020-01-28 Apple Inc. Device, method, and graphical user interface for navigating media content
US11816303B2 (en) 2015-06-18 2023-11-14 Apple Inc. Device, method, and graphical user interface for navigating media content
US10073591B2 (en) 2015-06-18 2018-09-11 Apple Inc. Device, method, and graphical user interface for navigating media content
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US20170068430A1 (en) * 2015-09-08 2017-03-09 Apple Inc. Device, Method, and Graphical User Interface for Providing Audiovisual Feedback
US10599394B2 (en) 2015-09-08 2020-03-24 Apple Inc. Device, method, and graphical user interface for providing audiovisual feedback
US10474333B2 (en) 2015-09-08 2019-11-12 Apple Inc. Devices, methods, and graphical user interfaces for moving a current focus using a touch-sensitive remote control
US10963130B2 (en) 2015-09-08 2021-03-30 Apple Inc. Devices, methods, and graphical user interfaces for moving a current focus using a touch-sensitive remote control
CN110275664A (en) * 2015-09-08 2019-09-24 苹果公司 For providing the equipment, method and graphic user interface of audiovisual feedback
US10152300B2 (en) * 2015-09-08 2018-12-11 Apple Inc. Device, method, and graphical user interface for providing audiovisual feedback
US11262890B2 (en) 2015-09-08 2022-03-01 Apple Inc. Devices, methods, and graphical user interfaces for moving a current focus using a touch-sensitive remote control
US11500672B2 (en) 2015-09-08 2022-11-15 Apple Inc. Distributed personal assistant
US11635876B2 (en) 2015-09-08 2023-04-25 Apple Inc. Devices, methods, and graphical user interfaces for moving a current focus using a touch-sensitive remote control
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US9697820B2 (en) 2015-09-24 2017-07-04 Apple Inc. Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US11526368B2 (en) 2015-11-06 2022-12-13 Apple Inc. Intelligent automated assistant in a messaging environment
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
US11069347B2 (en) 2016-06-08 2021-07-20 Apple Inc. Intelligent automated assistant for media exploration
US10354011B2 (en) 2016-06-09 2019-07-16 Apple Inc. Intelligent automated assistant in a home environment
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US11037565B2 (en) 2016-06-10 2021-06-15 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10733993B2 (en) 2016-06-10 2020-08-04 Apple Inc. Intelligent digital assistant in a multi-tasking environment
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US11152002B2 (en) 2016-06-11 2021-10-19 Apple Inc. Application integration with a digital assistant
US10521466B2 (en) 2016-06-11 2019-12-31 Apple Inc. Data driven natural language event detection and classification
US10269345B2 (en) 2016-06-11 2019-04-23 Apple Inc. Intelligent task discovery
US10089072B2 (en) 2016-06-11 2018-10-02 Apple Inc. Intelligent device arbitration and control
US10297253B2 (en) 2016-06-11 2019-05-21 Apple Inc. Application integration with a digital assistant
US10394381B2 (en) 2016-07-14 2019-08-27 Lenovo (Singapore) Pte. Ltd. False input reduction systems, apparatus, and methods for an information processing device
JP2018010512A (en) * 2016-07-14 2018-01-18 レノボ・シンガポール・プライベート・リミテッド Information processor, false input inhibiting method, and program
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US20180225033A1 (en) * 2017-02-08 2018-08-09 Fuji Xerox Co., Ltd. Information processing apparatus and non-transitory computer readable medium
US20180246569A1 (en) * 2017-02-27 2018-08-30 Fuji Xerox Co., Ltd. Information processing apparatus and method and non-transitory computer readable medium
US11405466B2 (en) 2017-05-12 2022-08-02 Apple Inc. Synchronization and task delegation of a digital assistant
US10791176B2 (en) 2017-05-12 2020-09-29 Apple Inc. Synchronization and task delegation of a digital assistant
US10810274B2 (en) 2017-05-15 2020-10-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
US11210961B2 (en) 2018-03-12 2021-12-28 Neurological Rehabilitation Virtual Reality, LLC Systems and methods for neural pathways creation/reinforcement by neural detection with virtual feedback
US11810474B2 (en) 2018-03-12 2023-11-07 Neuromersive, Inc Systems and methods for neural pathways creation/reinforcement by neural detection with virtual feedback
US11256322B2 (en) * 2018-05-09 2022-02-22 Neurological Rehabilitation Virtual Reality, LLC Systems and methods for responsively adaptable virtual environments
US11726556B2 (en) * 2018-05-09 2023-08-15 Neuromersive, Inc. Systems and methods for responsively adaptable virtual environments
US20220179481A1 (en) * 2018-05-09 2022-06-09 Neurological Rehabilitation Virtual Reality, LLC Systems and methods for responsively adaptable virtual environments
US11922006B2 (en) 2018-09-26 2024-03-05 Apple Inc. Media control for screensavers on an electronic device
US11921914B2 (en) * 2022-09-29 2024-03-05 Neuromersive, Inc. Systems and methods for responsively adaptable virtual environments

Similar Documents

Publication Publication Date Title
US5999895A (en) Sound operated menu method and apparatus
US6005549A (en) User interface method and apparatus
US9535494B2 (en) Apparatus and method for selecting from a display
US6160536A (en) Dwell time indication method and apparatus
Majaranta et al. Twenty years of eye typing: systems and design issues
Loncke Augmentative and alternative communication: Models and applications
EP0832480B1 (en) Dynamically redefining keys on a keyboard
Foley et al. The art of natural man-machine conversation
Edwards Soundtrack: An auditory interface for blind users
Oviatt et al. Perceptual user interfaces: multimodal interfaces that process what comes naturally
WO1996030822A1 (en) Method of and apparatus for data entry
Jones Human-computer interaction: A design guide
CA2345774A1 (en) Method and apparatus for displaying information
Myers et al. Using handhelds to help people with motor impairments
Simpson Computer Access for People with Disabilities: A Human Factors Approach
GB2332972A (en) Dwell time indicator in interactive display terminal
Brett Adaptive technology for special human needs
Lopes Designing user interfaces for severely handicapped persons
CA2214511A1 (en) Method of and apparatus for data entry
Goodenough-Trepagnier et al. Towards a method for computer interface design using speech recognition
Hill et al. Augmentative and alternative communication technology
DV et al. Eye gaze controlled adaptive virtual keyboard for users with SSMI
Treviranus Virtual reality technologies and people with disabilities
Schmauks et al. Integration of communicative hand movements into human-computer-interaction
Li Action theory and cognitive psychology in industrial design: user models and user interfaces

Legal Events

Date Code Title Description
STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

FPAY Fee payment

Year of fee payment: 12