US20110161081A1 - Speech Recognition Language Models - Google Patents

Speech Recognition Language Models Download PDF

Info

Publication number
US20110161081A1
US20110161081A1 US12/977,017 US97701710A US2011161081A1 US 20110161081 A1 US20110161081 A1 US 20110161081A1 US 97701710 A US97701710 A US 97701710A US 2011161081 A1 US2011161081 A1 US 2011161081A1
Authority
US
United States
Prior art keywords
website
query
input
queries
speech recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/977,017
Inventor
Brandon M. Ballinger
Johan Schalkwyk
Michael H. Cohen
Cyril Georges Luc Allauzen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Priority to US12/977,017 priority Critical patent/US20110161081A1/en
Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Ballinger, Brandon M., COHEN, MICHAEL H., SCHALKWYK, JOHAN, LUC ALLAUZEN, CYRIL GEORGES
Publication of US20110161081A1 publication Critical patent/US20110161081A1/en
Priority to US13/249,180 priority patent/US20120022873A1/en
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16Sound input; Sound output
    • G06F3/167Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/20Natural language analysis
    • G06F40/279Recognition of textual entities
    • G06F40/284Lexical analysis, e.g. tokenisation or collocates
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/40Processing or translation of natural language
    • G06F40/58Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/005Language recognition
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/183Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/26Speech to text systems
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/28Constructional details of speech recognition systems
    • G10L15/30Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L15/18Speech classification or search using natural language modelling
    • G10L15/183Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G10L15/19Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
    • G10L15/197Probabilistic grammars, e.g. word n-grams
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223Execution procedure of a spoken command
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/226Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics
    • G10L2015/228Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics of application context

Definitions

  • This document relates to systems and techniques for multi-modal input into an electronic device and conversion of spoken input to text.
  • keyboards are common input devices, and they typically include single-digit numbers (e.g., in a cellular telephone) each of the letters in the alphabet, and some characters (e.g., in Qwerty or Dvorak keyboards).
  • keyboards are frequently “virtual” in form, and are displayed on a touch screen of a device.
  • Such keyboards may be made available to various different applications running on a device, using a program known as an Input Method Editor, or IME, so that the IME receives the user input and then passes it to whatever application is currently active on the device.
  • IME Input Method Editor
  • An IME can also translate user input, such as when a user enters Roman characters in a written language like Pinyin, and the IME generates Chinese characters that correspond to the typed Pinyin. Where the Pinyin corresponds to multiple possible characters, the IME can display all such characters, the user can tap the intended character, and the IME can pass that character to the operating application.
  • keyboards Users of computer devices, and particularly of mobile computing devices, may be constrained in their use of a keyboard.
  • the keyboard itself may be constrained in size because mobile device displays are small, so that only a sub-set of relevant characters can be displayed or the keys may be too small to press accurately.
  • the user may be constrained, in that they cannot easily type on a keyboard while walking through a crowded airport or driving a car. In such situations, spoken input may be preferred over typed input.
  • speech-to-text conversion or translation typically requires lots of computer processing power, and mobile devices typically do not have much processing power. Also, such conversion often requires a particular user to “train” the system so that it better understands the user's voice and speech patterns.
  • an IME that provides keyboard services to a device may also provide for spoken input to the device.
  • Much of the IME services may be unchanged from an ordinary IME, but the IME may be augmented to provide for speech-to-text conversion.
  • the IME may take a user input in whatever form (e.g., typed, spoken, D Pad, etc.) and may convert it to a standard form for presentation to an active application (e.g., passing characters or other key presses).
  • Applications may subscribe to the IME's services just as they would subscribe if the IME did not accept voice input, and the provision of such speech-to-text functionality can be transparent to the applications.
  • a user could choose when to provide typed input and when to provide spoken input to the IME.
  • the user may be in an application that requires input, and may take an appropriate action to invoke the IME (e.g., pressing a particular button or soft key on the device, moving the device in a certain manner, and the like).
  • a keyboard may be initially displayed on the user's device, and the user may take another appropriate action to indicate that he or she will provide speech input. For example, the user may press a microphone button on the virtual keyboard, or may make a swipe across the virtual keyboard or another action that is inconsistent with an intent to provide typed input in the virtual keyboard.
  • the IME can begin “listening” to the device's microphone and after the user has spoken, may pass corresponding text to the application.
  • speech-to-text functionality may be provided on a computing device relatively simply, while re-using other IME functionality (e.g., interfaces to applications on a device) that is needed for keyboard-based IME translation.
  • IME functionality e.g., interfaces to applications on a device
  • Use of context-specific language models in the manners discussed above and below may also permit more accurate conversion of speech to text, regardless of whether the system is trained to a particular user.
  • Such context-specificity may also be provided automatically and transparently for a user, and at a level of specificity that is most suited to a particular situation.
  • the language model for the page may be used, but if the user is interacting with a form on the page, a more specific language model that is directed to that form or a particular field on the form may be used (with lower weightings, but higher-than-normal weightings, applied to the page-specific model).
  • Particular manners of using public user activity, such as search activity, to build contextual language models may also result in the generation of accurate models in a convenient manner.
  • Such data may be made available via natural user activity that is already occurring voluntarily by the users.
  • the activity may occur in large enough volumes to provide enough data needed to generate a complete and accurate model for many contexts, and in this case, for many web pages.
  • the models may also be updated over time, because the public user activity occurs continuously over time, so that continuously up-to-date models may be provided to users seeking speech-to-text conversion services.
  • a request can be received from a user of an electronic device for an application-independent input method editor having written and spoken input capabilities. That the user intends to provide spoken input to the application-independent input method editor can be identified, and the spoken input can be received from the user.
  • the spoken input can be input to an application executing on the electronic device.
  • the spoken input can be provided to a remote server.
  • the remote server includes a speech recognition system configured to recognize text based on the spoken input. Text can be received from the remote server, where the text represents the spoken input. The text can be provided to the application as user input.
  • a speech recognition language model In general, in one aspect, methods, computer program products and systems are described for forming a speech recognition language model.
  • Multiple query-website relationships are determined by identifying websites that are determined to be relevant to queries using one or more search engines.
  • Clusters are identified in the query-website relationships by connecting common queries and connecting common websites.
  • a speech recognition language model is created for a particular website based on at least one of analyzing at queries in a cluster that includes the website or analyzing webpage content of web pages in the cluster that includes the website.
  • Implementations of the methods, computer program products and systems can include one or more of the following features. Determining the multiple query-website relationships can include analyzing search results returned by the one or more search engines to the queries. Identifying clusters in the query-website relationships can include analyzing a graph of queries and websites at nodes of the graph, that are connected by edges defined by particular websites that are determined to be relevant to particular queries. A website can be determined to be relevant to a query if the website is a top n ranked search result for the query in a set of ranked search results relevant to the query, wherein n is a predetermined integer (e.g., is a top 10 ranked search result).
  • Creating the speech recognition model for the particular website can include creating cluster-based speech recognition models for each of multiple clusters, and creating the speech recognition model by interpolating among cluster-based speech recognition models for clusters that surround the particular website.
  • FIG. 1A is a conceptual diagram of an example system including a multi-modal input method editor.
  • FIG. 1B is a block diagram of an example system that can be used to implement the multi-modal input method editor.
  • FIG. 1C is a block diagram of example software that can be used to implement the input method editor.
  • FIG. 2 is a flow chart of an example process for generating text using a multi-modal input method editor.
  • FIGS. 3A-E show example screen shots of an electronic device including a multi-modal input method editor.
  • FIG. 4 is a block diagram of an example system for receiving speech input and training language models to interpret the speech input.
  • FIG. 5A is a block diagram of queries and associated websites in search results
  • FIG. 5B is a URL showing search terms in a query
  • FIG. 6 is a block diagram of an example system for recognizing speech input to a computing device.
  • FIGS. 7A and 7B show flowcharts of example processes of building an interpolated language model.
  • FIG. 8 shows a flowchart of an example process for recognizing text in an utterance.
  • FIG. 9 shows examples of generic computer devices that may be used to execute the actions discussed in this document.
  • FIG. 10 is a block diagram of an example data structure of a language model.
  • This document describes techniques and systems that may be used to provide speech-to-text conversion for a user of a computing device, such as a smartphone.
  • the speech input may be handled in a manner similar to other input (e.g., typed input) using an application such as an IME, where the IME can be switched into modes depending on the manner that the user chooses to enter data.
  • an application such as an IME
  • the input may in certain circumstances be transmitted (either in a raw or converted form) to a server system remote from the computing device that may be programmed to pass a transformed input back to the device, such as by providing text in response to receiving spoken inputs.
  • the computing device may also provide the server system with meta data that is passed with, or at essentially the same time as, the spoken inputs, and the meta data may be used by the server system to identify a context in which the user is entering the spoken input.
  • the server system may then use that meta data to identify a language model to be used and/or to build a language model on the fly, such as by dynamically applying particular weightings to different language models (which may each be derived from different input corpuses).
  • FIG. 1A is a conceptual diagram of an example system 100 that includes a multi-modal input method editor (IME) 102 .
  • the IME 102 is implemented in a mobile electronic device 104 , though it should be understood that the IME 102 can be implemented in a different electronic device, e.g., a PC, laptop computer, PDA, etc.
  • the electronic device 104 includes multiple user input devices 106 , including a microphone 105 to receive spoken user input.
  • Other user input mechanisms include a keyboard, which can include a soft or virtual keyboard (e.g., a touchscreen keyboard 108 ) or a hard or physical keyboard, a mouse, a trackball, and the like.
  • the user input mechanisms 106 are capable of receiving spoken input (i.e., by the microphone) and written input (i.e., by the keyboard 108 ).
  • the user input can be received by the electronic device 104 for use as input into one of various applications 110 that can execute on the electronic device 104 , e.g., a web browser, an e-mail application, a word processing application, a contacts book, and/or a calendar.
  • the user input is an input into a web form on a particular web page of a particular web site.
  • the IME is generally application-independent, i.e., can be used for most any of the applications 110 .
  • the spoken input can be provided to a remote server for conversion to text.
  • the speech utterance 111 can be transmitted over the network 112 to a remote server 114 that includes a speech service 116 and speech recognizer system 118 .
  • the network 112 can include one or more local area networks (LANs), a wide area network (WAN), such as the Internet, a wireless network, such as a cellular network, or a combination of all of the above.
  • LANs local area networks
  • WAN wide area network
  • wireless network such as a cellular network
  • the speech recognizer system 118 can use one or more language models 120 to recognize text from the speech utterance.
  • the text 113 which can be a selected best candidate or can be a list of n-best candidates that correspond to the speech utterance, is provided back to the electronic device 104 over the network 112 .
  • the text 113 can be displayed to the user on a display 122 of the electronic device 104 .
  • the user 101 can select a candidate from the list that corresponds to the user's spoken input, for example, using the keyboard 108 or another input mechanism, such as touching the touch screen over one of the candidates, to navigate the list and make a selection.
  • the user can also provide written input, and can provide input using a combination of written and spoken input.
  • the user can begin a search query in a web browser by speaking one or more words and can then add to the query string by typing additional input using the keyboard 108 .
  • the IME 102 can provide the combined user input to the relevant application, i.e., the web browser application in this example.
  • the language that the written input is written in can be determined and then provided as a language indicator to the remote server 114 .
  • the remote server 114 can use the language indicator when converting the speech utterance 111 to the text 113 . For example, by knowing the language in which the speech is spoken, an appropriate language model 120 can be selected for use by the speech recognizer 118 .
  • FIG. 1B is a block diagram of an example system 130 that can be used to implement a multi-modal IME.
  • the example system 130 can be implemented, for example, in a computer device, such as a personal computer device, or other electronic devices, such as a mobile phone, mobile communication device, personal digital assistant (PDA), Global Positioning System (GPS) navigation device, and the like.
  • a computer device such as a personal computer device
  • PDA personal digital assistant
  • GPS Global Positioning System
  • the example system 130 includes a processing device 132 , a first data store 134 , a second data store 136 , input devices 138 , output devices 140 , and a network interface 142 .
  • a bus system 144 including, for example, a data bus and a motherboard, can be used to establish and control data communication between the components 132 , 134 , 136 , 138 , 140 and 142 .
  • Other system architectures can also be used.
  • the processing device 132 can, for example, include one or more microprocessors.
  • the first data store 134 can, for example, include a random access memory storage device, such as a dynamic random access memory, or other types of computer-readable medium memory devices.
  • the second data store 136 can, for example, include one or more hard drives, a flash memory, and/or a read only memory, or other types of computer-readable medium memory devices.
  • the input devices 138 include at least one input device that is configured to receive spoken input and at least one input device configured to receive written input.
  • Example input devices 138 can include a microphone, keyboard, a mouse, a stylus, etc.
  • example output devices 140 can include a display device, an audio device, etc.
  • the network interface 142 can, for example, include a wired or wireless network device operable to communicate data to and from a network 146 .
  • the network 146 can include one or more local area networks (LANs), a wide area network (WAN), such as the Internet, a wireless network, such as a cellular network, or a combination of all of the above.
  • LANs local area networks
  • WAN wide area network
  • wireless network such as a cellular network
  • the system 130 can include input method editor (IME) code 131 from a data store, such as the data store 136 .
  • the input method editor code 131 can be defined by instructions that upon execution cause the processing device 132 to carry out input method editing functions.
  • the input method editor code 131 can, for example, include interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, that can be executed in a web browser environment.
  • Other implementations can also be used, e.g., a stand-alone application, an applet, a plug-in module, etc., for use in a user interface, such as a display that displays user inputs received by use of keypad mapping for a mobile device or keyboard mapping for a mobile device or personal computer.
  • Execution of the input method editor code 131 generates or launches an input method editor instance (IMEI) 133 .
  • the input method editor instance 133 facilitates the processing of one or more input methods at the system 130 , during which time the system 130 can receive inputs for characters or symbols, such as, for example, spoken or written input.
  • the user can use one or more of the input devices 138 , e.g., a microphone for spoken input or a keyboard for written input.
  • the user input can be Roman characters that represent input in a first writing system, e.g., Pinyin, and the input method editor can convert the input to a second writing system, e.g., Hanzi terms.
  • a Hanzi term can be composed of more than one Pinyin input.
  • the first data store 134 and/or the second data store 136 can store an association of inputs. Based on a user input, the input method editor instance 133 can use information in the data store 134 and/or the data store 136 to identify one or more candidate selections represented by the input. In some implementations, if more than one candidate selection is identified, the candidate selections are displayed on an output device 140 . For example, if the user input is spoken input, then a list of candidate selections showing written text representations of the spoken input can be presented to the user on the output device 140 . In another example, if the user input is Pinyin inputs, the user can select from the candidate selections a Hanzi term, for example, that the user desires to input.
  • a remote computing system 148 having access to the system 130 can be used to convert spoken user input to written user input.
  • the remote system 148 can be a server that provides a speech recognition service via the network 146 .
  • One or more speech utterances forming the spoken input can be transmitted to the remote system 148 over the network 146 .
  • the remote system 148 can determine a text conversion of the spoken input, for example, using a convenient form of speech recognizer system, and transmit the text conversion to the system 130 .
  • the text conversion can be a best candidate for text corresponding to the spoken input or can be a list of n-best candidate selections for presentation to the user for selection as the input.
  • the speech recognizer system can include Hidden Markov Modeling (HMM) encoded in a finite state transducer (FST). Other configurations of speech recognizer can be used by the remote system 148 .
  • HMM Hidden Markov Modeling
  • FST finite state transducer
  • the remote system 148 can also be used to edit a logographic script.
  • the remote system 148 may be a server that provides logographic script editing capability via the network 146 .
  • a user can edit a logographic script stored in the data store 134 and/or the data store 136 using a remote computing system, e.g., a client computer.
  • the system 130 can, for example, select a character and receive an input from a user over the network interface 142 .
  • the processing device 132 can, for example, identify one or more characters adjacent to the selected character, and identify one or more candidate selections based on the received input and the adjacent characters.
  • the system 130 can transmit a data communication that includes the candidate selections back to the remote computing system.
  • FIG. 1C includes a block diagram of example software that can be used to implement an input method editor in FIG. 1B (e.g., IMEI 133 ).
  • the system 160 includes a user interface 162 and software 164 .
  • a user 166 can access system 160 through the user interface 162 .
  • the software 164 includes applications 165 , IME engine 166 , an operating system (OS) 167 , a speech recognition system 169 including a language model 168 , and a detection engine 170 .
  • the operating system 167 is a particular piece of software that can provide the user interface 162 between the software 164 (e.g., applications 165 and IME engine 166 ) and the user 166 .
  • the speech recognition system 169 and language model 168 are separate from IME engine 166 .
  • the speech recognition system 169 and language model 168 (which can include two or more language models) are included within software 164 as a separate software component.
  • the speech recognition system 169 and language model 168 can be located remotely (e.g., at the remote system 148 of FIG. 1B ).
  • the speech recognition system 169 and language model 168 can be included within the IME engine 166 .
  • the language model 168 can define one or more language sub-models, each sub-model tailored to a particular application, or webpage, or webform on a particular webpage, or website, to name a few examples.
  • Each language sub-model can, for example, define a particular rule set, e.g., grammar particular to a language, phrase sets, verbals, etc., that can be used to determine a user's likely intent in entering a set of inputs (e.g., inputs for generating candidates that are translations, transliterations, or other types of phonetic representations).
  • each language sub-model can also include a user history of a particular user, e.g., a dictionary of words and phrased often used by a particular user.
  • the detection engine 170 includes an input module 172 and can include a timing module 174 .
  • the input module 172 can, for example, receive input (e.g., keystrokes representing characters or a speech utterance) to particular applications 165 and send the received input to the IME engine 166 .
  • the detection engine 170 is a component of the IME engine 166 .
  • the detection engine 170 can detect input and determine whether or not to send the input to the IME engine 166 .
  • the IME engine 166 can, for example, be implemented using the input method editor code 131 and associated data stores 134 and 136 , and provide output candidates in text converted from speech to an interface (e.g., user interface 162 ) as the input (e.g., speech utterances) is detected, as described with reference to FIGS. 2 and 3 A-E below.
  • the components of system 160 can be communicatively coupled to one or more of each other. Though the components identified above are described as being separate or distinct form each other, one or more of the components may be combined in a single system, or to perform a single process or routine.
  • the functional description provided herein including separation of responsibility for distinct functions is by way of example.
  • Other storage architectures can also be used. In particular, other groupings or other divisions of functional responsibilities can be made as necessary or in accordance with design preferences.
  • IME engine 166 can perform the functions of detection engine 170 .
  • input module 172 and timing module 174 can be combined into a single module.
  • FIG. 2 is a flowchart of an example process 200 for using an input method editor to receive spoken input from a user input device and to provide written, or textual, input to a corresponding application.
  • a request is received from a user for an application-independent input method editor that has written and spoken input capabilities (Step 202 ).
  • the request can be received by a mobile electronic device that has a touchscreen keyboard.
  • Example screenshots from such a mobile electronic device are shown in FIGS. 3A-E . These screenshots can be used to illustrate the example process 200 ; however, it should be understood that other devices can implement the process 200 , and the screenshots shown are not intended to be limiting.
  • FIG. 3A shows a screenshot 300 where a user has selected to activate a web browser application. Through the web browser application, the user has selected to navigate to the Google search page at the URL www.google.com 302 .
  • FIG. 3B shows a screen shot 304 with a soft touchscreen keyboard 306 displayed in a lower portion of the display screen. For example, the user can touch or tap the screen in the search query field 308 to automatically have the keyboard 306 displayed, although other mechanisms can be used to trigger the display of the keyboard 306 .
  • the example keyboard 306 shown includes a microphone key 310 .
  • An example of a request that can be received from the user includes the user selecting the microphone key 310 .
  • Another example includes the user selecting a graphical entity, such as a microphone icon or button, displayed next to or in an input field, e.g., in search query field 308 .
  • a graphical entity such as a microphone icon or button
  • Another example includes the user swiping his/her finger across the input field, e.g., in a left to right motion, or tapping the input field.
  • Yet another example includes the user picking up the device in a manner that is consistent with raising a microphone included in the device to the proximity of the user's mouth, which can be detected, for example, by an accelerometer reading.
  • Other forms of request can be received from the user for an application-independent input method editor having written and spoken input capabilities, and the above are but some examples.
  • a user's intention to provide spoken input to the application-independent input method editor is then identified in the process (Step 204 ). For example, receiving a speech utterance from the user can be used to identify that the user intends to provide spoken input. In other implementations, receiving the request from the user for the input method editor with written and spoken input capabilities can also be used to identify that the user intends to provide spoken input, i.e., the same user action can provide both the request and be used to identify the user's intention. In some implementations, as shown in the screenshot 312 in FIG. 3C , a graphical element can be displayed that prompts the user to speak, such as the microphone graphic 314 and the instructions “Speak now” 316 .
  • a spoken input i.e., a speech utterance
  • the user provides the spoken input as input to an application that is executing on the device (Step 206 ).
  • the spoken input is provided to a remote server that includes a speech recognition system configured to recognize text based on the spoken input (Step 208 ).
  • the spoken input can be sent over the network 146 to the remote system 148 , where the remote system 148 includes a speech recognition system to recognize text from a speech utterance.
  • a graphic is displayed to the user to indicate that the process is in progress, such as the “Working” graphic 320 shown in the screenshot 322 in FIG. 3D .
  • Text is then received from the remote server, where the text represents the spoken input (Step 210 ).
  • the remote server e.g., remote system 148
  • the corresponding text is sent back to the user's device and can be displayed for the user.
  • the best candidate for representation of the speech utterance is selected by the speech recognition system at the remote server and provided to the device.
  • an n-best list of candidates can be provided and presented to the user for selection of the correct candidate. For example, referring to FIG. 3E , a screen shot 324 shows a list of suggestions, with the best candidate “the man in the moon” displayed at the top of the list as the default selection.
  • the text i.e., the spoken input converted to written input
  • the application as user input (Step 212 ). That is, once the correct text conversion is selected, if a list of candidates was provided, or once the best candidate has been received, if only one was sent from the remote server, the written input can be passed to the application as the user input for processing by the application.
  • a context indicator can be sent with the spoken input to the remote system for conversion to text.
  • the remote system can use the context indicator to facilitate the speech-to-text conversion.
  • the context indicator can be used as a basis for selecting an appropriate language model to use by the speech recognition system.
  • the context indicator can specify the context in which the spoken user input was received.
  • the context indicator can specify a name of a field, e.g., in a web form, the name of the application in which the input was received, and/or identify a web page if the user input was received in a web browser application.
  • the context indicator can include metadata relating to a field in which the user input was received.
  • the metadata can specify that the field requires a one-word answer, or a date, or a name, and the like.
  • the context indicator information can be obtained by the input method editor from the operating system of the electronic device.
  • the device may pass a sound file (including streaming sound data) of the spoken input to a remote server system, and may take steps to improve the quality of the speech-to-text conversion.
  • the device may pass information that allows the server system to select a language model that is relatively small in size and is specific to the task that the user is currently facing. For example, when applications register with the IME, they may provide information about fields into which a user can enter information in the applications. The IME can pass such information to the server system, so that the server system may select an appropriate language model.
  • the IME can pass such information to the server system so that, for example, a user utterance that sounds like “scheet” is interpreted as “street” and not “sweet.”
  • the language model that is applied may also be a composite of, or interpolation of, multiple separate language models.
  • the different models may be relatively small models that have been derived from large data corpuses (e.g., SMS messages and e-mail messages).
  • the models may be prepared off-line by analyzing such corpuses, and the mixture weights that are applied to the models may be generated on-the-fly at run-time, including after data from a particular instance of speech input starts being received from a user.
  • the weightings may be a function, for example, of the field into which a user is currently making an utterance (e.g., the “to” or “from” fields of an email message versus the “body” field of an email message).
  • the language model may also be chosen more generally, without reference to a particular input field in which the cursor is located.
  • an application developer may register their application with an app store, and may indicate the type of application that it is, such as a music player.
  • a server system may have a number of topic-specific language models that it stores.
  • the IME may pass a “music” indication to the server system, so that the application is passed the word “heart,” and the user sees a song list for the female-led rock band.
  • the IME may pass the word “Hart” (assuming there are no great movies or television shows with the word “heart” in their titles) so that the user is shown an index of the episodes of the iconic detective drama “Hart to Hart.”
  • the domain or a sub-domain for the page may be provided, so that the that the language model will be specific to the particular type of web site.
  • the language model may be one in which “shopping” terms have higher prominence.
  • “product” may have a higher score than “protect” for similar sounds.
  • Such a model may be prepared to be directed to the site itself (e.g., by analyzing input forms on the site, and analyzing text on the site), or on a category that the site matches.
  • the same language model may be used for the sites Amazon.com, Buy.com, and the like.
  • FIG. 4 is a block diagram of an example system 400 for receiving speech input and training language models to interpret the speech input.
  • users of client devices enter data into text input fields, and a speech server analyzes that data to determine the type of text the users entered.
  • the speech server builds language models from this information, and uses the language models to recognize speech input associated with similar input fields.
  • Clients 402 can receive text input for web forms. When this input is used as part of a process to browse to another page, such as typing in a search field or order form field before pressing a submit button, the text and destination information can be saved in a toolbar log 404 .
  • Clients 406 such as computers and mobile computing devices, can receive text based web search queries from users. These queries can be resolved by a web search server (not shown) and the queries and search results can be stored in query logs 408 .
  • the client 406 can receive text input to applications, such as an email client, a messaging client, and/or a word processor and spoken input to applications. The text input and transcriptions of spoken input can be stored in input logs 409 .
  • a speech recognition server 410 can retrieve the data in the toolbar logs 404 , the query logs 408 , and/or the input logs 409 .
  • a speech recognition system 412 can group this data into categories or classifications.
  • the speech recognition system 412 creates a series of language models 414 a - 414 n .
  • the language models can contain words, phrases, sentences, etc. from the toolbar logs 404 , query logs 408 , and/or the input logs 409 based on a particular topic.
  • the language model A 414 a is a language model of British last name, it can contain “Churchill,” “Bean,” and “Pigou.”
  • the language model B 414 b contains technology-related language, it can contain “object oriented programming,” “reverse polish notation,” and “garbage in, garbage out.”
  • the speech recognition system 412 can build interpolated language models from the language models 414 a - 414 n .
  • the speech recognition system 412 can use an utterance with a known text result, and poll the language models 414 a - 414 n to determine the confidence level that each language model 414 a - 414 n would associate with the utterance/known text pair.
  • the confidence level returned by each language model 414 a - 414 n can be used by the speech recognition system 412 to create an interpolated language model. For example, using a test utterance and text of “Mr.
  • the language model A 414 a gives a confidence level of 50%
  • the language model B 414 b give a confidence level of 65%
  • the other language models 414 c - 414 n give substantially 0% confidence levels.
  • the speech recognition system 412 can create an interpolated language model that heavily weighs the language model A and B 414 a - b .
  • the particular weightings may be equal to or proportional to the confidence levels, in certain implementations.
  • a client device 416 can execute an application that accepts text input and can receive an utterance from a user for that text input.
  • the client device 416 can transmit the utterance, along with the context in which the utterance was made or received, to the speech recognition server 410 for translation into text.
  • the context in which an utterance is received is determined by the client device 416 with the field name or label associated with the text input.
  • some applications include metadata for an application input field such as package name, field number or name, and/or attribute flags (e.g. long sentence, email, street address, etc.).
  • a text label associated with or displayed near an input box is used to determine the context.
  • a speech service 418 can receive the utterance and context.
  • the speech service 418 can map the context to broader categories or to the categories of the language models 414 a - 414 n .
  • the speech service 418 can maintain a cache of utterances and resultant texts, and if a received utterance substantially matches a cached utterance, the speech service 418 can return the appropriate cached resultant texts.
  • the speech recognition system 412 uses or creates an interpolated language model to recognize the text in the utterance.
  • a single candidate text is identified.
  • the top n candidates are identified either wherein all candidates meet a certain confidence threshold, or wherein the top n candidates are selected.
  • the text or texts identified by the speech recognition system 412 is returned to the client 416 , where it is, for example, displayed in a text input field that has focus.
  • Such context-specific language models may be generated in a variety of ways, including by using search information from a public search engine. For example, bi-grams may be made between search queries and web sites that are highly relevant to those queries (e.g., are in the list of top 10 sites returned to the queries). The queries and sites may then be assigned as nodes in a graph, joined by the relationship between the two. Clusters of such relationships may then be identified, and language models may be generated for those clusters. A particular language model for a web site may then be generated as a weighted combination of the language models for the clusters around the site in the graph.
  • FIG. 5A is a block diagram 500 of queries and associated websites in search results.
  • a group of queries 502 is a collection of example queries that can be collected from a browser executing a toolbar.
  • the queries can be submitted to a web search engine, and a list of search results can be returned.
  • a search for “mike lebeau” can result in search results including facebook.com and linkedin.com.
  • a search for “blink” can result in search results including amazon.com.
  • the queries and associated websites can be used, for example, in determining a relationship between queries and related topics or categories.
  • facebook.com and linkedin.com are social network websites, it can be determined that the queries “bill byrne,” “will rusch,” “mike lebeau,” and “brian stope” may be the names of people.
  • amazon.com is a retailer with a reputation as a book retailer, it can be determined that “blink” and “great gatsby” may be the names or titles of a retail products, perhaps books.
  • a search for “william gibson” returns links to both facebook.com and amazon.com, it can be determined that “william gibson” may be an author.
  • FIG. 5B is a URL showing search terms in a query.
  • the URL in this example can be recorded by a web browser executing a toolbar.
  • a user submits a form via a GET request, the contents of the form can be encoded and logged in the URL parameters.
  • a search on the webpage facebook.com can generate the URL shown.
  • the name parameter 550 in this example “bill%20byrne”, indicates that “bill byrne” was entered into a field in a web form.
  • FIG. 6 is a block diagram of an example system 600 for recognizing speech input to a computing device.
  • a user enters speech input to an application via an IME.
  • the speech input is analyzed at a speech recognition server and text is returned to the application.
  • An application 604 executes on a computing device 602 , for example a mobile computing device.
  • An input method manager 608 can manage input methods that the application 604 accepts, including speech input from a user.
  • a speech IME 610 in the operating system of the computing device 602 records speech input and collects context data from an input context 612 .
  • Speech IME 160 can transmit the speech input and context to a voice data center 614 .
  • a speech service front end 616 can receive the speech input and context and prepare the speech input and context for analysis. For example, statistics can be collected, recognition jobs can be created and sorted, etc.
  • a speech recognition system 618 examines the context, and selects an associated language model from a big table of language models 622 .
  • the associated language model in the big table of language models 622 can contain a reference to a language model in a ProdLM 620 .
  • the speech recognition system 618 uses the referenced language model in the ProdLM 620 to analyze the speech input and determine a text string.
  • the text string is returned to the speech input method 610 , for example by way of the speech service front end 616 .
  • the text string can be presented to the user of the computing device 602 , and if approved by the user, can be sent to the application 604 as text input.
  • data obtained from web toolbar logs can be used in determining an interpolated language model.
  • a sample of content of user input (e.g., queries) into a web form can be obtained from web toolbar logs, e.g., in URLs. That is, when a user submits a web form via a GET request, the content of the user input into fields of the web form can be encoded and logged in the URL parameters (e.g., as was described above in reference to FIG. 5B ).
  • a toolbar log can therefore provide sample data of what users have input (i.e., typed) into each of the fields of a number of web forms.
  • the volume of this data may be too sparse to train a good language model for the web form.
  • a larger training set can be available if queries of the same category as the queries input into the particular web form are used. That is, queries that correspond to search results having similar websites in the top N search results (where N is an integer, e.g., 10) as each other are probably generally within the same category.
  • clusters in a search query—search result (website) graph (where queries and websites are assigned to nodes) can be identified, where an edge between a particular query node and a particular website node occurs whenever the website appears in the top N results for the particular query.
  • a language model can then be trained per cluster, that is, language models can be trained for a category (if a cluster is representative of a category of queries).
  • the interpolated language model can be built using multiple language models formed from a number of clusters.
  • the multiple language models can be selected as follows.
  • a sample of user input into the particular web form i.e., sample queries
  • the sample data may be the query “bill byrne” that is input into a web form on the www.facebook.com website, which is also reflected in the graph shown in FIG. 5A .
  • “bill byrne” belongs to a cluster that includes people-oriented websites (e.g., facebook.com and linkedin.com) and people names (Bill Byrne, Will Rusch, Mike Lebeau, Brian Strope and William Gibson).
  • the “bill byrne” query belongs to only the people names cluster. However, a query may belong to many clusters. In such instances, for the sample queries obtained from the web toolbar data, the K most significant clusters (K is a number/integer) can be identified and weighted, with the sum of their weights adding to 1.0.
  • the K most significant clusters can be determined by counting the number of sample queries (from the web toolbar log sample data) that fall into each cluster and then optimizing the weights of only the top K clusters. Other techniques to determine the K most significant clusters can be used.
  • the interpolated language model for the web form can be used to convert a speech input received for a field in the form into text. The converted text can be transmitted to an electronic device that initially received the speech input, or transmitted elsewhere, to be used as text input for the field in the web form.
  • FIG. 7A shows a flowchart of an example process 700 of building an interpolated language model for a website (or a web form of a website).
  • pairs of queries and results are collected, base language models are created, and interpolated language models are created.
  • pairs containing web queries and result sites are extracted from web search logs.
  • a web browser or web browser add on can report queries, query result URLs, or search results to a central repository.
  • a web search server can create a log of search terms and website domain names that were returned in response to those search terms.
  • a cluster bipartite graph also known as a bigraph, is formed by the pairs.
  • a bipartite graph is a graph whose vertices can be divided into two disjoint sets such that every edge connects a vertex in the first set to one in the second set.
  • the first set can contain nodes representing queries and the second set can contain nodes representing results.
  • Clusters are identified in the bipartite graph, for example so that each cluster consists of queries and results that may by semantically or syntactically similar.
  • step 706 language models are trained based on the queries in the clusters.
  • sample utterances are recorded by a range of speakers and associated with the queries.
  • the utterances for each query can be aggregated, such as by determining an average waveform and a deviation factor indicating how much deviation from the mean was found for most utterances.
  • web form sample data is obtained from toolbar logs, the web form sample data relating to the particular web form (or website) for which the interpolated language model is being generated (e.g., the interpolated language model can be for the web form).
  • a web browser toolbar can record and transmit the URLs of websites visited by the browser.
  • the URLs can include input text (i.e., sample queries) that has been entered into the web form and submitted. This information can be used to associate a particular website domain, or web page with a particular topic or idea.
  • the K most significant clusters for the input text i.e., the sample queries, obtained from the web form sample data are identified.
  • a list of cluster names is collected and ranked in order of significance or similarity to the data collected in the step 708 .
  • a predetermined number (K, in this case) of the most significant clusters are identified. For example, clusters are identified that include the sample queries.
  • the number of sample queries that fall into each cluster can be counted and the weights assigned to each cluster optimized for only the top K clusters.
  • an interpolated language model for the web form is built.
  • a data structure is created that contains links to the K most significant language models. This data structure represents a combined language model that delegates speech recognition functionality to the referenced language models.
  • each referenced language model can have a weight, such as a percentage, integer in a particular range, or probability, associated with the referenced language model in the interpolated language model. In some implementations, this weight can be determined by testing a known pairs of utterances and text. The pairs can be submitted to the referenced language models, which can return a confidence level representing the likelihood or accuracy of the language model to correctly identify the text in the utterance. The confidence levels, optionally normalized or otherwise converted, can be used as weighting values in the interpolated language model. Thus, reference language models likely to correctly identify text in a particular classification are most heavily weighted.
  • step 704 can be used in the step 704 .
  • input field flags and descriptions are obtained in the step 708 .
  • step 702 and step 714 can be combined using a process that builds an interpolation model and interpolation weights in one step.
  • FIG. 7B shows a flowchart of an example process 750 of building an interpolated language model.
  • text input and associated metadata is collected, base language models are created, and interpolated language models are created using the base models and interpolation factors such as weightings to be provided to each of multiple base models.
  • step 752 text input and input metadata pairs are extracted from input logs.
  • an application can report to a central repository input entered into an input field and metadata for the input field.
  • the text input can include voice input that is transcribed into a text format.
  • application wide or operating system wide metadata schemes can define or describe the type of input field, such as an address field, a free form text field, a search field, or a social status field.
  • a cluster bipartite graph also known as a bigraph, is formed by the pairs.
  • a bipartite graph is a graph whose vertices can be divided into two disjoint sets such that every edge connects a vertex in the first set to one in the second set.
  • the first set can contain nodes representing input metadata and the second set can contain nodes representing text input.
  • Clusters are identified in the bipartite graph, for example so that each cluster consists of similar input metadata.
  • step 756 language models are trained based on the text input in the clusters.
  • sample utterances are recorded by a range of speakers and associated with the metadata.
  • the utterances for each text input can be aggregated, such as by determining an average waveform and a deviation factor indicating how much deviation from the mean was found for most utterances.
  • input metadata is categorized.
  • the input metadata can be used to associate a particular input field or application with a particular topic or idea.
  • Example topics or ideas include, but are not limited to, address fields, free form text fields, search field, social status fields, and numeric fields.
  • step 760 the K most significant clusters for a category are identified.
  • a list of cluster names is collected and ranked in order of significance or similarity to the data collected in the step 758 .
  • a predetermined number (K, in this case) of the most significant clusters are then identified.
  • an interpolated language model is built.
  • a data structure is created that contains links to the K most significant language models. This data structure represents a combined language model that delegates speech recognition functionality to the referenced language models.
  • K interpolation weights are optimized.
  • Each referenced language model that is referenced in the data structure can have a weight, such as a percentage, integer in a particular range, or probability, associated with it. In some implementations, this weight can be determined by testing a known pair or pairs of utterances and text. The pairs can be submitted to the referenced language models, which can return a confidence level representing the likelihood or accuracy of the language model to correctly identify the text in the utterance. The confidence levels, optionally normalized or otherwise converted, can be used as weighting values in the interpolated language model. Thus, reference language models that are likely to identify text in a particular classification correctly are most heavily weighted.
  • a data structure with weighted links to the K most significant language models can be selected from a collection of some or all possible data structures with weighted links to language models.
  • FIG. 8 shows a flowchart of an example process 800 for recognizing text in an utterance.
  • an utterance is received, the speech in the utterance is recognized, and text, or a list of text options, is returned.
  • an utterance is received.
  • a speech recognition server can receive translation requests from clients.
  • the translation request can include an utterance (e.g. an audio file) and a context (e.g. text or other data describing how the utterance may be used or categorized).
  • a relevant language model is determined. For example, an interpolated language model for web search, free form text input, or social status can be determined.
  • language models including interpolated language models, can be indexed by keyword, web domain, application type, or other criteria. Using metadata associated with the utterance, such as context or source information, a relevant language model can be determined.
  • step 806 speech recognition is performed.
  • the utterance can used by a speech recognition application using the language model.
  • the speech recognition application can calculate one or more candidate text strings from the utterance.
  • a confidence level can be associated with each text string.
  • a K-best list of text strings representing the utterance is returned (where K is an integer).
  • K is an integer.
  • the text string with the highest confidence value is selected and returned.
  • a particular number (K, in this case) of text strings with the highest confidence level are returned.
  • step 808 all text strings with a confidence level above a particular threshold can be returned.
  • an utterance can be preprocessed before step 806 to improve recognition.
  • FIG. 9 shows an example of a generic computer device 900 and a generic mobile computer device 950 , which may be used with the techniques described here.
  • Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers.
  • Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices.
  • the components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.
  • Computing device 900 includes a processor 902 , memory 904 , a storage device 906 , a high-speed interface 908 connecting to memory 904 and high-speed expansion ports 910 , and a low speed interface 912 connecting to low speed bus 914 and storage device 906 .
  • Each of the components 902 , 904 , 906 , 908 , 910 , and 912 are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate.
  • the processor 902 may process instructions for execution within the computing device 900 , including instructions stored in the memory 904 or on the storage device 906 to display graphical information for a GUI on an external input/output device, such as display 916 coupled to high speed interface 908 .
  • multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory.
  • multiple computing devices 900 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).
  • the memory 904 stores information within the computing device 900 .
  • the memory 904 is a volatile memory unit or units.
  • the memory 904 is a non-volatile memory unit or units.
  • the memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.
  • the storage device 906 is capable of providing mass storage for the computing device 900 .
  • the storage device 906 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations.
  • a computer program product may be tangibly embodied in an information carrier.
  • the computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above.
  • the information carrier is a computer- or machine-readable medium, such as the memory 904 , the storage device 906 , memory on processor 902 , or a propagated signal.
  • the high speed controller 908 manages bandwidth-intensive operations for the computing device 900 , while the low speed controller 912 manages lower bandwidth-intensive operations.
  • the high-speed controller 908 is coupled to memory 904 , display 916 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 910 , which may accept various expansion cards (not shown).
  • low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914 .
  • the low-speed expansion port which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
  • input/output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
  • the computing device 900 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 920 , or multiple times in a group of such servers. It may also be implemented as part of a rack server system 924 . In addition, it may be implemented in a personal computer such as a laptop computer 922 . Alternatively, components from computing device 900 may be combined with other components in a mobile device (not shown), such as device 950 . Each of such devices may contain one or more of computing device 900 , 950 , and an entire system may be made up of multiple computing devices 900 , 950 communicating with each other.
  • Computing device 950 includes a processor 952 , memory 964 , an input/output device such as a display 954 , a communication interface 966 , and a transceiver 968 , among other components.
  • the device 950 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage.
  • a storage device such as a microdrive or other device, to provide additional storage.
  • Each of the components 950 , 952 , 964 , 954 , 966 , and 968 are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.
  • the processor 952 may execute instructions within the computing device 950 , including instructions stored in the memory 964 .
  • the processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors.
  • the processor may provide, for example, for coordination of the other components of the device 950 , such as control of user interfaces, applications run by device 950 , and wireless communication by device 950 .
  • Processor 952 may communicate with a user through control interface 958 and display interface 956 coupled to a display 954 .
  • the display 954 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology.
  • the display interface 956 may comprise appropriate circuitry for driving the display 954 to present graphical and other information to a user.
  • the control interface 958 may receive commands from a user and convert them for submission to the processor 952 .
  • an external interface 962 may be provide in communication with processor 952 , so as to enable near area communication of device 950 with other devices. External interface 962 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.
  • the memory 964 stores information within the computing device 950 .
  • the memory 964 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units.
  • Expansion memory 974 may also be provided and connected to device 950 through expansion interface 972 , which may include, for instance, a SIMM (Single In Line Memory Module) card interface.
  • SIMM Single In Line Memory Module
  • expansion memory 974 may provide extra storage space for device 950 , or may also store applications or other information for device 950 .
  • expansion memory 974 may include instructions to carry out or supplement the processes described above, and may include secure information also.
  • expansion memory 974 may be provide as a security module for device 950 , and may be programmed with instructions that permit secure use of device 950 .
  • secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.
  • the memory may include, for example, flash memory and/or NVRAM memory, as discussed below.
  • a computer program product is tangibly embodied in an information carrier.
  • the computer program product contains instructions that, when executed, perform one or more methods, such as those described above.
  • the information carrier is a computer- or machine-readable medium, such as the memory 964 , expansion memory 974 , memory on processor 952 , or a propagated signal that may be received, for example, over transceiver 968 or external interface 962 .
  • Device 950 may communicate wirelessly through communication interface 966 , which may include digital signal processing circuitry where necessary. Communication interface 966 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 968 . In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 970 may provide additional navigation- and location-related wireless data to device 950 , which may be used as appropriate by applications running on device 950 .
  • GPS Global Positioning System
  • Device 950 may also communicate audibly using audio codec 960 , which may receive spoken information from a user and convert it to usable digital information. Audio codec 960 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 950 . Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 950 .
  • Audio codec 960 may receive spoken information from a user and convert it to usable digital information. Audio codec 960 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 950 . Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 950 .
  • the computing device 950 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 980 . It may also be implemented as part of a smartphone 982 , personal digital assistant, or other similar mobile device.
  • Device 950 may also include one or more different devices that are capable of sensing motion. Examples include, but are not limited to, accelerometers and compasses. Accelerometers and compasses, or other devices that are capable of detecting motion or position are available from any number of vendors and may sense motion in a variety of ways. For example, accelerometers may detect changes in acceleration while compasses may detect changes in orientation respective to the magnetic North or South Pole. These changes in motion may be detected by the device 950 and used to update the display of the respective devices 950 according to processes and techniques described herein.
  • accelerometers may detect changes in acceleration while compasses may detect changes in orientation respective to the magnetic North or South Pole.
  • implementations of the systems and techniques described here may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof.
  • ASICs application specific integrated circuits
  • These various implementations may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • the systems and techniques described here may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user may provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
  • the systems and techniques described here may be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user may interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components.
  • the components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.
  • LAN local area network
  • WAN wide area network
  • the Internet the global information network
  • the computing system may include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • FIG. 10 is a block diagram of an example data structure 1000 of a language model.
  • the data structure 1000 can weightedly link a semantic idea or category with language models, for example, for use in speech recognition.
  • a semantic category 1002 such as a search query or type of input field, can be linked to one or more of a collection of interpolated language models 1004 .
  • the interpolated language models 1004 can be linked to one or more base language models 1006 .
  • the links between the interpolated language models 1004 and base language models 1006 can be weighted. In some examples, the sum of the weights of the links from one interpolated language model can be 1 or can be normalized to 1.
  • interpolated language models 1004 may exist that every weighted combination of base language models 1006 has a linked interpolated language model 1004 .
  • interpolated language model can be linked to more or fewer base language models 1006 .
  • different link weights between interpolated language models 1004 and base language models 1006 may exist, such as positive integers, probabilities, or dimensional distance (e.g. W,X,Y,Z values for four dimensional space.)
  • multiple semantic categories 1002 can be linked to a single interpolated language model 1004 .
  • interpolated language models 1004 can be created on demand, such as when a semantic category 1002 is added to the data structure.
  • the interpolated language models 1004 can persists after the removal of semantic categories (not shown). For example, previously removed semantic categories (not shown) may have prompted the creation of the interpolated language models 1004 a , 1004 b , and 1004 c .
  • a new semantic category 1008 can be added to the data structure 1004 , and linked to any of the interpolated language models 1004 .
  • every possible interpolated language model 1004 can be pre-created for use by a new semantic category 1008 . Some of these examples can be used in situations, such as when many new semantic categories 1008 are expected, when pre-processing time is available, and/or when few base language models 1006 are expected.

Abstract

Methods, computer program products and systems are described for forming a speech recognition language model. Multiple query-website relationships are determined by identifying websites that are determined to be relevant to queries using one or more search engines. Clusters are identified in the query-website relationships by connecting common queries and connecting common websites. A speech recognition language model is created for a particular website based on at least one of analyzing at queries in a cluster that includes the website or analyzing webpage content of web pages in the cluster that includes the website.

Description

    CROSS REFERENCE TO RELATED APPLICATION
  • This application claims priority to pending U.S. Provisional Application Ser. No. 61/289,968, entitled “Multi-Modal Input on an Electronic Device”, filed on Dec. 23, 2009, the entire contents of which are hereby incorporated herein by reference, and claims priority to pending U.S. Provisional Application Ser. No. 61/330,219, entitled “Multi-Modal Input on an Electronic Device”, filed on Apr. 30, 2010, the entire contents of which are hereby incorporated herein by reference.
  • TECHNICAL FIELD
  • This document relates to systems and techniques for multi-modal input into an electronic device and conversion of spoken input to text.
  • BACKGROUND
  • Computer users employ a number of mechanisms to provide input to their computing devices. Keyboards are common input devices, and they typically include single-digit numbers (e.g., in a cellular telephone) each of the letters in the alphabet, and some characters (e.g., in Qwerty or Dvorak keyboards). On mobile devices, keyboards are frequently “virtual” in form, and are displayed on a touch screen of a device. Such keyboards may be made available to various different applications running on a device, using a program known as an Input Method Editor, or IME, so that the IME receives the user input and then passes it to whatever application is currently active on the device. An IME can also translate user input, such as when a user enters Roman characters in a written language like Pinyin, and the IME generates Chinese characters that correspond to the typed Pinyin. Where the Pinyin corresponds to multiple possible characters, the IME can display all such characters, the user can tap the intended character, and the IME can pass that character to the operating application.
  • Users of computer devices, and particularly of mobile computing devices, may be constrained in their use of a keyboard. For example, the keyboard itself may be constrained in size because mobile device displays are small, so that only a sub-set of relevant characters can be displayed or the keys may be too small to press accurately. Also, the user may be constrained, in that they cannot easily type on a keyboard while walking through a crowded airport or driving a car. In such situations, spoken input may be preferred over typed input. However, speech-to-text conversion or translation typically requires lots of computer processing power, and mobile devices typically do not have much processing power. Also, such conversion often requires a particular user to “train” the system so that it better understands the user's voice and speech patterns.
  • SUMMARY
  • This document describes systems and techniques for multi-modal input into an electronic device, including speech input. In one example, an IME that provides keyboard services to a device may also provide for spoken input to the device. Much of the IME services may be unchanged from an ordinary IME, but the IME may be augmented to provide for speech-to-text conversion. Specifically, the IME may take a user input in whatever form (e.g., typed, spoken, D Pad, etc.) and may convert it to a standard form for presentation to an active application (e.g., passing characters or other key presses). Applications may subscribe to the IME's services just as they would subscribe if the IME did not accept voice input, and the provision of such speech-to-text functionality can be transparent to the applications.
  • A user could choose when to provide typed input and when to provide spoken input to the IME. First, the user may be in an application that requires input, and may take an appropriate action to invoke the IME (e.g., pressing a particular button or soft key on the device, moving the device in a certain manner, and the like). A keyboard may be initially displayed on the user's device, and the user may take another appropriate action to indicate that he or she will provide speech input. For example, the user may press a microphone button on the virtual keyboard, or may make a swipe across the virtual keyboard or another action that is inconsistent with an intent to provide typed input in the virtual keyboard. At that point, the IME can begin “listening” to the device's microphone and after the user has spoken, may pass corresponding text to the application.
  • In these manners, certain implementations may provide one or more benefits. For example, speech-to-text functionality may be provided on a computing device relatively simply, while re-using other IME functionality (e.g., interfaces to applications on a device) that is needed for keyboard-based IME translation. Use of context-specific language models in the manners discussed above and below may also permit more accurate conversion of speech to text, regardless of whether the system is trained to a particular user. Such context-specificity may also be provided automatically and transparently for a user, and at a level of specificity that is most suited to a particular situation. For example, where a user is simply interacting with a web page, the language model for the page may be used, but if the user is interacting with a form on the page, a more specific language model that is directed to that form or a particular field on the form may be used (with lower weightings, but higher-than-normal weightings, applied to the page-specific model).
  • Particular manners of using public user activity, such as search activity, to build contextual language models may also result in the generation of accurate models in a convenient manner. Such data may be made available via natural user activity that is already occurring voluntarily by the users. Also, the activity may occur in large enough volumes to provide enough data needed to generate a complete and accurate model for many contexts, and in this case, for many web pages. The models may also be updated over time, because the public user activity occurs continuously over time, so that continuously up-to-date models may be provided to users seeking speech-to-text conversion services.
  • In general, in one aspect, methods, computer program products and systems are described for a multi-modal input-method editor. A request can be received from a user of an electronic device for an application-independent input method editor having written and spoken input capabilities. That the user intends to provide spoken input to the application-independent input method editor can be identified, and the spoken input can be received from the user. The spoken input can be input to an application executing on the electronic device. The spoken input can be provided to a remote server. The remote server includes a speech recognition system configured to recognize text based on the spoken input. Text can be received from the remote server, where the text represents the spoken input. The text can be provided to the application as user input.
  • In general, in one aspect, methods, computer program products and systems are described for forming a speech recognition language model. Multiple query-website relationships are determined by identifying websites that are determined to be relevant to queries using one or more search engines. Clusters are identified in the query-website relationships by connecting common queries and connecting common websites. A speech recognition language model is created for a particular website based on at least one of analyzing at queries in a cluster that includes the website or analyzing webpage content of web pages in the cluster that includes the website.
  • Implementations of the methods, computer program products and systems can include one or more of the following features. Determining the multiple query-website relationships can include analyzing search results returned by the one or more search engines to the queries. Identifying clusters in the query-website relationships can include analyzing a graph of queries and websites at nodes of the graph, that are connected by edges defined by particular websites that are determined to be relevant to particular queries. A website can be determined to be relevant to a query if the website is a top n ranked search result for the query in a set of ranked search results relevant to the query, wherein n is a predetermined integer (e.g., is a top 10 ranked search result).
  • Creating the speech recognition model for the particular website can include creating cluster-based speech recognition models for each of multiple clusters, and creating the speech recognition model by interpolating among cluster-based speech recognition models for clusters that surround the particular website.
  • The details of one or more embodiments are set forth in the accompa-nying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.
  • DESCRIPTION OF DRAWINGS
  • FIG. 1A is a conceptual diagram of an example system including a multi-modal input method editor.
  • FIG. 1B is a block diagram of an example system that can be used to implement the multi-modal input method editor.
  • FIG. 1C is a block diagram of example software that can be used to implement the input method editor.
  • FIG. 2 is a flow chart of an example process for generating text using a multi-modal input method editor.
  • FIGS. 3A-E show example screen shots of an electronic device including a multi-modal input method editor.
  • FIG. 4 is a block diagram of an example system for receiving speech input and training language models to interpret the speech input.
  • FIG. 5A is a block diagram of queries and associated websites in search results
  • FIG. 5B is a URL showing search terms in a query
  • FIG. 6 is a block diagram of an example system for recognizing speech input to a computing device.
  • FIGS. 7A and 7B show flowcharts of example processes of building an interpolated language model.
  • FIG. 8 shows a flowchart of an example process for recognizing text in an utterance.
  • FIG. 9 shows examples of generic computer devices that may be used to execute the actions discussed in this document.
  • FIG. 10 is a block diagram of an example data structure of a language model.
  • Like reference symbols in the various drawings indicate like elements.
  • DETAILED DESCRIPTION
  • This document describes techniques and systems that may be used to provide speech-to-text conversion for a user of a computing device, such as a smartphone. In certain instances, the speech input may be handled in a manner similar to other input (e.g., typed input) using an application such as an IME, where the IME can be switched into modes depending on the manner that the user chooses to enter data. Where transformation of the input is needed, the input may in certain circumstances be transmitted (either in a raw or converted form) to a server system remote from the computing device that may be programmed to pass a transformed input back to the device, such as by providing text in response to receiving spoken inputs. The computing device may also provide the server system with meta data that is passed with, or at essentially the same time as, the spoken inputs, and the meta data may be used by the server system to identify a context in which the user is entering the spoken input. The server system may then use that meta data to identify a language model to be used and/or to build a language model on the fly, such as by dynamically applying particular weightings to different language models (which may each be derived from different input corpuses).
  • FIG. 1A is a conceptual diagram of an example system 100 that includes a multi-modal input method editor (IME) 102. In this example, the IME 102 is implemented in a mobile electronic device 104, though it should be understood that the IME 102 can be implemented in a different electronic device, e.g., a PC, laptop computer, PDA, etc. The electronic device 104 includes multiple user input devices 106, including a microphone 105 to receive spoken user input. Other user input mechanisms include a keyboard, which can include a soft or virtual keyboard (e.g., a touchscreen keyboard 108) or a hard or physical keyboard, a mouse, a trackball, and the like. The user input mechanisms 106 are capable of receiving spoken input (i.e., by the microphone) and written input (i.e., by the keyboard 108).
  • The user input can be received by the electronic device 104 for use as input into one of various applications 110 that can execute on the electronic device 104, e.g., a web browser, an e-mail application, a word processing application, a contacts book, and/or a calendar. In some implementations, the user input is an input into a web form on a particular web page of a particular web site. The IME is generally application-independent, i.e., can be used for most any of the applications 110.
  • If the user input is spoken input, i.e., a speech utterance, the spoken input can be provided to a remote server for conversion to text. For example, the speech utterance 111 can be transmitted over the network 112 to a remote server 114 that includes a speech service 116 and speech recognizer system 118. The network 112 can include one or more local area networks (LANs), a wide area network (WAN), such as the Internet, a wireless network, such as a cellular network, or a combination of all of the above.
  • The speech recognizer system 118 can use one or more language models 120 to recognize text from the speech utterance. The text 113, which can be a selected best candidate or can be a list of n-best candidates that correspond to the speech utterance, is provided back to the electronic device 104 over the network 112. The text 113 can be displayed to the user on a display 122 of the electronic device 104.
  • If the text 113 includes a list of n-best candidates, the user 101 can select a candidate from the list that corresponds to the user's spoken input, for example, using the keyboard 108 or another input mechanism, such as touching the touch screen over one of the candidates, to navigate the list and make a selection.
  • The user can also provide written input, and can provide input using a combination of written and spoken input. For example, the user can begin a search query in a web browser by speaking one or more words and can then add to the query string by typing additional input using the keyboard 108. The IME 102 can provide the combined user input to the relevant application, i.e., the web browser application in this example. In some implementations, the language that the written input is written in can be determined and then provided as a language indicator to the remote server 114. The remote server 114 can use the language indicator when converting the speech utterance 111 to the text 113. For example, by knowing the language in which the speech is spoken, an appropriate language model 120 can be selected for use by the speech recognizer 118.
  • FIG. 1B is a block diagram of an example system 130 that can be used to implement a multi-modal IME. The example system 130 can be implemented, for example, in a computer device, such as a personal computer device, or other electronic devices, such as a mobile phone, mobile communication device, personal digital assistant (PDA), Global Positioning System (GPS) navigation device, and the like.
  • The example system 130 includes a processing device 132, a first data store 134, a second data store 136, input devices 138, output devices 140, and a network interface 142. A bus system 144, including, for example, a data bus and a motherboard, can be used to establish and control data communication between the components 132, 134, 136, 138, 140 and 142. Other system architectures can also be used.
  • The processing device 132 can, for example, include one or more microprocessors. The first data store 134 can, for example, include a random access memory storage device, such as a dynamic random access memory, or other types of computer-readable medium memory devices. The second data store 136 can, for example, include one or more hard drives, a flash memory, and/or a read only memory, or other types of computer-readable medium memory devices.
  • The input devices 138 include at least one input device that is configured to receive spoken input and at least one input device configured to receive written input. Example input devices 138 can include a microphone, keyboard, a mouse, a stylus, etc., and example output devices 140 can include a display device, an audio device, etc. The network interface 142 can, for example, include a wired or wireless network device operable to communicate data to and from a network 146. The network 146 can include one or more local area networks (LANs), a wide area network (WAN), such as the Internet, a wireless network, such as a cellular network, or a combination of all of the above.
  • In some implementations, the system 130 can include input method editor (IME) code 131 from a data store, such as the data store 136. The input method editor code 131 can be defined by instructions that upon execution cause the processing device 132 to carry out input method editing functions. The input method editor code 131 can, for example, include interpreted instructions, such as script instructions, e.g., JavaScript or ECMAScript instructions, that can be executed in a web browser environment. Other implementations can also be used, e.g., a stand-alone application, an applet, a plug-in module, etc., for use in a user interface, such as a display that displays user inputs received by use of keypad mapping for a mobile device or keyboard mapping for a mobile device or personal computer.
  • Execution of the input method editor code 131 generates or launches an input method editor instance (IMEI) 133. The input method editor instance 133 facilitates the processing of one or more input methods at the system 130, during which time the system 130 can receive inputs for characters or symbols, such as, for example, spoken or written input. For example, the user can use one or more of the input devices 138, e.g., a microphone for spoken input or a keyboard for written input. In some implementations, the user input can be Roman characters that represent input in a first writing system, e.g., Pinyin, and the input method editor can convert the input to a second writing system, e.g., Hanzi terms. In some examples, a Hanzi term can be composed of more than one Pinyin input.
  • The first data store 134 and/or the second data store 136 can store an association of inputs. Based on a user input, the input method editor instance 133 can use information in the data store 134 and/or the data store 136 to identify one or more candidate selections represented by the input. In some implementations, if more than one candidate selection is identified, the candidate selections are displayed on an output device 140. For example, if the user input is spoken input, then a list of candidate selections showing written text representations of the spoken input can be presented to the user on the output device 140. In another example, if the user input is Pinyin inputs, the user can select from the candidate selections a Hanzi term, for example, that the user desires to input.
  • In some implementations, a remote computing system 148 having access to the system 130 can be used to convert spoken user input to written user input. For example, the remote system 148 can be a server that provides a speech recognition service via the network 146. One or more speech utterances forming the spoken input can be transmitted to the remote system 148 over the network 146. The remote system 148 can determine a text conversion of the spoken input, for example, using a convenient form of speech recognizer system, and transmit the text conversion to the system 130. The text conversion can be a best candidate for text corresponding to the spoken input or can be a list of n-best candidate selections for presentation to the user for selection as the input. In an example implementation, the speech recognizer system can include Hidden Markov Modeling (HMM) encoded in a finite state transducer (FST). Other configurations of speech recognizer can be used by the remote system 148.
  • In some implementations, the remote system 148 can also be used to edit a logographic script. For example, the remote system 148 may be a server that provides logographic script editing capability via the network 146. In one example, a user can edit a logographic script stored in the data store 134 and/or the data store 136 using a remote computing system, e.g., a client computer. The system 130 can, for example, select a character and receive an input from a user over the network interface 142. The processing device 132 can, for example, identify one or more characters adjacent to the selected character, and identify one or more candidate selections based on the received input and the adjacent characters. The system 130 can transmit a data communication that includes the candidate selections back to the remote computing system.
  • FIG. 1C includes a block diagram of example software that can be used to implement an input method editor in FIG. 1B (e.g., IMEI 133). The system 160 includes a user interface 162 and software 164. A user 166 can access system 160 through the user interface 162. The software 164 includes applications 165, IME engine 166, an operating system (OS) 167, a speech recognition system 169 including a language model 168, and a detection engine 170. The operating system 167 is a particular piece of software that can provide the user interface 162 between the software 164 (e.g., applications 165 and IME engine 166) and the user 166.
  • As shown in FIG. 1C, the speech recognition system 169 and language model 168 are separate from IME engine 166. In particular, the speech recognition system 169 and language model 168 (which can include two or more language models) are included within software 164 as a separate software component. Other implementations are possible. For example, the speech recognition system 169 and language model 168 can be located remotely (e.g., at the remote system 148 of FIG. 1B). As another example, the speech recognition system 169 and language model 168 can be included within the IME engine 166.
  • The language model 168 can define one or more language sub-models, each sub-model tailored to a particular application, or webpage, or webform on a particular webpage, or website, to name a few examples. Each language sub-model can, for example, define a particular rule set, e.g., grammar particular to a language, phrase sets, verbals, etc., that can be used to determine a user's likely intent in entering a set of inputs (e.g., inputs for generating candidates that are translations, transliterations, or other types of phonetic representations). In some implementations, each language sub-model can also include a user history of a particular user, e.g., a dictionary of words and phrased often used by a particular user.
  • The detection engine 170 includes an input module 172 and can include a timing module 174. The input module 172 can, for example, receive input (e.g., keystrokes representing characters or a speech utterance) to particular applications 165 and send the received input to the IME engine 166. In some implementations, the detection engine 170 is a component of the IME engine 166.
  • The detection engine 170 can detect input and determine whether or not to send the input to the IME engine 166. The IME engine 166 can, for example, be implemented using the input method editor code 131 and associated data stores 134 and 136, and provide output candidates in text converted from speech to an interface (e.g., user interface 162) as the input (e.g., speech utterances) is detected, as described with reference to FIGS. 2 and 3A-E below.
  • The components of system 160 can be communicatively coupled to one or more of each other. Though the components identified above are described as being separate or distinct form each other, one or more of the components may be combined in a single system, or to perform a single process or routine. The functional description provided herein including separation of responsibility for distinct functions is by way of example. Other storage architectures can also be used. In particular, other groupings or other divisions of functional responsibilities can be made as necessary or in accordance with design preferences. For example, IME engine 166 can perform the functions of detection engine 170. As another example, input module 172 and timing module 174 can be combined into a single module.
  • FIG. 2 is a flowchart of an example process 200 for using an input method editor to receive spoken input from a user input device and to provide written, or textual, input to a corresponding application. A request is received from a user for an application-independent input method editor that has written and spoken input capabilities (Step 202). By way of illustrative example, and without limitation, the request can be received by a mobile electronic device that has a touchscreen keyboard. Example screenshots from such a mobile electronic device are shown in FIGS. 3A-E. These screenshots can be used to illustrate the example process 200; however, it should be understood that other devices can implement the process 200, and the screenshots shown are not intended to be limiting.
  • FIG. 3A shows a screenshot 300 where a user has selected to activate a web browser application. Through the web browser application, the user has selected to navigate to the Google search page at the URL www.google.com 302. FIG. 3B shows a screen shot 304 with a soft touchscreen keyboard 306 displayed in a lower portion of the display screen. For example, the user can touch or tap the screen in the search query field 308 to automatically have the keyboard 306 displayed, although other mechanisms can be used to trigger the display of the keyboard 306. The example keyboard 306 shown includes a microphone key 310. An example of a request that can be received from the user includes the user selecting the microphone key 310. Another example includes the user selecting a graphical entity, such as a microphone icon or button, displayed next to or in an input field, e.g., in search query field 308. Another example includes the user swiping his/her finger across the input field, e.g., in a left to right motion, or tapping the input field. Yet another example includes the user picking up the device in a manner that is consistent with raising a microphone included in the device to the proximity of the user's mouth, which can be detected, for example, by an accelerometer reading. Other forms of request can be received from the user for an application-independent input method editor having written and spoken input capabilities, and the above are but some examples.
  • A user's intention to provide spoken input to the application-independent input method editor is then identified in the process (Step 204). For example, receiving a speech utterance from the user can be used to identify that the user intends to provide spoken input. In other implementations, receiving the request from the user for the input method editor with written and spoken input capabilities can also be used to identify that the user intends to provide spoken input, i.e., the same user action can provide both the request and be used to identify the user's intention. In some implementations, as shown in the screenshot 312 in FIG. 3C, a graphical element can be displayed that prompts the user to speak, such as the microphone graphic 314 and the instructions “Speak now” 316.
  • A spoken input, i.e., a speech utterance, is then received from the user. The user provides the spoken input as input to an application that is executing on the device (Step 206). The spoken input is provided to a remote server that includes a speech recognition system configured to recognize text based on the spoken input (Step 208). For example, referring again to FIG. 1B, the spoken input can be sent over the network 146 to the remote system 148, where the remote system 148 includes a speech recognition system to recognize text from a speech utterance. Because processing the speech to text conversion can take some time, in some implementations a graphic is displayed to the user to indicate that the process is in progress, such as the “Working” graphic 320 shown in the screenshot 322 in FIG. 3D.
  • Text is then received from the remote server, where the text represents the spoken input (Step 210). Once the remote server, e.g., remote system 148, has processed the speech utterance, the corresponding text is sent back to the user's device and can be displayed for the user. In some implementations, the best candidate for representation of the speech utterance is selected by the speech recognition system at the remote server and provided to the device. However, in some implementations, an n-best list of candidates can be provided and presented to the user for selection of the correct candidate. For example, referring to FIG. 3E, a screen shot 324 shows a list of suggestions, with the best candidate “the man in the moon” displayed at the top of the list as the default selection.
  • The text, i.e., the spoken input converted to written input, is then provided to the application as user input (Step 212). That is, once the correct text conversion is selected, if a list of candidates was provided, or once the best candidate has been received, if only one was sent from the remote server, the written input can be passed to the application as the user input for processing by the application.
  • In some implementations, a context indicator can be sent with the spoken input to the remote system for conversion to text. The remote system can use the context indicator to facilitate the speech-to-text conversion. For example, the context indicator can be used as a basis for selecting an appropriate language model to use by the speech recognition system. The context indicator can specify the context in which the spoken user input was received. For example, the context indicator can specify a name of a field, e.g., in a web form, the name of the application in which the input was received, and/or identify a web page if the user input was received in a web browser application. As another example, the context indicator can include metadata relating to a field in which the user input was received. For example, the metadata can specify that the field requires a one-word answer, or a date, or a name, and the like. In some implementations, the context indicator information can be obtained by the input method editor from the operating system of the electronic device.
  • The device may pass a sound file (including streaming sound data) of the spoken input to a remote server system, and may take steps to improve the quality of the speech-to-text conversion. As one example, the device may pass information that allows the server system to select a language model that is relatively small in size and is specific to the task that the user is currently facing. For example, when applications register with the IME, they may provide information about fields into which a user can enter information in the applications. The IME can pass such information to the server system, so that the server system may select an appropriate language model. For example, if the cursor is in an “address” field of an application, the IME can pass such information to the server system so that, for example, a user utterance that sounds like “scheet” is interpreted as “street” and not “sweet.”
  • The language model that is applied may also be a composite of, or interpolation of, multiple separate language models. The different models may be relatively small models that have been derived from large data corpuses (e.g., SMS messages and e-mail messages). The models may be prepared off-line by analyzing such corpuses, and the mixture weights that are applied to the models may be generated on-the-fly at run-time, including after data from a particular instance of speech input starts being received from a user. The weightings may be a function, for example, of the field into which a user is currently making an utterance (e.g., the “to” or “from” fields of an email message versus the “body” field of an email message).
  • The language model may also be chosen more generally, without reference to a particular input field in which the cursor is located. For example, an application developer may register their application with an app store, and may indicate the type of application that it is, such as a music player. Similarly, a server system may have a number of topic-specific language models that it stores. Thus, if a user of a music application speaks the word “Heart” or “Hart” (which would not be plain from the spoken word itself), the IME may pass a “music” indication to the server system, so that the application is passed the word “heart,” and the user sees a song list for the female-led rock band. If the media player is a video player, the IME may pass the word “Hart” (assuming there are no great movies or television shows with the word “heart” in their titles) so that the user is shown an index of the episodes of the iconic detective drama “Hart to Hart.”
  • When the user is interacting with a web page, such as in a web browser, different context information can be used for selecting the proper language model. In particular, the domain or a sub-domain for the page may be provided, so that the that the language model will be specific to the particular type of web site. For example, if the web site is Amazon, then the language model may be one in which “shopping” terms have higher prominence. For example, “product” may have a higher score than “protect” for similar sounds. Such a model may be prepared to be directed to the site itself (e.g., by analyzing input forms on the site, and analyzing text on the site), or on a category that the site matches. Thus, for example, the same language model may be used for the sites Amazon.com, Buy.com, and the like.
  • FIG. 4 is a block diagram of an example system 400 for receiving speech input and training language models to interpret the speech input. In the system 400, users of client devices enter data into text input fields, and a speech server analyzes that data to determine the type of text the users entered. The speech server builds language models from this information, and uses the language models to recognize speech input associated with similar input fields.
  • Clients 402, such as computers executing a web browser with an optional toolbar, can receive text input for web forms. When this input is used as part of a process to browse to another page, such as typing in a search field or order form field before pressing a submit button, the text and destination information can be saved in a toolbar log 404. Clients 406, such as computers and mobile computing devices, can receive text based web search queries from users. These queries can be resolved by a web search server (not shown) and the queries and search results can be stored in query logs 408. The client 406 can receive text input to applications, such as an email client, a messaging client, and/or a word processor and spoken input to applications. The text input and transcriptions of spoken input can be stored in input logs 409.
  • A speech recognition server 410 can retrieve the data in the toolbar logs 404, the query logs 408, and/or the input logs 409. A speech recognition system 412 can group this data into categories or classifications. The speech recognition system 412 creates a series of language models 414 a-414 n. The language models can contain words, phrases, sentences, etc. from the toolbar logs 404, query logs 408, and/or the input logs 409 based on a particular topic. For example, if the language model A 414 a is a language model of British last name, it can contain “Churchill,” “Bean,” and “Pigou.” In another example, if the language model B 414 b contains technology-related language, it can contain “object oriented programming,” “reverse polish notation,” and “garbage in, garbage out.”
  • In some implementations, the speech recognition system 412 can build interpolated language models from the language models 414 a-414 n. For example, the speech recognition system 412 can use an utterance with a known text result, and poll the language models 414 a-414 n to determine the confidence level that each language model 414 a-414 n would associate with the utterance/known text pair. The confidence level returned by each language model 414 a-414 n can be used by the speech recognition system 412 to create an interpolated language model. For example, using a test utterance and text of “Mr. Bean writes in reverse polish notation,” the language model A 414 a gives a confidence level of 50%, the language model B 414 b give a confidence level of 65%, and the other language models 414 c-414 n give substantially 0% confidence levels. In this example, the speech recognition system 412 can create an interpolated language model that heavily weighs the language model A and B 414 a-b. The particular weightings may be equal to or proportional to the confidence levels, in certain implementations.
  • A client device 416 can execute an application that accepts text input and can receive an utterance from a user for that text input. The client device 416 can transmit the utterance, along with the context in which the utterance was made or received, to the speech recognition server 410 for translation into text. In some implementations, the context in which an utterance is received is determined by the client device 416 with the field name or label associated with the text input. For example, some applications include metadata for an application input field such as package name, field number or name, and/or attribute flags (e.g. long sentence, email, street address, etc.). In some applications, a text label associated with or displayed near an input box is used to determine the context.
  • A speech service 418 can receive the utterance and context. In some implementations, the speech service 418 can map the context to broader categories or to the categories of the language models 414 a-414 n. In some implementations, the speech service 418 can maintain a cache of utterances and resultant texts, and if a received utterance substantially matches a cached utterance, the speech service 418 can return the appropriate cached resultant texts.
  • The speech recognition system 412 uses or creates an interpolated language model to recognize the text in the utterance. In some implementations, a single candidate text is identified. In some implementations, the top n candidates are identified either wherein all candidates meet a certain confidence threshold, or wherein the top n candidates are selected.
  • The text or texts identified by the speech recognition system 412 is returned to the client 416, where it is, for example, displayed in a text input field that has focus.
  • Such context-specific language models may be generated in a variety of ways, including by using search information from a public search engine. For example, bi-grams may be made between search queries and web sites that are highly relevant to those queries (e.g., are in the list of top 10 sites returned to the queries). The queries and sites may then be assigned as nodes in a graph, joined by the relationship between the two. Clusters of such relationships may then be identified, and language models may be generated for those clusters. A particular language model for a web site may then be generated as a weighted combination of the language models for the clusters around the site in the graph.
  • FIG. 5A is a block diagram 500 of queries and associated websites in search results. A group of queries 502 is a collection of example queries that can be collected from a browser executing a toolbar. The queries can be submitted to a web search engine, and a list of search results can be returned. For example, a search for “mike lebeau” can result in search results including facebook.com and linkedin.com. A search for “blink” can result in search results including amazon.com.
  • The queries and associated websites can be used, for example, in determining a relationship between queries and related topics or categories. In this example, if it is known that facebook.com and linkedin.com are social network websites, it can be determined that the queries “bill byrne,” “will rusch,” “mike lebeau,” and “brian stope” may be the names of people. Similarly, if it is known that amazon.com is a retailer with a reputation as a book retailer, it can be determined that “blink” and “great gatsby” may be the names or titles of a retail products, perhaps books. Similarly, if a search for “william gibson” returns links to both facebook.com and amazon.com, it can be determined that “william gibson” may be an author.
  • FIG. 5B is a URL showing search terms in a query. The URL in this example can be recorded by a web browser executing a toolbar. When a user submits a form via a GET request, the contents of the form can be encoded and logged in the URL parameters. In this example, a search on the webpage facebook.com can generate the URL shown. The name parameter 550, in this example “bill%20byrne”, indicates that “bill byrne” was entered into a field in a web form.
  • FIG. 6 is a block diagram of an example system 600 for recognizing speech input to a computing device. In the system 600, a user enters speech input to an application via an IME. The speech input is analyzed at a speech recognition server and text is returned to the application.
  • An application 604 executes on a computing device 602, for example a mobile computing device. An input method manager 608 can manage input methods that the application 604 accepts, including speech input from a user. A speech IME 610 in the operating system of the computing device 602 records speech input and collects context data from an input context 612.
  • Speech IME 160 can transmit the speech input and context to a voice data center 614. A speech service front end 616 can receive the speech input and context and prepare the speech input and context for analysis. For example, statistics can be collected, recognition jobs can be created and sorted, etc. A speech recognition system 618 examines the context, and selects an associated language model from a big table of language models 622. The associated language model in the big table of language models 622 can contain a reference to a language model in a ProdLM 620. The speech recognition system 618 uses the referenced language model in the ProdLM 620 to analyze the speech input and determine a text string. The text string is returned to the speech input method 610, for example by way of the speech service front end 616. The text string can be presented to the user of the computing device 602, and if approved by the user, can be sent to the application 604 as text input.
  • In some implementations, data obtained from web toolbar logs can be used in determining an interpolated language model. A sample of content of user input (e.g., queries) into a web form can be obtained from web toolbar logs, e.g., in URLs. That is, when a user submits a web form via a GET request, the content of the user input into fields of the web form can be encoded and logged in the URL parameters (e.g., as was described above in reference to FIG. 5B). A toolbar log can therefore provide sample data of what users have input (i.e., typed) into each of the fields of a number of web forms. In some instances, however, the volume of this data may be too sparse to train a good language model for the web form. However, for a particular web form a larger training set can be available if queries of the same category as the queries input into the particular web form are used. That is, queries that correspond to search results having similar websites in the top N search results (where N is an integer, e.g., 10) as each other are probably generally within the same category. To group queries by category, clusters in a search query—search result (website) graph (where queries and websites are assigned to nodes) can be identified, where an edge between a particular query node and a particular website node occurs whenever the website appears in the top N results for the particular query. A language model can then be trained per cluster, that is, language models can be trained for a category (if a cluster is representative of a category of queries).
  • To generate an interpolated language model for a particular web form (or web site), the interpolated language model can be built using multiple language models formed from a number of clusters. In some implementations, the multiple language models can be selected as follows. A sample of user input into the particular web form (i.e., sample queries) can be obtained, e.g., from a web toolbar log. For example, referring again to FIG. 5B, the sample data may be the query “bill byrne” that is input into a web form on the www.facebook.com website, which is also reflected in the graph shown in FIG. 5A. In this example, “bill byrne” belongs to a cluster that includes people-oriented websites (e.g., facebook.com and linkedin.com) and people names (Bill Byrne, Will Rusch, Mike Lebeau, Brian Strope and William Gibson). In this simple example, the “bill byrne” query belongs to only the people names cluster. However, a query may belong to many clusters. In such instances, for the sample queries obtained from the web toolbar data, the K most significant clusters (K is a number/integer) can be identified and weighted, with the sum of their weights adding to 1.0. In some implementations, the K most significant clusters can be determined by counting the number of sample queries (from the web toolbar log sample data) that fall into each cluster and then optimizing the weights of only the top K clusters. Other techniques to determine the K most significant clusters can be used. The interpolated language model for the web form can be used to convert a speech input received for a field in the form into text. The converted text can be transmitted to an electronic device that initially received the speech input, or transmitted elsewhere, to be used as text input for the field in the web form.
  • FIG. 7A shows a flowchart of an example process 700 of building an interpolated language model for a website (or a web form of a website). In the process 700, pairs of queries and results are collected, base language models are created, and interpolated language models are created.
  • In step 702, pairs containing web queries and result sites are extracted from web search logs. For example, a web browser or web browser add on can report queries, query result URLs, or search results to a central repository. In another example, a web search server can create a log of search terms and website domain names that were returned in response to those search terms.
  • In step 704, a cluster bipartite graph, also known as a bigraph, is formed by the pairs. A bipartite graph is a graph whose vertices can be divided into two disjoint sets such that every edge connects a vertex in the first set to one in the second set. In some implementations, the first set can contain nodes representing queries and the second set can contain nodes representing results. Clusters are identified in the bipartite graph, for example so that each cluster consists of queries and results that may by semantically or syntactically similar.
  • In step 706, language models are trained based on the queries in the clusters. In some implementations, sample utterances are recorded by a range of speakers and associated with the queries. The utterances for each query can be aggregated, such as by determining an average waveform and a deviation factor indicating how much deviation from the mean was found for most utterances.
  • In step 708, web form sample data is obtained from toolbar logs, the web form sample data relating to the particular web form (or website) for which the interpolated language model is being generated (e.g., the interpolated language model can be for the web form). In some implementations, a web browser toolbar can record and transmit the URLs of websites visited by the browser. The URLs can include input text (i.e., sample queries) that has been entered into the web form and submitted. This information can be used to associate a particular website domain, or web page with a particular topic or idea.
  • In step 710, the K most significant clusters for the input text, i.e., the sample queries, obtained from the web form sample data are identified. A list of cluster names is collected and ranked in order of significance or similarity to the data collected in the step 708. A predetermined number (K, in this case) of the most significant clusters are identified. For example, clusters are identified that include the sample queries. To limit the number of clusters to K, i.e., to identify the K most significant, in some implementations, the number of sample queries that fall into each cluster can be counted and the weights assigned to each cluster optimized for only the top K clusters.
  • In step 712, an interpolated language model for the web form (or website) is built. In one example, a data structure is created that contains links to the K most significant language models. This data structure represents a combined language model that delegates speech recognition functionality to the referenced language models.
  • In step 714, K interpolation weights are optimized. Each referenced language model can have a weight, such as a percentage, integer in a particular range, or probability, associated with the referenced language model in the interpolated language model. In some implementations, this weight can be determined by testing a known pairs of utterances and text. The pairs can be submitted to the referenced language models, which can return a confidence level representing the likelihood or accuracy of the language model to correctly identify the text in the utterance. The confidence levels, optionally normalized or otherwise converted, can be used as weighting values in the interpolated language model. Thus, reference language models likely to correctly identify text in a particular classification are most heavily weighted.
  • Although a particular number, type, and order of steps are shown, it will be understood by one skilled in the art that other number, types, and orders are possible. For example, other methods of clustering or graph creation can be used in the step 704. In another example, input field flags and descriptions are obtained in the step 708. In another example, step 702 and step 714 can be combined using a process that builds an interpolation model and interpolation weights in one step.
  • FIG. 7B shows a flowchart of an example process 750 of building an interpolated language model. In the process 750, text input and associated metadata is collected, base language models are created, and interpolated language models are created using the base models and interpolation factors such as weightings to be provided to each of multiple base models.
  • In step 752, text input and input metadata pairs are extracted from input logs. For example, an application can report to a central repository input entered into an input field and metadata for the input field. In some examples, the text input can include voice input that is transcribed into a text format. In some examples, application wide or operating system wide metadata schemes can define or describe the type of input field, such as an address field, a free form text field, a search field, or a social status field.
  • In step 754, a cluster bipartite graph, also known as a bigraph, is formed by the pairs. A bipartite graph is a graph whose vertices can be divided into two disjoint sets such that every edge connects a vertex in the first set to one in the second set. In some implementations, the first set can contain nodes representing input metadata and the second set can contain nodes representing text input. Clusters are identified in the bipartite graph, for example so that each cluster consists of similar input metadata.
  • In step 756, language models are trained based on the text input in the clusters. In some implementations, sample utterances are recorded by a range of speakers and associated with the metadata. The utterances for each text input can be aggregated, such as by determining an average waveform and a deviation factor indicating how much deviation from the mean was found for most utterances.
  • In step 758, input metadata is categorized. The input metadata can be used to associate a particular input field or application with a particular topic or idea. Example topics or ideas include, but are not limited to, address fields, free form text fields, search field, social status fields, and numeric fields.
  • In step 760, the K most significant clusters for a category are identified. A list of cluster names is collected and ranked in order of significance or similarity to the data collected in the step 758. A predetermined number (K, in this case) of the most significant clusters are then identified.
  • In step 762, an interpolated language model is built. In one example, a data structure is created that contains links to the K most significant language models. This data structure represents a combined language model that delegates speech recognition functionality to the referenced language models.
  • In step 764, K interpolation weights are optimized. Each referenced language model that is referenced in the data structure can have a weight, such as a percentage, integer in a particular range, or probability, associated with it. In some implementations, this weight can be determined by testing a known pair or pairs of utterances and text. The pairs can be submitted to the referenced language models, which can return a confidence level representing the likelihood or accuracy of the language model to correctly identify the text in the utterance. The confidence levels, optionally normalized or otherwise converted, can be used as weighting values in the interpolated language model. Thus, reference language models that are likely to identify text in a particular classification correctly are most heavily weighted.
  • Although a particular number, type, and order of steps are shown for the process in this figure, it will be understood by one skilled in the art that other number, types, and orders are possible. For example, in step 762 and step 764, a data structure with weighted links to the K most significant language models can be selected from a collection of some or all possible data structures with weighted links to language models.
  • FIG. 8 shows a flowchart of an example process 800 for recognizing text in an utterance. In the process 800, an utterance is received, the speech in the utterance is recognized, and text, or a list of text options, is returned.
  • In step 802, an utterance is received. For example, a speech recognition server can receive translation requests from clients. The translation request can include an utterance (e.g. an audio file) and a context (e.g. text or other data describing how the utterance may be used or categorized).
  • In step 804, a relevant language model is determined. For example, an interpolated language model for web search, free form text input, or social status can be determined. In some implementations, language models, including interpolated language models, can be indexed by keyword, web domain, application type, or other criteria. Using metadata associated with the utterance, such as context or source information, a relevant language model can be determined.
  • In step 806, speech recognition is performed. The utterance can used by a speech recognition application using the language model. The speech recognition application can calculate one or more candidate text strings from the utterance. A confidence level can be associated with each text string.
  • In step 808, a K-best list of text strings representing the utterance is returned (where K is an integer). In some implementations, the text string with the highest confidence value is selected and returned. In some implementations, a particular number (K, in this case) of text strings with the highest confidence level are returned.
  • Although a particular number, type, and order of steps are shown, it will be understood by one skilled in the art that other number, types, and orders are possible. For example, in step 808, all text strings with a confidence level above a particular threshold can be returned. In another example, an utterance can be preprocessed before step 806 to improve recognition.
  • FIG. 9 shows an example of a generic computer device 900 and a generic mobile computer device 950, which may be used with the techniques described here. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.
  • Computing device 900 includes a processor 902, memory 904, a storage device 906, a high-speed interface 908 connecting to memory 904 and high-speed expansion ports 910, and a low speed interface 912 connecting to low speed bus 914 and storage device 906. Each of the components 902, 904, 906, 908, 910, and 912, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 902 may process instructions for execution within the computing device 900, including instructions stored in the memory 904 or on the storage device 906 to display graphical information for a GUI on an external input/output device, such as display 916 coupled to high speed interface 908. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 900 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).
  • The memory 904 stores information within the computing device 900. In one implementation, the memory 904 is a volatile memory unit or units. In another implementation, the memory 904 is a non-volatile memory unit or units. The memory 904 may also be another form of computer-readable medium, such as a magnetic or optical disk.
  • The storage device 906 is capable of providing mass storage for the computing device 900. In one implementation, the storage device 906 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 904, the storage device 906, memory on processor 902, or a propagated signal.
  • The high speed controller 908 manages bandwidth-intensive operations for the computing device 900, while the low speed controller 912 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller 908 is coupled to memory 904, display 916 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 910, which may accept various expansion cards (not shown). In the implementation, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
  • The computing device 900 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 920, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 924. In addition, it may be implemented in a personal computer such as a laptop computer 922. Alternatively, components from computing device 900 may be combined with other components in a mobile device (not shown), such as device 950. Each of such devices may contain one or more of computing device 900, 950, and an entire system may be made up of multiple computing devices 900, 950 communicating with each other.
  • Computing device 950 includes a processor 952, memory 964, an input/output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The device 950 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 950, 952, 964, 954, 966, and 968, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.
  • The processor 952 may execute instructions within the computing device 950, including instructions stored in the memory 964. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device 950, such as control of user interfaces, applications run by device 950, and wireless communication by device 950.
  • Processor 952 may communicate with a user through control interface 958 and display interface 956 coupled to a display 954. The display 954 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 956 may comprise appropriate circuitry for driving the display 954 to present graphical and other information to a user. The control interface 958 may receive commands from a user and convert them for submission to the processor 952. In addition, an external interface 962 may be provide in communication with processor 952, so as to enable near area communication of device 950 with other devices. External interface 962 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.
  • The memory 964 stores information within the computing device 950. The memory 964 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 974 may also be provided and connected to device 950 through expansion interface 972, which may include, for instance, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 974 may provide extra storage space for device 950, or may also store applications or other information for device 950. Specifically, expansion memory 974 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 974 may be provide as a security module for device 950, and may be programmed with instructions that permit secure use of device 950. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.
  • The memory may include, for example, flash memory and/or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 964, expansion memory 974, memory on processor 952, or a propagated signal that may be received, for example, over transceiver 968 or external interface 962.
  • Device 950 may communicate wirelessly through communication interface 966, which may include digital signal processing circuitry where necessary. Communication interface 966 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 968. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 970 may provide additional navigation- and location-related wireless data to device 950, which may be used as appropriate by applications running on device 950.
  • Device 950 may also communicate audibly using audio codec 960, which may receive spoken information from a user and convert it to usable digital information. Audio codec 960 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 950. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 950.
  • The computing device 950 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 980. It may also be implemented as part of a smartphone 982, personal digital assistant, or other similar mobile device.
  • Device 950 may also include one or more different devices that are capable of sensing motion. Examples include, but are not limited to, accelerometers and compasses. Accelerometers and compasses, or other devices that are capable of detecting motion or position are available from any number of vendors and may sense motion in a variety of ways. For example, accelerometers may detect changes in acceleration while compasses may detect changes in orientation respective to the magnetic North or South Pole. These changes in motion may be detected by the device 950 and used to update the display of the respective devices 950 according to processes and techniques described herein.
  • Various implementations of the systems and techniques described here may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.
  • To provide for interaction with a user, the systems and techniques described here may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
  • The systems and techniques described here may be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user may interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.
  • The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • FIG. 10 is a block diagram of an example data structure 1000 of a language model. The data structure 1000 can weightedly link a semantic idea or category with language models, for example, for use in speech recognition.
  • A semantic category 1002, such as a search query or type of input field, can be linked to one or more of a collection of interpolated language models 1004. The interpolated language models 1004 can be linked to one or more base language models 1006. The links between the interpolated language models 1004 and base language models 1006 can be weighted. In some examples, the sum of the weights of the links from one interpolated language model can be 1 or can be normalized to 1.
  • It will be understood that, although a particular number and configuration of interpolated language models 1004, base language models 1006, and links are shown, other numbers and conffigurations are possible. For example, suffieicent interpolated language models 1004 may exist that every weighted combination of base language models 1006 has a linked interpolated language model 1004. In some examples, interpolated language model can be linked to more or fewer base language models 1006. In some examples, different link weights between interpolated language models 1004 and base language models 1006 may exist, such as positive integers, probabilities, or dimensional distance (e.g. W,X,Y,Z values for four dimensional space.) In some implementations, multiple semantic categories 1002 can be linked to a single interpolated language model 1004.
  • In some examples, interpolated language models 1004 can be created on demand, such as when a semantic category 1002 is added to the data structure. The interpolated language models 1004 can persists after the removal of semantic categories (not shown). For example, previously removed semantic categories (not shown) may have prompted the creation of the interpolated language models 1004 a, 1004 b, and 1004 c. A new semantic category 1008 can be added to the data structure 1004, and linked to any of the interpolated language models 1004.
  • In some examples, every possible interpolated language model 1004 can be pre-created for use by a new semantic category 1008. Some of these examples can be used in situations, such as when many new semantic categories 1008 are expected, when pre-processing time is available, and/or when few base language models 1006 are expected.
  • A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of what is described here.
  • In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.

Claims (15)

1. A computer-implemented method of forming a speech recognition language model, the method comprising:
determining, with a computer system, a plurality of query-website relationships by identifying websites that are determined to be relevant to queries using one or more search engines;
identifying clusters in the plurality of query-website relationships by connecting common queries and connecting common websites; and
creating a speech recognition language model for a particular website based on at least one of analyzing at queries in a cluster that includes the website or analyzing webpage content of web pages in the cluster that includes the website.
2. The method of claim 1, wherein determining the plurality of query-website relationships comprises analyzing search results returned by the one or more search engines to the queries.
3. The method of claim 1, wherein identifying clusters in the plurality of query-website relationships comprises analyzing a graph comprised of queries and websites at nodes of the graph, that are connected by edges defined by particular websites that are determined to be relevant to particular queries.
4. The method of claim 1, wherein a website is determined to be relevant to a query if the website is a top n ranked search result for the query in a set of ranked search results relevant to the query, wherein n is a predetermined integer.
5. The method of claim 1, wherein creating the speech recognition model for the particular website comprises creating cluster-based speech recognition models for each of a plurality of the clusters, and creating the speech recognition model by interpolating among cluster-based speech recognition models for clusters that surround the particular website.
6. A system for forming speech recognition language models comprising:
a data processing apparatus; and
storage coupled to the data processing apparatus storing code that when executed by the data processing apparatus causes the data processing apparatus to perform operations comprising:
determining a plurality of query-website relationships by identifying websites that are determined to be relevant to queries using one or more search engines;
identifying clusters in the plurality of query-website relationships by connecting common queries and common websites; and
creating a speech recognition language model for a particular website by at least one of analyzing queries in a cluster that holds the website or analyzing webpage content of web pages in the cluster that includes the website.
7. The system of claim 6, wherein determining the plurality of query-website relationships comprises analyzing search results returned by the one or more search engines to the queries.
8. The system of claim 6, wherein identifying clusters in the plurality of query-website relationships comprises analyzing a graph comprised of queries and websites at nodes of the graph, that are connected by edges defined by particular websites that are determined to be relevant to particular queries.
9. The system of claim 6, wherein a website is determined to be relevant to a query if the website is a top n ranked search result for the query in a set of ranked search results relevant to the query, wherein n is a predetermined integer.
10. The system of claim 6, wherein creating the speech recognition model for the particular website comprises creating cluster-based speech recognition models for each of a plurality of the clusters, and creating the speech recognition model by interpolating among cluster-based speech recognition models for clusters that surround the particular website.
11. A computer-readable storage device encoded with a computer program product, the computer program product including instructions for forming a speech recognition language model that, when executed, cause data processing apparatus to perform operations comprising:
determining a plurality of query-website relationships by identifying websites that are determined to be relevant to queries using one or more search engines;
identifying clusters in the plurality of query-website relationships by connecting common queries and connecting common websites; and
creating a speech recognition language model for a particular website based on at least one of analyzing queries in a cluster that includes the website or analyzing webpage content of web pages in the cluster that includes the website.
12. The computer-readable storage device of claim 11, wherein determining the plurality of query-website relationships comprises analyzing search results returned by the one or more search engines to the queries.
13. The computer-readable storage device of claim 11, wherein identifying clusters in the plurality of query-website relationships comprises analyzing a graph comprised of queries and websites at nodes of the graph, that are connected by edges defined by particular websites that are determined to be relevant to particular queries.
14. The computer-readable storage device of claim 11, wherein a website is determined to be relevant to a query if the website is a top n ranked search result for the query in a set of ranked search results relevant to the query, wherein n is a predetermined integer.
15. The computer-readable storage device of claim 11, wherein creating the speech recognition model for the particular website comprises creating cluster-based speech recognition models for each of a plurality of the clusters, and creating the speech recognition model by interpolating among cluster-based speech recognition models for clusters that surround the particular website.
US12/977,017 2009-12-23 2010-12-22 Speech Recognition Language Models Abandoned US20110161081A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US12/977,017 US20110161081A1 (en) 2009-12-23 2010-12-22 Speech Recognition Language Models
US13/249,180 US20120022873A1 (en) 2009-12-23 2011-09-29 Speech Recognition Language Models

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US28996809P 2009-12-23 2009-12-23
US33021910P 2010-04-30 2010-04-30
US12/977,017 US20110161081A1 (en) 2009-12-23 2010-12-22 Speech Recognition Language Models

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13/249,180 Continuation US20120022873A1 (en) 2009-12-23 2011-09-29 Speech Recognition Language Models

Publications (1)

Publication Number Publication Date
US20110161081A1 true US20110161081A1 (en) 2011-06-30

Family

ID=43858442

Family Applications (12)

Application Number Title Priority Date Filing Date
US12/976,920 Active 2031-06-07 US9495127B2 (en) 2009-12-23 2010-12-22 Language model selection for speech-to-text conversion
US12/976,972 Abandoned US20110161080A1 (en) 2009-12-23 2010-12-22 Speech to Text Conversion
US12/977,017 Abandoned US20110161081A1 (en) 2009-12-23 2010-12-22 Speech Recognition Language Models
US12/977,003 Active 2033-12-08 US9031830B2 (en) 2009-12-23 2010-12-22 Multi-modal input on an electronic device
US13/249,180 Abandoned US20120022873A1 (en) 2009-12-23 2011-09-29 Speech Recognition Language Models
US13/249,175 Abandoned US20120022866A1 (en) 2009-12-23 2011-09-29 Language Model Selection for Speech-to-Text Conversion
US13/249,181 Active US9047870B2 (en) 2009-12-23 2011-09-29 Context based language model selection
US13/249,172 Active 2031-06-30 US8751217B2 (en) 2009-12-23 2011-09-29 Multi-modal input on an electronic device
US14/299,837 Active US9251791B2 (en) 2009-12-23 2014-06-09 Multi-modal input on an electronic device
US14/988,408 Active US10157040B2 (en) 2009-12-23 2016-01-05 Multi-modal input on an electronic device
US16/169,279 Active 2031-02-19 US10713010B2 (en) 2009-12-23 2018-10-24 Multi-modal input on an electronic device
US17/812,320 Active US11914925B2 (en) 2009-12-23 2022-07-13 Multi-modal input on an electronic device

Family Applications Before (2)

Application Number Title Priority Date Filing Date
US12/976,920 Active 2031-06-07 US9495127B2 (en) 2009-12-23 2010-12-22 Language model selection for speech-to-text conversion
US12/976,972 Abandoned US20110161080A1 (en) 2009-12-23 2010-12-22 Speech to Text Conversion

Family Applications After (9)

Application Number Title Priority Date Filing Date
US12/977,003 Active 2033-12-08 US9031830B2 (en) 2009-12-23 2010-12-22 Multi-modal input on an electronic device
US13/249,180 Abandoned US20120022873A1 (en) 2009-12-23 2011-09-29 Speech Recognition Language Models
US13/249,175 Abandoned US20120022866A1 (en) 2009-12-23 2011-09-29 Language Model Selection for Speech-to-Text Conversion
US13/249,181 Active US9047870B2 (en) 2009-12-23 2011-09-29 Context based language model selection
US13/249,172 Active 2031-06-30 US8751217B2 (en) 2009-12-23 2011-09-29 Multi-modal input on an electronic device
US14/299,837 Active US9251791B2 (en) 2009-12-23 2014-06-09 Multi-modal input on an electronic device
US14/988,408 Active US10157040B2 (en) 2009-12-23 2016-01-05 Multi-modal input on an electronic device
US16/169,279 Active 2031-02-19 US10713010B2 (en) 2009-12-23 2018-10-24 Multi-modal input on an electronic device
US17/812,320 Active US11914925B2 (en) 2009-12-23 2022-07-13 Multi-modal input on an electronic device

Country Status (2)

Country Link
US (12) US9495127B2 (en)
EP (3) EP4318463A3 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110153324A1 (en) * 2009-12-23 2011-06-23 Google Inc. Language Model Selection for Speech-to-Text Conversion
US20120059810A1 (en) * 2010-09-08 2012-03-08 Nuance Communications, Inc. Method and apparatus for processing spoken search queries
US20130054238A1 (en) * 2011-08-29 2013-02-28 Microsoft Corporation Using Multiple Modality Input to Feedback Context for Natural Language Understanding
US20130173265A1 (en) * 2012-01-03 2013-07-04 Chiaka Chukwuma Okoroh Speech-to-online-text system
US20150121290A1 (en) * 2012-06-29 2015-04-30 Microsoft Corporation Semantic Lexicon-Based Input Method Editor
US9063636B2 (en) 2013-06-10 2015-06-23 International Business Machines Corporation Management of input methods
US9502032B2 (en) 2014-10-08 2016-11-22 Google Inc. Dynamically biasing language models
US9747895B1 (en) 2012-07-10 2017-08-29 Google Inc. Building language models for a user in a social network from linguistic information
US9978367B2 (en) 2016-03-16 2018-05-22 Google Llc Determining dialog states for language models
US10311860B2 (en) 2017-02-14 2019-06-04 Google Llc Language model biasing system
US20200404362A1 (en) * 2018-02-20 2020-12-24 Lg Electronics Inc. Display device
US11416214B2 (en) 2009-12-23 2022-08-16 Google Llc Multi-modal input on an electronic device

Families Citing this family (362)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7516190B2 (en) 2000-02-04 2009-04-07 Parus Holdings, Inc. Personal voice-based information retrieval system
US8181205B2 (en) 2002-09-24 2012-05-15 Russ Samuel H PVR channel and PVR IPG information
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US10002189B2 (en) 2007-12-20 2018-06-19 Apple Inc. Method and apparatus for searching using an active ontology
US9330720B2 (en) 2008-01-03 2016-05-03 Apple Inc. Methods and apparatus for altering audio output signals
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US20130275899A1 (en) * 2010-01-18 2013-10-17 Apple Inc. Application Gateway for Providing Different User Interfaces for Limited Distraction and Non-Limited Distraction Contexts
US20100030549A1 (en) 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
US8676904B2 (en) 2008-10-02 2014-03-18 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
USD625733S1 (en) 2009-03-04 2010-10-19 Apple Inc. Graphical user interface for a display screen or portion thereof
US20120311585A1 (en) 2011-06-03 2012-12-06 Apple Inc. Organizing task items that represent tasks to perform
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
JP5310389B2 (en) * 2009-08-27 2013-10-09 ソニー株式会社 Information processing apparatus, information processing method, and program
US10276170B2 (en) 2010-01-18 2019-04-30 Apple Inc. Intelligent automated assistant
US8626511B2 (en) * 2010-01-22 2014-01-07 Google Inc. Multi-dimensional disambiguation of voice commands
US20110184723A1 (en) * 2010-01-25 2011-07-28 Microsoft Corporation Phonetic suggestion engine
WO2011102246A1 (en) * 2010-02-18 2011-08-25 株式会社ニコン Information processing device, portable device and information processing system
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US9009040B2 (en) * 2010-05-05 2015-04-14 Cisco Technology, Inc. Training a transcription system
US20110314003A1 (en) * 2010-06-17 2011-12-22 Microsoft Corporation Template concatenation for capturing multiple concepts in a voice query
US8442827B2 (en) * 2010-06-18 2013-05-14 At&T Intellectual Property I, L.P. System and method for customized voice response
US10002608B2 (en) 2010-09-17 2018-06-19 Nuance Communications, Inc. System and method for using prosody for voice-enabled search
US8401853B2 (en) 2010-09-22 2013-03-19 At&T Intellectual Property I, L.P. System and method for enhancing voice-enabled search based on automated demographic identification
KR20120066530A (en) * 2010-12-14 2012-06-22 한국전자통신연구원 Method of estimating language model weight and apparatus for the same
US9063931B2 (en) * 2011-02-16 2015-06-23 Ming-Yuan Wu Multiple language translation system
WO2012116110A1 (en) * 2011-02-22 2012-08-30 Speak With Me, Inc. Hybridized client-server speech recognition
US9262612B2 (en) 2011-03-21 2016-02-16 Apple Inc. Device access using voice authentication
US9679561B2 (en) 2011-03-28 2017-06-13 Nuance Communications, Inc. System and method for rapid customization of speech recognition models
US9263045B2 (en) * 2011-05-17 2016-02-16 Microsoft Technology Licensing, Llc Multi-mode text input
US10057736B2 (en) 2011-06-03 2018-08-21 Apple Inc. Active transport based notifications
EP2721608B1 (en) * 2011-06-19 2019-03-13 MModal IP LLC Speech recognition using context-aware recognition models
CA2838283A1 (en) * 2011-06-20 2012-12-27 Tandemseven, Inc. System and method for building and managing user experience for computer software interfaces
GB2493413B (en) * 2011-07-25 2013-12-25 Ibm Maintaining and supplying speech models
US8650031B1 (en) * 2011-07-31 2014-02-11 Nuance Communications, Inc. Accuracy improvement of spoken queries transcription using co-occurrence information
KR20130016644A (en) * 2011-08-08 2013-02-18 삼성전자주식회사 Voice recognition apparatus, voice recognition server, voice recognition system and voice recognition method
US8676580B2 (en) * 2011-08-16 2014-03-18 International Business Machines Corporation Automatic speech and concept recognition
US9411970B2 (en) * 2011-08-19 2016-08-09 Microsoft Technology Licensing, Llc Sealing secret data with a policy that includes a sensor-based constraint
US8589160B2 (en) * 2011-08-19 2013-11-19 Dolbey & Company, Inc. Systems and methods for providing an electronic dictation interface
CN103918027B (en) * 2011-09-21 2016-08-24 纽安斯通信有限公司 Effective gradual modification of the optimum Finite State Transformer (FST) in voice application
US9129606B2 (en) * 2011-09-23 2015-09-08 Microsoft Technology Licensing, Llc User query history expansion for improving language model adaptation
US9053087B2 (en) * 2011-09-23 2015-06-09 Microsoft Technology Licensing, Llc Automatic semantic evaluation of speech recognition results
USD671558S1 (en) 2011-10-04 2012-11-27 Apple Inc. Display screen or portion thereof with icon
US8924853B2 (en) * 2011-10-07 2014-12-30 Blackberry Limited Apparatus, and associated method, for cognitively translating media to facilitate understanding
KR101185354B1 (en) * 2011-10-10 2012-09-21 한화에스앤씨주식회사 Apparatus for providing linked service among closed user groups based on smart television and smart set-top box
US8930189B2 (en) 2011-10-28 2015-01-06 Microsoft Corporation Distributed user input to text generated by a speech to text transcription service
US9245254B2 (en) 2011-12-01 2016-01-26 Elwha Llc Enhanced voice conferencing with history, language translation and identification
US9064152B2 (en) 2011-12-01 2015-06-23 Elwha Llc Vehicular threat detection based on image analysis
US9159236B2 (en) 2011-12-01 2015-10-13 Elwha Llc Presentation of shared threat information in a transportation-related context
US8934652B2 (en) 2011-12-01 2015-01-13 Elwha Llc Visual presentation of speaker-related information
US9368028B2 (en) 2011-12-01 2016-06-14 Microsoft Technology Licensing, Llc Determining threats based on information from road-based devices in a transportation-related context
US8811638B2 (en) 2011-12-01 2014-08-19 Elwha Llc Audible assistance
US10875525B2 (en) 2011-12-01 2020-12-29 Microsoft Technology Licensing Llc Ability enhancement
US9053096B2 (en) * 2011-12-01 2015-06-09 Elwha Llc Language translation based on speaker-related information
US9107012B2 (en) 2011-12-01 2015-08-11 Elwha Llc Vehicular threat detection based on audio signals
US9214157B2 (en) * 2011-12-06 2015-12-15 At&T Intellectual Property I, L.P. System and method for machine-mediated human-human conversation
US9348479B2 (en) 2011-12-08 2016-05-24 Microsoft Technology Licensing, Llc Sentiment aware user interface customization
US8903824B2 (en) * 2011-12-09 2014-12-02 International Business Machines Corporation Vertex-proximity query processing
US8788269B2 (en) 2011-12-15 2014-07-22 Microsoft Corporation Satisfying specified intent(s) based on multimodal request(s)
US9378290B2 (en) 2011-12-20 2016-06-28 Microsoft Technology Licensing, Llc Scenario-adaptive input method editor
US9324323B1 (en) * 2012-01-13 2016-04-26 Google Inc. Speech recognition using topic-specific language models
US11544750B1 (en) 2012-01-17 2023-01-03 Google Llc Overlaying content items with third-party reviews
USD692910S1 (en) * 2012-02-07 2013-11-05 Apple Inc. Display screen or portion thereof with graphical user interface
US10209954B2 (en) * 2012-02-14 2019-02-19 Microsoft Technology Licensing, Llc Equal access to speech and touch input
US10134385B2 (en) 2012-03-02 2018-11-20 Apple Inc. Systems and methods for name pronunciation
USD703231S1 (en) 2012-03-06 2014-04-22 Apple Inc. Display screen or portion thereof with graphical user interface
US8775177B1 (en) 2012-03-08 2014-07-08 Google Inc. Speech recognition process
US9317605B1 (en) 2012-03-21 2016-04-19 Google Inc. Presenting forked auto-completions
US8521539B1 (en) * 2012-03-26 2013-08-27 Nuance Communications, Inc. Method for chinese point-of-interest search
USD705808S1 (en) * 2012-03-27 2014-05-27 Apple Inc. Display screen or portion thereof with animated graphical user interface
WO2013169759A2 (en) * 2012-05-07 2013-11-14 Citrix Systems, Inc. Speech recognition support for remote applications and desktops
US9280610B2 (en) 2012-05-14 2016-03-08 Apple Inc. Crowd sourcing information to fulfill user requests
US10417037B2 (en) 2012-05-15 2019-09-17 Apple Inc. Systems and methods for integrating third party services with a digital assistant
CN103426089B (en) * 2012-05-17 2019-05-03 北京京东尚科信息技术有限公司 Commodity purchase guiding system and method
US9639676B2 (en) 2012-05-31 2017-05-02 Microsoft Technology Licensing, Llc Login interface selection for computing environment user login
US20130326347A1 (en) * 2012-05-31 2013-12-05 Microsoft Corporation Application language libraries for managing computing environment languages
US11023520B1 (en) 2012-06-01 2021-06-01 Google Llc Background audio identification for query disambiguation
US9081814B1 (en) 2012-06-01 2015-07-14 Google Inc. Using an entity database to answer entity-triggering questions
US9123338B1 (en) 2012-06-01 2015-09-01 Google Inc. Background audio identification for speech disambiguation
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
USD711408S1 (en) 2012-06-10 2014-08-19 Apple Inc. Display screen or portion thereof with graphical user interface
USD703695S1 (en) 2012-06-10 2014-04-29 Apple Inc. Display screen or portion thereof with graphical user interface
USD754159S1 (en) * 2012-06-11 2016-04-19 Apple Inc. Display screen or portion thereof with graphical user interface
US9734839B1 (en) * 2012-06-20 2017-08-15 Amazon Technologies, Inc. Routing natural language commands to the appropriate applications
US9043205B2 (en) * 2012-06-21 2015-05-26 Google Inc. Dynamic language model
US9594831B2 (en) 2012-06-22 2017-03-14 Microsoft Technology Licensing, Llc Targeted disambiguation of named entities
CN110488991A (en) 2012-06-25 2019-11-22 微软技术许可有限责任公司 Input Method Editor application platform
KR101605862B1 (en) 2012-06-29 2016-03-24 삼성전자주식회사 Display apparatus, electronic device, interactive system and controlling method thereof
KR20140004515A (en) 2012-07-03 2014-01-13 삼성전자주식회사 Display apparatus, interactive server and method for providing response information
US20140039893A1 (en) * 2012-07-31 2014-02-06 Sri International Personalized Voice-Driven User Interfaces for Remote Multi-User Services
US8831957B2 (en) * 2012-08-01 2014-09-09 Google Inc. Speech recognition models based on location indicia
US8959109B2 (en) 2012-08-06 2015-02-17 Microsoft Corporation Business intelligent in-document suggestions
EP2891078A4 (en) 2012-08-30 2016-03-23 Microsoft Technology Licensing Llc Feature-based candidate selection
US20140074466A1 (en) 2012-09-10 2014-03-13 Google Inc. Answering questions using environmental context
US9547647B2 (en) 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
RU2530268C2 (en) * 2012-11-28 2014-10-10 Общество с ограниченной ответственностью "Спиктуит" Method for user training of information dialogue system
US9190057B2 (en) * 2012-12-12 2015-11-17 Amazon Technologies, Inc. Speech model retrieval in distributed speech recognition systems
JPWO2014103568A1 (en) * 2012-12-28 2017-01-12 ソニー株式会社 Information processing apparatus, information processing method, and program
AU349920S (en) * 2013-01-05 2013-07-29 Samsung Electronics Co Ltd Display screen for an electronic device
CN103942230B (en) * 2013-01-21 2017-03-29 上海智臻智能网络科技股份有限公司 A kind of methods, devices and systems for carrying out voice web page navigation
US9697821B2 (en) * 2013-01-29 2017-07-04 Tencent Technology (Shenzhen) Company Limited Method and system for building a topic specific language model for use in automatic speech recognition
KR20230137475A (en) 2013-02-07 2023-10-04 애플 인크. Voice trigger for a digital assistant
USD736255S1 (en) * 2013-02-23 2015-08-11 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
US10424292B1 (en) * 2013-03-14 2019-09-24 Amazon Technologies, Inc. System for recognizing and responding to environmental noises
US10652394B2 (en) 2013-03-14 2020-05-12 Apple Inc. System and method for processing voicemail
US9875237B2 (en) * 2013-03-14 2018-01-23 Microsfot Technology Licensing, Llc Using human perception in building language understanding models
US10748529B1 (en) 2013-03-15 2020-08-18 Apple Inc. Voice activated device for use with a voice-based digital assistant
US9953630B1 (en) * 2013-05-31 2018-04-24 Amazon Technologies, Inc. Language recognition for device settings
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
WO2014197335A1 (en) 2013-06-08 2014-12-11 Apple Inc. Interpreting and acting upon commands that involve sharing information with remote devices
USD744529S1 (en) 2013-06-09 2015-12-01 Apple Inc. Display screen or portion thereof with icon
EP3937002A1 (en) 2013-06-09 2022-01-12 Apple Inc. Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant
US10176167B2 (en) 2013-06-09 2019-01-08 Apple Inc. System and method for inferring user intent from speech inputs
USD755240S1 (en) 2013-06-09 2016-05-03 Apple Inc. Display screen or portion thereof with graphical user interface
USD819649S1 (en) 2013-06-09 2018-06-05 Apple Inc. Display screen or portion thereof with graphical user interface
USD738889S1 (en) 2013-06-09 2015-09-15 Apple Inc. Display screen or portion thereof with animated graphical user interface
USD739873S1 (en) * 2013-06-10 2015-09-29 Huawei Technologies Co., Ltd. Display screen with icon
US9728184B2 (en) 2013-06-18 2017-08-08 Microsoft Technology Licensing, Llc Restructuring deep neural network acoustic models
US9311298B2 (en) 2013-06-21 2016-04-12 Microsoft Technology Licensing, Llc Building conversational understanding systems using a toolset
US9589565B2 (en) 2013-06-21 2017-03-07 Microsoft Technology Licensing, Llc Environmentally aware dialog policies and response generation
US9640182B2 (en) * 2013-07-01 2017-05-02 Toyota Motor Engineering & Manufacturing North America, Inc. Systems and vehicles that provide speech recognition system notifications
US9646606B2 (en) 2013-07-03 2017-05-09 Google Inc. Speech recognition using domain knowledge
WO2015018055A1 (en) 2013-08-09 2015-02-12 Microsoft Corporation Input method editor providing language assistance
CN105453080A (en) * 2013-08-30 2016-03-30 英特尔公司 Extensible context-aware natural language interactions for virtual personal assistants
USD748140S1 (en) * 2013-09-03 2016-01-26 Samsung Electronics Co., Ltd. Display screen portion with icon
KR102065409B1 (en) * 2013-09-04 2020-01-13 엘지전자 주식회사 Mobile terminal and method for controlling the same
USD746831S1 (en) 2013-09-10 2016-01-05 Apple Inc. Display screen or portion thereof with graphical user interface
US9229680B2 (en) * 2013-09-20 2016-01-05 Oracle International Corporation Enhanced voice command of computing devices
US10296160B2 (en) 2013-12-06 2019-05-21 Apple Inc. Method for extracting salient dialog usage from live data
US10873616B1 (en) 2013-12-10 2020-12-22 Google Llc Providing content to co-located devices with enhanced presentation characteristics
US9767803B1 (en) 2013-12-16 2017-09-19 Aftershock Services, Inc. Dynamically selecting speech functionality on client devices
US9626966B2 (en) * 2013-12-18 2017-04-18 Harman International Industries, Incorporated Voice recognition query response systems and methods for generating query responses using information from a vehicle
US9324321B2 (en) 2014-03-07 2016-04-26 Microsoft Technology Licensing, Llc Low-footprint adaptation and personalization for a deep neural network
US20150261968A1 (en) * 2014-03-12 2015-09-17 Ebay Inc. Visualizing random characters for keyboard-less devices
USD758442S1 (en) * 2014-03-26 2016-06-07 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
US9529794B2 (en) 2014-03-27 2016-12-27 Microsoft Technology Licensing, Llc Flexible schema for language model customization
US9614724B2 (en) 2014-04-21 2017-04-04 Microsoft Technology Licensing, Llc Session-based device configuration
US9520127B2 (en) 2014-04-29 2016-12-13 Microsoft Technology Licensing, Llc Shared hidden layer combination for speech recognition systems
US9430667B2 (en) 2014-05-12 2016-08-30 Microsoft Technology Licensing, Llc Managed wireless distribution network
US9384335B2 (en) 2014-05-12 2016-07-05 Microsoft Technology Licensing, Llc Content delivery prioritization in managed wireless distribution networks
US9384334B2 (en) 2014-05-12 2016-07-05 Microsoft Technology Licensing, Llc Content discovery in managed wireless distribution networks
US10111099B2 (en) 2014-05-12 2018-10-23 Microsoft Technology Licensing, Llc Distributing content in managed wireless distribution networks
US9874914B2 (en) 2014-05-19 2018-01-23 Microsoft Technology Licensing, Llc Power management contracts for accessory devices
USD772257S1 (en) * 2014-05-28 2016-11-22 Ricoh Company, Ltd. Display screen with graphical user interface
US9437189B2 (en) * 2014-05-29 2016-09-06 Google Inc. Generating language models
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9633004B2 (en) 2014-05-30 2017-04-25 Apple Inc. Better resolution when referencing to concepts
EP3480811A1 (en) 2014-05-30 2019-05-08 Apple Inc. Multi-command single utterance input method
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
USD771112S1 (en) 2014-06-01 2016-11-08 Apple Inc. Display screen or portion thereof with graphical user interface
US10037202B2 (en) 2014-06-03 2018-07-31 Microsoft Technology Licensing, Llc Techniques to isolating a portion of an online computing service
US9509799B1 (en) 2014-06-04 2016-11-29 Grandios Technologies, Llc Providing status updates via a personal assistant
US8995972B1 (en) 2014-06-05 2015-03-31 Grandios Technologies, Llc Automatic personal assistance between users devices
US10140981B1 (en) * 2014-06-10 2018-11-27 Amazon Technologies, Inc. Dynamic arc weights in speech recognition models
US9367490B2 (en) 2014-06-13 2016-06-14 Microsoft Technology Licensing, Llc Reversible connector for accessory devices
US9717006B2 (en) 2014-06-23 2017-07-25 Microsoft Technology Licensing, Llc Device quarantine in a wireless network
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
USD753696S1 (en) 2014-09-01 2016-04-12 Apple Inc. Display screen or portion thereof with graphical user interface
USD765114S1 (en) 2014-09-02 2016-08-30 Apple Inc. Display screen or portion thereof with graphical user interface
WO2016034492A1 (en) * 2014-09-02 2016-03-10 Philips Lighting Holding B.V. A method of applying a lighting arrangement to a surface and a lighting surface
USD753697S1 (en) 2014-09-02 2016-04-12 Apple Inc. Display screen or portion thereof with graphical user interface
US9953646B2 (en) 2014-09-02 2018-04-24 Belleau Technologies Method and system for dynamic speech recognition and tracking of prewritten script
USD791143S1 (en) * 2014-09-03 2017-07-04 Apple Inc. Display screen or portion thereof with graphical user interface
USD767595S1 (en) * 2014-09-03 2016-09-27 Apple Inc. Display screen or portion thereof with graphical user interface
CN105469793A (en) * 2014-09-11 2016-04-06 苗码信息科技(上海)股份有限公司 Full-automatic onsite driving control method via foreign language speech
US9818400B2 (en) 2014-09-11 2017-11-14 Apple Inc. Method and apparatus for discovering trending terms in speech requests
CN105469792A (en) * 2014-09-11 2016-04-06 苗码信息科技(上海)股份有限公司 Remote automatic navigating and driving automobile system via speaker-independent foreign language speech
US9606986B2 (en) * 2014-09-29 2017-03-28 Apple Inc. Integrated word N-gram and class M-gram language models
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10127911B2 (en) 2014-09-30 2018-11-13 Apple Inc. Speaker identification and unsupervised speaker adaptation techniques
US20160094491A1 (en) * 2014-09-30 2016-03-31 Genesys Telecommunications Laboratories, Inc. Pattern-controlled automated messaging system
US10074360B2 (en) 2014-09-30 2018-09-11 Apple Inc. Providing an indication of the suitability of speech recognition
CN104462262B (en) * 2014-11-21 2017-10-31 北京奇虎科技有限公司 A kind of method for realizing phonetic search, device and browser client
CN105827878B (en) * 2015-01-04 2019-06-25 中国移动通信集团公司 Voice messaging conversion method and voice transfer gateway
RU2646350C2 (en) * 2015-01-27 2018-03-02 Общество С Ограниченной Ответственностью "Яндекс" Method of entering data to electronic device, method of processing voice request, machine-readable media (options), electronic device, server and system
US10152299B2 (en) 2015-03-06 2018-12-11 Apple Inc. Reducing response latency of intelligent automated assistants
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
US9721566B2 (en) 2015-03-08 2017-08-01 Apple Inc. Competing devices responding to voice triggers
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
USD771670S1 (en) 2015-03-09 2016-11-15 Apple Inc. Display screen or portion thereof with animated graphical user interface
US10460227B2 (en) 2015-05-15 2019-10-29 Apple Inc. Virtual assistant in a communication session
US9966073B2 (en) * 2015-05-27 2018-05-08 Google Llc Context-sensitive dynamic update of voice to text model in a voice-enabled electronic device
US10083688B2 (en) 2015-05-27 2018-09-25 Apple Inc. Device voice control for selecting a displayed affordance
US10200824B2 (en) 2015-05-27 2019-02-05 Apple Inc. Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device
KR102394202B1 (en) * 2015-05-29 2022-05-04 삼성전자주식회사 Method for processing input between devices and electronic device thereof
USD760746S1 (en) 2015-06-04 2016-07-05 Apple Inc. Display screen or portion thereof with animated graphical user interface
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US11025565B2 (en) 2015-06-07 2021-06-01 Apple Inc. Personalized prediction of responses for instant messaging
US10403271B2 (en) 2015-06-11 2019-09-03 Nice Ltd. System and method for automatic language model selection
US20160378747A1 (en) 2015-06-29 2016-12-29 Apple Inc. Virtual assistant for media playback
US9704483B2 (en) * 2015-07-28 2017-07-11 Google Inc. Collaborative language model biasing
US9576578B1 (en) * 2015-08-12 2017-02-21 Google Inc. Contextual improvement of voice query recognition
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US10740384B2 (en) 2015-09-08 2020-08-11 Apple Inc. Intelligent automated assistant for media search and playback
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10331312B2 (en) 2015-09-08 2019-06-25 Apple Inc. Intelligent automated assistant in a media environment
US9858923B2 (en) * 2015-09-24 2018-01-02 Intel Corporation Dynamic adaptation of language models and semantic tracking for automatic speech recognition
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US11587559B2 (en) 2015-09-30 2023-02-21 Apple Inc. Intelligent device identification
US9946437B2 (en) * 2015-11-05 2018-04-17 International Business Machines Corporation Modifying an appearance of a GUI to improve GUI usability
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10956666B2 (en) 2015-11-09 2021-03-23 Apple Inc. Unconventional virtual assistant interactions
US10468016B2 (en) 2015-11-24 2019-11-05 International Business Machines Corporation System and method for supporting automatic speech recognition of regional accents based on statistical information and user corrections
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US10896681B2 (en) * 2015-12-29 2021-01-19 Google Llc Speech recognition with selective use of dynamic language models
US10049744B2 (en) * 2016-01-08 2018-08-14 Samsung Electronics Co., Ltd. Three-dimensional (3D) semiconductor memory devices and methods of manufacturing the same
US20170206899A1 (en) * 2016-01-20 2017-07-20 Fitbit, Inc. Better communication channel for requests and responses having an intelligent agent
US20170235724A1 (en) * 2016-02-11 2017-08-17 Emily Grewal Systems and methods for generating personalized language models and translation using the same
US10282417B2 (en) 2016-02-19 2019-05-07 International Business Machines Corporation Conversational list management
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
DK201670539A1 (en) * 2016-03-14 2017-10-02 Apple Inc Dictation that allows editing
US20170308289A1 (en) * 2016-04-20 2017-10-26 Google Inc. Iconographic symbol search within a graphical keyboard
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US11227589B2 (en) 2016-06-06 2022-01-18 Apple Inc. Intelligent list reading
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
DK179309B1 (en) 2016-06-09 2018-04-23 Apple Inc Intelligent automated assistant in a home environment
USD815649S1 (en) 2016-06-10 2018-04-17 Apple Inc. Display screen or portion thereof with graphical user interface
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
USD804502S1 (en) 2016-06-11 2017-12-05 Apple Inc. Display screen or portion thereof with graphical user interface
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
USD798893S1 (en) 2016-06-11 2017-10-03 Apple Inc. Display screen or portion thereof with animated graphical user interface
DK179343B1 (en) 2016-06-11 2018-05-14 Apple Inc Intelligent task discovery
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
DK179049B1 (en) 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
US10409488B2 (en) * 2016-06-13 2019-09-10 Microsoft Technology Licensing, Llc Intelligent virtual keyboards
WO2017217046A1 (en) * 2016-06-15 2017-12-21 ソニー株式会社 Information processing device and information processing method
CN109313895A (en) 2016-06-23 2019-02-05 索尼公司 Information processing unit and information processing method
US10481863B2 (en) * 2016-07-06 2019-11-19 Baidu Usa Llc Systems and methods for improved user interface
USD815110S1 (en) 2016-07-07 2018-04-10 Baidu Usa Llc Display screen or portion thereof with graphical user interface
USD817337S1 (en) 2016-07-07 2018-05-08 Baidu Usa Llc Display screen or portion thereof with graphical user interface
USD812635S1 (en) * 2016-07-07 2018-03-13 Baidu Usa Llc. Display screen or portion thereof with graphical user interface
US9691384B1 (en) 2016-08-19 2017-06-27 Google Inc. Voice action biasing system
US10832664B2 (en) 2016-08-19 2020-11-10 Google Llc Automated speech recognition using language models that selectively use domain-specific model components
US10474753B2 (en) 2016-09-07 2019-11-12 Apple Inc. Language identification using recurrent neural networks
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10366918B2 (en) * 2016-10-04 2019-07-30 International Business Machines Corporation Self-aligned trench metal-alloying for III-V nFETs
US9959864B1 (en) 2016-10-27 2018-05-01 Google Llc Location-based voice query recognition
US11281993B2 (en) 2016-12-05 2022-03-22 Apple Inc. Model and ensemble compression for metric learning
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
US10831366B2 (en) 2016-12-29 2020-11-10 Google Llc Modality learning on mobile devices
US11204787B2 (en) 2017-01-09 2021-12-21 Apple Inc. Application integration with a digital assistant
US10517975B2 (en) * 2017-01-12 2019-12-31 Industrial Technology Research Institute Light source apparatus and method of using the same
US10741174B2 (en) * 2017-01-24 2020-08-11 Lenovo (Singapore) Pte. Ltd. Automatic language identification for speech
US10268669B1 (en) * 2017-01-27 2019-04-23 John C. Allen Intelligent graphical word processing system and method
US10592577B2 (en) 2017-01-31 2020-03-17 Walmart Apollo, Llc Systems and methods for updating a webpage
US11609964B2 (en) 2017-01-31 2023-03-21 Walmart Apollo, Llc Whole page personalization with cyclic dependencies
US11010784B2 (en) 2017-01-31 2021-05-18 Walmart Apollo, Llc Systems and methods for search query refinement
US10628458B2 (en) * 2017-01-31 2020-04-21 Walmart Apollo, Llc Systems and methods for automated recommendations
US10554779B2 (en) 2017-01-31 2020-02-04 Walmart Apollo, Llc Systems and methods for webpage personalization
USD866575S1 (en) * 2017-03-20 2019-11-12 Exfo Inc. Display screen, or portion thereof, with graphical user interface for multi-fiber connector, fiber inspection probe testing
KR102398649B1 (en) * 2017-03-28 2022-05-17 삼성전자주식회사 Electronic device for processing user utterance and method for operation thereof
US20180316634A1 (en) * 2017-04-26 2018-11-01 Microsoft Technology Licensing, Llc Extending application functionality via conversational interfaces
KR102380717B1 (en) * 2017-04-30 2022-03-31 삼성전자주식회사 Electronic apparatus for processing user utterance and controlling method thereof
US10417266B2 (en) 2017-05-09 2019-09-17 Apple Inc. Context-aware ranking of intelligent response suggestions
DK201770383A1 (en) 2017-05-09 2018-12-14 Apple Inc. User interface for correcting recognition errors
US10726832B2 (en) 2017-05-11 2020-07-28 Apple Inc. Maintaining privacy of personal information
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
US10395654B2 (en) 2017-05-11 2019-08-27 Apple Inc. Text normalization based on a data-driven learning network
US11301477B2 (en) 2017-05-12 2022-04-12 Apple Inc. Feedback analysis of a digital assistant
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK201770428A1 (en) 2017-05-12 2019-02-18 Apple Inc. Low-latency intelligent automated assistant
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
US20180336275A1 (en) 2017-05-16 2018-11-22 Apple Inc. Intelligent automated assistant for media exploration
KR102355966B1 (en) * 2017-05-16 2022-02-08 애플 인크. Far-field extension for digital assistant services
US10403278B2 (en) 2017-05-16 2019-09-03 Apple Inc. Methods and systems for phonetic matching in digital assistant services
US10311144B2 (en) 2017-05-16 2019-06-04 Apple Inc. Emoji word sense disambiguation
US20180336892A1 (en) 2017-05-16 2018-11-22 Apple Inc. Detecting a trigger of a digital assistant
DK179560B1 (en) 2017-05-16 2019-02-18 Apple Inc. Far-field extension for digital assistant services
US10657328B2 (en) 2017-06-02 2020-05-19 Apple Inc. Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling
US10616036B2 (en) * 2017-06-07 2020-04-07 Accenture Global Solutions Limited Integration platform for multi-network integration of service platforms
US11900072B1 (en) * 2017-07-18 2024-02-13 Amazon Technologies, Inc. Quick lookup for speech translation
CN109471537A (en) * 2017-09-08 2019-03-15 腾讯科技(深圳)有限公司 Pronunciation inputting method, device, computer equipment and storage medium
USD843442S1 (en) 2017-09-10 2019-03-19 Apple Inc. Type font
US10445429B2 (en) 2017-09-21 2019-10-15 Apple Inc. Natural language understanding using vocabularies with compressed serialized tries
US10755051B2 (en) 2017-09-29 2020-08-25 Apple Inc. Rule-based natural language processing
US10395647B2 (en) * 2017-10-26 2019-08-27 Harman International Industries, Incorporated System and method for natural language processing
US10621282B1 (en) 2017-10-27 2020-04-14 Interactions Llc Accelerating agent performance in a natural language processing system
CN107808007A (en) * 2017-11-16 2018-03-16 百度在线网络技术(北京)有限公司 Information processing method and device
US10636424B2 (en) 2017-11-30 2020-04-28 Apple Inc. Multi-turn canned dialog
US20190214013A1 (en) * 2018-01-05 2019-07-11 Ca, Inc. Speech-to-text conversion based on user interface state awareness
US10733982B2 (en) 2018-01-08 2020-08-04 Apple Inc. Multi-directional dialog
JP7056185B2 (en) * 2018-01-31 2022-04-19 トヨタ自動車株式会社 Information processing equipment and information processing method
US10733375B2 (en) 2018-01-31 2020-08-04 Apple Inc. Knowledge-based framework for improving natural language understanding
US11468885B2 (en) 2018-02-15 2022-10-11 DMAI, Inc. System and method for conversational agent via adaptive caching of dialogue tree
WO2019161229A1 (en) 2018-02-15 2019-08-22 DMAI, Inc. System and method for reconstructing unoccupied 3d space
US20190371318A1 (en) * 2018-02-15 2019-12-05 DMAI, Inc. System and method for adaptive detection of spoken language via multiple speech models
US10832657B2 (en) * 2018-03-01 2020-11-10 International Business Machines Corporation Use of small unit language model for training large unit language models
US10789959B2 (en) 2018-03-02 2020-09-29 Apple Inc. Training speaker recognition models for digital assistants
US11106868B2 (en) * 2018-03-06 2021-08-31 Samsung Electronics Co., Ltd. System and method for language model personalization
US10592604B2 (en) 2018-03-12 2020-03-17 Apple Inc. Inverse text normalization for automatic speech recognition
US10818288B2 (en) 2018-03-26 2020-10-27 Apple Inc. Natural assistant interaction
US10623246B1 (en) * 2018-03-27 2020-04-14 Amazon Technologies, Inc. Device configuration by natural language processing system
US10909331B2 (en) 2018-03-30 2021-02-02 Apple Inc. Implicit identification of translation payload with neural machine translation
US10963273B2 (en) 2018-04-20 2021-03-30 Facebook, Inc. Generating personalized content summaries for users
US11715042B1 (en) 2018-04-20 2023-08-01 Meta Platforms Technologies, Llc Interpretability of deep reinforcement learning models in assistant systems
US11676220B2 (en) 2018-04-20 2023-06-13 Meta Platforms, Inc. Processing multimodal user input for assistant systems
US11307880B2 (en) 2018-04-20 2022-04-19 Meta Platforms, Inc. Assisting users with personalized and contextual communication content
US11886473B2 (en) 2018-04-20 2024-01-30 Meta Platforms, Inc. Intent identification for agent matching by assistant systems
US11145294B2 (en) 2018-05-07 2021-10-12 Apple Inc. Intelligent automated assistant for delivering content from user experiences
US10928918B2 (en) 2018-05-07 2021-02-23 Apple Inc. Raise to speak
US10984780B2 (en) 2018-05-21 2021-04-20 Apple Inc. Global semantic word embeddings using bi-directional recurrent neural networks
US11386266B2 (en) 2018-06-01 2022-07-12 Apple Inc. Text correction
DK201870355A1 (en) 2018-06-01 2019-12-16 Apple Inc. Virtual assistant operation in multi-device environments
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US10892996B2 (en) 2018-06-01 2021-01-12 Apple Inc. Variable latency device coordination
DK179822B1 (en) 2018-06-01 2019-07-12 Apple Inc. Voice interaction at a primary device to access call functionality of a companion device
USD863337S1 (en) 2018-06-03 2019-10-15 Apple Inc. Electronic device with animated graphical user interface
US11076039B2 (en) 2018-06-03 2021-07-27 Apple Inc. Accelerated task performance
US10832678B2 (en) 2018-06-08 2020-11-10 International Business Machines Corporation Filtering audio-based interference from voice commands using interference information
USD900830S1 (en) 2018-09-10 2020-11-03 Apple Inc. Electronic device with graphical user interface
US11010561B2 (en) 2018-09-27 2021-05-18 Apple Inc. Sentiment prediction from textual data
US11170166B2 (en) 2018-09-28 2021-11-09 Apple Inc. Neural typographical error modeling via generative adversarial networks
US10839159B2 (en) 2018-09-28 2020-11-17 Apple Inc. Named entity normalization in a spoken dialog system
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
US11475898B2 (en) 2018-10-26 2022-10-18 Apple Inc. Low-latency multi-speaker speech recognition
US11638059B2 (en) 2019-01-04 2023-04-25 Apple Inc. Content playback on multiple devices
US10949224B2 (en) 2019-01-29 2021-03-16 Walmart Apollo Llc Systems and methods for altering a GUI in response to in-session inferences
USD902221S1 (en) 2019-02-01 2020-11-17 Apple Inc. Electronic device with animated graphical user interface
USD900871S1 (en) 2019-02-04 2020-11-03 Apple Inc. Electronic device with animated graphical user interface
US11348573B2 (en) 2019-03-18 2022-05-31 Apple Inc. Multimodality in digital assistant systems
KR20200113349A (en) 2019-03-25 2020-10-07 삼성전자주식회사 Electronic Device and the Method for Supporting Multitasking thereof
US11423908B2 (en) 2019-05-06 2022-08-23 Apple Inc. Interpreting spoken requests
US11475884B2 (en) 2019-05-06 2022-10-18 Apple Inc. Reducing digital assistant latency when a language is incorrectly determined
US11307752B2 (en) 2019-05-06 2022-04-19 Apple Inc. User configurable task triggers
US11217251B2 (en) 2019-05-06 2022-01-04 Apple Inc. Spoken notifications
US11140099B2 (en) 2019-05-21 2021-10-05 Apple Inc. Providing message response suggestions
DK180129B1 (en) 2019-05-31 2020-06-02 Apple Inc. User activity shortcut suggestions
US11496600B2 (en) 2019-05-31 2022-11-08 Apple Inc. Remote execution of machine-learned models
DK201970510A1 (en) 2019-05-31 2021-02-11 Apple Inc Voice identification in digital assistant systems
US11289073B2 (en) 2019-05-31 2022-03-29 Apple Inc. Device text to speech
US11360641B2 (en) 2019-06-01 2022-06-14 Apple Inc. Increasing the relevance of new available information
KR20210016739A (en) 2019-08-05 2021-02-17 삼성전자주식회사 Electronic device and input method of the same
USD924912S1 (en) 2019-09-09 2021-07-13 Apple Inc. Display screen or portion thereof with graphical user interface
CN114127734A (en) * 2019-09-10 2022-03-01 谷歌有限责任公司 Location-based mode of biasing the provision of content when an automated assistant is responding to condensed natural language input
WO2021056255A1 (en) 2019-09-25 2021-04-01 Apple Inc. Text detection using global geometry estimators
US11308265B1 (en) 2019-10-11 2022-04-19 Wells Fargo Bank, N.A. Digitally aware neural dictation interface
KR20220010034A (en) * 2019-10-15 2022-01-25 구글 엘엘씨 Enter voice-controlled content into a graphical user interface
KR20210055387A (en) 2019-11-07 2021-05-17 삼성전자주식회사 Context based application providing server and controlling method thereof
US11675842B1 (en) * 2020-02-03 2023-06-13 Amazon Technologies, Inc. Generation of recommendation results using a verbal query
CN113555009A (en) * 2020-04-21 2021-10-26 京东数字科技控股有限公司 Method and apparatus for training a model
US11038934B1 (en) 2020-05-11 2021-06-15 Apple Inc. Digital assistant hardware abstraction
US11810578B2 (en) 2020-05-11 2023-11-07 Apple Inc. Device arbitration for digital assistant-based intercom systems
US11755276B2 (en) 2020-05-12 2023-09-12 Apple Inc. Reducing description length based on confidence
CN111711853B (en) 2020-06-09 2022-02-01 北京字节跳动网络技术有限公司 Information processing method, system, device, electronic equipment and storage medium
CN111738023A (en) * 2020-06-24 2020-10-02 宋万利 Automatic image-text audio translation method and system
US20220215833A1 (en) * 2021-01-07 2022-07-07 Lenovo (Singapore) Pte. Ltd. Method and device for converting spoken words to text form
CN113066480B (en) * 2021-03-26 2023-02-17 北京达佳互联信息技术有限公司 Voice recognition method and device, electronic equipment and storage medium
US11810558B2 (en) * 2021-05-26 2023-11-07 International Business Machines Corporation Explaining anomalous phonetic translations

Citations (93)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4820059A (en) * 1985-10-30 1989-04-11 Central Institute For The Deaf Speech processing apparatus and methods
US5632002A (en) * 1992-12-28 1997-05-20 Kabushiki Kaisha Toshiba Speech recognition interface system suitable for window systems and speech mail systems
US5638487A (en) * 1994-12-30 1997-06-10 Purespeech, Inc. Automatic speech recognition
US5715367A (en) * 1995-01-23 1998-02-03 Dragon Systems, Inc. Apparatuses and methods for developing and using models for speech recognition
US5737724A (en) * 1993-11-24 1998-04-07 Lucent Technologies Inc. Speech recognition employing a permissive recognition criterion for a repeated phrase utterance
US5768603A (en) * 1991-07-25 1998-06-16 International Business Machines Corporation Method and system for natural language translation
US6021403A (en) * 1996-07-19 2000-02-01 Microsoft Corporation Intelligent user assistance facility
US6182038B1 (en) * 1997-12-01 2001-01-30 Motorola, Inc. Context dependent phoneme networks for encoding speech information
US6356866B1 (en) * 1998-10-07 2002-03-12 Microsoft Corporation Method for converting a phonetic character string into the text of an Asian language
US20020062216A1 (en) * 2000-11-23 2002-05-23 International Business Machines Corporation Method and system for gathering information by voice input
US6397180B1 (en) * 1996-05-22 2002-05-28 Qwest Communications International Inc. Method and system for performing speech recognition based on best-word scoring of repeated speech attempts
US20020087309A1 (en) * 2000-12-29 2002-07-04 Lee Victor Wai Leung Computer-implemented speech expectation-based probability method and system
US20020087314A1 (en) * 2000-11-14 2002-07-04 International Business Machines Corporation Method and apparatus for phonetic context adaptation for improved speech recognition
US6418431B1 (en) * 1998-03-30 2002-07-09 Microsoft Corporation Information retrieval and speech recognition based on language models
US20020099818A1 (en) * 2000-11-16 2002-07-25 Russell Ethan George Method and system for monitoring the performance of a distributed application
US20020111990A1 (en) * 1999-11-01 2002-08-15 Wood Christopher Noah Internet based message management system
US20030050778A1 (en) * 2001-09-13 2003-03-13 Patrick Nguyen Focused language models for improved speech input of structured documents
US6539358B1 (en) * 2000-05-24 2003-03-25 Delphi Technologies, Inc. Voice-interactive docking station for a portable computing device
US6581033B1 (en) * 1999-10-19 2003-06-17 Microsoft Corporation System and method for correction of speech recognition mode errors
US6678415B1 (en) * 2000-05-12 2004-01-13 Xerox Corporation Document image decoding using an integrated stochastic language model
US20040013882A1 (en) * 2002-07-10 2004-01-22 Rudiger Gorny Multi-layer product containing polycarbonate
US20040024583A1 (en) * 2000-03-20 2004-02-05 Freeman Robert J Natural-language processing system using a large corpus
US20040043758A1 (en) * 2002-08-29 2004-03-04 Nokia Corporation System and method for providing context sensitive recommendations to digital services
US20040049388A1 (en) * 2001-09-05 2004-03-11 Roth Daniel L. Methods, systems, and programming for performing speech recognition
US6714778B2 (en) * 2001-05-15 2004-03-30 Nokia Corporation Context sensitive web services
US20040098571A1 (en) * 2002-11-15 2004-05-20 Falcon Stephen R. Portable computing device-integrated appliance
US6839670B1 (en) * 1995-09-11 2005-01-04 Harman Becker Automotive Systems Gmbh Process for automatic control of one or more devices by voice commands or by real-time voice dialog and apparatus for carrying out this process
US20050005240A1 (en) * 1999-10-05 2005-01-06 Microsoft Corporation Method and system for providing alternatives for text derived from stochastic input sources
US20050091037A1 (en) * 2003-10-24 2005-04-28 Microsoft Corporation System and method for providing context to an input method
US20050108017A1 (en) * 2003-10-27 2005-05-19 John-Alexander Esser Determining language for word recognition event
US20050114474A1 (en) * 2003-11-20 2005-05-26 International Business Machines Corporation Automatic configuration of the network devices via connection to specific switch ports
US6912499B1 (en) * 1999-08-31 2005-06-28 Nortel Networks Limited Method and apparatus for training a multilingual speech model set
US20060004850A1 (en) * 2004-07-01 2006-01-05 Chowdhury Abdur R Analyzing a query log for use in managing category-specific electronic content
US20060004572A1 (en) * 2004-06-30 2006-01-05 Microsoft Corporation Homonym processing in the context of voice-activated command systems
US6985147B2 (en) * 2000-12-15 2006-01-10 International Business Machines Corporation Information access method, system and storage medium
US20060009974A1 (en) * 2004-07-09 2006-01-12 Matsushita Electric Industrial Co., Ltd. Hands-free voice dialing for portable and remote devices
US20060035632A1 (en) * 2004-08-16 2006-02-16 Antti Sorvari Apparatus and method for facilitating contact selection in communication devices
US20060048055A1 (en) * 2004-08-25 2006-03-02 Jun Wu Fault-tolerant romanized input method for non-roman characters
US7027987B1 (en) * 2001-02-07 2006-04-11 Google Inc. Voice interface for a search engine
US20060095248A1 (en) * 2004-11-04 2006-05-04 Microsoft Corporation Machine translation system incorporating syntactic dependency treelets into a statistical framework
US7043422B2 (en) * 2000-10-13 2006-05-09 Microsoft Corporation Method and apparatus for distribution-based language model adaptation
US20060111907A1 (en) * 2004-11-24 2006-05-25 Microsoft Corporation Generic spelling mnemonics
US7107204B1 (en) * 2000-04-24 2006-09-12 Microsoft Corporation Computer-aided writing system and method with cross-language writing wizard
US7174288B2 (en) * 2002-05-08 2007-02-06 Microsoft Corporation Multi-modal entry of ideogrammatic languages
US20070060114A1 (en) * 2005-09-14 2007-03-15 Jorey Ramer Predictive text completion for a mobile communication facility
US7203645B2 (en) * 2001-04-27 2007-04-10 Intel Corporation Speech recognition system loading different recognition engines for different applications
US7203731B1 (en) * 2000-03-03 2007-04-10 Intel Corporation Dynamic replication of files in a network storage system
US20070124507A1 (en) * 2005-11-28 2007-05-31 Sap Ag Systems and methods of processing annotations and multimodal user inputs
US20080027723A1 (en) * 2000-11-30 2008-01-31 Verizon Services Organization, Inc. Methods and apparatus for performing speech recognition over a network and using speech recognition results
US20080091406A1 (en) * 2006-10-16 2008-04-17 Voicebox Technologies, Inc. System and method for a cooperative conversational voice user interface
US20080091435A1 (en) * 2006-10-13 2008-04-17 Brian Strope Business listing search
US20080091443A1 (en) * 2006-10-13 2008-04-17 Brian Strope Business listing search
US7383553B2 (en) * 2003-05-15 2008-06-03 International Business Machines Corporation Accessing a platform independent input method editor from an underlying operating system
US20080133228A1 (en) * 2006-11-30 2008-06-05 Rao Ashwin P Multimodal speech recognition system
US20080131851A1 (en) * 2006-12-04 2008-06-05 Dimitri Kanevsky Context-sensitive language learning
US7392188B2 (en) * 2003-07-31 2008-06-24 Telefonaktiebolaget Lm Ericsson (Publ) System and method enabling acoustic barge-in
US20080155111A1 (en) * 2006-12-20 2008-06-26 Hitachi, Ltd. Delivery system, communication apparatus and delivery method
US20090030696A1 (en) * 2007-03-07 2009-01-29 Cerra Joseph P Using results of unstructured language model based speech recognition to control a system-level function of a mobile communications facility
US20090030687A1 (en) * 2007-03-07 2009-01-29 Cerra Joseph P Adapting an unstructured language model speech recognition system based on usage
US20090055162A1 (en) * 2007-08-20 2009-02-26 Microsoft Corporation Hmm-based bilingual (mandarin-english) tts techniques
US20090063455A1 (en) * 2007-08-30 2009-03-05 Microsoft Corporation Bipartite Graph Reinforcement Modeling to Annotate Web Images
US7526431B2 (en) * 2001-09-05 2009-04-28 Voice Signal Technologies, Inc. Speech recognition using ambiguous or phone key spelling and/or filtering
US20090164216A1 (en) * 2007-12-21 2009-06-25 General Motors Corporation In-vehicle circumstantial speech recognition
US20090292529A1 (en) * 2002-12-18 2009-11-26 At&T Corp. System and method of providing a spoken dialog interface to a website
US20100004930A1 (en) * 2008-07-02 2010-01-07 Brian Strope Speech Recognition with Parallel Recognition Tasks
US20100042400A1 (en) * 2005-12-21 2010-02-18 Hans-Ulrich Block Method for Triggering at Least One First and Second Background Application via a Universal Language Dialog System
US20100049502A1 (en) * 2000-07-24 2010-02-25 Microsoft Corporation Method and system of generating reference variations for directory assistance data
US7672833B2 (en) * 2005-09-22 2010-03-02 Fair Isaac Corporation Method and apparatus for automatic entity disambiguation
US20100057687A1 (en) * 2008-09-04 2010-03-04 Microsoft Corporation Predicting future queries from log data
US20100088303A1 (en) * 2008-10-03 2010-04-08 Microsoft Corporation Mining new words from a query log for input method editors
US7698136B1 (en) * 2003-01-28 2010-04-13 Voxify, Inc. Methods and apparatus for flexible speech recognition
US20100100377A1 (en) * 2008-10-10 2010-04-22 Shreedhar Madhavapeddi Generating and processing forms for receiving speech data
US20100111059A1 (en) * 2007-04-02 2010-05-06 Benjamin Bappu Content delivery
US20100153219A1 (en) * 2008-12-12 2010-06-17 Microsoft Corporation In-text embedded advertising
US20100153370A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation System of ranking search results based on query specific position bias
US20110004462A1 (en) * 2009-07-01 2011-01-06 Comcast Interactive Media, Llc Generating Topic-Specific Language Models
US7877385B2 (en) * 2007-09-21 2011-01-25 Microsoft Corporation Information retrieval using query-document pair information
US7881936B2 (en) * 1998-12-04 2011-02-01 Tegic Communications, Inc. Multimodal disambiguation of speech recognition
US7907705B1 (en) * 2006-10-10 2011-03-15 Intuit Inc. Speech to text for assisted form completion
US20110066577A1 (en) * 2009-09-15 2011-03-17 Microsoft Corporation Machine Learning Using Relational Databases
US20110077943A1 (en) * 2006-06-26 2011-03-31 Nec Corporation System for generating language model, method of generating language model, and program for language model generation
US7921107B2 (en) * 2007-04-10 2011-04-05 Yahoo! Inc. System for generating query suggestions using a network of users and advertisers
US20110093265A1 (en) * 2009-10-16 2011-04-21 Amanda Stent Systems and Methods for Creating and Using Geo-Centric Language Models
US7941189B2 (en) * 2007-02-07 2011-05-10 Denso Corporation Communicating road noise control system, in-vehicle road noise controller, and server
US7953692B2 (en) * 2007-12-07 2011-05-31 Microsoft Corporation Predicting candidates using information sources
US20110137635A1 (en) * 2009-12-08 2011-06-09 Microsoft Corporation Transliterating semitic languages including diacritics
US20110162035A1 (en) * 2009-12-31 2011-06-30 Apple Inc. Location-based dock for a computing device
US20120022868A1 (en) * 2010-01-05 2012-01-26 Google Inc. Word-Level Correction of Speech Input
US20120022867A1 (en) * 2009-12-23 2012-01-26 Ballinger Brandon M Speech to Text Conversion
US20120060113A1 (en) * 2010-09-08 2012-03-08 Nuance Communications, Inc. Methods and apparatus for displaying content
US8156129B2 (en) * 2009-01-15 2012-04-10 Microsoft Corporation Substantially similar queries
US20120143611A1 (en) * 2010-12-07 2012-06-07 Microsoft Corporation Trajectory Tiling Approach for Text-to-Speech
US20120191745A1 (en) * 2011-01-24 2012-07-26 Yahoo!, Inc. Synthesized Suggestions for Web-Search Queries

Family Cites Families (134)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5267345A (en) 1992-02-10 1993-11-30 International Business Machines Corporation Speech recognition apparatus which predicts word classes from context and words from word classes
US5819225A (en) * 1996-05-30 1998-10-06 International Business Machines Corporation Display indications of speech processing states in speech recognition system
US5822730A (en) 1996-08-22 1998-10-13 Dragon Systems, Inc. Lexical tree pre-filtering in speech recognition
US6167377A (en) 1997-03-28 2000-12-26 Dragon Systems, Inc. Speech recognition language models
US6119186A (en) 1997-05-30 2000-09-12 Texas Instruments Incorporated Computer system with environmental manager for detecting and responding to changing environmental conditions
EP0979497A1 (en) * 1997-10-08 2000-02-16 Koninklijke Philips Electronics N.V. Vocabulary and/or language model training
US6317712B1 (en) 1998-02-03 2001-11-13 Texas Instruments Incorporated Method of phonetic modeling using acoustic decision tree
AU2981099A (en) * 1998-03-09 1999-09-27 Trustees Of Tufts College Treatment of compulsive behaviours in man and animals
IL140805A0 (en) * 1998-10-02 2002-02-10 Ibm Structure skeletons for efficient voice navigation through generic hierarchical objects
US8938688B2 (en) 1998-12-04 2015-01-20 Nuance Communications, Inc. Contextual prediction of user words and user actions
US6922669B2 (en) 1998-12-29 2005-07-26 Koninklijke Philips Electronics N.V. Knowledge-based strategies applied to N-best lists in automatic speech recognition systems
US6851115B1 (en) * 1999-01-05 2005-02-01 Sri International Software-based architecture for communication and cooperation among distributed electronic agents
US7030863B2 (en) 2000-05-26 2006-04-18 America Online, Incorporated Virtual keyboard system with automatic correction
JP4292646B2 (en) 1999-09-16 2009-07-08 株式会社デンソー User interface device, navigation system, information processing device, and recording medium
FI112978B (en) * 1999-09-17 2004-02-13 Nokia Corp Entering Symbols
US6778959B1 (en) 1999-10-21 2004-08-17 Sony Corporation System and method for speech verification using out-of-vocabulary models
US6446041B1 (en) 1999-10-27 2002-09-03 Microsoft Corporation Method and system for providing audio playback of a multi-source document
US7403888B1 (en) * 1999-11-05 2008-07-22 Microsoft Corporation Language input user interface
US20020055844A1 (en) * 2000-02-25 2002-05-09 L'esperance Lauren Speech user interface for portable personal devices
US7149970B1 (en) 2000-06-23 2006-12-12 Microsoft Corporation Method and system for filtering and selecting from a candidate list generated by a stochastic input method
US7219058B1 (en) 2000-10-13 2007-05-15 At&T Corp. System and method for processing speech recognition results
US7457750B2 (en) 2000-10-13 2008-11-25 At&T Corp. Systems and methods for dynamic re-configurable speech recognition
US6876966B1 (en) 2000-10-16 2005-04-05 Microsoft Corporation Pattern recognition training method and apparatus using inserted noise followed by noise reduction
US20020087325A1 (en) * 2000-12-29 2002-07-04 Lee Victor Wai Leung Dialogue application computer platform
US20020087315A1 (en) * 2000-12-29 2002-07-04 Lee Victor Wai Leung Computer-implemented multi-scanning language method and system
DE10100725C1 (en) 2001-01-10 2002-01-24 Philips Corp Intellectual Pty Automatic dialogue system for speech interrogation of databank entries uses speech recognition system assisted by speech model obtained before beginning of dialogue
US7076738B2 (en) * 2001-03-02 2006-07-11 Semantic Compaction Systems Computer device, method and article of manufacture for utilizing sequenced symbols to enable programmed application and commands
US7778816B2 (en) 2001-04-24 2010-08-17 Microsoft Corporation Method and system for applying input mode bias
US20030008680A1 (en) 2001-05-24 2003-01-09 Huh Stephen S. Using identification information obtained from a portable phone
US6959276B2 (en) 2001-09-27 2005-10-25 Microsoft Corporation Including the category of environmental noise when processing speech signals
US6950796B2 (en) 2001-11-05 2005-09-27 Motorola, Inc. Speech recognition by dynamical noise model adaptation
US7610556B2 (en) * 2001-12-28 2009-10-27 Microsoft Corporation Dialog manager for interactive dialog with computer user
US6999931B2 (en) 2002-02-01 2006-02-14 Intel Corporation Spoken dialog system using a best-fit language model and best-fit grammar
US7149694B1 (en) 2002-02-13 2006-12-12 Siebel Systems, Inc. Method and system for building/updating grammars in voice access systems
US8213917B2 (en) 2006-05-05 2012-07-03 Waloomba Tech Ltd., L.L.C. Reusable multimodal application
US7403890B2 (en) 2002-05-13 2008-07-22 Roushar Joseph C Multi-dimensional method and apparatus for automated language interpretation
US7224981B2 (en) 2002-06-20 2007-05-29 Intel Corporation Speech recognition of mobile devices
JP4109063B2 (en) 2002-09-18 2008-06-25 パイオニア株式会社 Speech recognition apparatus and speech recognition method
US7328155B2 (en) 2002-09-25 2008-02-05 Toyota Infotechnology Center Co., Ltd. Method and system for speech recognition using grammar weighted based upon location information
JP4352790B2 (en) 2002-10-31 2009-10-28 セイコーエプソン株式会社 Acoustic model creation method, speech recognition device, and vehicle having speech recognition device
US7149688B2 (en) 2002-11-04 2006-12-12 Speechworks International, Inc. Multi-lingual speech recognition with cross-language context modeling
US7457745B2 (en) 2002-12-03 2008-11-25 Hrl Laboratories, Llc Method and apparatus for fast on-line automatic speaker/environment adaptation for speech/speaker recognition in the presence of changing environments
EP2017828A1 (en) 2002-12-10 2009-01-21 Kirusa, Inc. Techniques for disambiguating speech input using multimodal interfaces
EP1574093B1 (en) * 2002-12-20 2009-04-01 Nokia Corporation Method and device for organizing user provided information with meta-information
AU2003300143A1 (en) * 2002-12-31 2004-07-29 Digispeech Marketing Ltd Comprehensive spoken language learning system
US7805299B2 (en) 2004-03-01 2010-09-28 Coifman Robert E Method and apparatus for improving the transcription accuracy of speech recognition software
US20040243415A1 (en) * 2003-06-02 2004-12-02 International Business Machines Corporation Architecture for a speech input method editor for handheld portable devices
JP4548646B2 (en) 2003-09-12 2010-09-22 株式会社エヌ・ティ・ティ・ドコモ Noise model noise adaptation system, noise adaptation method, and speech recognition noise adaptation program
CA2486128C (en) 2003-10-30 2011-08-23 At&T Corp. System and method for using meta-data dependent language modeling for automatic speech recognition
US7634095B2 (en) 2004-02-23 2009-12-15 General Motors Company Dynamic tuning of hands-free algorithm for noise and driving conditions
US7400878B2 (en) 2004-02-26 2008-07-15 Research In Motion Limited Computing device with environment aware features
US7696136B2 (en) * 2004-03-11 2010-04-13 Crompton Corporation Lubricant compositions containing hydroxy carboxylic acid and hydroxy polycarboxylic acid esters
US7624018B2 (en) * 2004-03-12 2009-11-24 Microsoft Corporation Speech recognition using categories and speech prefixing
US20050246325A1 (en) 2004-04-30 2005-11-03 Microsoft Corporation Method and system for recording and accessing usage of an item in a computer system
US20090018829A1 (en) 2004-06-08 2009-01-15 Metaphor Solutions, Inc. Speech Recognition Dialog Management
JP3923513B2 (en) 2004-06-08 2007-06-06 松下電器産業株式会社 Speech recognition apparatus and speech recognition method
US7848927B2 (en) 2004-11-30 2010-12-07 Panasonic Corporation Speech recognition device and method of recognizing speech using a language model
US7409344B2 (en) * 2005-03-08 2008-08-05 Sap Aktiengesellschaft XML based architecture for controlling user interfaces with contextual voice commands
US7739286B2 (en) 2005-03-17 2010-06-15 University Of Southern California Topic specific language models built from large numbers of documents
US8009678B2 (en) 2005-03-17 2011-08-30 Microsoft Corporation System and method for generating a dynamic prioritized contact list
US20060277466A1 (en) * 2005-05-13 2006-12-07 Anderson Thomas G Bimodal user interaction with a simulated object
EP1750253B1 (en) 2005-08-04 2012-03-21 Nuance Communications, Inc. Speech dialog system
US7895193B2 (en) * 2005-09-30 2011-02-22 Microsoft Corporation Arbitration of specialized content using search results
US8620667B2 (en) * 2005-10-17 2013-12-31 Microsoft Corporation Flexible speech-activated command and control
US20070106685A1 (en) * 2005-11-09 2007-05-10 Podzinger Corp. Method and apparatus for updating speech recognition databases and reindexing audio and video content using the same
ES2330758T3 (en) * 2005-11-25 2009-12-15 Swisscom Ag PROCEDURE FOR CUSTOMIZING A SERVICE.
JP4961755B2 (en) 2006-01-23 2012-06-27 富士ゼロックス株式会社 Word alignment device, word alignment method, word alignment program
US7818279B2 (en) 2006-03-13 2010-10-19 Microsoft Corporation Event detection based on evolution of click-through data
US7818315B2 (en) 2006-03-13 2010-10-19 Microsoft Corporation Re-ranking search results based on query log
US8301448B2 (en) 2006-03-29 2012-10-30 Nuance Communications, Inc. System and method for applying dynamic contextual grammars and language models to improve automatic speech recognition accuracy
US20080005067A1 (en) 2006-06-28 2008-01-03 Microsoft Corporation Context-based search, retrieval, and awareness
US8001130B2 (en) 2006-07-25 2011-08-16 Microsoft Corporation Web object retrieval based on a language model
JP4957110B2 (en) 2006-08-03 2012-06-20 日亜化学工業株式会社 Light emitting device
US8564544B2 (en) * 2006-09-06 2013-10-22 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
US7827033B2 (en) 2006-12-06 2010-11-02 Nuance Communications, Inc. Enabling grammars in web page frames
US7856351B2 (en) * 2007-01-19 2010-12-21 Microsoft Corporation Integrated speech recognition and semantic classification
WO2008097194A1 (en) 2007-02-09 2008-08-14 Agency For Science, Technology And Research Keyword classification and determination in language modelling
US20080215541A1 (en) * 2007-03-01 2008-09-04 Microsoft Corporation Techniques for searching web forums
US8880405B2 (en) * 2007-03-07 2014-11-04 Vlingo Corporation Application text entry in a mobile environment using a speech processing facility
US8949266B2 (en) * 2007-03-07 2015-02-03 Vlingo Corporation Multiple web-based content category searching in mobile search application
US20080221900A1 (en) 2007-03-07 2008-09-11 Cerra Joseph P Mobile local search environment speech processing facility
US20110060587A1 (en) * 2007-03-07 2011-03-10 Phillips Michael S Command and control utilizing ancillary information in a mobile voice-to-speech application
US8949130B2 (en) * 2007-03-07 2015-02-03 Vlingo Corporation Internal and external speech recognition use with a mobile communication facility
US7584294B2 (en) 2007-03-12 2009-09-01 Citrix Systems, Inc. Systems and methods for prefetching objects for caching using QOS
US7945851B2 (en) * 2007-03-14 2011-05-17 Nuance Communications, Inc. Enabling dynamic voiceXML in an X+V page of a multimodal application
US20080228903A1 (en) * 2007-03-16 2008-09-18 Yahoo! Inc. System and method of serving advertisements for web applications
US8670987B2 (en) * 2007-03-20 2014-03-11 Nuance Communications, Inc. Automatic speech recognition with dynamic grammar rules
US8060373B2 (en) 2007-03-21 2011-11-15 At&T Intellectual Property I, L.P. System and method of identifying contact information
CN101286094A (en) * 2007-04-10 2008-10-15 谷歌股份有限公司 Multi-mode input method editor
US7895177B2 (en) 2007-05-29 2011-02-22 Yahoo! Inc. Enabling searching of user ratings and reviews using user profile location, and social networks
CN102124459B (en) 2007-06-14 2013-06-12 谷歌股份有限公司 Dictionary word and phrase determination
US7831427B2 (en) 2007-06-20 2010-11-09 Microsoft Corporation Concept monitoring in spoken-word audio
US8275615B2 (en) 2007-07-13 2012-09-25 International Business Machines Corporation Model weighting, selection and hypotheses combination for automatic speech recognition and machine translation
CA2694327A1 (en) 2007-08-01 2009-02-05 Ginger Software, Inc. Automatic context sensitive language correction and enhancement using an internet corpus
US8321219B2 (en) 2007-10-05 2012-11-27 Sensory, Inc. Systems and methods of performing speech recognition using gestures
TWI399966B (en) 2007-12-31 2013-06-21 Htc Corp The mobile phone and the dialing method thereof
US8473276B2 (en) * 2008-02-19 2013-06-25 Google Inc. Universal language input
US8065143B2 (en) * 2008-02-22 2011-11-22 Apple Inc. Providing text input using speech data and non-speech data
EP2101250B1 (en) 2008-03-14 2014-06-11 BlackBerry Limited Character selection on a device using offset contact-zone
US8224656B2 (en) * 2008-03-14 2012-07-17 Microsoft Corporation Speech recognition disambiguation on mobile devices
US8831950B2 (en) 2008-04-07 2014-09-09 Nuance Communications, Inc. Automated voice enablement of a web page
US8121837B2 (en) 2008-04-24 2012-02-21 Nuance Communications, Inc. Adjusting a speech engine for a mobile computing device based on background noise
TWI352970B (en) 2008-04-30 2011-11-21 Delta Electronics Inc Voice input system and voice input method
US8090738B2 (en) 2008-05-14 2012-01-03 Microsoft Corporation Multi-modal search wildcards
US8589161B2 (en) 2008-05-27 2013-11-19 Voicebox Technologies, Inc. System and method for an integrated, multi-modal, multi-device natural language voice services environment
US20100030549A1 (en) * 2008-07-31 2010-02-04 Lee Michael M Mobile device having human language translation capability with positional feedback
US8027973B2 (en) 2008-08-04 2011-09-27 Microsoft Corporation Searching questions based on topic and focus
US8385971B2 (en) * 2008-08-19 2013-02-26 Digimarc Corporation Methods and systems for content processing
US20110161072A1 (en) 2008-08-20 2011-06-30 Nec Corporation Language model creation apparatus, language model creation method, speech recognition apparatus, speech recognition method, and recording medium
US20100045611A1 (en) * 2008-08-21 2010-02-25 Microsoft Corporation Touch screen mobile device as graphics tablet input
US8775154B2 (en) * 2008-09-18 2014-07-08 Xerox Corporation Query translation through dictionary adaptation
US8326785B2 (en) * 2008-09-30 2012-12-04 Microsoft Corporation Joint ranking model for multilingual web search
US9798720B2 (en) 2008-10-24 2017-10-24 Ebay Inc. Hybrid machine translation
WO2010061507A1 (en) 2008-11-28 2010-06-03 日本電気株式会社 Language model creation device
US9330165B2 (en) 2009-02-13 2016-05-03 Microsoft Technology Licensing, Llc Context-aware query suggestion by mining log data
US8509398B2 (en) 2009-04-02 2013-08-13 Microsoft Corporation Voice scratchpad
US20100257171A1 (en) 2009-04-03 2010-10-07 Yahoo! Inc. Techniques for categorizing search queries
US20100318531A1 (en) 2009-06-10 2010-12-16 Microsoft Corporation Smoothing clickthrough data for web search ranking
US20100315266A1 (en) 2009-06-15 2010-12-16 Microsoft Corporation Predictive interfaces with usability constraints
US8589163B2 (en) * 2009-12-04 2013-11-19 At&T Intellectual Property I, L.P. Adapting language models with a bit mask for a subset of related words
US11416214B2 (en) * 2009-12-23 2022-08-16 Google Llc Multi-modal input on an electronic device
US8782556B2 (en) * 2010-02-12 2014-07-15 Microsoft Corporation User-centric soft keyboard predictive technologies
KR101477530B1 (en) * 2010-03-12 2014-12-30 뉘앙스 커뮤니케이션즈, 인코포레이티드 Multimodal text input system, such as for use with touch screens on mobile phones
US8694304B2 (en) 2010-03-26 2014-04-08 Virtuoz Sa Semantic clustering and user interfaces
US8428759B2 (en) * 2010-03-26 2013-04-23 Google Inc. Predictive pre-recording of audio for voice input
US8265928B2 (en) 2010-04-14 2012-09-11 Google Inc. Geotagged environmental audio for enhanced speech recognition accuracy
US8694313B2 (en) 2010-05-19 2014-04-08 Google Inc. Disambiguation of contact information using historical data
US8468012B2 (en) 2010-05-26 2013-06-18 Google Inc. Acoustic model adaptation using geographic information
US8359311B2 (en) * 2010-06-01 2013-01-22 Microsoft Corporation Federated implicit search
US8813060B2 (en) * 2011-06-17 2014-08-19 Microsoft Corporation Context aware application model for connected devices
US8847775B2 (en) 2012-11-30 2014-09-30 Panasonic Corporation Tangible charge level awareness method and apparatus using augmented batteries
US9075846B2 (en) * 2012-12-12 2015-07-07 King Fahd University Of Petroleum And Minerals Method for retrieval of arabic historical manuscripts
US20140173440A1 (en) 2012-12-13 2014-06-19 Imimtek, Inc. Systems and methods for natural interaction with operating systems and application graphical user interfaces using gestural and vocal input
US9626960B2 (en) * 2013-04-25 2017-04-18 Nuance Communications, Inc. Systems and methods for providing metadata-dependent language models

Patent Citations (102)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4820059A (en) * 1985-10-30 1989-04-11 Central Institute For The Deaf Speech processing apparatus and methods
US5768603A (en) * 1991-07-25 1998-06-16 International Business Machines Corporation Method and system for natural language translation
US5632002A (en) * 1992-12-28 1997-05-20 Kabushiki Kaisha Toshiba Speech recognition interface system suitable for window systems and speech mail systems
US5737724A (en) * 1993-11-24 1998-04-07 Lucent Technologies Inc. Speech recognition employing a permissive recognition criterion for a repeated phrase utterance
US5638487A (en) * 1994-12-30 1997-06-10 Purespeech, Inc. Automatic speech recognition
US5715367A (en) * 1995-01-23 1998-02-03 Dragon Systems, Inc. Apparatuses and methods for developing and using models for speech recognition
US6839670B1 (en) * 1995-09-11 2005-01-04 Harman Becker Automotive Systems Gmbh Process for automatic control of one or more devices by voice commands or by real-time voice dialog and apparatus for carrying out this process
US6397180B1 (en) * 1996-05-22 2002-05-28 Qwest Communications International Inc. Method and system for performing speech recognition based on best-word scoring of repeated speech attempts
US6021403A (en) * 1996-07-19 2000-02-01 Microsoft Corporation Intelligent user assistance facility
US6182038B1 (en) * 1997-12-01 2001-01-30 Motorola, Inc. Context dependent phoneme networks for encoding speech information
US6418431B1 (en) * 1998-03-30 2002-07-09 Microsoft Corporation Information retrieval and speech recognition based on language models
US6356866B1 (en) * 1998-10-07 2002-03-12 Microsoft Corporation Method for converting a phonetic character string into the text of an Asian language
US7881936B2 (en) * 1998-12-04 2011-02-01 Tegic Communications, Inc. Multimodal disambiguation of speech recognition
US6912499B1 (en) * 1999-08-31 2005-06-28 Nortel Networks Limited Method and apparatus for training a multilingual speech model set
US20050005240A1 (en) * 1999-10-05 2005-01-06 Microsoft Corporation Method and system for providing alternatives for text derived from stochastic input sources
US6581033B1 (en) * 1999-10-19 2003-06-17 Microsoft Corporation System and method for correction of speech recognition mode errors
US20020111990A1 (en) * 1999-11-01 2002-08-15 Wood Christopher Noah Internet based message management system
US7203731B1 (en) * 2000-03-03 2007-04-10 Intel Corporation Dynamic replication of files in a network storage system
US20040024583A1 (en) * 2000-03-20 2004-02-05 Freeman Robert J Natural-language processing system using a large corpus
US7107204B1 (en) * 2000-04-24 2006-09-12 Microsoft Corporation Computer-aided writing system and method with cross-language writing wizard
US6678415B1 (en) * 2000-05-12 2004-01-13 Xerox Corporation Document image decoding using an integrated stochastic language model
US6539358B1 (en) * 2000-05-24 2003-03-25 Delphi Technologies, Inc. Voice-interactive docking station for a portable computing device
US20100049502A1 (en) * 2000-07-24 2010-02-25 Microsoft Corporation Method and system of generating reference variations for directory assistance data
US7043422B2 (en) * 2000-10-13 2006-05-09 Microsoft Corporation Method and apparatus for distribution-based language model adaptation
US20020087314A1 (en) * 2000-11-14 2002-07-04 International Business Machines Corporation Method and apparatus for phonetic context adaptation for improved speech recognition
US20020099818A1 (en) * 2000-11-16 2002-07-25 Russell Ethan George Method and system for monitoring the performance of a distributed application
US20020062216A1 (en) * 2000-11-23 2002-05-23 International Business Machines Corporation Method and system for gathering information by voice input
US20080027723A1 (en) * 2000-11-30 2008-01-31 Verizon Services Organization, Inc. Methods and apparatus for performing speech recognition over a network and using speech recognition results
US6985147B2 (en) * 2000-12-15 2006-01-10 International Business Machines Corporation Information access method, system and storage medium
US20020087309A1 (en) * 2000-12-29 2002-07-04 Lee Victor Wai Leung Computer-implemented speech expectation-based probability method and system
US7366668B1 (en) * 2001-02-07 2008-04-29 Google Inc. Voice interface for a search engine
US7027987B1 (en) * 2001-02-07 2006-04-11 Google Inc. Voice interface for a search engine
US7203645B2 (en) * 2001-04-27 2007-04-10 Intel Corporation Speech recognition system loading different recognition engines for different applications
US6714778B2 (en) * 2001-05-15 2004-03-30 Nokia Corporation Context sensitive web services
US7526431B2 (en) * 2001-09-05 2009-04-28 Voice Signal Technologies, Inc. Speech recognition using ambiguous or phone key spelling and/or filtering
US20040049388A1 (en) * 2001-09-05 2004-03-11 Roth Daniel L. Methods, systems, and programming for performing speech recognition
US20030050778A1 (en) * 2001-09-13 2003-03-13 Patrick Nguyen Focused language models for improved speech input of structured documents
US7174288B2 (en) * 2002-05-08 2007-02-06 Microsoft Corporation Multi-modal entry of ideogrammatic languages
US20040013882A1 (en) * 2002-07-10 2004-01-22 Rudiger Gorny Multi-layer product containing polycarbonate
US20040043758A1 (en) * 2002-08-29 2004-03-04 Nokia Corporation System and method for providing context sensitive recommendations to digital services
US20040098571A1 (en) * 2002-11-15 2004-05-20 Falcon Stephen R. Portable computing device-integrated appliance
US20090292529A1 (en) * 2002-12-18 2009-11-26 At&T Corp. System and method of providing a spoken dialog interface to a website
US7698136B1 (en) * 2003-01-28 2010-04-13 Voxify, Inc. Methods and apparatus for flexible speech recognition
US7383553B2 (en) * 2003-05-15 2008-06-03 International Business Machines Corporation Accessing a platform independent input method editor from an underlying operating system
US7392188B2 (en) * 2003-07-31 2008-06-24 Telefonaktiebolaget Lm Ericsson (Publ) System and method enabling acoustic barge-in
US20050091037A1 (en) * 2003-10-24 2005-04-28 Microsoft Corporation System and method for providing context to an input method
US7370275B2 (en) * 2003-10-24 2008-05-06 Microsoft Corporation System and method for providing context to an input method by tagging existing applications
US20050108017A1 (en) * 2003-10-27 2005-05-19 John-Alexander Esser Determining language for word recognition event
US20050114474A1 (en) * 2003-11-20 2005-05-26 International Business Machines Corporation Automatic configuration of the network devices via connection to specific switch ports
US20060004572A1 (en) * 2004-06-30 2006-01-05 Microsoft Corporation Homonym processing in the context of voice-activated command systems
US20060004850A1 (en) * 2004-07-01 2006-01-05 Chowdhury Abdur R Analyzing a query log for use in managing category-specific electronic content
US20060009974A1 (en) * 2004-07-09 2006-01-12 Matsushita Electric Industrial Co., Ltd. Hands-free voice dialing for portable and remote devices
US20060035632A1 (en) * 2004-08-16 2006-02-16 Antti Sorvari Apparatus and method for facilitating contact selection in communication devices
US20060048055A1 (en) * 2004-08-25 2006-03-02 Jun Wu Fault-tolerant romanized input method for non-roman characters
US7505894B2 (en) * 2004-11-04 2009-03-17 Microsoft Corporation Order model for dependency structure
US20060111892A1 (en) * 2004-11-04 2006-05-25 Microsoft Corporation Extracting treelet translation pairs
US20060111891A1 (en) * 2004-11-04 2006-05-25 Microsoft Corporation Order model for dependency structure
US7698124B2 (en) * 2004-11-04 2010-04-13 Microsoft Corporaiton Machine translation system incorporating syntactic dependency treelets into a statistical framework
US20060095248A1 (en) * 2004-11-04 2006-05-04 Microsoft Corporation Machine translation system incorporating syntactic dependency treelets into a statistical framework
US20060111896A1 (en) * 2004-11-04 2006-05-25 Microsoft Corporation Projecting dependencies to generate target language dependency structure
US7200550B2 (en) * 2004-11-04 2007-04-03 Microsoft Corporation Projecting dependencies to generate target language dependency structure
US20060111907A1 (en) * 2004-11-24 2006-05-25 Microsoft Corporation Generic spelling mnemonics
US20070060114A1 (en) * 2005-09-14 2007-03-15 Jorey Ramer Predictive text completion for a mobile communication facility
US7672833B2 (en) * 2005-09-22 2010-03-02 Fair Isaac Corporation Method and apparatus for automatic entity disambiguation
US20070124507A1 (en) * 2005-11-28 2007-05-31 Sap Ag Systems and methods of processing annotations and multimodal user inputs
US20100042400A1 (en) * 2005-12-21 2010-02-18 Hans-Ulrich Block Method for Triggering at Least One First and Second Background Application via a Universal Language Dialog System
US20110077943A1 (en) * 2006-06-26 2011-03-31 Nec Corporation System for generating language model, method of generating language model, and program for language model generation
US7907705B1 (en) * 2006-10-10 2011-03-15 Intuit Inc. Speech to text for assisted form completion
US20080091443A1 (en) * 2006-10-13 2008-04-17 Brian Strope Business listing search
US20080091435A1 (en) * 2006-10-13 2008-04-17 Brian Strope Business listing search
US7890326B2 (en) * 2006-10-13 2011-02-15 Google Inc. Business listing search
US20080091406A1 (en) * 2006-10-16 2008-04-17 Voicebox Technologies, Inc. System and method for a cooperative conversational voice user interface
US20080133228A1 (en) * 2006-11-30 2008-06-05 Rao Ashwin P Multimodal speech recognition system
US20080131851A1 (en) * 2006-12-04 2008-06-05 Dimitri Kanevsky Context-sensitive language learning
US20080155111A1 (en) * 2006-12-20 2008-06-26 Hitachi, Ltd. Delivery system, communication apparatus and delivery method
US7941189B2 (en) * 2007-02-07 2011-05-10 Denso Corporation Communicating road noise control system, in-vehicle road noise controller, and server
US20090030687A1 (en) * 2007-03-07 2009-01-29 Cerra Joseph P Adapting an unstructured language model speech recognition system based on usage
US20090030696A1 (en) * 2007-03-07 2009-01-29 Cerra Joseph P Using results of unstructured language model based speech recognition to control a system-level function of a mobile communications facility
US20100111059A1 (en) * 2007-04-02 2010-05-06 Benjamin Bappu Content delivery
US7921107B2 (en) * 2007-04-10 2011-04-05 Yahoo! Inc. System for generating query suggestions using a network of users and advertisers
US20090055162A1 (en) * 2007-08-20 2009-02-26 Microsoft Corporation Hmm-based bilingual (mandarin-english) tts techniques
US20090063455A1 (en) * 2007-08-30 2009-03-05 Microsoft Corporation Bipartite Graph Reinforcement Modeling to Annotate Web Images
US7877385B2 (en) * 2007-09-21 2011-01-25 Microsoft Corporation Information retrieval using query-document pair information
US7953692B2 (en) * 2007-12-07 2011-05-31 Microsoft Corporation Predicting candidates using information sources
US20090164216A1 (en) * 2007-12-21 2009-06-25 General Motors Corporation In-vehicle circumstantial speech recognition
US20100004930A1 (en) * 2008-07-02 2010-01-07 Brian Strope Speech Recognition with Parallel Recognition Tasks
US20100057687A1 (en) * 2008-09-04 2010-03-04 Microsoft Corporation Predicting future queries from log data
US20100088303A1 (en) * 2008-10-03 2010-04-08 Microsoft Corporation Mining new words from a query log for input method editors
US20100100377A1 (en) * 2008-10-10 2010-04-22 Shreedhar Madhavapeddi Generating and processing forms for receiving speech data
US20100153219A1 (en) * 2008-12-12 2010-06-17 Microsoft Corporation In-text embedded advertising
US20100153370A1 (en) * 2008-12-15 2010-06-17 Microsoft Corporation System of ranking search results based on query specific position bias
US8156129B2 (en) * 2009-01-15 2012-04-10 Microsoft Corporation Substantially similar queries
US20110004462A1 (en) * 2009-07-01 2011-01-06 Comcast Interactive Media, Llc Generating Topic-Specific Language Models
US20110066577A1 (en) * 2009-09-15 2011-03-17 Microsoft Corporation Machine Learning Using Relational Databases
US20110093265A1 (en) * 2009-10-16 2011-04-21 Amanda Stent Systems and Methods for Creating and Using Geo-Centric Language Models
US20110137635A1 (en) * 2009-12-08 2011-06-09 Microsoft Corporation Transliterating semitic languages including diacritics
US20120022867A1 (en) * 2009-12-23 2012-01-26 Ballinger Brandon M Speech to Text Conversion
US20110162035A1 (en) * 2009-12-31 2011-06-30 Apple Inc. Location-based dock for a computing device
US20120022868A1 (en) * 2010-01-05 2012-01-26 Google Inc. Word-Level Correction of Speech Input
US20120060113A1 (en) * 2010-09-08 2012-03-08 Nuance Communications, Inc. Methods and apparatus for displaying content
US20120143611A1 (en) * 2010-12-07 2012-06-07 Microsoft Corporation Trajectory Tiling Approach for Text-to-Speech
US20120191745A1 (en) * 2011-01-24 2012-07-26 Yahoo!, Inc. Synthesized Suggestions for Web-Search Queries

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Lin Li, Query-URL Bipartite Based Approach to Personalized Query Recommendation, 2008, Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008) *
Oren Kurland, Re-ranking search results using language models of query-specific clusters, ACM, Volume 12 Issue 4, August 2009, Pages 437 - 460 *

Cited By (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10713010B2 (en) 2009-12-23 2020-07-14 Google Llc Multi-modal input on an electronic device
US9031830B2 (en) 2009-12-23 2015-05-12 Google Inc. Multi-modal input on an electronic device
US9251791B2 (en) 2009-12-23 2016-02-02 Google Inc. Multi-modal input on an electronic device
US10157040B2 (en) 2009-12-23 2018-12-18 Google Llc Multi-modal input on an electronic device
US20110153324A1 (en) * 2009-12-23 2011-06-23 Google Inc. Language Model Selection for Speech-to-Text Conversion
US11914925B2 (en) 2009-12-23 2024-02-27 Google Llc Multi-modal input on an electronic device
US20110161080A1 (en) * 2009-12-23 2011-06-30 Google Inc. Speech to Text Conversion
US8751217B2 (en) 2009-12-23 2014-06-10 Google Inc. Multi-modal input on an electronic device
US11416214B2 (en) 2009-12-23 2022-08-16 Google Llc Multi-modal input on an electronic device
US9495127B2 (en) 2009-12-23 2016-11-15 Google Inc. Language model selection for speech-to-text conversion
US9047870B2 (en) 2009-12-23 2015-06-02 Google Inc. Context based language model selection
US8666963B2 (en) * 2010-09-08 2014-03-04 Nuance Communications, Inc. Method and apparatus for processing spoken search queries
US20120259636A1 (en) * 2010-09-08 2012-10-11 Nuance Communications, Inc. Method and apparatus for processing spoken search queries
US8239366B2 (en) * 2010-09-08 2012-08-07 Nuance Communications, Inc. Method and apparatus for processing spoken search queries
US20120059810A1 (en) * 2010-09-08 2012-03-08 Nuance Communications, Inc. Method and apparatus for processing spoken search queries
US10332514B2 (en) * 2011-08-29 2019-06-25 Microsoft Technology Licensing, Llc Using multiple modality input to feedback context for natural language understanding
US9576573B2 (en) * 2011-08-29 2017-02-21 Microsoft Technology Licensing, Llc Using multiple modality input to feedback context for natural language understanding
US20170169824A1 (en) * 2011-08-29 2017-06-15 Microsoft Technology Licensing, Llc Using multiple modality input to feedback context for natural language understanding
US20130054238A1 (en) * 2011-08-29 2013-02-28 Microsoft Corporation Using Multiple Modality Input to Feedback Context for Natural Language Understanding
US11264023B2 (en) * 2011-08-29 2022-03-01 Microsoft Technology Licensing, Llc Using multiple modality input to feedback context for natural language understanding
US20130173265A1 (en) * 2012-01-03 2013-07-04 Chiaka Chukwuma Okoroh Speech-to-online-text system
US20150121290A1 (en) * 2012-06-29 2015-04-30 Microsoft Corporation Semantic Lexicon-Based Input Method Editor
US9959340B2 (en) * 2012-06-29 2018-05-01 Microsoft Technology Licensing, Llc Semantic lexicon-based input method editor
US9747895B1 (en) 2012-07-10 2017-08-29 Google Inc. Building language models for a user in a social network from linguistic information
US9063636B2 (en) 2013-06-10 2015-06-23 International Business Machines Corporation Management of input methods
US9069446B2 (en) 2013-06-10 2015-06-30 International Business Machines Corporation Management of input methods
US9502032B2 (en) 2014-10-08 2016-11-22 Google Inc. Dynamically biasing language models
US10553214B2 (en) 2016-03-16 2020-02-04 Google Llc Determining dialog states for language models
US9978367B2 (en) 2016-03-16 2018-05-22 Google Llc Determining dialog states for language models
US11037551B2 (en) 2017-02-14 2021-06-15 Google Llc Language model biasing system
US10311860B2 (en) 2017-02-14 2019-06-04 Google Llc Language model biasing system
US11682383B2 (en) 2017-02-14 2023-06-20 Google Llc Language model biasing system
US20200404362A1 (en) * 2018-02-20 2020-12-24 Lg Electronics Inc. Display device
US11706482B2 (en) * 2018-02-20 2023-07-18 Lg Electronics Inc. Display device

Also Published As

Publication number Publication date
US9495127B2 (en) 2016-11-15
US20120022873A1 (en) 2012-01-26
US20120022867A1 (en) 2012-01-26
US9047870B2 (en) 2015-06-02
US20160132293A1 (en) 2016-05-12
US20220405046A1 (en) 2022-12-22
US20190056909A1 (en) 2019-02-21
US10157040B2 (en) 2018-12-18
US11914925B2 (en) 2024-02-27
EP2339576A2 (en) 2011-06-29
EP4318463A3 (en) 2024-02-28
EP3091535A2 (en) 2016-11-09
US10713010B2 (en) 2020-07-14
US9031830B2 (en) 2015-05-12
US8751217B2 (en) 2014-06-10
US20120022866A1 (en) 2012-01-26
US9251791B2 (en) 2016-02-02
EP2339576A3 (en) 2011-11-23
US20110161080A1 (en) 2011-06-30
EP3091535A3 (en) 2017-03-01
US20140288929A1 (en) 2014-09-25
EP2339576B1 (en) 2019-08-07
US20110153324A1 (en) 2011-06-23
US20120022853A1 (en) 2012-01-26
US20110153325A1 (en) 2011-06-23
EP3091535B1 (en) 2023-10-11
EP4318463A2 (en) 2024-02-07

Similar Documents

Publication Publication Date Title
US11914925B2 (en) Multi-modal input on an electronic device
US11416214B2 (en) Multi-modal input on an electronic device
US10395654B2 (en) Text normalization based on a data-driven learning network
CN111710333B (en) Method and system for generating speech transcription
US10181322B2 (en) Multi-user, multi-domain dialog system
KR102390940B1 (en) Context biasing for speech recognition
US20170154623A1 (en) Pronunciation learning through correction logs
US8380512B2 (en) Navigation using a search engine and phonetic voice recognition
US10643603B2 (en) Acoustic model training using corrected terms
US11016968B1 (en) Mutation architecture for contextual data aggregator
US11170765B2 (en) Contextual multi-channel speech to text
JP7400112B2 (en) Biasing alphanumeric strings for automatic speech recognition
CN112262382A (en) Annotation and retrieval of contextual deep bookmarks
CN1965349A (en) Multimodal disambiguation of speech recognition
AT&T
TWM453219U (en) Semantic identification device

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044142/0357

Effective date: 20170929