WO1998020405A2

WO1998020405A2 - Method and system for screen text capturing

Info

Publication number: WO1998020405A2
Application number: PCT/US1997/020118
Authority: WO
Inventors: Ambin Cen; Yueping Wei
Original assignee: U.C. Media, Inc.
Priority date: 1996-11-06
Filing date: 1997-11-06
Publication date: 1998-05-14
Also published as: AU5197498A; AU5167598A; WO1998020406A2

Abstract

A method and system is provided, in standalone and client-server versions, for capturing text on a computer screen, regardless of the application or graphical user interface in which the user is operating, and providing information about the text from a related database.

Description

Method and System for Screen Text Capturing

TECHNICAL FIELD

The present invention relates to the field of computer software and more particularly to the field of screen text capture in a client-server environment.

BACKGROUND OF THE INVENTION

In many situations it is desirable to obtain additional real-time information about text that appears on a computer screen. Such information can be of a wide variety of forms. For example, a user might wish to obtain a dictionary definition for the text that appears on the screen, a list of synonyms for the text, or a translation of the text into a foreign language. Computer software systems that provide information about screen text are well-known and include spelling programs, grammar checking programs, and translation programs. Prior art computer software programs for providing information about screen text have the significant limitation that they are application-specific; that is, the code for providing further information about screen text is part of the code for a specific application; thus, only text converted to that application can be operated on by the programs. Most computer text is not formatted to permit the use of these application specific programs; therefore, important information about screen text is inaccessible to the user. Also known in the art are "flying tips." A flying tip is a feature built into a specific application program, e.g., Microsoft Word, that permits the user to obtain additional information about text at particular screen locations. Flying tips operate only at a limited set of points on the user's screen, rather than at every point on the user's screen and at every point in a user's documents.

A wide variety of databases have been developed that include valuable information about particular words, including same-language and foreign language dictionaries, "help" files, encyclopedias, thesaurases, and the like; however, current technology does not permit the user to access all of these databases, because current technology only permits access to databases that relate to specific applications and specific graphic user interfaces. Thus, opportunities for improved understanding of screen text are foregone.

Conventional programs and devices that accomplish screen capture, such as optical character recognition devices, have the further limitation that the functionality is only available to users who obtain and install the screen capture program on their computer. Thus, screen capture capability is not available to a large number of users on a cost-effective basis, even though information providers may wish for such users to have screen capture capability. BRIEF SUMMARY OF THE INVENTION

The present invention is a computer program that permits the capture of computer screen text regardless of the format of the text, the application in which the text is generated, or the graphical user interface in which the user is operating. The computer program may be used as a stand-alone program, or it can be inserted into another program in order to permit the other program to accomplish the screen capturing function. The present invention works in the background; that is, it requires minimal user interaction. The program is driven by the user's mouse cursor movement. If the cursor is pointed to any set of text characters for more than a pre- specified amount of time, the program captures the resulting text. The captured result is then fed as an index to another program for a look-up search in a database that contains information about text characters. Any matching entries in the database are instantly displayed on the screen. Clicking the mouse permits the user to cycle through multiple matching entries, if more than one entry exists for a particular text fragment.

A further object of the present invention is to provide the above-described functionality in a client-server environment. In the present invention, the screen capturing computer software is ported to any network applications, including internet web or other client-server applications. When the user loads a page from a server, the capturing engine can be downloaded with the page. The capturing engine then interacts with the server engine to capture and deliver information about the text and characters to which the user's cursor is pointed, such as translation, visual images, or the like. Thus, the functionality of the stand-alone version may be provided in connection with web applications, without requiring the user to obtain application specific software. The client-server functionality of the program permits multiple users to have screen-capture capability on a cost-effective basis; thus, information providers can ensure that users have access to full information about the information content of their web sites or other computer applications, without requiring users to obtain and install their own application programs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart that describes the operation of the screen-capturing and retrieval functions of the present invention.

FIG. 2 is a flow chart that describes the server-based version of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention may be further understood by reference to the attached figures. Referring to FIG. 1, the screen-capturing function of the present invention is further described.

The present invention resides in the background on a host computer, and it operates in the background of any other application program, such as a conventional word processing program. Thus, the computer user may initiate and operate another word processing program without any interference from the present invention. In addition, the present invention can apply to any graphical user interface environments, e.g., Macintosh, UNIX or Microsoft Windows. In Upon initiation, the functionality of the present invention is automatically initiated in the background and is available to the user. Upon initiation, a mouse hook function is initiated in the present invention. The mouse hook determines whether the user's cursor is located on text on the computer screen, or whether text is highlighted on the screen via the cursor. If the cursor is not located on text, the program resides in the background until the cursor is moved onto text. When the cursor is moved onto text on the screen and remains there for a prespecified amount of time, preferably about one second, the program determines by conventional means whether the text at which the cursor is located relates to a system resource of the resident application program. For example, the text might be the word "File," a system resource under which the user can create new files, open existing files, close open files, and the like. If the cursor location relates to a system resource, then the present invention calls the system function, such as "GetWindowText( )" in MSWindows, to retrieve the text. If the cursor is not located on a system function, the present invention obtains the cartesian coordinates of the text on the screen.

Once the cartesian coordinates of the text are retrieved and delivered to the program, the present invention captures the screen text. Screen text capture may be by a variety of methods. The preferred method of the stand-alone version of the present invention is to trap the system "DrawText Messages" by sending the system's user-defined messages to repaint a portion of the window proximal to the captured text. Many systems, such as Microsoft Windows, for example, include system "DrawText" messages. In the Microsoft Windows system, these "DrawText" messages are referred to as "Textout( )" or "ExtTextOut( )" functions. The present invention can apply to any graphical user interface environment which is based upon a system messaging mechanism. Among other things, these functions include the text located on the screen at the identified coordinates. The present invention captures these system messages. Referring to FIG. 1, this is accomplished by modification of the system functions TextOut( ) and ExtTextOut( ). These system functions are modified so that the first instruction upon any initiation of these functions will be to jump to a new set of functions, NewTextOut( ), or NewExtTextOut( ). The new functions are programmed so that any arguments passed by the TextOut( ) and ExtTextOut( ) functions are copied to the new functions. Next, a conventional filtering method is used to filter out information other than the captured text from system messages NewTextOut( ) and NewExtTextOut( ). Other methods of text capture include optical character recognition. With this method the program calculates and isolates the image area at which the cursor is located. Next, optical character recognition techniques are used to recognize the text. Another method of capturing text requires the user to use the mouse click. The program determines whether the mouse has clicked on and highlighted text. If text is highlighted, the program then copies the text to a clipboard of the conventional type. The program then automatically retrieves the text from the clipboard.

Once the text is captured by one of these means, the database part of the program is initiated. The captured text serves as an index for the searching engine of any database program. Any type of database program can be used with the present invention; for example, the database program could be a translator, a cross-reference database, or a help file. The database search engine searches the database for an entry corresponding to the captured text. For example, if the database program is a foreign-language translator, the captured text is searched and an entry is retrieved that includes a foreign-language translation of the captured text. If the database program is a "help" file, the database program retrieves the help message in the database that relates to the captured text. In the preferred embodiment of the present invention, the inventor's own database of English and Chinese words is used to permit translation of screen text from one of those languages to the other.

Once an entry is retrieved from the database, the entry is displayed in the user's screen in the repainted window of the program. Thus, the user obtains information about text on the screen from a related database by merely pointing the mouse cursor to the text. In the embodiment of a foreign-language translator, the user can obtain rapid translation of text by pointing to the foreign-language text as the user reads. The present invention has the flexibility to operate with any database, thus, the user may translate text into any language or obtain any kind of information that relates to text, provided that a database is available to the user.

Referring to FIG. 2, a server-based version of the present invention is provided. The server-based version will operate in any client-server environment. The server-based version uses remote object technology, such as that provided by JAVA or Microsoft Active X. In the server version of the present invention, the program is ported to and embedded in an HTML server or any other server. The screen capture technology is built into a web page or other application as a JAVA Applet or as an object in ActiveX. The page and the embedded screen capture JAVA Applets or objects are then downloaded with the web page or application to the client- side machine. Once downloaded, the JAVA Applets or the objects provide the same functionality as the stand-alone version on the client's machine, without requiring the client-side user to obtain its own version of the program. Thus, for example, a web page can include its own foreign-language translation function for users who access the page.

When running on the client side, as in the stand-alone version, the program determines whether the cursor is located on text or words. If not, the program resides in the background and no functionality is apparent to the user. Once the cursor is located on text, the screen capture technology is initiated by one of a variety of methods. In the preferred method, the Applet or object running on the client machine reports to the server the cartesian coordinates of the cursor on the client-side user's screen, and the text from the screen can be extracted from the server based on the cartesian coordinates (x,y) of the text on the screen and the page of the text in the HTML file running on the server. In alternative embodiments, the server captures the text via one of the methods described above in connection with the stand-alone version of the present invention.

Once the text is captured by any of the above methods, the server then delivers the text to any database program, which uses the text as an index for the database engine. The server retrieves the related entry from the database search, as in the stand-alone version. After retrieving the related entry from the database, such as a foreign language translation, the server relays the result to the client machine as a remote object vehicle. The remote object vehicle repaints the client-side user's screen with a box that includes the information retrieved from the database.

In the alternative methods, wherein the screen capture function is accomplished in the server version by the same means as in the stand-alone version, the server version still reports the result to the client side machine via a remote object vehicle. The server version thus permits a user operating a client machine to obtain information from a database about screen text from a web page or application running on a server machine by merely pointing the mouse cursor to the screen text without requiring the client machine to run its own screen capture application. Conversely, the server version permits an information provider to include screen capture technology in its server-side applications, e.g., web pages, without requiring the user to obtain or install client-side applications.

While the invention has been disclosed in connection with the preferred embodiments shown and described in detail, various modifications and improvements thereon will become readily apparent to those skilled in the art.

Claims

What is claimed is:

1. A method of capturing text on a computer screen, comprising:

tracking movement of a cursor; if the cursor is pointed to the text for more than a pre-specified amount of time, capturing the resulting text; searching for the resulting text in a database; and displaying a database entry associated with the text on the computer screen.