WO1999017235A1

WO1999017235A1 - Method and apparatus for storing and retrieving labeled interval data for multimedia recordings

Info

Publication number: WO1999017235A1
Application number: PCT/US1998/020446
Authority: WO
Inventors: Christopher J. Macey; David M. Weimer; Pierre David Wellner
Original assignee: At & T Corp.
Priority date: 1997-10-01
Filing date: 1998-09-30
Publication date: 1999-04-08
Also published as: CA2271745A1; JP2001511991A

Abstract

A teleconference system (200) is disclosed for digitally recording and playing a conference telephone call that includes a plurality of intervals. The teleconference system includes a skim server (55) that detects a first set of the plurality of intervals and a conference bridge (100) that detects a second set of the plurality of intervals during the conference call. An interval database server (65) generates labeled interval data for all detected intervals and stores the labeled interval data in a database. The labeled interval data includes an interval data element that defines each interval. After the conference call is recorded, the labeled interval data can be searched and retrieved based on assorted criteria. Portions of the recorded conference call associated with the retrieved labeled interval data can also be retrieved and played back. This facilitates easy retrieval and playback of desired portions or a recorded conference call. Further, during playback of the conference call (85), a user interface is generated. The user interface displays the stored labeled interval data. A user can easily select or skip to desired portions of the conference call by selecting portions of the user interface.

Description

METHOD AND APPARATUS FOR STORING AND RETRIEVING LABELED INTERVAL DATA FOR MULTIMEDIA RECORDINGS

FIELD OF THE INVENTION

The present invention is directed to storage and retrieval of multimedia data. More particularly, the invention is directed to storage and retrieval of labeled interval data in a database.

BACKGROUND OF THE INVENTION

Unlike records of written communications, records of speech communication are rarely recorded, let alone stored, even though storage of digital speech may be readily achieved. It is presently feasible to store gigabytes and even terabytes of digitally recorded speech or other types of multimedia information (e.g., video). Other than for archival purposes, there is no practical reason for storing such data without having a mechanism by which a user can identify and retrieve only those portions of the stored data, which may be of interest.

The difficulty inherent in searching and retrieving digital speech records stored in a database stems from the traditional approaches to querying a database to locate particular records. Most database queries are logical queries based upon the presence or absence of specified characteristics in the records being searched. Boolean logic and fuzzy logic have been used to increase the utility of database queries, but these techniques merely extend the fundamental basis of most typical database queries, whether one or more terms, indices, or other identifying characteristics are present (or absent) in the records being searched.

Digital speech records, without being converted into text by speech- to-text conversion or transcription or otherwise parsed cannot be located and/or identified using traditional database query techniques as it is not practical to determine whether a word (or phrase) appears in a selected portion of recorded speech. Therefore, review of non-transcribed digital speech records is frequently limited to listening to the digitally recorded speech until the item or items of interest are heard. Unfortunately, this frequently requires listening to a considerable degree of extraneous or irrelevant speech which can be extremely time-consuming without providing any significant elucidation. Moreover, digital speech records frequently contain lengthy pauses and, if the digital speech record is between more than two speakers, it is frequently difficult, if not impossible, to identify the speakers, further exacerbating the problem of identifying a specific segment in recorded digital speech.

Even when a digital speech record is divided into separate digital recordings, and each recording is individually accessible and identified, the digitally recorded data is of limited use. For example, if ten conference calls were recorded in a digital storage medium, a user might be able to locate a particular conference call on a particular date, if the user were fortunate enough to know that the information he or she sought was in that specific conference call. Even, so, the user would still have to listen to the entire recording of the conference call. For a user seeking to identity a specific comment made by a specific participant to the conference call, it is extremely inefficient for the user to have to listen to the entire conference call. Moreover, if the user does not know the specific date and time of the conference call in which the person spoke, the user might have to listen to several conference call recordings before finding the desired information. Clearly, as soon as a greater than minimal number of recordings were stored, it becomes impractical for a user to locate desired information merely by listening to the conference call recordings.

Based on the foregoing, there is a need for a method and apparatus for readily identifying, locating, and retrieving stored digital speech and other digital multimedia records.

SUMMARY OF THE INVENTION

One embodiment of the present invention is a teleconference system for digitally recording and playing a conference telephone call that includes a plurality of intervals. The teleconference system includes a skim server that detects a first set of the plurality of intervals and a conference bridge that detects a second set of the plurality of intervals during the conference call. An interval database server generates labeled interval data for all detected intervals and stores the labeled interval data in a database. The labeled interval data includes an interval data element that defines each interval. After the conference call is recorded, the labeled interval data can be searched and retrieved based on assorted criteria. Portions of the recorded conference call associated with the retrieved labeled interval data can also be retrieved and played back. This facilitates easy retrieval and playback of desired portions of a recorded conference call. Further, during playback of the conference call, a user interface is generated. The user interface displays the stored labeled interval data. A user can easily select or skip to desired portions of the conference call by selecting portions of the user interface.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention.

Fig. 2 illustrates the format of an interval data element that forms the labeled interval data associated with a recorded conference.

Fig. 3 illustrates a conference playback document in accordance with one embodiment of the present invention.

Fig. 4 illustrates in detail how overlapping intervals are displayed.

DETAILED DESCRIPTION In one embodiment of the present invention, intervals within recorded digital speech or other multimedia data are specifically identified and labeled. The labeled interval data provides a mechanism by which a user can specifically identity an interval within digitally recorded multimedia, and having identified that interval, retrieve it and other intervals sharing desired characteristics.

Fig. 1 illustrates a teleconference system in accordance with one embodiment of the present invention. Teleconference system 200 records and stores a teleconference call and associated labeled interval data. Teleconference system 200 f rther allows a recorded teleconference to be played back using the stored labeled interval data. The main components of teleconference system 200 are a conference recorder 110, a skim server 55, an interval database ("IDB") server 65, and a Java user interface 85.

In teleconference system 200, a plurality of telephones 31, 32, and 33 are interconnected through the public switched telephone network

("PSTN") 40. One or more individuals may participate in a teleconference through each telephone 31-33. The participants may be identified by the telephone they are calling from or, alternatively, by voice recognition or other forms of identification during the teleconference. A teleconference may be initiated by a conference host accessing a

WebRoom interface on a WebRooms server 50. A WebRoom interface provides a mechanism by which participants may be actively added to and/or deleted from a teleconference. In one embodiment, the WebRoom interface for all teleconference participants is implemented as Common Gateway Interface ("CGI") program 60 on an HyperText Transport

Protocol Web Server ("Httpd") 70 that provides interactive control of the teleconference through Hyper-Text Markup Language ("HTML") documents. The HTML documents are accessible as conference pages 80 through a Web browser 90 such as Netscape^® Navigator or Internet Explorer^®.

At record time, the conference host uses WebRooms server 50 to dial a conference scribe. The conference scribe acts as an additional participant to the teleconference. At the same time, conference recorder 110 tells IDB Server 65 to create a new collection point, referred to as a "depot" for storing all data related to this particular recording, and it tells skim server 55 to begin recording an audio file using, for example, a Dialogic board 57 from Dialogic Corp., or its equivalent. A depot in teleconference system 200 can be a structured query language ("SQL") database 35 coupled to an Open DataBase Connectivity ("ODBC") interface 36. While the conference is running, conference bridges 100 detects call control events (e.g., which participant is talking, new participants being added, etc.) and sends these events through WebRooms server 50 and conference recorder 110 into the new depot (i.e., SQL database 35). Meanwhile, skim server 55 detects pauses in speech and adds these events as well to the depot. The events detected by both conference bridges 100 and skim server 65 are referred to as "intervals". When playing back a recorded conference on teleconference system

200, the user brings up a Java user interface 85 to select a recording accessed via IDB server 65. The user interface 85 retrieves labeled interval data for the recording and uses them to display a visual time-line of events. The user enters a phone number that is passed to Skim Server 55 so it can call the user's telephone for conference playback through

Dialogic board 57. As the audio plays on the user's phone, Java user interface 85 continuously updates the graphical display and controls how the recording is played using skim server 55. All clients like Java user interface 85 and conference recorder 110 communicate with skim server 55 and IDB server 65 through a CORBA application programming interface in one embodiment of the present invention. CORBA was chosen because it allows a simple interface between programs written in different languages running on different platforms. Both servers 50 and 55 and conference recorder 110 are written in C+ + and run on Sun Solaris platforms in one embodiment of the present invention.

Skim server 55 performs the following functions: 1. Records audio from telephone line to file. 2. Detects speech events while recording and posts them to the database.

3. Plays from file to telephone line

- from any point in recording - in variable speeds

- with pauses removed or not.

In one embodiment, skim server 55 is based on the same type of hardware as standard voice mail servers, and it performs many of the same functions. One difference between skim server 55 and a more traditional voice mail server is that it processes speech events and posts them to IDB server 65, and also that it provides fine control over what parts of the audio file are played and what parts are skipped.

One function of IDB server 65 is to store and retrieve labeled interval data associated with a recorded conference. This is data that describes properties about specific intervals within the speech, such as who is talking, pauses in speech, telephone call control data, etc. This can be further extended to applications that require intervals that mark video scene changes, or relate automatic speech recognition output to a recording. The labeled interval data can be created, stored, and retrieved by a number of different applications. Some are automatically derived from raw speech data, some are side effects of user activity, and others may be entered manually at record time or at playtime.

Fig. 2 illustrates the format of an interval data element 130 that forms the labeled interval data associated with a recorded conference. Every interval during the recorded conference will be associated with an interval data element 130. In one embodiment, each interval data element 130 includes the following: 1. Recording ID or Depot 122: Refers to the recording that is associated with the interval and the collection point where the recording is stored.

2. Start time 123: Applications need both absolute time and time relative to recording start time. Relative time is more compact, and it is easy to convert to absolute as long as an absolute start time is stored with the recording.

3. Duration or end time 124.

4. Type: A code to identify the meaning of this interval. Is it a pause in speech, a scene change, etc.?

5. Type-specific data values 126: Depending on the type, this data could be a string of text, a number, a URL, etc.

Labeled interval data must be able to be stored, retrieved, and manipulated more than one at a time. Some applications will deal with large collections of intervals that share everything except start time and end time (e.g., all times when a specific person was speaking).

Applications must be able to store interval data in the database at any time: before recording has begun, during recording, and after. For example, for a teleconference it may be necessary to record caller-id and ringing events before the call, record who is speaking during the call, and make annotations about the call afterwards. Some applications need to display incomplete interval data while a recording is in progress (e.g., catch up to live conference), so it should be possible to post an interval that has started but not ended yet, and post the end time later. It should also be possible to adjust interval data, for example to realign them with other data. All applications that post events to IDB server 65 must specify precise millisecond offsets for start and end times of each interval. All offsets are from an absolute start-time for the recording. Posting intervals from different machines in real-time requires all clients that are posting events have synchronized clocks, so standard network time protocol

("NTP") software is run on all of these machines.

Browse, search, and playback applications need to query and display subsets of interval data. Examples of queries that can be supported by the present invention include: • All interval data for a specific recording, sorted by time and type.

• All intervals of a specific type with specific values, or values within a particular range.

• Intervals within an absolute or relative time range. • Intervals of a specific duration.

The present invention provides for logical/set operations. For example, assume a user wants to see and/or hear only the parts of a recording when person A or person B was talking, and wants to leave all the pauses out. This can be expressed by making three queries: intervals when A was speaking (set A), intervals when B was speaking (set B), and pause intervals (set P). The desired set can be expressed as "A union B less P" , or if these sets are thought of as long bit masks, then they can be described as logical operations: (A B) & ( P).

Some types of intervals may not have clear start and end times. Instead of a binary on/off state at each time increment, some data has an associated probability curve over time because the exact times of the events are not certain. Output from automatic speech recognition (e.g., phoneme lattices) can include several overlapping hypotheses about what words are being said at any given moment. In one embodiment of the present invention, IDB server 65 provides support for "fuzzy" intervals. In another embodiment, IDB server 65 uses binary intervals along with a probability value in the type-specific numeric data field to achieve a similar effect as fuzzy intervals, but without fuzzy logical operations.

Transcriptions can be stored as interval data, perhaps one sentence per interval, or one word per interval depending on how fine a mapping is desired between words and time. The transcriptions may be produced from close caption text, higher quality off-line transcriptions, or a lower quality automatic speech recognition system.

Teleconference system 200 provides playback of recorded conferences using conference playback documents. The system utilizes stored labeled interval data associated with the conference. Fig. 3 illustrates a conference playback document 300 in accordance with one embodiment of the present invention. Conference playback document 300 is implemented as a Java applet through Java user interface 85 of Fig. 1. It uses a visual structuring of the recording as a series of color-coded intervals (e.g., intervals 305 and 310) plotted on a horizontal time axis in an area referred to as a time-line window 315. Each participant in a call

(e.g., participants 316-320) is allocated a separate time-line for graphically depicting all labeled intervals that are associated with that person (e.g., dialing, connected, muted, talking, etc.).

Fig. 4 illustrates in detail how overlapping intervals are displayed. As shown in Fig. 4, by plotting each interval type one at a time, starting with taller bars, the document displays overlapping intervals on the same line. Referring again to Fig. 3, intervals that are not associated with an individual person are plotted separately above the participants, (e.g., hyperlinks 330, speech segments, etc.). Time-line window 315 provides a snapshot of every participants' activity, and can be used to navigate through the recording.

In one embodiment, once users have established a phone connection to the recorded conference player, they can use a tool bar 350 below the time-line to begin playing the audio and adjust the skimming parameters. In another embodiment, a separate phone connection is not necessary because the audio conference recording can be "streamed" in conjunction with conference playback document 300.

Toolbar 350 provides five buttons to control the player: "goto beginning 351", "jump back 352", "stop 353", "play 354", and "jump forward 355". It also contains a slider 356 for adjusting the playing speed (0.7x, l.Ox, 1.3x, 1.7x, and 2. Ox), a zoom menu 357 for selecting the zoom factor (none, 20min., lOmin., and 5min.), and an on/off pause button 358 for pause removal.

As the recorded conference audio plays, a vertical red needle 360 moves across the time-line. When needle 360 moves, every participant's name tag is colored to reflect that person's state at that time in the meeting.

Fig. 3 shows a one hour conference with the entire duration visible (zoom = none). In this view, the visual structures help make some details of the call immediately obvious. For example, the number and span of the light colored bars can identify the most/least dominant talkers. The initial long uninterrupted talking bands show who gave the formal presentations.

Finally the point where the question and answer session began is visible roughly half way into the call, where many short talking intervals are scattered among many participants. More detailed information must be found by either listening to the audio or by searching through linked annotations, images, and other documents.

The zooming feature allows the user to narrow the duration displayed in the time-line window. A numbered scroll bar allows the user to register the zoomed-in portion with the full duration, and scroll using mouse clicks or arrow keys on the keyboard. Scrolling is independent of player location needle 360, so the user can separately glance at regions, without disrupting listening. Player needle 360 can be moved by clicking on the time-line, or by pressing a jump forward/backward button. When this happens, the skim server plays a short non-speech audio cue and begins to play at the new location.

Clicking the time-line near the top is used to select hyperlinks (e.g., link 330) rather than to move the needle. When a link is selected, or a "links" button 340 is pressed, a dialog displays all the links in the recording. This dialog can be used to visit a link, edit a link, or create a link both in and out of the time-line. One embodiment of the present invention supports the following types of links: annotations, audio, documents, images, and general URL. All links are implemented using URLs except annotations, which store textual content as interval data.

Each type of link is displayed on the time-line with a representative icon.

Hyperlinks into and out of the time-line are stored as intervals, and contain both a beginning and ending time offset. Thus a link can refer to a particular point or region of the time-line, allowing a rich set of skimming alternatives. For example, following a link can cause play to begin at a certain point, end at a certain point, or sequence through selected regions. This means that following a link can have multiple effects, including moving the player needle and changing the document page.

As disclosed, one embodiment of the present invention is a teleconference recorder and player. When a conference is recorded, an interval database stores labeled interval data associated with the conference. The labeled interval data allows searching and retrieving of the recorded conference, and facilitates playback of the recorded conference.

Several embodiments of the present invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.

For example, although the embodiments disclosed are implemented over the Internet, the present invention can be implemented using a private network, or using any other known or future data communication methods.

Claims

WHAT IS CLAIMED IS: 1. A system for recording and playing multimedia data that includes a plurality of intervals, said system comprising: a skim server that detects a first set of the plurality of intervals; an interval database server coupled to said skim server, said interval database server generating labeled interval data for the first set of the plurality of intervals detected by said skim server; and a database coupled to said interval database server and storing said labeled interval data; wherein said labeled interval data comprises an interval data element for each of the detected plurality of intervals. 2. The system of claim 1, further comprising: a conference bridge coupled to said interval database server that detects a second set of the plurality of intervals; wherein said interval database server further generates labeled interval data for the second set of the plurality of intervals detected by said skim server. 3. The system of claim 2, wherein said first set of the plurality of intervals comprise pauses in speech. 4. The system of claim 2, wherein said second set of the plurality of intervals comprise call control events. 5. The system of claim 1, wherein the multimedia data comprises a conference telephone call. 6. The system of claim 1, wherein said interval data element comprises: a type of the detected interval; a start time of the detected interval; and a duration of the detected interval. 7. The system of claim 6, wherein said interval data element further comprises: a recording identification of the detected interval; and a type-specific data value of the detected interval. 8. The system of claim 1, wherein said interval database server comprises: means for searching said stored labeled interval data. 9. The system of claim 8, wherein said interval database server further comprises: means for retrieving said stored labeled interval data and associated multimedia data. 10. The system of claim 5, further comprising: a user interface generated during playback of the conference call, wherein said user interface displays the stored labeled interval data. 11. A method for recording and playing multimedia data that includes a plurality of intervals, said method comprising: detecting the plurality of intervals; generating labeled interval data for the plurality of intervals; and storing the labeled interval data in a database; wherein said labeled interval data comprises an interval data element associated with each of the plurality of intervals. 12. The method of claim 11, wherein said interval data element comprises: a type of the associated interval; a start time of the associated interval; and a duration of the associated interval. 13. The method of claim 12, wherein said interval data element further comprises: a recording identification of the associated interval; and a type-specific data value of the associated interval. 14. The method of claim 11, further comprising: storing the multimedia data in the database. 15. The method of claim 14, further comprising: querying said database based on one or more labeled interval data parameters; and retrieving at least one interval data element and associated multimedia data from the database. 16. The method of claim 11, wherein the multimedia data comprises a conference telephone call. 17. The method of claim 16, further comprising: generating a user interface that displays the labeled interval data; and playing the conference call based on selections of the user interface. 18. A method of recording and playing a teleconference telephone call, said method comprising: detecting a plurality of intervals during the telephone call; generating labeled interval data for each of said plurality of intervals; and storing said labeled interval data in a database. 19. The method of claim 18, wherein said labeled interval data comprises a plurality of interval data elements, said method further comprising: 81 querying said database and retrieving one or more of the stored

82 interval data elements; and

83 playing a portion of the teleconference telephone call that is

84 associated with each of said retrieved interval data elements.

85 20. The method of claim 18, wherein said detected intervals

86 comprise:

87 an identity of a speaker;

88 pauses in speech; and

89 telephone call control.