CONTROL OF SEQUENCE OF VIDEO MODIFYING OPERATIONS
Background of the Invention The invention relates to modifying video.
When a video program is created at a nonlinear video editor, analog video source materials are often digitized, compressed and stored on disc for later random access for use in creating a video program. Segments of the video, also referred to as "video clips," can be taken from different sources, and the beginning and end points can be trimmed as desired during editing. The order of clips is established in a time line to determine the sequence, and the position of a clip can be changed. Clips can be deleted or inserted as desired. During the editing process, the clips of compressed video can be randomly accessed, decompressed and viewed in real time.
The source video can be modified by various "pixel manipulations," which are fine grain processes (such as scale, rotate, blur), and "tools," which are larger grain processes (such as digitizing, color correcting, add to timeline, layer with another clip and master out).
Compositing involves use of fine grain processes. In compositing applications, it is known to use a process tree to specify a set of transformations, filters, and effects to be applied to one clip. A process tree uses a dataflow metaphor, and processes are connected together with no storage nodes in between.1 As used herein, "video editing" includes traditional editing operations, compositing operations, and tool operations.
Summary of the Invention
In one aspect, the invention features, in general, a method of modifying video that includes receiving a first video clip, modifying the first video clip by one or more video tools in sequence to create a second video clip that is modified with respect to the first video clip, and recording the sequence of tools and the associated tool parameters used to modify the first video clip. Each tool modifies a clip input to it according to tool parameters and user interaction to generate an output clip.
In another aspect, the invention features, in general, a method of modifying source video that includes receiving a first video clip, modifying the first video clip by one or more operations (e.g., bit manipulations and/or tools) in sequence to create a
second video clip that is modified with respect to the first video clip, recording the sequence of operations used to modify the first video clip, and displaying a history graph of the operations used to modify the first video clip.
Preferred embodiments of the invention may include one or more of the following features. The tool history graph can be used in workflow navigation. E.g., video tools displayed on the tool history graph can be used with a different parameter file. There can be an intermediate video clip that is generated by a first tool and input to a second tool, and the second video tool can be used with a different parameter file. The first video tool can be used with a different parameter file, and the tool history graph will be updated to show that the intermediate video clip and the second video clip are outdated. The first tool can then be run with the different parameter file to result in an updated intermediate video clip, and the second video tool can be run on the updated intermediate video clip to result in an updated second video clip.
The tool history graph can be used to perform a media management function, such as moving unnecessary backing media, recreating clip backing media, and redigitizing clip backing media. The tool history graph can also be used to modify a clip history to create workflow steps. E.g., the user can create additional tools and new clips on the history. The tool history graph can specify an external tool, and the system will export a clip to the external tool and import the resulting clip. The tool history graph can be used to locate a clip on the history, and a tool can be added to modify the located clip.
In another aspect, the invention features, in general, a method of controlling media workflow by selecting a sequence of one or more operations to be used to modify a first video clip to result in a second video clip, displaying a graph of the operations, and thereafter modifying the first video clip by the selected and displayed operations.
In preferred embodiments the operations include tools. A control decision can be used determine which of a plurality of possible tools could be used to modify the first video clip, and the display could include a control block showing the first clip as an input, and a plurality of workflow branches as output, each having a respective tool and a respective output clip. The control decision can be based on an if/then/else
decision. The control decision can be based on a selection control input that indicates one of the possible tools and associated workflow branch. There can be a plurality of clips as possible inputs, and the control decision can also determine which of the first clips is input, and the displaying would include the plurality of first clips.
Embodiments of the invention may include one or more of the following advantages. The history display feature usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material. One can simply and quickly specify a new processing scheme or create a modified processing scheme for a clip by specifying new tools on the tool history diagram.
Other advantages and features of the invention will be apparent from the following description of a preferred embodiment thereof and from the claims.
Description of Drawings
Fig. 1 is a block diagram of a nonlinear video editing system.
Fig. 2 is a more detailed block diagram of some of the components of the Fig. 1 system.
Fig. 3 shows a time line and editing control box used on the display of the Fig. 1 system.
Fig. 4 shows a video editing software application and storage for clips and media.
Fig. 5 shows a standard unit of work flow used in workflow recording and display in the Fig. 1 system.
Fig. 6 shows an exemplary clip history displayed on the Fig. 1 system.
Fig. 7 shows a standard unit of workflow including an if/then/else control block.
Fig. 8 shows a standard unit of workflow including a control block control by a selection input.
Description of Particular Embodiments
Referring to Fig. 1, nonlinear video editing system 10 includes host computer 12 having monitor 14, user input devices 16 (e.g., keyboard and mouse), video editing software 18, video editing peripheral board 20 in an expansion slot of computer 12, video tape recorder (NTR) 24, and mass storage disc 26. Referring to Fig. 2, host computer 12 includes CPU 30, PCI bus 32, and input/output ports 34 for connection to monitor 14 and user input devices 16. Peripheral board 20 is connected to PCI bus 32. Board 20 includes coder/decoder (codec) 34, video input/output (I/O) port 36 for source video, color corrector 38 and busmaster circuit 40, which transfers data to and from PCI bus 32. Board 20 also has audio I/O ports, other video processing circuits, buffers, and hardware tools 102 to carry out various video editing and data transfer functions. The invention can also be used with software tools and third party hardware and software tools.
Referring to Fig. 3, during editing a user can access various control windows on monitor 14 to carry out the editing functions. The user can place small picture icons 48 and associated clip names identifying particular video clips in bin window 50, edit the beginning and end points of a particular video clip in edit window 52, and place a video clip in the program at time line 54.
Each time that video material is accessed or modified in system 10 at a clip level, a software application program keeps track of the history of processing of a video clip and its relation to prior or subsequent video clips. A representation of the history, called a tool history graph, can then be created and displayed at any time in tool history window 56, shown in Fig. 6
In creating the tool history graphs, system 10 employs (1) clips, (2) clip history information, (3) used-on lists, (4) parameter files, and (5) tools. These objects are defined and described as follows:
1. Clip
A clip is a unit of multimedia, usually video, with at least the following attributes:
a well defined start and end time (in-out), and
a description of the location in the media storage pool where the clip backing media is located.
The clip may or may not have actual backing media. Users of nonlinear video editors typically manage hundreds to thousands of clips for a given project.
2. Clip History Information
Clip history information includes:
the name of the process (tool) that created the clip,
the names of all of the input and output clips in the history of the generation of the clip, and
parameter information such as a file name and a tool-dependent format for the information in the file.
Clip history information can also include additional attributes such as name, create date, and media type.
3. Used-on List
A used-on list is associated with a particular clip and is a list of processes that use the clip. This list is the forward linlc in the tool history graph; the clip history information (2. above) is the backward link. For a given clip, either the tool history information (back link), or the used-on-list (forward linlc), or both will be stored with the clip attributes in a clip store.
4. Parameter Files
Parameter files are auxiliary input used by a process (tool). The parameter file data does not necessarily have to be stored as a separate file so long as the parameters can be saved to and restored from disk in some form. Examples of parameter file data include scalar data such as clip background color, a motion slow down - speed up amount, the key color and acceptable range for a chroma keying operation. Parameter file data can also contain time varying data such as key framed position data and color values. In general, a parameter file contains all of the data needed by a process (tool) to operate on an input clip to produce a specific output clip. The parameter file encapsulates the use of the tool by the user. The content of a parameter file is the direct or indirect result of user input.
5. Tool
A tool is a discreet transformation performed on a clip that results in a new clip being produced. A tool is usually designed to perform one type of nonlinear video editing and/or compositing operation such as motion effect, keying, painting, etc. A tool is a unit of workflow as compared to a unit of pixel manipulation. A tool is most often started and provided an input clip on which the tool will work. The output of a tool is not directly connected to another tool; it is always connected to a clip. Examples of tools include: input digitize tool, matte tool, master out tool, paint tool, color corrector, keyer (e.g., chroma and/or luminance key process), motion effects tools (e.g., fast or slow motion), timeline tool (which can do editing (horizontal) and layering (vertical) operations), motion tracking and third party applications.
Referring to Fig. 4, there is shown nonlinear editor application 104 (used in system 10) with clip storage area 106 (also called "clip store") for clip data (described as 1. above) and media storage area 108 (also called "media store") for the backing media files to which the clip data refer. Note that a clip can exist without actual media in the media storage area 108. The clip history (described as 2. above) is also stored in clip storage area 108.
Referring to Fig. 5, a standard unit of work flow can be represented by an input clip Cl (110), an output clip C2 (112), a tool 114, and parameters file 116. The input clip Cl is usually kept in the clip storage area 106 after tool 114 is finished. The media backing for clip Cl, however, may be deleted. The output clip C2 almost always has media backing in the media storage area 108. The parameters file 116 serves as input to the tool 114 and as a place to save all user input on tool exit. Parameters file 116 may be empty on tool start-up, and contain a list of all of the user commands on exit. At a minimum, parameters file 116 must contain the information necessary to produce the output clip, C2, at a later stage, given Cl and a copy of parameter file 116. In the aggregate history of all clips, a single clip can feed more than one process. That is, work flow can fork into two paths at a clip node in the graph. A tool 114 can accept more than one clip as input and produce more than one clip as output.
The processing involved in implementing a tool history graph includes (1) recording the history of a clip, (2) displaying the history of one or more clips in an interactive graph, and (3) allowing user interaction with that graph to, e.g., accomplish (a) workflow navigation, i.e., random access to steps performed over the life of a project, (b) media management, e.g., removing unwanted media, regenerating media, redigitizing media, and other tedious media management functions, and (c) workflow step creation, i.e., outlining the steps to be taken during an editing session. For example, to create a workflow step, the user can digitize one clip, color correct it, add it to a timeline, and then take that sequence of events and apply it to a whole set of clips. The steps just mentioned for implementing a tool history graph are described in the following sections, which also include a section (d) dealing with further workflow controls.
1. Recording History
As the user creates a clip within application 104 (Fig. 4), a description of the clip is created and maintained in the clip store 1.06, which is where all clips and clip history information reside. Clips are created as a result of running a tool. The input
digitize tool (which imports raw video into the application) is the most common creator of clips. The history of a digitized clip will indicate the digitize tool as the creating process, and, rather than an input clip, the history will have a tape name as input. Similarly the master out tool will write to a tape (usually) as output and have clips as inputs.
2. Displaying a Clip History
At one level, a tool history graph can be simply used to display the history of a clip or clips without user interaction with the graph (other than pan/zoom and similar presentation operations.) The user can display the history of one or more clips graphically or textually in purpose designed windows. A graphic representation can appear as in Fig. 5 (discussed above) for one a one-tool history or Fig. 6 (discussed below) for a history involving many tools. A text based clip history display lists hierarchically the clip name, the name of the creating process, and the names of the input clips and parameters. Then a similar history is also prepared for each input clip of that tool. The result is a text tree listing the complete ancestry of the initial clip in outline format. The user can specify how many links backward to trace the clip ancestry. For example, the user could display the history of the final output program clip and request that all history paths leading to that final clip be displayed back to where each initial clip was digitized.
3. User/Graph Interaction
A. Navigating Clip History - Workflow Navigation
At a further level of tool history graph usage, the user interacts with the graph to navigate the history of a clip to access a previously completed step to re-execute a process (tool). Tool re-launch is implemented by a user selecting (e.g., by clicking on) a tool in the tool history graph, which causes that tool to be launched with the
appropriate clip and parameter input(s) such that the state of the tool is the same as when it was last executed for that clip and parameters. The user can change one or more of the tool's parameters to change the tool's result. This will outdate (make obsolete) the output clip(s) of the tool as well as any subsequent clips in the history that depend on those output clips. The outdated nature of the clips is graphically indicated (e.g., different color, lines or shading) on the tool history graph. If desired, the user can select that all downstream processes are automatically run in order to update each affected subsequent clip, resulting in a great savings in time over manual operation.
B. Media Management
The tool history graph can also be used to perform at least the following media management tasks:
i. move unnecessary clip backing media offline,
ii. recreate or redigitize clip backing media files, possibly at different resolutions,
iii. cut, copy and paste clips, and
iv. consolidate backing material of a selected set of clips.
C. Modifying Clip History - Creating Workflow Steps
A further, powerful use of the tool history graph is defining sequences of workflow steps. This is accomplished by dragging tool icons 118 (Fig. 6) into the graph display, dragging clip icons 48 (Fig. 3) from bin window 50 or other places, and connecting them together to form a sub-graph (i.e., new workflow sequence.) Users can
access existing clips or tools, or create a new clip and use an unparameterized tool. Some of the workflow operations that can be performed are:
i. copy an existing sequence of steps, paste it as a new sequence, and connect a new input clip to the sub-graph,
ii. convert media by dragging an existing input clip (having RGB color) icon, a convert-RGB-to-NUV tool icon, and output clip (having NUN color) into the graph and connecting them together and executing the workflow,
iii. setup for a batch digitize by starting with a timeline program clip, selecting all initial input clips (ancestor clips whose creating process is the digitizer), and specifying a re-digitize operation,
iv. export clips (similar to the convert media example above, with the tool exporting media and having no clip output, per se),
v. call an external tool, which causes an input clip to be directed to an external tool for processing and return, and
vi. fork project development to cause a particular clip or clips to branch into more than one tool downstream in the workflow.
Fork project management can be used for "versioning." The user can add a set of clips into a timeline, add tool icons to the graph and produce an output. The user can then select all of the steps back to the set of clips, copy that sequence, paste it as a new sequence, and modify the parameters controlling the tool (or use different tools) to produce a different look. The user can thus very easily try different approaches to content design and delete the versions (media and process parameters) that are not going to be used.
Fig. 6 shows an exemplary tool history graph that displays the following sequence of steps:
1. The user digitizes video tape 120 at tool 122 into a clip, clip 1 (124) using user parameters 126.
2. Then the user places clip 1 (124) (or a copy of clip 1) directly in the timeline 126; this is indicated by arrow 128.
3. The user also takes clip 1 (124) (or a copy of clip 1) and runs it through color corrector 130, which produces a color-corrected clip, clip 2 (132) using user parameters 134.
4. The user then puts clip 2 (132) in timeline 126, where the two clips are edited together according to user parameters 136.
5. The user then renders timeline 126 to produce program clip 138.
If the user subsequently decides that he or she is not satisfied with the color correction, instead of making further modifications to the color corrected version 132, or reaccessing the material from video tape 120, the user can access the digital version, clip 1 (124), as it existed prior to color correction. The clip history feature of system 10 usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material.
System 10 also permits one to specify a new processing scheme or to create a modified processing scheme for a clip using tool history palette 140, which includes tool icons 118 to specify, e.g., a digitizer, color corrector, keyer (e.g., chroma luminance key process), motion effects (e.g., fast or slow motion) and other tools. The user accesses a tool by clicking on it and dragging it to the desired location in the processing diagram, and can insert the tool as an additional process or to replace a displayed process. The user can also add a new source video or add a new clip (from the bin window 50 or identified by name). When a new clip is inserted part way through a process, the prior steps and image source will be replaced with the prior history for the selected clip. The subsequent clips will also be replaced with new clip identifiers for new clips to be created. When the user is satisfied, he or she can enable system 10 to then create the
new clips (the ones that come after the substituted clip in the process flow) by running indicated tools on the substituted clip, unless the specified tools have already been run for the substituted clip.
The tool history graph can also be modified to record and display pixel manipulations. In this case, instead of tool icons and tool parameter files, or in addition to tool icons, there would be pixel manipulation icons that indicate the operation (e.g., scale, rotate, blur, layer with another clip) and associated pixel manipulation parameter files specifying the operation.
D. Further Workflow Control
The tool history graph as described herein can be used in the absence of a "history" to define and create workflow in the first instance by entering icons indicating clips and tools onto a blank graph.
The graph can, as a part of forked project management, include control constructs in the diagram to provide different possible flowpaths on the graph and control blocks used to determine the flow of control on the graph based upon inputs or predefined conditions. In particular, if/then/else control blocks can be added to the diagram as shown for the control block and workflow units shown in Fig. 7. The input clip 150 is shown connected to control block 152, which in this example determines which workflow branch to use in processing clip 150 based upon whether clip 150 is the PAL video type or not. If clip 150 is PAL, then tool 154 is used to generate clip 156. If clip 150 is not PAL, then tool 158 is used to generate clip 160. In addition to media type, any other media attribute, or any other variable in general, can be used to determine the workflow branch at control block 152. Another example of a control block, a switch tree, is shown in Fig. 8. In this case, clip 162 is the input clip, and switch tree 164 has five possible output paths 1-5, employing respective tools 166-174 to generate respective output clips 176-184. Switch tree 164 selects one of the five output paths depending on the value of input S being 1-5. It is also possible for a switch control block to have multiple inputs (each
with a respective input clip) and multiple outputs, with one or more control inputs used to select the input clip and output workflow path. The use of control blocks in the tool history graph workflow diagram provides a large number of possibilities for efficiently creating media and doing batch processing. Other embodiments of the invention are within the scope of the following claims.
What is claimed is: