WO2001060060A1 - Control of sequence of video modifying operations - Google Patents

Control of sequence of video modifying operations Download PDF

Info

Publication number
WO2001060060A1
WO2001060060A1 PCT/US2001/004264 US0104264W WO0160060A1 WO 2001060060 A1 WO2001060060 A1 WO 2001060060A1 US 0104264 W US0104264 W US 0104264W WO 0160060 A1 WO0160060 A1 WO 0160060A1
Authority
WO
WIPO (PCT)
Prior art keywords
clip
tool
video
video clip
tools
Prior art date
Application number
PCT/US2001/004264
Other languages
French (fr)
Inventor
Paul N. Sears
Original Assignee
Media 100, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Media 100, Inc. filed Critical Media 100, Inc.
Priority to AU2001236845A priority Critical patent/AU2001236845A1/en
Publication of WO2001060060A1 publication Critical patent/WO2001060060A1/en

Links

Classifications

    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34Indicating arrangements 
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G11B27/034Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/76Television signal recording
    • H04N5/765Interface circuits between an apparatus for recording and another apparatus

Definitions

  • the invention relates to modifying video.
  • Video clips When a video program is created at a nonlinear video editor, analog video source materials are often digitized, compressed and stored on disc for later random access for use in creating a video program. Segments of the video, also referred to as "video clips,” can be taken from different sources, and the beginning and end points can be trimmed as desired during editing. The order of clips is established in a time line to determine the sequence, and the position of a clip can be changed. Clips can be deleted or inserted as desired. During the editing process, the clips of compressed video can be randomly accessed, decompressed and viewed in real time.
  • the source video can be modified by various "pixel manipulations,” which are fine grain processes (such as scale, rotate, blur), and “tools,” which are larger grain processes (such as digitizing, color correcting, add to timeline, layer with another clip and master out).
  • pixel manipulations are fine grain processes (such as scale, rotate, blur)
  • tools which are larger grain processes (such as digitizing, color correcting, add to timeline, layer with another clip and master out).
  • Compositing involves use of fine grain processes.
  • compositing applications it is known to use a process tree to specify a set of transformations, filters, and effects to be applied to one clip.
  • a process tree uses a dataflow metaphor, and processes are connected together with no storage nodes in between. 1
  • video editing includes traditional editing operations, compositing operations, and tool operations.
  • the invention features, in general, a method of modifying video that includes receiving a first video clip, modifying the first video clip by one or more video tools in sequence to create a second video clip that is modified with respect to the first video clip, and recording the sequence of tools and the associated tool parameters used to modify the first video clip.
  • Each tool modifies a clip input to it according to tool parameters and user interaction to generate an output clip.
  • the invention features, in general, a method of modifying source video that includes receiving a first video clip, modifying the first video clip by one or more operations (e.g., bit manipulations and/or tools) in sequence to create a second video clip that is modified with respect to the first video clip, recording the sequence of operations used to modify the first video clip, and displaying a history graph of the operations used to modify the first video clip.
  • operations e.g., bit manipulations and/or tools
  • the tool history graph can be used in workflow navigation.
  • video tools displayed on the tool history graph can be used with a different parameter file.
  • the first video tool can be used with a different parameter file, and the tool history graph will be updated to show that the intermediate video clip and the second video clip are outdated.
  • the first tool can then be run with the different parameter file to result in an updated intermediate video clip, and the second video tool can be run on the updated intermediate video clip to result in an updated second video clip.
  • the tool history graph can be used to perform a media management function, such as moving unnecessary backing media, recreating clip backing media, and redigitizing clip backing media.
  • the tool history graph can also be used to modify a clip history to create workflow steps. E.g., the user can create additional tools and new clips on the history.
  • the tool history graph can specify an external tool, and the system will export a clip to the external tool and import the resulting clip.
  • the tool history graph can be used to locate a clip on the history, and a tool can be added to modify the located clip.
  • the invention features, in general, a method of controlling media workflow by selecting a sequence of one or more operations to be used to modify a first video clip to result in a second video clip, displaying a graph of the operations, and thereafter modifying the first video clip by the selected and displayed operations.
  • the operations include tools.
  • a control decision can be used determine which of a plurality of possible tools could be used to modify the first video clip, and the display could include a control block showing the first clip as an input, and a plurality of workflow branches as output, each having a respective tool and a respective output clip.
  • the control decision can be based on an if/then/else decision.
  • the control decision can be based on a selection control input that indicates one of the possible tools and associated workflow branch.
  • Embodiments of the invention may include one or more of the following advantages.
  • the history display feature usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material.
  • Fig. 1 is a block diagram of a nonlinear video editing system.
  • Fig. 2 is a more detailed block diagram of some of the components of the Fig. 1 system.
  • Fig. 3 shows a time line and editing control box used on the display of the Fig. 1 system.
  • Fig. 4 shows a video editing software application and storage for clips and media.
  • Fig. 5 shows a standard unit of work flow used in workflow recording and display in the Fig. 1 system.
  • Fig. 6 shows an exemplary clip history displayed on the Fig. 1 system.
  • Fig. 7 shows a standard unit of workflow including an if/then/else control block.
  • Fig. 8 shows a standard unit of workflow including a control block control by a selection input. Description of Particular Embodiments
  • nonlinear video editing system 10 includes host computer 12 having monitor 14, user input devices 16 (e.g., keyboard and mouse), video editing software 18, video editing peripheral board 20 in an expansion slot of computer 12, video tape recorder (NTR) 24, and mass storage disc 26.
  • host computer 12 includes CPU 30, PCI bus 32, and input/output ports 34 for connection to monitor 14 and user input devices 16.
  • Peripheral board 20 is connected to PCI bus 32.
  • Board 20 includes coder/decoder (codec) 34, video input/output (I/O) port 36 for source video, color corrector 38 and busmaster circuit 40, which transfers data to and from PCI bus 32.
  • Board 20 also has audio I/O ports, other video processing circuits, buffers, and hardware tools 102 to carry out various video editing and data transfer functions.
  • the invention can also be used with software tools and third party hardware and software tools.
  • a user can access various control windows on monitor 14 to carry out the editing functions.
  • the user can place small picture icons 48 and associated clip names identifying particular video clips in bin window 50, edit the beginning and end points of a particular video clip in edit window 52, and place a video clip in the program at time line 54.
  • a software application program keeps track of the history of processing of a video clip and its relation to prior or subsequent video clips.
  • a representation of the history called a tool history graph, can then be created and displayed at any time in tool history window 56, shown in Fig. 6
  • system 10 In creating the tool history graphs, system 10 employs (1) clips, (2) clip history information, (3) used-on lists, (4) parameter files, and (5) tools. These objects are defined and described as follows: 1. Clip
  • a clip is a unit of multimedia, usually video, with at least the following attributes:
  • the clip may or may not have actual backing media. Users of nonlinear video editors typically manage hundreds to thousands of clips for a given project.
  • Clip history information includes:
  • parameter information such as a file name and a tool-dependent format for the information in the file.
  • Clip history information can also include additional attributes such as name, create date, and media type.
  • a used-on list is associated with a particular clip and is a list of processes that use the clip. This list is the forward linlc in the tool history graph; the clip history information (2. above) is the backward link. For a given clip, either the tool history information (back link), or the used-on-list (forward linlc), or both will be stored with the clip attributes in a clip store. 4. Parameter Files
  • Parameter files are auxiliary input used by a process (tool).
  • the parameter file data does not necessarily have to be stored as a separate file so long as the parameters can be saved to and restored from disk in some form.
  • Examples of parameter file data include scalar data such as clip background color, a motion slow down - speed up amount, the key color and acceptable range for a chroma keying operation.
  • Parameter file data can also contain time varying data such as key framed position data and color values.
  • a parameter file contains all of the data needed by a process (tool) to operate on an input clip to produce a specific output clip.
  • the parameter file encapsulates the use of the tool by the user.
  • the content of a parameter file is the direct or indirect result of user input.
  • a tool is a discreet transformation performed on a clip that results in a new clip being produced.
  • a tool is usually designed to perform one type of nonlinear video editing and/or compositing operation such as motion effect, keying, painting, etc.
  • a tool is a unit of workflow as compared to a unit of pixel manipulation. A tool is most often started and provided an input clip on which the tool will work. The output of a tool is not directly connected to another tool; it is always connected to a clip.
  • tools include: input digitize tool, matte tool, master out tool, paint tool, color corrector, keyer (e.g., chroma and/or luminance key process), motion effects tools (e.g., fast or slow motion), timeline tool (which can do editing (horizontal) and layering (vertical) operations), motion tracking and third party applications.
  • keyer e.g., chroma and/or luminance key process
  • motion effects tools e.g., fast or slow motion
  • timeline tool which can do editing (horizontal) and layering (vertical) operations
  • motion tracking and third party applications include: input digitize tool, matte tool, master out tool, paint tool, color corrector, keyer (e.g., chroma and/or luminance key process), motion effects tools (e.g., fast or slow motion), timeline tool (which can do editing (horizontal) and layering (vertical) operations), motion tracking and third party applications.
  • nonlinear editor application 104 (used in system 10) with clip storage area 106 (also called “clip store”) for clip data (described as 1. above) and media storage area 108 (also called “media store”) for the backing media files to which the clip data refer.
  • clip storage area 106 also called “clip store”
  • media storage area 108 also called “media store”
  • a clip can exist without actual media in the media storage area 108.
  • the clip history (described as 2. above) is also stored in clip storage area 108.
  • a standard unit of work flow can be represented by an input clip Cl (110), an output clip C2 (112), a tool 114, and parameters file 116.
  • the input clip Cl is usually kept in the clip storage area 106 after tool 114 is finished.
  • the media backing for clip Cl may be deleted.
  • the output clip C2 almost always has media backing in the media storage area 108.
  • the parameters file 116 serves as input to the tool 114 and as a place to save all user input on tool exit. Parameters file 116 may be empty on tool start-up, and contain a list of all of the user commands on exit. At a minimum, parameters file 116 must contain the information necessary to produce the output clip, C2, at a later stage, given Cl and a copy of parameter file 116. In the aggregate history of all clips, a single clip can feed more than one process. That is, work flow can fork into two paths at a clip node in the graph. A tool 114 can accept more than one clip as input and produce more than one clip as output.
  • the processing involved in implementing a tool history graph includes (1) recording the history of a clip, (2) displaying the history of one or more clips in an interactive graph, and (3) allowing user interaction with that graph to, e.g., accomplish (a) workflow navigation, i.e., random access to steps performed over the life of a project, (b) media management, e.g., removing unwanted media, regenerating media, redigitizing media, and other tedious media management functions, and (c) workflow step creation, i.e., outlining the steps to be taken during an editing session. For example, to create a workflow step, the user can digitize one clip, color correct it, add it to a timeline, and then take that sequence of events and apply it to a whole set of clips.
  • workflow navigation i.e., random access to steps performed over the life of a project
  • media management e.g., removing unwanted media, regenerating media, redigitizing media, and other tedious media management functions
  • workflow step creation i.e., outlining
  • a description of the clip is created and maintained in the clip store 1.06, which is where all clips and clip history information reside.
  • Clips are created as a result of running a tool.
  • the input digitize tool (which imports raw video into the application) is the most common creator of clips.
  • the history of a digitized clip will indicate the digitize tool as the creating process, and, rather than an input clip, the history will have a tape name as input.
  • the master out tool will write to a tape (usually) as output and have clips as inputs.
  • a tool history graph can be simply used to display the history of a clip or clips without user interaction with the graph (other than pan/zoom and similar presentation operations.)
  • the user can display the history of one or more clips graphically or textually in purpose designed windows.
  • a graphic representation can appear as in Fig. 5 (discussed above) for one a one-tool history or Fig. 6 (discussed below) for a history involving many tools.
  • a text based clip history display lists hierarchically the clip name, the name of the creating process, and the names of the input clips and parameters. Then a similar history is also prepared for each input clip of that tool.
  • the result is a text tree listing the complete ancestry of the initial clip in outline format.
  • the user can specify how many links backward to trace the clip ancestry. For example, the user could display the history of the final output program clip and request that all history paths leading to that final clip be displayed back to where each initial clip was digitized.
  • Tool re-launch is implemented by a user selecting (e.g., by clicking on) a tool in the tool history graph, which causes that tool to be launched with the appropriate clip and parameter input(s) such that the state of the tool is the same as when it was last executed for that clip and parameters.
  • the user can change one or more of the tool's parameters to change the tool's result. This will outdate (make obsolete) the output clip(s) of the tool as well as any subsequent clips in the history that depend on those output clips.
  • the outdated nature of the clips is graphically indicated (e.g., different color, lines or shading) on the tool history graph. If desired, the user can select that all downstream processes are automatically run in order to update each affected subsequent clip, resulting in a great savings in time over manual operation.
  • the tool history graph can also be used to perform at least the following media management tasks:
  • a further, powerful use of the tool history graph is defining sequences of workflow steps. This is accomplished by dragging tool icons 118 (Fig. 6) into the graph display, dragging clip icons 48 (Fig. 3) from bin window 50 or other places, and connecting them together to form a sub-graph (i.e., new workflow sequence.) Users can access existing clips or tools, or create a new clip and use an unparameterized tool.
  • v. call an external tool, which causes an input clip to be directed to an external tool for processing and return, and
  • Fork project management can be used for "versioning."
  • the user can add a set of clips into a timeline, add tool icons to the graph and produce an output.
  • the user can then select all of the steps back to the set of clips, copy that sequence, paste it as a new sequence, and modify the parameters controlling the tool (or use different tools) to produce a different look.
  • the user can thus very easily try different approaches to content design and delete the versions (media and process parameters) that are not going to be used.
  • Fig. 6 shows an exemplary tool history graph that displays the following sequence of steps: 1.
  • the user digitizes video tape 120 at tool 122 into a clip, clip 1 (124) using user parameters 126.
  • clip 1 (124) (or a copy of clip 1) directly in the timeline 126; this is indicated by arrow 128.
  • the user also takes clip 1 (124) (or a copy of clip 1) and runs it through color corrector 130, which produces a color-corrected clip, clip 2 (132) using user parameters 134.
  • the user can access the digital version, clip 1 (124), as it existed prior to color correction.
  • the clip history feature of system 10 usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material.
  • System 10 also permits one to specify a new processing scheme or to create a modified processing scheme for a clip using tool history palette 140, which includes tool icons 118 to specify, e.g., a digitizer, color corrector, keyer (e.g., chroma luminance key process), motion effects (e.g., fast or slow motion) and other tools.
  • tool icons 118 to specify, e.g., a digitizer, color corrector, keyer (e.g., chroma luminance key process), motion effects (e.g., fast or slow motion) and other tools.
  • the user accesses a tool by clicking on it and dragging it to the desired location in the processing diagram, and can insert the tool as an additional process or to replace a displayed process.
  • the user can also add a new source video or add a new clip (from the bin window 50 or identified by name). When a new clip is inserted part way through a process, the prior steps and image source will be replaced with the prior history for the selected clip.
  • the subsequent clips will also be replaced with new clip identifiers for new clips to be created.
  • system 10 can enable system 10 to then create the new clips (the ones that come after the substituted clip in the process flow) by running indicated tools on the substituted clip, unless the specified tools have already been run for the substituted clip.
  • the tool history graph can also be modified to record and display pixel manipulations.
  • pixel manipulation icons that indicate the operation (e.g., scale, rotate, blur, layer with another clip) and associated pixel manipulation parameter files specifying the operation.
  • the tool history graph as described herein can be used in the absence of a "history" to define and create workflow in the first instance by entering icons indicating clips and tools onto a blank graph.
  • the graph can, as a part of forked project management, include control constructs in the diagram to provide different possible flowpaths on the graph and control blocks used to determine the flow of control on the graph based upon inputs or predefined conditions.
  • control blocks can be added to the diagram as shown for the control block and workflow units shown in Fig. 7.
  • the input clip 150 is shown connected to control block 152, which in this example determines which workflow branch to use in processing clip 150 based upon whether clip 150 is the PAL video type or not. If clip 150 is PAL, then tool 154 is used to generate clip 156. If clip 150 is not PAL, then tool 158 is used to generate clip 160.
  • any other media attribute, or any other variable in general, can be used to determine the workflow branch at control block 152.
  • a control block a switch tree
  • clip 162 is the input clip
  • switch tree 164 has five possible output paths 1-5, employing respective tools 166-174 to generate respective output clips 176-184.
  • Switch tree 164 selects one of the five output paths depending on the value of input S being 1-5.
  • a switch control block it is also possible for a switch control block to have multiple inputs (each with a respective input clip) and multiple outputs, with one or more control inputs used to select the input clip and output workflow path.
  • the use of control blocks in the tool history graph workflow diagram provides a large number of possibilities for efficiently creating media and doing batch processing. Other embodiments of the invention are within the scope of the following claims.

Abstract

A method of modifying video that includes receiving a first video clip (124), modifying the first video clip by one or more video tools (134) in sequence to create a second video clip (132) that is modified with respect to the first video clip, and recording the sequence of tools and the associated tool parameters used to modify the first video clip. Each tool modifies clip input to it according to tool parameters to generate an output clip. Also described are the display of a tools history graph (140) useful in monitoring workflow, workflow navigation, media management and workflow creation.

Description

CONTROL OF SEQUENCE OF VIDEO MODIFYING OPERATIONS
Background of the Invention The invention relates to modifying video.
When a video program is created at a nonlinear video editor, analog video source materials are often digitized, compressed and stored on disc for later random access for use in creating a video program. Segments of the video, also referred to as "video clips," can be taken from different sources, and the beginning and end points can be trimmed as desired during editing. The order of clips is established in a time line to determine the sequence, and the position of a clip can be changed. Clips can be deleted or inserted as desired. During the editing process, the clips of compressed video can be randomly accessed, decompressed and viewed in real time.
The source video can be modified by various "pixel manipulations," which are fine grain processes (such as scale, rotate, blur), and "tools," which are larger grain processes (such as digitizing, color correcting, add to timeline, layer with another clip and master out).
Compositing involves use of fine grain processes. In compositing applications, it is known to use a process tree to specify a set of transformations, filters, and effects to be applied to one clip. A process tree uses a dataflow metaphor, and processes are connected together with no storage nodes in between.1 As used herein, "video editing" includes traditional editing operations, compositing operations, and tool operations.
Summary of the Invention
In one aspect, the invention features, in general, a method of modifying video that includes receiving a first video clip, modifying the first video clip by one or more video tools in sequence to create a second video clip that is modified with respect to the first video clip, and recording the sequence of tools and the associated tool parameters used to modify the first video clip. Each tool modifies a clip input to it according to tool parameters and user interaction to generate an output clip.
In another aspect, the invention features, in general, a method of modifying source video that includes receiving a first video clip, modifying the first video clip by one or more operations (e.g., bit manipulations and/or tools) in sequence to create a second video clip that is modified with respect to the first video clip, recording the sequence of operations used to modify the first video clip, and displaying a history graph of the operations used to modify the first video clip.
Preferred embodiments of the invention may include one or more of the following features. The tool history graph can be used in workflow navigation. E.g., video tools displayed on the tool history graph can be used with a different parameter file. There can be an intermediate video clip that is generated by a first tool and input to a second tool, and the second video tool can be used with a different parameter file. The first video tool can be used with a different parameter file, and the tool history graph will be updated to show that the intermediate video clip and the second video clip are outdated. The first tool can then be run with the different parameter file to result in an updated intermediate video clip, and the second video tool can be run on the updated intermediate video clip to result in an updated second video clip.
The tool history graph can be used to perform a media management function, such as moving unnecessary backing media, recreating clip backing media, and redigitizing clip backing media. The tool history graph can also be used to modify a clip history to create workflow steps. E.g., the user can create additional tools and new clips on the history. The tool history graph can specify an external tool, and the system will export a clip to the external tool and import the resulting clip. The tool history graph can be used to locate a clip on the history, and a tool can be added to modify the located clip.
In another aspect, the invention features, in general, a method of controlling media workflow by selecting a sequence of one or more operations to be used to modify a first video clip to result in a second video clip, displaying a graph of the operations, and thereafter modifying the first video clip by the selected and displayed operations.
In preferred embodiments the operations include tools. A control decision can be used determine which of a plurality of possible tools could be used to modify the first video clip, and the display could include a control block showing the first clip as an input, and a plurality of workflow branches as output, each having a respective tool and a respective output clip. The control decision can be based on an if/then/else decision. The control decision can be based on a selection control input that indicates one of the possible tools and associated workflow branch. There can be a plurality of clips as possible inputs, and the control decision can also determine which of the first clips is input, and the displaying would include the plurality of first clips.
Embodiments of the invention may include one or more of the following advantages. The history display feature usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material. One can simply and quickly specify a new processing scheme or create a modified processing scheme for a clip by specifying new tools on the tool history diagram.
Other advantages and features of the invention will be apparent from the following description of a preferred embodiment thereof and from the claims.
Description of Drawings
Fig. 1 is a block diagram of a nonlinear video editing system.
Fig. 2 is a more detailed block diagram of some of the components of the Fig. 1 system.
Fig. 3 shows a time line and editing control box used on the display of the Fig. 1 system.
Fig. 4 shows a video editing software application and storage for clips and media.
Fig. 5 shows a standard unit of work flow used in workflow recording and display in the Fig. 1 system.
Fig. 6 shows an exemplary clip history displayed on the Fig. 1 system.
Fig. 7 shows a standard unit of workflow including an if/then/else control block.
Fig. 8 shows a standard unit of workflow including a control block control by a selection input. Description of Particular Embodiments
Referring to Fig. 1, nonlinear video editing system 10 includes host computer 12 having monitor 14, user input devices 16 (e.g., keyboard and mouse), video editing software 18, video editing peripheral board 20 in an expansion slot of computer 12, video tape recorder (NTR) 24, and mass storage disc 26. Referring to Fig. 2, host computer 12 includes CPU 30, PCI bus 32, and input/output ports 34 for connection to monitor 14 and user input devices 16. Peripheral board 20 is connected to PCI bus 32. Board 20 includes coder/decoder (codec) 34, video input/output (I/O) port 36 for source video, color corrector 38 and busmaster circuit 40, which transfers data to and from PCI bus 32. Board 20 also has audio I/O ports, other video processing circuits, buffers, and hardware tools 102 to carry out various video editing and data transfer functions. The invention can also be used with software tools and third party hardware and software tools.
Referring to Fig. 3, during editing a user can access various control windows on monitor 14 to carry out the editing functions. The user can place small picture icons 48 and associated clip names identifying particular video clips in bin window 50, edit the beginning and end points of a particular video clip in edit window 52, and place a video clip in the program at time line 54.
Each time that video material is accessed or modified in system 10 at a clip level, a software application program keeps track of the history of processing of a video clip and its relation to prior or subsequent video clips. A representation of the history, called a tool history graph, can then be created and displayed at any time in tool history window 56, shown in Fig. 6
In creating the tool history graphs, system 10 employs (1) clips, (2) clip history information, (3) used-on lists, (4) parameter files, and (5) tools. These objects are defined and described as follows: 1. Clip
A clip is a unit of multimedia, usually video, with at least the following attributes:
a well defined start and end time (in-out), and
a description of the location in the media storage pool where the clip backing media is located.
The clip may or may not have actual backing media. Users of nonlinear video editors typically manage hundreds to thousands of clips for a given project.
2. Clip History Information
Clip history information includes:
the name of the process (tool) that created the clip,
the names of all of the input and output clips in the history of the generation of the clip, and
parameter information such as a file name and a tool-dependent format for the information in the file.
Clip history information can also include additional attributes such as name, create date, and media type.
3. Used-on List
A used-on list is associated with a particular clip and is a list of processes that use the clip. This list is the forward linlc in the tool history graph; the clip history information (2. above) is the backward link. For a given clip, either the tool history information (back link), or the used-on-list (forward linlc), or both will be stored with the clip attributes in a clip store. 4. Parameter Files
Parameter files are auxiliary input used by a process (tool). The parameter file data does not necessarily have to be stored as a separate file so long as the parameters can be saved to and restored from disk in some form. Examples of parameter file data include scalar data such as clip background color, a motion slow down - speed up amount, the key color and acceptable range for a chroma keying operation. Parameter file data can also contain time varying data such as key framed position data and color values. In general, a parameter file contains all of the data needed by a process (tool) to operate on an input clip to produce a specific output clip. The parameter file encapsulates the use of the tool by the user. The content of a parameter file is the direct or indirect result of user input.
5. Tool
A tool is a discreet transformation performed on a clip that results in a new clip being produced. A tool is usually designed to perform one type of nonlinear video editing and/or compositing operation such as motion effect, keying, painting, etc. A tool is a unit of workflow as compared to a unit of pixel manipulation. A tool is most often started and provided an input clip on which the tool will work. The output of a tool is not directly connected to another tool; it is always connected to a clip. Examples of tools include: input digitize tool, matte tool, master out tool, paint tool, color corrector, keyer (e.g., chroma and/or luminance key process), motion effects tools (e.g., fast or slow motion), timeline tool (which can do editing (horizontal) and layering (vertical) operations), motion tracking and third party applications.
Referring to Fig. 4, there is shown nonlinear editor application 104 (used in system 10) with clip storage area 106 (also called "clip store") for clip data (described as 1. above) and media storage area 108 (also called "media store") for the backing media files to which the clip data refer. Note that a clip can exist without actual media in the media storage area 108. The clip history (described as 2. above) is also stored in clip storage area 108. Referring to Fig. 5, a standard unit of work flow can be represented by an input clip Cl (110), an output clip C2 (112), a tool 114, and parameters file 116. The input clip Cl is usually kept in the clip storage area 106 after tool 114 is finished. The media backing for clip Cl, however, may be deleted. The output clip C2 almost always has media backing in the media storage area 108. The parameters file 116 serves as input to the tool 114 and as a place to save all user input on tool exit. Parameters file 116 may be empty on tool start-up, and contain a list of all of the user commands on exit. At a minimum, parameters file 116 must contain the information necessary to produce the output clip, C2, at a later stage, given Cl and a copy of parameter file 116. In the aggregate history of all clips, a single clip can feed more than one process. That is, work flow can fork into two paths at a clip node in the graph. A tool 114 can accept more than one clip as input and produce more than one clip as output.
The processing involved in implementing a tool history graph includes (1) recording the history of a clip, (2) displaying the history of one or more clips in an interactive graph, and (3) allowing user interaction with that graph to, e.g., accomplish (a) workflow navigation, i.e., random access to steps performed over the life of a project, (b) media management, e.g., removing unwanted media, regenerating media, redigitizing media, and other tedious media management functions, and (c) workflow step creation, i.e., outlining the steps to be taken during an editing session. For example, to create a workflow step, the user can digitize one clip, color correct it, add it to a timeline, and then take that sequence of events and apply it to a whole set of clips. The steps just mentioned for implementing a tool history graph are described in the following sections, which also include a section (d) dealing with further workflow controls.
1. Recording History
As the user creates a clip within application 104 (Fig. 4), a description of the clip is created and maintained in the clip store 1.06, which is where all clips and clip history information reside. Clips are created as a result of running a tool. The input digitize tool (which imports raw video into the application) is the most common creator of clips. The history of a digitized clip will indicate the digitize tool as the creating process, and, rather than an input clip, the history will have a tape name as input. Similarly the master out tool will write to a tape (usually) as output and have clips as inputs.
2. Displaying a Clip History
At one level, a tool history graph can be simply used to display the history of a clip or clips without user interaction with the graph (other than pan/zoom and similar presentation operations.) The user can display the history of one or more clips graphically or textually in purpose designed windows. A graphic representation can appear as in Fig. 5 (discussed above) for one a one-tool history or Fig. 6 (discussed below) for a history involving many tools. A text based clip history display lists hierarchically the clip name, the name of the creating process, and the names of the input clips and parameters. Then a similar history is also prepared for each input clip of that tool. The result is a text tree listing the complete ancestry of the initial clip in outline format. The user can specify how many links backward to trace the clip ancestry. For example, the user could display the history of the final output program clip and request that all history paths leading to that final clip be displayed back to where each initial clip was digitized.
3. User/Graph Interaction
A. Navigating Clip History - Workflow Navigation
At a further level of tool history graph usage, the user interacts with the graph to navigate the history of a clip to access a previously completed step to re-execute a process (tool). Tool re-launch is implemented by a user selecting (e.g., by clicking on) a tool in the tool history graph, which causes that tool to be launched with the appropriate clip and parameter input(s) such that the state of the tool is the same as when it was last executed for that clip and parameters. The user can change one or more of the tool's parameters to change the tool's result. This will outdate (make obsolete) the output clip(s) of the tool as well as any subsequent clips in the history that depend on those output clips. The outdated nature of the clips is graphically indicated (e.g., different color, lines or shading) on the tool history graph. If desired, the user can select that all downstream processes are automatically run in order to update each affected subsequent clip, resulting in a great savings in time over manual operation.
B. Media Management
The tool history graph can also be used to perform at least the following media management tasks:
i. move unnecessary clip backing media offline,
ii. recreate or redigitize clip backing media files, possibly at different resolutions,
iii. cut, copy and paste clips, and
iv. consolidate backing material of a selected set of clips.
C. Modifying Clip History - Creating Workflow Steps
A further, powerful use of the tool history graph is defining sequences of workflow steps. This is accomplished by dragging tool icons 118 (Fig. 6) into the graph display, dragging clip icons 48 (Fig. 3) from bin window 50 or other places, and connecting them together to form a sub-graph (i.e., new workflow sequence.) Users can access existing clips or tools, or create a new clip and use an unparameterized tool. Some of the workflow operations that can be performed are:
i. copy an existing sequence of steps, paste it as a new sequence, and connect a new input clip to the sub-graph,
ii. convert media by dragging an existing input clip (having RGB color) icon, a convert-RGB-to-NUV tool icon, and output clip (having NUN color) into the graph and connecting them together and executing the workflow,
iii. setup for a batch digitize by starting with a timeline program clip, selecting all initial input clips (ancestor clips whose creating process is the digitizer), and specifying a re-digitize operation,
iv. export clips (similar to the convert media example above, with the tool exporting media and having no clip output, per se),
v. call an external tool, which causes an input clip to be directed to an external tool for processing and return, and
vi. fork project development to cause a particular clip or clips to branch into more than one tool downstream in the workflow.
Fork project management can be used for "versioning." The user can add a set of clips into a timeline, add tool icons to the graph and produce an output. The user can then select all of the steps back to the set of clips, copy that sequence, paste it as a new sequence, and modify the parameters controlling the tool (or use different tools) to produce a different look. The user can thus very easily try different approaches to content design and delete the versions (media and process parameters) that are not going to be used.
Fig. 6 shows an exemplary tool history graph that displays the following sequence of steps: 1. The user digitizes video tape 120 at tool 122 into a clip, clip 1 (124) using user parameters 126.
2. Then the user places clip 1 (124) (or a copy of clip 1) directly in the timeline 126; this is indicated by arrow 128.
3. The user also takes clip 1 (124) (or a copy of clip 1) and runs it through color corrector 130, which produces a color-corrected clip, clip 2 (132) using user parameters 134.
4. The user then puts clip 2 (132) in timeline 126, where the two clips are edited together according to user parameters 136.
5. The user then renders timeline 126 to produce program clip 138.
If the user subsequently decides that he or she is not satisfied with the color correction, instead of making further modifications to the color corrected version 132, or reaccessing the material from video tape 120, the user can access the digital version, clip 1 (124), as it existed prior to color correction. The clip history feature of system 10 usefully permits one to access and visualize the history of a particular program clip. It also permits one to access earlier digitized versions of the clip without the need to reaccess the original source material.
System 10 also permits one to specify a new processing scheme or to create a modified processing scheme for a clip using tool history palette 140, which includes tool icons 118 to specify, e.g., a digitizer, color corrector, keyer (e.g., chroma luminance key process), motion effects (e.g., fast or slow motion) and other tools. The user accesses a tool by clicking on it and dragging it to the desired location in the processing diagram, and can insert the tool as an additional process or to replace a displayed process. The user can also add a new source video or add a new clip (from the bin window 50 or identified by name). When a new clip is inserted part way through a process, the prior steps and image source will be replaced with the prior history for the selected clip. The subsequent clips will also be replaced with new clip identifiers for new clips to be created. When the user is satisfied, he or she can enable system 10 to then create the new clips (the ones that come after the substituted clip in the process flow) by running indicated tools on the substituted clip, unless the specified tools have already been run for the substituted clip.
The tool history graph can also be modified to record and display pixel manipulations. In this case, instead of tool icons and tool parameter files, or in addition to tool icons, there would be pixel manipulation icons that indicate the operation (e.g., scale, rotate, blur, layer with another clip) and associated pixel manipulation parameter files specifying the operation.
D. Further Workflow Control
The tool history graph as described herein can be used in the absence of a "history" to define and create workflow in the first instance by entering icons indicating clips and tools onto a blank graph.
The graph can, as a part of forked project management, include control constructs in the diagram to provide different possible flowpaths on the graph and control blocks used to determine the flow of control on the graph based upon inputs or predefined conditions. In particular, if/then/else control blocks can be added to the diagram as shown for the control block and workflow units shown in Fig. 7. The input clip 150 is shown connected to control block 152, which in this example determines which workflow branch to use in processing clip 150 based upon whether clip 150 is the PAL video type or not. If clip 150 is PAL, then tool 154 is used to generate clip 156. If clip 150 is not PAL, then tool 158 is used to generate clip 160. In addition to media type, any other media attribute, or any other variable in general, can be used to determine the workflow branch at control block 152. Another example of a control block, a switch tree, is shown in Fig. 8. In this case, clip 162 is the input clip, and switch tree 164 has five possible output paths 1-5, employing respective tools 166-174 to generate respective output clips 176-184. Switch tree 164 selects one of the five output paths depending on the value of input S being 1-5. It is also possible for a switch control block to have multiple inputs (each with a respective input clip) and multiple outputs, with one or more control inputs used to select the input clip and output workflow path. The use of control blocks in the tool history graph workflow diagram provides a large number of possibilities for efficiently creating media and doing batch processing. Other embodiments of the invention are within the scope of the following claims.
What is claimed is:

Claims

1. A method of modifying video comprising receiving a first video clip, modifying said first video clip by one or more video tools in sequence to create a second video clip that is modified with respect to said first video clip, each said tool modifying a clip input to it according to tool parameters to generate an output clip, and recording the sequence of tools and the associated tool parameters used to modify said first video clip.
2. A method of modifying video comprising receiving a first video clip, modifying said first video clip by one or more operations in sequence to create a second video clip that is modified with respect to said first video clip, recording the sequence of operations used to modify said first video clip, and displaying a history graph of said operations used to modify said first video clip.
3. The method of claim 2 wherein said operations include tools, each said tool modifying a clip input to it according to tool parameters to generate an output clip.
4.The method of claim 2 or 3 wherein said operations include pixel manipulations
5. The method of claim 1 further comprising displaying a history of said tools used to modify said first video clip.
6. The method of claim 1 or 5 further comprising repeating use of a said video tool with a different parameter file.
7. The method of claim 1 or 5 wherein there are first and second said tools and an intermediate video clip that is generated by said first tool and input to said second tool, and further comprising repeating use of said second video tool with a different parameter file.
8. The method of claim 5 wherein there are first and second said tools and an intermediate video clip that is generated by said first tool and input to said second tool, and further comprising changing said display to show a use of said first video tool with a different parameter file and to show that said intermediate video clip and second video clip are outdated.
9. The method of claim 8 further comprising repeating said use of said first tool with said different parameter file to result in an updated intermediate video clip and repeating use of said second video tool on said updated intermediate video clip to result in an updated second video clip.
10. The method of claim 2 or 5 further comprising performing a media management function based upon said displaying of said history.
11. The method of claim 2 or 5 further comprising performing a media management function based upon said displaying of said history, and wherein said media management function is a function selected from the list of removing unnecessary backing media, recreating clip backing media, redigitizing clip backing media, and cutting, copying and/or pasting clips.
12. The method of claim 5 further comprising creating additional tools and/or new clips on said history.
13. The method of claim 5 wherein a said tool is an external tool, and further comprising exporting to said external tool and importing the resulting clip.
14. The method of claim 5 further comprising selecting a clip on said history and using a tool to modify the located clip.
15. Computer apparatus for modifying video comprising a processor, a user input device, a video editing application, a clip store that is accessible by said application and stores clips, a backing media store that is accessible by said application and stores media specified by said clips, tools for performing operations on clips, and parameter files specifying parameters for said tools, each said tool modifying a clip input to it according to said tool parameters to generate an output clip, said application recording the sequence of tools and the associated tool parameters used to modify a first video clip into a second video clip.
16. Computer apparatus for modifying video comprising a processor, a user input device, a video editing application, a clip store that is accessible by said application and stores clips, a backing media store that is accessible by said application and stores media specified by said clips, said application recording the sequence of operations used to modify a first video clip, and a display that shows a history graph of the operations used to modify said first video clip.
17. The apparatus of claim 16 further comprising tools and wherein said display displays operations that include said tools.
18. The apparatus of claim 16 or 17 wherein said display displays operations that include pixel manipulations.
19. The apparatus of claim 15 further comprising a display that shows a history graph of said tools used to modify said first video clip.
20. The apparatus of claim 15 or 19 wherein a said video tool can be used with a different parameter file.
21. The apparatus of claim 15 or 19 wherein there are first and second said tools and an intermediate video clip that is generated by said first tool and input to said second tool, and wherein use of said second video tool is repeated with a different parameter file.
22. The apparatus of claim 19 wherein there are first and second said tools and an intermediate video clip that is generated by said first tool and input to said second tool, and wherein said display is changed to show a use of said first video tool with a different parameter file and to show that said intermediate video clip and second video clip are outdated.
23. The apparatus of claim 22 wherein use of said first tool is repeated with said different parameter file to result in an updated intermediate video clip and use of said second video tool is repeated on said updated intermediate video clip to result in an updated second video clip.
24. The apparatus of claim 19 wherein a said tool is an external tool.
25. A method of modifying video comprising selecting a sequence of one or more tools to be used to modify a first video clip to create a second video clip that is modified with respect to said first video clip, displaying a graph of said one or more tools used to modify said first video clip to create said second video clip, and thereafter modifying said first video clip by said one or more tools in sequence.
26. A method of modifying video comprising selecting a sequence of one or more operations to be used to modify a first video clip to create a second video clip that is modified with respect to said first video clip, displaying a graph of said one or more operations used to modify said first video clip to create said second video clip, and thereafter modifying said first video clip by said one or more operations in sequence, wherein said selecting includes selecting a control decision to determine which of a plurality of possible operations will be used to modify said first video clip, and said displaying includes displaying a control block showing said first clip as an input, and a plurality of workflow branches as output, each having a said possible operation and a respective output clip.
27. The method of claim 26 wherein said operations include pixel manipulations.
28. The method of claim 25 wherein said selecting includes selecting a control decision to determine which of a plurality of possible tools will be used to modify said first video clip, and said displaying includes displaying a control block showing said first clip as an input, and a plurality of workflow branches as output, each having a said possible tool and a respective output clip.
29. The method of claim 26 or 28 wherein said control decision is based on an if/then/else decision.
30. The method of claim 26 or 28 wherein said control decision is based on an a selection control input that indicates a said possible tool and associated workflow branch.
31. The method of claim 26 or 28 wherein there are a plurality of said first clips as possible inputs, and said control decision also determines which said first clip is input, and said displaying includes displaying said plurality of first clip.
PCT/US2001/004264 2000-02-09 2001-02-09 Control of sequence of video modifying operations WO2001060060A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2001236845A AU2001236845A1 (en) 2000-02-09 2001-02-09 Control of sequence of video modifying operations

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US50137700A 2000-02-09 2000-02-09
US09/501,377 2000-02-09

Publications (1)

Publication Number Publication Date
WO2001060060A1 true WO2001060060A1 (en) 2001-08-16

Family

ID=23993307

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/004264 WO2001060060A1 (en) 2000-02-09 2001-02-09 Control of sequence of video modifying operations

Country Status (2)

Country Link
AU (1) AU2001236845A1 (en)
WO (1) WO2001060060A1 (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002021272A2 (en) * 2000-09-08 2002-03-14 Corel Inc. Method and apparatus for enhancing reliability of automated data processing
US6651121B1 (en) 2000-09-08 2003-11-18 Corel Inc. Method and apparatus for facilitating scalability during automated data processing
US6757888B1 (en) 2000-09-08 2004-06-29 Corel Inc. Method and apparatus for manipulating data during automated data processing
US6850956B1 (en) 2000-09-08 2005-02-01 Corel Inc. Method and apparatus for obtaining and storing data during automated data processing
US6868193B1 (en) 2000-09-08 2005-03-15 Corel Inc. Method and apparatus for varying automated data processing
US6925593B1 (en) 2000-09-08 2005-08-02 Corel Corporation Method and apparatus for transferring data during automated data processing
US6938030B1 (en) 2000-09-08 2005-08-30 Corel Corporation Method and apparatus for facilitating accurate automated processing of data
US6944865B1 (en) 2000-09-08 2005-09-13 Corel Corporation Method and apparatus for saving a definition for automated data processing
US6961922B1 (en) 2000-09-08 2005-11-01 Corel Corporation Method and apparatus for defining operations to be performed during automated data processing
US7000223B1 (en) 2000-09-08 2006-02-14 Corel Corporation Method and apparatus for preparing a definition to control automated data processing
JP2007158757A (en) * 2005-12-06 2007-06-21 Sharp Corp Information editing system
US7296238B1 (en) 2000-09-08 2007-11-13 Corel Corporation Method and apparatus for triggering automated processing of data
US7747673B1 (en) 2000-09-08 2010-06-29 Corel Corporation Method and apparatus for communicating during automated data processing
US9432711B2 (en) 2012-11-19 2016-08-30 John D. Steinberg System and method for creating customized, multi-platform video programming

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5404316A (en) * 1992-08-03 1995-04-04 Spectra Group Ltd., Inc. Desktop digital video processing system
US5644674A (en) * 1993-07-16 1997-07-01 Sony Corporation Imaging parameter recording apparatus imaging method and editing system
US5874988A (en) * 1996-07-08 1999-02-23 Da Vinci Systems, Inc. System and methods for automated color correction
US5982350A (en) * 1991-10-07 1999-11-09 Eastman Kodak Company Compositer interface for arranging the components of special effects for a motion picture production

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5982350A (en) * 1991-10-07 1999-11-09 Eastman Kodak Company Compositer interface for arranging the components of special effects for a motion picture production
US5404316A (en) * 1992-08-03 1995-04-04 Spectra Group Ltd., Inc. Desktop digital video processing system
US5644674A (en) * 1993-07-16 1997-07-01 Sony Corporation Imaging parameter recording apparatus imaging method and editing system
US5874988A (en) * 1996-07-08 1999-02-23 Da Vinci Systems, Inc. System and methods for automated color correction

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7000223B1 (en) 2000-09-08 2006-02-14 Corel Corporation Method and apparatus for preparing a definition to control automated data processing
US6651121B1 (en) 2000-09-08 2003-11-18 Corel Inc. Method and apparatus for facilitating scalability during automated data processing
WO2002021272A2 (en) * 2000-09-08 2002-03-14 Corel Inc. Method and apparatus for enhancing reliability of automated data processing
US6757888B1 (en) 2000-09-08 2004-06-29 Corel Inc. Method and apparatus for manipulating data during automated data processing
US6850956B1 (en) 2000-09-08 2005-02-01 Corel Inc. Method and apparatus for obtaining and storing data during automated data processing
US6868193B1 (en) 2000-09-08 2005-03-15 Corel Inc. Method and apparatus for varying automated data processing
US6925593B1 (en) 2000-09-08 2005-08-02 Corel Corporation Method and apparatus for transferring data during automated data processing
US6938030B1 (en) 2000-09-08 2005-08-30 Corel Corporation Method and apparatus for facilitating accurate automated processing of data
US7296238B1 (en) 2000-09-08 2007-11-13 Corel Corporation Method and apparatus for triggering automated processing of data
US6961922B1 (en) 2000-09-08 2005-11-01 Corel Corporation Method and apparatus for defining operations to be performed during automated data processing
US8694601B2 (en) 2000-09-08 2014-04-08 8324450 Canada Inc. Method and apparatus for communicating during automated data processing
WO2002021272A3 (en) * 2000-09-08 2002-07-18 Corel Inc Method and apparatus for enhancing reliability of automated data processing
US6944865B1 (en) 2000-09-08 2005-09-13 Corel Corporation Method and apparatus for saving a definition for automated data processing
US7747673B1 (en) 2000-09-08 2010-06-29 Corel Corporation Method and apparatus for communicating during automated data processing
US7853833B1 (en) 2000-09-08 2010-12-14 Corel Corporation Method and apparatus for enhancing reliability of automated data processing
US7962618B2 (en) 2000-09-08 2011-06-14 Corel Corporation Method and apparatus for communicating during automated data processing
US8271576B2 (en) 2000-09-08 2012-09-18 Corel Corporation Method and apparatus for communicating during automated data processing
JP2007158757A (en) * 2005-12-06 2007-06-21 Sharp Corp Information editing system
US11671645B2 (en) 2012-11-19 2023-06-06 John Douglas Steinberg System and method for creating customized, multi-platform video programming
US9432711B2 (en) 2012-11-19 2016-08-30 John D. Steinberg System and method for creating customized, multi-platform video programming
US10158901B2 (en) 2012-11-19 2018-12-18 Steinberg John D System and method for creating customized, multi-platform video programming
US11178442B2 (en) 2012-11-19 2021-11-16 John Douglas Steinberg System and method for creating customized, multi-platform video programming

Also Published As

Publication number Publication date
AU2001236845A1 (en) 2001-08-20

Similar Documents

Publication Publication Date Title
JP3219027B2 (en) Scenario editing device
EP0892976B1 (en) Media editing system and method with improved effect management
US7359617B2 (en) Dual mode timeline interface
US6539163B1 (en) Non-linear editing system and method employing reference clips in edit sequences
EP0915469A2 (en) Digital video editing method and system
WO2001060060A1 (en) Control of sequence of video modifying operations
JP2001512266A (en) Method and system for editing or modifying 3D video in a non-linear editing environment
WO1993007554A1 (en) A compositer interface for arranging the components of special effects for a motion picture production
US7318203B2 (en) Selecting image processing functions
JPH0855019A (en) Visual programming method
CA2720256A1 (en) Method for automated television production
CA2697347A1 (en) Defining an interactive user interface
US20060181545A1 (en) Computer based system for selecting digital media frames
CN111798128A (en) Interface operation method of process flow, computer equipment and storage medium
US7484201B2 (en) Nonlinear editing while freely selecting information specific to a clip or a track
EP3246921B1 (en) Integrated media processing pipeline
JPH11327980A (en) Version management device
CN1280717C (en) Generationg of multi-media content structure using marking langaage to describe
CN111798126A (en) Process flow creation method, computer device, and storage medium
US7873260B2 (en) Video and audio processing control
JP2003158710A (en) Video object editing device and its program
JPH0954688A (en) Gui design supporting method
JPH05227477A (en) Moving picture image editing method
CN116993870A (en) Project management method, device, equipment and computer readable storage medium
JPH06309158A (en) State transition specification generator

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP