US9547937B2

US9547937B2 - Three-dimensional annotation system and method

Info

Publication number: US9547937B2
Application number: US13/691,509
Authority: US
Inventors: Jared Sandrew; Jill Hunt
Original assignee: Legend3D Inc
Current assignee: Legend3D Inc
Priority date: 2012-11-30
Filing date: 2012-11-30
Publication date: 2017-01-17
Also published as: US20140152648A1; WO2014085735A1

Abstract

Embodiments enable a three-dimensional annotation system and method that accepts desired depths for regions of input images and annotates two-dimensional/three-dimensional images with three-dimensional annotations for viewing at the desired depth(s) in any three-dimensional manner. Enables rapid and intuitive specification of desired depth and application of depth to regions in the two-dimensional images, or when editing three-dimensional images, as annotated by the three-dimensional annotations having at least one depth associated with the annotation. Enables rapid and intuitive depth augmentation or editing of an input image.

Description

BACKGROUND OF THE INVENTION

Field of the Invention

One or more embodiments of the invention are related to the field of image analysis and image enhancement and computer graphics processing of two-dimensional images into three-dimensional images. More particularly, but not by way of limitation, one or more embodiments of the invention enable a three-dimensional annotation system and method. Embodiments accept a desired depth for a region in a two-dimensional image, or three-dimensional image, and annotate the image with three-dimensional annotations at the desired depth for example. This enables rapid and intuitive depth alteration in three-dimensional images and conversion of two-dimensional images to three-dimensional images by enabling stereographers to specify depths for regions of images in an intuitive manner. Embodiments may display an annotated image with a corresponding stereoscopic image or pair of images for left and right eye viewing, or any other three-dimensional viewing enabled image, such as an anaglyph image.

Description of the Related Art

Three-dimensional images include any type of image or images that provide different left and right eye views to encode depth, some types of three-dimensional images require use of special glasses to ensure the left eye viewpoint is shown to the left eye and the right eye viewpoint is shown to the right eye of an observer. Existing systems that are utilized to convert two-dimensional images to three-dimensional images typically require rotoscoping of images to create outlines of regions in the images. The rotoscoped regions are then individually depth adjusted by hand to produce a left and right eye image, or single anaglyph image, or other three-dimensionally viewable image, such as a polarized three-dimensional image viewed with left and right lenses having different polarization angles for example. There is no easy way for stereographers to specify specific depths to apply to regions in a natural way. Thus, ad hoc depths are applied to images, and if the images are not acceptable, for example after client review, then there is no easy manner in which to provide easy feedback. Without a visual language to give creative and technical notes for the placement of objects, feedback can be unclear, causing more creative and technical iteration.

In addition, typical methods for converting movies from 2D to 3D in an industrial setting capable of handling the conversion of hundreds of thousands of frames of a movie with large amounts of labor or computing power, make use of an iterative workflow. The iterative workflow includes rotoscoping or modeling objects in each frame, adding depth and then rendering the frame into left and right viewpoints forming an anaglyph image or a left and right image pair. If there are errors in the edges of the masked objects for example, then the typical workflow involves an “iteration”, i.e., sending the frames back to the workgroup responsible for masking the objects, (which can be in a country with cheap unskilled labor half way around the world), after which the masks are sent to the workgroup responsible for rendering the images, (again potentially in another country), wherein rendering is accomplished by either shifting input pixels left and right for cell animation images for example or ray tracing the path of light through each pixel in left and right images to simulate the light effects the path of light interacts with and for example bounces off of or through, which is computationally extremely expensive. After rendering, the rendered image pair is sent back to the quality assurance group. It is not uncommon in this workflow environment for many iterations of a complicated frame to take place. This is known as “throw it over the fence” workflow since different workgroups work independently to minimize their current workload and not as a team with overall efficiency in mind. With hundreds of thousands of frames in a movie, the amount of time that it takes to iterate back through frames containing artifacts can become high, causing delays in the overall project. Even if the re-rendering process takes place locally, the amount of time to re-render or ray-trace all of the images of a scene can cause significant processing and hence delays on the order of at least hours. Each iteration may take a long period of time to complete as the work may be performed by groups in disparate locations having shifted work hours. Elimination of iterations such as this would provide a huge savings in wall-time, or end-to-end time that a conversion project takes, thereby increasing profits and minimizing the workforce needed to implement the workflow.

Hence there is a need for a three-dimensional annotation system and method.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the invention accept inputs from a stereographer that indicate depths at which to place regions or object volumes within a two dimensional image that are utilized to create stereoscopic viewable images, e.g., two horizontally offset left and right eye viewpoints or images. In one or more embodiments of the invention, the input is accepted by the system and displayed at the depth indicated on the three-dimensional version of the two-dimensional input image. In one or more embodiments, the depth may be specified using a graphical input device, such as a graphics drawing tablet. In other embodiments or in combination, depths may be input via a keyboard, obtained through analysis of the input, e.g., script or text to annotate with, or via voice commands while drawing annotation information or symbols for example.

In one scenario of the conversion workflow, a mask group takes source images and creates masks for items, areas or human recognizable objects in each frame of a sequence of images that make up a movie. Stereographers utilize embodiments of the invention to specify depths, for example with annotations that are shown at the desired depth along with any other information, to apply to particular regions, for example the masked regions from the mask group, in each image. The depth augmentation group applies the specified depths, and for example shapes, to the masks created by the mask group. Embodiments of the invention make this process extremely intuitive as the depth to apply is shown with information at the desired depth. Optionally, the depth may be applied before or independent of the masking process for example.

When rendering an image pair, left and right viewpoint images and left and right absolute translation files, or a single relative translation file may be generated and/or utilized by one or more embodiments of the invention. The translation files specify the pixel offsets for each source pixel in the original 2D image, for example in relative or absolute form respectively. These files are generally related to an alpha mask for each layer, for example a layer for an actress, a layer for a door, a layer for a background, etc. These translation files, or maps are passed from the depth augmentation group that renders 3D images, to the quality assurance workgroup or depending on the project size, a stereographer and/or associate stereographer. This allows the quality assurance workgroup (or other workgroup such as the depth augmentation group) to perform real-time editing of 3D images without re-rendering for example to alter layers/colors/masks and/or remove artifacts such as masking errors without delays associated with processing time/re-rendering and/or iterative workflow that requires such re-rendering or sending the masks back to the mask group for rework, wherein the mask group may be in a third world country with unskilled labor on the other side of the globe. In addition, when rendering the left and right images, i.e., 3D images, the Z depth of regions within the image, such as actors for example, may also be passed along with the alpha mask to the quality assurance group, who may then adjust depth as well without re-rendering with the original rendering software. This may be performed for example with generated missing background data from any layer so as to allow “downstream” real-time editing without re-rendering or ray-tracing for example.

Quality assurance may give feedback to the masking group or depth augmentation group for individuals so that these individuals may be instructed to produce work product as desired for the given project, without waiting for, or requiring the upstream groups to rework anything for the current project. This allows for feedback yet eliminates iterative delays involved with sending work product back for rework and the associated delay for waiting for the reworked work product. Elimination of iterations such as this provide a huge savings in wall-time, or end-to-end time that a conversion project takes, thereby increasing profits and minimizing the workforce needed to implement the workflow.

In summary, embodiments of the invention minimize iterative workflow by providing more intuitive instructions regarding depth for another workgroup to utilize. For example, embodiments of the invention enable eliminate iterative workflow paths back through different workgroups by enabling other workers or workgroups to have an intuitive method in which to view depth instructions and successfully input the correct depth. Great amounts of time are saved by eliminating re-rendering by other work groups, and allow depth to be correctly input local to a work group. Embodiments of the system thus greatly aid the artist in the enhancement of images to include depth by providing realistic depth information once, to minimize manual manipulation of images.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

FIG. 1 shows an architectural view of an embodiment of the system.

FIG. 2 shows an input two-dimensional image.

FIG. 3 shows a masked version of the two-dimensional image showing regions within each object to apply depth to.

FIG. 4 shows annotations for desired depth at a specific depth for general messages, or at the depth of the desired region for example, wherein the annotations may be viewed in three-dimensional depth with anaglyph glasses.

FIG. 5 shows the input image converted to three-dimensional image in anaglyph format, which may be viewed in three-dimensional depth with anaglyph glasses to view separate left and right eye viewpoints from one image.

FIG. 6 shows a logical side view of the depth applied to the annotations and optionally to the regions that may be masked for example and depth augmented as per the associated annotation.

FIG. 7 illustrates a flowchart illustrating an embodiment of the method implemented by one or more embodiments of the system of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows an architectural view of an embodiment of the system 100. As illustrated, computer 101 is coupled with any combination of input devices including graphics tablet 102 a, keyboard 102 b, mouse 102 c and/or microphone 102 d. Computer 101 may obtain a two-dimensional image and display the image on screen 103. Screen 103 may display a single image that may be viewed at depth, for example as an anaglyph using two different colors shifted left and right that may be viewed with glasses with lenses of two different colors, e.g., Red and Blue to view the image as a three-dimensional image for example. In general, the two-dimensional image may have multiple regions that are to be converted to different depths, for example first region 151 a, e.g., a fish and second region 151 b, coral for ease of illustration. In other scenarios, embodiments of the invention may be utilized to amend or otherwise change or alter depth of three-dimensional input images. Other embodiments may be utilized to annotate convergence for blending a feature film and/or alteration of native stereo elements with positive or negative depth with respect to the screen plane for example. Regardless of the input image type, embodiments of the system accept annotation associated with desired depths from input devices 102 a-d, for example first annotation 152 a and second annotation 152 b. Any number of regions or annotations may be accepted by embodiments of the system. In one or more embodiments of the invention, the annotation itself may be analyzed to obtain the desired depth associated with a given region, or any input from the same or other input device may be utilized to obtain the desired depth. The annotation is then placed at the depth thus obtained, which results in three-

dimensional annotations

152 a and 152 b displayed at the depth thus obtained from the annotation itself, e.g., from numbers in the annotation via optical character recognition or other handwriting recognition software for example. The depth may be the desired depth of an associated object or for example the depths of the four corners of the screen or any other depth associated with the annotation for example. The annotation may include general comments at a particular depth and not associated with a specific region for example.

In one or more embodiments of the invention, obtaining the depth includes analyzing the annotation with text recognition software to determine the depth. For example, if mouse or

graphics tablet

102 c or 102 a is utilized to cursively drawing the annotation, the input may be analyzed by text recognition software to determine if a numerical value exists within the cursive text, for example “10” or “5” as obtained from

annotations

152 a and 152 b. In addition, keywords or characters such as “+”, “−”, “forward”, “back”, etc., may be obtained via text recognition software and applied to the depth of the annotation automatically for example. Alternatively, or in combination, the mouse input may be utilized to for example drag up or down to adjust the annotation and add text next to an arrow annotation for example to show that the annotation is “10” or “5”, for example which changes as the mouse is dragged and automatically updated in the annotation, while the annotation itself moves forward or backward. Alternatively, or in combination, the keyboard may be utilized to accept annotations or depths associated with annotations. The input text from the keyboard may be parsed to obtain keywords, characters or numbers for example to automatically either augment the annotation or move the annotation in depth or both. Alternatively, or in combination, the microphone may be monitored and depth accepted therefrom to automatically augment the annotation or move the annotation in depth or by asserting voice recognition software to determine keywords, characters or numbers.

Embodiments of the system may thus be utilized in obtaining a two-dimensional source image, displaying the two-dimensional source image on the screen associated with the computer, accepting an annotation associated with a desired depth of a region within the two-dimensional source image via any of the input devices coupled with the computer, obtaining a depth associated with the annotation as described above, and annotating the two-dimensional image with the annotation at the depth in a three-dimensional image, i.e., an image that has at least the annotations displayed at depth.

Embodiments of the system annotate the two-dimensional image with the annotation at the depth by generating an image encoded with left and right viewpoints or a pair of images comprising an image for viewing with a left and right eye respectively wherein the pair of images includes the annotation and the two-dimensional source image. In one or more embodiments the resulting image is a single anaglyph image, or polarized image, or any other type of image that includes the annotation shown at depth along with the two-dimensional source image.

Before or after accepting the desired depth of any portions or regions of a two-dimensional image, the computer or any other computer that may access the resulting annotated image, may accept at least one mask associated with the region of the two-dimensional source image. In other words, masking may take place before or after the annotation of the two-dimensional image. Embodiments of the system may then displace at least a portion of the region, for example a particular side, or middle or any other portion, in the two-dimensional source image left and right based on the depth to create a resulting output three-dimensional image.

When the resulting depth appears to be acceptable based on the requirements of the particular project, the system may output a three-dimensional image without the annotation. In movie-based projects, this may entail large numbers of images and tweening for example between key frames or other images generated with one or more embodiments of the system.

FIG. 2 shows an input two-dimensional image. Embodiments of the invention may be utilized on cell animation or photographic or rendered or any other type of images. As shown, an exemplary object such as a fish is shown near vertically oriented structures, which may represent coral or other structures. FIG. 3 shows a masked version of the two-dimensional image showing regions within each object to apply depth to. In one or more embodiments, the regions are utilized to apply depths that vary over the region to create regions that are not flat, i.e., not at the same depth across the entire region. As shown, region 151 a includes many sub-regions or masks, shown as different colors along the sides and back of the fish, which are not shown in the unmasked version of FIG. 2. FIG. 4 shows annotations for desired depth at a specific depth for general messages or at the depth of the desired region for example, wherein the annotations may be viewed in three-dimensional depth with anaglyph glasses. As shown, the two-dimensional image is still in two-dimensions, i.e., the depth across the entire image does not vary. In other words, the two-dimensional image along with the three-dimensional annotations specify the depths to apply to particular areas or regions and is used as an input to the depth augmentation group for example. The depth group then moves the associated regions in depth to match the annotations in an intuitive manner that is extremely fast and provides a built-in sanity check for depth. Using this method, it is inherently verifiable whether a depth of a region is at or about at the depth of the associated annotation.

FIG. 5 shows the input image converted to three-dimensional image in anaglyph format, which may be viewed in three-dimensional depth with anaglyph glasses. As shown, the individual coral pieces are at the specified depths, for example, nearest ones at “−10” at region 151 b having associated annotation 152 b, with the furthest ones at “4”, “5”, and “7”, while the region of the nose of the fish 151 a is at “0” and the fins are at offset “−2” as shown associated with annotation 152 a. In one or more embodiments these numbers may indicate the left and right shift in pixels or the depth in feet/meters of the particular regions, or any other quantitative value associated with distance or depth. In other embodiments of the invention, the polarity may be such that positive numbers represent depths further away from the viewer.

As illustrated, embodiments of the invention minimize iterative workflow by providing more intuitive instructions regarding depth for another workgroup to utilize. Thus, the system and method implemented by the system eliminate iterative workflow paths back through different workgroups by enabling other workers or workgroups to have an intuitive method in which to view depth instructions and successfully input the correct depth. Great amounts of time are saved by eliminating re-rendering by other work groups, and allow depth to be correctly input local to a work group. Embodiments of the system thus greatly aid the artist in the enhancement of images to include depth by providing realistic depth information once, to minimize manual manipulation of images.

In one or more embodiments, a particular annotation may itself have a differing depth along the annotation to show how a depth varies, i.e., is not constant or flat across a region. For example, an annotation may show a curve from a first depth to a second depth along the annotation so that the annotation has a depth range. In this case more than one number for depth may be associated with a particular annotation and analyzed by the system to shift a portion of the annotation nearer or further than another portion of the same annotation. There is no limit to the number of depths that a particular annotation may be placed at. As shown in FIG. 5, the bottom right annotation shows a depth of −14 and −4 with a “far” depth of −20, which is analyzed by an embodiment of the invention to designate that region of the image as having a depth that ranges between the three annotated depths, wherein an embodiment of the invention may thus set the depth of the masked region to as shown by shifting closer annotated portions farther left and right that deeper areas respectively.

FIG. 6 shows a logical side view of the depth applied to the annotations and optionally to the regions that may be masked for example and depth augmented as per the associated annotation. FIG. 6 illustrates the depth applied to FIGS. 1 and 2 from a side view of screen 103 to show the depth applied to the

annotations

152 a and 152 b (see also FIG. 4 with anaglyph glasses on), and optionally to the associated

regions

151 a and 151 b, once the associated depth notated in the annotations is applied to the regions (see also FIG. 5 with anaglyph glasses on). As shown, the annotations are at depth for three-dimensional or stereoscopic viewing 602 to aid in the application of depth to the associated regions for example wherein a viewer 601 is shown at the right side of screen 103.

FIG. 7 illustrates a flowchart illustrating an embodiment of the method implemented by one or more embodiments of the system of FIG. 1. As shown, the method includes obtaining the source image at 701, displaying the source image on the screen of the computer shown in FIG. 1, as per 702, accepting annotation associated with the desired depth of the region at 703, obtaining a depth associated with the annotation at 704 in a number of ways previously described with respect to the system, annotating the source image with the annotation in three-dimensions, for stereoscopic viewing at 705. From the viewpoint of depth workers viewing the annotations at depth, the annotations are utilized to show where depth should be applied and the system may accept masks for regions in the source image at 706 and then optionally display the regions as well at 707, and which is shown in FIG. 6. Although the annotations may not be at the same depth as the associated regions or may not even have associated regions, i.e., may simply be annotations at depth to aid in understanding something associated with the source image, the annotations at depth greatly speed and aid the process of working on images that may include depth.

While the invention herein disclosed has been described by means of specific embodiments and applications thereof, numerous modifications and variations could be made thereto by those skilled in the art without departing from the scope of the invention set forth in the claims.

Claims

What is claimed is:

1. A three-dimensional annotation method comprising:

obtaining a source image that is two-dimensional or three-dimensional;

displaying said source image on a screen associated with a first computer;

accepting an annotation associated with a desired depth of a region within said source image via an input device coupled with said first computer;

obtaining at least one depth associated with said annotation;

wherein said at least one depth corresponds with said desired depth of said region; and,

annotating said source image with said annotation at said at least one depth in a three-dimensional image;

generating an annotated stereoscopic image that comprises left and right eye views that differ from each other having said annotation at said at least one depth that differs from a depth of said region; and,

generating an output stereoscopic image with said region at said same depth as said at least one depth of said annotation.

2. The method of claim 1 wherein said input device comprises a graphics tablet, a mouse, or a keyboard and wherein said accepting said annotation comprises accepting input from said graphics tablet, said mouse or said keyboard respectively.

3. The method of claim 1 wherein said input device comprises a microphone and wherein said accepting said annotation comprises accepting input from said microphone.

4. The method of claim 1 wherein said obtaining said at least one depth comprises analyzing said annotation with text recognition software to determine said at least one depth.

5. The method of claim 1 wherein said obtaining said at least one depth comprises analyzing motion of a mouse to determine said at least one depth.

6. The method of claim 1 wherein said obtaining said at least one depth comprises parsing alphanumeric data from a keyboard to determine said at least one depth.

7. The method of claim 1 wherein said obtaining said at least one depth comprises asserting voice recognition software.

8. The method of claim 1 wherein said generating said annotated stereoscopic image comprises generating a pair of images comprising a left image to view with a left eye and a right eye image to view with a right eye respectively wherein said pair of images includes said annotation and said source image.

9. The method of claim 1 wherein said generating said annotated stereoscopic image comprises generating an anaglyph image comprising a left eye colored image and a right eye colored image that are combined and that includes said annotation and said source image.

10. The method of claim 1 wherein said generating said annotated stereoscopic image comprises generating a polarized image comprising a left eye image polarized in a first axis and a right eye image polarized in a second axis orthogonal to said first axis that are combined and that includes said annotation and said source image.

11. The method of claim 1 wherein said generating said annotated stereoscopic image comprises generating a single image capable of displaying left and right eye viewpoints to a left eye and right eye respectively with differing depths that includes said annotation and said source image.

12. The method of claim 1 further comprising:

accepting at least one mask associated with said region of said source image.

13. The method of claim 1 further comprising:

displacing at least a portion of said region in said source image left and right based on said at least one depth to create said three-dimensional image.

14. The method of claim 1 further comprising:

displacing at least a portion of said region in said source image left and right based on said at least one depth to create an output three-dimensional image without said annotation.

15. The method of claim 1 wherein said annotating said source image with said annotation at said at least one depth occurs before moving at least a portion of said region in said source image left and right to alter depth within the source image.

16. The method of claim 1 wherein said annotating said source image with said annotation at said at least one depth comprises annotating said source image with a plurality of annotations that each comprise a different depth.

17. A three-dimensional annotation method comprising:

obtaining a source image that is two-dimensional or three-dimensional;

displaying said source image on a screen associated with a first computer;

accepting an annotation associated with a desired depth of a region within said source image via an input device coupled with said first computer wherein said input device comprises any combination of graphics tablet, mouse, keyboard or microphone;

obtaining a at least one depth associated with said annotation by analyzing said annotation with text recognition software or by analyzing motion of a mouse or by parsing alphanumeric data from said keyboard or by asserting voice recognition software or any combination thereof;

wherein said at least one depth corresponds with said desired depth of said region;

and, annotating said source image with said annotation at said at least one depth in a three-dimensional image wherein said annotating said source image with said annotation at said at least one depth occurs before moving at least a portion of said region in said source image left and right to alter depth within the source image; and,

18. The method of claim 17 further comprising:

accepting at least one mask associated with said region of said source image.

19. The method of claim 17 further comprising:

20. The method of claim 17 further comprising:

21. The method of claim 17 wherein said annotating said source image with said annotation at said at least one depth comprises annotating said source image with a plurality of annotations that each comprise a different depth.