US20070183679A1 - Image processing method, image processing device and mobile communication terminal - Google Patents

Image processing method, image processing device and mobile communication terminal Download PDF

Info

Publication number
US20070183679A1
US20070183679A1 US11/498,983 US49898306A US2007183679A1 US 20070183679 A1 US20070183679 A1 US 20070183679A1 US 49898306 A US49898306 A US 49898306A US 2007183679 A1 US2007183679 A1 US 2007183679A1
Authority
US
United States
Prior art keywords
image
information
noted
mask
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US11/498,983
Other versions
US7864198B2 (en
Inventor
Mineko Moroto
Akiho Shibata
Takashi Katayama
Junko Asakura
Yusaku Nagai
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vodafone Group PLC
Original Assignee
Vodafone KK
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vodafone KK filed Critical Vodafone KK
Assigned to VODAFONE K.K. reassignment VODAFONE K.K. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ASAKURA, JUNKO, MOROTO, MINEKO, NAGAI, YUSAKU, KATAYAMA, TAKASHI, SHIBATA, AKIHO
Publication of US20070183679A1 publication Critical patent/US20070183679A1/en
Assigned to SOFTBANK MOBILE CORP. reassignment SOFTBANK MOBILE CORP. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: VODAFONE K.K.
Assigned to VODAFONE GROUP PLC reassignment VODAFONE GROUP PLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SOFTBANK MOBILE CORP.
Application granted granted Critical
Publication of US7864198B2 publication Critical patent/US7864198B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/002D [Two Dimensional] image generation
    • G06T11/60Editing figures and text; Combining figures or text
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3871Composing, repositioning or otherwise geometrically modifying originals the composed originals being of different kinds, e.g. low- and high-resolution originals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3872Repositioning or masking

Definitions

  • the present invention relates to an image processing method, an image processing device and a mobile communication terminal, and in more detail relates to an image processing method for forming a composite image having a mask image superimposed on an original image, an image processing device using the image processing method, and a mobile communication terminal provided with the image processing device.
  • Image processing technology of the above described related art such as image recognition, image authentication, image tracking etc. are extremely good at being capable of application to a moving picture, and various application have been considered.
  • a user processes an original image according to their own preferences, and expressing their own innermost thoughts is not within the technical scope. For this reason, for example, even if the image processing technology of the related art is applied to video telephone, it is insufficient from the point of view of holding an enjoyable conversation in accordance with an image that reflects the interests of the transmission side caller while demonstrating the function of the video telephone.
  • the present invention has been conceived in view of the above described problems, and an object of the invention is to provide an image processing method capable of simply and appropriately forming an image that a user has produced on an original image, regardless of whether the original image is a moving picture or a still image.
  • Another object of the present invention is to provide a mobile communication terminal capable of transmitting an image the user has appropriately produced for an original image, regardless of whether the original image is a moving picture or a still image.
  • An image processing method of the present invention is an image processing method for carrying out processing of an image, and comprises the steps of: specifying a noted image region, being a region for an image to be focused on, in an original image, and extracting noted information containing information relating to position and size of the noted image within the original image; forming a mask image to be displayed superimposed display on the original image, based on a mask image information file containing the noted image information, at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information of specified element images respectively corresponding to the relationship information, and at least one element image information selected using the element image selection information; and creating a composite image having the mask image superimposed on a noted image comprising image that includes the noted image of the noted image region at the same location and same size as for the original image.
  • At least one relationship information representing a relationship between position and size of the noted image region, element image selection information representing selection of specified element image information respectively corresponding to the relationship information, and mask image information containing at least one element image information selected using the element image selection information, are registered in a predetermined mask image information file.
  • “noted image region” is a region for a face in the original image containing a figure, for example, or a region capable of being specified by pictorial characteristics within an image assumed to be obtained from an original image.
  • a mask image information file As a premise for the existence of a mask image information file, first of all, in a noted image extraction step, the noted image region is specified in the original image, and noted image information containing information relating to the position and size of the noted image region within the original image is extracted. Continuing on, in a mask image data creation step, a mask image is created by displaying superimposed on the original image, based on noted image information and mask image information registered in the mask image information file.
  • a mask image formed from element images selected using element image selection information contained in mask image information is formed on the original image at a position determined using information connected to position of a noted image region and information representing a relationship between noted image region in relationship information contained in mask image information, at a size determined by information representing a relationship between information connected to size of the noted position in the original image and size of the noted image region in the relationship information.
  • a composite image creating step a composite image having the mask image superimposed on a noted image comprising image that includes a noted image inside the noted image region, such as a facial image within a facial image region, for example, at the same location and same size as for the original image, is formed.
  • the image processing method of the present invention it is possible to simply and appropriately form an image that a user has produced according their interests for a noted image comprising image, regardless of whether the original image is a moving picture or a still image.
  • the image processing method of the present invention it is possible to make the noted image comprising image the original image.
  • rendering of the original image according to the user's interests can be carried out by selecting a mask image.
  • the mask image information file further comprises background image selection information representing selection of information for a specified background image
  • background image selection information representing selection of information for a specified background image
  • a specified display image containing the noted image is cut from the original image, and the composite image can be formed after a background addition image has been made the noted image comprising image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
  • a specified display image containing a noted image such as a figure is cut from the original image for the case where the noted image is a facial image.
  • the specified display image is superimposed on a background image designated using background image selection information contained in the mask image information file, to form a background addition image.
  • the above described mask image is then superimposed on this noted image comprising image, with the background addition image as the noted image comprising image. Accordingly, rendering of the specified display image, such as a figure and so forth, according to the user's interests can be carried out by selecting a mask image and a background image.
  • the information relating to the position of the noted image region may contain position information for a representative point of the noted image region of the original image.
  • position information for a representative point of the noted image region of the original image it is possible to make information representing a relationship with position of a noted region, in relationship information contained in the mask image information, information representing a relative relationship for a representative point of the noted image, and it is possible to simplify information representing a relationship with position of the noted image region in the relationship information.
  • the information relating to the position of the noted image region can further contain inclination information for the noted image region of the original image.
  • inclination information for the noted region of the original image it is possible to form a composite image displaying a desired element image superimposed at a position corresponding to the inclination of the noted image region in the original image. It is also possible to cause display by holding inclination in the displayed element image corresponding to inclination of the noted image region in the original image.
  • the inclination information can be a single angle information representing a two dimensional inclination of the noted image, or two or three angle information representing three dimensional inclination of the noted image.
  • representative point of the noted image region is made the origin in the relationship information, and it is possible for coordinate position data of a coordinate system determining direction of a coordinate axis based on inclination of the noted image region to be included as display position information for the specified element image.
  • the information in the relationship information with respect to a representative point of the noted image region can be simply interpreted as coordinate position information of a coordinate system that determined by direction of coordinate axis based on inclination of the noted image region, wherein the system has an origin as a representative point of the noted image region.
  • the noted information is the information representing a relative position of information relating to the position.
  • the relationship information it is possible for the relationship information to contain proportion information for size of the noted image region and size of the specified graphic at the time of display. In this case, even when size of the noted image within the original image is changed from moment to moment, it is possible to form a mask image containing element images of size appropriate to the size of the noted image region.
  • the mask image information may further comprise respective execution time information for element image display fixed by combining the relationship information and element image selection information corresponding to the relationship information.
  • the mask image information by forming a mask image according to the execution time information, it is possible to display respective desired element images for only display execution times set to respective to desired times, and it is possible to form a mask image to vary the dynamic.
  • the mask image information further comprises information for display of a plurality of types of element image display in time sequence, fixed by combining the relationship information and element image selection information corresponding to the relationship information, and for time sequence display designation for designating repeat of time sequence display.
  • information for display of a plurality of types of element image display in time sequence fixed by combining the relationship information and element image selection information corresponding to the relationship information, and for time sequence display designation for designating repeat of time sequence display.
  • time sequence display designation information it is possible to display a plurality of types of element image on the original image repeatedly in time sequence every desired time.
  • the mask image information to further comprise simultaneous display designation information for designating display of a plurality of types of element image display at the same time, fixed by combining the relationship information and element image selection information corresponding to the relationship information.
  • simultaneous display designation information for designating display of a plurality of types of element image display at the same time, fixed by combining the relationship information and element image selection information corresponding to the relationship information.
  • the image processing method of the present invention it is possible to make the original image a moving picture.
  • a moving picture in which position, inclination and size of the noted image in the original image is changing it is possible to form a composite image that is dynamic and not unpleasant, in accordance with the user's interests.
  • a fixed phenomenon occurrence determination step for determining whether or not a predetermined phenomenon has occurred in the noted image region based on the noted image information is further provided, and the mask image data formation step and composite image data formation step are commenced after positive determination in the specified phenomenon occurrence determination step.
  • the fixed phenomenon occurrence detection step when a predetermined phenomenon has occurred in the noted image region, since the mask image data formation step and the composite image formation step are commenced, it is possible to form a mask image where rendering is carried out in accordance with that phenomenon, and it is possible to form a composite image having that mask image superimposed on the original image.
  • the eyelids in the noted image can be closed, it is possible to form a composite image with tears flowing from the eyes.
  • An image processing device of the present invention is an image processing device for carrying out processing of an image, and comprises noted image information extracting means for specifying a noted image region, being a region for an image to be focused on, in an original image, and extracting noted information relating to position and size of the noted image region within the original image, storage means for storing a mask image information file registering mask image information containing at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information for a specified element image respectively corresponding to the relationship information, and at least one element image information selected using the element image selection information, mask image formation means for forming a mask image displayed superimposed on the original image, based on the noted image information and the mask image information, and composite image creation means for creating a composite image having the mask image superimposed on a noted image comprising image that includes the noted image inside the noted image region at the same location and same size as for the original image.
  • noted image information extracting means specifies a noted image region in an original image, and extracts noted information relating to position and size of the noted image region within the original image.
  • mask image data formation means forms a mask image by displaying superimposed on the original image, based on noted image information and mask image information registered in the mask image information file.
  • composite image formation means forms a composite image having the mask image superimposed on a noted image comprising image that includes the noted image within the noted image region at the same location and same size as for the original image.
  • a composite image is formed using the image processing method of the present invention. Accordingly, according to the image processing device of the present invention it is possible to simply and appropriately form an image that a user has rendered on the noted image comprising image according to their interests, regardless of whether the original image is a still image or a moving picture.
  • the noted image comprising image can comprise the original image.
  • rendering of the original image according to the user's interests can be carried out by selecting a mask image.
  • the image processing device of the present invention it is possible to have a structure where the mask image information file further comprises background image selection information representing selection of information for a specified background image, and the composite image formation means cuts a specified display image containing the noted image from the original image, to form the composite image after a background addition image has been made the noted image comprising image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
  • the composite image formation means first of all extracts a specified display image containing a noted image from the original image. Continuing on, the composite image formation means superimposes the specified display image on a background image designated using background image selection information contained in the mask image information file, to form a background addition image. The composite image formation means then superimposes the above described mask image on the background addition image. Accordingly, rendering of the specified display image, such as a figure and so forth, according to the user's interests can be carried out by selecting a mask image and a background image.
  • the image processing device of the present invention it is possible have a structure further comprising display means for displaying the composite image.
  • display means for displaying the composite image.
  • the image processing device of the present invention it is possible have a structure further comprising display means for displaying the composite image.
  • display means for displaying the composite image because it is possible to transmit a composite image formed by a user of the composite image carrying out rendering according to their interests at the transmission side towards the receive side, it is possible to notify a composite image formed by the user rendering according to their interests to the receiving party.
  • an image processing device of the present invention it is possible have a structure further comprising imaging means for acquiring the original image.
  • imaging means for acquiring the original image.
  • an image acquired by the imaging means it is also possible for an image acquired by the imaging means to be a still image or a moving picture.
  • a mobile communication terminal of the present invention is a mobile communication terminal provided with the imaging processing device of the present invention.
  • this mobile communication terminal since it is possible to transmit a composite image formed by the image processing device of the present invention to a receive side terminal, it is possible to transmit an image that a user renders in an original image according to their tastes regardless of whether it is a still image or a moving picture.
  • the image processing method of the present invention it is possible to simply and appropriately form an image that a user rendered in an original image, regardless of whether it is a still image or a moving picture.
  • the image processing device of the present invention since it is possible to form a composite image using the image processing method of the present invention, it is possible to easily and appropriately form an image that was rendered by a user in an original image regardless of whether the original image is a moving picture or a still image.
  • the image processing device of the present invention since the image processing device of the present invention is provided it is possible to transmit an image a user appropriately produced for an original image, regardless of whether it is a moving picture or a still image.
  • FIG. 1A schematically shows the external structure of a front side of a mobile telephone unit of a first embodiment of the present invention
  • FIG. 1B schematically shows the external structure of a rear side of a mobile telephone unit of the first embodiment of the present invention
  • FIG. 2 is functional block diagram for describing the internal structure of the mobile telephone unit of FIG. 1A and FIG. 1B ;
  • FIG. 3 is functional block diagram for describing the internal structure of a mask image processing section of FIG. 2 ;
  • FIG. 4A is a first drawing for describing noted image region information.
  • FIG. 4B is a second drawing for describing noted image region information
  • FIG. 5A is drawing for describing the mask image information file structure of FIG. 2 ;
  • FIG. 5B is drawing for describing the element image file structure of FIG. 5A ;
  • FIG. 6 is a first drawing showing an example of an element image
  • FIG. 7 is a first drawing showing example contents of the display designation file of FIG. 5A ;
  • FIG. 8A is a second drawing showing example contents of the display designation file of FIG. 5A ;
  • FIG. 8 b is a third drawing showing example contents of the display designation file of FIG. 5A ;
  • FIG. 9A is a drawing showing an example of a noted image
  • FIG. 9B is a drawing showing an example for describing the noted image coordinate system in FIG. 9A ;
  • FIG. 10A is a drawing showing an example of an original image
  • FIG. 10B is a first drawing showing an example of a mask image
  • FIG. 11 is a first drawing showing an example of a composite image of the first embodiment
  • FIG. 12A is a second drawing showing an example of an element image
  • FIG. 12B is a third drawing showing an example of an element image
  • FIG. 12C is a fourth drawing showing an example of an element image
  • FIG. 13 is a second drawing showing an example of a mask image
  • FIG. 14 is a second drawing showing an example of a composite image of the first embodiment
  • FIG. 15A is a fifth drawing showing an example of an element image
  • FIG. 15B is a sixth drawing showing an example of an element image
  • FIG. 15C is a seventh drawing showing an example of an element image
  • FIG. 16 is a third drawing showing an example of a mask image
  • FIG. 17 is a third drawing showing an example of a composite image of the first embodiment
  • FIG. 18 is a functional block diagram for describing the internal structure of a mobile telephone unit of a second embodiment of the present invention.
  • FIG. 19 is a functional block diagram for describing the internal structure of a mask image processing section of FIG. 18 ;
  • FIG. 20 is a drawing for describing the mask image information file structure of FIG. 18 ;
  • FIG. 21 is a drawing showing example contents of the display designation file of FIG. 20 ;
  • FIG. 22 is a drawing showing an example of a background image
  • FIG. 23 is a drawing showing an example of a specified display image.
  • FIG. 24 is a flowchart for explaining a composite image formation operation for composite image formation section in FIG. 19 .
  • FIG. 1A A first embodiment of the present invention will be described in the following with reference to FIG. 1A - FIG. 17 .
  • the structure of a mobile telephone unit 10 is shown schematically in FIG. 1 and FIG. 2 .
  • a front view of the external appearance of the mobile telephone unit 10 is shown in FIG. 1A
  • a rear view of the external appearance of the mobile telephone unit 10 is shown in FIG. 1B .
  • the functional block structure of the mobile telephone unit 10 is shown in FIG. 2 .
  • the portable telephone unit 10 is comprised of (a) a portable telephone body 11 provided with a control section 21 and so forth, which will be described later, (b) an operating section 12 having a numerical keypad for input of telephone numbers and function keys for inputting various commands such as operation mode switching, to the control section 21 , and (c) a display section 13 having a liquid crystal display for displaying operation guidance, operating status, received messages etc. in response to commands from the control section 12 .
  • the portable telephone 10 is also comprised of (d) a call speaker 14 for reproducing voice signals transmitted from a calling party at the time of a call and a microphone 15 for inputting sound at the time of a call, and (e) a guidance speaker 16 for generating ring alerts and guidance voice in response to commands from the control section 21 . Further, the portable telephone unit 10 is further comprised of (f) an antenna 17 connected to a transmission section 22 for transceiving wireless signals between the unit and a base station.
  • the portable telephone unit 10 comprises (g) an imaging optical system for taking pictures of an operator side operating the operating section 12 while looking at the display on the display section 13 , that is, for carrying photography in a so-called self-facing state, and (h) an imaging optical system 19 for taking pictures facing towards the field of vision of the operator, that is, taking pictures in a so-called other-party state.
  • the portable telephone body 11 comprises (i) a control section 21 for unified control of all operations of the portable telephone unit 10 , and also performing various data processing, (ii) a transceiver section 22 for transceiving wireless signals to and from a base station via the antenna 17 , and (iii) a storage section 23 for storing various data.
  • the portable telephone body 11 also comprises a switching optical system 24 for making either light passing through the imaging optical system 18 or light passing through the imaging optical system 19 be emitted towards an imaging element 25 that will be described later, under the control of the control section 21 , and (iv) an imaging element 25 such as a CCD element for imaging an optical image formed by light on a light receiving surface via the switching optical system 24 .
  • the imaging element 25 carries out imaging operations under the control of the control section 21 , and imaging results are notified to the control section 21 .
  • the control section 21 is provided with a basic control section 29 and a mask image processing section 30 .
  • the basic control section 29 controls starting and stopping of operation of the mask image processing section 30 , and also controls each element arranged external to the control section 21 and performs data processing besides image processing for mask image formation, that will be described later.
  • the mask image processing section 30 mentioned above comprises a noted image extraction section 31 , a mask image formation section 32 and a composite image formation section 33 .
  • the noted image extraction section 31 specifies a noted image region, such as a facial region of a specified person (for example, the user of the portable telephone unit 10 ) in the original image, for example, based on original image data OMD from the imaging element 25 , and extracts noted image information WMD made up of position information for a representative point of the noted image region in the original image, inclination information of the noted image region in the original image, and size information of the noted image region.
  • Each element position in the original image region OMR is represented using coordinate values of an X S Y S coordinate system, which is the original image coordinate system.
  • Inclination angle ⁇ W is represented by a counterclockwise angle.
  • the noted image information extraction section 31 first of all obtains coordinate position O W (X SW1 ,Y SW1 ) of the original image coordinate system for the representative point O W , as position information for the representative point in the noted image WMR.
  • the representative point of the noted image region WMR it is possible to adopt a distinctive point of a noted image region acquired using image analysis, such as taking a tip position of a nose in a facial image, in the case where the noted image us a facial image of a specified person.
  • the noted image information extraction section 31 obtains a crossing angle ⁇ W of the X S axis and the X W axis (that is, the above described inclination angle ⁇ W ) as inclination information of the noted image WMR in the original image region OMR.
  • Inclination angle ⁇ W is obtained by calculating a difference between a crossing angle of the X S axis and a characteristic direction as a noted image in the noted image region WMR defined with a predetermined inclination angle ⁇ W of “0”, and a crossing angle of the X S axis and the characteristic direction in the noted image inside the original image.
  • the noted image information extraction section 31 obtains a characteristic length relating to the noted image region WMR, as size information for the noted image region WMR.
  • the width of the noted image region on the X W axis is made the characteristic length (width LX W in FIG. 4A ).
  • the size information for the noted image region WMR it is possible to adopt the width of the noted image region WMR on the Y W axis, maximum width of the noted image region WMR in the X W axis (or Y W axis) direction, a distance between two characteristic points in the noted image region (for example, a distance between the centers of two eyes in a facial image when the noted image is a facial image of a specified person), or the length of an X W axis component of the distance or the length of a Y W axis component of the distance, as well as the width LX W of the noted image region WMR on the X W axis.
  • the noted image information extraction section 31 obtains position information, inclination information and size information for the representative point in the noted image WMR.
  • a coordinate position O W (X SW2 ,Y SW2 ) for the original image coordinate system of the representative point O W is obtained as the position information of the representative point
  • a width LX W of the noted image region on the X W axis is obtained as the size information of the noted image region WMR.
  • the above mentioned mask image formation section 32 forms mask image data MMD based on noted image information WMD extracted by the noted image information extraction section 31 , and mask image information MDD stored in the above mentioned storage section 23 . Note that, formation of the mask image data MMD using the mask image information MDD and the mask image formation section 32 will be described later.
  • the above mentioned composite image formation section 33 forms composite image data SMD having the mask image superimposed on the original image based on the original image data OMD and the mask image data MMD. Note that, the composite image data SMMD formed by the composite image formation section 33 is notified to the basic control section 29 .
  • control section 21 is provided with a central processing unit (CPU), digital signal processor (DSP) and so forth. Then, various data processing is carried out by the basic image processing section 29 and mask image processing section 30 , constituted by programs executed by the control section 21 , in order to realize portable telephone functions, including mail interchange processing, and besides that operational control of other structural elements is also performed.
  • CPU central processing unit
  • DSP digital signal processor
  • a mask image information file 40 is stored in the storage section 23 .
  • this mask image information file is comprised of (i) a display designation file 41 j storing display designation descriptors for designating as to with what relationship to the noted image region in the original image the mask image is displayed, and (ii) an element image file 42 j storing element image data designated using display designation descriptors in the display designation file 41 j .
  • the mask image information file 40 is provided to the portable telephone unit 10 from a contents server via a wireless circuit, or is provided after being formed by the user using a personal computer or the like by means of storage medium or an external interface, not shown in this figure.
  • element images are stored in the element image file 42 j .
  • the i th element image, stored in the element image file 42 j will be denoted element image EMM ji .
  • FIG. 5B a case is shown where there are a plurality of element images stored in the element image file 42 j , but there may be cases where the number of element images stored in the element image file 42 j is one, or cases where it is 2 or more.
  • a element image EMM ji for example, there is a heart shaped image or a lip shaped image and so forth.
  • the element image EMM ji is formed in a coordinate system inherent to the element image, that is in the X E Y E coordinate system making a representative point (in FIG. 6 the center point of element image EMM 11 ) of element image EMM ji an origin O E , as shown representatively for element image EMM 11 in FIG. 6 .
  • a maximum width LX Ei in FIG. 6 maximum width LX E1 ) in the direction of the X E is used as the size information for element image EMM ji .
  • the element image EMM ji has been made as a heart shaped element image EMM 11 , but the same applies even if it is an element image of a different shape or size.
  • Example of display designation descriptors for the above described display designation file 41 j are shown in FIG. 7 , FIG. 8A and FIG. 8B . Note that, the display designation descriptors of this embodiment are described in a format conforming to the SMIL (Synchronized Multimedia Integration Language) format.
  • SMIL Synchronized Multimedia Integration Language
  • FIG. 7 an example for carrying out display designation to display the same element image steadily as two positions (for example, positions of two eyes in the case where the noted image is a facial image of a specified person) fixed in accordance with position, inclination and size of the noted image region is shown as the display designation file 41 1 .
  • display of additional image 1 and additional image 2 designated by ⁇ additional image 1 designation> and ⁇ addition image 2 designation> described between ⁇ simultaneous display designation> and ⁇ end simultaneous display designation> is designated.
  • Additional image 1 is designated using the descriptors between ⁇ additional image 1 designation> and ⁇ end additional image 1 designation>.
  • element image EMM 11 designated by “element image designation” is display at a coordinate position (X W11 , Y W11 ) of the X W Y W coordinate system in the case where size information for the noted image region WMR is a value LX W0 constituting a predetermined reference, as designated using “display position designation”.
  • size designation a maximum width in a X W axis direction of the element image EMM 11 at the time of display is made a length of proportion R 1 with respect to size LX W of the noted image region WMR. Note that, from the fact that display time designation, which will be described later, is not carried out, the element image EMM 11 for additional image 1 is designated to be displayed continuously.
  • Additional image 2 is designated using the descriptors between ⁇ additional image 2 designation> and ⁇ end additional image 2 designation>.
  • element image EMM 11 designated by “element image designation” is display at a coordinate position (X W12 ,Y W12 ) of the X W Y W coordinate system in the case where size information for the noted image region WMR is a value LX W0 constituting a predetermined reference, as designated using “display position designation”.
  • the same display designation as for the case of additional image 1 is carried out because the same designations as for the case of additional image 1 are carried out.
  • FIG. 8A an example for carrying out display designation to display the one additional image 1 whose display contents vary with time at a single position (for example, position of a mouth in the case where the noted image is a facial image of a specified person) fixed in accordance with position, inclination and size of the noted image region is shown as the display designation file 41 2 .
  • the display designation file 41 2 of FIG. 8A a plurality of element images can not be displayed at the same time, which means that ⁇ simultaneous display designation> and ⁇ end simultaneous display designation> relating to the simultaneous display in FIG. 7 are not described.
  • Additional image 1 is designated using the descriptors between ⁇ additional image 1 designation> and ⁇ end additional image 1 designation>. Specifically, first of all an element image EMM 21 designated by a first “element image designation” is displayed at a coordinate position (X W21 ,Y W21 ) of the X W Y W coordinate system in the case where size information LX E for a noted image region designated using a first “display position designation” is a value LX E0 constituting a predetermined reference. Also, as designated by the first “size designation”, a maximum width in a X E axis direction of the element image EMM 2 , at the time of display is made a length of proportion R 21 with respect to size LX W of the noted image region WMR. Also, as designated by the first “display time designation”, the element image EMM 21 is displayed for a time T 21 .
  • an element image EMM 22 designated using a second “element image designation” is displayed at a position designated using a second “display position designation”, that is, at the same position as for the case of element image EMM 21 .
  • a maximum width in a X E axis direction of the element image EMM 22 at the time of display is made a length of proportion R 22 with respect to size LX W of the noted image region WMR.
  • the element image EMM 22 is displayed for a time T 22 .
  • an element image EMM 23 designated using a third “element image designation” is displayed at a position designated using a third “display position designation”, that is, at the same position as for the case of element image EMM 21 .
  • a maximum width in a X E axis direction of the element image EMM 23 at the time of display is made a length of proportion R 23 with respect to size LX W of the noted image region WMR.
  • the element image EMM 23 is displayed for a time T 23 .
  • FIG. 8B An example for carrying out display designation in order to display one element image all the time at a single position fixed in accordance with position, inclination and size of the noted image region, and also displaying two different element images selectively in time order at two other positions determined in accordance with position, inclination and size of the noted image region is shown in FIG. 8B as an example of a display designation file 413 .
  • display of additional image 1 and addition image 2 designated by ⁇ additional image 1 designation> and ⁇ additional image 2 designation> described between ⁇ simultaneous display designation> and ⁇ end simultaneous display designation> is designated.
  • Additional image 1 is designated using the descriptors between ⁇ additional image 1 designation> and ⁇ end additional image 1 designation>.
  • element image EMM 3 designated by “element image designation” is displayed at a coordinate position (X W31 ,Y W31 ) of the X W Y W coordinate system in the case where size information for the noted image region WMR is a value LX W0 constituting a predetermined reference, as designated using “display position designation”.
  • size designation a maximum width in a X W axis direction of the element image EMM 31 at the time of display is made a length of proportion R 31 with respect to size LX W of the noted image region WMR. Note that, because display time designation is not carried out, the element image EMM 31 for additional image 1 is designated to be displayed continuously.
  • Additional image 2 is designated using the descriptors between ⁇ additional image 2 designation> and ⁇ end additional image 2 designation>.
  • first of all an element image EMM 32 designated by a first “element image designation” is displayed at a coordinate position (X W32 ,Y W32 ) of the X W Y W coordinate system in the case where size information for a noted image region WMR, as designated using a first “display position designation”, is a value LX W0 constituting a predetermined reference.
  • size designation a maximum width in a X W axis direction of the element image EMM 32 at the time of display is made a length of proportion R 32 with respect to size LX W of the noted image region WMR.
  • the element image EMM 32 is displayed for a time T 32 .
  • an element image EMM 33 designated by a second “element image designation” is displayed at a coordinate position (X W33 ,Y W33 ) of the X W Y W coordinate system in the case where size information for a noted image region WMR, as designated using a second “display position designation”, is a value LX 0 constituting a predetermined reference. Also, as designated by “size designation”, a maximum width in a X W axis direction of the element image EMM 33 at the time of display is made a length of proportion R 33 with respect to size LX W of the noted image region WMR. Further, as designated by the second “display time designation”, the element image EMM 33 is displayed for a time T 33 .
  • Simultaneous display of element image EMM 31 and element image EMM 32 , and simultaneous display of element image EMM 31 and element image EMM 33 , by combining additional image 1 and additional image 2 in this way, are carried in time sequence. If these displays are completed, simultaneous display of element image EMM 31 and element image EMM 32 , and simultaneous display of element image EMM 31 and element image EMM 33 , are again carried out in time sequence under the same conditions as described above. Simultaneous display of element image EMM 31 and element image EMM 32 , and simultaneous display of element image EMM 31 and element image EMM 33 in this way, are repeatedly carried out in time sequence.
  • the mask image formation section 32 has already read out contents of the mask information file 40 in the storage section 23 as mask image information MDD.
  • the original image is an image that is formed on an imaging screen of the imaging element 25 by means of the imaging optical system 18 and the switching optical system 24 .
  • the noted image WMM is a facial image of a specified person, such as shown in FIG. 9A . Then, as shown in FIG.
  • a tip position of a nose in the noted image WMM is made an origin O W
  • a direction connecting both sides is made an X W axis direction
  • an X W Y W coordinate system (noted image coordinate system) with a direction orthogonal to the X W axis direction being a Y W axis direction is defined.
  • the origin O W and the X W axis direction are obtained by analyzing the noted image in the original image, and the Y W axis direction is obtained based on the obtained X W axis direction.
  • FIG. 7 described above is an example of a composite image having a position of two eyes as display positions, and element image EMM 11 being the heart shaped graphic image shown in FIG. 6 , for displaying a mask image that is the element image EMM 11 displayed in a size of proportion R 1 with respect to length value LX W representing size of noted image WMM, superimposed on an original image.
  • original image data OMD being data for an original image OMM imaged on the imaging element 25 , is notified to the noted image information extraction section 31 .
  • the original image data OMD is also notified to the composite image formation section 33 at the same time (refer to FIG. 3 ).
  • the acquired original image OMM is shown in FIG. 10A .
  • the noted image information extraction section 31 image processes the notified original image OMM to extract a region of the noted image WMM.
  • the noted image information extraction section 31 analyzes the noted image WMM, to obtain a position of a tip of a nose (X SW ,Y SW ) in the noted image WMM of the original image coordinate system (X S Y S coordinate system) as a position of an origin O W for the noted image coordinates (X W Y W coordinate system), and obtain a direction in which a line segment linking two central position of pupils of two eyes extends as an X W axis direction for the X W Y W coordinate system.
  • a crossing angle ⁇ W of the obtained X W axis direction and the X S axis direction is then obtained.
  • the noted image information extraction section 31 obtains a width LX W of the noted image WMM on the X W axis. The noted image information extraction section 31 then notifies position (X SW ,Y SW ) as position information for the noted image region, angle ⁇ W as inclination information for the noted image region, and length LX W as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • the mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (X E11 ,Y E11 ) in the X S Y S coordinate system of the element image EMM 11 in the additional image 1 , using the following equations (1) and (2).
  • X E11 ( LX W /LX 0 )( X W11 *cos ⁇ W +Y W11 *sin ⁇ W )+ X SW (1)
  • Y E11 ( LX W /LX 0 )( ⁇ X W11 *sin ⁇ W +Y W11 *cos ⁇ W )+ Y SW (2)
  • the mask image formation section 32 calculates a magnification M 11 for expansion or contraction of the element image EMM 11 in the additional image 1 from the following equation (3).
  • M 11 R 1 *LX W /LX E1 (3)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 11 with a magnification M 11 and with origin O E as a center in the X S Y S coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 11 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 11 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E11 ,Y E11 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the mask image formation section 32 calculates display position (X E12 ,Y E21 ) in the X S Y S coordinate system of the element image EMM 11 in the additional image 2 , using the following equations (4) and (5).
  • X E12 ( LX W /LX 0 )( X W12 *cos ⁇ W +Y W12 *sin ⁇ W )+ X SW (4)
  • Y E12 ( LX W /LX 0 )( ⁇ X W12 *sin ⁇ W +Y W12 *cos ⁇ W )+ Y SW (5)
  • the mask image formation section 32 calculates a magnification M 12 for expansion or contraction of the element image EMM 11 in the additional image 2 from the following equation (6).
  • M 12 R 1 *LX W /LX E1 (6)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 11 with a magnification M 12 and with origin O E a center in the X S Y S coordinate system as. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 11 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 11 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E12 ,Y E12 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMMA such as is shown by image F 10 B in FIG. 10B is formed.
  • the mask image formation section 32 then notifies data of the mask image MMMA to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received the mask image data MMD and the original image data OMD forms a composite image SMMA having the mask image MMMA superimposed on the original image data OMM.
  • the composite image SMMA formed in this way is represented as image F 11 A in FIG. 11 .
  • the composite image for the case of this example is formed. Therefore, even if the noted image WMM of the original image is moved parallel from the above described example during imaging, as shown in image F 11 B, the same as with the above described case a composite image SMMB is formed having a mask image with the element image EMM 11 displayed at size that is a proportion designated according to size of the noted image region at a position of two eyes in the noted image WMM superimposed on the original image.
  • a composite image SMMD is formed having a mask image with the element image EMM 11 displayed having a size that is a proportion designated according to size of the noted image region at a position of two eyes in the noted image WMM, and having the same inclination as the inclination of the noted image WMM within in the original image, superimposed on the original image.
  • the noted image WMM in the original image is contracted, expanded or rotated during imaging, it is generally accompanied by change in coordinate position in the original image coordinate system (X S Y S coordinate system) for nose tip position in the noted image region.
  • This example is an example of a composite image where, in FIG. 8A described above, a mask image having element image EMM 21 shown in FIG. 12A , element image EMM 22 shown in FIG. 12B and element image EMM 23 shown in FIG. 12C displayed in time sequence, with the position of a mouth as display position, is superimposed on the original image.
  • element images EMM 21 , EMM 22 and EMM 23 are displayed in a size corresponding to “size information” being designated by the descriptor of FIG. 8A .
  • element image EMM 21 is displayed in a size of proportion R 21 with respect to length value LX W representing size of the noted image WMM.
  • element image EMM 22 is displayed in a size of proportion R 22 with respect to length value LX W .
  • element image EMM 23 is displayed in a size of proportion R 23 with respect to length value LX W .
  • data for an original image OMM imaged on the imaging element 25 is notified to the noted image information extraction section 31 as original image data OMD.
  • the noted image extraction section 31 carries out image processing of the notified original image OMM and extracts the noted image information WMR specifying the region of the noted image WMM.
  • the noted image information extraction section 31 then notifies position (X SW ,Y SW ) as position information for the noted image region, angle ⁇ W as inclination information for the noted image region, and length LX W as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • the mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (X E2 ,Y E2 ) in the X S Y S coordinate system of the first element image EMM 21 in the additional image 1 , using the following equations (7) and (8).
  • X E2 ( LX W /LX 0 )( X W2 *cos ⁇ W +Y W2 *sin ⁇ W )+ X SW (7)
  • Y E2 ( LX W /LX 0 )( ⁇ X W2 *sin ⁇ W +Y W2 *cos ⁇ W )+Y SW (8)
  • the mask image formation section 32 calculates a magnification M 21 for expansion or contraction of the first element image EMM 21 from the following equation (9).
  • M 21 R 21 *LX W /LX E21 (9)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 21 with a magnification M 21 and with origin O E as a center in the X S Y S coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 2 , by angle ⁇ W with origin O E in the X S Y S coordinate system as a center. The element image EMM 21 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E2 ,Y E2 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMM 21 such as is shown by image F 13 A in FIG. 13 is formed.
  • the mask image formation section 32 then notifies data of the mask image MMM 21 to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM 21 having the mask image MMM superimposed on the original image OMM.
  • the composite image SMM 21 formed in this way is represented as image F 14 A in FIG. 14 .
  • the mask image formation section 32 forms the mask image displaying element image EMM 21 in a size of proportion R 21 with respect to length value LX W representing the size of the noted image WMM, with mouth position in the noted image WMM as display position.
  • the composite image formation section 33 sequentially forms the composite image having the mask image that is the element image EMM 21 that has been expanded or contracted at magnification M 21 at the position of the mouth in the noted image WMM of the original image.
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 22 with a magnification M 22 and with origin O E as a center in the X E Y E coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 22 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 22 that has been expanded or contracted and rotated is then arranged in the X s Y S coordinate system so that a coordinate position (X E2 ,Y E2 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMM 22 such as is shown by image F 13 B in FIG. 13 is formed.
  • the mask image formation section 32 then notifies data of the mask image MMM 22 to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM 22 having the mask image MMM superimposed on the original image OMM.
  • the composite image SMM 22 formed in this way is represented as image F 14 B in FIG. 14 .
  • the mask image formation section 32 forms the mask image displaying element image EMM 22 in a size of proportion R 22 with respect to length value LX W representing the size of the noted image WMM, with mouth position in the noted image WMM as display position.
  • the composite image formation section 33 forms the composite image having the mask image that is the element image EMM 22 that has been expanded or contracted at magnification M 22 at the position of the mouth in the noted image WMM of the original image.
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 23 with a magnification M 23 and with origin O E as a center in the X E Y E coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 23 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 23 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E2 ,Y E2 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMM 23 such as is shown by image F 13 C in FIG. 13 is formed.
  • the mask image formation section 32 then notifies data of the mask image MMM 23 to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM 23 having the mask image MMM 23 superimposed on the original image OMM.
  • the composite image SMM 23 formed in this way is represented as image F 14 C in FIG. 14 .
  • the mask image formation section 32 forms the mask image displaying element image EMM 23 in a size of proportion R 23 with respect to length value LX W representing the size of the noted image WMM, with mouth position in the noted image WMM as display position.
  • the composite image formation section 33 forms the composite image having the mask image that is the element image EMM 23 that has been expanded or contracted at magnification M 23 at the position of the mouth in the noted image WMM of the original image.
  • the mask image formation section 32 similarly to above, repeatedly forms the mask image that uses the above described element image EMM 21 , the mask image that uses the element image EMM 22 , and the mask image that uses the element image EMM 23 , in time sequence.
  • the composite image formation section 33 then sequentially forms composite images with mask images comprised of the element image EMM 21 , element image EMM 22 and element image EMM 23 that have been expanded or contracted to a designated size superimposed at a position of a mouth of the noted image in the original image, based on original image data OMD and mask image data MMD.
  • This example corresponds to the mask image designation of FIG. 8 described above, and is an example of a composite image that displays a mask image, in which an additional image 1 that is displayed in a steady manner at a display position corresponding to an element image and a noted image, and an additional image 2 that is displayed at a display position that changes with time corresponding to an element image and a noted image, are displayed simultaneously, superimposed on an original image.
  • the element image EMM 31 for the additional image 1 is shown in FIG. 15A .
  • the first element image EMM 32 for the additional image 2 is shown in FIG. 15B
  • the second element image EMM 33 for the additional image 2 is shown in FIG. 15C .
  • element images EMM 31 , EMM 32 and EMM 33 are displayed in a size corresponding to “size information” being designated by the descriptor of FIG. 8 .
  • element image EMM 31 is displayed in a size of proportion R 31 with respect to length value LX W representing size of the noted image WMM.
  • element image EMM 32 is displayed in a size of proportion R 32 with respect to length value LX W .
  • element image EMM 33 is displayed in a size of proportion R 33 with respect to length value LX W representing size of the noted image WMM.
  • original image data OMD imaged on the imaging element 25 is notified to the noted image information extraction section 31 .
  • the noted image extraction section 31 carries out image processing of the notified original image OMM and extracts the noted image information specifying the region of the noted image WMM.
  • the noted image information extraction section 31 then notifies position (X SW ,Y SW ) as position information for the noted image region, angle ⁇ W as inclination information for the noted image region, and length LX W as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • the mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (X E31 ,Y E31 ) in the X S Y S coordinate system of the first element image EMM 31 in the additional image 1 , using the following equations (12) and (13).
  • X E31 ( LX W /LX 0 )( X W31 *cos ⁇ W +Y W31 *sin ⁇ W )+ X SW (12)
  • Y E31 ( LX W /LX 0 )( ⁇ X W31 *sin ⁇ W +Y W31 *cos ⁇ W )+ Y SW (13)
  • the mask image formation section 32 calculates a magnification M 3 , for expansion or contraction of the first element image EMM 31 from the following equation (14).
  • M 31 R 31 *LX W /LX E33 (14)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 31 with a magnification M 31 and with origin O E as a center in the X E Y E coordinate system.
  • the mask image formation section 32 rotates the expanded or contracted element image EMM 3 , by angle ⁇ W with origin O E as a center in the X S Y S coordinate system.
  • the element image EMM 31 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E31 ,Y E31 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the mask image formation section 32 calculates display position (X E32 ,Y E32 ) in the X S Y S coordinate system of the second element image EMM 32 in the additional image 2 , using the following equations (15) and (16).
  • X E32 ( LX W /LX 0 )( X W32 *cos ⁇ W +Y W32 *sin ⁇ W )+X SW (15)
  • Y E32 ( LX W /LX 0 )( ⁇ X W32 *sin ⁇ W +Y W32 *cos ⁇ W )+ Y SW (16)
  • the mask image formation section 32 calculates a magnification M 32 for expansion or contraction of the second element image EMM 32 from the following equation (17).
  • M 32 R 32 *LX W /LX E32 (17)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 32 with a magnification M 32 and with origin O E as a center in the X E Y E coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 32 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 31 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E32 ,Y E32 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMM 31 such as is shown by image F 16 A in FIG. 16 is formed.
  • the mask image formation section 32 then notifies data of the mask image MMM 31 to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM 31 having the mask image MMM 31 superimposed on the original image OMM.
  • the composite image SMM 31 formed in this way is represented as image F 17 A in FIG. 17 .
  • the mask image formation section 32 forms the mask image displaying element image EMM 31 in a size of proportion R 31 with respect to length value LX W representing the size of the noted image WMM, and displaying element image EMM 32 in a size of proportion R 32 with respect to length value LX W at a position that is a relationship designating the noted image WMM.
  • the composite image formation section 33 then forms a composite image having the mask image represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • the mask image formation section 32 calculates display position (X E31 ,Y E31 ) in the X S Y S coordinate system, and magnification M 31 , of the first element image EMM 31 in the additional image 1 , using the above described equations (12) to (14). Then, after expanding or contracting element image EMM 31 with a magnification M 31 , the mask image formation section 32 then rotates by angle ⁇ W with origin O E as a center in the X E Y E coordinate system.
  • the element image EMM 2 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E31 , Y E31 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the mask image formation section 32 calculates display position (X E33 ,Y E33 ) in the X S Y S coordinate system of the element image EMM 33 in the additional image 2 , using the following equations (18) and (19).
  • X E33 ( LX W /LX 0 )( X W33 *cos ⁇ W +Y W33 *sin ⁇ )+ X SW (18)
  • Y E33 ( LX W /LX 0 )( ⁇ X W33 *sin ⁇ W +Y W33 *cos ⁇ )+ Y SW (19)
  • the mask image formation section 32 calculates a magnification M 33 for expansion or contraction of the third element image EMM 33 from the following equation (20).
  • M 33 R 33 *LX W /LX E33 (20)
  • the mask image formation section 32 carries out expansion or contraction of the element image EMM 33 with a magnification M 33 and with origin O E as a center in the X E Y E coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM 33 by angle ⁇ W with origin O E as a center in the X S Y S coordinate system. The element image EMM 33 that has been expanded or contracted and rotated is then arranged in the X S Y S coordinate system so that a coordinate position (X E33 ,Y E33 ) in the X S Y S coordinate system becomes the position of the origin O E .
  • the result of the above processing is that a mask image MMM 32 such as is shown by image F 16 B in FIG. 16 is formed.
  • the mask image formation section 32 then notifies data of the mask image MMM 32 to the composite image formation section 33 as mask image data MMD.
  • the composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM 32 having the mask image MMM 32 superimposed on the original image OMM.
  • the composite image SMM 32 formed in this way is represented as image F 17 B in FIG. 17 .
  • the mask image formation section 32 forms the mask image displaying element image EMM 31 in a size of proportion R 31 with respect to length value LX W representing the size of the noted image WMM, and displaying element image EMM 33 in a size of proportion R 33 with respect to length value LX W , at a position that is a relationship designating the noted image WMM.
  • the composite image formation section 33 then forms a composite image having the mask image represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • the mask image formation section 32 similarly to above, repeatedly forms the mask image that uses the above described element image EMM 31 and the element image EMM 32 , and the mask image that uses the element image EMM 31 and the element image EMM 33 , in time sequence.
  • the composite image formation section 33 then sequentially forms composite images having the mask images represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • Information data SMD of the composite images formed in the above described first, second and third composite image formation examples is notified from the composite image formation section 33 to the basic control section 29 .
  • the basic control section 29 that has received this composite image information data displays the composite image SMM on the display section 13 , and transmits to a fellow caller via the transceiver section 21 .
  • a region of a noted image WMM is specified in an original image, and noted image information WMD containing information relating to position and size of the noted image WMM region within the original image is extracted.
  • a mask image MMM displayed superimposed on the original image OMM is formed based on noted image information WMD and mask image information registered in the mask image information file 40 .
  • a mask image comprised of element images selected using element image selection information contained in mask image information is formed on the original image OMM at a position determined using information connected to position of a region of noted image WMM and information representing a relationship to position of the noted image region contained in mask image information, and at a size determined by information relating to size of the region of the noted image WMM in the original image OMM, and information representing a relationship to size of the noted image region.
  • a composite image having the mask image superimposed on the original image is then formed.
  • the information relating to the position of the noted image region contained in the mask image information it is possible for the information relating to the position of the noted image region contained in the mask image information to contain position information for a representative point of the noted image region WMR in the original image OMM. Therefore, it is possible to make information representing a relationship to position of a noted region contained in the mask image information, information representing a relative relationship for a representative point of the noted image, and it is possible to simplify information representing a relationship with position of the noted image region in the relationship information.
  • the information relating to the position of the noted image region contained in the mask image information to further contain inclination information for the noted image region in the original image OMM. Therefore, by using the inclination information for the noted region of the original image OMM it is possible to form a composite image displaying a desired element image superimposed at a position corresponding to the inclination of the noted image region in the original image. It is also possible to display by holding inclination in the displayed element image corresponding to inclination of the noted image region in the original image.
  • a representative point of the noted image is made an origin, and coordinate position information in a coordinate system determining direction of a coordinate axis based on inclination of the noted image region in the original image is included as display position information for an element image. Therefore, the information with respect to a representative point of the noted image region can be simply interpreted as coordinate position information of a coordinate system that determined by direction of coordinate axis based on inclination of the noted image region, wherein the system has an origin as a representative point of the noted image region.
  • the noted information is the information representing a relative position of information relating to the position.
  • the mask image information it is possible for the mask image information to contain proportion information for size of the noted image region and size of the element image EMM at the time of displaying. In this case, even when size of the noted image within the original image OMM is changed from moment to moment, it is possible to form a mask image MMM containing element images EMM of size appropriate to the size of the noted image region.
  • the mask image information may contain execution time information for respective display of the element image EMM.
  • the mask image WMM by forming a mask image WMM according to the execution time information, it is possible to display respective desired element images EMM for only display execution times set to respective to desired times, and it is possible to form a mask image MMM to vary the dynamic.
  • the mask image information may contain information time sequence display designation, for designating display a plurality of element images EMM in time sequence, and repeat of time sequence display.
  • the mask image information by forming a mask image WMM according to the time sequence display designation information, it is possible to display a plurality of element images on the original image OMM repeatedly in time sequence every desired time.
  • the mask image information it is possible for the mask image information to contain information for simultaneous display designation, for designating display of a plurality of element images at the same time.
  • the mask image information it is possible to form a mask image having a desired plurality of element images EMM respectively displayed as desired positions, and it is possible to form mask images WMM of numerous variations.
  • this embodiment is different in that it is possible to take different images to the original image in the background to a person's image.
  • description will mainly focus on points of difference.
  • a portable telephone unit 10 ′ of this embodiment is different from the portable telephone unit of the first embodiment in that the mask image processing section 30 ′ is provided in the control section 21 and in that the mask image information file 40 ′ is stored in the storage section 23 , inside of the portable telephone body 11 .
  • the mask image processing section 30 ′ comprises a noted image extraction section 31 , a mask image formation section 32 ′ and a composite image formation section 33 ′.
  • the noted image information extraction section 31 operates in the same manner as in the case of the first embodiment.
  • the mask image formation section 32 ′ forms mask image data MMD based on noted image information WMD extracted by the noted image information extraction section 31 , and mask image information MDD stored in the mask image information file 40 ′ in the above mentioned storage section 23 . Also, the mask image formation section 32 ′ creates background image information BPN made up of background image designation information representing whether or not there is background image designation and storage position information for the background image data BDD in the background designation information in the event that there is background image designation, based on mask image information MDD stored in the mask image information file 40 ′.
  • background image information BPN made up of background image designation information representing whether or not there is background image designation and storage position information for the background image data BDD in the background designation information in the event that there is background image designation
  • the composite image formation 33 ′ first of all forms an image that has a person's image, being a specified display image within an original image, superimposed on a background image using the background image data BDD, as a background image containing image, based on original image data OMD and background image data BDD.
  • the composite image formation section 33 ′ forms composite image data SMD′, wherein a mask image represented using mask image information MDD is superimposed on the image containing background image.
  • the composite image formation means 33 ′ forms the original image as a background image containing image.
  • the composite image formation section 33 ′ forms composite image data SMD having a mask image represented using mask image information MMD superimposed on the background image containing image.
  • the composite image data SMD′ is the same as the composite image data SMD of the first embodiment.
  • the composite image data SMD′ formed by the composite image formation section 33 is notified to the basic control section 29 .
  • the mask image information file 40 ′ inside the storage section 23 is comprised of (i) a display designation file 41 ′ storing display designation descriptors for designating as to with what relationship to the noted image region in the original image the mask image is displayed and background image designation descriptors, (ii) the above described element image file 42 , and (iii) a background image file 43 storing background image data BDD.
  • the mask image information file 40 ′ is provided to the portable telephone unit 10 from a contents provider via wireless circuit, or is provided after being formed by the user using a personal computer or the like by means of storage medium or a non-shown external interface.
  • FIG. 21 An example of display designation descriptors for the display designation file 41 ′ of this embodiment is shown in FIG. 21 .
  • the descriptor example in FIG. 21 has the fact that there is background image designation added to the descriptor example of FIG. 7 for the case of the first embodiment.
  • a background image to be displayed simultaneously using the descriptor “background image designation” from ⁇ background image designation> to ⁇ end background image designation> is designated.
  • a header address BMMA for the background image file 43 in the storage section 23 is designated as “background image designation”.
  • background image data BDD representing a background image BMM inside the background image region BMR having the same shape and size as the original image region OMR shown in FIG. 22 is stored in the background image file 43 .
  • the person image SDM inside the original image region OMR shown in FIG. 23 is a specified display image to be cut from the original image OMM.
  • original image data OMD being data for an original image OMM imaged on the imaging element 25
  • noted image information data WMD is formed by the noted image information extraction section 31 .
  • the noted image data WMD formed in this way is notified from the noted image information extraction section 31 to the mask image formation section 32 ′.
  • the mask image formation section 32 ′ that has received the noted image information data WMD creates data of a mask image MMMA based on noted image information data WMD and mask image information MDD, and notifies the data to the composite image formation section 33 ′ as mask image data MMD. Also, the mask image formation section 32 ′ creates background image information BPN comprising whether or not there is background image designation, and a header address of a background image file 43 for when there is background image designation, based on mask image information MDD, and notifies the background image information BPN to the composite image formation section 33 ′. In this case, as shown in FIG. 21 described above, background image designation is described in the display designation file 41 ′ and so the fact that there is background image designation and a header address of the background image file 43 is included as background image information BPN.
  • the composite image formation section 33 ′ that has received the mask image data MMD, original image data OMD and background image data BPN first of all determines whether or not there is background image designation based on the background image information BPN. In the present case, since there is background image designation, as described above, the determination result is affirmative. Continuing on, the composite image formation section 33 ′ reads out background image data BDD from the background image file 43 using the header address of the background image file 43 in the background image information.
  • the composite image formation section 33 ′ that has acquired the background image data BDD, in addition to the mask image data MMD and the original image data OMD creates composite image data SMD as follows.
  • a personal image SDM is extracted from the original image OMM represented by the image F 24 A in FIG. 24 as a specified display image represented by the image F 24 B. This extraction is carried out after specifying the outline of the personal image SDM from general characteristics of a personal image, by extracting an image inside the outline.
  • the composite image formation region 33 ′ superimposes the specified display image SDM on the background image BMM represented by image F 24 C, at the same position and same size as for the case of the original image OMR.
  • a background additional image BAM represented as image F 24 D is formed.
  • the composite image formation section 33 ′ superimposes the mask image MMMA represented as the image F 24 E on the background additional image BAM. As a result, a composite image SMMA′ represented as image F 24 F is formed.
  • the data SMD′ of the composite image SMMA′ formed in this manner is notified from the composite image formation section 33 to the basic control section 29 .
  • the basic control section 29 that has received this composite image information data displays the composite image SMMA′ on the display section 13 , and transmits to a fellow caller via the transceiver section 21 .
  • the mask image formation section 32 ′ forms background image information BPN comprising only the fact that there is no background image designation and notifies this data to the composite image formation section 33 ′.
  • the composite image formation section 33 ′ that receives this type of background image information BPN determines that there is no background image designation and superimposes the mask image MMMA on the original image OMM to form a composite image.
  • a composite image SMMA that is the same as for the case of the composite image formation example of the first embodiment is formed.
  • a background additional image BAM is formed in the composite image formation section 33 ′ as a completed composite image.
  • the composite image formation section 33 ′ cuts a personal image SDM containing a facial image, being a noted image, from the original image, as a specified display image, superimposes it on the designated background image BMM and forms a background additional image BAM as a noted image comprising image. Then, the composite image formation section 33 ′ superimposes the mask image MMMA on the background additional image BAM. Accordingly, rendering of the specified display image, such as a personal image SDM and so forth, according to the user's interests, can be carried out by selecting a mask image and a background image.
  • the mask image information file 40 ′ comprises a background image file 43 .
  • the background image file 43 it is also possible for the background image file 43 to be located in a region outside the mask image information file.
  • the original image is a moving picture, but the present invention can also be applied to cases where the original image is a still image.
  • image formation resulting from light passing through the imaging optical system 18 that images in a so-called self portrait state is made the original image, but it is also possible to make image formation resulting from light passing through the imaging optical system 19 that images in a so-called other party shooting state to be made the original image. It is also possible for an image stored in the storage section 23 to be made the original image.
  • the number of element imaged displayed at the same time is arbitrary.
  • the number of element images carrying out time sequence display is also arbitrary.
  • the mask image information file is comprised of the display designation file and the element information file. It is also possible, however, for the mask image information file to further comprise an element image preview file, and when element image preview is designated, to perform element image preview on the display section 13 by referencing the element image preview file.
  • specified phenomenon occurrence is determined for determining whether or not a predetermined phenomenon has occurred on the noted image region based on noted image information, and it is possible commence formation of the mask image using occurrence of the specified phenomenon as a trigger.
  • the noted image is a facial image of a specified person
  • the eyelids are closed, it is possible to form a composite image with tears flowing from positions of the eyes.
  • the present invention has been applied to a portable telephone unit, but it is also possible to apply the present invention to various other mobile communication terminals. It is also possible to apply the present invention to general electronic appliances such as a personal computer.
  • the image processing method and image processing method of the present invention can be applied to processing of an acquired original image.
  • the mobile communication terminal of the present invention can be applied to a mobile communication terminal having van image mail function and a television telephone function.

Abstract

A noted image information extraction section 31 specifies a region of a noted image in an original image, and noted image information containing information relating to position and size of the noted image WMM region within the original image is extracted. Continuing on, mask image formation means 31 forms a mask image by displaying superimposed on the original image, based on noted image information and mask image information registered in a mask image information file. As a result, a mask image comprised of element images selected using element image selection information contained in mask image information is formed at a specified position and specified size in the noted image region of the original image. A composite image having the mask image superimposed on the original image is then formed by composite image formation means 33. As a result, an image that a user has rendered in an original image is simply and appropriately formed, regardless of whether it is a still image or a moving picture.

Description

    RELATED APPLICATION
  • This a continuation application of the international patent application No. PCT/JP2005/001753 filed with Application date: Feb. 7, 2005. The present application is based on, and claims priority from, J.P. Application 2004-029943, filed on Feb. 5, 2004, the disclosure of which is hereby incorporated by reference herein its entirety.
  • BACKGROUND OF THE INVENTION
  • The present invention relates to an image processing method, an image processing device and a mobile communication terminal, and in more detail relates to an image processing method for forming a composite image having a mask image superimposed on an original image, an image processing device using the image processing method, and a mobile communication terminal provided with the image processing device.
  • BACKGROUND ART
  • There have conventionally been various means for performing image processing to process an acquired original image. As technology for this type of image processing, there is facial recognition technology for recognizing that there is a facial image in an original image acquired using a digital camera or the like. There is also facial authentication technology that is a further advancement of this facial recognition technology, for extracting feature characteristics of a face and specifying an individual (refer to non-patent publication 1, and patent publications 1 and 2).
  • As background to this type of existing image processing technology relating to facial recognition, for example, there has also been proposed video telephone technology, where an image acquired by a transmission side terminal is sent as is, and not only displayed on a receive side terminal, but in order to protect privacy of the user of the transmission side terminal sent to the receive side terminal after carrying out image processing to change a background image resulting from acquisition to another image (refer to patent reference 3). With this technology, first of all, an image of the transmission side caller is extracted from an image acquired by a camera of the transmission side terminal. Then, a composite image is formed by superimposing the extracted caller image onto a pre-prepared background image, and the composite image is transmitted from the transmission side terminal to the receive side terminal. As a result, privacy information such as current location of the transmission side caller is not notified to the receiving side caller, and it becomes possible to hold a face-to-face conversation that feels comfortable.
  • Also, in order to protect the privacy of the transmission side user, technology has been proposed where a real time image of the transmission side caller is not used, but an image that has been selected by matching voice of the transmission side user from a pre-prepared plurality of still images of the transmission side caller or another caller is transmitted to the receive side terminal (refer to patent reference 3). Note that, In this case also, it is possible to form a composite image by superimposing the selected caller image onto a pre-prepared background image, and transmit the composite image from the transmission side terminal to the receive side terminal.
  • With this technology, with respect to the predetermined user or other user, still images are prepared for each time a vowel and the nasal “n” sound occurs, as required. The generated sound during the user's call is then analyzed, and as a result of analysis an image is selected for the obtained generated sound. Continuing on, a composite image is formed by superimposing the selected image onto a pre-prepared background image, and the composite image is transmitted from the transmission side terminal to the receive side terminal. As a result, also in cases where the fact that a real time image is transmitted would be inappropriate from the point of view of protecting privacy, such as when bathing, it is possible to hold a call that does not feel uncomfortable while referencing the images.
  • [non-patent reference 1] Eyematic Japan Ltd. “technical information”, published Dec. 29, 2003, internet [URL:http://www.eyematic.co.jp/tech.html].
  • [Patent reference 1] Japanese patent laid-open No. 2003-271958
  • [Patent reference 2] Japanese patent laid-open No. 2003-296735
  • [Patent reference 3] Japanese patent laid-open No. 2003-309829
  • SUMMARY OF THE INVENTION Problems to be Solved by the Invention
  • Image processing technology of the above described related art, such as image recognition, image authentication, image tracking etc. are extremely good at being capable of application to a moving picture, and various application have been considered. However, with this type of image processing technology, a user processes an original image according to their own preferences, and expressing their own innermost thoughts is not within the technical scope. For this reason, for example, even if the image processing technology of the related art is applied to video telephone, it is insufficient from the point of view of holding an enjoyable conversation in accordance with an image that reflects the interests of the transmission side caller while demonstrating the function of the video telephone.
  • Specifically, with the image processing technology of the related art, it has been considered that as long as it is possible to carry out production matching the interests of the user such as emphasizing the user's emotions in an original image, it would be possible to bring enjoyment to the user, but carrying out this type of production on an image is not possible, particularly when the original image is a moving picture.
  • When the original image is a fixed still image, this type of image production can be carried out by superimposing desired features and colored accessory images at desired sizes on desired positions of the stillimage. However, when a transmitted image is a moving picture changing from moment to moment with the passing of time, or an image equivalent to a moving picture, by displaying the accessory images at fixed position of a display screen a display image that is unpleasant arises.
  • For example, for an original image that contains a facial image of a user, when carrying out production to cause display of heart marks at positions of the eyes of on the facial image at almost the same size as the eyes, by displaying the heart marks superimposed at fixed positions of the display screen it is not possible to display the heart marks at positions of the eyes in the facial image for a moving picture that changes from moment to moment. Also, when the superimposed display corresponds not only to display position but also the size of the face in the moving picture, the display image becomes unpleasant.
  • The present invention has been conceived in view of the above described problems, and an object of the invention is to provide an image processing method capable of simply and appropriately forming an image that a user has produced on an original image, regardless of whether the original image is a moving picture or a still image.
  • Another object of the present invention is to provide a mobile communication terminal capable of transmitting an image the user has appropriately produced for an original image, regardless of whether the original image is a moving picture or a still image.
  • MEANS FOR SOLVING THE PROBLEMS
  • An image processing method of the present invention is an image processing method for carrying out processing of an image, and comprises the steps of: specifying a noted image region, being a region for an image to be focused on, in an original image, and extracting noted information containing information relating to position and size of the noted image within the original image; forming a mask image to be displayed superimposed display on the original image, based on a mask image information file containing the noted image information, at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information of specified element images respectively corresponding to the relationship information, and at least one element image information selected using the element image selection information; and creating a composite image having the mask image superimposed on a noted image comprising image that includes the noted image of the noted image region at the same location and same size as for the original image.
  • With this image processing method, at least one relationship information representing a relationship between position and size of the noted image region, element image selection information representing selection of specified element image information respectively corresponding to the relationship information, and mask image information containing at least one element image information selected using the element image selection information, are registered in a predetermined mask image information file. In this case, “noted image region” is a region for a face in the original image containing a figure, for example, or a region capable of being specified by pictorial characteristics within an image assumed to be obtained from an original image.
  • As a premise for the existence of a mask image information file, first of all, in a noted image extraction step, the noted image region is specified in the original image, and noted image information containing information relating to the position and size of the noted image region within the original image is extracted. Continuing on, in a mask image data creation step, a mask image is created by displaying superimposed on the original image, based on noted image information and mask image information registered in the mask image information file. As a result, a mask image formed from element images selected using element image selection information contained in mask image information is formed on the original image at a position determined using information connected to position of a noted image region and information representing a relationship between noted image region in relationship information contained in mask image information, at a size determined by information representing a relationship between information connected to size of the noted position in the original image and size of the noted image region in the relationship information. Then, in a composite image creating step, a composite image having the mask image superimposed on a noted image comprising image that includes a noted image inside the noted image region, such as a facial image within a facial image region, for example, at the same location and same size as for the original image, is formed.
  • Accordingly, with the image processing method of the present invention, it is possible to simply and appropriately form an image that a user has produced according their interests for a noted image comprising image, regardless of whether the original image is a moving picture or a still image.
  • With the image processing method of the present invention, it is possible to make the noted image comprising image the original image. In this case, rendering of the original image according to the user's interests can be carried out by selecting a mask image.
  • Also, with the image processing method of the present invention, the mask image information file further comprises background image selection information representing selection of information for a specified background image, and in the composite image formation step a specified display image containing the noted image is cut from the original image, and the composite image can be formed after a background addition image has been made the noted image comprising image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
  • In this case, in the composite image formation step, first of all a specified display image containing a noted image such as a figure is cut from the original image for the case where the noted image is a facial image. Continuing on, the specified display image is superimposed on a background image designated using background image selection information contained in the mask image information file, to form a background addition image. The above described mask image is then superimposed on this noted image comprising image, with the background addition image as the noted image comprising image. Accordingly, rendering of the specified display image, such as a figure and so forth, according to the user's interests can be carried out by selecting a mask image and a background image.
  • Also, with the image processing method of the present invention, it is possible for the information relating to the position of the noted image region to contain position information for a representative point of the noted image region of the original image. In this way, it is possible to make information representing a relationship with position of a noted region, in relationship information contained in the mask image information, information representing a relative relationship for a representative point of the noted image, and it is possible to simplify information representing a relationship with position of the noted image region in the relationship information. Note that, when information representing a relationship to position of the noted image region, in the relationship information, is made information representing a relative position with respect to the representative point of the noted image region, that relative position information is made relative position information for the case where the noted image region has a reference size, and it is possible to convert value of the relative position information to a value corresponding to the size of the noted image region in the original image, according to a proportion of the size of the noted image in the original image and the size of the noted image region constituting the reference. It is also possible to make a value of the relative position information a value after normalization using size of the noted image region in the original image, and to convert a value of the relative position information to a value corresponding to the size of the noted image region in the original image.
  • Here, the information relating to the position of the noted image region can further contain inclination information for the noted image region of the original image. In this case, by using the inclination information for the noted region of the original image it is possible to form a composite image displaying a desired element image superimposed at a position corresponding to the inclination of the noted image region in the original image. It is also possible to cause display by holding inclination in the displayed element image corresponding to inclination of the noted image region in the original image. Incidentally, the inclination information can be a single angle information representing a two dimensional inclination of the noted image, or two or three angle information representing three dimensional inclination of the noted image.
  • As described above, when position information for a representative point of the noted image in the original image and inclination information for the noted image region in the original image are contained in the information relating to position of the noted image region, representative point of the noted image region is made the origin in the relationship information, and it is possible for coordinate position data of a coordinate system determining direction of a coordinate axis based on inclination of the noted image region to be included as display position information for the specified element image. In this case, the information in the relationship information with respect to a representative point of the noted image region can be simply interpreted as coordinate position information of a coordinate system that determined by direction of coordinate axis based on inclination of the noted image region, wherein the system has an origin as a representative point of the noted image region. Herein, the noted information is the information representing a relative position of information relating to the position.
  • With the image processing method of the present invention, it is possible for the relationship information to contain proportion information for size of the noted image region and size of the specified graphic at the time of display. In this case, even when size of the noted image within the original image is changed from moment to moment, it is possible to form a mask image containing element images of size appropriate to the size of the noted image region.
  • Also, with the image processing method of the present invention, it is possible for the mask image information to further comprise respective execution time information for element image display fixed by combining the relationship information and element image selection information corresponding to the relationship information. In this case, by forming a mask image according to the execution time information, it is possible to display respective desired element images for only display execution times set to respective to desired times, and it is possible to form a mask image to vary the dynamic.
  • The mask image information further comprises information for display of a plurality of types of element image display in time sequence, fixed by combining the relationship information and element image selection information corresponding to the relationship information, and for time sequence display designation for designating repeat of time sequence display. In this case, by forming a mask image according to the time sequence display designation information, it is possible to display a plurality of types of element image on the original image repeatedly in time sequence every desired time.
  • Also, with the image processing method of the present invention, it is possible for the mask image information to further comprise simultaneous display designation information for designating display of a plurality of types of element image display at the same time, fixed by combining the relationship information and element image selection information corresponding to the relationship information. In this case, it is possible to form a mask image having a desired plurality of element images respectively displayed as desired positions, and it is possible to form mask images of numerous variations.
  • Note that, with the image processing method of the present invention, it is possible to make the original image a moving picture. In this case, for a moving picture in which position, inclination and size of the noted image in the original image is changing, it is possible to form a composite image that is dynamic and not unpleasant, in accordance with the user's interests.
  • A fixed phenomenon occurrence determination step for determining whether or not a predetermined phenomenon has occurred in the noted image region based on the noted image information is further provided, and the mask image data formation step and composite image data formation step are commenced after positive determination in the specified phenomenon occurrence determination step. In this case, in the fixed phenomenon occurrence detection step, when a predetermined phenomenon has occurred in the noted image region, since the mask image data formation step and the composite image formation step are commenced, it is possible to form a mask image where rendering is carried out in accordance with that phenomenon, and it is possible to form a composite image having that mask image superimposed on the original image. For example, in the case of a person's face, when the eyelids in the noted image can be closed, it is possible to form a composite image with tears flowing from the eyes.
  • An image processing device of the present invention is an image processing device for carrying out processing of an image, and comprises noted image information extracting means for specifying a noted image region, being a region for an image to be focused on, in an original image, and extracting noted information relating to position and size of the noted image region within the original image, storage means for storing a mask image information file registering mask image information containing at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information for a specified element image respectively corresponding to the relationship information, and at least one element image information selected using the element image selection information, mask image formation means for forming a mask image displayed superimposed on the original image, based on the noted image information and the mask image information, and composite image creation means for creating a composite image having the mask image superimposed on a noted image comprising image that includes the noted image inside the noted image region at the same location and same size as for the original image.
  • With this image processing device, noted image information extracting means specifies a noted image region in an original image, and extracts noted information relating to position and size of the noted image region within the original image. Continuing on, mask image data formation means forms a mask image by displaying superimposed on the original image, based on noted image information and mask image information registered in the mask image information file. Then, composite image formation means forms a composite image having the mask image superimposed on a noted image comprising image that includes the noted image within the noted image region at the same location and same size as for the original image.
  • Specifically, with the image processing device of the present invention, a composite image is formed using the image processing method of the present invention. Accordingly, according to the image processing device of the present invention it is possible to simply and appropriately form an image that a user has rendered on the noted image comprising image according to their interests, regardless of whether the original image is a still image or a moving picture.
  • With the image processing device of the present invention, it is possible for the noted image comprising image to comprise the original image. In this case, rendering of the original image according to the user's interests can be carried out by selecting a mask image.
  • With the image processing device of the present invention, it is possible to have a structure where the mask image information file further comprises background image selection information representing selection of information for a specified background image, and the composite image formation means cuts a specified display image containing the noted image from the original image, to form the composite image after a background addition image has been made the noted image comprising image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
  • In this case, the composite image formation means first of all extracts a specified display image containing a noted image from the original image. Continuing on, the composite image formation means superimposes the specified display image on a background image designated using background image selection information contained in the mask image information file, to form a background addition image. The composite image formation means then superimposes the above described mask image on the background addition image. Accordingly, rendering of the specified display image, such as a figure and so forth, according to the user's interests can be carried out by selecting a mask image and a background image.
  • Also, with the image processing device of the present invention, it is possible have a structure further comprising display means for displaying the composite image. In this case, it is possible to confirm a composite image formed by a user of the composite image rendering according to their own interests by displaying the composite image using display means.
  • Also, with the image processing device of the present invention, it is possible have a structure further comprising display means for displaying the composite image. In this case, because it is possible to transmit a composite image formed by a user of the composite image carrying out rendering according to their interests at the transmission side towards the receive side, it is possible to notify a composite image formed by the user rendering according to their interests to the receiving party.
  • With the image processing device of the present invention, it is possible have a structure further comprising imaging means for acquiring the original image. In this case, it is possible to make an image acquired using the imaging means the original image, and it is possible to easily and appropriately form an image that a user has rendered on the original image according to their interests. Note that, it is also possible for an image acquired by the imaging means to be a still image or a moving picture.
  • A mobile communication terminal of the present invention is a mobile communication terminal provided with the imaging processing device of the present invention. With this mobile communication terminal, since it is possible to transmit a composite image formed by the image processing device of the present invention to a receive side terminal, it is possible to transmit an image that a user renders in an original image according to their tastes regardless of whether it is a still image or a moving picture.
  • EFFECTS OF THE INVENTION
  • As is described above, according to the image processing method of the present invention, it is possible to simply and appropriately form an image that a user rendered in an original image, regardless of whether it is a still image or a moving picture.
  • Also, according to the image processing device of the present invention, since it is possible to form a composite image using the image processing method of the present invention, it is possible to easily and appropriately form an image that was rendered by a user in an original image regardless of whether the original image is a moving picture or a still image.
  • Also, according to the mobile communication terminal since the image processing device of the present invention is provided it is possible to transmit an image a user appropriately produced for an original image, regardless of whether it is a moving picture or a still image.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1A schematically shows the external structure of a front side of a mobile telephone unit of a first embodiment of the present invention;
  • FIG. 1B schematically shows the external structure of a rear side of a mobile telephone unit of the first embodiment of the present invention;
  • FIG. 2 is functional block diagram for describing the internal structure of the mobile telephone unit of FIG. 1A and FIG. 1B;
  • FIG. 3 is functional block diagram for describing the internal structure of a mask image processing section of FIG. 2;
  • FIG. 4A is a first drawing for describing noted image region information.
  • FIG. 4B is a second drawing for describing noted image region information;
  • FIG. 5A is drawing for describing the mask image information file structure of FIG. 2;
  • FIG. 5B is drawing for describing the element image file structure of FIG. 5A;
  • FIG. 6 is a first drawing showing an example of an element image;
  • FIG. 7 is a first drawing showing example contents of the display designation file of FIG. 5A;
  • FIG. 8A is a second drawing showing example contents of the display designation file of FIG. 5A;
  • FIG. 8 b is a third drawing showing example contents of the display designation file of FIG. 5A;
  • FIG. 9A is a drawing showing an example of a noted image;
  • FIG. 9B is a drawing showing an example for describing the noted image coordinate system in FIG. 9A;
  • FIG. 10A is a drawing showing an example of an original image;
  • FIG. 10B is a first drawing showing an example of a mask image;
  • FIG. 11 is a first drawing showing an example of a composite image of the first embodiment;
  • FIG. 12A is a second drawing showing an example of an element image;
  • FIG. 12B is a third drawing showing an example of an element image;
  • FIG. 12C is a fourth drawing showing an example of an element image;
  • FIG. 13 is a second drawing showing an example of a mask image;
  • FIG. 14 is a second drawing showing an example of a composite image of the first embodiment;
  • FIG. 15A is a fifth drawing showing an example of an element image;
  • FIG. 15B is a sixth drawing showing an example of an element image;
  • FIG. 15C is a seventh drawing showing an example of an element image;
  • FIG. 16 is a third drawing showing an example of a mask image;
  • FIG. 17 is a third drawing showing an example of a composite image of the first embodiment;
  • FIG. 18 is a functional block diagram for describing the internal structure of a mobile telephone unit of a second embodiment of the present invention;
  • FIG. 19 is a functional block diagram for describing the internal structure of a mask image processing section of FIG. 18;
  • FIG. 20 is a drawing for describing the mask image information file structure of FIG. 18;
  • FIG. 21 is a drawing showing example contents of the display designation file of FIG. 20;
  • FIG. 22 is a drawing showing an example of a background image;
  • FIG. 23 is a drawing showing an example of a specified display image; and
  • FIG. 24 is a flowchart for explaining a composite image formation operation for composite image formation section in FIG. 19.
  • DETAILED DESCRIPTION First Embodiment
  • A first embodiment of the present invention will be described in the following with reference to FIG. 1A-FIG. 17. The structure of a mobile telephone unit 10, being a mobile communication terminal of a first embodiment, is shown schematically in FIG. 1 and FIG. 2. In this embodiment, a front view of the external appearance of the mobile telephone unit 10 is shown in FIG. 1A, and a rear view of the external appearance of the mobile telephone unit 10 is shown in FIG. 1B. Also, the functional block structure of the mobile telephone unit 10 is shown in FIG. 2.
  • As shown comprehensively in FIG. 1A, FIG. 1B and FIG. 2, the portable telephone unit 10 is comprised of (a) a portable telephone body 11 provided with a control section 21 and so forth, which will be described later, (b) an operating section 12 having a numerical keypad for input of telephone numbers and function keys for inputting various commands such as operation mode switching, to the control section 21, and (c) a display section 13 having a liquid crystal display for displaying operation guidance, operating status, received messages etc. in response to commands from the control section 12. The portable telephone 10 is also comprised of (d) a call speaker 14 for reproducing voice signals transmitted from a calling party at the time of a call and a microphone 15 for inputting sound at the time of a call, and (e) a guidance speaker 16 for generating ring alerts and guidance voice in response to commands from the control section 21. Further, the portable telephone unit 10 is further comprised of (f) an antenna 17 connected to a transmission section 22 for transceiving wireless signals between the unit and a base station. Still further, the portable telephone unit 10 comprises (g) an imaging optical system for taking pictures of an operator side operating the operating section 12 while looking at the display on the display section 13, that is, for carrying photography in a so-called self-facing state, and (h) an imaging optical system 19 for taking pictures facing towards the field of vision of the operator, that is, taking pictures in a so-called other-party state.
  • As shown in FIG. 2, the portable telephone body 11 comprises (i) a control section 21 for unified control of all operations of the portable telephone unit 10, and also performing various data processing, (ii) a transceiver section 22 for transceiving wireless signals to and from a base station via the antenna 17, and (iii) a storage section 23 for storing various data. The portable telephone body 11 also comprises a switching optical system 24 for making either light passing through the imaging optical system 18 or light passing through the imaging optical system 19 be emitted towards an imaging element 25 that will be described later, under the control of the control section 21, and (iv) an imaging element 25 such as a CCD element for imaging an optical image formed by light on a light receiving surface via the switching optical system 24. In this case, the imaging element 25 carries out imaging operations under the control of the control section 21, and imaging results are notified to the control section 21.
  • The control section 21 is provided with a basic control section 29 and a mask image processing section 30. In this case, the basic control section 29 controls starting and stopping of operation of the mask image processing section 30, and also controls each element arranged external to the control section 21 and performs data processing besides image processing for mask image formation, that will be described later.
  • As shown in FIG. 3, the mask image processing section 30 mentioned above comprises a noted image extraction section 31, a mask image formation section 32 and a composite image formation section 33. The noted image extraction section 31 specifies a noted image region, such as a facial region of a specified person (for example, the user of the portable telephone unit 10) in the original image, for example, based on original image data OMD from the imaging element 25, and extracts noted image information WMD made up of position information for a representative point of the noted image region in the original image, inclination information of the noted image region in the original image, and size information of the noted image region.
  • Description will now be given of the extraction of the noted image information WMD. In the following, to simplify the description a 2-dimensional case will be dealt with disregarding the structure with respect to a display surface of body to be imaged in the original image, and the structure of the body to be images along a length direction in the original image will not be considered. Also, as shown in FIG. 4A and FIG. 4B, a noted image region WMR is specified in the original image region OMR. In this case, the case where the noted image region WMR is inclined on the reference (that is inclination angle θW=0) is shown schematically in FIG. 4A. Also, the case where inclination is only θ0 from the reference inclination (inclination angle θW0) is shown in FIG. 4B. Each element position in the original image region OMR is represented using coordinate values of an XS YS coordinate system, which is the original image coordinate system. Also, in the noted image WMR, the representative point OW is made the origin, and when inclination angle θW=0 an axis parallel to the XS axis is made an XW axis, and an axis parallel to the YS axis is made a YW axis to define a noted image coordinate system (that is an XW YW coordinate system). Inclination angle θW is represented by a counterclockwise angle.
  • In the case of FIG. 4A, after specifying a noted image region WMR in the original image region OMR, the noted image information extraction section 31 first of all obtains coordinate position OW (XSW1,YSW1) of the original image coordinate system for the representative point OW, as position information for the representative point in the noted image WMR. In this case, as the representative point of the noted image region WMR, it is possible to adopt a distinctive point of a noted image region acquired using image analysis, such as taking a tip position of a nose in a facial image, in the case where the noted image us a facial image of a specified person.
  • Next, the noted image information extraction section 31 obtains a crossing angle θW of the XS axis and the XW axis (that is, the above described inclination angle θW) as inclination information of the noted image WMR in the original image region OMR. Inclination angle θW is obtained by calculating a difference between a crossing angle of the XS axis and a characteristic direction as a noted image in the noted image region WMR defined with a predetermined inclination angle θW of “0”, and a crossing angle of the XS axis and the characteristic direction in the noted image inside the original image. As this characteristic direction, for example, it is possible to adopt a long axis direction or short axis direction of an approximate ellipse in the case where the shape of the noted region WMR can approximate to elliptical, or a direction linked to a central position both eyes in a facial image in the case where the noted image is a facial image of a specified person. If the inclination angle θW (in the case of FIG. 4A, θW=0) is obtained in this manner, it becomes possible to obtain the XW YW coordinate system (noted coordinate system) within the original image region OMR.
  • Next, the noted image information extraction section 31 obtains a characteristic length relating to the noted image region WMR, as size information for the noted image region WMR. In this embodiment, the width of the noted image region on the XW axis is made the characteristic length (width LXW in FIG. 4A). Note that, as the size information for the noted image region WMR, it is possible to adopt the width of the noted image region WMR on the YW axis, maximum width of the noted image region WMR in the XW axis (or YW axis) direction, a distance between two characteristic points in the noted image region (for example, a distance between the centers of two eyes in a facial image when the noted image is a facial image of a specified person), or the length of an XW axis component of the distance or the length of a YW axis component of the distance, as well as the width LXW of the noted image region WMR on the XW axis.
  • In the case of FIG. 4B also, similarly to the case of FIG. 4A described above, after specifying a noted image region WMR in the original image region OMR, the noted image information extraction section 31 obtains position information, inclination information and size information for the representative point in the noted image WMR. Note that, in the case of FIG. 4B, a coordinate position OW (XSW2,YSW2) for the original image coordinate system of the representative point OW is obtained as the position information of the representative point, inclination angle θW0 is obtained as the inclination information of the noted region WMR, and a width LXW of the noted image region on the XW axis is obtained as the size information of the noted image region WMR.
  • The above mentioned mask image formation section 32 forms mask image data MMD based on noted image information WMD extracted by the noted image information extraction section 31, and mask image information MDD stored in the above mentioned storage section 23. Note that, formation of the mask image data MMD using the mask image information MDD and the mask image formation section 32 will be described later.
  • The above mentioned composite image formation section 33 forms composite image data SMD having the mask image superimposed on the original image based on the original image data OMD and the mask image data MMD. Note that, the composite image data SMMD formed by the composite image formation section 33 is notified to the basic control section 29.
  • Note that, in this embodiment, the control section 21 is provided with a central processing unit (CPU), digital signal processor (DSP) and so forth. Then, various data processing is carried out by the basic image processing section 29 and mask image processing section 30, constituted by programs executed by the control section 21, in order to realize portable telephone functions, including mail interchange processing, and besides that operational control of other structural elements is also performed.
  • Returning to FIG. 2, a mask image information file 40 is stored in the storage section 23. As shown in FIG. 5A, this mask image information file is comprised of (i) a display designation file 41 j storing display designation descriptors for designating as to with what relationship to the noted image region in the original image the mask image is displayed, and (ii) an element image file 42 j storing element image data designated using display designation descriptors in the display designation file 41 j. Note that, the mask image information file 40 is provided to the portable telephone unit 10 from a contents server via a wireless circuit, or is provided after being formed by the user using a personal computer or the like by means of storage medium or an external interface, not shown in this figure.
  • As shown in FIG. 5B, element images are stored in the element image file 42 j. Note that, hereafter, the ith element image, stored in the element image file 42 j, will be denoted element image EMMji. With FIG. 5B, a case is shown where there are a plurality of element images stored in the element image file 42 j, but there may be cases where the number of element images stored in the element image file 42 j is one, or cases where it is 2 or more. A element image EMMji, for example, there is a heart shaped image or a lip shaped image and so forth.
  • In this manner, the element image EMMji is formed in a coordinate system inherent to the element image, that is in the XE YE coordinate system making a representative point (in FIG. 6 the center point of element image EMM11) of element image EMMji an origin OE, as shown representatively for element image EMM11 in FIG. 6. Then, a maximum width LXEi (in FIG. 6 maximum width LXE1) in the direction of the XE is used as the size information for element image EMMji. In FIG. 6, the element image EMMji has been made as a heart shaped element image EMM11, but the same applies even if it is an element image of a different shape or size.
  • Example of display designation descriptors for the above described display designation file 41 j are shown in FIG. 7, FIG. 8A and FIG. 8B. Note that, the display designation descriptors of this embodiment are described in a format conforming to the SMIL (Synchronized Multimedia Integration Language) format.
  • In FIG. 7, an example for carrying out display designation to display the same element image steadily as two positions (for example, positions of two eyes in the case where the noted image is a facial image of a specified person) fixed in accordance with position, inclination and size of the noted image region is shown as the display designation file 41 1. Specifically, display of additional image 1 and additional image 2, designated by <additional image 1 designation> and <addition image 2 designation> described between <simultaneous display designation> and <end simultaneous display designation> is designated.
  • Additional image 1 is designated using the descriptors between <additional image 1 designation> and <end additional image 1 designation>. Specifically, element image EMM11 designated by “element image designation” is display at a coordinate position (XW11, YW11) of the XW YW coordinate system in the case where size information for the noted image region WMR is a value LXW0 constituting a predetermined reference, as designated using “display position designation”. Also, as designated by “size designation”, a maximum width in a XW axis direction of the element image EMM11 at the time of display is made a length of proportion R1 with respect to size LXW of the noted image region WMR. Note that, from the fact that display time designation, which will be described later, is not carried out, the element image EMM11 for additional image 1 is designated to be displayed continuously.
  • Additional image 2 is designated using the descriptors between <additional image 2 designation> and <end additional image 2 designation>. Specifically, element image EMM11 designated by “element image designation” is display at a coordinate position (XW12,YW12) of the XW YW coordinate system in the case where size information for the noted image region WMR is a value LXW0 constituting a predetermined reference, as designated using “display position designation”. With respect to designation other than this, the same display designation as for the case of additional image 1 is carried out because the same designations as for the case of additional image 1 are carried out.
  • Note that, in the case of the display designation file 41 1 as in FIG. 7, only element image EMM11 is stored in the element image file 42 1 corresponding to the display designation file 41 1, as an element image.
  • In FIG. 8A, an example for carrying out display designation to display the one additional image 1 whose display contents vary with time at a single position (for example, position of a mouth in the case where the noted image is a facial image of a specified person) fixed in accordance with position, inclination and size of the noted image region is shown as the display designation file 41 2. With the display designation file 41 2 of FIG. 8A, a plurality of element images can not be displayed at the same time, which means that <simultaneous display designation> and <end simultaneous display designation> relating to the simultaneous display in FIG. 7 are not described.
  • Additional image 1 is designated using the descriptors between <additional image 1 designation> and <end additional image 1 designation>. Specifically, first of all an element image EMM21 designated by a first “element image designation” is displayed at a coordinate position (XW21,YW21) of the XW YW coordinate system in the case where size information LXE for a noted image region designated using a first “display position designation” is a value LXE0 constituting a predetermined reference. Also, as designated by the first “size designation”, a maximum width in a XE axis direction of the element image EMM2, at the time of display is made a length of proportion R21 with respect to size LXW of the noted image region WMR. Also, as designated by the first “display time designation”, the element image EMM21 is displayed for a time T21.
  • If display of element image EMM21 for time T21 is completed, next an element image EMM22 designated using a second “element image designation” is displayed at a position designated using a second “display position designation”, that is, at the same position as for the case of element image EMM21. Also, as designated by the second “size designation”, a maximum width in a XE axis direction of the element image EMM22 at the time of display is made a length of proportion R22 with respect to size LXW of the noted image region WMR. Further, as designated by the second “display time designation”, the element image EMM22 is displayed for a time T22.
  • If display of element image EMM22 for time T22 is completed, next an element image EMM23 designated using a third “element image designation” is displayed at a position designated using a third “display position designation”, that is, at the same position as for the case of element image EMM21. Also, as designated by the third “size designation”, a maximum width in a XE axis direction of the element image EMM23 at the time of display is made a length of proportion R23 with respect to size LXW of the noted image region WMR. Further, as designated by the third “display time designation”, the element image EMM23 is displayed for a time T23.
  • If display of the element image EMM21, element image EMM22, and element image EMM23, in time sequence in this way is completed, the element image EMM21, element image EMM22, and element image EMM23 are displayed again in time sequence under the same conditions as described above. Display of the element image EMM21, element image EMM22, and element image EMM23 in this way using designation conditions is repeated in time order.
  • An example for carrying out display designation in order to display one element image all the time at a single position fixed in accordance with position, inclination and size of the noted image region, and also displaying two different element images selectively in time order at two other positions determined in accordance with position, inclination and size of the noted image region is shown in FIG. 8B as an example of a display designation file 413. Specifically, display of additional image 1 and addition image 2, designated by <additional image 1 designation> and <additional image 2 designation> described between <simultaneous display designation> and <end simultaneous display designation> is designated.
  • Additional image 1 is designated using the descriptors between <additional image 1 designation> and <end additional image 1 designation>. Specifically, element image EMM3, designated by “element image designation” is displayed at a coordinate position (XW31,YW31) of the XW YW coordinate system in the case where size information for the noted image region WMR is a value LXW0 constituting a predetermined reference, as designated using “display position designation”. Also, as designated by “size designation”, a maximum width in a XW axis direction of the element image EMM31 at the time of display is made a length of proportion R31 with respect to size LXW of the noted image region WMR. Note that, because display time designation is not carried out, the element image EMM31 for additional image 1 is designated to be displayed continuously.
  • Additional image 2 is designated using the descriptors between <additional image 2 designation> and <end additional image 2 designation>. Specifically, first of all an element image EMM32 designated by a first “element image designation” is displayed at a coordinate position (XW32,YW32) of the XW YW coordinate system in the case where size information for a noted image region WMR, as designated using a first “display position designation”, is a value LXW0 constituting a predetermined reference. Also, as designated by “size designation”, a maximum width in a XW axis direction of the element image EMM32 at the time of display is made a length of proportion R32 with respect to size LXW of the noted image region WMR. Also, as designated by the first “display time designation”, the element image EMM32 is displayed for a time T32.
  • With respect to additional image 2, if display of element image EMM32 for T32 is completed, next an element image EMM33 designated by a second “element image designation” is displayed at a coordinate position (XW33,YW33) of the XW YW coordinate system in the case where size information for a noted image region WMR, as designated using a second “display position designation”, is a value LX0 constituting a predetermined reference. Also, as designated by “size designation”, a maximum width in a XW axis direction of the element image EMM33 at the time of display is made a length of proportion R33 with respect to size LXW of the noted image region WMR. Further, as designated by the second “display time designation”, the element image EMM33 is displayed for a time T33.
  • Simultaneous display of element image EMM31 and element image EMM32, and simultaneous display of element image EMM31 and element image EMM33, by combining additional image 1 and additional image 2 in this way, are carried in time sequence. If these displays are completed, simultaneous display of element image EMM31 and element image EMM32, and simultaneous display of element image EMM31 and element image EMM33, are again carried out in time sequence under the same conditions as described above. Simultaneous display of element image EMM31 and element image EMM32, and simultaneous display of element image EMM31 and element image EMM33 in this way, are repeatedly carried out in time sequence.
  • In the following description will be given of the operation of forming a composite image with a mask image superimposed on an original image, in the portable telephone unit constructed as described above.
  • As a prerequisite, the mask image formation section 32 has already read out contents of the mask information file 40 in the storage section 23 as mask image information MDD. Also, the original image is an image that is formed on an imaging screen of the imaging element 25 by means of the imaging optical system 18 and the switching optical system 24. The noted image WMM is a facial image of a specified person, such as shown in FIG. 9A. Then, as shown in FIG. 9B, a tip position of a nose in the noted image WMM is made an origin OW, a direction connecting both sides is made an XW axis direction, and an XW YW coordinate system (noted image coordinate system) with a direction orthogonal to the XW axis direction being a YW axis direction is defined. Note that, the origin OW and the XW axis direction are obtained by analyzing the noted image in the original image, and the YW axis direction is obtained based on the obtained XW axis direction.
  • FIRST COMPOSITE IMAGE FORMATION EXAMPLE
  • First of all, a first composite image formation method will be described. This example, in FIG. 7 described above, is an example of a composite image having a position of two eyes as display positions, and element image EMM11 being the heart shaped graphic image shown in FIG. 6, for displaying a mask image that is the element image EMM11 displayed in a size of proportion R1 with respect to length value LXW representing size of noted image WMM, superimposed on an original image.
  • First of all, original image data OMD, being data for an original image OMM imaged on the imaging element 25, is notified to the noted image information extraction section 31. Note that, the original image data OMD is also notified to the composite image formation section 33 at the same time (refer to FIG. 3). In this case, the acquired original image OMM is shown in FIG. 10A.
  • The noted image information extraction section 31 image processes the notified original image OMM to extract a region of the noted image WMM. Continuing on, the noted image information extraction section 31 analyzes the noted image WMM, to obtain a position of a tip of a nose (XSW,YSW) in the noted image WMM of the original image coordinate system (XS YS coordinate system) as a position of an origin OW for the noted image coordinates (XW YW coordinate system), and obtain a direction in which a line segment linking two central position of pupils of two eyes extends as an XW axis direction for the XW YW coordinate system. A crossing angle θW of the obtained XW axis direction and the XS axis direction is then obtained. Further, the noted image information extraction section 31 obtains a width LXW of the noted image WMM on the XW axis. The noted image information extraction section 31 then notifies position (XSW,YSW) as position information for the noted image region, angle θW as inclination information for the noted image region, and length LXW as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • The mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (XE11,YE11) in the XS YS coordinate system of the element image EMM11 in the additional image 1, using the following equations (1) and (2).
    X E11=(LX W /LX 0)(X W11*cos θW +Y W11*sin θW)+X SW  (1)
    Y E11=(LX W /LX 0)(−X W11*sin θW +Y W11*cos θW)+Y SW  (2)
  • Next the mask image formation section 32 calculates a magnification M11 for expansion or contraction of the element image EMM11 in the additional image 1 from the following equation (3).
    M 11 =R 1 *LX W /LX E1  (3)
  • Next, the mask image formation section 32 carries out expansion or contraction of the element image EMM11 with a magnification M11 and with origin OE as a center in the XS YS coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM11 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM11 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE11,YE11) in the XS YS coordinate system becomes the position of the origin OE.
  • Next, similarly to the case for additional image 1, the mask image formation section 32 calculates display position (XE12,YE21) in the XS YS coordinate system of the element image EMM11 in the additional image 2, using the following equations (4) and (5).
    X E12=(LX W /LX 0)(X W12*cos θW +Y W12*sin θW)+X SW  (4)
    Y E12=(LX W /LX 0)(−X W12*sin θW +Y W12*cos θW)+Y SW  (5)
  • Next, similarly to the case for additional image 1, the mask image formation section 32 calculates a magnification M12 for expansion or contraction of the element image EMM11 in the additional image 2 from the following equation (6).
    M 12 =R 1 *LX W /LX E1  (6)
  • Next, the mask image formation section 32 carries out expansion or contraction of the element image EMM11 with a magnification M12 and with origin OE a center in the XS YS coordinate system as. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM11 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM11 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE12,YE12) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMMA such as is shown by image F10B in FIG. 10B is formed. The mask image formation section 32 then notifies data of the mask image MMMA to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received the mask image data MMD and the original image data OMD forms a composite image SMMA having the mask image MMMA superimposed on the original image data OMM. The composite image SMMA formed in this way is represented as image F11A in FIG. 11.
  • As described above, the composite image for the case of this example is formed. Therefore, even if the noted image WMM of the original image is moved parallel from the above described example during imaging, as shown in image F11B, the same as with the above described case a composite image SMMB is formed having a mask image with the element image EMM11 displayed at size that is a proportion designated according to size of the noted image region at a position of two eyes in the noted image WMM superimposed on the original image.
  • Also, even if the noted image WMM of the original image is contracted during imaging, as shown in image F11C, the same as with the above described case a composite image SMMC is formed having a mask image with the element image EMM11 displayed at size that is a proportion designated according to size of the noted image region at a position of two eyes in the noted image WMM superimposed on the original image. Also, even if the noted image WMM of the original image is rotated during imaging, as shown in image F11D, the same as with the above described case a composite image SMMD is formed having a mask image with the element image EMM11 displayed having a size that is a proportion designated according to size of the noted image region at a position of two eyes in the noted image WMM, and having the same inclination as the inclination of the noted image WMM within in the original image, superimposed on the original image. Note that, when the noted image WMM in the original image is contracted, expanded or rotated during imaging, it is generally accompanied by change in coordinate position in the original image coordinate system (XS YS coordinate system) for nose tip position in the noted image region.
  • SECOND COMPOSITE IMAGE FORMATION EXAMPLE
  • Next, a second composite image formation method will be described. This example is an example of a composite image where, in FIG. 8A described above, a mask image having element image EMM21 shown in FIG. 12A, element image EMM22 shown in FIG. 12B and element image EMM23 shown in FIG. 12C displayed in time sequence, with the position of a mouth as display position, is superimposed on the original image.
  • Note that, element images EMM21, EMM22 and EMM23 are displayed in a size corresponding to “size information” being designated by the descriptor of FIG. 8A. Specifically, element image EMM21 is displayed in a size of proportion R21 with respect to length value LXW representing size of the noted image WMM. Also, element image EMM22 is displayed in a size of proportion R22 with respect to length value LXW. Further, element image EMM23 is displayed in a size of proportion R23 with respect to length value LXW.
  • First of all, similarly to the first composite image forming example described above, data for an original image OMM imaged on the imaging element 25 is notified to the noted image information extraction section 31 as original image data OMD. Continuing on, the noted image extraction section 31 carries out image processing of the notified original image OMM and extracts the noted image information WMR specifying the region of the noted image WMM. The noted image information extraction section 31 then notifies position (XSW,YSW) as position information for the noted image region, angle θW as inclination information for the noted image region, and length LXW as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • The mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (XE2,YE2) in the XS YS coordinate system of the first element image EMM21 in the additional image 1, using the following equations (7) and (8).
    X E2=(LX W /LX 0)(X W2*cos θW +Y W2*sin θW)+X SW  (7)
    Y E2=(LX W /LX 0)(−X W2*sin θW +Y W2*cos θW)+YSW  (8)
  • Next the mask image formation section 32 calculates a magnification M21 for expansion or contraction of the first element image EMM21 from the following equation (9).
    M 21 =R 21 *LX W /LX E21  (9)
  • Next, similarly to the first composite image formation described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM21 with a magnification M21 and with origin OE as a center in the XS YS coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM2, by angle θW with origin OE in the XS YS coordinate system as a center. The element image EMM21 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE2,YE2) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMM21 such as is shown by image F13A in FIG. 13 is formed. The mask image formation section 32 then notifies data of the mask image MMM21 to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM21 having the mask image MMM superimposed on the original image OMM. The composite image SMM21 formed in this way is represented as image F14A in FIG. 14.
  • After that, during time T21, the mask image formation section 32 forms the mask image displaying element image EMM21 in a size of proportion R21 with respect to length value LXW representing the size of the noted image WMM, with mouth position in the noted image WMM as display position. As a result, during time T21, the composite image formation section 33 sequentially forms the composite image having the mask image that is the element image EMM21 that has been expanded or contracted at magnification M21 at the position of the mouth in the noted image WMM of the original image.
  • In this manner, once time T21 has elapsed, the mask image formation section 32 calculates display position (XE2,YE2) in the XS YS coordinate system of the second element image EMM22 in the additional image 1, using the above described equations (7) and (8). Continuing on, the mask image formation section 32 calculates a magnification M22 for expansion or contraction of the second element image EMM22 from the following equation (10).
    M 22 =R 22 *LX W /LX E22  (10)
  • Next, similarly to the case of the first element image EMM21 described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM22 with a magnification M22 and with origin OE as a center in the XE YE coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM22 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM22 that has been expanded or contracted and rotated is then arranged in the Xs YS coordinate system so that a coordinate position (XE2,YE2) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMM22 such as is shown by image F13B in FIG. 13 is formed. The mask image formation section 32 then notifies data of the mask image MMM22 to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM22 having the mask image MMM superimposed on the original image OMM. The composite image SMM22 formed in this way is represented as image F14B in FIG. 14.
  • After that, during time T22, the mask image formation section 32 forms the mask image displaying element image EMM22 in a size of proportion R22 with respect to length value LXW representing the size of the noted image WMM, with mouth position in the noted image WMM as display position. As a result, during time T22 the composite image formation section 33 forms the composite image having the mask image that is the element image EMM22 that has been expanded or contracted at magnification M22 at the position of the mouth in the noted image WMM of the original image.
  • In this manner, once time T22 has elapsed, the mask image formation section 32 calculates display position (XE2,YE2) in the XS YS coordinate system of the third element image EMM23 in the additional image 1, using the above described equations (7) and (8). Continuing on, the mask image formation section 32 calculates a magnification M23 for expansion or contraction of the second element image EMM23 from the following equation (11).
    M 23 =R 23 *LX W /LX E23  (11)
  • Next, similarly to the case of the first element image EMM21 described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM23 with a magnification M23 and with origin OE as a center in the XE YE coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM23 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM23 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE2,YE2) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMM23 such as is shown by image F13C in FIG. 13 is formed. The mask image formation section 32 then notifies data of the mask image MMM23 to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM23 having the mask image MMM23 superimposed on the original image OMM. The composite image SMM23 formed in this way is represented as image F14C in FIG. 14.
  • After that, during time T23, the mask image formation section 32 forms the mask image displaying element image EMM23 in a size of proportion R23 with respect to length value LXW representing the size of the noted image WMM, with mouth position in the noted image WMM as display position. As a result, during time T23 the composite image formation section 33 forms the composite image having the mask image that is the element image EMM23 that has been expanded or contracted at magnification M23 at the position of the mouth in the noted image WMM of the original image.
  • In this manner, once time T23 has elapsed, the mask image formation section 32, similarly to above, repeatedly forms the mask image that uses the above described element image EMM21, the mask image that uses the element image EMM22, and the mask image that uses the element image EMM23, in time sequence. The composite image formation section 33 then sequentially forms composite images with mask images comprised of the element image EMM21, element image EMM22 and element image EMM23 that have been expanded or contracted to a designated size superimposed at a position of a mouth of the noted image in the original image, based on original image data OMD and mask image data MMD.
  • THIRD COMPOSITE IMAGE FORMATION EXAMPLE
  • Next, a third composite image formation method will be described. This example corresponds to the mask image designation of FIG. 8 described above, and is an example of a composite image that displays a mask image, in which an additional image 1 that is displayed in a steady manner at a display position corresponding to an element image and a noted image, and an additional image 2 that is displayed at a display position that changes with time corresponding to an element image and a noted image, are displayed simultaneously, superimposed on an original image.
  • In this matter, the element image EMM31 for the additional image 1 is shown in FIG. 15A. Also, the first element image EMM32 for the additional image 2 is shown in FIG. 15B, and the second element image EMM33 for the additional image 2 is shown in FIG. 15C.
  • Note that, element images EMM31, EMM32 and EMM33 are displayed in a size corresponding to “size information” being designated by the descriptor of FIG. 8. Specifically, element image EMM31 is displayed in a size of proportion R31 with respect to length value LXW representing size of the noted image WMM. Also, element image EMM32 is displayed in a size of proportion R32 with respect to length value LXW. Further, element image EMM33 is displayed in a size of proportion R33 with respect to length value LXW representing size of the noted image WMM.
  • First of all, similarly to the first and second composite image formation described above, original image data OMD imaged on the imaging element 25 is notified to the noted image information extraction section 31. Continuing on, the noted image extraction section 31 carries out image processing of the notified original image OMM and extracts the noted image information specifying the region of the noted image WMM. The noted image information extraction section 31 then notifies position (XSW,YSW) as position information for the noted image region, angle θW as inclination information for the noted image region, and length LXW as size information of the noted image region to the mask image formation section 32 as noted image information data WMD.
  • The mask image formation section 32 that has received the noted image information data WMD first of all calculates display position (XE31,YE31) in the XS YS coordinate system of the first element image EMM31 in the additional image 1, using the following equations (12) and (13).
    X E31=(LX W /LX 0)(X W31*cos θW +Y W31*sin θW)+X SW  (12)
    Y E31=(LX W /LX 0)(−X W31*sin θW +Y W31*cos θW)+Y SW  (13)
  • Next the mask image formation section 32 calculates a magnification M3, for expansion or contraction of the first element image EMM31 from the following equation (14).
    M 31 =R 31 *LX W /LX E33  (14)
  • Next, similarly to the first and second composite image formation described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM31 with a magnification M31 and with origin OE as a center in the XE YE coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM3, by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM31 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE31,YE31) in the XS YS coordinate system becomes the position of the origin OE.
  • Next, similarly to the case for the first element image EMM31 of the additional image 1, the mask image formation section 32 calculates display position (XE32,YE32) in the XS YS coordinate system of the second element image EMM32 in the additional image 2, using the following equations (15) and (16).
    X E32=(LX W /LX 0)(X W32*cos θW +Y W32*sin θW)+XSW  (15)
    Y E32=(LX W /LX 0)(−X W32*sin θW +Y W32*cos θW)+Y SW  (16)
  • Next the mask image formation section 32 calculates a magnification M32 for expansion or contraction of the second element image EMM32 from the following equation (17).
    M 32 =R 32 *LX W /LX E32  (17)
  • Next, similarly to the case of the element image EMM31 described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM32 with a magnification M32 and with origin OE as a center in the XE YE coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM32 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM31 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE32,YE32) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMM31 such as is shown by image F16A in FIG. 16 is formed. The mask image formation section 32 then notifies data of the mask image MMM31 to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM31 having the mask image MMM31 superimposed on the original image OMM. The composite image SMM31 formed in this way is represented as image F17A in FIG. 17.
  • After that, during time T31, the mask image formation section 32 forms the mask image displaying element image EMM31 in a size of proportion R31 with respect to length value LXW representing the size of the noted image WMM, and displaying element image EMM32 in a size of proportion R32 with respect to length value LXW at a position that is a relationship designating the noted image WMM. During time T31, the composite image formation section 33 then forms a composite image having the mask image represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • In this manner, once time T31 has elapsed, the mask image formation section 32 calculates display position (XE31,YE31) in the XS YS coordinate system, and magnification M31, of the first element image EMM31 in the additional image 1, using the above described equations (12) to (14). Then, after expanding or contracting element image EMM31 with a magnification M31, the mask image formation section 32 then rotates by angle θW with origin OE as a center in the XE YE coordinate system. The element image EMM2 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE31, YE31) in the XS YS coordinate system becomes the position of the origin OE.
  • Next, similarly to the case for the first element image EMM11 of the additional image 1, the mask image formation section 32 calculates display position (XE33,YE33) in the XS YS coordinate system of the element image EMM33 in the additional image 2, using the following equations (18) and (19).
    X E33=(LX W /LX 0)(X W33*cos θW +Y W33*sin θ)+X SW  (18)
    Y E33=(LX W /LX 0)(−X W33*sin θW +Y W33*cos θ)+Y SW  (19)
  • Next the mask image formation section 32 calculates a magnification M33 for expansion or contraction of the third element image EMM33 from the following equation (20).
    M 33 =R 33 *LX W /LX E33  (20)
  • Next, similarly to the case of the first element image EMM31 described above, the mask image formation section 32 carries out expansion or contraction of the element image EMM33 with a magnification M33 and with origin OE as a center in the XE YE coordinate system. Continuing on, the mask image formation section 32 rotates the expanded or contracted element image EMM33 by angle θW with origin OE as a center in the XS YS coordinate system. The element image EMM33 that has been expanded or contracted and rotated is then arranged in the XS YS coordinate system so that a coordinate position (XE33,YE33) in the XS YS coordinate system becomes the position of the origin OE.
  • The result of the above processing is that a mask image MMM32 such as is shown by image F16B in FIG. 16 is formed. The mask image formation section 32 then notifies data of the mask image MMM32 to the composite image formation section 33 as mask image data MMD.
  • The composite image formation section 33 that has received this mask image data MMD and the original image data OMD forms a composite image SMM32 having the mask image MMM32 superimposed on the original image OMM. The composite image SMM32 formed in this way is represented as image F17B in FIG. 17.
  • After that, during time T32, the mask image formation section 32 forms the mask image displaying element image EMM31 in a size of proportion R31 with respect to length value LXW representing the size of the noted image WMM, and displaying element image EMM33 in a size of proportion R33 with respect to length value LXW, at a position that is a relationship designating the noted image WMM. During time T32, the composite image formation section 33 then forms a composite image having the mask image represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • In this manner, once time T32 has elapsed, the mask image formation section 32, similarly to above, repeatedly forms the mask image that uses the above described element image EMM31 and the element image EMM32, and the mask image that uses the element image EMM31 and the element image EMM33, in time sequence. The composite image formation section 33 then sequentially forms composite images having the mask images represented by mask image data MMD notified from the mask image formation section 32 superimposed on the original image.
  • Information data SMD of the composite images formed in the above described first, second and third composite image formation examples is notified from the composite image formation section 33 to the basic control section 29. In accordance with designation carried out by a predetermined user operating the operation section 12, the basic control section 29 that has received this composite image information data displays the composite image SMM on the display section 13, and transmits to a fellow caller via the transceiver section 21.
  • As has been described above, with this embodiment, a region of a noted image WMM is specified in an original image, and noted image information WMD containing information relating to position and size of the noted image WMM region within the original image is extracted. Continuing on, a mask image MMM displayed superimposed on the original image OMM is formed based on noted image information WMD and mask image information registered in the mask image information file 40. As a result, a mask image comprised of element images selected using element image selection information contained in mask image information is formed on the original image OMM at a position determined using information connected to position of a region of noted image WMM and information representing a relationship to position of the noted image region contained in mask image information, and at a size determined by information relating to size of the region of the noted image WMM in the original image OMM, and information representing a relationship to size of the noted image region. A composite image having the mask image superimposed on the original image is then formed.
  • Therefore, according to this embodiment, it is possible to simply and appropriately form an image that a user has rendered on original image according to their interests.
  • Also, with this embodiment, it is possible for the information relating to the position of the noted image region contained in the mask image information to contain position information for a representative point of the noted image region WMR in the original image OMM. Therefore, it is possible to make information representing a relationship to position of a noted region contained in the mask image information, information representing a relative relationship for a representative point of the noted image, and it is possible to simplify information representing a relationship with position of the noted image region in the relationship information.
  • Also, with this embodiment, it is possible for the information relating to the position of the noted image region contained in the mask image information to further contain inclination information for the noted image region in the original image OMM. Therefore, by using the inclination information for the noted region of the original image OMM it is possible to form a composite image displaying a desired element image superimposed at a position corresponding to the inclination of the noted image region in the original image. It is also possible to display by holding inclination in the displayed element image corresponding to inclination of the noted image region in the original image.
  • Also with this embodiment, as information relating to position of the noted image contained in the mask image information, a representative point of the noted image is made an origin, and coordinate position information in a coordinate system determining direction of a coordinate axis based on inclination of the noted image region in the original image is included as display position information for an element image. Therefore, the information with respect to a representative point of the noted image region can be simply interpreted as coordinate position information of a coordinate system that determined by direction of coordinate axis based on inclination of the noted image region, wherein the system has an origin as a representative point of the noted image region. Herein, the noted information is the information representing a relative position of information relating to the position.
  • Also with this embodiment, it is possible for the mask image information to contain proportion information for size of the noted image region and size of the element image EMM at the time of displaying. In this case, even when size of the noted image within the original image OMM is changed from moment to moment, it is possible to form a mask image MMM containing element images EMM of size appropriate to the size of the noted image region.
  • Also with this embodiment, it is possible for the mask image information to contain execution time information for respective display of the element image EMM. In this case, by forming a mask image WMM according to the execution time information, it is possible to display respective desired element images EMM for only display execution times set to respective to desired times, and it is possible to form a mask image MMM to vary the dynamic.
  • Also, with this embodiment, it is possible for the mask image information to contain information time sequence display designation, for designating display a plurality of element images EMM in time sequence, and repeat of time sequence display. In this case, by forming a mask image WMM according to the time sequence display designation information, it is possible to display a plurality of element images on the original image OMM repeatedly in time sequence every desired time.
  • Also, with this embodiment, it is possible for the mask image information to contain information for simultaneous display designation, for designating display of a plurality of element images at the same time. In this case, it is possible to form a mask image having a desired plurality of element images EMM respectively displayed as desired positions, and it is possible to form mask images WMM of numerous variations.
  • Second Embodiment
  • Next, a second embodiment of the present invention will be described mainly with reference to FIG. 18 to FIG. 24, and with reference to appropriate other drawings. Note that, elements that are the same or similar to the above described first embodiment have the same reference numbers assigned, and repeat description will be omitted.
  • Compared to the first embodiment, this embodiment is different in that it is possible to take different images to the original image in the background to a person's image. In the following, description will mainly focus on points of difference.
  • As shown in FIG. 18, a portable telephone unit 10′ of this embodiment is different from the portable telephone unit of the first embodiment in that the mask image processing section 30′ is provided in the control section 21 and in that the mask image information file 40′ is stored in the storage section 23, inside of the portable telephone body 11.
  • As shown in FIG. 19, the mask image processing section 30′ comprises a noted image extraction section 31, a mask image formation section 32′ and a composite image formation section 33′. In this case, the noted image information extraction section 31 operates in the same manner as in the case of the first embodiment.
  • As with the mask image formation section 32 of the first embodiment, the mask image formation section 32′ forms mask image data MMD based on noted image information WMD extracted by the noted image information extraction section 31, and mask image information MDD stored in the mask image information file 40′ in the above mentioned storage section 23. Also, the mask image formation section 32′ creates background image information BPN made up of background image designation information representing whether or not there is background image designation and storage position information for the background image data BDD in the background designation information in the event that there is background image designation, based on mask image information MDD stored in the mask image information file 40′.
  • When the fact that there is background image designation has been notified using the background image information BPN, the composite image formation 33′ first of all forms an image that has a person's image, being a specified display image within an original image, superimposed on a background image using the background image data BDD, as a background image containing image, based on original image data OMD and background image data BDD. Continuing on, the composite image formation section 33′ forms composite image data SMD′, wherein a mask image represented using mask image information MDD is superimposed on the image containing background image.
  • On the other hand, when the fact that there is no background image designation is notified using the background image information BPN, the composite image formation means 33′ forms the original image as a background image containing image. Continuing on, the composite image formation section 33′ forms composite image data SMD having a mask image represented using mask image information MMD superimposed on the background image containing image. In this case, the composite image data SMD′ is the same as the composite image data SMD of the first embodiment.
  • The composite image data SMD′ formed by the composite image formation section 33 is notified to the basic control section 29.
  • As shown in FIG. 20A, the mask image information file 40′ inside the storage section 23 is comprised of (i) a display designation file 41′ storing display designation descriptors for designating as to with what relationship to the noted image region in the original image the mask image is displayed and background image designation descriptors, (ii) the above described element image file 42, and (iii) a background image file 43 storing background image data BDD. Note that, the mask image information file 40′ is provided to the portable telephone unit 10 from a contents provider via wireless circuit, or is provided after being formed by the user using a personal computer or the like by means of storage medium or a non-shown external interface.
  • An example of display designation descriptors for the display designation file 41′ of this embodiment is shown in FIG. 21. The descriptor example in FIG. 21 has the fact that there is background image designation added to the descriptor example of FIG. 7 for the case of the first embodiment. Specifically, with the display designation descriptor of FIG. 21, in addition to the descriptor for additional image 1 and additional image 2 in FIG. 7 from <simultaneous display designation> to <end simultaneous display designation>, a background image to be displayed simultaneously using the descriptor “background image designation” from <background image designation> to <end background image designation> is designated. Here, a header address BMMA for the background image file 43 in the storage section 23 is designated as “background image designation”. In the following description, background image data BDD representing a background image BMM inside the background image region BMR having the same shape and size as the original image region OMR shown in FIG. 22 is stored in the background image file 43.
  • Note that, when there is no descriptor from <background image designation> to <end background image designation>, it means that there is no background image designation. Also in the case where “0” is described as the header address BMMA value for the background image file 43, it means that there is no background image designation.
  • In the following description will be given of the operation of forming a composite image in the portable telephone unit 10′ constructed as described above.
  • In addition to the prerequisites for the composite image formation operation in the first embodiment described above, the person image SDM inside the original image region OMR shown in FIG. 23 is a specified display image to be cut from the original image OMM.
  • First of all, similarly to the case of the first composite image forming example of the first embodiment, original image data OMD, being data for an original image OMM imaged on the imaging element 25, is notified to the noted image information extraction section 31, and noted image information data WMD is formed by the noted image information extraction section 31. The noted image data WMD formed in this way is notified from the noted image information extraction section 31 to the mask image formation section 32′.
  • Similarly to the case of the first composite image formation example for the first embodiment, the mask image formation section 32′ that has received the noted image information data WMD creates data of a mask image MMMA based on noted image information data WMD and mask image information MDD, and notifies the data to the composite image formation section 33′ as mask image data MMD. Also, the mask image formation section 32′ creates background image information BPN comprising whether or not there is background image designation, and a header address of a background image file 43 for when there is background image designation, based on mask image information MDD, and notifies the background image information BPN to the composite image formation section 33′. In this case, as shown in FIG. 21 described above, background image designation is described in the display designation file 41′ and so the fact that there is background image designation and a header address of the background image file 43 is included as background image information BPN.
  • The composite image formation section 33′ that has received the mask image data MMD, original image data OMD and background image data BPN first of all determines whether or not there is background image designation based on the background image information BPN. In the present case, since there is background image designation, as described above, the determination result is affirmative. Continuing on, the composite image formation section 33′ reads out background image data BDD from the background image file 43 using the header address of the background image file 43 in the background image information.
  • In this manner, the composite image formation section 33′ that has acquired the background image data BDD, in addition to the mask image data MMD and the original image data OMD creates composite image data SMD as follows.
  • First of all, a personal image SDM is extracted from the original image OMM represented by the image F24A in FIG. 24 as a specified display image represented by the image F24B. This extraction is carried out after specifying the outline of the personal image SDM from general characteristics of a personal image, by extracting an image inside the outline.
  • Continuing on, the composite image formation region 33′ superimposes the specified display image SDM on the background image BMM represented by image F24C, at the same position and same size as for the case of the original image OMR. As a result, a background additional image BAM represented as image F24D is formed.
  • Next, the composite image formation section 33′ superimposes the mask image MMMA represented as the image F24E on the background additional image BAM. As a result, a composite image SMMA′ represented as image F24F is formed.
  • The data SMD′ of the composite image SMMA′ formed in this manner is notified from the composite image formation section 33 to the basic control section 29. In accordance with designation carried out by a predetermined user operating the operation section 12, the basic control section 29 that has received this composite image information data displays the composite image SMMA′ on the display section 13, and transmits to a fellow caller via the transceiver section 21.
  • Note that, in a case where background image designation is not described in the display designation file 41′, or when the storage header address of the background image data BDD is “0”, the mask image formation section 32′ forms background image information BPN comprising only the fact that there is no background image designation and notifies this data to the composite image formation section 33′. The composite image formation section 33′ that receives this type of background image information BPN determines that there is no background image designation and superimposes the mask image MMMA on the original image OMM to form a composite image. As a result, a composite image SMMA that is the same as for the case of the composite image formation example of the first embodiment is formed.
  • Also, with the above description when mask image designation is not described in the display designation file 41′ but only background image designation is described, a background additional image BAM is formed in the composite image formation section 33′ as a completed composite image.
  • Also, with the above description, description has been given of an example with background image designation in addition to mask image designation that is the same as for the case of the first composite image formation example for the first embodiment, but naturally it is also possible to have background image designation in addition to mask image designation that is the same as the second and third composite image designation examples of the first embodiment.
  • As has been described above, with this embodiment the composite image formation section 33′ cuts a personal image SDM containing a facial image, being a noted image, from the original image, as a specified display image, superimposes it on the designated background image BMM and forms a background additional image BAM as a noted image comprising image. Then, the composite image formation section 33′ superimposes the mask image MMMA on the background additional image BAM. Accordingly, rendering of the specified display image, such as a personal image SDM and so forth, according to the user's interests, can be carried out by selecting a mask image and a background image.
  • Note that, with the above described second embodiment, the mask image information file 40′ comprises a background image file 43. On the other hand, it is also possible for the background image file 43 to be located in a region outside the mask image information file.
  • Also, with the above described first and second embodiments, the original image is a moving picture, but the present invention can also be applied to cases where the original image is a still image.
  • Also, with the above described first and second embodiments, image formation resulting from light passing through the imaging optical system 18 that images in a so-called self portrait state is made the original image, but it is also possible to make image formation resulting from light passing through the imaging optical system 19 that images in a so-called other party shooting state to be made the original image. It is also possible for an image stored in the storage section 23 to be made the original image.
  • Also, with the above-described first and second embodiments, no consideration has been given to structure along the depth of the noted objects in the noted image region of the original image, and image elements using in the mask image are also created with two-dimensional model. However, it is also possible to consider structure along the depth of the noted objects in the noted image region, and to create element image used in the mask imaged with three-dimensional models. In this case, even if noted objects are moving, having a rotational component about an axis parallel to the display screen, it is possible to form a mask image that affords no discomfort by tracking this rotation.
  • Also, the number of element imaged displayed at the same time is arbitrary. The number of element images carrying out time sequence display is also arbitrary.
  • Also with the above described first and second embodiments, the mask image information file is comprised of the display designation file and the element information file. It is also possible, however, for the mask image information file to further comprise an element image preview file, and when element image preview is designated, to perform element image preview on the display section 13 by referencing the element image preview file.
  • Also, specified phenomenon occurrence is determined for determining whether or not a predetermined phenomenon has occurred on the noted image region based on noted image information, and it is possible commence formation of the mask image using occurrence of the specified phenomenon as a trigger. For example, when the noted image is a facial image of a specified person, when the eyelids are closed, it is possible to form a composite image with tears flowing from positions of the eyes.
  • Also, with the above-described first and second embodiments, the present invention has been applied to a portable telephone unit, but it is also possible to apply the present invention to various other mobile communication terminals. It is also possible to apply the present invention to general electronic appliances such as a personal computer.
  • As has been described above, it is possible to apply the image processing method and image processing method of the present invention to processing of an acquired original image. Also the mobile communication terminal of the present invention can be applied to a mobile communication terminal having van image mail function and a television telephone function.

Claims (19)

1. An image processing method for carrying out processing of an image, comprising the steps of:
specifying an noted image region, being a region for an image to be focused on, in an original image, and extracting noted image information containing information relating to position and size of the noted image within the original image;
forming a mask image to be displayed superimposed on the original image, based on a mask image information file containing the noted image information, at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information of specified element images respectively corresponding to the relationship information, respective execution time information for graphic display determined by combining the relationship information and element image information corresponding to the relationship information, and at least one element image information selected using the element image selection information; and
forming a composite image having the mask image superimposed on a noted image containing image that contains the noted image inside the noted image region at the same location and same size as for the original image.
2. The image processing method of claim 1, wherein the noted image comprising image is the original image.
3. The image processing method of claim 1, wherein the mask image information file further comprise background image selection information showing that information of a specified background image has been selected, and
in the step of forming the composite image, a specified display image containing the noted image is cut from the original image, and the composite image is formed after a background addition image has been made the noted image containing image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
4. The image processing method of claim 1, wherein the information relating to the position of the noted image region contains position information for a representative point of the noted image region of the original image.
5. The image processing method of claim 4, wherein the information relating to the position of the noted image region contains inclination information for the noted image region of the original image.
6. The image processing method of claim 4, wherein, in the relationship information, the representative point of the noted image region is made an origin, and coordinate position information determining direction of a coordinate axis based on inclination of the noted image region is included as display position information for the specified image.
7. The image processing method of claim 1, wherein the relationship information comprises proportion information for size of the noted image region and size for display time of a specified graphic.
8. (canceled)
9. The image processing method of claim 1, wherein the mask image information further comprises information for display of a plurality of types of graphic display in time sequence, fixed by combining the relationship information and element image selection information corresponding to the relationship information, and for time sequence display designation for designating repeat of time sequence display.
10. The image processing method of claim 1, wherein the mask image information further comprises simultaneous display designation information for designating display of a plurality of types of graphic display at the same time, fixed by combining the relationship information and element image selection information corresponding to the relationship information.
11. The image processing method of claim 1, wherein the original image is a moving image.
12. The image processing method of claim 11, further comprising a step of determining whether or not a predetermined phenomenon has occurred in the noted image region based on the noted image information, and wherein
the mask image formation commences after positive determination in the specified phenomenon occurrence determination step.
13. An image processing device for carrying out processing of an image, comprising:
noted image information extracting means for specifying an noted image region, being a region for an image to be focused on, in an original image, and extracting noted image information relating to position and size of the noted image region within the original image;
storage means for storing a mask image information file registering mask image information comprising at least one relationship information representing a relationship to position and size of the noted image region, element image selection information representing selection of information for a specified element image respectively corresponding to the relationship information, respective execution time information for graphic display determined by combining the relationship information and element image information corresponding to the relationship information, and at least one element image information selected using the element image selection information;
mask image formation means for forming a mask image displayed superimposed on the original image, based on the noted image information and the mask image information; and
composite image formation means for forming a composite image having the mask image superimposed on an noted image containing image that includes the noted image inside the noted image region at the same location and same size as for the original image.
14. The image processing device of claim 13, wherein the noted image containing image is the original image.
15. The image processing device of claim 13, wherein the mask image information file further comprises background image selection information showing that information of a specified background image has been selected, and
the composite image forming means cuts a specified display image containing the noted image from the original image, and the composite image is formed after a background addition image has been made the noted image containing image having the specified display image superimposed on the specified background image at the same position and size as for the original image.
16. The image processing device of claim 13, further comprising display means for displaying the composite image.
17. The image processing device of claim 13, further comprising transmission means for transmitting the composite image data.
18. The image processing device of claim 13, further comprising imaging means for acquiring the original image.
19. A mobile communication terminal provided with the image processing device of claim 13.
US11/498,983 2004-02-05 2006-08-04 Image processing method, image processing device and mobile communication terminal Active 2026-01-21 US7864198B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2004-029943 2004-02-05
JP2004029943 2004-02-05
PCT/JP2005/001753 WO2005076210A1 (en) 2004-02-05 2005-02-07 Image processing method, image processing apparatus, and mobile communication terminal apparatus

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2005/001753 Continuation WO2005076210A1 (en) 2004-02-05 2005-02-07 Image processing method, image processing apparatus, and mobile communication terminal apparatus

Publications (2)

Publication Number Publication Date
US20070183679A1 true US20070183679A1 (en) 2007-08-09
US7864198B2 US7864198B2 (en) 2011-01-04

Family

ID=34835975

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/498,983 Active 2026-01-21 US7864198B2 (en) 2004-02-05 2006-08-04 Image processing method, image processing device and mobile communication terminal

Country Status (4)

Country Link
US (1) US7864198B2 (en)
EP (1) EP1713030A4 (en)
JP (1) JPWO2005076210A1 (en)
WO (1) WO2005076210A1 (en)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090169074A1 (en) * 2008-01-02 2009-07-02 General Electric Company System and method for computer assisted analysis of medical image
US20090240666A1 (en) * 2008-03-19 2009-09-24 Sony Ericsson Mobile Communications Japan, Inc. Mobile terminal device and computer program
US20100235336A1 (en) * 2009-03-12 2010-09-16 Samsung Electronics Co., Ltd. Method and apparatus for managing image files
US20110038533A1 (en) * 2009-08-17 2011-02-17 Fuji Xerox Co., Ltd. Image processing apparatus and computer readable medium
CN103403765A (en) * 2012-03-05 2013-11-20 松下电器产业株式会社 Content processing apparatus, integrated circuit thereof, method, and program
CN103455978A (en) * 2012-05-31 2013-12-18 索尼电脑娱乐欧洲有限公司 Apparatus and method for augmenting a video image
US20130342575A1 (en) * 2012-05-23 2013-12-26 1-800 Contacts, Inc. Systems and methods to display rendered images
US20140197940A1 (en) * 2011-11-01 2014-07-17 Aisin Seiki Kabushiki Kaisha Obstacle alert device
US9208608B2 (en) 2012-05-23 2015-12-08 Glasses.Com, Inc. Systems and methods for feature tracking
US9236024B2 (en) 2011-12-06 2016-01-12 Glasses.Com Inc. Systems and methods for obtaining a pupillary distance measurement using a mobile computing device
US9286715B2 (en) 2012-05-23 2016-03-15 Glasses.Com Inc. Systems and methods for adjusting a virtual try-on
CN108537741A (en) * 2017-03-03 2018-09-14 佳能株式会社 Image processing apparatus and the control method for controlling image processing apparatus
RU2673467C1 (en) * 2013-05-31 2018-11-27 Кэнон Кабусики Кайся Image capturing device, image processing device, method of controlling image capturing device, method of controlling image processing device and a program for them

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4417292B2 (en) * 2005-05-25 2010-02-17 ソフトバンクモバイル株式会社 Object output method and information processing apparatus
JP2008053855A (en) * 2006-08-22 2008-03-06 Olympus Imaging Corp Imaging apparatus
EP2225878A1 (en) * 2007-11-22 2010-09-08 Koninklijke Philips Electronics N.V. Methods and devices for receiving and transmitting an indication of presence
EP2113881A1 (en) * 2008-04-29 2009-11-04 Holiton Limited Image producing method and device
JP2009278325A (en) * 2008-05-14 2009-11-26 Seiko Epson Corp Image processing apparatus and method, and program
US20110018961A1 (en) * 2009-07-24 2011-01-27 Huboro Co., Ltd. Video call device and method
EP2355472B1 (en) * 2010-01-22 2020-03-04 Samsung Electronics Co., Ltd. Apparatus and method for transmitting and receiving handwriting animation message
JP5525923B2 (en) * 2010-06-09 2014-06-18 任天堂株式会社 Image processing program, image processing apparatus, image processing system, and image processing method
US8854298B2 (en) 2010-10-12 2014-10-07 Sony Computer Entertainment Inc. System for enabling a handheld device to capture video of an interactive application
JP5791256B2 (en) * 2010-10-21 2015-10-07 キヤノン株式会社 Display control apparatus and display control method
JP5805503B2 (en) * 2011-11-25 2015-11-04 京セラ株式会社 Portable terminal, display direction control program, and display direction control method
JP6099874B2 (en) * 2012-03-22 2017-03-22 キヤノン株式会社 Control device, processing device, control method, processing method, and program
KR101948692B1 (en) 2012-10-09 2019-04-25 삼성전자주식회사 Phtographing apparatus and method for blending images
WO2015144209A1 (en) 2014-03-25 2015-10-01 Metaio Gmbh Method and system for representing a virtual object in a view of a real environment

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US2002A (en) * 1841-03-12 Tor and planter for plowing
US4845641A (en) * 1986-09-19 1989-07-04 Hoya Corporation Method of forming a synthetic image in simulation system for attachment of spectacles
US5469536A (en) * 1992-02-25 1995-11-21 Imageware Software, Inc. Image editing system including masking capability
US5831591A (en) * 1995-09-01 1998-11-03 Samsung Electronics Co., Ltd. Information picture display device using a double-picture type screen
US6141433A (en) * 1997-06-19 2000-10-31 Ncr Corporation System and method for segmenting image regions from a scene likely to represent particular objects in the scene
US6222637B1 (en) * 1996-01-31 2001-04-24 Fuji Photo Film Co., Ltd. Apparatus and method for synthesizing a subject image and template image using a mask to define the synthesis position and size
US20020015514A1 (en) * 2000-04-13 2002-02-07 Naoto Kinjo Image processing method
US6400374B2 (en) * 1996-09-18 2002-06-04 Eyematic Interfaces, Inc. Video superposition system and method
US6542936B1 (en) * 1997-07-03 2003-04-01 Ipac Acquisition Subsidiary I, Llc System for creating messages including image information
US6556704B1 (en) * 1999-08-25 2003-04-29 Eastman Kodak Company Method for forming a depth image from digital image data
US6961446B2 (en) * 2000-09-12 2005-11-01 Matsushita Electric Industrial Co., Ltd. Method and device for media editing
US6993553B2 (en) * 2000-12-19 2006-01-31 Sony Corporation Data providing system, data providing apparatus and method, data acquisition system and method, and program storage medium
US7200247B2 (en) * 1998-05-19 2007-04-03 Sony Computer Entertainment Inc. Image processing device and method, and distribution medium
US7227976B1 (en) * 2002-07-08 2007-06-05 Videomining Corporation Method and system for real-time facial image enhancement
US7230653B1 (en) * 1999-11-08 2007-06-12 Vistas Unlimited Method and apparatus for real time insertion of images into video
US7236622B2 (en) * 1999-08-25 2007-06-26 Eastman Kodak Company Method for forming a depth image
US7317826B2 (en) * 1999-04-26 2008-01-08 Adobe Systems Incorporated Identifying intrinsic pixel colors in a region of uncertain pixels
US7330100B2 (en) * 2003-10-31 2008-02-12 Benq Corporation Mobile device and related method for displaying text message with background images
US7463788B2 (en) * 2003-03-07 2008-12-09 Fujifilm Corporation Method, device and program for cutting out moving image
US7522206B2 (en) * 2002-03-14 2009-04-21 Kyocera Corporation Photographed image display device and photographed image display method

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU4307499A (en) 1998-05-03 1999-11-23 John Karl Myers Videophone with enhanced user defined imaging system
JP3810943B2 (en) * 1999-05-06 2006-08-16 株式会社東芝 Image processing apparatus, image processing method, and recording medium recording image processing program
JP2001292305A (en) * 2000-02-02 2001-10-19 Casio Comput Co Ltd Image data synthesizer, image data synthesis system, image data synthesis method and recording medium
JP4291963B2 (en) * 2000-04-13 2009-07-08 富士フイルム株式会社 Image processing method
JP2003309829A (en) * 2002-04-15 2003-10-31 Matsushita Electric Ind Co Ltd Mobile moving picture phone

Patent Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US2002A (en) * 1841-03-12 Tor and planter for plowing
US4845641A (en) * 1986-09-19 1989-07-04 Hoya Corporation Method of forming a synthetic image in simulation system for attachment of spectacles
US5469536A (en) * 1992-02-25 1995-11-21 Imageware Software, Inc. Image editing system including masking capability
US5831591A (en) * 1995-09-01 1998-11-03 Samsung Electronics Co., Ltd. Information picture display device using a double-picture type screen
US6222637B1 (en) * 1996-01-31 2001-04-24 Fuji Photo Film Co., Ltd. Apparatus and method for synthesizing a subject image and template image using a mask to define the synthesis position and size
US6525836B2 (en) * 1996-01-31 2003-02-25 Fuji Photo Film Co., Ltd. Apparatus for and method of synthesizing image
US6400374B2 (en) * 1996-09-18 2002-06-04 Eyematic Interfaces, Inc. Video superposition system and method
US6141433A (en) * 1997-06-19 2000-10-31 Ncr Corporation System and method for segmenting image regions from a scene likely to represent particular objects in the scene
US6542936B1 (en) * 1997-07-03 2003-04-01 Ipac Acquisition Subsidiary I, Llc System for creating messages including image information
US7200247B2 (en) * 1998-05-19 2007-04-03 Sony Computer Entertainment Inc. Image processing device and method, and distribution medium
US7317826B2 (en) * 1999-04-26 2008-01-08 Adobe Systems Incorporated Identifying intrinsic pixel colors in a region of uncertain pixels
US6556704B1 (en) * 1999-08-25 2003-04-29 Eastman Kodak Company Method for forming a depth image from digital image data
US7236622B2 (en) * 1999-08-25 2007-06-26 Eastman Kodak Company Method for forming a depth image
US7230653B1 (en) * 1999-11-08 2007-06-12 Vistas Unlimited Method and apparatus for real time insertion of images into video
US20050008246A1 (en) * 2000-04-13 2005-01-13 Fuji Photo Film Co., Ltd. Image Processing method
US20020015514A1 (en) * 2000-04-13 2002-02-07 Naoto Kinjo Image processing method
US6961446B2 (en) * 2000-09-12 2005-11-01 Matsushita Electric Industrial Co., Ltd. Method and device for media editing
US6993553B2 (en) * 2000-12-19 2006-01-31 Sony Corporation Data providing system, data providing apparatus and method, data acquisition system and method, and program storage medium
US7522206B2 (en) * 2002-03-14 2009-04-21 Kyocera Corporation Photographed image display device and photographed image display method
US7227976B1 (en) * 2002-07-08 2007-06-05 Videomining Corporation Method and system for real-time facial image enhancement
US7463788B2 (en) * 2003-03-07 2008-12-09 Fujifilm Corporation Method, device and program for cutting out moving image
US7330100B2 (en) * 2003-10-31 2008-02-12 Benq Corporation Mobile device and related method for displaying text message with background images

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090169074A1 (en) * 2008-01-02 2009-07-02 General Electric Company System and method for computer assisted analysis of medical image
US20090240666A1 (en) * 2008-03-19 2009-09-24 Sony Ericsson Mobile Communications Japan, Inc. Mobile terminal device and computer program
US8386458B2 (en) * 2008-03-19 2013-02-26 Sony Mobile Communications Japan, Inc. Mobile terminal device and computer program
US20100235336A1 (en) * 2009-03-12 2010-09-16 Samsung Electronics Co., Ltd. Method and apparatus for managing image files
US9239847B2 (en) * 2009-03-12 2016-01-19 Samsung Electronics Co., Ltd. Method and apparatus for managing image files
US20110038533A1 (en) * 2009-08-17 2011-02-17 Fuji Xerox Co., Ltd. Image processing apparatus and computer readable medium
US9002103B2 (en) * 2009-08-17 2015-04-07 Fuji Xerox Co., Ltd. Image processing apparatus and computer readable medium
US9773172B2 (en) * 2011-11-01 2017-09-26 Aisin Seiki Kabushiki Kaisha Obstacle alert device
US20140197940A1 (en) * 2011-11-01 2014-07-17 Aisin Seiki Kabushiki Kaisha Obstacle alert device
US9236024B2 (en) 2011-12-06 2016-01-12 Glasses.Com Inc. Systems and methods for obtaining a pupillary distance measurement using a mobile computing device
CN103403765A (en) * 2012-03-05 2013-11-20 松下电器产业株式会社 Content processing apparatus, integrated circuit thereof, method, and program
US9117275B2 (en) 2012-03-05 2015-08-25 Panasonic Intellectual Property Corporation Of America Content processing device, integrated circuit, method, and program
US9483853B2 (en) * 2012-05-23 2016-11-01 Glasses.Com Inc. Systems and methods to display rendered images
US20170046863A1 (en) * 2012-05-23 2017-02-16 Glasses.Com Inc. Systems and methods to display rendered images
US9208608B2 (en) 2012-05-23 2015-12-08 Glasses.Com, Inc. Systems and methods for feature tracking
US9286715B2 (en) 2012-05-23 2016-03-15 Glasses.Com Inc. Systems and methods for adjusting a virtual try-on
US9311746B2 (en) 2012-05-23 2016-04-12 Glasses.Com Inc. Systems and methods for generating a 3-D model of a virtual try-on product
US9378584B2 (en) 2012-05-23 2016-06-28 Glasses.Com Inc. Systems and methods for rendering virtual try-on products
US20130342575A1 (en) * 2012-05-23 2013-12-26 1-800 Contacts, Inc. Systems and methods to display rendered images
US9235929B2 (en) 2012-05-23 2016-01-12 Glasses.Com Inc. Systems and methods for efficiently processing virtual 3-D data
US10147233B2 (en) 2012-05-23 2018-12-04 Glasses.Com Inc. Systems and methods for generating a 3-D model of a user for a virtual try-on product
US9996959B2 (en) * 2012-05-23 2018-06-12 Glasses.Com Inc. Systems and methods to display rendered images
CN103455978A (en) * 2012-05-31 2013-12-18 索尼电脑娱乐欧洲有限公司 Apparatus and method for augmenting a video image
RU2673467C1 (en) * 2013-05-31 2018-11-27 Кэнон Кабусики Кайся Image capturing device, image processing device, method of controlling image capturing device, method of controlling image processing device and a program for them
US10356305B2 (en) 2013-05-31 2019-07-16 Canon Kabushiki Kaisha Image-capturing apparatus, image processing apparatus, method for controlling image-capturing apparatus, method for controlling image processing apparatus, and program for the same
CN108537741A (en) * 2017-03-03 2018-09-14 佳能株式会社 Image processing apparatus and the control method for controlling image processing apparatus

Also Published As

Publication number Publication date
JPWO2005076210A1 (en) 2007-10-18
US7864198B2 (en) 2011-01-04
WO2005076210A1 (en) 2005-08-18
EP1713030A1 (en) 2006-10-18
EP1713030A4 (en) 2007-05-02

Similar Documents

Publication Publication Date Title
US7864198B2 (en) Image processing method, image processing device and mobile communication terminal
US6961446B2 (en) Method and device for media editing
JP3946676B2 (en) Captured image processing apparatus and method
US7496215B2 (en) Information device and computer product
JP2004178163A (en) Image processing method and device
CN108600632B (en) Photographing prompting method, intelligent glasses and computer readable storage medium
KR20050078136A (en) Method for providing local information by augmented reality and local information service system therefor
CN116648729A (en) Head portrait display device, head portrait generation device, and program
JP2006163871A (en) Image processor and processing method, and program
US8290299B2 (en) Object outputting method and information processing apparatus
KR20120086810A (en) Terminal and method for processing image thereof
JP2000322588A (en) Device and method for image processing
TW201908949A (en) Emoticon generating method and device, electronic apparatus, and storage medium
JP2004072685A (en) Device, method and program for compositing image and recording medium recording the program
JP2004171543A (en) Method and device for image processing
JP2011192008A (en) Image processing system and image processing method
CN113286073A (en) Imaging method, imaging device, and storage medium
KR101738896B1 (en) Fitting virtual system using pattern copy and method therefor
JP4202330B2 (en) Mask image information creation method, mask image information creation apparatus, and mask image information creation program
JP7128347B2 (en) Image processing device, image processing method and program, imaging device
JP2007122561A (en) Portable terminal device and portrait data providing system
JP2004118563A (en) Method, device and program for processing character image
JP2007166383A (en) Digital camera, image composing method, and program
JP2005124161A (en) Video telephone set and program
JP2013242835A (en) Image communication system, image generation device and program

Legal Events

Date Code Title Description
AS Assignment

Owner name: VODAFONE K.K., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MOROTO, MINEKO;SHIBATA, AKIHO;KATAYAMA, TAKASHI;AND OTHERS;SIGNING DATES FROM 20060108 TO 20060113;REEL/FRAME:018373/0315

Owner name: VODAFONE K.K., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MOROTO, MINEKO;SHIBATA, AKIHO;KATAYAMA, TAKASHI;AND OTHERS;REEL/FRAME:018373/0315;SIGNING DATES FROM 20060108 TO 20060113

AS Assignment

Owner name: SOFTBANK MOBILE CORP., JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:VODAFONE K.K.;REEL/FRAME:020401/0001

Effective date: 20061001

Owner name: SOFTBANK MOBILE CORP.,JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:VODAFONE K.K.;REEL/FRAME:020401/0001

Effective date: 20061001

AS Assignment

Owner name: VODAFONE GROUP PLC, UNITED KINGDOM

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOFTBANK MOBILE CORP.;REEL/FRAME:020409/0037

Effective date: 20070927

Owner name: VODAFONE GROUP PLC,UNITED KINGDOM

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOFTBANK MOBILE CORP.;REEL/FRAME:020409/0037

Effective date: 20070927

STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552)

Year of fee payment: 8

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12