Three-dimensional (3D) augmentation digital processing method for two-dimensional (2D) images

TECHNICAL FIELD OF THE INVENTION:

[0001] This invention belongs to the field of digital image processing. To be precise, this invention augments 3D effects of 2D images using a digital image processing method.

BACKGROUND OF THE INVENTION:

[0002] The 3D information of an image includes binocular disparity, the focusing of eyes, convergence, motion parallax, aerial perspective (which causes changes in color and luminance), occlusion of objects, geometric perspective, and more. Among all information mentioned above, binocular disparity is the strongest 3D information. Because of the pupillary distance between a person's eyes, when looking at a scene, the views obtained by each of the eyes are different from each other, and the brain uses this information (binocular disparity) to form a sense of 3D. Most 3D movies, 3D televisions and other 3D display devices achieve 3D effects by making use of binocular disparity. However, when viewing a plane 2D image, eyes focus on pixels on a plane, and the brain knows this is a 2D image on a plane because of binocular disparity. As the strongest 3D information, binocular disparity weakens the 3D effects of other 3D information in 2D images. The aim of this invention is to augment other 3D information to produce and enhance 3D effects when viewing 2D images with naked eyes.

SUMMARY OF THE INVENTION:

[0003] Binocular disparity weakens other 3D information in 2D images, and there are methods to generate and augment 3D effects when viewing 2D images on a flat display by augmenting other 3D information of 2D images. It is possible to augment the 3D information of 2D images displayed on traditional devices (e g. traditional televisions) by adjusting luminance, color, saturation, contrast, and so on, but adjusting these properties does not have a significant effect. Among all 3D information of 2D images, geometric perspective is a very strong and important information. When drawing a picture, making use of geometric perspective is a very effective way to augment 3D effects, and it is important to locate a vanishing point (as shown by Figure 1) when making use of geometric perspective. Many traditional image processing methods to augment or generate 3D information are complicated. These traditional methods usually take a long time to be processed, so it is more difficult to apply them in real-time. This invention provides a method which sets a vanishing point or vertical vanishing line on the image and augments geometric perspective based on this vanishing point or vertical vanishing line. By applying this method, the sense of 3D of 2D images is augmented.

[0004] In the geometric perspective of an image, there can be one or more vanishing points. Setting only one vanishing point can make the image processing easier. The vanishing point can be replaced by a vertical vanishing line, and this can also make the image processing easier. For a 2D image like the one shown in Figure 2, a vertical vanishing line can be set at the horizontal center of the image, and in this case, the vertical vanishing line is also the center line of the image.

[0005] When viewed by people, farther objects are usually smaller in the scene compared to closer objects. The objects near the vertical vanishing line of an image are usually the farthest from the viewer compared to other objects on the image, so the image processing method of this invention makes the horizontal widths of the objects near the vertical vanishing line smaller compared to their original horizontal widths on the image. The objects away from the vertical vanishing line of the image are usually the closest to the viewer compared to other objects on the image, so this invention makes the horizontal widths of the objects away from the vertical vanishing line larger compared to their original horizontal widths on the image. When the vertical vanishing line is also the center line of the image, on the image, the left and right edges of the image are the farthest away from the vertical vanishing line. Therefore, when the vertical vanishing line is also the center line of the image, the image processing method of this invention makes the horizontal widths of objects near the center line of the image smaller compared to their original horizontal widths on the image, and this method makes the horizontal widths of objects near the left and right edges of the image larger compared to their original horizontal widths on the image. This method has two features: 1. Augmenting 3D information of images by setting a vertical vanishing line of geometric perspective. 2. Making the objects near the vertical vanishing line of images visually farther from the viewer, and making the objects away from the vertical vanishing line of images visually closer to the viewer, so the geometric perspective of the processed images is augmented. This invention is applicable and effective to a wide range of scenes and video sources, such as movies, sports shows, TV shows, TV series, fashion shows, and more.

[0006] 4 examples of this image processing method are shown in Figures 3-6 (top view from above the display).

[0007] In [0004], [0005], and [0006], an image processing method that sets a vertical vanishing line is introduced, and this method can be extended to set a vanishing point instead of a vertical vanishing line. For example, in Figure 15, the vanishing point is set to be at the center point c of the image, and the image is divided into 5 sections (dl2, d3, d4, d5, d6) to be processed. When setting a vanishing point, this method makes the sections near the vanishing point smaller both horizontally and vertically. For example, in Figure 15, section dl2 contains the vanishing point, so section dl2 is processed to be smaller. When setting a vanishing point, this method makes the sections away from the vanishing point larger both horizontally and vertically. For example, in Figure 15, section d6 is the outmost section and the section farthest from the vanishing point, so section d6 is processed to be larger.

BRIEF DESCRIPTION OF DRAWINGS:

[0008] Figure 1 illustrates a vanishing point.

[0009] Figure 2 provides an example for a rectangular image and its center line. When the image is rectangular and fills up the whole display, the center line of the image is also the center line of the display.

[0010] Figures 3-6 show 4 examples of this image processing method. Figure 7 shows the original image before being processed. To make the examples easier to understand, assume the images in Figures 3-7 are filling up the whole display. All of Figures 3-7 are showing the top view of the display, where the display is facing to the bottom of the Figures, and the viewer is viewing the display from the bottom of the Figures. In all of Figures 3-7, C represents the center line of the image, L represents the left edge of the image, and R represents the right edge of the image. Because all of Figures 3-7 are showing the top view of the display, Figures 3-7 are also showing the top view of the center line C, left edge L, and right edge R, so the three lines C, L, and R are visible as points in Figures 3-7.

[0011] Figure 3 shows the 1st example: In Figure 3, the vertical vanishing line of the image is set to be at the center line C. After being processed, the image is reduced in horizontal size at the regions near the vertical vanishing line, and the image is increased in horizontal size at the regions away from the vertical vanishing line. Because the original image is on the display, after being processed, the regions near the vertical vanishing line (which is at the center line C in Figure 3) are visually behind the display and farther from the viewer's eyes. Also, because the vertical vanishing line of the image is set to be at the center line C, the left edge L and right edge R are the farthest away from the vertical vanishing line in Figure 3. Therefore, in Figure 3, this method makes the horizontal size of the image increases at the regions near the left edge L and the right edge R (which are away from the vertical vanishing line in Figure 3), so after being processed, these regions near the left and right edges are visually in front of the display and closer to the viewer's eyes. In Figure 3, when the original image is processed at the horizontal 1/4 and 3/4 points from the left edge, the horizontal magnification ratio is set to be 1. Therefore, at the horizontal 1/4 and 3/4 points from the left edge, the horizontal size of the processed image does not change compared to the original horizontal size, and these two points of the image are still visually on the display. In Figure 3, the processed image has a symmetrical shape, which is visually behind the display at the horizontal center and visually in front of the display at the left and right edges. By contrast, in Figure 7, the center line C, left edge L, and right edge R are all visually on the display. In Figure 3, the horizontal magnification ratio decreases linearly from L to C and from R to C, and the horizontal magnification ratio is equal to 1 at the horizontal 1/4 and 3/4 points from the left edge. In Figure 3, the value of the horizontal magnification ratio is the largest at L and R, smallest at C.

[0012] Figure 4 shows the 2nd example. In Figure 4, the vertical vanishing line of the image is set to be at the center line C, and L and R represent the left and right edges of the image. In Figure 4, the horizontal magnification ratios for all regions of the image are less than or equal to 1.

[0013] Figure 5 shows the 3rd example. In Figure 5, the vertical vanishing line of the image is set to be at the center line C, and L and R represent the left and right edges of the image. In Figure 5, the horizontal magnification ratios for all regions of the image are greater than or equal to 1.

[0014] Figure 6 shows the 4th example. In Figure 6, the processed image is a tilted plane. In Figure 6, C represents the center line of the image, and L and R represent the left and right edges of the image. In Figure 6, the vertical vanishing line is not within the image. When the processed image is a tilted plane, in the processed image, one of the left or right edges has the smallest horizontal magnification ratio among the whole image, and the other left or right edge has the biggest horizontal magnification ratio among the whole image.

[0015] In each of the 4 examples shown by Figures 3-6, it is also possible to set the vertical vanishing line to be not at the center line of the image.

[0016] Figure 8 shows an example of processing 8 sections of an image to augment 3D information. In Figure 8, the original image is divided into 8 sections of the same horizontal width, but this method can also set some or all sections to have different horizontal widths in the original image, and the number of sections can also be not 8.

[0017] Figure 9 shows an example of processing 8 sections of an image to augment 3D information with one more step compared to Figure 8, and this additional step is rounding up or rounding down the number of horizontal pixels to an integer when calculating the processed horizontal width of each section.

[0018] Figure 10 shows processing the image when taking photos or recording videos. [0019] Figure 11 shows processing the image during TV program post production.

[0020] Figure 12 shows processing the image when a TV or set-top box receives the TV program. [0021] Figure 13 shows processing the image when playing movies.

[0022] Figure 14 shows processing the image for TV programs that are finished products.

[0023] Figure 15 shows how to process the image to augment 3D information by setting a vanishing point, and the image in Figure 15 fills up the whole display. In Figure 15, the vanishing point of the image is set to be at the center point c of the image, and the image is divided into 5 sections (dl2, d3, d4, d5, d6) to be processed.

DESCRIPTION OF EMBODIMENTS:

[0024] In the example shown in Figure 8, the original image is horizontally split into 8 sections of the same horizontal width, and each section of the image is processed respectively. The vertical vanishing line of the image is set to be at the center line. Divided by the vertical vanishing line, the 4 sections at left are symmetrical to the 4 sections at right. When being processed, a section can be stretched to be wider, get narrower, or stay the same in terms of horizontal width. This stretching or narrowing process is evenly applied within each section, and each section maintains its information after being processed. In the example shown in Figure 8, the differences in the number of horizontal pixels expanded or narrowed between any two neighboring sections are all equal (except for the two sections next to the center line because these two center sections have the same horizontal width after being processed). In Figure 8, there are Width*Hight (e.g. 1920*1080) pixels, and n is a positive integer value being used to represent the change in the number of horizontal pixels. The value n/3 shown in the 2nd section from the left means the 2nd section of the original image will have its horizontal width increases by n/3 pixels after being processed, and similarly, any positive value in a section means this section will be processed to be wider, and any negative value in a section means this section will be processed to be narrower. Among all 8 sections in Figure 8, the 2 sections next to the vertical vanishing line (which is set to be at the center line in this example) have value -n, and this means the horizontal width of each of these two sections will decrease by n pixels. The number of horizontal pixels expanded for each of the 8 sections (from left to right) are n, n/3, -n/3, -n, -n, -n/3, n/3, and n respectively. Divided by the center line, there is a group of 4 sections at left and a group of 4 sections at right. Among each of these two groups of 4 sections, each two neighboring section has the difference in the number of horizontal pixels expanded or narrowed to be 2n/3 pixels, so in Figure 8, the change of the horizontal magnification ratio is linear from the vertical vanishing line to the left and right edges of the image. In the example shown in Figure 8, the total number of horizontal pixels expanded among all sections is equal to the total number of horizontal pixels narrowed among all sections, so the total number of horizontal pixels of the entire image does not change after being processed. As shown by Figure 9, when this method is applied, rounding up or rounding down can be used to convert the processed horizontal width of each section to an integer. In Figure 9, rounding is applied, and the rounded number of horizontal pixels expanded for each of the sections (from left to right) are n, [n/3], -[n/3], -n, -n, -[n/3], [n/3], and n.

[0025] For example, the resolutions of standard-definition television, high-definition television, 4K ultra-high definition television, and 8K ultra-high definition television are 720*576, 1920*1080, 3840*2160, and 7680*4320 respectively. The numbers of horizontal pixels are 720, 1920, 3840, and 7680 respectively. When processing images of different resolutions, this method can apply the same set of horizontal magnification ratios and therefore has the same effect on the output (processed) images. In the example shown in Figure 8, let k be a preset parameter, and the value n (being used to represent the change in the number of horizontal pixels) in [0024] can be calculated by having the number of horizontal pixels divided by k. For instance, a high-definition television has 1920 horizontal pixels, and when the image is being processed, n can be calculated by n=1920/k.

[0026] In the example shown in Figure 8 , the original image is divided into 8 sections of the same horizontal width, and from the vertical vanishing line (which is set to be at the center line in this example) to the left and right edges of the image, the value of the horizontal magnification ratio increases from smaller than 1 to greater than 1.

[0027] When applying this method, the original image can be divided into any number of sections to be processed, and the number of sections can be either odd or even. The vertical vanishing line can be set either at or not at the center line of the image. Prior to being processed, each pair of sections in the original image can have either identical or non-identical horizontal widths, and the arrangement of sections can be either symmetrical or asymmetrical. From the vertical vanishing line to each of the two left or right edges of the image, the change of the horizontal magnification ratio can be either linear or non-linear. After a section is processed, the change of the horizontal width for this section can be as few as 0. The total number of horizontal pixels expanded in all sections can be either equal or not equal to the total number of horizontal pixels narrowed in all sections, and after the entire image is processed, the total horizontal width of the entire image can either change or not change. The original image can be divided into sections in any shapes, and the arrangement of the sections can be horizontal or non-horizontal.

[0028] Some examples of different locations and settings of the vertical vanishing line (e.g. whether the vertical vanishing line is at the center line of the image or not) and different settings of horizontal magnification ratios are shown in the 4 examples in Figures 3-6.

[0029] The vertical processing of geometric perspective can also be added to this method by replacing the vertical vanishing line with a vanishing point. An example of setting a vanishing point is shown in Figure 15, where the vanishing point is set to be at the center point c of the image. When setting a vanishing point, the original image is also divided into sections to be processed, and the sections can be in any shapes. When setting a vanishing point, this method makes the sections near the vanishing point smaller both horizontally and vertically, and this method makes the sections away from the vanishing point larger both horizontally and vertically. The vanishing point can be set either at or not at the center point of the image.

[0030] After applying the image processing method of this invention, if there is a need to adjust the vertical or horizontal size of the processed image, the processed image can be stretched or reduced evenly in size as a whole, and the previously augmented 3D effects will be retained.

[0031] As shown in Figures 10-14, this image processing method can be used during the camera shooting process, the intermediate image and video production process, display device, finished images and videos, and more.