Background of the Invention Panoramic images contain a field of view that covers the entire perimeter around the imaging device. Stereoscopy is the ability to simulate a third dimension (depth) in images. The combination of panoramic imaging with stereoscopy produces an image, which possesses a full panoramic field of view with the addition of a simulated depth dimension and enables orientation in space and in depth. The enhanced orientation enabled by stereoscopic panoramic images is considered of great value in certain applications, such as Tele-Operating and applications in the field of virtual reality.

As is implied by its name, panoramic stereoscopic imaging involves the acquiring of two images of the same panoramic space from two different viewpoints. Each viewpoint presents a slightly shifted image with respect to the other and they can then be combined to give the right and left views required for stereoscopic viewing. Prior art systems for producing the panoramic images generally are based on creating mosaics of a series of images that are captured by physically rotating a camera lens 360 degrees about a vertical axis, optically rotating the field of view of a stationary lens, using multiple camera lenses aimed in different directions to obtain full 360 degree coverage, or some combination of these. The images from the second viewpoint are obtained either separately, by moving the camera lens used to capture the first series of images to another position and then capturing a second series of images, or simultaneously with the first series by using a second lens mounted in a fixed position relative to the first. Computational methods also exist for computer generating a second set of images from a first set taken with a camera. The mechanical and optical systems required to acquire the images are often massive and bulky and are generally complex and expensive as are the image processing techniques necessary to create the panoramic mosaics and final panoramic stereoscopic image.

One example of a prior art system to obtain panoramic stereoscopic images is disclosed in U. S. patent no. 6,023, 588. In this patent is described a method by which two panoramic images are produced from two viewpoints positioned on the same vertical axis. The two images created are manipulated to enable panoramic stereoscopic display. One significant problem with this method is that it is not compatible with the way the human brain is accustomed to receive, analyze and interpret images in order to effectively grasp their stereoscopic qualities. Human eyes are positioned on the same horizontal axis ; therefore, a stereoscopic image that is based on viewpoints positioned on the same vertical axis, rather than on a horizontal axis, is less suitable and most likely will force a viewer to tilt their head in order to truly grasp the stereoscopic qualities of the image.

Another example of a prior art system for obtaining panoramic stereoscopic images is disclosed in U. S. patent no. 6,301, 447. In this patent is described an apparatus that provides support for a rotatable camera and shifting means for moving the camera between two offset positions. The camera contains a fish-eye lens or other wide-angle lens. In order to create a panoramic image from each of the two positions, the camera is rotated around its axis as necessary in order to capture several images that can be seamed together to form a single panoramic image. The shifting means are then activated to move the imaging device to the second position, where the camera is rotated again around its axis to form a second, shifted, panoramic image. Image processing techniques are described that are. used to manipulate the two panoramic images to enable panoramic stereoscopic display. The described process is very cumbersome and requires a relatively complex mechanical structure and accurate movement. The most serious shortcoming of the method divulged in this patent, however, is the length of time needed to create a single stereoscopic image. The long time period makes this method unsuitable for panoramic stereoscopic video imaging in real-time applications.

Methods of eliminating the necessity of creating a mosaic from multiple images by the use of reflective surfaces to acquire panoramic images at a single shot are known in the art. For example, published International Patent Application WO 02/059676 by the same applicant hereof, the description of which is incorporated herein by reference, describes a method of using reflective surfaces and additional optical elements to acquire a nearly spherical field of view image at a single shot.

It is therefore an object of the present invention to provide a panoramic stereoscopic imaging assembly that overcomes the shortcomings of the prior art.

It is another object of the present invention to provide a panoramic stereoscopic imaging apparatus enabling the production of at least two panoramic images which are suitable for creation of stereoscopic panoramic views. It is yet another object of the present invention to provide a method of combination of the acquired panoramic images to enable a stereoscopic view of the panoramic scene.

It is a further object of the present invention to provide a means for creation of a panoramic stereoscopic image, simultaneously with a stereoscopic image of a narrow, optically zoomed sector.

It is a still further object of the present invention to provide devices and methods capable of producing real-time panoramic stereoscopic video.

Further purposes and advantages of this invention will appear as the description proceeds.

Summary of the Invention The present invention refers to a method and apparatus for panoramic stereoscopic imaging. The apparatus of the invention utilizes, as its basic elements, reflective panoramic lenses as panoramic image sources. Each panoramic lens produces a reflection of the panoramic surroundings and enables creation of a panoramic image of the surroundings in a single shot.

The apparatus design is based on the use of more than one such lens in order to provide more than one image-source for the same panoramic environment, and thus to enable the production of a panoramic image having stereoscopic qualities.

A preferred embodiment of the invention is an apparatus that consists of two reflective panoramic lenses, whose entrance pupils are positioned on the same horizontal axis and at the same height. Each of the two lenses covers a full panoramic field of view. While all of the panoramic field of view is covered by the two lenses, only two sectors of it are covered simultaneously by both of the lenses, therefore stereoscopy can be achieved in these two sectors only. The entire system can be rotated around its axis in order to achieve stereoscopy in additional sectors.

Another embodiment of the invention is an apparatus that consists of three reflective panoramic lenses, whose entrance pupils are positioned on the same horizontal plane at the same height, and preferably in a way that forms a virtual regular triangle. In this embodiment every sector of the panoramic space is covered by at least two different lenses simultaneously, therefore stereoscopy can be achieved for the entire panoramic space.

In both of the above embodiments, each panoramic lens can incorporate an optical zoom lens, which enables zooming in on a specific sector in the panoramic space. An alignment of at least two such zooming lenses towards the same sector will enable both optical zoom and stereoscopy of that sector.

Also included in the present invention are methods of transformation of the acquired panoramic images to enable utilizing them for panoramic stereoscopic display.

In a first aspect the present invention provides an imaging apparatus comprising: a. Two or more lenses, each providing at least a panoramic scene.

Each entrance pupil of each lens is positioned at the same height with respect to a common horizontal plane and each lens has a vertical axis of symmetry. The vertical axes of symmetry are parallel to each other and perpendicular to the common plane. b. An image capture device is associated with each of the lenses.

Each of the image capture devices is directed towards a different one of the lenses and the optical axis of each of the image capture devices coincides with the vertical axis of symmetry of its associated lens.

Each of the lenses reflects a panoramic field of view of the same scene towards the associated image capture device, which is located coaxially with it, and thus two or more panoramic images of the same scene are provided.

The imaging apparatus of the invention further comprises a support structure suitable to maintain the spatial relationship between the lenses, the image capture devices and their common plane. In preferred embodiments of the invention each of the lenses comprise at least an axi- symmetric reflective surface, suitable to reflect a panoramic scene. In other preferred embodiments, each of the lenses may provide a nearly spherical field of view comprised of the panoramic scene and at least one additional scene.

In the embodiment of the invention in which the number of lenses in the imaging apparatus is two, there can be further provided a vertical axis of rotation, perpendicular to the common plane. The axis of rotation is located between the two lenses equidistantly from each of them. In this embodiment of the invention, a rotation mechanism, designed to rotate the imaging apparatus around the axis of rotation can be provided.

In another preferred embodiment of the invention, the number of lenses in the imaging apparatus is three. The entrance pupils of the lenses are positioned on the common plane such that virtual connection of the points of intersection of the vertical axes of symmetry of each lens with the plane creates a virtual regular triangle. In preferred embodiments of the imaging apparatus of the invention, an optical zoom lens is incorporated into one or more of the lenses which reflect a panoramic field of view of the scene. The optical zoom lens can be rotated around the vertical axis of symmetry of the lens in which it is incorporated.

In the preferred embodiments of the invention, the imaging apparatus further comprise image processing means. The image processing means are designed to receive the images that are acquired by the image capture devices and process them for viewing. The image viewed after processing can be either a static stereoscopic image or a real-time stereoscopic video image of a scene.

In another aspect the invention provides a method of creating a right image and a left image from two or more panoramic images that are produced by panoramic reflective lenses. The method comprises the following steps: a) Providing an imaging apparatus comprising: (1) Two or more panoramic axi-symmetric reflective lenses, each providing at least a panoramic scene. The entrance pupil of each lens is positioned at the same height with respect to a common horizontal plane and each lens has a vertical axis of symmetry. The vertical axes of symmetry are parallel to each other and perpendicular to the common plane.

(2) An image capture device associated with each of the lenses.

Each of the image capture devices is directed towards a different one of the lenses and the optical axis of each of the image capture devices coincides with the vertical axis of symmetry of its associated lens.

Each of the lenses reflects a panoramic field of view of the same scene towards the associated image capture device, which is located coaxially with it, and thus two or more panoramic images of. the same scene are provided. b) Imaging a reflection of a panoramic scene received from the two or more panoramic axi-symmetric reflective lenses by the two or more image capture devices located coaxially with them. c) Forming a group consisting of pairs of the panoramic axi-symmetric reflective lenses. The group comprises all possible unique pairs of lenses that can be formed from the lenses. d) Determining the identity of the common sectors of the panoramic scene that appear in each pair of images received from each of the pairs of lenses that comprise the group. e) Determining which of the panoramic lenses provides a right viewpoint and which a left viewpoint in each of the common sectors.

Creating a right image comprising, for all pairs in the group, the images that are from the common sectors and which provide a right viewpoint. g) Creating a left image comprising, for all pairs in the group, the images that are from the common sectors and which provide a left viewpoint.

In the preferred embodiment of the method of the invention wherein the number of panoramic images is two, the right and/or left image may further comprise at least one of the sectors that are covered by only one panoramic lens. This sector/s is positioned in the image according to its position in the panoramic scene.

All the above and other characteristics and advantages of the invention will be further understood through the following illustrative and non-limitative description of preferred embodiments thereof, with reference to the appended drawings.

Brief Description of the Drawings Fig. 1 schematically shows a panoramic monolithic optical lens, which provides a reflection of a panoramic field of view; Fig. 2 schematically shows an alternative design of a panoramic monolithic optical lens, which provides a reflection of a panoramic field of view; - Fig. 3 schematically shows an optical lens enabling simultaneous view of a panoramic scene together with an optically zoomed sector; Fig. 4 schematically shows a panoramic stereoscopic imaging apparatus comprising two panoramic imaging assemblies; Fig. 5 schematically shows the fields of view, which are acquired by the two panoramic imaging assemblies of the apparatus of Fig. 4 ; Fig. 6 schematically shows the fields of view, which are acquired by two panoramic imaging assemblies with optical zoom capability; Fig. 7 schematically shows the image transformation that needs to be performed in order to make the images produced by two panoramic imaging assemblies suitable for further stereoscopic display; Fig. 8 schematically shows a panoramic stereoscopic imaging apparatus comprising three panoramic imaging assemblies; Fig. 9 schematically shows the preferred layout of the three panoramic imaging assemblies of the apparatus of Fig. 8; Figs. 10A to 10C schematically show the field of view, which is acquired by each of the three panoramic imaging assemblies of the apparatus of Fig. 8; Fig. 11 schematically shows the field of view, in which stereoscopy can by achieved, which is covered by two different imaging assemblies of the apparatus of Fig. 8; Fig. 12 schematically shows the transformation of a panoramic image as acquired by the focal plane array to rectangular form; Fig. 13 schematically shows the incorporation of zooming lenses in an apparatus comprising three imaging assemblies; Fig. 14 schematically shows a method of dividing the panoramic field of view, which aids in the determination of the right and the left images of the stereoscopic pair in each of the sectors; Fig. 15 schematically shows the transformation that needs to be performed on the rectangular images produced by an imaging apparatus incorporating two imaging assemblies in order to create a left image and a right image; and Fig. 16 schematically shows the transformation that needs to be performed on the rectangular images produced by an imaging apparatus incorporating three imaging assemblies in order to create a left image and a right image.

Detailed Description of Preferred Embodiments In order to achieve the purposes of the present invention, omni-directional vision systems, which acquire an omni-directional scene in a single shot, are employed. Those systems are based on the use of reflective surfaces that reflect an omni-directional scene towards an image capture device. Such reflective surfaces, and optical lenses which utilize them, have been described in the art, for example in the above cited publication WO 02/059676.

In Figs. 1 to 3 are schematically shown several optical designs of monolithic omni-directional lenses. The incorporation of those examples in this description is in order to demonstrate the principle of omni-directional lenses and representative possibilities of their designs. It is to be noted that the purpose of the present invention is not to provide designs of innovative omni-directional lenses, but rather to describe the incorporation of those lenses for the purposes of the present invention, it being understood that the skilled person will be easily able to design or select the appropriate lenses to be employed in a particular situation.

In this application the terms"panoramic"and"omni-directional"are used interchangeably, whenever reference is made herein to either of them, the term should not be taken literally, and it should be understood that it is meant to indicate a lens means capable of providing a large field of view, <BR> including"fish-eye"and similar types of lenses, e. g. , a panoramic field of view, and therefore the term is not limited to any specific type of lens means.

In this application, the term"panoramic imaging assembly"is used to refer to the system comprising, among other components, the panoramic lens, the image capture device, and support means that maintain the correct spatial relationship between the elements of and provides physical support for the system that is used to obtain the individual panoramic images.

In this application, the term"stereoscopic panoramic imaging apparatus"is used to refer to the system comprising, among other components, two or more panoramic imaging devices and support means that provide physical support for the system and maintain the correct spatial relationship between the panoramic imaging devices that are used to obtain the two or more individual panoramic images that are combined to form the stereoscopic panoramic image.

Fig. 1 is a schematic description of a monolithic optical structure, which provides coverage of a panoramic field of view. The lens (1) has an axi- symmetric aspheric shape, comprising several surfaces having a co- dependent design to compensate for aberrations, allowing optimal acquiring of the panoramic image. The lens (1) comprises a perimeter refractive surface (2) an upper transparent surface (3), coated with reflective material on its exterior side, and a lower refractive surface (4). Each ray that is within the vertical field of view covered by the lens (1), is refracted by the perimeter refractive surface (2), penetrates the lens, traveling towards the upper reflective surface (3) where it is reflected downwards towards the lower refractive surface (4), where it is refracted again and exits the lens (1) towards an image capture device (not shown) located coaxially with the lens (1). Dotted line (5) is a schematic optical path of a ray originating at the upper limit of the field of view which is covered by the lens (1) and a second dotted line (6) represents a schematic optical path for a ray originating at the lower limit of that field of view.

Fig. 2 is a schematic description of yet another monolithic optical structure, which provides coverage of a panoramic field of view. The lens (7) has an axi-symmetric aspheric shape, comprising several surfaces having a co- dependent design to compensate aberrations, allowing optimal acquiring of the panoramic image. The lens (7) comprises a perimeter refractive surface (8) a lower transparent surface (9), covered with reflective material on its exterior surface, an upper transparent surface (10) coated with reflective material on its exterior surface, and a lower refractive surface (11). Each ray which is within the vertical field of view (12) which is covered by the lens (7), is refracted by the perimeter refractive surface (8) and travels through the lens towards the lower reflective surface (9) where it is reflected upwards towards the upper reflective surface (10). At the upper reflective surface (10) it is reflected downwards towards the lower refractive surface (11), where it is refracted again and exits lens (7) towards an image capture device (not shown) located coaxially with lens (7). Dashed line (13) represents schematically the optical path of a ray originating within the field of view (12) covered by the lens (7).

Fig. 3 schematically shows a lens assembly which provides simultaneous coverage of a panoramic field of view together with an optically zoomed sector. The lens assembly comprises an axi-symmetric reflective surface (14), and a second, smaller, reflective surface (15). The axi-symmetric reflective surface (14) is designed to reflect a panoramic field of view towards an image capture device (not shown) located coaxially with it. The second reflective surface (15), which has a different radius of curvature than the axi-symmetric reflective surface (14), is positioned to reflect a limited sector towards the center of the image, thus providing an optically zoomed image of a sector which appears also in the panoramic image, but appears there in smaller proportions. The second reflective surface (15) may be connected directly to the axi-symmetric reflective surface (14), providing a fixed image of a fixed sector or it may be connected to a motor (16), by a connector (17), which enables it to turn and tilt and provide images of different sectors. The described arrangement will enable reflection of the zoomed sector towards a portion of the focal plane array, which is not used for the panoramic image.

The preferred embodiments of the present invention provide a panoramic stereoscopic imaging apparatus, which enables capture of a full panoramic field of view from several different viewpoints simultaneously, thus supplying the source images required for production of panoramic stereoscopic images. The current invention makes use of several omni- directional imaging assemblies, each comprising an omni-directional lens and an image capture device. Each omni-directional imaging assembly collects an omni-directional image of the surroundings from a different viewpoint. Since each of the lenses covers a full 360° field of view, each lens serves as either a right viewpoint or as a left viewpoint in different sectors in the panoramic space. The invention provides a method, to be described hereinbelow, of determining which of the lenses is considered to provide the right viewpoint and which the left viewpoint in each sector and a method of translating the acquired panoramic images to two separate images, a right image and a left image, suitable for panoramic stereoscopic display.

Figure 4 shows schematically a preferred embodiment of the present invention comprising a panoramic stereoscopic imaging apparatus which comprises two panoramic imaging assemblies. Each assembly contains a panoramic lens which reflects an omni-directional image towards an image capture device located coaxially with it. The description of the inner structure of each imaging assembly is not presented herein, but is based on the description of the lenses shown in Figs. 1 to 3 and to other prior art designs. The apparatus (18) comprises a first panoramic imaging assembly (19) located at a first location (20), and a second panoramic imaging assembly (22) located at a second location (23). The locations (20) and (23) are located on a platform (24), which is a common horizontal plane. In this embodiment it is preferred that the two imaging assemblies are structured in the same way and are designed using the same parameters, i. e. are essentially identical. It is stressed that in order to create the most effective images, the first imaging assembly (19) and the second imaging assembly (22) should be at the same height. More specifically, the entrance pupils of both lenses should be at the same height, above the platform.

The positions of the two imaging assemblies should be such that their central optical axes are parallel to each other and orthogonal to the common horizontal plane. The preferred distance between the two imaging assemblies is chosen to be as close as possible to the distance that exists between the members of a pair of human eyes, thus enabling production of a pair of images that closely resemble those normally received by the brain.

Since each of the imaging assemblies is able to capture an entire panoramic field of view, each imaging assembly will actually view the entire perimeter around it, including its neighboring imaging assembly. More explicitly, part of the field of view of each imaging assembly will be blocked by the other imaging assembly. Since'stereoscopy can be produced only if two images of the same sector are created from two different viewpoints and since parts of the panoramic field of view are covered only by one lens, stereoscopy cannot be achieved in these parts.

In Fig. 5 are schematically shown, from an overhead viewpoint, the fields of view, of each of the two imaging assemblies. A common panoramic field of view (27) is within the range of coverage of both the first imaging assembly (28) and the second imaging assembly (29). However, the existence of the imaging assemblies themselves creates blockage of parts of the field of view, i. e. the first imaging assembly (28), blocks part of the field of view of the second imaging assembly (29), preventing it from imaging a first blocked sector (30). In the same manner, the second imaging assembly (29) blocks part of the field of view of the first imaging assembly (28), preventing it from covering a second blocked sector (31). It can, however, be seen that the first blocked sector (30) is well within the field of view of the first imaging assembly (28), and the second blocked sector (31) is well within the field of view of the second imaging assembly (29). Since stereoscopy production requires coverage of a field of view from two different viewpoints, and since that condition is not supplied in respect to the first blocked sector (30) and the second blocked sector (31), stereoscopy cannot be achieved in these sectors, however stereoscopy can be achieved in all other sectors (32) of the panoramic field of view.

In order to change the sector that is covered by both imaging assemblies, i. e. the stereoscopic sector (32), the entire apparatus can be rotated as described now with reference to Fig. 4. The platform (24) is part of a physical structure that holds the imaging assemblies in their place to insure their stability and relative locations as well as supports the entire apparatus. If an axis of rotation (25) and a rotation mechanism (26) are provided, then the rotation mechanism (26) enables rotation of the apparatus (18) around axis of rotation (25) to allow coverage, of the previously blocked sectors, by both of the imaging assemblies. Preferably the rotation mechanism enables rotation of at least 180 degrees in either direction and the axis of rotation (25) is positioned at the center of the platform (24), equidistant from locations (20) and (23). The rotation mechanism (26) can either allow auto- mechanical or manual rotation of the apparatus. It is preferable that special paths would be maintained in the platform (24) for deployment of wires from the imaging assemblies to an external processing unit, recording unit, power sources or other peripherals as required by the application. Details of the design and operation of the platform, the physical structure of which it is a part, and the rotation mechanism will not be discussed herein for brevity, since they are well known in the art and can easily be provided by the skilled person.

A limitation of the embodiment of the invention described hereinabove, is that a full stereoscopic coverage of the entire panoramic field of view (27) cannot be achieved at a single instant, due to mutual blockage by the imaging assemblies. This embodiment is however sufficient, and most effective, for applications that require stereoscopic qualities in specific sectors of the panoramic space, but not in all of it at the same instant. To provide such coverage, the preferred embodiment of the invention to be described hereinbelow, with reference to Figs. 8 to 16 has been developed.

In Fig. 6 is schematically shown an embodiment in which optical zoom lenses are incorporated into each of the two imaging assemblies and the field of view covered by these lenses. The incorporation of zoom lenses together with an omni-directional lens can be performed, for example, by an optical structure such as that shown in Fig. 3. Two separate optical zoom lenses are used, one in the first imaging assembly (33) and another in the second imaging assembly (34). The first optical zoom lens, i. e. the lens incorporated into the first imaging assembly (33), has a field of view covering a first sector (35). The second optical zoom lens, i. e. the lens incorporated in the second imaging assembly (34), has a field of view covering a second sector (36). The sector that is covered by both zoom lenses (37) is the sector in which both optical zoom and stereoscopy are achieved.

The two optical zoom lenses are independent of each other and can be independently rotated to any direction, therefore it is possible to utilize these lenses in order to achieve optical zoom of two different sectors, and not necessarily for the purpose of creating a stereoscopic view of a single zoomed sector. It will be understood by the skilled person that the operation or rotation of the zoom lenses does not affect or compromise the panoramic view that is achieved by the panoramic lenses.

As discussed hereinabove, the creation of a stereoscopic image requires the ability to distinguish between a left image and a right image of a stereoscopic pair. When using zoom lenses, which can rotate and cover different sectors of a full panoramic field of view, it is important to pay special attention to which of the zoom lenses is considered to be that providing the image with the left viewpoint, and which is considered to be providing the right image. Referring to Fig. 6, it can be seen how this distinction can be achieved in the zoomed sectors. A common virtual horizontal axis (38) connects the centers of the first imaging assembly (33) and the second imaging assembly (34) and creates a first field of view (39) and a second field of view (40). For any zoomed sector, which is covered by both zooming lenses and located in the first field of view (39), the first imaging assembly (33) is considered to be providing the left image, and the second imaging assembly (34) is considered to be providing the right one. It is important to notice that the imaging assemblies switch their roles while observing a zoomed sector located in the second field of view (40). For any zoomed sector, which is covered by both zooming lenses and located in the second field of view (40), the first imaging assembly (33) is considered to provide the right image, and the second imaging assembly (34) is considered to provide the left one.

The abovementioned considerations should also be made in respect to the entire panoramic space, not only the zoomed sector. The first imaging assembly (33) provides the image having the left viewpoint and the second imaging assembly (34) provides the image having the right viewpoint in the first sector (39). In the second sector (40), the first imaging assembly (33) provides the image having the right viewpoint and the second imaging assembly (34) provides the left image.

Fig. 7 schematically shows a transformation that is applied to the panoramic images in order to make them suitable for stereoscopic use. As previously described, in order to accurately simulate a stereoscopic display, special attention must be made to distinguish between the images that present-right and left viewpoints. The unique panoramic lenses incorporated in the imaging assemblies used in the present invention create images of the reflections of the entire 360 degrees surroundings on the focal plane arrays of the image capture devices. These reflections appear as a circular image on the focal plane arrays. A first circular image (41) is acquired on the first focal plane array (42) of the first imaging assembly and a second circular image (43) is acquired on the second focal plane array (44) of the second imaging assembly. A straight virtual division line (48) is drawn dividing the space surrounding the imaging assemblies into a first sector of the image and a second sector of the image, where the terms"first"and"second"are used in the same sense as in the description accompanying Fig. 6 hereinabove. The virtual division line (48) passes through the centers of the focal plane arrays of both of the image capture devices, and creates a division of the images as follows: the first circular image (41) is divided to a first half (49) and a second half (45), and the second circular image (43) is divided to a first half (46) and a second half (47). As previously described in reference to Fig. 6; where, in the first sector (39), the first imaging assembly (33) provides the left image and the second imaging device (34) provides the right image ; In the second sector (40) the roles are reversed.

Referring again to Fig. 7, special attention must be made to the effect of the changing angular positions of the imaging assemblies on the viewpoint presented by the images. In the case of the first image (41), images appearing in the first sector (49) are considered as being produced from the left viewpoint and images appearing in the second sector (45) are considered as being produced from the right viewpoint. A similar situation exists for the case of the second image (43), wherein images appearing in the first sector (46) are considered as being produced from the right viewpoint and images appearing in the second sector (45) are considered as being produced from the left viewpoint.

Therefore, a transformation must be made that will create separate left and right images that each cover the complete panoramic view and will be suitable for projection as a stereoscopic pair. The result of the transformation is a left image consisting of the first sector (49) of the first image (41) and the second sector (47) of the second image (43) and a right image consisting of the first sector (46) of the second image (43) and the second sector (45) of the first image (41). As is well known to experienced persons, the images as acquired by the focal plane arrays are not well suited for display to the human eye and interpretation by the human brain, because of their distorted circular shapes; therefore, they are normally subjected to computerized processing and converted to images having a rectangular shape. The transformation described hereinabove, involving switching between halves of the images can be performed either on the circular images or on the rectangular images, as will be described hereinbelow with reference to Fig. 15. In any case, the transformation should be performed before projecting the images to be viewed.

Fig. 8 schematically shows a preferred embodiment of the present invention which overcomes the limitation of the embodiment of the panoramic stereoscopic imaging apparatus described hereinabove. This embodiment utilizes three omni-directional imaging assemblies, in order to allow simultaneous coverage of every sector in the panoramic field of view by at least two different lenses.

In Fig. 8 is shown schematically apparatus (50), which consists of a platform (51), a first panoramic imaging assembly (52) located at a first location (53), a second panoramic imaging assembly (54) located at a second location (55), and a third panoramic imaging assembly (56) located at a third location (57). Locations (53), (55), and (57) are on a platform (51) which is a common horizontal plane for all three imaging assemblies. Platform (51) is part of a physical structure (not shown in the figures or described herein) that holds the imaging assemblies in their place to insure their stability and relative locations as well as supports the entire apparatus. It is preferable that special paths be maintained in platform (51) to enable deployment of wires from the image assemblies to an external processing unit, recording unit, power supply or other peripherals as required by the application.

In this embodiment it is preferred that the three imaging assemblies are structured in the same way and are designed using the same parameters, i. e. are essentially identical. All three panoramic imaging assemblies should be positioned at the same height, more specifically, the panoramic lenses incorporated within the imaging assemblies, should be at the same height above the platform. The orientation of the three imaging assemblies should be such that their central axes are parallel to each other and vertical to the common horizontal plane. In order to provide coverage of every sector in the panoramic field of view by at least two different lenses, without the use of a rotation mechanism to allow stereoscopy as described hereinabove with reference to the embodiment of the invention shown in Fig. 4, the three panoramic imaging assemblies must not be located on a straight line. The preferred layout of the imaging assemblies is shown in Fig. 9.

Fig. 9 schematically shows the preferred layout of the three panoramic imaging assemblies, as seen from an overhead position. In the figure is seen the platform (58), the first panoramic imaging assembly (59), the second panoramic imaging assembly (60) and the third panoramic imaging assembly (61). Virtually connecting the centers of the three imaging assemblies, i. e. the three locations described in reference to Fig. 8, a regular triangle (62) is formed. Since each lens blocks part of the field of view of each other lens, around each of the lenses there exists an area that is covered by only that specific lens. The distance between the lenses is set as a function of the desired size of the area that is covered by only one of the lenses. Moving the lenses closer together increases the sizes of the areas that are covered by only one of the lenses, i. e. the area of overlapping images is decreased. Therefore, special attention must be made not to reduce the distance between the lenses beyond the minimum that allows a total stereoscopic panoramic image to be achieved.

The positioning of the imaging assemblies such that lines drawn between their locations on platform 51 (Fig. 8) form a regular triangle (62) (Fig. 9) is the preferred layout since it insures that every sector in the panoramic space is covered by at least two different lenses. It is also the most suitable structure since it provides symmetric coverage of each sector, which is most convenient for further computerized processing.

Figs. 10A to 10C schematically show the fields of view of each of the three lenses of the apparatus shown in Fig. 8. Figure 10A shows the useable sector (63) of the field of view of the first panoramic lens (64). The rest of the panoramic sector, sector (65), is either hidden behind one of the other two imaging assemblies or is located between them and disregarded. Figure 10B shows-the useable sector (66) of the field of view of the second panoramic lens (67). The rest of the panoramic sector, sector (68), is either hidden behind one of the other two imaging assemblies or is located between them and disregarded. Figure 10C shows the useable sector (69) of the field of view of the third panoramic lens (70). The rest of the panoramic sector, sector (71), is either hidden behind one of the other two imaging assemblies or is located between them and disregarded. It is to be noted that the description of the sectors given herein is a general schematic description and is subject to changes, which result from the size and shape of the panoramic lenses and by the distances between them.

Fig. 11 schematically shows the sector of a panoramic field of view, which is covered by two different lenses. As in Figs. 10A and 10B, and incorporated in the current figure, the first imaging assembly (64) covers a first sector (63), and the second imaging assembly (67) covers a second sector (66). The two sectors overlap in a first overlap sector (72) and a second overlap sector (72'). The sector (72') is redundant and therefore disregarded since it contains stereoscopic information that is available from the primary overlap areas of the other two pairs of lenses. The overlap sector (72) is the sector in which stereoscopy can be achieved, since it is viewed from two different viewpoints, which are those of the first lens (64) and the second lens (67).

The situation described hereinabove applies mutandis mutatis to any pair of lenses; therefore, there are three different sectors in which overlapping of the fields of view occur. All three of them together cover the full panoramic field of view. Therefore, stereoscopy can be achieved in the entire panoramic space, without the need to'move or rotate any component of the system.

Fig. 12 shows schematically the shape of the image that is acquired by each of the panoramic imaging assemblies. As described hereinabove, each of the imaging assemblies comprises a reflective lens, with an optional additional optical zoom lens. The reflective panoramic lens reflects the surrounding scenery as a circular shape (73) towards the image capture device, so that the image that is acquired by the focal plane array is also circular in its shape. The circular image (73) comprises an inner area (74). and an outer area (75). If the optical zoom lens is not incorporated, the inner area (74) will be a reflection of the image capture device. In embodiments which include an optical zoom lens, e. g. those described in Fig. 3, the inner area (74) comprises the reflection of the optically zoomed sector and the outer area (75) comprises the surrounding scenery from around the lens.

In the preferred lens layout of the apparatus, which was describes with reference to Fig. 9, a part of the panoramic field of view of each of the lenses is blocked by the other two lenses. Since part of the field of view is blocked, a sector of the image (76) comprises an image of the two neighboring lenses and the space between them. The image that appears in sector (76) is of no use and may be disregarded.

The circular shape of the image (73) makes it difficult to understand and is therefore generally considered unsuitable for viewing. For this reason, computerized processing techniques are implemented on the circular image (73) to transform it to a rectangular, more understandable, form. The computerized processing creates two separate images: a rectangular image (77), which comprises the surrounding scenery and a separate image (78), which shows the zoomed portion, if a zooming lens was incorporated. The computerized process is done by compatible software that is based on the lens parameters in order to produce the most accurate outcome. The first rectangular image (77) contains a portion (79), which is the portion that includes a part of the image that should be disregarded, i. e. sector (79) in the rectangular image (77) is the equivalent of the disregarded sector (76) of the original circular image (73).

The techniques of computer image processing are well known and therefore, for the sake of brevity, neither these techniques nor the various embodiments of compatible software that are based on these techniques nor the embodiments of the computing and peripheral equipment necessary to run this software are further discussed herein. Experienced practitioners of the art will be able to provide suitable embodiments of all of the software and the hardware necessary to implement the present invention.

Fig. 13 schematically shows the effect of rotation of the zoom lenses, when they are used in conjunction with the panoramic lenses in the apparatus comprising three panoramic imaging assemblies. Each of the panoramic imaging assemblies can optionally incorporate an optical zoom lens (i. e. as described in Fig. 3), whose operation does not influence, and is not influenced by, the operation of the panoramic reflective lenses. Each of the zooming lenses can be independently rotated towards a sector of interest and will create a zoomed reflection of that sector on the focal plane array, as described hereinabove with reference to Fig. 12. The rotation of the zoom lenses can be accomplished either by separate rotation means that rotate the only zoom lenses or, in other embodiments, by rotating the panoramic lens, to which the zoom lens is permanently attached. Since, in the preferred embodiments, the panoramic lenses are each reflective axi-symmetric lenses, their rotation, in order to rotate the attached zoom lens, will not change the panoramic image produced.

Referring to Fig. 13, there are schematically shown: a first panoramic lens (80) which has an optical zoom lens, which covers a first sector (81), incorporated into it; and a second panoramic lens (82) which has an optical zoom lens, which covers a first sector (83), incorporated into it. Each of the panoramic lenses can be rotated to any desired direction in order to dynamically change the sector that is viewed by the respective zoom lens.

When two different zoom lenses are directed towards the same sector (84), stereoscopy can be achieved in that sector. It is to be understood that the use of the zoom lenses does not have to be exclusively for production of stereoscopic images, and each of the zoom lenses can be directed to a different sector in order to view three independent zoomed sectors. It is also to be noted that it is sufficient to direct only two zoom lenses towards the same sector in order to create a stereoscopic image; therefore, the third optical zoom lens can be utilized to zoom on a different sector. Although the description hereinabove is in terms of a particular pair of lenses, it is understood that-the description applies to any pair of zoom lenses. Those skilled in the art will appreciate that in order to achieve stereoscopy in a zoomed sector by utilizing zoom mirrors, it is imperative that the zoom factor is identical for all of the lenses.

Figure 14 schematically shows the division of the panoramic space into three equal parts, in each of which a stereoscopic zoomed sector can be created by the overlapping fields of view of two different zoom lenses. The three parts referred to in this figure are the parts of the panoramic space located between the solid division lines (a), (b), and (c). The division lines are determined by virtually connecting the center point of the system (which is also the center point of the regular triangle which connects the imaging assemblies) with each center of each imaging assembly. Any sector in the first part (85) can be covered by the zoom lenses of the first panoramic imaging assembly (86) and the second panoramic imaging assembly (87).

For any such sector in (85), the zoom lens incorporated within the first imaging assembly (86) is considered to produce the left image and the zoom lens incorporated within the second imaging assembly (87) is considered to produce the right image of the stereo pair. Any sector in the second part (88) can be covered by the zoom lenses of the second panoramic imaging assembly (87) and the third panoramic imaging assembly (89). For any such sector, the zoom lens incorporated within the second imaging assembly (87) is considered to produce the left image and the zoom lens incorporated within the third imaging assembly (89) is considered to produce the right image. Any sector in the third part (90) can be covered by the zoom lenses of the third panoramic imaging assembly (89) and the first panoramic imaging assembly (86). For any such sector, the zoom lens incorporated within the third imaging assembly (89) is considered to produce the left image and the zoom lens incorporated within the first imaging assembly (86) is considered to produce the right image. The determination of which of the overlapping images is to be the right image and which the left image of the stereoscopic pair is important for the projection of the images to the appropriate eye of the observer. This is because a meaningful stereoscopic image can only be viewed if an image produced having a left viewpoint is projected to the left eye and an image produced having a right viewpoint is projected to the right eye.

The method described hereinabove for determination of which image should be the right image and which should be the left is applicable also to the panoramic lenses themselves and not only to the zoom lenses. More explicitly and referring again to Fig. 14, the first part (85) can be simultaneously. viewed by the first panoramic lens (86), which produces the left image in that part, and by the second panoramic lens (87), which produces the right image in that part. The second part (88) is simultaneously viewed by the second panoramic lens (87), which produces the left image in that part, and by the third panoramic lens (89), which produces the right image in that part. The third part (90) is simultaneously viewed by the third panoramic lens (89), which produces the left viewpoint in that part, and by the first panoramic lens (86), which produces the right viewpoint in that part. In Fig. 16 hereinbelow will be demonstrated how these considerations are put into practice in the production of source images suitable for stereoscopic projection.

Fig. 15 shows schematically a pair of images produces by the apparatus incorporating two imaging assemblies, which was described hereinabove in reference to Fig. 4. In Fig. 15 are shown the different sectors, which appear in each of the images, after transformation to a rectangular form. The figure refers only to the panoramic images and not to the zoomed portions, which are processed separately. Image A (91) is generated by the first imaging assembly, and contains a sector which is viewed only by the first imaging assembly (92), a sector that is viewed by both imaging assemblies, in which the first imaging assembly is considered to produce the left image (93) of the stereo pair, a sector that is blocked by the second imaging assembly (94) and a sector that is viewed by both imaging assemblies, in which the first imaging assembly is considered to produce the right image. (95). Image B (96) is generated by the second imaging assembly, and contains a sector that is blocked by the first imaging assembly (97), a sector which is viewed by both imaging assemblies, in which the second imaging assembly is considered to produce the right image (98) of the stereo pair, a sector that is viewed only by the second imaging assembly (99) and a sector, which is viewed by both imaging assemblies, in which the second imaging assembly is considered to produce the left image (100). Since image A (91) and image B (96), each comprise a sector that is a left viewpoint and a sector that is a right viewpoint, and since stereoscopy requires a left image and a separate right image of the same scene, a swap of parts of the images needs to be performed between image A (91) and image B (96) to create two separate images, a totally right image and a totally left image that will constitute the stereoscopic pair. The left image (101) comprises the left oriented part (93) of the first image (91) and the left oriented part (100) of the second image (96). The right image (102) should be comprised of the right oriented part (98) of the second image (96) and the right oriented part (95) of the first image (91). The part of the first image that contains the reflection of the second imaging assembly (94) and the part of the second image that contains the reflection of the first imaging assembly (97) do not contain valuable data and can be disregarded. The area of the first image, which is covered only by the first imaging assembly (92) and the area of the second image that is covered only by the second imaging assembly (99) do not contribute to the stereoscopic qualities of the image, however they do contribute to the general spatial orientation ability, since they include additional sectors of the panoramic scene, not included in the stereoscopic sectors, and they can be incorporated in either picture or both, in the appropriate place, respectively to the sector they image and depending on considerations regarding the specific application.

Fig. 16 schematically shows the three images produced by the apparatus incorporating three imaging assemblies, which was described hereinabove with reference to Fig. 8. In Fig. are shown the different sectors, which appear in each of the images, after transformation to a rectangular form.

The figure refers only to the panoramic images and not to the zoomed portions, which are processed separately. Image A (103), which is generated by the first imaging assembly, comprises a first sector (104) in which the first imaging assembly is considered to produce the image having the left viewpoint, a second sector (105) in which the first imaging assembly is considered to produce the image having the right viewpoint, and a third sector (106) which contains the reflection of the two neighboring imaging assemblies. Image B (107), which is generated by the second imaging assembly, comprises a first sector (108) in which the second imaging assembly is considered to produce the image having the right viewpoint, a second sector (109) in which the second imaging assembly is considered to produce the image having the left viewpoint, and a third sector (110) which contains the reflection of the two neighboring imaging assemblies. Image C (111), which is generated by the third imaging assembly, comprises a first sector (112) in which the third imaging assembly is considered to produce an image having the right viewpoint, a second sector (113) in which the third imaging assembly is considered to produce an image having the left viewpoint, and a third sector (114) which contains the reflection of the two neighboring imaging assemblies.

Since image A (103), image B (107) and image C (111), each comprise a sector that is considered left and a sector that is considered right, and stereoscopy requires a left image and a separate right image, a swap of parts of the images needs to be performed. The right image (115) should comprise the right oriented sector (105) of the first image (103), the right oriented sector (108) of the second image (107) and the right oriented sector (112) of the third image (112). The left image (116) should comprise the left oriented sector (113) of the third image (111), the left oriented sector (104) of the first image (103) and the left oriented sector (109) of the second image (107) in the order described here. The right image (115) and the left image (116) will each separately comprise the full panoramic field of view and will each show a slightly shifted image of the same panoramic perimeter. The sectors of the images, which contain the reflection of the other two imaging assemblies, do not contain any valuable data and can be disregarded. Their presence or absence do not contribute to or influence the stereoscopic qualities of the entire panoramic stereoscopic image. Those sectors are described in this figure as sector (106) in the first image (103), sector (110) in the second image (107) and sector (114) in the third image (111).

It is to be noted that the methods of determining of which lens produces the image having the right or left viewpoint at a particular location in the panoramic field of view are dependant on the configuration and layout of the lenses. The size of the panoramic lenses, their shape and their distance from each other will affect the field of view covered by each individual lens and, therefore affect the fields of view mutually covered by the two lenses of each pair. These effects will have further impact on the sectors of the images themselves, influencing which should be utilized as part of the right image and which as part of the left image. Therefore, special attention should be given to adapting the methods described herein, mutandis mutatis, to the specific design parameters of the apparatus employed. The transformations described hereinabove with respect to Figs. 15 and 16 can be performed at one of three stages of processing the images: - on the circular image that is directly acquired, by swapping the appropriate sectors between the circular images, before transforming the images to rectangular shape; - simultaneously with transforming the circular images to rectangular form, by mapping the appropriate sectors from the circular source images to the relevant rectangular output images; or - only after each circular image is transformed to rectangular shape.

The acquisition and storage of the images, tracking of the positions of the lenses, and other steps required for creating and displaying the final images, including carrying out the transformations described hereinabove that are necessary to produce the final stereoscopic images, are all performed with the aid of computer processing. It is to be noted that the present invention deals with the apparatus and method of producing stereoscopic images of a panoramic scene. Additional aspects, such as image processing and the actual projection methods of the images, are not covered by the present invention, and not described in detail, since they are well within the knowledge of those skilled in the art. Commercially available display systems, such as stereoscopic projectors, active and passive stereoscopic display systems, auto-stereoscopic screens, HMD (Head Mounted Display), anaglyph images and other known methods can be used to display the images that are acquired by the apparatus of the present invention. The image capturing devices incorporated into the panoramic imaging assemblies can be of any type known in the art for collecting images of a scene. If the image collecting device is capable of operating at a suitable speed then not only static images of a scene can be projected but also the apparatus of the invention is capable of providing real-time panoramic stereoscopic video imaging of a scene.

Although the present invention provides a panoramic stereoscopic image, it is often preferable to display to the viewer a limited sector at a time, such that would enable him to fully focus and grasp the stereoscopic effect. The present invention, therefore, provides a constantly available panoramic stereoscopic image, from which any sector may be selected to display to the viewer, and may be changed according to defined parameters, for instance compatibly to the viewer's head movement.

The methods demonstrated herein can also be used with optical structures which enable acquiring a nearly spherical field of view. Examples for such optical structures are presented in WO 02/059676, cited hereinabove. Nearly spherical field of view lenses acquire two separate scenes. The first scene, which is a panoramic scene, is acquired by utilizing reflective surfaces such as those described hereinabove. The second scene comprises an additional sector which is at least partially above the panoramic scene, and which is a partial view of the scene located in front of the imaging device. The second scene is acquired through refractive optics. Therefore, those skilled in the art would appreciate that among the lenses relevant for providing a panoramic field, of view, exist nearly spherical view lenses. It can be further understood that while utilizing nearly spherical view lenses, stereoscopy in the first scene is achieved in accordance with the methods described herein, and stereoscopy in the second scene is achieved by utilizing conventional stereoscopic methods, since the second scene is actually a conventional forward view of the imaging device.

Although embodiments of the invention have been described by way of illustration, it will be understood that the invention may be carried out with many variations, modifications, and adaptations, without departing from its spirit or exceeding the scope of the claims.