Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Applications Nos. 63/287,119, filed December 8, 2021. This application is also a Continuation-in-Part of PCT application PCT/US22/19553, filed March 9, 2022, entitled "System and Method for Using Non-Axis Aligned Bounding Boxes for Retail Detection". Both of which are incorporated herein in their entireties.

Background

[0002] In a retail setting, it is desirable to be able to use computer vision methods to detect and identify products on a retail shelf to aid in management of the retail establishment. For example, computer vision may be used to detect and identify products for various tasks, such as tracking product inventory, determining out-of-stock products and determining misplaced products.

Product detection is one of the fastest-moving areas and plays a fundamental role in many retail applications such as product recognition, planogram compliance, out-of-stock management, and check-out free shopping.

[0003] To this end, numerous computer vision methods have been developed and many real-world applications based on those computer vision methods perform at a satisfactory level. Currently, various visual sensors (e.g., fixed cameras, robots, drones, and mobile phones) have been deployed in retail stores, enabling the application of advanced technologies to ease shopping and store management tasks.

[0004] Object detectors typically comprise a localization sub-network that feeds downstream tasks, such as pose estimation, fine-grained classification, and similarity matching. Most downstream tasks require that the localization sub- network provide a bounding area for each object, for example, products in a retail setting. Therefore, for scene understanding in 2D images, the first step is to detect the objects and represent them by 2D bounding boxes. It is crucial to ensure that the bounding boxes are well aligned with the detected objects to provide accurate information about the products for the downstream tasks. The bounding box is expected to cover the most representative pixels and accurately locate the product while concurrently excluding as much noisy context, as possible, such as background. As shown in FIG. 1(a), retail scene product detectors typically output axis-aligned bounding boxes (AABB) or rotated bounding boxes (RBOX), regardless of the pose of the product. As shown in FIG. 1(b), conventional detectors using AABBs and RBOXs cover the visible entirety of the products, creating inconsistent appearances of the same products and extra difficulties for detection and categorization.

[0005] However, products can be of arbitrary poses in a real-world retail scene, especially when the image is taken by a camera not facing straight towards the shelf, as shown in FIG. 1(b). Additionally, products may have non-regular or unusually-shaped packaging. Because of mutual occlusion, rotation, distortion, and restricted shooting angles in retail scenarios, previous datasets and detectors have difficulty drawing proper bounding boxes to satisfy these requirements because neither an AABB nor a RBOX are able to be perfectly aligned with the actual boundaries of the ill-posed or oddly- shaped products. If AABBs or RBOXs are used as the bounding box shape to annotate the products, there will always be irrelevant background 102 or multiple sides of the product 104 included in the boxes, as shown in FIG. 1(b), or parts of the products may be cut out. As such, the most precise object regions cannot be retrieved. Therefore, the features extracted from these object regions may not be accurate for the downstream tasks.

Summary

[0006] To address the issues identified above, disclosed herein is a system and method implementing an object detector for predicting non-AABB, convex- shaped regions of interest whose edges are tightly aligned with the boundaries of arbitrarily posed objects. In one embodiment, the objects may be retail products.

[0007] In one embodiment, the system and method generates quadrilateral boxes which tightly cover the most representative faces of the retail products. The detector disclosed herein represents the quadrilateral boxes by a central point and four offsets. The system and method provides two benefits compared to the conventional AABB format. First, the quadrilateral boxes do not include unnecessary background information or miss parts of the objects, so that features extracted from the predicted bounding boxes are precise and informative. Second, a quadrilateral box itself already encodes some pose information of the enclosed object. With a simple 2D projection transformation, the pose can be normalized as if the camera is facing straight towards the object. Thus, a simple projection transformation can be applied to correct the pose of products for downstream tasks.

[0008] In other embodiments, other convex shapes, for example, triangles or ellipses may be predicted as bounding boxes. In yet another embodiment, complex concave polygons, that is, concave polygons having greater than 4 sides, are predicted as the bounding boxes. The complex concave polygons provide a tighter fit of the bounding box, especially when the product is of non-regular shape.

Brief Description of the Drawings

[0009] By way of example, a specific exemplary embodiment of the disclosed system and method will now be described, with reference to the accompanying drawings, in which:

[0010] FIG. 1(a) is an illustration showing the difficulties associated with AABB or

RBOX bounding boxes. [0011] FIG. 1(b) is an illustration showing that axis-aligned bounding boxes often include irrelevant background information or cut out parts of the product when the products are ill-posed.

[0012] FIG. 2 is an illustration showing the bounding boxes of the quadrilateral detector disclosed herein.

[0013] FIG. 3(a) is an illustration showing how an axis-aligned bounding box captures multiple faces of the same product, whereas FIG. 3(b) shows that the quadrilateral detector captures only the front-facing surface of the product container.

[0014] FIG. 4 is a representation of an exemplary quadrilateral bounding box produced by the quadrilateral detector disclosed herein, represented by a central point and four distance offsets to specify the coordinates of the corners of the box.

[0015] FIG. 5 is an architectural diagram of the base network.

[0016] FIG. 6 is an illustration showing an AABB center applied on a QUAD (FIG.

6(a)), as opposed to a gravity center (FIG. 6(b)). FIG. 6(c) shows the quad- centeredness map with a shrunk ratio applied.

[0017] FIG. 7 is a representation of the elliptical shaped bounding box represented using a center point and two distance offsets representing the lengths of the major and minor axes of the ellipse. [0018] FIG. 8 is a representation of a triangular shaped bounding box represented using a central point and three distance offsets representing the coordinates of the vertices of the triangle.

Detailed Description

[0019] There are two aspects to the disclosed invention. In the first aspect, a training dataset containing images of retail products annotated with quadrilateral- shaped or complex polygonal-shaped bounding boxes is developed and used to train the quadrilateral or polygonal detector. In the second aspect, a strong quadrilateral or polygonal detector is disclosed that out-performs prior art detectors on the training dataset. The detector, in one embodiment, produces quadrilateral bounding boxes, as shown in FIG. 2, and, in another embodiment, produce complex polygonal bounding boxes (i.e., bounding boxes having 5 or more sides) that provide the advantages of excluding unnecessary background information and more precisely including pixels representing the actual product.

[0020] Training Dataset

[0021] The training dataset is designed with three features to solve the aforementioned challenges: (1) bounding boxes of products are densely labeled in quadrilateral or polygonal style by well-trained annotators and multiple rounds of re-correction. Exemplary bounding box annotations are illustrated in FIG. 2. Quadrilaterals can adequately reflect the shape and pose of most products like boxes, bottles, cans, chests, and bags regardless of the shooting angles, and efficiently handle the irregular cases like clothes, balls, sunglasses, etc. Polygonal bounding boxes may be used for oddly-shaped products; (2) the bounding box annotations only cover the front face of products (See FIG. 3(b)) when multiple faces are visible, as opposed to AABB, which would cover all exposed faces (See FIG. 3(a)). The front face of the products provide the most distinguishing information about the product and keeps the appearance consistent for the same products; and (3) two different testing sets support two product detection tasks: origin-domain detection and cross-domain detection. One testing set shares the domain with the training set, while another is independently collected from different stores, with different shooting equipment, and at more difficult shooting angles.

[0022] Image Collection - Practically, a variety of sensors are utilized under different conditions for on-shelf product detection. The resolution and shooting angles cover an extensive range by different types of sensors. Specifically, robots usually take high brightness pictures from the bottom up using high-quality cameras and build-in light source, shaping most products into a trapezoid shape. Fixed cameras are, in most cases, mounted on the ceiling, creating low-resolution images from top to bottom; staff and customers prefer to photograph with mobile phones from the front or side, shaping products into a rhomboid shape. The product categories sold in different stores also show a great deal of variety. [0023] Considering these factors, images are collected from two sources to support origin-domain and cross-domain detection. In the origin domain, training and testing images share a similar style and are pictured at similar angles in the same stores by the same sensors. As a result, images are selected from a prior product dataset to form the origin domain.

[0024] These images have three properties: (1) They are collected from a limited number (e.g., <5) of stores worldwide; (2) All images are shot by humans holding mobile phones from side or front perspectives; and (3) The diversity of categories is rich but still highly limited.

[0025] In the cross domain, approximately 500 images are collected in 5 different stores (100 for each) from multiple sensors, cover unseen categories, and mimic the view angles of fixed cameras and robots.

[0026] Annotation - Each product is annotated with a quadrilateral bounding box, referred to here as a "QUAD" or a polygonal bounding box. A QUAD refers to 4 points with 8 degrees of freedom while a polygonal bounding box may have any number of points, depending on the number of sides.

[0027] For regular shaped products mainly in cuboid and cylinder containers, the is defined as the top-left corner of the front face of the product, and the other points represent the other corners in clockwise order. For spheres, cones, and other cases, for which it is hard to identify corners, or front faces, and for irregular-shaped products where such defined quadrilaterals cannot cover the entire front face, the minimum AABB is first drawn and the four corners are then adjusted following the perspective transformation. The front face has the most representative information and is also critical for consistent appearance, but the side face is still annotated if the front face is invisible.

[0028] In one embodiment, in total, 1,777,108 QUADs are annotated by 13 well- trained annotators in 3 rounds of correction. The origin domain is split to training (8,216 images, 1,215,013 QUADs), validation (588 images, 92,128 QUADs), and origin-domain testing set (2,940 images, 432,896 QUADs). The cross domain composes the cross-domain testing set (500 images, 37,071 QUADs).

[0029] In embodiments wherein complex, concave polygons are used for the bounding boxes, the training dataset, and its creation, are identical to the dataset and process for creating the dataset described above, except that the dataset comprises images annotated with the polygons having the desired number of sides instead of with quadrilaterals. For example, hexagonal polygons, and are described by 6 points and having 12 degrees of freedom.

[0030] Detector

[0031] A strong baseline detector designed exclusively for quadrilateral or polygonal product detection is first disclosed. The base network will be introduced first. Afterward, a ground-truth assignment strategy is disclosed. Finally, a corner refinement module is disclosed. [0032] The detector extends the localization subnet to have different output definitions. In one embodiment, a quadrilateral box is represented as Q = , where are vertices of the bounding box, as shown in FIG. 4. The localization subnet has 4 additional channels (8 channels total, 4 pairs) in the output map. Each pair of channels correspond to the distance offset from the central points to if the central points are positive. During training, the offsets are normalized by the feature stride and smooth LI loss is applied for optimization. The polygonal embodiment is a generalization of the quadrilateral embodiment, and the representation of each polygon depends on the number of sides.

[0033] Base Network - An architectural diagram of the base network appears in FIG.

5. The design of the base network applies a prior-art DenseBox-style head to multiple feature pyramid levels. The feature pyramid 502 is generated via a feature pyramid network (FPN) which utilizes a deep convolutional network as the backbone. As an image 504 is fed into the backbone and several feature maps are extracted to compose the initial feature pyramid. The ResNet family is adopted as the backbone, and the extracted feature maps are from C ₃ to C ₅ . Generally, low feature pyramid levels have high resolution but weak semantic information (i.e., these feature maps are gradually down- sampled but semantically enhanced). The FPN leverages a top-down module that up-samples the high feature pyramid levels and sums them to the adjacent lower levels to enhance semantic information. The feature maps after the FPN are denoted as P ₃ , P ₄ , P ₅ . An anchor free detection head is then attached. The head contains two branches. One is a binary classification branch 506 to predict a heatmap for product/background. Another is a regression branch 508 to predict the offset from the pixel location to the corner points of the QUAD or polygon. Each branch consists of 3 stacks of convolutional layers followed by another c channel convolutional layer, wherein c = 1 for the classification branch and, for et quadrilateral embodiment, c = 8 for the regression branch (or 2x the number of sides in the generalized polygonal embodiment).

[0034] In the alternate embodiment, which is generalized for bounding boxes of any shape, the head contains an additional branch determining which shape is suitable to fit the product, named the shape-fit branch (not shown). The shape-fit branch operates multi-class classification which may include classes for all or any subset of quadrilateral, triangular, axis-aligned rectangular, rotated rectangular and complex polygonal shaped bounding boxes. The regression branch consists of c channel convolutional layers, where c equals the largest number of degrees of freedom of any shapes to be classified by the shape-fit branch. For example, 8 for quadrilaterals, 4 for axis-aligned rectangles, 6 for triangles, 5 for rotated rectangles, 12 for hexagons, etc. So, c is equal to max(12,8,4,6,5) = 12.

[0035] During inference, the shape-fit branch will determine which shape should be used for the product being detected and the regression branch will produce the corresponding offsets. If the shape-fit result is a quadrilateral, the first 8 values produced from the regression branch are used; if the shape fit result is a triangle, the first 6 values are used, etc.

[0036] Ground-truth Assignment - The ground-truth assignment strategy plays a vital role in the training phase. Here, two aspects are focused on: (1) on-map assignment; and (2) cross-pyramid assignment.

[0037] On-map: Centerness - The common definition of the centerness of an AABB is shown in Eq. (1):

[0038] By Eq. (1), the feature pixel at position (i, j) is considered as the "center point" if it keeps the same distances to the left and right AABB boundaries and, concurrently, keeps the same distances to the top and bottom AABB boundaries . This is denoted it as the "AABB center". The AABB center has the highest centerness as 1, and the other pixels have degraded centerness calculated by Eq.(l). However, when adopting the AABB center to quadrilaterals, as shown in FIG. 6(a), the center can be far away from a corner, which leads to unbalanced regression difficulty and lack of receptive field from that corner. [0039] To solve the above problem, for the quadrilateral detector, the "QUAD center" is defined as the center of gravity, not only because it is the geometric center of the QUAD but also because it represents the mean position of all the points in the shape, which mitigates the unbalanced regression difficulties, as shown in FIG. 6(b). Eq. (2) can then be used to calculate the quad-centerness for any where: g denotes the gravity center; denotes the distances between the gravity center g and the left/right/top/bottom boundaries; and denotes the distances between the and the boundaries.

[0040] If locates on the gravity center, its quad-centerness has the highest value as 1. Otherwise, the quad-centerness are gradually degraded, as shown in FIG. 6(c).

[0041] It is mentionable that the centerness calculated by Eq. (1) is a special instantiation of the quad-centerness calculated by Eq. (2). This is because, when QUAD is specialized to an AABB, such that Eq.

(2) is mathematically equivalent to Eq. (1). [0042] For the generalized detector, the center of gravity of the complex polygon pij can be calculated using a generalization of Eq. (2): where: k is the index of the boundary (edge) of the shape;

K is the total number of boundaries (sides); denotes the distances between the gravity center g and the K ^th boundary; and y is a hyper-parameter for normalization, typically 0.5.

[0043] Cross-Pyramid: Soft Scale - A fast assignment strategy across pyramid levels is crucial for training where each image contains hundreds of objects. Prior strategies are typically scale-based (i.e., assigning objects to different levels in terms of their scales). The larger the scale, the higher the level to which the objects are assigned, so that the needs of receptive field and resolution of feature maps are well balanced. Herein, a loss-based strategy (termed Soft Selection) is used, where object scale does not indicate pyramid level. Instead, it first assigns each object to all pyramid levels P ₃ , P ₄ , P ₅ and calculates for each level P (which, in this case, = 3, 4, 5). Then, the level that produces the minimal loss is converted to a one-hot vector (i.e., (1,0,0)) if the minimal loss is from P ₃ ; (0,1,0) if it is from P ₄ , and so on). The vector is used as the ground-truth to train an auxiliary network that simultaneously predicts a vector (F ₃ , F ₄ , F ₅ ). Each element F _t is a down- weighting factor for . The final loss of each object is

[0044] Soft Selection outperforms scale-based strategies on generic datasets. However, it is highly inefficient because it independently calculates losses for each object and slowly trains the auxiliary network. In practice, when the number of instances per image becomes large, the training process takes exceptionally longer (~4x — 5x ) than scale-based strategies.

[0045] The merit of Soft Selection can be maintained while accelerating the assignment by accounting for the relationship between loss and scale. By Soft Selection, the minimal loss from level indicates that the auxiliary network is trained to generate a relatively larger but the loss is not independent of scales. On the contrary, object scale inherently determines which level will produce the minimal loss. The reason is as follows. First, when assigning objects (e.g., object A with size 8 x 8 and B with size 16 x 16) to the pyramid, their regression targets (denoted as T _A , T _B ) are normalized by the level stride. Specifically, on a lower level (like F ₃ ), the target is divided by stride 8, while on a higher level (like F ₄ ), the target is divided by 16, and so on. Therefore, when assigning A to F ₃ and P ₄ , T _A is 1 x 1 and 0.5 x 0.5, respectively; when assigning B, T _B is 2 x 2 and l x l, respectively. Note that all levels share the detection head. The combination of and T _A = 1 x 1 and T _B = 1 x 1 leads to the smallest regression difficulty for the regression head. Naturally, it produces minimal regression losses, which means the smaller object is assigned to a lower level. Second, because A has a smaller scale, it requires more local fine-grained information beneficial for classification, which is more available from high-resolution, lower levels. In comparison, B has a larger scale and needs a larger receptive field, which is more available from higher levels. Therefore, the "loss-based" Soft Selection, in essence, follows the scale-based law.

[0046] Nevertheless, Soft Selection outperforms scale-based strategies. The improvement can be credited to its loss reweighting mechanism. This mechanism involves multiple levels during training and reweights the loss in terms of the regression and classification difficulties, making optimization easier. Because the pyramid is discrete, if an object scale falls into the gap of two adjacent levels, the difficulty of both levels will be similar. The auxiliary network has opportunities to learn to predict proper for both levels.

[0047] The analysis motivates the abandonment of the auxiliary network and the design of a scale-based solution based on Soft Scale (SS). For an arbitrary shaped object 0 with area Area ₀ , SS assigns the object to two adjacent levels by Eqs. (4) and (5) and calculates the loss-reweighting factors by Eqs. (6) and (7) respectively.

[0048] Eq. (5) is borrowed from FPN, where, in one embodiment, 224 is the

ImageNet pre-training size. Objects with exact area 224 ² are assigned to in which case . If an object is with area 223 ² , FPN assigns it to while SS assigns it to = 0.994 and to with is fixed at 5. SS operates as rapidly as scale- based strategies, keeps the loss-reweighting like Soft Selection, and greatly improves the performance of the quadrilateral detector.

[0049] Corner Refinement Module - A corner refinement module (CRM) is provided to make the quadrilateral detector two-stage. For each predicted bounding box from the detector, the locations of its corners and center are obtained.

Bilinear interpolation is then used to extract X + 1 features (X corners and 1 center) from the feature map generated by the 3rd stacked convolution in the regression branch. These features are concatenated and fed into a 1 x 1 convolutional layer to predict the difference between ground-truth and the previous prediction. The same operation and convolution are also inserted in the classification branch to predict object/background as a 2nd-stage classification. During testing, the regression results from the two stages are combined but only the classification result from the first stage is trusted.

CRM shares the sprits with Faster-RCNN, but the 5 points mentioned above are enough for quadrilateral products, and the 2nd-stage classification supervision helps training, though not involved in testing.

[0050] Losses - During training, the bounding boxes are first shrunk by a ratio according to the gravity centers. If one feature pixel locates inside the shrunk bounding box, the pixel is considered responsible for learning the ground- truth. Focal loss is utilized for classification and SmoothLl loss is used for regression. Both losses are re-weighted by the production of quad- centerness and level reweighting factor F. The total loss is the summation of the classification and regression losses. If two-stage, additional focal loss and LI loss for CRM are added to the total loss.

[0051] Alternate Embodiments

[0052] In alternate embodiments of the invention, shapes other than a quadrilateral or complex polygonal may be chosen for the bounding box.

[0053] In one alternate embodiment, an elliptical or circle bounding shape may be used, as shown in FIG. 7. The box is represented as R = {x, y, a, b}, where (x, y) are the coordinates of the center and (a, b) are the length of two axes as the vertices for the bounding shape. The localization subnet has a total of 4 channels in the output map. Two of them correspond to the distance offset from the central points to (x,y) if the central points are positive. The other two correspond to the lengths of the major and minor axes (a, b) of the ellipse. During training, the offsets are normalized by the feature stride and a smooth L1 loss is applied for optimization.

[0054] In a second alternate embodiment, a triangular bounding shape may be used, as shown in FIG. 8. In this case, the box is represented as T = are vertices of the triangle. The localization subnet has a total of 6 channels in the output map. Each two channels correspond to the distance offset from the central points to p _i if the central points are positive. During training, the offsets are normalized by the feature stride and smooth L1 loss is applied for optimization.

[0055] In yet a third alternate embodiment, the quadrilateral detector can be extended for generating 3D bounding boxes with arbitrary poses. The detector can output N channels from the localization subnet, where N is the minimal number of parameters to represent the 3D shape. For example, in a cuboid with rectangle faces, N is twice the number of vertices. In a sphere, N is 3 which corresponds to the distance offset from the central point to the sphere center and the radius. [0056] As would be realized, other 2D and 3D shapes for the bounding box may be contemplated to be within the scope of the invention.

[0057] This disclosed detector may be used for pose normalization of the detected bounding boxes. The bounding boxes, are not necessarily bound by width and height displacement from the center. Thus, the bounding boxes can provide independent point we cried about s on the detected object and can capture the shear in the objects. The bounding boxes may be pose corrected by projecting the sheared boxes onto a reference plane through a homography matrix or any other transformation to help in pose normalization for matching.

[0058] Product detection is challenging and fundamental in the retail industry.

Herein is disclosed a new dataset and a customized quadrilateral detector, which detects products as quadrilaterals as opposed to AABBs. As would be realized by one of skill in the art, the disclosed method described herein can be implemented by a system comprising a processor and memory, storing software that, when executed by the processor, performs the functions comprising the method.

[0059] As would further be realized by one of skill in the art, many variations on implementations discussed herein which fall within the scope of the invention are possible. Moreover, it is to be understood that the features of the various embodiments described herein were not mutually exclusive and can exist in various combinations and permutations, even if such combinations or permutations were not made express herein, without departing from the spirit and scope of the invention. Accordingly, the method and apparatus disclosed herein are not to be taken as limitations on the invention but as an illustration thereof. The scope of the invention is defined by the claims which follow.