TECHNICAL FIELD

[0001] The disclosure relates to a system and method for identification of objects and prediction of object class.

BACKGROUND

[0002] Consumer-grade waste sorting industry has been given increasing attention due to its contributions to developing a more sustainable method of handling recyclable waste. Examples of objects that are sorted may be in accordance with classes such as plastics, metals, cardboards, paper, and glass. A large part of the plastic segregation process is typically carried out manually, which raises concerns with regards to the labor capacity, and the quality of sorting.

[0003] A solution uses robotic arm(s) to segregate metal, plastic, and organic waste into different baskets with the aid of an infrared sensor. This application, however, does not account for the different types of plastics (such as PET and HDPE), and is limited in that it only uses a single sensor type to analyse the thermal images produced by an infrared sensor. [0004] Another solution disclosed in US patent publication US20180016096A1 focuses on segregating recyclable items from non-recyclable items, and further categorising the recyclable items based on object class such as plastic, paper, metal, cardboard, cloth and glass. However, this is a relatively costly solution in that a large number of sensors are used to produce digital images for classifying these recyclable items, and the solution does not classify between different types of plastics.

[0005] Another solution disclosed in EP patent publication EP0824042A1 utilises a laser source directing a polarised beam on to different types of plastics (such as PET, HDPE, PVC), which may be expensive and requires extensive or onerous safety considerations.

[0006] Another solution disclosed in US20140367316A1 is an industrial grade solution rather than a consumer-grade solution. The disclosure utilised Terahertz (THz) spectroscopy to identify and distinguish industrial plastics and flat sheet plastics (black HIPS, PC, PS, ABS, PC-ABS, acrylic and ACETAL), not post-consumer plastics (especially PET and HDPE bottles) and item level sorting (arbitrary 3D shape). This system requires complex pre-sorting and long waste stream chain and requires a huge investment, i.e., this THz spectroscopic approach is relatively slow and costly. [0007] While near infra-red (NIR) and RGB camera-based sensors are the among most commonly sensors in residential waste sorting process, there are limitations of each sensor which can only provide reliable predictions in certain conditions. For example, NIR cannot penetrate plastics and could not get any color information from objects. In addition, NI R optical sorters are not able to sort labels on plastic bottles, black plastics, and multi-layers plastics. Poor sorting performance may also result from cross-contamination with food waste. The limitation of RGB cameras includes its sensitivity towards varying lighting conditions, optical illusions of overlapping items, and inability to view and differentiate bottle-type objects that are covered with labels or irregularities such as dirt and dust.

[0008] There exists a need to provide a solution to alleviate at least one of the aforementioned problems.

SUMMARY

[0009] The disclosure provides a relatively fast and intuitive automated object identification and class prediction based on a sensor-fusion based solution. The solution utilizes power intensity heatmap image generated based on THz electromagnetic radiation and RGB images for specific plastic stream (for example PET and HDPE bottles) classification in small-scale or consumer grade M R F and recycling companies, providing a novel combination of high contrast heatmap imaging, high spatial resolution imaging (for RGB images) and intuitive machine learning analysis. Aspects of the disclosure include a unique sensor-fusion (RGB and THz sensor) and their data (features)-driven machine learning algorithm for enabling a waste sorting solution.

[0010] According to the present invention, a method for predicting a class of an object as claimed in claim 1 is provided. A system for predicting a class of an object according to the invention is defined in claim 7.

[0011] The dependent claims define some examples associated with the method and system, respectively.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The invention will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which: - FIG. 1 is a flow chart of a method for identifying an object and predicting a class of the object;

- FIG. 2 A is a system setup for identification of objects and prediction of object class; FIG. 2B shows an example of the images captured by a first sensor and a second sensor;

- FIG. 3A to 3E illustrate various image processing modules and techniques associated with the identification of objects and prediction of object class;

- FIG. 4A shows a process flow for the prediction of object class; FIG. 4B shows an accuracy test result of the prediction model according to some embodiments;

- FIG. 5 illustrates an example of an ensemble module operable to combine multiple prediction results to form a final prediction result; and

- FIG. 6 shows a schematic illustration of a processor 210 for identification of objects and/or prediction of object class according to some embodiments.

DETAILED DESCRIPTION

[0013] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0014] Embodiments described in the context of one of the systems or methods are analogously valid for the other systems or methods.

[0015] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and/or combinations and/or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0016] In the context of some embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements. [0017] As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

[0018] As used herein, the term “object” includes any object, particularly recyclable or reusable object that may be sorted according to type and/or class. For example, plastic objects may be sorted according to whether they belong to the class of High-Density Poly Ethylene (HDPE) or Polyethylene terephthalate (PET). The objects may include bottles, jars, containers, plates, bowls etc. of various shapes and sizes.

[0019] As used herein, the term “associate”, “associated”, “associate”, and “associating” indicate a defined relationship (or cross-reference) between at least two items. For instance, an image associated with an object may include a defined region of interest which focuses on the object for further processing via object detection and/or tracking algorithm(s).

[0020] As used herein, the term “module” refers to, or forms part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

[0021] As used herein, the term “RGB image” broadly refers to images that comprises image data having colors encoded as three numbers associated with the color ‘red’, ‘green’ and ‘blue’.

[0022] FIG. 1 shows a flow chart of a method 100 for predicting a class of an object comprising the steps of: receiving an RGB image and a power intensity image (which may be in the form of a heatmap image) associated with the object (step S102); identifying on the RGB image and the heatmap image, a first region of interest and a second region of interest associated with the object (step S104); obtaining a first segmented image associated with the first region of interest and a second segmented image associated with the second region of interest (step S106); and obtaining a prediction of the class of the object using the first segmented image and the second segmented image (step S108). The step of obtaining a prediction of the object class may include a step of pairing the first segmented image and the second segmented image. The step of pairing may include steps of assigning a common identifier to the first segmented image and the second segment image.

[0023] FIG. 2A shows a setup of a system 200 for predicting classes of objects according to the method 100. The system 200 for predicting a class (e.g. HDPE plastic, PET plastic, paper cardboard, metal glass etc.) of an object 201 comprises a first sensor 202 configured to capture at least one RGB image 204 associated with the object; a second sensor 206 configured to capture at least one power intensity image, i.e. heat map 208 associated with the object 201. The object 201 may be positioned on a platform 205, such as a conveyor belt. The system 200 comprises a processor 210 arranged in data or signal communication with the first sensor 202 and the second sensor 206 to receive the at least one captured RGB image 204 and heatmap image 208. It is contemplated that multiple RGB images and multiple heatmap images of the same object may be captured at different time stamps, each image pair (RGB, heatmap) may yield a predicted class of the object. In some embodiments, multiple predicted classes associated with the image pairs may be ensembled to derive a final prediction of the class.

[0024] The first sensor 202 may be an RGB capturing device, such as a camera, a video recorder, or a camcorder, to capture continuous RGB images of the object at different positions on the platform 205. The second sensor 206 is a Tera-hertz (THz) based image capturing device configured to capture heatmap images of the object at different positions on a platform. The THz based image capturing device may include a THz source configured to emit an electromagnetic radiation in a THz, super-THz or sub-THz frequency range. For example, a sub-THz region (0.1-0.7 THz) and higher frequency region (1.0-3.0 THz). The THz based image capturing device may also include a scanner arranged to receive electromagnetic radiation (emitted from the THz source) incident on an object and to generate a power intensity heatmap image.

[0025] In some embodiments, the first sensor 202 and second sensor 206 may be remotely operated to capture images of the object simultaneously or concurrently at predetermined intervals. In some embodiments, the first sensor 202 and the second sensor 206 may be configured to capture or record their respective images continuously.

[0026] FIG. 2B shows an example of a captured RGB image 204 and a heatmap image 208 (also referred to as THz output image). In some embodiments, multiple RGB images of the object (known as frames) may be captured and stored on an associated database of the first sensor 202. The captured THz heatmap image frames at a specific THz spectrum may be stored in an associated database of the second sensor 206 and converted into multiple frames of heatmap images 208.

[0027] The captured images 204, 208 may be sent to the processor 210 for further processing. The processing of captured heatmap image(s) may include the step of obtaining image data via Fourier-transform spectroscopy and/or THz time-domain spectroscopy techniques as would be known to those skilled in the art. [0028] In some embodiments as shown in FIG. 3A, the processor 210 comprises an image label module 212, an object detection and tracking module 214, an image segmentation module 216, a pairing and classification module 218, and an ensemble module 220. The modules 212, 214, 216, 218 and 220 are described for the purpose of clarity and for ease of understanding. It is contemplated that one or more of the aforementioned modules may be combined. It is further contemplated that all the modules may be combined.

[0029] The image label module 212 is operable to label each image with a graphical annotation tool and may be saved as Extensible Markup Language (XML) files. In some embodiments, the images were labelled to belong to only one class, “object”. The annotation may be in the form of bounding boxes 302, 304 around the objects of interest. FIG. 3B shows an example of the bounding boxes 302 annotated around objects of interests in an RGB image 204 and a bounding box 304 around an object of interest in a THz heatmap image 208.

[0030] The object detection and tracking module 214 comprises an object detection algorithm and a tracking algorithm to receive the image data of the RGB image(s), the image data of the heatmap image(s) for analysis so as to obtain the (Region of Interest) ROI of each object in different time stamps, that is, images of the same object tracked across a time period. The object may be positioned on a moving platform such as a conveyor belt, which moves at a predetermined speed, ranging from 0.45 to 1.5 meters per second

[0031] In some embodiments, the object detection algorithm may be a pre-trained deep learning machine learning or artificial intelligence-based algorithm, such as, but not limited to a deep convolutional neural network model such as the YOLO (You Only Look Once) v4 model, which is based on a DenseNet classification model.

[0032] In some embodiments, the tracking algorithm may be a feature matching-based algorithm to track objects over multiple image frames for the RGB image capturing device and the THz based image capturing device. An example of the tracking algorithm may be the DeepSORT algorithm. The algorithm checks the important features of the object in one frame and compares the features in the following frames to track the object throughout the conveyor belt.

[0033] In some embodiments, the object detection algorithm and the object tracking algorithm may be combined or cascaded. In the combination, both the algorithms are configured to receive image data from the image capturing devices 202, 206. The tracking algorithm is configured to receive output from the object detection algorithm (i.e. boundary boxes, defined ROI) as input parameters for pre-processing. The combined output of the object detection algorithm and object tracking algorithm is an output video showing the ROI around each image captured object across each image frame. [0034] The image segmentation module 216 is operable to receive the output video data from the module 214 as input, and generate segmented images of the RGB images and THz images as output based on the ROI identified on each image frame. In other words, when the object is tracked through the movement of the conveyor belt, one segmented RGB image and one segmented heatmap image per object at a particular time stamp is saved in a database. The image segmentation module 216 may comprise a segmentation algorithm to model the segmentation as a max-flow/min-cut optimization problem to be solved by a GrabCut algorithm which iteratively extracts the foreground of an image (i.e. the ROI) using graph cuts. The algorithm may use a Gaussian Mixture Model (GMM) to model the foreground and the background of the image based on color distribution of the target object (as defined by the ROI) and that of the background to get the segmented foreground image of the objects on the platform. An example of the extracted segmented images 306, 308 in the form of GrabCut images for RGB (segmented image 306) and THz (segmented image 308) is shown in FIG. 3C.

[0035] The pairing and classification module 218 is operable to process the segmented images output from the image segmentation module 216. Several image processing methods may be implemented on the segmented images, including, but not limited to, (i) adding a background (for example a black canvas) in the background, (ii) augmentation and (iii) normalization. The addition of black canvas is motivated by a requirement that the input dimension of the RGB segmented image and the corresponding heatmap image captured at the same time stamp should be the re-sized, so that all RGB images and THz images have the same size. In order to minimize distortion and deformation of the segmented images 306, 308 caused by image resizing, the black canvas with fixed width and height was added in the background for each image, after which the THz input image 312 is 356x371 pixels, and RGB input image 310 is 350x480 pixels as shown in FIG. 3D.

[0036] The augmentation may be based on a tensorflow-based augmentation applied to eliminate the impact of both position and orientation of objects and yet retain the black background colour. Flipped or rotated images for the input images 310, 312 may be generated as additional input images. FIG. 3E shows the additional flipped or rotated images 314, 316 generated for the RGB input image 310 and the THz input image 312 respectively.

[0037] Normalization on all the input images 310, 312, 314, 316 may be performed before classification takes place. Every pixel value in an image may be divided by a reference value (e.g. 255), ensuring all values to be within the range between 0 and 1.

[0038] The classification module 218 is further operable to fuse or combine the processed input images 310, 312, 314 and 316 using one or more multi-input deep convolutional neural- networks. FIG. 4A shows a process flow of the classification process 400. In some embodiments, the one or more multi-input deep convolutional neural networks comprise two pre-trained convolutional neural network model 402, 404 with deep learning capabilities for large scale image recognition (such as VGG16 models) for feature extraction from the input images 310, 312, 314 and 316. In some embodiments, a shared layer 406 and one or more dense layers 408 are configured to receive and concatenate the extracted features to produce a single prediction of the object class 410. In some embodiments, the deep learning is modeled as an error minimization problem, with a loss function, such as a categorical_crossentroy loss function applied with an optimizer, such as an Adam optimizer for training. The parameters are updated as the optimization progresses based on forward and backward propagation techniques, until the error reaches a minimum value or below an acceptable reference value (i.e. optimized or near optimized).

[0039] Although it is possible to provide a final prediction based on a single predicted result, multiple predictions of the same object using multiple processed images reduce the risk of wrong prediction which could be caused by improperly captured images (e.g. blurred images). The ensemble module 220 operates to combine multiple predicted results associated with each object to be classified.

[0040] As shown in FIG. 5, for each object (having an assigned identifier idl), there will be three pairs of segmented images (502, 504, 506) passed to the classification module 218. Raw images associated with the three pairs of segmented images 502, 504, 506 are obtained from the first sensor 202 and second sensor 206 at three different time ti, ta, and ts. Each object to be classified will be predicted for three times to produce result 1, result 2, and result 3. The three results will be ensembled to produce or generate the final result. In some embodiments, the ensembling may be in the form of voting. In other words, the most predicted class will be the final result of the object and saved to the database with the respective track ID idl. It is contemplated that other statistical methods for ensembling may be used.

[0041] It is appreciable that the RGB camera 202 and THz sensor 206 are synchronized with each other to capture input video data and heatmap data into the processor 210 respectively. Following the input, the processor 210 performs object detection to localize the object and tracks the object throughout the frames (at different time stamps). For this process, the processor 210 saves a segmented image in RGB and a segmented image in THz per object per time stamp into a database. These images are then fed into the hybrid classification model which predicts the class.

[0042] It is contemplated that the machine learning and/or artificial intelligence-based algorithms deployed in various modules may undergo training and testing before operationally deployed. Such training may include the formation of testing and training dataset, and supervised learning, unsupervised learning.

[0043] In some embodiments, upon data collection from the first sensor 202 and second sensor 206, the image frames are extracted from the video files at/with 30 frames per second (FPS). After extraction, there may be 535 RGB and 535 corresponding THz images, forming a total of 1070 images. The extracted images may then be labelled using the image label module 212 and the annotations are saved as XML files. Once the images were labelled, the dataset was split into training and test dataset. The training dataset consisted of 85% (910 images) of the data and the test dataset consisted of 15% (160 images) of the data. Following the splitting, the training dataset may be augmented (this augmentation is different from that performed by the pairing/classification module 218). The augmentation includes horizontal and vertical flipping, 90-degree rotations, 10% crop, and 1.5px blur, etc. Upon augmentation, there were 2,727 images in the training dataset. Based on the 2,727 images, there were 3,545 labelled objects. This dataset is then used fortraining the object detection and tracking module 214.

[0044] In some embodiments, the augmentation performed on the segmented images by the pairing and classification module 218 includes the generation of training and validation images for training and validation. For example, after flipping and rotation, 3120 training images and 634 validation images for both THz and RGB as shown in FIG. 3E may be generated to train the hybrid model.

[0045] In some embodiments, there comprises 256 testing images in total, among which 81% are correctly predicted, as shown in the fl-score column under the accuracy row of FIG. 4B. The accuracy increases to 0.84 (84 %) and 0.93 (93 %) in validation and training dataset. [0046] It is contemplated that other computer vision/ image processing techniques known to a skilled person may be combined/supplemented to form further embodiments to supplement and/or replace the ML/AI algorithms. For example, instead of a one-stage feature detector, a multi-stage feature detector may be envisaged.

[0047] In some embodiments, the processor 210 may include hardware components such as server computer(s) arranged in a distributed or non-distributed configuration to implement the modules 212, 214, 216, 218, 220. The hardware components may be supplemented by a database management system configured to compile one or more industry-specific characteristic databases. In some embodiments, the industry-specific characteristic databases may include analysis modules to correlate one or more dataset with an industry. Such analysis modules may include an expert rule database, a fuzzy logic system, or any other artificial intelligence module. [0048] FIG. 6 shows a server computer system 600 according to an embodiment. The server computer system 600 includes a communication interface 602 (e.g. configured to receive captured images from the image capturing device(s) 206). The server computer 600 further includes a processing unit 604 and a memory 606. The memory 606 may be used by the processing unit 604 to store, for example, data to be processed, such as data associated with the captured images and intermediate results output from the modules 212, 214, 216, 218 and/or final results output from the module 220. The server computer is configured to perform the method of FIG. 1. It should be noted that the server computer system 600 can be a distributed system including a plurality of computers.

[0049] In some embodiments, a computer-readable medium is provided including program instructions, which, when executed by one or more processors, cause the one or more processors to perform the methods according to the embodiments described above. The computer-readable medium may include a non-transitory computer-readable medium.

[0050] In some embodiments, the ML/AI algorithms may include algorithms such as neural networks, fuzzy logic, evolutionary algorithms etc.

[0051] It is to be appreciated that the use of electromagnetic radiation operating in the THz frequency range (THz radiation) to generate heatmap images is a complement to the RGB images, and vice versa. Heatmap images generated using the THz radiation source are able to capture salient image data relating to translation, vibration, and rotation according to molecular structure due to the sensitivity of the THz radiation, thereby facilitating molecular spectroscopy in materials investigation. However, such sensitivity of the THz radiation may unduly affect the resultant heatmap image for an object when the object is subject to dimension changes. For example, dimensional changes made to the same object, including, but not limited to, changes in shape(s), size(s), volume(s), thickness(es) of layer or wall(s), curvature(s), form(s) of an object result in heterogenous dimension differences which may unduly affect how the object looks like in the heatmap view. Some of such dimension changes may be caused by external stresses/forces applied to the object, such as distortion, compression, deformation, which result in various states of the objects such as flattened state, deformed state, compressed state, crushed state, torn state, etc. of objects, which can in turn affect the prediction of object class. As these states may not be obvious in RGB camera, the image data obtained from RGB images may be paired and combined with the image data obtained from the heatmap images generated by the THz radiation source to achieve a higher predicted precision/accuracy.

[0052] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.