BACKGROUND

Field

[0001] The present disclosure is generally directed to multi-modal sensing data system.

Related Art

[0002] A surge in demand of the curbside space in urban areas has allowed for digital transformation of curbside from a parking spot into an extensively monitored dynamic economic hub. A large city’s curb space can run to several miles and may be used for parking, commercial loading, last-mile docking, mobility, drive-through, and/or retail services. This transformation has propelled the use of a variety of sensors to parse information about congestion, occupancy, usage, and above all safety and security of this public zone.

[0003] The number of cameras and/or sensors for observing a large area (e.g., a city) is large and may include multiple imaging and/or sensing modalities (e.g., Night vision, light detection and ranging (LiDAR), and radio detection and ranging (RADAR)), observing multiple locations. This sensor fusion or multi-modal system adds robustness to existing field-of-view (FoV) coverage at the cost of high bandwidth. Such data streams pose a challenge in terms of analytics, variety, volume, and velocity considering city-scale sensing infrastructure.

[0004] FIG. 1 illustrates an example architecture of a current data analysis system. Current systems as shown in FIG. 1, a set of sensors in a sensor network 110 may transfer images 115 and/or their features over a network to a feature database server 120 on a private network that maintains the images 115 and/or their features for analysis and/or searching. Such a feature database server 120 is typically queried, e.g., by a client computing device 130, based on a set of image data and/or query features 125 transmitted over an intervening network for similarity and/or retrieval using a nearest-neighbor- and/or neural-network-based matching algorithm that may be storage- and compute-intensive. The network load and query complexity are quite high for a low fidelity verification, membership, or association query in a large information management system. [0005] In prior art, an image-based query, retrieval, or matching task transfers the image data (e.g., the set of image data and/or query features 125) over the network to image databases (e.g., feature database server 120). This retrieval query can be done by nearest neighbor or a deep neural network to compute similarity of image features. This causes large loads on the network by transferring large numbers of image files and on the compute infrastructure that processes each received image and many stored images to identify similar images for even a low fidelity membership or count queries.

[0006] To solve the problems associated with transmitting large numbers of images across a network and performing extensive processing at a central server or database, presented herein is a system that (1) enables a city-wide fast membership or information existence check, (2) reduces network bandwidth, and (3) identifies associations between image, point-cloud, and event-sensing modalities.

SUMMARY

[0007] Example implementations described herein include an innovative method for multimodal perceptual hash-based indexing of data for fast membership verification queries. The method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The method may further include storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.

[0008] Example implementations described herein include an innovative computer-readable medium storing computer executable code for multi-modal perceptual hash-based indexing of data for fast membership verification queries. The computer executable code may include instructions for receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The computer executable code may also include instructions for computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The computer executable code may further include instructions for storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.

[0009] Example implementations described herein include an innovative apparatus for multimodal perceptual hash-based indexing of data for fast membership verification queries. The apparatus may include a memory and at least one processor configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The at least one processor may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The at least one processor may further be configured to store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.

[0010] Example implementations described herein include an innovative method for an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries. The method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The method may further include transmitting, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.

[0011] Example implementations described herein include an innovative computer-readable medium storing computer executable code for an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries. The computer executable code may include instructions for receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The computer executable code may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The computer executable code may further include transmitting, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes. [0012] Example implementations described herein include an innovative apparatus that is an edge computing device for multi-modal perceptual hash-based indexing of data for fast queries. The apparatus may include a memory and at least one processor configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The at least one processor may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The at least one processor may further be configured to transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes.

[0013] Example implementations described herein include an innovative method for one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries. The method may include receiving, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. The method may also include storing the received plurality of hash indexes in a probabilistic data structure. The method may further include receiving, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index. The method may additionally include verifying an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.

[0014] Example implementations described herein include an innovative computer-readable medium storing computer executable code for one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries. The computer executable code may include instructions for receiving, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. The computer executable code may also include instructions for storing the received plurality of hash indexes in a probabilistic data structure. The computer executable code may further include instructions for receiving, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index. The computer executable code may additionally include instructions for verifying an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.

[0015] Example implementations described herein include an innovative apparatus that is one of a centralized server or a distributed server for multi-modal perceptual hash-based indexing of data for fast queries. The apparatus may include a memory and at least one processor configured to receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. The at least one processor may also be configured to store the received plurality of hash indexes in a probabilistic data structure. The at least one processor may further be configured to receive, from a client computing device, one of a membership query or an association query, the membership query or the association query including at least one hash index. The at least one processor may additionally be configured to verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query.

BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. 1 illustrates an example architecture of a current data analysis system.

[0017] FIG. 2 is a diagram illustrating a high-level view of a system for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0018] FIG. 3 is a diagram illustrating elements of a system for multi-modal perceptual hashbased indexing of data for fast membership verification queries.

[0019] FIG. 4 is a diagram illustrating a set of components in an example multi-modal deep hash composer, in accordance with some aspects of the disclosure.

[0020] FIG. 5 is a diagram illustrating an example process for processing data captured in accordance with some aspects disclosed herein.

[0021] FIG. 6 is a diagram illustrating a set of calculated indexes being transmitted to, and received by, a distributed multi-modal hash registry. [0022] FIG. 7 is a diagram illustrating a process for verifying occupancy and/or a membership in the distributed hash index registry.

[0023] FIG. 8 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

|0024| FIG. 9 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0025 [ FIG. 10 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0026] FIG. 11 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0027] FIG. 12 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0028] FIG. 13 is a flow diagram illustrating a method for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0029] FIG. 14 illustrates an example computing environment with an example computer device suitable for use in some example implementations.

DETAILED DESCRIPTION

[0030] The following detailed description provides details of the FIGs. and example implementations of the present application. Reference numerals and descriptions of redundant elements between FIGs. are omitted for clarity. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic” may involve fully automatic or semi-automatic implementations involving user or administrator control over certain aspects of the implementation, depending on the desired implementation of one of the ordinary skills in the art practicing implementations of the present application. Selection can be conducted by a user through a user interface or other input means, or can be implemented through a desired algorithm. Example implementations as described herein can be utilized either singularly or in combination and the functionality of the example implementations can be implemented through any means according to the desired implementations.

[0031] Example implementations described herein relate to an innovative method for multimodal perceptual hash-based indexing of data for fast membership verification queries. The method may include receiving a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The method may also include computing a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The method may further include storing information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries.

[0032] In some aspects, the presented invention exchanges the hashed index rather than the dense data frames from sensor(s) data stream and persists it to (e.g., stores it in) a distributed optimized data structures on an index registry. This index registry is made available for membership query based on a shared set of hash functions.

[0033] The discussion below relates to data management for information verification in a cityscale scenario (e.g., for a city like New York or London with approximately 1.5 Million closed circuit television (CCTV) sensors). The discussion specifically relates to (1) sensor fusion for multiple modalities of sensors deployed at a city-scale (e.g., fusing data received from a sensor network including two or more types of sensors such as CCTV, IR/Night Camera, RADAR, and LiDAR), (2) imaging data types captured by sensors in the sensor network such as an RGB image, a point cloud, or error codes and reports, (3) distributed infrastructure including, e.g., an edge device, a gateway system, cloud nodes, a private network, and TCP/HTTP communication systems. One aspect disclosed herein relates to a multi-modal sensor network distributed indexing and data membership verification for curbside monitoring system.

[0034] In some aspects, more than one billion images may be associated with a city-scale membership or association query. Accordingly, a membership verification algorithm with quasi-linear complexity, e.g., O(nlogn), or sub-linear complexity, e.g., O(logn)is desired. A membership verification algorithm may be used to check if particular information (e.g., parking occupancy, bike dock occupancy, curbside loading zone usage) exists, is valid, or to retrieve statistics such as a count associated with the particular information from one or more distributed index and probabilistic data structures. The presented query can be based on a visual data type or an associated data type like text to determine membership and/or association in one or more of a uni-modal and/or cross-modal sensor fusion space.

[0035] In some aspects, low fidelity perceptual hashes are used to index information in a large distributed data system. Using the low-fidelity perceptual hashes, in some aspects, involves propagating only the hashed index from multi-modal sensor(s) data stream and persisting it to one or more search-optimized data structures on a distributed registry and metadata store. In some aspects, an index may contain low fidelity information across data types and is referred to as cross-modal, multi-modal, or sensor-fusion indexes.

[0036] The data system described herein, in some aspects, benefits from two core technologies, (a) sensor fusion or multi-modal distributed sensor(s) data indexing and registry that relies on a family of perceptual deep hash composer functions that produce similar results if features in the source material are similar and (b) fast multi-modal distributed verification for membership or an association test across modalities & distributed data index on the deployment network. In some aspects, a trade-off is made between dense image feature transfer over the network and low-latency m-bit binary coded perceptual hashes to enable fast membership checks over a large distributed city-scale sensor network. The verification query, based on the low latency m-bit binary coded perceptual hashes, may be capable of fast membership tests in a large distributed index registry. In some aspects, there are four components that enable this workflow, (1) a distributed multi-modal deep hash composer, (2) a set of perceptual hashes and fusion hashes, (3) a multi-modal hash index registry, and (4) a membership verification and/or an association module. For example, a system in accordance with some aspects of the disclosure may (1) enable a city-wide fast membership or information existence check due to sub-linear query complexity in probabilistic data structure, (2) reduce network bandwidth by leveraging perceptual hash code to represent and transmit sensing data, and (3) identify associations between image, point-cloud, and event-sensing modalities by using hash composition and metadata.

[0037] FIG. 2 is a diagram 200 illustrating a high-level view of a system for multi-modal perceptual hash-based indexing of data for fast membership verification queries. Diagram 200 illustrates that, as opposed to the system described in FIG. 1, the system for multi-modal perceptual hash-based indexing of data for fast membership verification queries may include a local multi-modal sensor network 210 including a set of sensors 250 that produce sensor data sets 255 and an edge computing device 270 that produce hash indexes 215. The system may further include a multi-modal index registry 220 that interacts with at least one client computing device 241 in a verification-client local network 240.

[0038] The local multi-modal sensor network 210 may include a set of sensors, e.g., sensor 250 and sensor 260, that may each capture one of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, the sensor 250 or the sensor 260 of the local multi-modal sensor network 210 may include one or more of a CCTV, an IR/Night Camera, a RADAR, or a LiDAR Sensor Network which may generate imaging data types such as an RGB image, a point cloud, an error code, and/or reports. The local multi-modal sensor network 210 may monitor a monitored zone 280 that includes, e.g., a set of parking spaces 281, 282, and 283.

[0039] The local multi-modal sensor network 210 may also include an edge computing device 270 that receives sensor data sets from the sensors, e.g., sensors 250 and 260 in the local multimodal sensor network 210. For example, the edge computing device 270 may receive sensor data sets 255 (e.g., an image data set, a point cloud data set, a processed information data set, or an inferred data set) over a local network 265. The edge computing device 270 may compute a hash index for each received sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The edge computing device 270 may also compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. The edge computing device 270 may transmit the computed indexes, e.g., as indexes 215, to the multi-modal index registry 220.

[0040] The multi-modal index registry 220 may store the transmitted indexes for fast membership verification queries. The multi-modal index registry 220, in some aspects, may be a distributed index registry, while in other aspects, the multi-modal index registry 220 may be a centralized index registry or a combination of distributed and centralized components. The index may be stored in a probabilistic data structure such as a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. The multi-modal index registry may also store a set of perceptual hash functions that may be retrieved by the local multi-modal sensor network 210 , e.g., an edge computing device 270, and applied to received sensor data sets. Additionally, a client device may retrieve the set of perceptual hash functions in order to generate an index-based query. For example, the client device may be provided with a single- mode data set or a multi-modal data set and apply the retrieved perceptual hash functions to the provided data set(s) to generate and/or compute a set of indexes, e.g., indexes 225, for an index-based query that is then transmitted to the multi-modal index registry 220 to perform a search (a membership verification) based on the received set of indexes.

[0041] The multi-modal index registry 220, in some aspects, may also include a data processing module 221 for data ingestion from edge computing devices that process input sensor data sets (e.g., edge computing device 270 of the local multi-modal sensor network 210). The data processing module 221 may provide the processed data to a distributed data storage 223 of the multi-modal index registry 220. In some aspects, the multi-modal index registry 220 also includes a query-processing module 222 for interacting with an edge computing device 242 associated with a verification-client local network 240. The distributed multi-modal index registry 220 and the edge computing devices 270 and 242 may be considered a system 230 for multi-modal perceptual hash-based indexing of data for fast membership verification queries.

[0042] FIG. 3 is a diagram 300 illustrating elements of a system 330 for multi-modal perceptual hash-based indexing of data for fast membership verification queries. The diagram 300 illustrates includes a set of client-side applications 310 that may access a distributed data storage 340 including a distributed multi-modal hash index registry 341 and a metadata registry 342 stored in a set of databases 343 of the system 330 through interface(s) 331. The set of client-side applications 310 may include a verification module 311 and a set of data analytics 312. The verification module 311 may provide an image or data set to the system via the interface(s) 331. The interface(s) 331 may include one or more edge computing devices that include and/or implement a query generator 360. In some aspects, for a verification operation, the query generator 360 may receive an image or other data set for the verification operation and may generate a membership query 365. For example, a deep hash composer 362 may calculate a set of hash indexes for inclusion in a hash exchange format. The routing and evaluation module 361 may generate the membership query 365 including the set of hash indexes generated by the deep hash composer 362.

(0043] A set of sensor-side applications 320 may include a sensor network 321 a set of external systems 322 and a set of external data stores 323 (e.g., local data stores). The sensor-side applications 320 may provide data to the system 330, and more specifically to the distributed data storage 340, via the interface(s) 332. The interface(s) 332, in some aspects, may include one or more edge computing devices that include and/or implement a distributed multi-modal deep hash composer 350 in a same local network as one or more sensor devices in the sensor network 321.

[0044] In some aspects, a distributed multi-modal deep hash composer 350, may be distributed and shared across the system 330 for client-side and/or sensor-side (which may collectively be referred to as client-side) index generation using perceptual hashing. The distributed multimodal deep hash composer 350 uses ^-perceptual hash functions to encode low fidelity image information analogous to image or object features in an m-bit binary hash code. In some aspects, calculated uni-modal and cross-modal perceptual hashes (m-bit binary hash codes) may be embedded in an exchange format (a format of hash exchange and metadata 355) including metadata that is persisted and/or stored in an index registry, e.g., distributed multimodal hash index registry 341 or a metadata registry 342. For example, the distributed (across a network) data storage 340, in some aspects, enables low-latency membership queries.

[0045] The structure of the registry, in some aspects, includes a plurality of spatial, domain, and mode-specific databases that may use probabilistic data structures such as one or more scalable bloom filters, cuckoo filters, or other probabilistic data structures. For example, bloom and cuckoo filters, in some aspects, may be used to determine (with a given degree of certainty) whether an item is present or absent from a collection. The data included in the one or more probabilistic data structures may also be stored in, and/or transformed to, a tree structure for distribution, search, comparison, and version management.

[0046] In some aspects, diagram 300 may represent a deployment diagram for a multi-modal membership verification system (e.g., the system 330) that is capable of ingesting sensor data from a plurality of sensors (e.g., sensors in the sensor network 321) and code the frame data using the &-hash functions. The perceptual hash functions may be orchestrated by a replicated hash/filter escrow system 351 that may be managed remotely across the system 330. In some aspects, the hash functions run on client-side edge systems (e.g., edge computing devices) and share/transmit the binary code (e.g., in hash exchange and metadata 355) to the distributed data storage 340 including the distributed multi-modal hash index registry 341. The parameters (k and ni) and hash models and/or filters may be chosen and distributed across the device network.

[0047] A verification test system (e.g., as implemented using verification module 311) can present an image or event to check for membership and/or the presence of corresponding or associated information in the distributed data storage 340. Th verification test system, in some aspects, may be deployed as a service and may communicate with the distributed data storage 340 including the distributed multi-modal hash index registry 341 to perform membership checks. Verification, in some aspects, is conducted as a fast hash-based membership check and, once the membership and or the presence of corresponding data is verified, corresponding metadata may be retrieved for additional information. The system, in some aspects, provides the ability to provide membership verification at a city-scale and may be integrated as a layer on a data bus and other middleware platforms.

[0048] In some aspects, the system 330 may be targeted towards application of multi-modal perceptual hashes-based indexing of low-fidelity data for fast membership query in a large cityscale camera sensor network to monitor and manage curbside information. This can cater to short term parking monitoring and just-in-time existence check. Some aspects may incorporate versions to verify existence over a larger time frame such as for the case with long-term parking at facilities like airport and railway station. In some aspects, the system may reduce a need for manual intervention and may provide a large computational benefit in search or verification operations over a large infrastructure.

[0049] Some specific monitoring functions include but are not limited to monitoring any one of a vehicle parked in a parking area, a repeat violator vehicle on a commercial curbside, docked bikes, sensor data fingerprint for validity, equipment (e.g., a barrier, cone, etc.) in a curbside. The system may also be configured to retrieve associated night vision index based on daytime imaging data or the other way round.

[0050] Additional applications may include monitoring systems deployment by city or transit authorities for real-time zone regulation, people flow by video analytics for dynamic curbside management and productivity estimation. These applications may contribute to realizing new ROI (Return on Investment) for transforming a cityscape by effectively monitoring, regulating, and allocating access to urban curbside space. Some aspects may incorporate versions of the distributed data storage and sets of perceptual hash functions for monitoring long-term airport parking. The long-term airport parking monitoring, in some embodiments, may include incorporating temporal information in the structures and an ability to compute existence in those temporal or even geo-spatial segments. Additional aspects, may include a combination of image hash index and metadata to compute a metadata hashing for reducing collision and increasing parameter awareness, security assessments and audits for the sensor network and information quality, and/or blockchain extensions for single source of truth for various assessments and manipulations.

[0051] The discussion below focuses on an example workflow associated with a short-term curbside parking occupancy and/or a loading zone occupancy use-case. For example, the discussion below may relate to a system as illustrated in FIG. 2. The camera network setup, in some aspects, is such that one or more CCTV cameras constantly monitors at least one zone (e.g., monitored zone 280), where the at least one zone may include multiple parking spots (e.g., parking spaces 281-283). When a curbside user parks their vehicle in the at least one zone the event information may be captured into the system and the hashes index workflow is initiated. The query to this index registry can be used to test occupancy, membership, and/or an association in a distributed multi-modal sensor index management system.

[0052] FIG. 4 is a diagram 400 illustrating a set of components in an example multi-modal deep hash composer, in accordance with some aspects of the disclosure. In some aspects, the multi-modal deep hash composer may be included in an edge computing device, e.g., edge computing device 270 of FIG. 2. The multi-modal deep hash composer may be shared or distributed across the system to distribute algorithms, hash functions, and orchestrate its operations. The multi-modal deep hash composer, in some aspects, may be executed on an edge client or device and enable analytics on a hash-based index in our system.

[0053] The multi-modal deep hash composer, in some aspects, may interact with a hash escrow component (not shown) of a multi-modal index registry (e.g., the multi-modal index registry 220) that may perform identification, verification, orchestration, and generation beyond providing the sets of k perceptual hash functions (e.g., k-hash functions) for computing a set of one or more hash indexes from input data sets to the various client-side hashing modules that may be executed in an edge computing device (e.g., edge computing device 270). In some aspects, A hash functions can rotate based on escrow orchestration. A family of perceptual hash functions, in some aspects, include existing technologies like a uni-modal locality-sensitive perceptual hash generating a hash code for a particular mode for inclusion in a uni-mode sensor indexing structure (e.g., point structure 437), or cross-modal (or multi-modal) hash leveraging Deep Neural Network (DNN) that is capable of generating an n-wise cross-modal sensor fusion indexing structure (e.g., triplet structure 434 or pair structure 436). Such an n-wise cross-modal sensor fusion indexing structure can be used to encode pair, triplet, or n-mode sensor data into a common subspace. A device-optimized cross-modal neural network such as a vision- language model, in some aspects, may be used to generate perceptual hashes enabling cross- modal membership verification.

[0054] The multi-modal deep hash composer, in some aspects, may receive one or more datatypes (e.g., image, point cloud, text events) in sensor data 415 from a sensor network (e.g., the local multi-modal sensor network 210 of FIG. 2). The multi-modal deep hash composer may include a multi-modal data stream handler 420 that may evaluate a data format (or type), may register and index the received data for local storage and identify, or associate, metadata for, or with, the received sensor data 415. The multi-modal data stream handler, in some aspects may provide the processed data 425, including register and index data 425a and metadata 425b, to a multi-modal hash calculation module 430.

[0055] The multi-modal hash calculation module 430, in some aspects, may retrieve sets of uni-modal hash functions (sets of k perceptual hash functions) corresponding to data types received from hash function escrow 440. In some aspects, the multi-modal hash calculation module 430, in some aspects, may retrieve sets of cross-modal hash functions (sets of k' cross- modal perceptual hash functions) corresponding to combinations of data types received from the hash function escrow 440. The sets of uni-modal and cross-modal hash functions, in some aspects, may be associated with a mode aware &-hash models module 431 (e.g., implemented as a DNN). The multi-modal hash calculation module 430, in some aspects, may include a unimodel hash calculation module 432 that calculates the uni-modal hashes based on the set of uni-modal hash functions for a particular type of received data. The multi-modal hash calculation module 430 may also include, in some aspects, a cross-modal hash calculation module 433 that calculates one or more cross-modal hashes associated with one or more combinations of data sets. In some aspects, the cross-modal hash calculation module 433 may calculate the cross-modal hashes based on calculated hashes from the uni-model hash calculation module 432 and/or processed data sets from the multi-modal data stream handler 420.

[0056] The multi-modal hash calculation module 430 may store the calculated hashes in one or more data structures such as point structure 437 (e.g., for storing uni-modal hash indexes), triplet structure 434, and/or pair structure 436. The multi-modal deep hash composer, e.g., via a registry encoder 450, may generate a data set 455 for transmission over a network to a multimodal hash index registry. For example, in an aspect using a bloom filter probabilistic data structure a vector or binary code of m-bits per image data frame for each perceptual hash function may be generated. The binary code of m bits may initially be set to 0, for each of a set of k independent perceptual hash functions (e.g., [/i ₁₍ h ₂ , ..., h _k ]). For each element x G X , the bits at positions h ₂ x ₁ ), .... h^x^] in the vectors associated with the set of k perceptual hash functions [/i ₁₍ h ₂ , ... , h _k ], respectively, may be set to 1. In some aspects, the binary code of m bits is associated with the set of k perceptual hash functions such that for each perceptual hash h _L of the k perceptual hashes are associated with a set of bits in the m bits. The generated data set 455 may be formatted in an exchange format that, in some aspects, includes one or more of a registry target 455a (as an example of registry data), metadata 455b (e.g., including parameters (m, n, k) indicating a size ‘m’ of a hash index, a number ‘n’ of data types, and a number ‘k’ of hashes used in the hash index calculation), a security watermark (e.g., watermark fingerprint 455c), multi-modal hash exchange data 455d, an index structure identification in a common standard persisted on a distributed index and metadata registry server (e.g., multi-modal hash index registry 220).

[0057] FIG. 5 is a diagram 500 illustrating an example process for processing data captured in accordance with some aspects disclosed herein. For example, the process may be applied to a short-term curbside parking monitoring with low-latency just-in-time information management at a city-scale with multiple observation areas in each camera coverage zone. The data processing may be used to verify occupancy or membership of a query image based on the hash index.

[0058] In some aspects, a set of sensors 510 may capture a set of N data points 515 (e.g., data sets from ‘N’ sensors). The sets of data points may be associated with a same time period and a same or overlapping set of areas. For example, the set of N data points 515 may include image data from a CCTV and a night vision and/or IR camera. The captured sets of data points may be stored locally based on metadata associated with a captured data point (e.g., a time associated with the data point, a coverage area of the data point, one or more zones associated with the data point, a data point type, and/or a capture device associated with the data point).

(0059) A detector component 520, in some aspects, performs a zone-based object and/or event detection and registration test. In some aspects, the detector component 520 may perform the zone-based object and/or event detection and registration test frequently to reduce redundant downstream operations on image. A deep hash composer 530, in some aspects, may apply a hash family (e.g., a set of perceptual hashes) on single-mode data to generate a uni-modal hash index 555. The deep hash composer 530 may also, in some aspects, create a pairwise (n=2) hash composition for generating a cross-modal hash 545. In some aspects, the deep hash composer 530 may flatten one or more of the uni -modal hash index and the cross-modal hash index to an m-bit binary code. For example, a 32-bit binary code (m=32) can address a 4-GB address space and, with k > 3 perceptual hash functions, a likelihood of collisions may be reduced while preserving the feature similarity. The deep hash composer 530, or an edge device including the deep hash composer 530, may transmit (or distribute) the calculated hash indexes to a distributed multi-modal hash registry.

[0060] FIG. 6 is a diagram 600 illustrating a set of calculated indexes being transmitted to, and received by, a distributed multi-modal hash registry. In some aspects, the set of calculated indexes may be one of a plurality of sets of calculated indexes (e.g., a multi-modal hash index 615) for a plurality of different data sets (e.g., sets of captured data associated with different times or different coverage areas) that may be formatted in a standardized exchange format. The set(s) of calculated indexes, in some aspects, may be transmitted in accordance with a batching operation. The batching operation may collect sets of calculated hash indexes until some threshold is met. The threshold, in some aspects, may be one or more of a threshold time (for periodic transmission) and/or a threshold amount of data for transmission. For example, every few minutes an edge computing device may share an index (or set of calculated hash indexes) with hash exchange information and metadata about a set of observed parking spots (or coverage areas).

[00611 The transmitted, and received, set(s) of calculated hash indexes and any additional data received from an edge computing device, in some aspects, may be persisted in a probabilistic data structure and distributed so as to be available for responding to a query. A distributed multi-modal hash registry may allocate and register the received set(s) of calculated hash indexes such that the hash indexes in the set(s) of calculated hash indexes are added to the distributed multi-modal hash registry if not already included in the distributed multi-modal hash registry or may be used to update an existing version of a corresponding (or equivalent) hash index stored in the distributed multi-modal hash registry.

[0062] The distributed multi-modal hash registry 620, in some aspects, may store the set(s) of calculated hash indexes in multiple different data structures and/or formats. For example, in some aspects, each a uni -modal hash index 615a and a cross-modal hash index 615b may be stored in a linear index registry 621 of the distributed multi-modal hash registry 620. The linear index registry 621 may be configured as a cuckoo or counting bloom filter structure used for a membership and/or verification query (or test). A linear index registry 621, in some aspects, may be persisted in a probabilistic data structure such as a bloom filter or a cuckoo filter which may be used to verify membership and/or determine whether a given index exists in a set. In another aspect of the probabilistic data structure based linear registry, a count-min-sketch data structure or a Top-k data structure may identify a frequency of events and/or matching data in a data sample. The membership and/or verification query may be associated with a configured matching criteria or threshold. For example, for a uni-modal match, the matching criteria may specify that each bit, or a threshold number that is less than the number of bits in a uni-modal hash index, match a set of bits in a hash index-based query. Similarly, for a cross-modal association, a threshold number of bits that may be different from, or the same as, the threshold number of bits for the uni -modal hash index match criteria, may be configured to determine an association between a uni-modal hash index-based query or a cross-modal hash index-based query with stored information in the linear index registry 621.

[0063] A tree-based index registry 622 may be created to allow for quick search and comparison of an index, metadata, and its versions (e.g., snapshots of a linear index registry 621 for different time periods). The tree-based index registry 622 may include pointers to a metadata registry 623 that may be used to store metadata associated with the data included in the tree-based index registry 622. The metadata registry 623 may include data relating to a registration, an event, a sensor location, a zone identifier, a hash family (e.g., a set of k hash functions), or other information relating to data (e.g., multi-modal hash index 615) received from an edge computing device. A hash index distribution within the distributed multi-modal hash registry 620 may be done based on a geo-spatial sensor network, observation time, and registry type (e.g., a data like zonal parking occupancy). The various stores may be available for membership information query.

[0064] FIG. 7 is a diagram 700 illustrating a process for verifying occupancy (e.g., of a parking spot) and/or a membership in the distributed hash index registry. For example, for a low latency check for membership of a car image across an indexed parked-car-information registry the system may verify membership information to verify if a presented car image (F) exists in a short-term parking occupancy registry. Given a query for the image 715 (e.g., the image Y) received from a client computing device, a deep hash composer 720 at an edge computing device may calculate a uni-modal hash index based on a same family of k perceptual hash functions 721, e.g., [/i ₁₍ h ₂ , — , h _k ], as used to process captured data from a sensor network. The family of hash of k perceptual hash functions 721, in some aspects, may be received, or retrieved, from a distributed hash escrow that provides families of perceptual hash functions to edge computing devices throughout the system.

[0065] The deep hash composer 720 may also generate an associated query. The generated query may include or indicate a set of bits at positions [^(K), h ₂ (K), ..., h _k (Y ] and, in some aspects, may indicate a particular hashed index registry (e.g., a hashed index registry maintained for a particular application). The generated query may be passed to a routing & evaluation module 730 for evaluation. The routing & evaluation module 730 may be responsible for mapping, routing, and executing the query (e.g., performing a membership evaluation operation) on an identified hashed index registry in the distributed index registry server 740. Executing the query, in some aspects, may include identifying at 731 whether bits in the identified hashed index registry at positions [h ₁ (K), h ₂ (Y), ... , i _k (K)] are each evaluated to be 1. If any of the bits at positions [h ₁ Y), h ₂ Y), ..., h _k (Y ] are evaluated to be 0, in some aspects, the image associated with the query may be determined to not match an image captured by the sensor network and stored in the distributed hash registry. If all of the bits at the positions [hi(K), /I ₂ (K), ... , h _k (Yy] are evaluated to be 1, the image associated with the query may be determined to likely match an image captured by the sensor network and stored in the distributed hash registry. The determination, in some aspects, may be erroneous based on the possibility and/or the probability of a hash collision. The parameters k and m, in some aspects, may be chosen such that the probability of a false positive (and hence a false hit) is calculated to be below a threshold (acceptable) probability. For example, a check for membership may be applied to a set of indices in a zonal parking occupancy registry.

[0066] The deep hash composer 720 may also apply a set of cross-modal hash functions to create a pairwise hash composition for cross-modal hash index evaluation operation. The deep hash composer 720 may also select the type of registry a query in the distributed index registry server 740 should target based on a mode and data type associated with the query regarding image 715 and the type of analysis for each type of registry (e.g., identification 731 or a check 732). In some aspects, a verification and/or membership query may include a check 732 for an association between the image 715 (i . e. , a first type of captured data) and data captured by other types of sensors in the distributed index registry server 740. For example, a query related to a daytime image may additionally include a check 732 for an association with another type of captured data (e.g., a nighttime image or a point cloud) in a cross-modal registry. The results 745 (e.g., a status or an index) of the identification 731 and the check 732 may then be provided to the client computing device from which the query regarding image 715 was received.

[0067] In some aspects, a parking system may, continuously, periodically, or on an event-based basis, observe occupation of a target zone (e.g., monitored zone 280 including the set of parking spaces 281-283). Various computer vision models may be used to track, detect, extract a region of interest, and encode features for uni-modal or cross-modal hash index generation. A multimodal hash registry, in some aspects, maintains multi-modal indices for each object that occupies a set of observed regions. Based on the maintained multi-modal indices, a specific image verification operation may be performed over the occupancy membership registry for the target zone or a set of target zones in a region of interest (e.g., a city, zip code, block, etc.). Performing the specific image verification operation may reduce an overhead associated with frequently looking for parked vehicles in a specific locale or tracking a specific vehicle continuously. The specific image verification operation may also include an association check to determine if alternate modality sensing data exists. It can serve as a security verification, fingerprinting, or validation for image frames that are served by the sensor network. This invention can be applied for consolidated parking revenue collection in curbside management scenario.

[0068] FIG. 8 is a flow diagram 800 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270). At 810, the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2, 3, and 4, a system 230 or 330 (or an edge computing device 270 or an interface 332) may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0069] At 820, the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions. The set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions. For example, referring to FIGs. 3-5, the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530).

[0070] The system may further compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. For example, referring to FIGs. 3-5, the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433. The cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing. The hash indexes may be computed at an edge computing device. After computing the hash indexes, the computed hash indexes may be transmitted to a centralized data storage or a distributed data storage.

[0071] At 830, the system may store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries. For example, referring to FIGs. 3 and 6, the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623). The stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target. In some aspects, storing the information includes storing additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.

[0072] In some aspects, the system may also receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set as described in relation to FIGs. 3 and 7. Based on receiving the query, the system may perform a search of the probabilistic data structure based on the query-related hash index and may transmit a result of the search in response to the received query. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIG. 7, the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.

[0073] FIG. 9 is a flow diagram 900 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270). At 910, the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2, 3, and 4, a system 230 or 330 (or an edge computing device 270 or an interface 332) may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0074] At 920, the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions. The set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions. For example, referring to FIGs. 3-5, the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530). [0075] At 930, the system may compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. For example, referring to FIGs. 3-5, the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433. The cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing. The hash indexes may be computed at an edge computing device.

[0076] After computing the hash indexes, the computed hash indexes may be transmitted, at 940, to a centralized data storage or a distributed data storage. The transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes. The information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage. For example, referring to FIGs. 3-6, a distributed multi-modal deep hash composer 350, a multi-modal hash calculation module 430, or a deep hash composer 530, may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.

[0077] At 950, the system may store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries. For example, referring to FIGs. 3 and 6, the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623). The stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.

[0078] In some aspects, the system may store, at 960, additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database. The additional information, in some aspects, may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry. For example, referring to FIGs. 3 and 6, the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623.

[0079] At 970, the system may receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set. The set of perceptual hash functions used to generate the query, in some aspects, may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage. In some aspects, the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs. 3 and 7, the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.

[0080] At 980, the system may perform a search of the probabilistic data structure based on the query-related hash index. In some aspects, the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index. The probabilistic data structure, in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. For example, referring to FIGs. 3 and 7, the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index.

[0081 ] Finally, at 990, the system may transmit a result of the search in response to the received query. The result may include a membership existence status, a storage location associated with one of the sensor data identified by the search, or a storage location of metadata associated with the sensor data. For example, referring to FIG. 7, the routing & evaluation module 730 may receive a query-related hash index generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.

[0082] FIG. 10 is a flow diagram 1000 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by an edge computing device (e.g., edge computing device 270). At 1010, the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2, 3, and 4, an edge computing device of the system 230 or 330 (e.g., the edge computing device 270 or an edge computing device implementing interface(s) 332) may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0083] At 1020, the system may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions. The set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be retrieved from a centralized hash escrow or a distributed hash escrow that is an orchestrator for the families of k perceptual hash functions. For example, referring to FIGs. 3-5, the system 330 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530).

[0084] The system may further compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. For example, referring to FIGs. 3-5, the system 330 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433. The cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing. The hash indexes may be computed at an edge computing device. After computing the hash indexes, the computed hash indexes may be transmitted to a centralized data storage or a distributed data storage.

[0085] At 1030, the edge computing device may transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes. The transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes. The information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage. For example, referring to FIGs. 3-6, a distributed multi-modal deep hash composer 350, a multi-modal hash calculation module 430, or a deep hash composer 530, may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.

[0086] The system may store the transmitted information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries. For example, referring to FIGs. 3 and 6, the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623). The stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target. In some aspects, storing the information includes storing additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database.

[0087] In some aspects, the system may also receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set as described in relation to FIGs. 3 and 7. Based on receiving the query, the system may perform a search of the probabilistic data structure based on the query-related hash index and may transmit a result of the search in response to the received query. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIG. 7, the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query.

[0088] FIG. 11 is a flow diagram 1100 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by an edge computing device (e.g., edge computing device 270). At 1110, the system may receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2, 3, and 4 a system 230 or 330 (or an edge computing device 270 or an interface 332) may receive sensor data sets for the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0089] At 1120, the edge computing device may retrieve the set of perceptual hash functions for the plurality of types of sensors from a one of a centralized hash escrow or a distributed hash escrow. In some aspects, the centralized hash escrow server or the distributed hash escrow is an orchestrator for a family of k hash functions. The centralized or distributed hash escrow, in some aspects, may ensure that the same set of k perceptual hash functions are being used across the system so that the hash indexes produced based on sensor data and hash indexes produced based on test data are consistent (e.g., produce a same, or similar, hash for the same, or similar, image feature). For example, referring to FIGs. 2-4, the edge computing device 270, an edge computing device implementing distributed multi-modal deep hash composer 350, or an edge computing device implementing multi-modal hash calculation module 430 may retrieve, from a hash function escrow 440, one or more sets of perceptual hash functions to populate the replicated hash/filter escrow system 351.

[0090] At 1130, the edge computing device may compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The hash index may be a hash index in an m-bit binary structure (e.g., for inclusion in a probabilistic data structure) based on a family of k perceptual hash functions. The set of perceptual hash functions (e.g., for each type of sensor) in some aspects, may be a set of perceptual hashes retrieved, at 1120, from the centralized hash escrow or the distributed hash escrow that is an orchestrator for the families of k perceptual hash functions. For example, referring to FIGs. 3-5, an edge computing device implementing distributed multi-modal deep hash composer 350, an edge computing device implementing multi-modal hash calculation module 430, or an edge computing device implementing deep hash composer 530 may compute a hash index for each particular sensor data set at the distributed multi-modal deep hash composer 350, multi-modal hash calculation module 430, or deep hash composer 530, respectively.

[0091 ] At 1140, the edge computing device may compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. For example, referring to FIGs. 3-5, an edge computing device implementing distributed multi-modal deep hash composer 350, an edge computing device implementing multi-modal hash calculation module 430, or an edge computing device implementing deep hash composer 530 may compute a cross-modal hash 434, 436 or 545, at a cross-modal hashing module 433. The cross-modal hashing may include one or more of pair-wise hashing or n-wise hashing.

[0092] At 1150, the edge computing device may transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes. The transmitted information regarding the computed set of hash indexes may include the computed set(s) of hash indexes. The information may be computed via an intervening network from an edge computing device of the system to a centralized data storage or a distributed data storage. For example, referring to FIGs. 3-6, a distributed multi-modal deep hash composer 350, a multi-modal hash calculation module 430, or a deep hash composer 530, may transmit a calculated hash index (or set of hash indexes) to a distributed data storage 340 or a distributed multi-modal hash registry 620.

[0093] At 1160, the edge computing device may store, at least in part, the received plurality of sensor data sets in a local data storage. For example, referring to FIGs. 3 and 6, an edge computing device may store the received plurality of sensor data sets in an external data store 323 (e.g., a local data store). In some aspects, storing, at least in part, the raw data locally and transmitting the hash indexes allows for a search for matching data to be performed quickly and efficiently at a central (or distributed) server while maintaining the accessibility of the sensor data sets based on the results of the search for matching data. For example, the system may store the transmitted hash index data and associated metadata in a distributed data storage 340, e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342 or a distributed multi-modal hash registry 620, e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623. The stored information regarding the computed set of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.

|0094| At 1170, the edge computing device may receive a request for at least one particular sensor data set of the received plurality of sensor data sets stored in the local data storage based on a query received at one of the centralized or the distributed server. The request may be based on a storage location identified in the result of a search based on the query received at one of the centralized or the distributed server. The set of perceptual hash functions used to generate the query (e.g., at a client-side edge computing device), in some aspects, may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage. In some aspects, the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs. 3 and 7, an edge computing device in the system 330 may receive an indication (e.g., in a set of results 745) of a storage location in an external data store 323.

|0095| Finally, at 1180, the edge computing device may transmit the requested at least one particular sensor data set to one of the centralized server or the distributed server. For example, referring to FIGs. 3 and 7, an edge computing device in the system 330 may retrieve the requested at least one particular sensor data and transmit the requested at least one particular sensor data. [0096] FIG. 12 is a flow diagram 1200 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270). At 1210, the system may receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2-5, a system 230 or 330 may receive a plurality of hash indexes generated by a distributed multimodal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530) based on sensor data sets collected by the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0097] At 1220, the system may store the received plurality of hash indexes in a probabilistic data structure. For example, referring to FIGs. 3 and 6, the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623). The stored plurality of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.

[0098] In some aspects, the system may store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database. The additional information, in some aspects, may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry. For example, referring to FIGs. 3 and 6, the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623. [0099] At 1230, the system may receive, from a client computing device, one of a membership query or an association query. In some aspects, the membership query or the association query may include at least one hash index. The set of perceptual hash functions used to generate the query, in some aspects, may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage. In some aspects, the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs. 3 and 7, the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.

[0100] At 1240, the system may verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query. In some aspects, the verification includes performing a search of the probabilistic data structure based on the query-related hash index. In some aspects, the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index. The probabilistic data structure, in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. For example, referring to FIGs. 3 and 7, the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index to verify and/or determine whether matching sensor data set or an associated sensor data set is stored in a searched distributed data storage. For example, referring to FIG. 7, the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query. [0101 ] FIG. 13 is a flow diagram 1300 illustrating a method for multi-modal perceptual hashbased indexing of data for fast membership verification queries. The method may be performed by a system, and more specifically an edge computing device (e.g., system 230, 330; edge computing device 270). At 1310, the system may receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. In some aspects, the plurality of sensor data sets includes at least two of an image data set, a point cloud data set, a processed information, or an inferred data set. For example, referring to FIGs. 2-5, a system 230 or 330 may receive a plurality of hash indexes generated by a distributed multimodal deep hash composer 350 (or one of multi-modal hash calculation module 430 or deep hash composer 530) based on sensor data sets collected by the local multi-modal sensor network 210, the sensor network 321, or the same network 410. In some aspects, the plurality of sensor data sets are associated with a same time period and a same area of interest. The plurality of sensor data sets, in some aspects, is one of multiple pluralities of sensor data sets for multiple areas of interest and/or for multiple time periods.

[0102] At 1320, the system may store the received plurality of hash indexes in a probabilistic data structure. For example, referring to FIGs. 3 and 6, the system 330 may store hash exchange and metadata 355 in a distributed data storage 340 (e.g., in distributed multi-modal hash index registry 341 and/or metadata registry 342) or a distributed multi-modal hash registry 620 (e.g., in one or more of the linear index registry 621, the tree-based index registry 622 or the metadata registry 623). The stored plurality of hash indexes may include one or more of a registry parameter; at least one metadata parameter indicating one or more a number of bits of a hash index, a dimensionality of a common encoding subspace, and a number of perceptual hash functions associated with the sets of perceptual hash functions; an index structure, a security watermark, and a registry target.

[0103] In some aspects, the system may store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database. The additional information, in some aspects, may include metadata relating to the hash indexes stored in a distributed multi-modal hash registry or a linear index registry. For example, referring to FIGs. 3 and 6, the system may store additional data in a metadata registry 342 or one or more of a tree-based index registry 622 or a metadata registry 623. [0104] At 1330, the system may receive, from a client computing device, one of a membership query or an association query. In some aspects, the membership query or the association query may include at least one hash index. The set of perceptual hash functions used to generate the query, in some aspects, may be a same set of perceptual hash functions that are used to generate hash indexes stored in the distributed data storage. In some aspects, the query may be generated by applying the set of perceptual hash functions to a test image or other input data for a verification operation and the generated query is based on the calculated query-related hash index. In some aspects, the query may be related to identifying whether information related to an object matching, or associated with, the input data set is stored in the probabilistic data structure. The query, in some aspects, may further include an indication of a domain or the type of sensor associated with the query. For example, referring to FIGs. 3 and 7, the system 330 may receive an image (e.g., image 715) via the interface(s) 331 and apply a family of k perceptual hash functions 721 at a query generator 360 (or a deep hash composer 362 or 720) and receive the generated query at the distributed data storage 340 for searching the distributed multi-modal hash index registry 341.

[0105] At 1340, the system may verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query. In some aspects, the verification includes performing a search of the probabilistic data structure based on the query-related hash index. In some aspects, the search is of a particular probabilistic data structure associated with a particular application that is associated with (e.g., indicated in) the query-related hash index. The probabilistic data structure, in some aspects, is one of a bloom filter, a cuckoo filter, a count-min-sketch data structure, or a Top-k data structure. For example, referring to FIGs. 3 and 7, the system 330 may search the distributed multi-modal hash index registry 341 or the distributed index registry server 740 based on the received query-related hash index to verify and/or determine whether matching sensor data set or an associated sensor data set is stored in a searched distributed data storage. For example, referring to FIG. 7, the routing & evaluation module 730 may receive a query generated by a deep hash composer 720, perform the identification 731 of a match and the check 732 for an association, and return results 745. As described in relation to FIG. 7, the identification 731 and the check 732 may be performed in a particular registry or index identified in a received query. [0106] At 1350, the system may identify, based on verifying the existence of the matching sensor data set or the associated sensor data, a location of the matching or the associated sensor data set. After identifying the location at 1350, the system may, at 1360, retrieve the matching or the associated sensor data set from the identified location. The identified location may be in an external storage (e.g., a local storage to the sensor network) For example, referring to FIGs. 3 and 7, the system 330 may identify, based on the search of the distributed multi-modal hash index registry 341 or the distributed index registry server 740, a location associated with a set of matching or associated data in a local storage (e.g., external data store 323). For example, referring to FIG. 3 the system 330 may retrieve, from the location identified at 1350, data from the external data store 323.

[0107] Finally, at 1370, the system may transmit the retrieved matching or the retrieved associated sensor data set to the client computing device. The retrieved data may include the matching or associated sensor data set and/or metadata associated with the matching or associated sensor data set. For example, referring to FIG. 3, the system 330 may transmit a sensor data set retrieved from an external data store 323 via interface(s) 332 to a verification module 311 via interface(s) 331.

[0108] FIG. 14 illustrates an example computing environment with an example computer device suitable for use in some example implementations. Computer device 1405 in computing environment 1400 can include one or more processing units, cores, or processors 1410, memory 1415 (e.g., RAM, ROM, and/or the like), internal storage 1420 (e.g., magnetic, optical, solid-state storage, and/or organic), and/or IO interface 1425, any of which can be coupled on a communication mechanism or bus 1430 for communicating information or embedded in the computer device 1405. IO interface 1425 is also configured to receive images from cameras or provide images to projectors or displays, depending on the desired implementation.

10109] Computer device 1405 can be communicatively coupled to input/user interface 1435 and output device/interface 1440. Either one or both of the input/user interface 1435 and output device/interface 1440 can be a wired or wireless interface and can be detachable. Input/user interface 1435 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch-screen interface, keyboard, a pointing/cursor control, microphone, camera, braille, motion sensor, accelerometer, optical reader, and/or the like). Output device/interface 1440 may include a display, television, monitor, printer, speaker, braille, or the like. In some example implementations, input/user interface 1435 and output device/interface 1440 can be embedded with or physically coupled to the computer device 1405. In other example implementations, other computer devices may function as or provide the functions of input/user interface 1435 and output device/interface 1440 for a computer device 1405.

[0110] Examples of computer device 1405 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and/or coupled thereto, radios, and the like).

[0111] Computer device 1405 can be communicatively coupled (e.g., via IO interface 1425) to external storage 1445 and network 1450 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configuration. Computer device 1405 or any connected computer device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.

[0112] IO interface 1425 can include but is not limited to, wired and/or wireless interfaces using any communication or IO protocols or standards (e.g., Ethernet, 1402.1 lx, Universal System Bus, WiMax, modem, a cellular network protocol, and the like) for communicating information to and/or from at least all the connected components, devices, and network in computing environment 1400. Network 1450 can be any network or combination of networks (e.g., the Internet, local area network, wide area network, a telephonic network, a cellular network, satellite network, and the like).

(0113] Computer device 1405 can use and/or communicate using computer-usable or computer readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like. Non- transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid- state storage), and other non-volatile storage or memory.

(0114] Computer device 1405 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transitory media, and stored on and retrieved from non-transitory media. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0115] Processor(s) 1410 can execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed that include logic unit 1460, application programming interface (API) unit 1465, input unit 1470, output unit 1475, and inter-unit communication mechanism 1495 for the different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can be varied in design, function, configuration, or implementation and are not limited to the descriptions provided. Processor(s) 1410 can be in the form of hardware processors such as central processing units (CPUs) or in a combination of hardware and software units.

(0116] In some example implementations, when information or an execution instruction is received by API unit 1465, it may be communicated to one or more other units (e.g., logic unit 1460, input unit 1470, output unit 1475). In some instances, logic unit 1460 may be configured to control the information flow among the units and direct the services provided by API unit 1465, the input unit 1470, the output unit 1475, in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 1460 alone or in conjunction with API unit 1465. The input unit 1470 may be configured to obtain input for the calculations described in the example implementations, and the output unit 1475 may be configured to provide an output based on the calculations described in example implementations.

(0117[ Processor(s) 1410 can be configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The processor(s) 1410 may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The processor(s) 1410 may further be configured to store information regarding the computed set of hash indexes in a probabilistic data structure for the fast membership verification queries. The processor(s) 1410 may further be configured to compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. The processor(s) 1410 may also be configured to transmit, from the edge computing device over an intervening network, the information regarding the computed set of hash indexes to the centralized data storage or the distributed storage. The processor(s) 1410 may also be configured to store additional information regarding the plurality of sensor data sets in one or more of a tree data structure or a metadata database. The processor(s) 1410 may also be configured to receive a query based on a query-related hash index computed for an input data set based on the set of perceptual hash functions for the type of sensor associated with the input data set. The processor(s) 1410 may further be configured to perform a search of the probabilistic data structure based on the query-related hash index. The processor(s) 1410 may further be configured to transmit a result of the search in response to the received query.

[0118] Processor(s) 1410 can be configured to receive a plurality of sensor data sets from a plurality of sensors including a plurality of types of sensors. The processor(s) 1410 may also be configured to compute a hash index for each particular sensor data set in the plurality of sensor data sets based on a set of perceptual hash functions for a type of sensor associated with the particular sensor data set. The processor(s) 1410 may further be configured to transmit, to a probabilistic data structure of a device configured to respond to queries relating to the plurality of sensor data sets, information regarding the computed set of hash indexes. The processor(s) 1410 may further be configured to compute at least one additional hash index based on one or more of (1) the computed hash index for two of more of the sensor data sets in the plurality of sensor data sets or (2) two or more of the sensor data sets in the plurality of sensor data sets. The processor(s) 1410 may further be configured to retrieve the set of perceptual hash functions for the plurality of types of sensors from a one of a centralized hash escrow or a distributed hash escrow. The processor(s) 1410 may further be configured to store, at least in part, the received plurality of sensor data sets in a local data storage. The processor(s) 1410 may further be configured to receive, from one of a centralized or a distributed server, a request for at least one particular sensor data set of the received plurality of sensor data sets stored in the local data storage based on a query received at one of the centralized or the distributed server. The processor(s) 1410 may further be configured to transmit the requested at least one particular sensor data set to one of the centralized server or the distributed server.

[0119] Processor(s) 1410 can be configured to receive, from an edge computing device, a plurality of hash indexes relating to a plurality of sensor data sets received by the edge computing device from a plurality of sensors including a plurality of types of sensors. The processor(s) 1410 may further be configured to store the received plurality of hash indexes in a probabilistic data structure. The processor(s) 1410 may further be configured to receive, from a client computing device, one of a membership query or an association query. The processor(s) 1410 may further be configured to verify an existence of a matching sensor data set or an associated sensor data set using the probabilistic data structure based on the received membership query or the association query. The processor(s) 1410 may further be configured to identify, based on verifying the existence of the matching sensor data set or the associated sensor data, a location of the matching or the associated sensor data set. The processor(s) 1410 may further be configured to retrieve the matching or the associated sensor data set from the identified location. The processor(s) 1410 may further be configured to transmit the retrieved matching or the retrieved associated sensor data set to the client computing device.

[0120] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In example implementations, the steps carried out require physical manipulations of tangible quantities for achieving a tangible result.

[0121] Unless specifically stated otherwise, as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system’s memories or registers or other information storage, transmission or display devices.

[ 01221 Example implementations may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer readable medium, such as a computer readable storage medium or a computer readable signal medium. A computer readable storage medium may involve tangible mediums such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid-state devices, and drives, or any other types of tangible or non-transitory media suitable for storing electronic information. A computer readable signal medium may include mediums such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.

[0123] Various general-purpose systems may be used with programs and modules in accordance with the examples herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the example implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the example implementations as described herein. The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), processors, or controllers.

10124] As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application. Further, some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format.

[0125] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and/or components of the described example implementations may be used singly or in any combination. It is intended that the specification and example implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.