CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] None.

TECHNICAL FIELD

[0002] The present disclosure is generally related to the field of graphical pattern scanning and, in particular, to graphical pattern scanning by an electronic device in standby mode.

BACKGROUND

[0003] Barcodes are graphical patterns that encode data. Barcodes may be configured as linear (or one-dimensional) barcodes, quick response (QR) codes (matrix barcodes), or other graphical patterns. The data encoded in a barcode may be interpreted as information relating to an object or location where the barcode is displayed. The data encoded in a barcode may be interpreted as a locator or identifier that points to a web page that displays information, multimedia content, or an ecommerce portal. The data encoded in a barcode may activate and/or provide information to an application in a scanning device.

SUMMARY

[0004] The disclosed aspects/embodiments provide techniques that allow an electronic device to remain in a standby mode while performing early steps of scanning an N-dimensional (N-D) code that graphically encodes data. Such improved techniques reduce the amount of one or more of the time, attention, and/or interaction a user must dedicate to the scanning process relative to existing techniques.

[0005] A first aspect relates to method performed by an electronic device to process an N-D code, the method comprising: detecting by an electronic device in a standby mode a trigger gesture performed by a user of the electronic device; in response to detecting the trigger gesture: obtaining while in the standby mode an image from a camera of the electronic device; determining while in the standby mode whether the image includes an N-D code; in response to determining that the image includes an N-D code: bringing the electronic device out of the standby mode; determining whether the user of the electronic device is an authorized user; and processing the N-D code in response to determining that the user is an authorized user of the electronic device. [0006] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the electronic device is one of a smartphone, a tablet computer, a virtual reality headset, an augmented reality headset, an electronic subsystem of a vehicle, and a static display device.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides obtaining an image from the camera of the electronic device comprises obtaining video data; and the image is a frame from the video data.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides detecting by the electronic device the trigger gesture comprises sensing a predetermined characteristic of a signal from an accelerometer of the electronic device.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides detecting by the electronic device the trigger gesture comprises sensing a difference between two images from the camera.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining whether the user is an authorized user of the electronic device comprises bringing the electronic device out of the standby mode.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides bringing the electronic device out of standby comprises sending a signal to the user indicating activation of an N-D code processing mode of the electronic device.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the signal is one or more of a haptic signal, a visual signal, and an auditory signal. [0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining whether the user is an authorized user of the electronic device comprises comparing biometric information of the user to stored biometric information of one or more authorized users of the electronic device.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides processing the N-D code comprises activating an application on the electronic device, accessing information stored in a network, displaying information stored in the electronic device, and accessing a web page.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides processing the N-D code comprises processing the N-D code in response to obtaining from the user a signal approving processing of the N-D code.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides processing the N-D code comprises no further interaction with the user. [0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides processing the N-D code comprises: obtaining from the user a signal indicating a selected N-D code from a plurality of N-D codes in the image; and processing the selected N-D code.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining whether the N-D code is present in the image comprises whether the N-D code is located in a central area of the image, and whether the N-D code is in focus in the image. [0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining whether the N-D code is present in the image further comprises determining whether the N-D code is skewed in the image.

[0020] A second aspect relates to a non-transitory computer readable medium comprising a computer program product for use by an electronic device, the computer program product comprising computer executable instructions stored on the non-transitory computer readable medium that, when executed by one or more processors, cause the transit network node to execute any of the preceding aspects.

[0021] A third aspect relates to an electronic device configured to perform the methods of any of the preceding aspects.

[0022] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0023] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0025] FIG. 1 is a method performed by an electronic device to process an N-D code.

[0026] FIG. 2 is a method performed by an electronic device to determine whether an N-D code that may be processed is present in an image.

[0027] FIG. 3 is a schematic diagram of an electronic device according to an embodiment of the disclosure.

[0028] FIG. 4A shows a barcode.

[0029] FIG. 4B shows a QR code. [0030] FIG. 5 shows a person using an electronic device to perform the method of FIG. 1.

DETAILED DESCRIPTION

[0031] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and/or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0032] As described above, barcodes (1 -dimensional codes) and QR codes (2-dimensional codes) are two types of graphical patterns that encode data. FIG. 4A shows an example barcode and FIG. 4B shows an example QR code. The methods and devices described herein may be used with barcodes and QR codes, but may also be used with other types of graphical patterns that encode data. Some examples are a graphical pattern that surrounds or is integrated into an image or logo, a graphical pattern that is superimposed on an image (e.g., a watermark), a graphical pattern that is a subliminal code incorporated into a video, a graphical pattern that is perceptible to a digital camera but not to the human eye, or a graphical pattern that varies with time or has depth (e.g., varies with distance from the camera. Collectively, such graphical patterns that encode data are referred to herein as N-dimensional codes (or N-D codes).

[0033] Scanning a QR code with a smartphone typically requires that a user of the smartphone perform several steps. With some smartphones, the user must:

1. bring the smartphone out of standby,

2. authenticate the user to the smartphone,

3. navigate the smartphone to a required screen,

4. start an application, open a menu,

5. select a QR scan function,

6. aim a camera of the smartphone at the QR code to be scanned, and

7. wait for the application to locate and process the QR code.

[0034] With other smartphones, the user must:

1. bring the smartphone out of standby,

2. activate a camera of the smartphone,

3. aim the camera at the QR code to be scanned,

4. confirm the smartphone’s identification of the QR code to be scanned,

5. authenticate the user to the smartphone, and 6. wait for the application to process the QR code.

[0035] Due to the many steps required, the user may choose not to scan an available QR code because, for example, the process takes more time than the user is willing to spend (e.g., the user is walking past the QR code and cannot stop to have the phone process the QR code), because the user is unable to dedicate the attention to the smartphone that the process requires (e.g., the user is in a conversation or is driving), or because the user is unable to interact as needed with the smartphone screen (e.g., the user has only one hand free).

[0036] Disclosed herein is a method to process an N-D code that addresses such problems by reducing the amount of time and/or the amount of interaction with the user that are required to process an N-D code, while maintaining secure scanning, preventing use in inappropriate locations, and reducing accidental use. Furthermore, the method disclosed herein may be performed on electronic devices other than smartphones, such as a camera-based electronic subsystem of a vehicle or a static display device having a camera

[0037] FIG. 1 is a method 100 performed by an electronic device to process an N-D code. In block 102, the electronic device detects, while in a standby mode, a trigger gesture performed by a user of the electronic device.

[0038] In various embodiments, an electronic device is in standby mode when it is in a state that is neither fully active, nor fully powered-off Such an electronic device in standby mode may be operating with less than its full processing capacity and may consume less energy than when operating out of standby mode. Such an electronic device in standby mode is not powered off and has some active functionality. Such an electronic device in standby mode may have its display turned off, but be sensitive to activity on its touch screen, pushbuttons, or accelerometers or be receiving signals on its cellular interface, wired network interface, wireless interface (e.g., a Bluetooth interface) or other communication interface.

[0039] In some embodiments, the electronic device is a portable device (e.g., a smartphone, a tablet computer, a virtual reality headset, or an augmented reality headset). In such embodiments, the trigger gesture performed by the user may comprise the user moving the electronic device from a current orientation to a new orientation (e.g., an upright or partially- upright position) and holding the electronic device steady in the new orientation for a predetermined amount of time (a time threshold). In some such embodiments, the time threshold is 0.5 seconds. In other embodiments where the electronic device is a portable device, the trigger gesture performed by the user may comprise the user moving the electronic device in a predetermined pattern. In some such embodiments, the predetermined pattern may be rotating the electronic device by at least a predetermined angle and back again at least a predetermined number of times. In other such embodiments, the predetermined pattern may be moving the electronic device to a predetermined orientation and moving the electronic device linearly by at least a predetermined distance in a predetermined direction, possibly within a predetermined amount of time (e.g., holding the smartphone in an orientation to view an N-D code and moving the smartphone toward the N-D code with a ‘pushing’ motion). Such movements of the electronic device may be detected using accelerometers, differences between two or more images from the camera, differences in successive frames of video data, or other such techniques. In various embodiments, the trigger gesture may be any combination of the above gestures.

[0040] In other embodiments, the electronic device is in a fixed position relative to the user (e.g., the electronic device is a camera-based electronic subsystem of a vehicle where the camera is viewing an interior of the vehicle or a static display device having a camera (e.g., a connected TV or a kiosk). In such embodiments, the trigger gesture performed by the user may comprise the user presenting an N-D code to the camera of the electronic device or holding an object bearing the N-D code up to the camera of the electronic device and ‘pushing’ the object toward the camera In other such embodiments, the trigger gesture may be an actual physical gesture performed by the user and visible to the camera (e.g., waving a hand in a predetermined gesture). Such trigger gestures may be detected by sensing a difference between two or more images from the camera.

[0041] In still other embodiments, the electronic device is a camera-based electronic subsystem of a vehicle where the camera is facing outward from the vehicle. In such embodiments, the trigger gesture performed by the user may comprise the user positioning the vehicle so that an N-D code is in view of the outward-facing camera. One example of such a trigger gesture is pulling the vehicle up to an entrance of a parking lot, where an N-D code is displayed on a pole next to the entering vehicle. Such trigger gestures may be detected by sensing a difference between two or more images from the camera.

[0042] In various embodiments, the trigger gesture performed by the user may comprise a predefined combination of gestures like those described above and similar gestures. Such a combination may be recognized as a trigger gesture when performed in a predetermined sequence (or order) or when performed in any order.

[0043] In block 104, in response to detecting the trigger gesture in block 102 and while remaining in the standby mode, the electronic device obtains an image from its camera In some embodiments, the electronic device obtains video data from its camera and the image comprises a static image (or frame) from the video data. In block 106, while remaining in the standby mode, the electronic device determines whether the image includes one or more N-D codes. One process that may be used by the electronic device in block 106 is described with reference to FIG.

2.

[0044] In some embodiments, where the camera of the electronic device is configured to operate in a plurality of modes, the method 100 may activate the camera in a mode that uses less battery power than another mode. For example, the camera may be activated in one or more of a black-and-white mode and a low resolution mode. Since block 104 is being performed while the electronic device is in standby mode, the user will likely not have visual feedback of whether they have successfully pointed the camera at an intended N-D code. Thus, in some embodiments, the camera may be activated in a wide angle mode, to increase the likelihood that the intended N-D code is within the field of view of the camera

[0045] In the method 100, if no N-D code is present in the image, the method 100 returns to block 102 to await another trigger gesture. In other embodiments, if no N-D code is present in the image, the method 100 returns to block 104 to obtain another image. In such embodiments, the method 100 loops between blocks 104 and 106 until the user successfully positions the electronic device (or the N-D code) for block 106 to determine that one or more N-D codes are present in the image. In such embodiments, the method 100 may loop between steps 104 and 106 only a predetermined number of times or for only a predetermined amount of time and then return to block 102 to await another trigger gesture.

[0046] In various embodiments, block 106 may detect whether image includes some other predetermined object. In such embodiments the object might be a person, a painting, a cartoon image, a logo, or other recognizable object. In such embodiments, decoding such an object may comprise matching the image of the detected object to one or more stored images and, if the object matches by at least a predetermined threshold measurement, causing the electronic device to perform an action associated with the stored image. In some embodiments, the associated action may be a similar type of action to those associated with decoding an N-D code.

[0047] In block 108, the electronic device enters N-D code processing mode. In various embodiments, in step 108 and the remaining steps of the method 100, the electronic device is brought out of standby mode only to the extent needed to perform the steps of the method. In some embodiments, block 108 includes storing in memory for later use the image in which the N-D code was determined to be present in block 106. Some embodiments include block 110, where the electronic device sends a signal to the user indicating the activation of N-D code processing mode. Such a signal may be a haptic signal (e.g., a vibration of the electronic device), a visual signal (e.g., a brief flashing pattern or written message on a display of the electronic device), or an auditory signal (e.g., a beep). [0048] In block 112, the user of the electronic device is authenticated as an authorized user of the electronic device. In some embodiments, any user activating the N-D code processing mode may be considered as an authorized user (e.g., where the electronic device is a public kiosk). However, where the electronic device is a smartphone or connected TV, only some users may be authorized (e.g., the owner of the smartphone or a user designated to the connected TV as an ‘adult’). Such authentication of the user may be disabled in a setting menu of the electronic device.

[0049] Authentication may comprise comparing biometric information of a current user of the electronic device to stored biometric information of one or more authorized users of the electronic device. In some embodiments, authentication may comprise facial or fingerprint recognition. The stored biometric information may be stored locally in the electronic device or in secure storage on a network.

[0050] Block 110, signaling to the user indicating the activation of N-D code processing mode can also optionally be provided after or during user authentication. In some embodiments, if it is determined in block 106 that the image includes an N-D code, the electronic device may initiate a user authentication process in block 112 and the signal indicating the activation of N-D code processing mode in block 110 is provided at the same time as user authentication, for example as an additional element of a lock screen. In other embodiments, the signal indicating the activation of N-D code processing mode in block 110 is provided only after user authentication.

[0051] In some embodiments, in block 114, the user is requested to confirm that N-D processing should proceed and further processing is performed only in response to obtaining from the user a signal approving such processing. If the user cancels such processing or fails to respond to the request, the method 100 terminates. The electronic device may be returned to standby mode or may return later in response to other conditions, such as inactivity. After the electronic device returns to the standby mode, the method 100 resumes at block 102.

[0052] In some embodiments, where a plurality of N-D codes has been determined to be present in the image, in block 114 the user is requested to indicate a selected N-D code to be processed. In some such embodiments, the electronic device displays the image to the user and the user selects an N-D code by tapping on the N-D code in the image on a touch screen device, producing a signal indicating the selected N-D code. In other such embodiments, the electronic device displays the image to the user and highlights one of the N-D codes. The user may interact with the electronic device to shift the highlight to a desired N-D code and then perform an action with a user interface of the electronic device to send a signal indicating the selection of the desired N-D code.

[0053] In block 116, the electronic device decodes and processes the N-D code. In some embodiments, this processing is the same as that provided by user-time and -interaction intensive functions for smartphone QR code scanning as described above. Such processing may include activating an application on the electronic device, accessing information stored in a network, displaying information stored in the electronic device, and accessing a web page. The QR code may be interpreted as a locator or identifier that causes the electronic device to access a web page (e.g., a page that displays information or multimedia content, or a page that is an ecommerce portal). The data encoded in the QR code may cause the electronic device to activate an application, provide information to the application, and/or cause the application to access information stored in a network or in the electronic device.

[0054] Other embodiments may additionally determine an identity of an N-D code generator that has generated an N-D code to be processed. If the N-D code generator is included in a list of approved N-D code generators, the electronic device decodes and processes the N-D code without further user interaction. In some such embodiments, when the N-D code generator is not included in the list of approved N-D code generators, the electronic device requests further confirmation from the user that N-D processing should proceed, before decoding and processing the N-D code.

[0055] In various embodiments, performance of the method 100 or of a block of the method 100 (e.g., block 108) may be disabled so that automatic scanning of N-D codes in standby mode is not performed. In some embodiments, the user may be able to disable such scanning in a setting menu of the electronic device. In some embodiments, a location sensing function of the device (e.g., Global Positioning System (GPS) sensing) may determine that the electronic device is in a location in which such scanning should not be activated.

[0056] In various embodiments, the electronic device may be configured to perform the method 100 while the electronic device is not in standby mode. Where the electronic device has already authenticated the user, block 112 may be skipped. In some embodiments, the confirmation of N-D processing by the user in block 114 may not be implemented.

[0057] It may be seen that the method of processing an N-D code disclosed herein eliminates many steps required in other electronic devices. For example, there is no need for the user to bring the electronic device out of standby mode, activate the camera, manually activate an N-D processing mode by starting an application or selecting a function, or tap on an image of an N-D code to initiate processing. [0058] FIG. 2 is a method 200 performed by an electronic device to determine whether an image includes one or more N-D codes and whether those codes may be decodable. The method 200 may be performed by the electronic device as part of block 106, as described with reference to FIG. 1.

[0059] In block 202, the electronic device detects whether the image includes one or more N-D codes. If no N-D codes are detected in the image, the method 200 terminates with an indication that the image does not include an N-D code. In some embodiments, the image is searched by the electronic device for alignment markings, fiducials, or position detection patterns that indicate the presence of N-D codes. In other embodiments, the image is presented to a classification algorithm that may use, for example, an artificial intelligence (Al) model that has been trained to recognize N-D codes using training data that includes training images that do include N-D codes and training images that do not include N-D codes.

[0060] In block 204, the electronic device determines which of the one or more N-D codes are in a predetermined central area of the image. If no N-D codes are detected in the central area of the image, the method 200 terminates with an indication that the image does not include an N- D code. In some embodiments, the portion of the image considered a central portion may be selected to exclude N-D codes that were inadvertently included in the image because the user had pointed the camera inaccurately toward one or more intended N-D codes, or had not intended to point the camera toward any N-D code. In other embodiments, the portion of the image considered a central portion may be selected based upon camera characteristics that would cause N-D codes in the periphery of the image to be decoded incorrectly or to be undecodable.

[0061] In block 206, the electronic device determines which of the one or more N-D codes are sufficiently sharp (or in focus) to be accurately decoded. If no N-D codes are sufficiently in focus, the method 200 terminates with an indication that the image does not include an N-D code. In some embodiments, such a determination may be made by measuring the one or more N-D codes using a spatial frequency response (SFR) or modulation transfer function (MTF). Any N- D code that measures above a predetermined threshold value may be determined to be sufficiently sharp to be accurately decoded.

[0062] In block 208, the electronic device determines which of the one or more N-D codes are rotated relative to the plane of the image (or skewed) by too great an angle to be accurately decoded. If all N-D codes are too skewed, the method 200 terminates with an indication that the image does not include an N-D code. In some embodiments, the amount by which an N-D code is skewed may be determined using the alignment markings, fiducials, or position detection patterns of the N-D code. In such embodiments, if one or more lines between orthogonal pairs of fiducials form angles with the sides of the image that are greater than a predetermined threshold angle, then the N-D code is too skewed in the plane of the image. In such embodiments, if the lengths of lines between orthogonal pairs of fiducials differ by more than a predetermined threshold amount, then the N-D code is too skewed about an axis that is parallel to the plane of the image. Any N-D code that is not skewed by more than a threshold amount may be determined to be accurately decodable. Further, in block 208 an N-D code can be skewed but still readable, but nevertheless lead to the method terminating because the user is determined to have not intended to read an N-D code with the camera.

[0063] Block 202 determines whether any N-D codes are included in the image. Blocks 204, 206, and 208 determine whether any N-D codes in the image are potentially decodable. In various embodiments, the determinations (tests) made in blocks 204, 206, and 208 may be made in any order and/or combination. N-D codes identified in block 202 may be stored in a set and, as the tests of blocks 204, 206, and 208 are performed, N-D codes that do not satisfy one of the tests may be removed from the set and not considered in subsequent tests. In such embodiments, if one or more N-D codes remain in the set after all tests have been performed, the method 200 terminates with an indication that the image includes one or more N-D codes.

[0064] FIG. 3 is a schematic diagram of an electronic device 300 (e.g., a smartphone, atablet computer, an electronic subsystem of a vehicle, a static display device, etc.) according to an embodiment of the disclosure. The electronic device 300 is suitable for implementing the disclosed embodiments as described herein.

[0065] The electronic device 300 includes a processor 302 that can implement various processing operations of the electronic device 300. The processor 302 is coupled to and in communication with at least one memory 304 and is implemented by hardware and software. The processor 302 may be implemented as one or more central processing unit (CPU) chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 302 comprises an N-D code scanning module 306. The N-D code scanning module 306 is able to implement the methods disclosed herein. The inclusion of the N-D code scanning module 306 therefore provides a substantial improvement to the functionality of the electronic device 300 and effects a transformation of the electronic device 300 to a different state. Alternatively, the N-D code scanning module 306 is implemented as instructions stored in the memory 304 and executed by the processor 302.

[0066] The memory 304 stores instructions and data used, generated, or collected by the electronic device 300. For example, the memory 304 could store software or firmware instructions executed by the processor 302 and data used to perform the methods disclosed herein. The memory 304 includes any suitable volatile and/or non-volatile storage and retrieval device(s). Any suitable type of memory may be used, such as random access memory (RAM), read only memory (ROM), hard disk, optical disc, subscriber identity module (SIM) card, memory stick, secure digital (SD) memory card, and the like.

[0067] The processor 302 is also coupled to and in communication with a transceiver 308, which is coupled to an antenna 310 and/or a wired communication terminal 312. The transceiver 308 is configured to transmit and receive signals using Ethernet or other network protocol via the wired communication terminal 312. The transceiver 308 is also configured to wirelessly transmit and receive signals using cellular or Bluetooth or other wireless protocol via the antenna 310.

[0068] The processor 302 is further coupled to and in communication with a user interface 314 for communication with one or more user interface devices. Such user interface devices may include a display, a touch screen, a keyboard, a remote control, a fingerprint scanner, a speaker, a mouse, a trackball, and other devices suitable to provide interaction with a user of the electronic device 300.

[0069] The processor 302 is also coupled to and in communication with a first camera 316 and, in some embodiments, to a second camera 318. In a smartphone or a tablet computer, the camera 316 may be a camera facing away from the display screen of the electronic device 300 and, if present, the camera 318 may be a camera facing in the same direction as the display screen of the electronic device 300 (e.g., to view a user of the electronic device 300). In an electronic subsystem of a vehicle, the camera 316 may be one or more cameras facing the interior of the vehicle (e.g., to view occupants of the vehicle) and the camera 318 may be one or more cameras facing away from the vehicle (e.g., to view the vehicle’s surroundings). In a static display device (e.g., a kiosk or connected TV), the camera 316 may face a user of the static display device.

[0070] The disclosed embodiments may be a system, an apparatus, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure. The computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.

[0071] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0072] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.