DEVICE AND METHOD FOR PROVIDING AN E-READER INTERACTIVE SURFACE WITH HAPTIC FEEDBACK FOR AN AUTOMOTIVE SETTING

TECH N ICAL FI E LD

[0001] The present disclosure relates to a device and operation method for providing an e- reader interactive surface with haptic capabilities, in particular an e-reader device and operation method arranged for helping users distinguish between different user-interface options using haptic feedback.

BACKG ROU N D

[0002] Document KR20160010843 discloses a method for playing an audio book provided with a vibration function, comprising the steps of: loading sound source data to play the audio book and obtaining total play time of the sound source data loaded; playing the sound source data; loading one or more from vibration data and illustration data corresponding to the sound source data; obtaining one or more from the vibration information and the illustration information corresponding to the play time of the sound source data from one or more from the vibration data and the illustration data; and outputting vibration or illustration based on one or more from the vibration data or the illustration data obtained.

[0003] Document US2016240102 discloses a system for enhanced learning that combines tactile surfaces and audio recordings, allowing the user to explore a 3-dimensional object through touch while also hearing pre-recorded audio explaining regions or features of interest. One of a collection of 3-dimensional tactile modules is used with the system at a given time. When a tactile module is inserted into the system (or otherwise connected) by the user, the system automatically recognizes the inserted module, locates the set of audio recordings and region maps associated with that tactile module, and then plays modulespecific audio recordings. As the user explores the tactile surface, the system is continually alert for a signal from the user that he/she would like to know more about a particular feature or sub-region of the surface. When such a signal is detected, the system automatically plays a specific audio recording containing information associated with that region of the tactile surface. A region map, stored in memory along with the set of audio recordings for a specific tactile module links specific audio recordings to specific regions of interest.

[0004] Document DE102012215397 Al discloses a method for interactive attentiveness enhancement of a vehicle driver includes steps of determining that a driver assistance function for semiautomatic control of the motor vehicle is active, detecting a stimulus which may be experienced by the vehicle driver upon an observation of the surroundings of the motor vehicle, detecting an input of the vehicle driver in response to the stimulus, determining a game result based on the input and the stimulus, and outputting a notice concerning the game result.

[0005] These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure.

GENERAL DESCRIPTION

[0006] The present disclosure relates to a device for providing an e-reader interactive surface with haptic capabilities for helping users distinguish between different user-interface options.

[0007] Some currently available book reader/audiobook mobile applications (apps) can be connected to a vehicle's infotainment system (e.g., Kobo Audiobooks and Apple CarPlay, Audible and Apple CarPlay or Android Auto). The connection between these apps and the in- vehicle infotainment system is designed so that users may receive their intended entertainment content with minimal distraction from the primary task of driving the vehicle. Due to this, the functioning of these apps inside a vehicle is limited to being only capable of delivering audio content through the vehicles' speakers, with no option to display the books' written content. As the automotive industry evolves, it is expected that users will spend less time driving their vehicles, which will in turn result in users having more free time for themselves during their commutes. Additionally, most in-vehicle infotainment displays and surfaces are not equipped with haptic feedback capabilities, so the interactions between users and these devices are very basic. Haptic feedback in devices and surfaces has commonly been achieved through vibrotactile stimuli, generated from mechanical actuators (e.g., KHOSHKAVA et al., 2019). Recently, technologies have emerged that make use of other actuation types, such as electrovibration, to generate other types of tactile feedback beyond just vibrations (e.g., Olley et al., 2020). While the technology for creating richer display and surface interactions is available, apps that make use of these capabilities still need to be developed.

[0008] With the expected increase in autonomous vehicles that will become available to the general consumer market, these users will find themselves with more free time during their commute. Users might choose to engage in a variety of activities to make use of this free time, either for leisure or work purposes, such as reading documents, reading books, or listening to audiobooks. To the best of our knowledge, due to current safety concerns, currently there is no book reader app which has the function of displaying a book's written content on the in- vehicle infotainment display. Additionally, as display surface technology develops, more focus is being given to display devices and surfaces that can provide new and innovative ways of enriching the interactions users have with them such as generating richer haptic feedback sensations.

[0009] An aspect of the present disclosure relates to a book reader/audiobook application (app) for use in either level 5 autonomous vehicle's infotainment displays, or in vehicle's infotainment displays aimed at passengers. The app makes use of the richer interaction opportunities that some haptic feedback display and surface technologies currently offer, while also planning for what these technologies and other haptic feedback technologies will be able to provide in the future.

[0010] An aspect of the present disclosure relates to use of a haptic surface, comprising an insulative layer fitted atop a transparent and conductive electrode sheet layer, which, when charged with an electric current and when in contact with the human skin, produces a frictional stimulus by controlling the electric current. The conductive electrode sheet layer is in turn fitted atop a touch-sensitive layer fitted atop a layer of LED light, which, in turn, is fitted on top of a layer of vibrotactile actuators. The layer of LEDs is optional and when not present the touch-sensitive layer is fitted atop a layer of vibrotactile actuators. This haptic surface can be employed either as: a) the interactive surface of a display to which it is directly connected to (e.g., the haptic surface is part of the display); or b) as the interactive surface of an external device (i.e., a device to which it is not directly connected to) (e.g., touch surface interface communicating wirelessly with a device in close proximity). Depending on the intended use, the inclusion of the layer of LED lights might not be necessary. The touch-sensitive layer can be made of a variety of materials, depending on the intended use and aesthetics (e.g., transparent glass - display; translucent plastic - interactive surface for an external device with a layer of LED lights underneath it, providing location cues, such has button icons, or identifying the edges of the interactive area by lighting up the edge of said area). This surface can generate both frictional and vibrotactile haptic sensations. LEDs are controlled though external software commands, which are interpreted and acted on by the smart surface's own technology. Also, The LEDs can be controlled by the same systems that controls the haptic surface.

[0011] Frictional haptic sensations are generated via the insulative and conductive layers, which allow for friction between the haptic surface and the user's skin to be modulated through electrostatic actuation, while the skin is in motion over the surface. Vibrotactile haptic sensations, in turn, are generated via the vibrotactile actuators which can generate vibrations strong enough to be felt through the user's skin, while in direct contact (stationary or in motion) with the haptic surface. When the haptic surface is used as the interactive surface for an external device, haptic feedback sensations can be generated by the external device's software. When software (be it from an external device's software, or from the device to which the surface is directly connected to) sends a command to the surface, regarding the actuation of either or both haptic sensations, the surface's own technology must be capable of interpreting said command and actuate on the intended layer of actuators at the intended locations. The surface's own technology must also be capable of interpreting commands regarding the activation of the layer of LED lights, be it to activate them all, or to activate only certain areas.

[0012] In an embodiment, the haptic surface is fitted on at least two regions inside the vehicle: on the display of the central stack, where it serves as said device's interactive surface, as well as on a seat's armrest where it acts as an external interactive surface for the central stack.

[0013] It is disclosed a device for providing an e-reader interactive surface with haptic feedback for an automotive setting, comprising: a display for displaying images of objects of said e-reader interactive surface; a haptic surface comprising actuators for exhibiting tactile textures, comprising: a touch-sensitive layer, a frictional haptic layer comprising an insulative sheet for user touch and a transparent conductive electrode film for providing frictional haptic sensations, and a vibrotactile haptic layer for providing vibrotactile haptic sensations; an electronic data processor configured to: provide the e-reader interactive surface on the display for displaying images of objects of said e-reader interactive surface; display an image on said display, said image comprising a user-interface element of an object of said e-reader interactive surface; and drive the haptic layers to exhibit a tactile texture when the user touches the image of the object, wherein the texture mimics tactile sensations corresponding to the object whose image is being displayed.

[0014] In an embodiment, the object is a user-interface slider or a scrollbar.

[0015] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent on the length or size of a data record corresponding to the slider or scrollbar.

[0016] In an embodiment, the object is a user-interface icon.

[0017] In an embodiment, the electronic data processor is further configured to drive the haptic layers to exhibit a tactile texture whose roughness is defined dependent on the length or size of a data record corresponding to the user-interface icon.

[0018] In an embodiment, the object is an audio file or an audio streaming object.

[0019] In an embodiment, the touch-sensitive layer is made of translucent plastic comprising LEDs underneath it.

[0020] In an embodiment, the automotive setting is an autonomous vehicle and the display is a central stack display of the autonomous vehicle.

[0021] In an embodiment, the display is arranged to be mounted upon an armrest of the autonomous vehicle.

[0022] In an embodiment, the touch-sensitive layer is a capacitive based touch-sensitive layer.

[0023] It is also disclosed a system for providing an e-reader interactive surface with haptic capabilities comprising a first device for providing an e-reader interactive surface with haptic capabilities according to any of the previous embodiments for a central stack display and a second device for providing an e-reader interactive surface with haptic capabilities according to any of the previous claims for an armrest.

[0024] In an embodiment, the method of operating a device further comprises to use said electronic data processor for: provide the e-reader interactive surface on the display for displaying images of objects of said e-reader interactive surface; display an image on said display, said image comprising a user-interface element of an object of said e-reader interactive surface; and drive the haptic layers to exhibit a tactile texture when the user touches the image of the object, wherein the texture mimics tactile sensations corresponding to the object whose image is being displayed.

[0025] In an embodiment, the method of operating a system comprises to use said electronic data processor for: provide the e-reader interactive surface on both displays for displaying images of objects of said e-reader interactive surface; display an image on both displays, said image comprising a user-interface element of an object of said e-reader interactive surface; and drive the haptic layer where the user touches the image of the object, to exhibit a tactile texture when the user touches the image of the object, wherein the texture mimics tactile sensations corresponding to the object whose image is being displayed.

BRI EF DESCRI PTION OF THE DRAWI NGS

[0026] The following figures provide preferred embodiments for illustrating the present disclosure and should not be seen as limiting the scope of the invention.

[0027] Figure 1 illustrates a schematic operation method of the home screen/library screen according to an embodiment.

[0028] Figure 2 illustrates a schematic operation method of the next/previous page navigation according to an embodiment.

[0029] Figure 3 illustrates a schematic operation method of the book options menu process according to an embodiment.

[0030] Figure 4 illustrates a schematic operation method of the text highlighting process according to an embodiment. [0031] Figure 5 illustrates a schematic operation method of the process with the audiobook open according to an embodiment.

[0032] Figure 6 shows a schematic illustration of the main menu according to an embodiment.

[0033] Figure 7 shows a schematic illustration of the finger resting over an item which then triggers a vibrotactile stimulus and a read aloud command according to an embodiment.

[0034] Figure 8 shows a schematic illustration of the dragging of an item to the bottom of the screen to open it according to an embodiment.

[0035] Figure 9 shows a schematic illustration of the drawing of a circle around an icon to open it according to an embodiment.

[0036] Figure 10 shows a schematic illustration of the dragging of a finger across the surface to switch page according to an embodiment.

[0037] Figure 11 shows a schematic illustration of the options menu when a book is opened according to an embodiment.

[0038] Figure 12 shows a schematic illustration of the app window when audiobook is playing according to an embodiment.

[0039] Figure 13 shows a schematic illustration of the options menu when audiobook is opened according to an embodiment.

DETAILED DESCRI PTION

[0040] The present disclosure relates to a device for providing an e-reader interactive surface with haptic capabilities for helping users distinguish between different user interface options.

[0041] In an embodiment, the haptic feedback book reader/audiobook player app of the present disclosure can be installed and accessed through several different devices, such as the vehicle's infotainment system or the user's personal smartphone, as long as the software permits it. However, the full benefits of the haptic feedback sensations can only be felt when the interaction between the user and the app is done through a device or surface which possesses actuators capable of generating the intended sensations. Therefore, the interaction between user and app is more fruitful when done through the haptic surface of the present disclosure, especially when said interaction occurs inside the vehicle during the user's commute.

[0042] In an embodiment, the haptic feedback sensations of the present disclosure make use of frictional and/or vibrotactile stimuli which can be generated through electrostatic and vibrotactile actuation, respectively.

[0043] When the app is accessed through a device without a surface that can produce one or both of the intended haptic feedback stimuli types, an adequate external interactive surface can be used to generate said stimuli. Although the main innovation of this app is the employment of haptic feedback sensations during the interactions with the user, there are no impediments regarding installing and using this app on devices without actuators that can produce the stimuli types from which this app benefits, as long as users keep in mind that the complete experience cannot be achieved in said devices. There is also no impediment to the user choosing to use an external interactive surface to interact with the app when displayed on a device which also possesses a surface with haptic feedback capabilities. For example, users might choose to interact with the app through the armrest's haptic surface, while leaning back on their seat, for extra comfort, while others might prefer to interact with it through the central stack, interacting "directly" with the visual content displayed there. The visual content of the app is always displayed on a screen (e.g., in the central stack).

[0044] The app of the present disclosure synchronizes information between multiple instances of the service across different devices (e.g., automotive display app, android app, web app). It also provides all the basic functionalities found on other book reader/audiobook apps, such as library/bookshelf organization, word search function, alphabetical or genre sorting, text highlights, jump to chapter, as well as background colour and font size control (when supported by the file type).

[0045] In an embodiment, the app of the present disclosure is used inside a vehicle with both a central stack with haptic feedback and an external interactive surface embedded on the seat's armrest.

[0046] In an embodiment, the navigation of the home screen/library screen is shown in Figure

1. [0047] As illustrated in Figure 1, this device according to an embodiment includes the navigation in home screen/library screen 1, the user is not sure of what they want to read yet, so decide to explore their library 2, the user slides finger across icons representing bookshelves and unorganized books and audiobooks 3, different haptic feedback sensations are felt while sliding the finger across each icon. Bookshelves with more books, books with more words, and audiobooks with longer audio files have a rougher texture 4, the user stops finger on top of icon 5, the name of item pops up on the display and voice assistant reads it aloud A. If the icon does not grab user's attention 14, the user is not sure of what they want to read yet, so decide to explore their library 2. If the icon grabs user's attention 6 and the user decides to open it B. If the icon is a bookshelf 7 and the user clicks the icon and receives audio and vibrotactile feedback. Bookshelf view opens, showing icons for all the books/audiobooks on it 8, the user slides finger across icons representing books/audiobooks 9 and bookshelves with more books, books with more words, and audiobooks with longer audio files have a rougher texture 4. If the icon is a book/audiobook 10, the user has three options to open it C, if the user drags icon to the bottom of the display (a rougher texture sensation is felt the more words the book has/the longer the audiobook file is) 11. If the user uses finger to draw circle clockwise around the icon (a rougher texture sensation is felt the more words the book has/the longer the audiobook file is) 12. If the user clicks the book icon like a normal button (a vibrotactile click sensation, accompanied by an audio cue, is transmitted to the user) 13. If the name of item pops up on the display and voice assistant reads it aloud A and the icon does not grab user's attention, the user is not sure of what they want to read yet, so decide to explore their library 2.

[0048] As illustrated in Figure 2, this device according to an embodiment includes the operation method of the next/previous page navigation with a written book open 15, a user intends to switch page 16, a user slides finger on touchscreen display surface/smart surface portion of the armrest 17 and the textural sensation of the paper is generated D. If the textural sensation is from right to center 18, the user is not sure of what to read yet, so decide to explore their library 19 or if the textural sensation is from left to center 19', the user is not sure of what they want to read yet, so decide to explore their library 21.

[0049] Users can slide their fingers across icons representing different books/audiobooks/bookshelves. The rougher an icon feels, the more words it has/the longer the audio file is/the more books are assigned to said bookshelf (Figure 6). When the finger stops on top of an icon, its name pops up and the voice assistant reads it aloud through the vehicle's speakers (this option can be disabled in the app's settings) (Figure 7). While navigating the app's user interface, each button click is accompanied by audio and vibrotactile feedback. To open a book/audiobook, the user has three options: i) drag the icon to the bottom of the screen (a rougher texture sensation is felt the more words the book/the longer the audiobook file is) (Figure 8); ii) draw a circle clockwise around the icon (a rougher texture is felt the more words the book has/the longer the audiobook file is) (Figure 9); iii) click the icon to open it (a vibrotactile click sensation, accompanied by an audio cue, is transmitted to the user).

[0050] If a book is selected and opened, users can slide their finger(s) (from edge to centre) in order to flip to the previous/next page. As they do so, the haptic surface will generate the sensation of dragging a paper page (together with page-flip audio and visual feedback, the latter being customizable by the user's preference) (Figures 2 and 10).

[0051] Users can tap the centre area of the haptic surface to hide/unhide a reading menu, which includes "jump to chapter" option, "text size adjustment" option, and a slider showing the current position on the book and which can also be used to jump to specific parts/pages of the book. Clicking an option generates a small vibrotactile stimuli on the surface to indicate a button press. Sliding the slider generates a unique textural sensation while the icon is dragged along the slider. When exploring this menu by sliding the finger, distinct texture sensations are used to help users distinguish between the different option buttons of the menu, without the use of visual cues. For eye-free interaction, placing the finger to rest on top of a button for a short period of time will result in the voice assistant reading aloud the name of the option in which the finger is resting on (these audio cues can be disabled in the app's settings). (Figures 3 and 11).

[0052] As illustrated in Figure 3, this device according to an embodiment includes the operation method of the book options menu process with a book open 22, a user intends to use an option from the hidden menu 23, the user taps centre of touchscreen display surface/smart surface portion of the armrest 24, a short vibrotactile stimuli is generated 25. If hidden menu is revealed E and the user intends to press a button, they already know the location of 26, the user moves finger towards the intended option button's position and clicks it, which is accompanied by a vibrotactile and audio confirmation cue 27 and so an option is the one user was searching for 33. If hidden menu is revealed E and the user is searching for an option button without visual cues 28, the user slides finger across the display surface/armrest' smart surface 29, the finger slides on top of a button 30 and the user feels a noticeable difference in the texture sensation, indicating the finger has slided on top of a button/different button 31, the user rests finger on top of button for a short period 32 and the voice assistant reads aloud the name of the option F. If the voice assistant reads aloud the name of the option F and the option is not the one user was searching for 34, the user slides finger across the display surface/armrest' smart surface 29. If hidden menu is revealed E and the user intends to use the slider to move along the pages 35, the user presses the slider icon and drags it in the intended direction 36, the user feels a texture sensation that is specific to the slider while dragging the slider icon 37.

[0053] Users can select words/sentences that they would like to highlight/comment on. This information is automatically synchronized (as well as reading progress) with the user's account through the internet, and readily available the next time they open said document in another instance of the app in any device/web browser (this feature can already be found in other book readers/document reader apps/services). The selection can either be performed by touch/long pressing and dragging across the intended text to select, by controlling a "mouse pointer" and using it to highlight the selection by touch/long pressing and dragging across the intended text to select (Figure 4).

[0054] As illustrated in Figure 4, this device according to an embodiment includes an operation method of the text highlighting process with a written book open 38 and if a user intends to highlight a segment of the text G using a display 39, a user taps and hold finger over the first word of the desired section 40, selection Ul appears 41, the user slides finger to the end of the section 42 and a selection Ul continues to be visible, user can now touch the "highlight selection" button 43. If a user intends to highlight a segment of the text G using the haptic surface portion of the armrest 44, the user slides finger across surface to move mouse pointer on the display to the desired section 45 and selection Ul appears, user can now select the "highlight selection" button.

[0055] If an audiobook is selected and opened (Figure 5): Control elements for play/pause, stop, next paragraph/page, and previous page/paragraph are presented on the visual interface (Figure 12). The LEDs, installed under the armrest surface's touch-sensitive layer, are turned on to present these same control elements, allowing users to control the app from a comfortable resting position through the armrest when the central stack' software sends the respective command(s) to the surface. As with written books, users can tap the centre area of the display or of the armrest's haptic surface to hide/unhide a reading menu presented on the visual interface. The audiobook is paused while this menu is being shown (Figure 13). In an embodiment, the LEDs can be controlled by the same systems that controls the haptic surface.

[0056] As illustrated in Figure 5, this device according to an embodiment includes an operation method of the process with the audiobook open 47, control elements for play/pause, stop, next paragraph/page, and previous page/paragraph are presented on both the display and the armrest' smart surface, through light up LED elements. On the display, users can find a horizontal slider through which they can control the file's timeline, as well as a vertical slider, through which they can control the volume. Clicking any of this button is accompanied by vibrotactile and audio feedback 48, the user intends to use an option from the hidden menu 49, the user taps centre of touchscreen display/surface 50 and a short vibrotactile stimuli is generated on the interacted surface 51.

[0057] According Figure 5, if a hidden menu is revealed H and the user intends to press a button they know the location of 52, the user moves finger towards the intended option button's position and clicks it, which is accompanied by a vibrotactile and audio confirmation cue 53, and an option is the one user was searching for 54, if hidden menu is revealed I. if a hidden menu is revealed H and the user is searching for an option button without visual cues 55, user feels a noticeable difference in the texture sensation, indicating the finger has slided on top a button/different button 56, the finger slides on top of a button 57, the user presses the slider icon and drags it in the intended direction 58, the user rests finger on top of button for a short period 59 if a hidden menu is revealed I. If a hidden menu is revealed I and an option is not the one user was searching for 60, the user feels a noticeable difference in the texture sensation, indicating the finger has slided on top a button/different button 56.

[0058] Flow diagrams of particular embodiments of the presently disclosed methods are depicted in figures. The flow diagrams illustrate the functional information that one of ordinary skill in the art requires to perform said methods in accordance with the present disclosure.

[0059] It will be appreciated by those of ordinary skill in the art that unless otherwise indicated herein, the particular sequence of steps described is illustrative only and can be varied without departing from the disclosure. Thus, unless otherwise stated, the steps described are unordered, meaning that, when possible, the steps can be performed in any convenient or desirable order.

[0060] It is to be appreciated that certain embodiments of the disclosure as described herein may be incorporated as code (e.g., a software algorithm or program) residing in firmware and/or on computer-compatible medium having control logic for enabling execution on a computer system having a computer processor, such as any of the services described herein. Such a computer system typically includes memory storage configured to provide output from execution of the code which configures a processor in accordance with the execution. The code can be arranged as firmware or software, and can be organized as a set of modules, including the various modules and algorithms described herein, such as discrete code modules, function calls, procedure calls or objects in an object-oriented programming environment. If implemented using modules, the code can comprise a single module or a plurality of modules that operate in cooperation with one another to configure the computer in which it is executed to perform the associated functions, as described herein.

[0061] The term "comprising" whenever used in this document is intended to indicate the presence of stated features, whole, parts, components, but not to preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0062] The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above-described embodiments are combinable. The following dependent claims further set out particular embodiments of the disclosure. [0063] References

1. KHOSHKAVA, V., CRUZ-HERNANDEZ, J. M., & SHAH, K. (2019). HAPTIC ACTUATOR ASSEMBLY WITH A SPRING PRE-LOAD DEVICE (Patent No. EP3582074A1).

2. Olley, M. F. D., Peshkin, M. A., & Colgate, J. E. (2020). Electronic controller haptic display with simultaneous sensing and actuation (Patent No. US10768749B2).

3. Tanvas Inc. (2020). Tanvas Automotive Solutions [Brochure], Tanvas.

4. TDK Corporation. (2020). PowerHapTM Actuators.