FIELD OF THE INVENTION

The invention relates to the field of de -identifying sensitive information. In particular, the invention relates to de -identifying sensitive information captured from a 3D imaging system.

BACKGROUND OF THE INVENTION

For any communication system, privacy is often an essential feature. Sharing information with a receiver often requires any sensitive information, that is not needed by the receiving party, to be hidden from the receiver.

This is a well-known problem and is relatively easy to solve for some cases. For example, when there is a single camera at a fixed position the position of the sensitive information in the frames can be blurred/removed. The same logic can be applied when a video is captured, post processed, and only later made available to others.

However, if the video is captured from a moving camera (e.g., body camera from a surgeon), or from a multiple-sensor 3D capture/imaging system, and is transmitted in real-time, then a new method for de -identifying sensitive information is needed.

For example, in the case of a surgeon performing an operation that is being transmitted to students remotely, it may be that the face, or other identifying features (e.g., tattoos), of the patient and/or health care providers need to be blurred. Additionally, details such as names, patient identification numbers, social security numbers, etc., which may be on monitoring screens, may need to be removed.

If a room such as a surgical suite is captured in 3D and rendered in such a way that the users and/or camera may freely move throughout the scene, then it can never be known where the camera is at any point. If there are objects, persons and/or information in the captured scene that should not be displayed to the viewer, then those items must be dynamically removed or blurred in real-time so as to ensure that privacy is maintained.

Thus, there is a need for a method capable of de -identifying sensitive information imaged by 3D capture systems.

SUMMARY OF THE INVENTION According to examples in accordance with an aspect of the invention, there is provided a method for de -identifying sensitive information, the method comprising:obtaining a three dimensional, 3D, model of a scene; identifying a region of interest, ROI, in the 3D model of the scene corresponding to sensitive information; and modifying the 3D model of the scene to de-identify the sensitive information contained in the ROI; and outputting the 3D model including the de-identified sensitive information in the ROI, wherein based on the 3D model output, a real-time two-dimensional, 2D, image of the scene is rendered at a target viewport. Modifying the 3D model of the scene, to de-identify sensitive information, enables the user to be sure that, when an image is rendered from the 3D model at any viewport, the rendered image will also be de-identified. As such, it is not necessary to de-identify each image individually.

The ROI is a 3D region of interest. In some embodiments, the ROI might comprise sensitive information or other information of interest that should be de-identified. The 3D model can be a mesh, a 3D mesh, multiple 2D meshes, a point cloud, a Light Detection And Ranging, LIDAR, pointcloud representation, or any other volumetric or 3D representation of a room or environment, for instance based on space partition, voxel-based segmentation or other known techniques. It is to be understood that a LIDAR point-cloud representation refers to a point-cloud representation that is preferrably obtained by a LIDAR technology, but other point cloud representations can be envisioned within the context of the present application.

In some embodiments, de-identification of the sensitive information comprises any one of: removal of texture values, vertex removal, vertex filtering, vertex attachment, colour removal in objects of interest, replacement of sensitive information with non-sensitive information, removal of at least some objects of interest, such as removal of at least part of points in the point cloud.

In some embodiments, vertex removal comprises removal of characteristics of the 3D model, such as a subset of vertices, in the ROI of the 3D model, wherein the ROI corresponding to the removed characteristics of the 3D model is reconstructed using information from another image set not part of the 3D model.

Identifying the ROI within the 3D model can be achieved through automatic detection, user input, or other any other means. Of course, more than one ROI can be identified in the 3D model.

Modifying the 3D model removes, blurs, replaces or otherwise distorts the information in the ROI such that no privacy sensitive data in that region remains or is identifiable.

The method may further comprise generating a two dimensional, 2D, image of the scene at a target viewport, such as at a parent node, using the modified 3D model. Any image generated from the modified 3D model will be de-identified as the 3D model itself is de-identified after being modified.

Modifying the 3D model for example comprises modifying texture values of the ROI in the 3D model. In some embodiments, modification of texture values might refer to change of characteristics of texture values, such as the shape of textures, color of textures, brightness, opacity, roughness, or any combination of these texture values. These are non-limiting examples of texture values, and other examples of texture values can be envisioned within the context of the present application.

Modifying the 3D model for example comprises modifying a geometry of the ROI in the 3D model. Modifying a geometry distorts the 3D model such that the modified 3D model does not contain any information which can lead to the identification of the information in the ROI. Thus, the modification can be achieved without requiring the texture of the 3D model to be modified, which may make the resulting 3D model look more natural whilst still being de-identified.

For example, the face-shape of a person in the 3D model can be modified to a different shape, whilst keeping the original texture values. Thus, the person will still look like a person in the 3D model but cannot be identified from the model itself.

The method may further comprise: receiving a non-modified image of the scene taken at a camera viewport; identifying an area of the non-modified image corresponding to the projection of the ROI from the camera viewport; and modifying the identified area in the non-modified image to de-identify the information contained in the ROI.

This allows other images to be de-identified by using the 3D model. For example, an additional camera may provide a better image (e.g. higher quality/resolution) than a rendered image. Thus, these images can be efficiently modified by converting (i.e. projecting) the ROI to the viewport of the camera. Thus, the modified images can be used alongside the modified 3D model to view the scene.

Identifying the ROI may comprise identifying a person in the scene, estimating the pose of the identified person and determining whether the pose corresponds to a pre -determined pose. It may for example be known that only people in certain pose (e.g. lying down on a patient couch) need to be deidentified.

The method may further comprise: receiving a verification that the ROI is in the correct position relative to the position of the sensitive information in the 3D model; and modifying the position of the ROI based on the verification.

This enables a check to be implemented that the de -identification will correctly take place. The invention also provides a computer program carrier comprising computer program code which, when executed on a processing system, cause the processing system to perform all of the steps of the method described above.

The invention also provides a system for de -identifying sensitive information, the system comprising a processor configured to: obtain a three dimensional, 3D, model of a scene; identify a region of interest, ROI, in the 3D model corresponding to sensitive information; and modify the 3D model of the scene to de-identify the sensitive information contained in the ROI.

This system performs the processing of the 3D model using the method described above.

The processor is for example further configured to generate a two dimensional, 2D, image of the scene at a target viewport using the modified 3D model.

The processor is for example configured to modify the 3D model by modifying texture values of the ROI in the 3D model and/or modifying geometry values of the ROI in the 3D model.

The processor may be further configured to: receive a non-modified image of the scene taken at a camera viewport; identify pixels in the non-modified image corresponding to the projection of the ROI from the camera viewport; and modify the identified pixels in the non-modified image to de-identify the information contained in the ROI.

The processor may be configured to identify the ROI by identifying a person in the scene, estimating the pose of the identified person and determining whether the pose corresponds to a predetermined pose. Predetermined pose may be defined by the system, defined by the operator, or obtained in other ways.

The system may further comprise a 3D imaging system configured to generate the 3D model of the scene. Examples of such systems include video cameras, photo cameras, Light Detection And Ranging of Laser Imaging Detection (LIDAR) and other 3D imaging systems configured to generate a 3D model of a scene. furthermore, a system for remote communication comprising at least two nodes in a network, comprising a parent node and at least one child node, wherein a two-way communication between the parent node and the at least one child node is established is described. The two-way communication comprises de -identification of sensitive information according to any one of the methods according to any of the embodiments of the present application. It is to be understood that a parent node and a child node are interconnected and the information can flow from the child node to the parent node and vice-versa. In some embodiments the parent node can overtake control of the child node. This might be done in e.g., a medical imaging scanning seting in e.g., Computed Tomography and/or Magnetic Resonance Imaging seting.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

For a beter understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

Fig. 1 shows a first method for de -identifying sensitive information in a 3D model of a scene;

Fig. 2 shows a second method for de-identifying sensitive information in a 3D model of a scene;

Fig. 3 shows a third method for de-identifying sensitive information in a 3D model of a scene; and

Fig. 4 shows a system for de-identifying sensitive information in a 3D model of a scene.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The invention will be described with reference to the Figures.

It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become beter understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

The invention provides a method for de-identifying sensitive information. The method comprises obtaining a three dimensional, 3D, model of a scene, identifying a region of interest, ROI, in the 3D model corresponding to sensitive information and modifying the 3D model of the scene to de- identify the sensitive information contained in the ROI.

In a an embodiment, a method for de-identifying sensitive information is described, the method comrises the steps of: obtaining (102) a three dimensional, 3D, model of a scene; identifying (104) a region of interest, ROI, in the 3D model of the scene corresponding to sensitive information; and modifying (106) the 3D model of the scene to de-identify the sensitive information contained in the ROI; and outputting the 3D model including the de-identified sensitive information in the ROI, wherein based on the 3D model output, a real-time two-dimensional, 2D, image of the scene is rendered at a target viewport.

It is to be understood that the present approach works directly on 3D scene representation rather than 2D scene representation that is used in conventional methods. In some cases, the de-identified information is represented by a 3D model that is displayed in target viewport (such as a remote communciation node) directly as a 3D model. In some other cases, the 3D model is converted in a 2D representation of the scene. The latter might be beneficial to establish a direct real-time communication link with a parent node in a parent-child communication system in order to transmit the information in a secure way while preserving the needed information for analysis of situation, such a medical imaging study scan done in an emergency room, while considering specific limitations for such a medical imaging study, such as internet bandwith or processing power of equipment.

Note that the scene could be any scene containing an object, person and/or information which is sensitive. Sensitive information may include the face, or any feature, of a person as well as any information which that person may consider sensitive. Of course, whether information is sensitive or not can often depend the particular case. Thus, the sensitivity of information may need to be assessed on a case-by-case basis. For example, a particular piece of information may, or may not, be sensitive information depending on the receiver.

It is to be understood that if a “vertex” or “vertices” are mentioned, it might refer to vertices alone, alteration of vertices and/or edges, points in point cloud representations or other approaches. “Vertices” is a general term used for simplicity. For instance, in a case of a vertex filter, specific vertices or edges in the image, or a specific part of the image such as a mesh, will be filtered.

It is to be further understood that a replacement of sensitive information with nonsensitive information is applied in some embodiments of the present application. As a non-limiting example, replacement of sensitive information could be done through removal of some points in the point cloud. With this, the privacy features will be preserved as some sensitive information that might lead towards identification of e.g., a person would be removed. As another example, multiple points might be replaced in e.g., replacing a face from the scene with an abstract face from an atlas.

Fig. 1 shows a first method for de -identifying sensitive information in a 3D model. In step 102, a 3D model of a scene is obtained. The 3D model can be obtained directly from a 3D capture/imaging system.

A number of different technologies exist for the real-time 3D capture of an area/scene. These include systems with multiple cameras and laser imaging, detection, and ranging (LIDAR) based systems. These systems can produce a geometric representation of the scene they are imaging (e.g. mesh representation, point cloud representation etc.). In other words, these systems produce a 3D model of the scene. This can then be used to render an image onto a 2D display such that a viewer can move a virtual camerato any place within the scene. The 3D model could also be used to render views for an augmented reality (AR)/virtual reality (VR) device for a more immersive experience. In these cases, two 2D images are rendered at viewports which simulate the positions of the eyes of the user, thereby providing a 3D experience to the user. These immersive environments can be used for remote teaching and education.

The 3D capture system used to obtain the 3D model in step 102 may be one of the already known methods of capturing a scene in real time as discussed above (e.g. LIDAR or camera based system). The 3D capture system will produce a 3D model of the scene (e.g. a room) and all objects inside it as, for example, a textured mesh, point cloud or other volumetric representation. This is done in realtime, similar to a video camera, but rather than producing a 2D video image, it produces a 3D model that can be rendered to a 2D projection from any angle (viewport), or rendered into a virtual environment for viewing in VR/AR. Typically the 3D model will consist of geometric information that defines the shape, size and location of all objects in the space. The 3D model can also include textural information (e.g. color and transparency) that defines the color of the points, or faces, of the geometric information.

In step 104, a region of interest (ROI) in the 3D model is identified. In essence, regions (ROIs) within the 3D model that need to be blurred (or otherwise modified) for privacy reasons, or due to commercial sensitivity, are identified. This may be achieved through a number of methods.

The ROI(s) may be placed at a predetermined location in the scene. In the case where sensitive information/objects cannot move in the scene, the ROI can be placed at a static position in the 3D model. For example, the head location of a patient on a fixed surgical table, or the location of the table of an MRI machine, are fixed objects which can be identified based on their position in the scene. For standardized scenes (e.g. an operating room), the ROI placements can be taken from a database of ROIs.

Visual markers could be attached to locations in the scene that may physically move. This enables the movement of the object/information (e.g., a display) to be tracked as it moves around the scene. As such, the ROI can be placed, and their position modified, based on the tracking of the object/information.

The ROI can also be linked to known information of the scene. For example, in the case where a screen in present in the scene, the information currently being displayed on the screen can determine whether a ROI is placed around the screen. This means that when the screen changes, the ROI can adjust automatically to modify only specific sections of the display. For example, the screen may change from showing non-sensitive information (e.g. heart rate, blood pressure etc.) to showing personal information of the patient (e.g. age, gender etc.). Thus, the ROI can change when a change in the screen is detected. Of course, only certain parts of the screen could be determined to be a ROI if it is known where on the screen the sensitive information will be shown. In some cases, devices in the scene can provide additional information. For example, the position of the table in an MRI scanner can be obtained from the MRI scanner itself. Thus, it can be assumed that location of the patient will be fixed relative to the table. As such, the position of the table, obtained from the MRI scanner, can be used as an anchor point to a ROI covering the patient. Thus, the ROI can track the patient as the table moves into/out of the machine.

Similarly, pose estimation of the people within the scene can be used to identify the ROI. In the case where the scene is an operating room (or other medical scene), the patient is likely the only person in the scene that will be horizontal. As such, when a person is determined to be horizontal (or at least not vertical), one can assume that a ROI can be placed on that person.

The ROI could be determined by using trackable hardware, such as Bluetooth trackers. This would be beneficial if an individual person, in a group of people, does not want to be captured, as this would allow easy determination of the individual people within the room/scene which, for example, do not want to be imaged.

Of course, image recognition algorithms, such as facial recognition or object recognition could be used to identify the ROI. Additionally, an AR headset, or other interface device, can be used by a user to manually specify a 3D region within the space to treat as a ROI.

All ROIs can be stored into a database, such as an onsite or cloud database, for future use.

In step 106, the 3D model is modified to de-identify the information contained in the ROI. This can be referred to as a “privacy enhancement” of the 3D model. Privacy enhancement can be achieved through a number of means, such as blurring, removing and/or replacing the information in the ROI. In one example, the texture values of the ROI can be blurred, removed and/or replaced (e.g. the color of the ROI can be modified). Additionally, or alternatively, the geometry values (e.g. the coordinates of points/meshes in the 3D model) can be blurred, removed and/or replaced thereby to distort the geometry of the ROI in the 3D model.

The texture information of the ROI in the 3D model could be modified whilst keeping the geometry as it is. This would typically be sufficient to remove privacy sensitive information from display screens, for example.

If there is a particular reason for the ROI selection (e.g. ROI metadata specifying the reason for ROI selection), it may possible to replace the information contained in the ROI rather than remove it completely. Sensitive information can comprise, but not limited to, sensitive personal information, such as medical records including e.g., date of birth, disease type and other sensitive information. Furthermore, sensitive information could comprise information that could help in identfying a person, such as faces, tattoos, etc. Furthermore, sensitive information could comprise information that could help to identify a specific 3D scene, such as hospital name, room name, etc. It is to be understood, that these examples are non-limiting representative examples, and other examples can be provided within the context of the present application. For example, in a medical setting, it may be appropriate and/or relevant to know the patient’s approximate age. However, displaying a date of birth would violate privacy. Thus, the date of birth on a screen could be replaced with, for example, a range of ages (e.g. patient is between 60 and 65).

Any personal indicators/features on a person (e.g. tattoos, scars etc.) should preferably be removed and/or replaced (e.g. make the skin appear tattoo-less). This is because, if personal indicators/features are blurred, their position is still apparent in the 3D model. This may be considered sensitive information.

As discussed above, the geometry of the ROI in the 3D model can be modified to deidentify the information in the ROI. For example, the geometry of the ROI could be completely removed from the scene. As such, the sensitive information would not be visible in the 3D model.

Removal of the sensitive information in the 3D model is of particular benefit compared to removal of sensitive information in a 2D image. It means multiple 2D images from any desired viewing location may be generated without requiring further image processing. Removal of the sensitive information in the 3D model also will not cause a blank area in the rendered images where the information was removed (i.e. objects behind the sensitive information will be shown in the rendered images).

Of course, the geometry of the ROI in the 3D model can also be blurred, distorted or replaced to make the sensitive information appear geometrically different in the modified 3D model.

Modification of the geometry of the ROI can be achieved by, for example, modifying the coordinates of the points in a point cloud, modifying the relative position of vertices in a mesh or moving/removing voxels in a volumetric 3D model.

The modified 3D model can then be stored, transmitted or used to render a 2D (deidentified) image of the scene in step 108. For example, if the modified 3D model is not required for immediate use, it can be stored and later used to render 2D images at a target viewport. Of course, the modified 3D model can be used to render real-time 2D images/frames to a user. Thus, the user can view the scene in real-time from a target viewport of their choice and there is no risk of the target viewport containing sensitive information, as the sensitive information has already been modified at the 3D model before any images/frames are rendered.

In some examples, a target viewport refers to a node in a network used in communication. For instance, this could refer to a Radiology Operations Command Center (ROCC) type of device that is provided by Royal Philips. In a ROCC setting there are multiple nodes in a communication and remote support system that are interconnected. For instance, several MR systems (child nodes) might be connected to a parent node (remote support system). In some examples, the target viewport might refer to the parent node. In some other examples, the target viewport might refer to a child viewport. In some embodiments, de-identification of the sensitive information comprises any one of: removal of texture values, vertex removal, vertex filtering, vertex attachment, colour removal in objects of interest, replacement of sensitive information with non-sensitive information, removal of at least some objects of interest, such as removal of at least part of points in the point cloud.

In a non-limiting example, vertex removal might be done through analyzing, by the system, similarity between U and V vertices, obtaining vertex-deleted subgraphs G-U and G-V, comparing the vertex-deleted subgraphs G-U and G-V according to a certain criterion (e.g., isomorphism, degree sequence, index, characteristic polynomial, complementarity spectrum) and removing the at least part of the G-U or G-V certice. It is to be understood that this example is a nonlimiting example for the case of 2 vertices, but in general other vertex removal approaches might be applied. Also, the approaches might be more than for 2 vertices.

In a non-limiting example of a vertex attachment, expanded graphs GU 1 and GV 1 are obtained by attaching a pendant vertex to U and V, respectively.

In a non-limited example, color removal in objects of interest might comprise any one of: colour removal (e.g., completely removing green), colour alteration (e.g., changing the shades of green colour), colour substitution (e.g., substituting green with red colours), or other ways of changing colour. Objects of interest might refer to any objects in an image that contain a colour, and this might range from macro objects (face, limbs, devices, plants, humans, etc.) to micro-level objects (voxels, vertices, points in the cloud point representation, etc.).

Removal of at least some objects of interest might refer to removal of certain objects of interest, such as removal of at least part of points in the point cloud or removal of e.g., faces in the 3D scene. Removal of objects of interest can be enhanced by substitution of removed objects of interest by other objects. For instance, if a face is removed, it can be substituted by another obtained (e.g., from a cloud or atlas) or randomly generated faces (e.g., with the help of a ChatGPT Image Generator).

In some embodiments, vertex removal comprises removal of characteristics of the 3D model, such as a subset of vertices, in the ROI of the 3D model, wherein the ROI corresponding to the removed characteristics of the 3D model is reconstructed using information from another image set not part of the 3D model. Characteristics of 3D model comprise any objects of interest, but it is to be understood that characteristics of the 3D model represent the 3D model using a collection of points in 3D space, connected by various geometric entities such as triangles, lines, curved surfaces, etc, and hence representing an physical object. Whereas objects of interest might comprise even a single vertice. In other words, characteristics of 3D model define an object, whereas objects of interest is a term that encompasses individual elemements of an object of 3D model from which the 3D model or an object in the model is constructed. In the provided example, some information is removed from the 3D scene, and then is substituted by information from another set of images (e.g., from a database of images). In this way, a realistic scene is generated that comprises information that is relevant for a reviewer, but yet does not comprise any sensitive information of e.g., people in the 3D scene. For instance, a 3D model of a scene is obtained, sensitive information is identified and removed, a set of images is obtained that is not part of this 3D model (the current scene), the removed information is substituted with information from another scene, a final image is generated and then provided to a reviewer of the image. In a ROCC setting this an image is obtained from a scene, 3D model is constructed, then the information is substituted, and the image, which looks realistic, but does not contain sensitive information is transmitted to a remote operator in a ROCC device. ROCC is a representative example, and other settings might be envisioned containing parent and child communication.

In some embodiments, vertex filtering is applied through any one of the following filters: lowpass filter, static/dynamic filter, edge filter, graph filter, wherein the any one of the following filters remove non-essential features , while preserving essential features of the model. For instance, it might be beneficial to preserve the general appearance of a human being in an image, such as a patient preserving the volumetric represenation of limbs, torso, face etc. By applying e.g., a static/dynamic filter these volumetric representations might be preserved. Static/dynamic filter utilizes both static and dynamic guidances and may be based on a nonlinear optimization that enforces smoothness of the signal while preserving variations that correspond to features of certain scales. The static/dynamic filter can be used for scale-aware filtering of mesh geometry, allowing us to separate geometry signals according to their scales. In general, such decomposition can be used for manipulating geometric details, boosting or attenuating features at different scales. For instance, vertices or points in a point cloud relating to eyes might be attenuated in order to remove the specific characteristics of eyes, while the vertices or points in a point cloud related to e.g., limbs might be preserved or even enhanced. This selective enhancement might enhance the 3D model in places where it needs to be enhanced (e.g., enhancing injuries of patient), while de -identifying the sensitive information such as eyes. It is to be understood that vertex removal, vertex filtering and other “vertex-related” features generally refer not only to vertex-specific approaches, but also to approaches that are broader than vertices, such as point-cloid representation approaches.

In a preferred embodiment, assuming good ROI selection and a 3D scene as a mesh representation, privacy enhancement is carried out on the subset of vertices / faces that exist within the ROI, in the following way:

Mesh is defined as a list of vertices and faces;

Each vertex (V) has coordinates in space [x,y,z]

Each face has 3 or more vertices [VI, V2, . . . Vn]

Each face has a normal axis defined by the order of vertices.

Each face has either a colour associated with it, or texture information.

If texture is used, each vertex has a corresponding location in the texture map [U,V] Privacy enhancement is then carried out on the subset of vertices / faces that exist within the ROI. Privacy enhancement may be based on vertex removal, where for instance, a random subset of vertices is deleted, and faces are reconstructed to fdl holes in the mesh. New faces can use an average colour of the previous faces, or texture as defined by the remaining vertices.

Other approaches discussed in the present application may be used, such as vertex filtering, wherein a filter may be applied to the coordinates of the vertice for removing the small features, but leaving the coarse features (e.g., smoothing the face, but leaving the head as roughly spherical object).

In some embodiments, static/dynamic filter can be used to removing details from the mesh, but leaving coarse features. Static/dynamic filter can be tuned based on mesh resolution of object to be removed.

In some embodiments, texture and/or colour removal approaches might be used. Texture may be represented by an image, with sections of image applied to e.g., a face. Any typical image manipulation can be used on only those parts of the texture map that represent the faces of the mesh in the ROI. In some embodiments, colours can be smoothed, removed and/or randomly changed. In the case of randomly changed colours, the colours are altered in such a way that the object dimensions are preserved, but yet the the colours are rendered so that sensitive information cannot be identified.

In some embodiments when the 3D scene is represented by a point cloud, the following steps can be performed to de-identify sensitive information.

Assigning a coordinate in space [x,y,z] and a colour [r,g,b] for at least some points in the point cloud representation.

Removing points that may contain sensitive information.

Further filtering the remaining points using e.g., a lowpass flter to remove fine details in sensitive regions.

Optionally removing / modifying colours to furhter enhance privacy.

For a volumetric 3D scene based on voxels the following approach may be used: Dividing the voxels into uniform cubes to construct geometry.

For at least some cubes defining a texture of the cube.

Fine detail can be removed by decreasing the voxel resolution, averaging neighbouring voxels, etc.

These are some representative non-limiting examples, and other examples may be envisioned within the context of the present application.

Fig. 2 shows a second method for de-identifying sensitive information from a 3D capture system. The method shown in Fig. 2 contains all of the steps of the method of Fig. 1. However, in this second method, a verification step 202 has been added between the privacy enhancement step 106 and the storing/rendering step 108. Rendering, or image synthesis, refers to the process of generating a photorealistic or non-photorealistic image from a 2D/3D model by means of a computer program. The verification step 202 is used to verify the correct detection, identification and application of the ROIs in the 3D model. For example, this could be achieved by having a trusted third party observer manually checking that the ROIs are in the correct place in real-time. Of course, the verification step 202 can be performed afterwards if the modified 3D model is stored for later use. If the verification step 202 determines a ROI is not correct (e.g. it is in the wrong position, or a ROI is present where there shouldn’t be one), the ROI can be modified in step 204 to ensure it is correct. For example, modifying the ROI may include modifying the position of the ROI to the correct position, corresponding to the sensitive information, or removing a ROI which does not correspond to sensitive information. Of course, the verification step 202 may also lead to the identification of further ROIs which may not have been previously identified.

In some embodiments, receiving (202) a verification that the ROI is in the correct position for scanning as indicated by the operator of the device relative to the position of the sensitive information in the 3D model is being received. In this case the operator (in the child or parent nodes) or a trusted third-party observer can directly verify that the scanning is done in a correct position.

Fig. 3 shows a third method for de -identifying sensitive information from a 3D capture system. The method shown in Fig. 3 contains all of the steps of the method of Fig. 1. However, in this method, an additional, non-modified image is also obtained from an additional 2D capture system (e.g. a conventional camera) independently from the 3D model in step 302.

Using the viewport of the camera in the 3D scene (i.e. the angle and position of the camera), the position of the ROI can be projected/converted into a 2D area in the non-modified image which corresponds to the ROI in step 304. Thus, the area of the non-modified image corresponding to the ROI can be modified (e.g. blurred, removed and/or replaced) to de-identify the non-modified image in step 306. As such, a user can choose to render a 2D image from the modified 3D model or view the now- modified image at the camera viewport. The modified image can also be stored together with the modified 3D model.

For example, the additional camera may be a body-worn camera on a surgeon. The feed from this additional camera also needs to be privacy enhanced. This can be achieved by translating the ROI from the 3D domain in the 3D model to the 2D domain of the additional camera so long as the relative position of the additional camera within the scene is known. Thus, a user can view the scene from the camera viewport as obtained by the additional camera.

Fig. 4 shows a system for de-identifying sensitive information in a 3D model of a scene. A 3D capture system 402 is used to capture/generate a 3D model of the scene it is imaging. For example, a plurality of cameras may be used by the 3D capture system. The images from the plurality of cameras can be registered to each other (e.g. from knowledge of the relative positions of the cameras) and used to generate a 3D geometry (e.g. depth map, point clouds, meshes or otherwise) of the scene. The 3D model 404 comprises the 3D geometry of the scene and may comprise the texture values (e.g. color and transparency) of objects in the scene. Of course, other 3D imaging methods could also be used (e.g. LIDAR).

The 3D model 404 is fed into a processor 406 which identifies the ROIs in the scene and modifies them accordingly, thereby to de-identify sensitive information. This may be achieved by checking a ROI database 408 indicating possible positions and/or objects to look out for in the scene. Of course, when a ROI is detected, this can be fed back into the ROI database 408.

As such, the processor can output a modified, de-identified, 3D model 410 of the scene. The modified 3D model 410 can now be used to render a modified, de-identified, 2D image 412 of the scene. Of course, in the case of AR and/or VR, the 3D model can be used to render two separate 2D images of the scene simulating two views from the eyes of a user, thereby generating an immersive experience when viewed on an AR/VR headset.

As will be appreciated, the concepts described herein are useful for any situation where 3D scene capture is used, particularly for remote education, training and monitoring in clinical spaces.

In education and research, for example, video recordings are already used for medical students and healthcare professionals. Videos of diagnoses and treatments are used as examples for these users to learn from. As these videos are commonly distributed, the confidentiality of subjects in the video can be maintained by the concepts proposed herein.

In diagnosis and treatment, 3D scene capture allows a remote expert to guide a local health-care professional. The proposed solution ensures data confidentiality of the patient, staff and any others (e.g. family) to be maintained.

Of course, non-medical 3D captures can also benefit from de-identification at the 3D model, such as court rooms. In the case of a 3D capture of a court room, for example, the witness stand may be identified as a ROI and thus the witnesses can be de-identified in the 3D model.

The invention may be applied to 3D imaging of any space containing information which is to be de-identified, before images are rendered from the 3D model created by the 3D imaging system.

Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.

It is to be understood that any of the previous steps described in relation to embodiments and/or training steps described above can be performed by a specific-purpose computer system or general-purpose computer system, or a computer-readable medium, or data carrier system configured to carry out any of the steps described previously. The computer system can include a set of software instructions that can be executed to cause the computer system to perform any of the methods or computer-based functions disclosed herein. The computer system may operate as a standalone device or may be connected, for example using a network, to other computer systems or peripheral devices. In embodiments, a computer system performs logical processing based on digital signals received via an analogue-to-digital converter.

Some portions of the description are presented in terms of symbolic representations of operations on non-transient signals stored within a computer memory. These descriptions and representations are used by those skilled in the data processing arts to convey the substance of their work most effectively to others skilled in the art. Such operations typically require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times to refer to certain arrangements of steps requiring physical manipulation of physical quantities as modules or code devices, without loss of generality.

However, all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage devices. Portions of the present disclosure include processes and instructions that may be embodied in software, firmware, or hardware, and when embodied in software, may be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

In a networked deployment, the computer system operates in the capacity of a server, or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer or distributed network environment. The computer system can also be implemented as or incorporated into various devices, such as a server or another type of computer such as a workstation that includes a controller, a stationary computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a set of software instructions sequentially or non-sequentially that specify actions to be taken by that machine. The computer system can be incorporated as an integrated system part of a larger system that includes additional devices. In an embodiment, the computer system can be implemented using electronic devices that provide voice, video, or data communication possibilities. Further, while the computer system is illustrated in the singular, the term “system” shall also be taken to include any collection of systems or sub-systems that individually or jointly execute a set or multiple sets, of software instructions to perform one or more computer functions. The computer system may also include a processor. The processor executes instructions to implement some, or all aspects of methods and processes described herein. The processor is tangible and non-transitory. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non- transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The processor is an article of manufacture and/or a machine component. The processor is configured to execute software instructions to perform functions as described in the various embodiments herein. The processor may be a general- purpose processor or may be part of an application specific integrated circuit (ASIC). The processor may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device, a logical circuit, including a programmable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit that includes discrete gate and/or transistor logic. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices. The processor can include one or more internal levels of cache, and a bus controller or bus interface unit to direct interaction with a bus. The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. References to a computing device comprising “a processor” should be interpreted to include more than one processor or processing core, as in a multi -core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted to include a collection, or network, of computing devices each including a processor or processors. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices. Further, the software instructions, when executed by the processor, perform one or more steps of the methods and processes as described herein.

The computer system further includes a main memory and a static memory, where memories in the computer system communicate with each other and the processor via a bus. Either or both main memory and the static memory may be considered representative examples of the memory of the controller, and store instructions used to implement some, or all aspects of methods and processes described herein. Memories described herein are tangible storage mediums for storing data and executable software instructions and are non-transitory during the time software instructions are stored therein. The main memory and the static memory are articles of manufacture and/or machine components. The main memory and the static memory are computer-readable mediums from which data and executable software instructions can be read by a computer (or e.g., the processor). Each of the main memory and the static memory may be implemented as one or more of random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM (Compact Disk - Read Only Memory)), digital versatile disk (DVD), floppy disk, Blu-ray disk, or any other form of storage medium known in the art. The memories may be volatile or non-volatile, secure and/or encrypted, unsecure and/or unencrypted.

“Memory” is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include RAM memory, registers, and register files. References to “computer memory” or “memory” should be interpreted as possibly being multiple memories. The memory may for instance be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices. The memory may store various software applications including computer executable instructions, that when run on the processor, implement the methods and systems set out herein. Other forms of memory, such as a storage device and a mass storage device, may also be included and accessible by the processor (or processors) via the bus. The storage device and mass storage device can each contain any or all of the methods and systems discussed herein.

The computer system can further include a communications interface by way of which the computer system can connect to networks and receive data useful in executing the methods and system set out herein as well as transmitting information to other devices. The computer system further includes a video display unit as an output device by which information can be output, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT), for example. Additionally, the computer system includes an input device, such as a keyboard/virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and a cursor control device, such as a mouse or touch-sensitive input screen or pad. The computer system also optionally includes a disk drive unit, a signal generation device, such as a speaker or remote control, and/or a network interface device.

The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform one or more method steps. The structure for a variety of these systems is discussed in the description below. In addition, any programming language that is sufficient for achieving the techniques and implementations of the present disclosure may be used. In addition, the language used in the specification has been principally selected for readability and instructional purposes and may not have been selected to delineate or circumscribe the disclosed subject matter. Accordingly, the present disclosure is intended to be illustrative, and not limiting, of the scope of the concepts discussed herein.

In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes software programs. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component/object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functionalities as described herein, and a processor described herein may be used to support a virtual processing environment.

The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to fully describe all the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

Although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art.

The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents and shall not be restricted or limited by the foregoing detailed description.

Functions implemented by a processor may be implemented by a single processor or by multiple separate processing units which may together be considered to constitute a “processor”. Such processing units may in some cases be remote from each other and communicate with each other in a wired or wireless manner.

The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. If the term “adapted to” is used in the claims or description, it is noted the term “adapted to” is intended to be equivalent to the term “configured to”. If the term “arrangement” is used in the claims or description, it is noted the term “arrangement” is intended to be equivalent to the term “system”, and vice versa. Any reference signs in the claims should not be construed as limiting the scope.