Inventors: Ping-Heng Kuo and Ralf Rossbach

Priority/ Incorporation By Reference

[0001] This application claims priority to U.S. Provisional Application Serial No. 63/376, 643 filed on September 22, 2022 and entitled "Methods for AI/ML Models Training and Reporting in Wireless Networks," the entirety of which is incorporated herein by reference.

Background

[0002] 5G New Radio (NR) has introduced many radio access network (RAN) and core network (CN) enhancements, as well as an enhanced security architecture. Artificial intelligence (Al) and/or machine learning (ML) processes, e.g., deep learning neural networks, may be used to facilitate and optimize certain decision makings in one or more network functionalities (e.g., in the RAN or CN) . For example, the use cases for AI/ML for the air interface include channel state information (CSI) feedback enhancement (e.g. , overhead reduction, improved accuracy, prediction) ; beam management (e.g. , beam prediction in time, and/or spatial domain for overhead and latency reduction, beam selection accuracy improvement) ; and positioning accuracy enhancements. Additionally, the AI/ML services can be used by applications at the UE, the RAN, or external to the UE/RAN (e.g., Al-as-a-Service (AlaaS) .

[0003] In any of these use cases, one or multiple UEs served by the RAN, or the RAN itself (e.g. , a RAN node such as a gNB) , can function as an Al agent that trains all or part of the AI/ML model (s) . For example, a UE can train the model based on, e.g. , data collected by the UE (e.g. , radio-related measurements, application-related measurements, sensor input, etc. ) . In Federated Learning (FL) use cases, multiple UEs may report/transf er respective trained models to the RAN for model fusion/aggregation . Some FL applications include autonomous driving or autonomous railway.

[0004] In many scenarios, it is crucial to ensure that the trained AI/ML models meet a minimum required quality and can be trusted by the clients (e.g. , network functions, UEs and/or external applications) using the AI/ML services. Particularly in FL use cases, if the training results from one Al agent do not meet the required quality, the aggregated global model may become misleading. For critical applications (e.g. , autonomous driving) , a poor quality AI/ML model can have disastrous effects .

Summary

[0005] Some further exemplary embodiments are related to a processor of an artificial intelligence (Al) agent configured to perform operations. The operations include collecting a dataset for training an Al or machine learning (ML) (AI/ML) model, determining context information for the collected dataset, an AI/ML training method to be used, or a metric related to a trustworthiness of a previously trained AI/ML model and, prior to either training the AI/ML model or reporting a trained AI/ML model, reporting the context information to an Al manager.

[0006] Other exemplary embodiments are related to a processor of an artificial intelligence (Al) manager configured to perform operations. The operations include receiving, from an Al agent, context information for a dataset collected by the Al agent to train an Al or machine learning (ML) (AI/ML) model, an AI/ML training method to be used, or a metric related to a trustworthiness of a previously trained AI/ML model, determining, based on the context information, whether one or more criteria are satisfied and based on whether the criteria are satisfied, transmitting a positive response or a negative response to the Al agent regarding whether to train the AI/ML model or report a trained AI/ML model.

Brief Description of the Drawings

[0007] Fig. 1 shows a network arrangement according to various exemplary embodiments.

[0008] Fig. 2 shows an exemplary UE according to various exemplary embodiments.

[0009] Fig. 3 shows a method for selective AI/ML model training and reporting based on evaluated metrics related to a trustworthiness or quality for the AI/ML model according to various exemplary embodiments.

[0010] Fig. 4 shows a method for selective AI/ML model training and reporting based on criteria related to the validity of a training dataset and/or training methods for the AI/ML model according to various exemplary embodiments.

[0011] Fig. 5 shows a method for AI/ML model training adaptation based on performance feedback according to various exemplary embodiments. [0012] Fig. 6 shows a method for multi-stage training of a global AI/ML model from multiple partial models according to various exemplary embodiments.

Detailed Description

[0013] The exemplary embodiments may be further understood with reference to the following description and the related appended drawings, wherein like elements are provided with the same reference numerals. The exemplary embodiments describe operations for ensuring that artificial intelligence (Al) and/or machine learning (ML) models trained by Al agents in a network are trustworthy with regard to quality. The exemplary embodiments provide signaling and reporting mechanisms for providing an Al manager or consumer with information sufficient to determine that an AI/ML model trained remotely by an Al agent can be trusted.

[0014] In some aspects, the Al manager (e.g., 5G NR RAN or a network-side function) can indicate to the Al agent (e.g., UE) one or more types of metrics and/or parameters for the Al agent to evaluate regarding the Al model trained (or to be trained) by the Al agent. These metrics can indicate the trustworthiness of the Al model. For example, the Al agent can be instructed to evaluate a confidence level for the AI/ML model (e.g., low, medium or high confidence) or an accuracy metric related to the inferencing error of the AI/ML model. In another example, the Al agent can be provided with certain criteria to evaluate regarding the dataset used to train the AI/ML model, e.g., a size of, age of, or method for collecting the data used to train the model, prior to training and/or reporting the Al model. In other aspects, the Al agent can evaluate these metrics/criteria without an explicit indication from the Al manager. [0015] Still other aspects of these exemplary embodiments relate to performance feedback operations and multi-stage training operations coordinated by the Al manager.

[0016] The exemplary aspects are described with regard to a UE . However, the use of a UE is provided for illustrative purposes. The exemplary aspects may be utilized with any electronic component that may establish a connection with a network and is configured with the hardware, software, and/or firmware to exchange information and data with the network. Therefore, the UE as described herein is used to represent any electronic component that is capable of accessing a wireless network and performing AI/ML training or inferencing operations.

[0017] The exemplary aspects are described with regard to the network being a 5G New Radio (NR) network and a base station being a next generation Node B (gNB) . However, the use of the 5G NR network and the gNB are provided for illustrative purposes. The exemplary aspects may apply to any type of network that utilizes similar functionalities. For example, some AI/ML operations can be RAT-independent .

[0018] The exemplary embodiments are further described with regard to artificial intelligence (Al) and/or machine learning (ML) based operations. Any number of different AI/ML models may be used, depending on UE and network implementation. For example, in some embodiments, advanced AI/ML technigues (e.g., a deep learning neural network (NN) ) may be used while in other embodiments simpler AI/ML techniques (e.g., a decision tree) may be used. Further, the various types of models may use different types of data for training the model, including, e.g., radio- related measurements, application-related measurements or sensor data. Thus, reference to any particular AI/ML-based model is provided for illustrative purposes. The exemplary aspects described herein may apply to any type of AI/ML-based modeling that uses a training phase and an inference phase that can be executed at a UE, a RAN (e.g. , a network node such as a base station) , and/or a network-side function or entity (e.g., a core network element such as a location management function (LMF) for providing UE positioning services; an application server; etc. ) .

[0019] In some embodiments, the Al agent can be a user equipment (UE) in the 5G New Radio (NR) radio access network (RAN) while in other embodiments, the Al agent is a node of the RAN (e.g., a gNB) or a network-side entity, e.g., the core network, RAN or an application server. It should be understood that the techniques described herein may be used regardless of whether the Al agent is a UE, the RAN, or a network-side node and regardless of whether the Al manager is a UE, the RAN, or a network-side node. Those skilled in art will ascertain that the methods by which the UE, RAN or network-side node are enabled with Al agent or Al manager functionalities are varied and can depend on any combination of preconfigured functionalities, RAN conf iguration/ indication, CN entity conf iguration/indication, UE indication, etc.

[0020] Thus, although some techniques are described with respect to a UE being enabled for Al agent functionalities and a RAN (or network-side) node being enabled for Al manager functionalities, any one of the aforementioned entities can serve as the Al manager (e.g. , providing one or more types of metrics, assistance information, etc. ) or as the Al agent (e.g. , training the model, evaluating the metrics, and reporting the trained model) . Additionally, in some scenarios, the Al agent and the Al manager can both be network-side nodes or functionalities (e.g., the Al agent is a base station and the Al manager is a core network entity) or can both be UEs (e.g. , the Al agent is a first UE and the Al manager is a second UE connected to the first UE via a sidelink) .

[0021] Fig. 1 shows an exemplary network arrangement 100 according to various exemplary embodiments. The exemplary network arrangement 100 includes a user equipment (UE) 110. Those skilled in the art will understand that the UE may be any type of electronic component that is configured to communicate via a network, e.g. , mobile phones, tablet computers, smartphones, phablets, embedded devices, wearable devices, Cat-M devices, Cat-Mi devices, MTC devices, eMTC devices, other types of Internet of Things (loT) devices, etc. It should also be understood that an actual network arrangement may include any number of UEs being used by any number of users. Thus, the example of a single UE 110 is merely provided for illustrative purposes .

[0022] The UE 110 may communicate directly with one or more networks. In the example of the network configuration 100, the networks with which the UE 110 may wirelessly communicate are a 5G NR radio access network (5G NR-RAN) 120, an LTE radio access network (LTE-RAN) 122 and a wireless local access network (WLAN) 124. Therefore, the UE 110 may include a 5G NR chipset to communicate with the 5G NR-RAN 120, an LTE chipset to communicate with the LTE-RAN 122 and an ISM chipset to communicate with the WLAN 124. However, the UE 110 may also communicate with other types of networks (e.g. , legacy cellular networks) and the UE 110 may also communicate with networks over a wired connection. With regard to the exemplary aspects, the UE 110 may establish a connection with the 5G NR-RAN 122.

[0023] The 5G NR-RAN 120 and the LTE-RAN 122 may be portions of cellular networks that may be deployed by cellular providers (e.g., Verizon, AT&T, T-Mobile, etc.) . These networks 120, 122 may include, for example, cells or base stations (Node Bs, eNodeBs, HeNBs, eNBS, gNBs, gNodeBs, macrocells, microcells, small cells, femtocells, etc.) that are configured to send and receive traffic from UEs that are equipped with the appropriate cellular chip set. The WLAN 124 may include any type of wireless local area network (WiFi, Hot Spot, IEEE 802. llx networks, etc . ) .

[0024] The UE 110 may connect to the 5G NR-RAN via at least one of the next generation nodeB (gNB) 120A and/or the gNB 120B. Reference to two gNBs 120A, 120B is merely for illustrative purposes. The exemplary aspects may apply to any appropriate number of gNBs.

[0025] In addition to the networks 120, 122 and 124 the network arrangement 100 also includes a cellular core network 130, the Internet 140, an IP Multimedia Subsystem (IMS) 150, and a network services backbone 160. The cellular core network 130, e.g., the 5GC for the 5G NR network, may be considered to be the interconnected set of components that manages the operation and traffic of the cellular network. The cellular core network 130 also manages the traffic that flows between the cellular network and the Internet 140. The core network 130 may include, e.g., a location management function (LMF) to support location determinations for a UE . [0026] The IMS 150 may be generally described as an architecture for delivering multimedia services to the UE 110 using the IP protocol. The IMS 150 may communicate with the cellular core network 130 and the Internet 140 to provide the multimedia services to the UE 110. The network services backbone 160 is in communication either directly or indirectly with the Internet 140 and the cellular core network 130. The network services backbone 160 may be generally described as a set of components (e.g. , servers, network storage arrangements, etc. ) that implement a suite of services that may be used to extend the functionalities of the UE 110 in communication with the various networks .

[0027] Fig. 2 shows an exemplary UE 110 according to various exemplary embodiments. The UE 110 will be described with regard to the network arrangement 100 of Fig. 1. The UE 110 may represent any electronic device and may include a processor 205, a memory arrangement 210, a display device 215, an input/output (I/O) device 220, a transceiver 225, and other components 230. The other components 230 may include, for example, an audio input device, an audio output device, a battery that provides a limited power supply, a data acquisition device, ports to electrically connect the UE 110 to other electronic devices, sensors to detect conditions of the UE 110, etc. Additionally, the UE 110 may be configured to access an SNPN.

[0028] The processor 205 may be configured to execute a plurality of engines for the UE 110. For example, the engines may include an AI/ML engine 235 for performing various operations related to training an AI/ML model (as an Al agent) or facilitating the training and generation of a trained AI/ML model via one or more remote Al agents (as an Al manager) . In some embodiments, when the UE 110 is the Al agent, the AI/ML engine 235 may assess a trustworthiness of an AI/ML model trained (or to be trained) by the UE 110. These operations will be described in greater detail below.

[0029] The above referenced engine being an application (e.g., a program) executed by the processor 205 is only exemplary. The functionality associated with the engines may also be represented as a separate incorporated component of the UE 110 or may be a modular component coupled to the UE 110, e.g., an integrated circuit with or without firmware. For example, the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information. The engines may also be embodied as one application or separate applications. In addition, in some UEs, the functionality described for the processor 205 is split among two or more processors such as a baseband processor and an applications processor. The exemplary aspects may be implemented in any of these or other configurations of a UE .

[0030] The memory 210 may be a hardware component configured to store data related to operations performed by the UE 110. The display device 215 may be a hardware component configured to show data to a user while the I/O device 220 may be a hardware component that enables the user to enter inputs. The display device 215 and the I/O device 220 may be separate components or integrated together such as a touchscreen.

[0031] The transceiver 225 may be a hardware component configured to establish a connection with the 5G-NR RAN 120, the LTE RAN 122 etc. Accordingly, the transceiver 225 may operate on a variety of different frequencies or channels (e.g. , set of consecutive frequencies) . The transceiver 225 includes circuitry configured to transmit and/or receive signals (e.g. , control signals, data signals) . Such signals may be encoded with information implementing any one of the methods described herein. The processor 205 may be operably coupled to the transceiver 225 and configured to receive from and/or transmit signals to the transceiver 225. The processor 205 may be configured to encode and/or decode signals (e.g., signaling from a base station of a network) for implementing any one of the methods described herein.

[0032] The exemplary network base station, in this case gNB 120A, may represent a serving cell for the UE 110. The gNB 120A may represent any access node of the 5G NR network through which the UE 110 may establish a connection and manage network operations. The gNB 120A may include a processor, a memory arrangement, an input/output (I/O) device, a transceiver, and other components. The other components may include, for example, an audio input device, an audio output device, a battery, a data acquisition device, ports to electrically connect the gNB 120A to other electronic devices, etc. The functionality associated with the processor of the gNB 120A may also be represented as a separate incorporated component of the gNB 120A or may be a modular component coupled to the gNB 120A, e.g., an integrated circuit with or without firmware. For example, the integrated circuit may include input circuitry to receive signals and processing circuitry to process the signals and other information. In addition, in some gNBs, the functionality described for the processor is split among a plurality of processors (e.g. , a baseband processor, an applications processor, etc. ) . The exemplary aspects may be implemented in any of these or other configurations of a gNB. [0033] The memory may be a hardware component configured to store data related to operations performed by the UEs 110, 112. The I/O device may be a hardware component or ports that enable a user to interact with the gNB 120A. The transceiver may be a hardware component configured to exchange data with the UE 110 and any other UE in the system 100. The transceiver may operate on a variety of different freguencies or channels (e.g., set of consecutive frequencies) . Therefore, the transceiver may include one or more components (e.g. , radios) to enable the data exchange with the various networks and UEs. The transceiver includes circuitry configured to transmit and/or receive signals (e.g., control signals, data signals) . Such signals may be encoded with information implementing any one of the methods described herein. The processor may be operably coupled to the transceiver and configured to receive from and/or transmit signals to the transceiver. The processor may be configured to encode and/or decode signals (e.g. , signaling from a UE) for implementing any one of the methods described herein.

[0034] Artificial Intelligence (Al) and Machine Learning (ML) is envisioned to be an integral part of Beyond 5G (B5G) (Rel-18 and beyond) , as well as 6G. In particular, AI/ML may play a role for the optimization of network functionalities. AI/ML models trained by the Al agent (s) in the network may be used to facilitate certain decision makings in one or more network functionalities (e.g., in RAN or Core Network) , including but not limited to: beam management; positioning, resource allocation; network management (operation and management (0AM) ) ; route election; energy saving; and load Balancing. In addition, in Al-as-a-Service (AlaaS) , the AI/ML services can be consumed by applications initiated at either the user or network side. The trained AI/ML model can be provided by any Al agent reachable in the network, including the UE . In various use cases, one or more UEs in a network may function as Al agents who can train at least a part of AI/ML models based on, e.g., data collected locally by each UE (e.g., radio-related or application-related measurements, sensor input, etc.) .

[0035] When the AI/ML model is trained by the UE for provision by the network as services to be consumed by some functions externally instantiated (e.g., on the network side or in an application server) , the UE needs to report/transf er the trained models to the network. Similarly, when Federated Learning (FL) is used, the UE reports/transf ers the trained models to the network for model fusion.

[0036] FL operation for the 5G system is specified in 3GPP TS 22.261. In FL, a cloud server hosting a model aggregator trains a global model by aggregating local models partially trained by multiple end devices, e.g., UEs. Within each training iteration, a UE downloads an untrained model from the Al server and performs the training based on local training data. The UE reports the interim training results to the cloud server via 5G UL channels and the server aggregates the interim training results from the UEs and updates the global model. The updated global model is then distributed back to the UEs and the UEs can perform the training for the next iteration.

[0037] In many scenarios, it is crucial to ensure that the trained AI/ML models meet a minimum required quality and can be trusted by the clients (e.g., network functions, UEs and/or external applications) using the AI/ML services. Particularly in FL use cases, if the training results from one Al agent do not meet the required quality, the aggregated global model may become misleading. For critical applications (e.g., autonomous driving) , a poor quality AI/ML model can have disastrous effects .

[0038] The quality of a trained Al model can be assessed in a variety of manners. For example, key metrics of model quality relate to accuracy, robustness, stability and data quality. The accuracy of a trained Al model can be assessed by performing an error analysis using test examples to compare expected (known) results with the inferencing results generated by the trained Al model. If the inferencing error (or probability of inferencing error) is sufficiently high, the parameters of the model may be adjusted or the model may be retrained to achieve a higher degree of accuracy. In other examples, the robustness of the model can be assessed by subjecting the model to large variances in input data, e.g., to simulate poor input data, and the stability of the model can be assessed by determining the consistency in the results when only small variances are applied in the input data. The data quality relates to attributes such as the size, age and source of the training data set.

[0039] The quality or trustworthy level of an AI/ML model may be influenced by the following factors (not an exhaustive list) : the size of the dataset used for model training; the age of the dataset used for model training; the collection method of the dataset used for model training; the correctness of the dataset used for model training; the "integrity" of the dataset collection; the algorithm used for model training; and other factors. Thus, in the context of this description it may be considered that trustworthy or trustworthiness may also be synonymous with "valid," "adequate" or "integrity." [0040] It is crucial to ensure the AI/ML models trained at an Al agent can meet a minimum required quality, and therefore can be trusted by the clients (including, e.g. , UEs, RAN nodes, network functions and/or external applications) using AI/ML services .

[0041] According to various exemplary embodiments described herein, operations are described for ensuring AI/ML models trained by Al agents are trustworthy with respect to quality, and therefore can be applied for inferencing by other network functions and/or third-party applications. It should be understood that the exemplary embodiments may be described with respect to the Al agent (the entity training/reporting the AI/ML model) being a UE . However, certain aspects of the present disclosure may be applicable to other entities serving as the Al agent, e.g. , a RAN node or network-side node, as described further below. Additionally, in some scenarios, the Al agent and the Al manager can both be network-side nodes or functionalities (e.g., the Al agent is a base station and the Al manager is a core network entity) or can both be UEs (e.g. , the Al agent is a first UE and the Al manager is a second UE connected to the first UE via a sidelink) . Thus, the Al agent (or Al agent node) can refer to any type of UE or network node and the Al manager (or Al manager node) can refer to any type of UE or network node.

[0042] The exemplary embodiments provide signaling and reporting mechanisms for providing an Al manager or consumer with information sufficient to determine that an AI/ML model trained remotely by an Al agent can be trusted. In some aspects, the Al manager (e.g. , 5G NR RAN or a network function) can indicate to the Al agent (e.g., UE) one or more types of metrics for the Al agent to evaluate regarding the Al model trained (or to be trained) by the Al agent. For example, the Al agent can be instructed to evaluate a confidence level for the AI/ML model or a metric related to the inferencing error of the AI/ML model. In another example, the Al agent can be provided with certain criteria to evaluate, e.g., a size of, age of, or method of data collection, prior to training and/or reporting the Al model. In other aspects, the Al agent can evaluate these metrics/criteria without an explicit indication from the Al manager. Still other aspects of these exemplary embodiments relate to performance feedback operations and multi-stage training operations coordinated by the Al manager.

[0043] According to one aspect of these exemplary embodiments, one or more metrics relating to the trustworthiness of the AI/ML model may be determined by the Al agent (e.g., the UE) and reported or provided to the Al manager/ consumer in association with the trained AI/ML model. Based on the reported metrics, the Al manager (e.g., the 5G NR RAN) can determine whether the trained model has a sufficient quality or trustworthiness to be used for inferencing. In some embodiments, the metrics can relate to the accuracy of the trained AI/ML model and include, e.g., the probability that the inferencing error of the AI/ML model exceeds a threshold; the probability distribution parameter (s) of the inferencing error of the AI/ML model (e.g., the mean and standard deviation, the type of distribution, etc.) ; or the maximum possible value of the inferencing error of this AI/ML model.

[0044] In other embodiments, the metric can be an integer value that marks the overall confidence level of this AI/ML model. For example, the confidence level can be selected from among values indicating low confidence, medium confidence or high confidence (e.g., 0=Low, l=Medium, 2=High) . Those skilled in the art will ascertain that additional values can also be used, or the indication can be a binary flag, e.g., trustable or not trustable.

[0045] The method by which the Al agent (e.g., UE) evaluates these metrics for trustworthiness may be based on the particular implementation of the node (e.g., the evaluation algorithm is not mandated by specifications) . To ensure that the Al manager entity can trust that the Al agent entity will use trustworthiness evaluation methods that are acceptable to the Al manager, security certificate ( s ) may be exchanged between the Al agent and the Al manager prior to evaluating the metric and/or training the AI/ML model.

[0046] The Al manager may indicate the type of trustworthy level metric to be evaluated before the Al agent initiates its training functionalities so that the Al agent knows what metric should be evaluated and reported. In some embodiments, the Al agent may report the trustworthy level metrics only when the evaluated metrics meet (or fail to meet) certain conditions, e.g., when the trustworthy level is lower than a threshold. In one example, when the AI/ML model is evaluated by the Al agent to be trustworthy, the Al agent can skip the reporting of such metric (s) . In this example, if the model is evaluated to be not trustworthy, the Al agent can provide the metric (s) to the Al manager so that the Al manager can, e.g., suggest ways to improve the training of the Al model. In another example, when the AI/ML model is evaluated by the Al agent to be trustworthy, the Al agent can report the trustworthy level/metric . This information can be used by the Al manager to, e.g., select a group of models with very high trustworthy levels as a first group of partial models to fuse into an aggregated global model (e.g., in federated learning (FL) operations) . In still another example, the AI/ML model and the associated metrics can be reported automatically and regardless of the values of the evaluated metrics.

[0047] The Al manager may provide some assistance information, e.g., parameters relating to acceptable or unacceptable trustworthiness metrics, for the Al agent to evaluate the trustworthy level metrics. For example, the Al agent may be provided with a targeted inferencing error, e.g., the maximum inferencing error that can be tolerated. In another example, the Al agent may be provided with a threshold of inferencing error, e.g., when the trustworthy level metric is to be characterized by the probability where the inferencing error of the AI/ML model exceeds a threshold.

[0048] In still another example, the assistance information can comprise parameters relating to the dataset collection by the Al agent. For example, in positioning methods where the Al manager is the 5G NR RAN (or the LMF in the core network) and the Al agent is the UE, the Al manager may first provide satellite health conditions if the one or more entries in the dataset corresponds to GNSS positioning. The UE can consider this assistance information when assessing the trustworthy metric, e.g., an integer value associated with a confidence level for the model quality (e.g., low, medium or high confidence) . [0049] In another aspect, the Al agent can evaluate the one or more metrics and based on the evaluation, determine whether the AI/ML model should be trained and/or reported. In these aspects, the Al agent may be provided with an indication of the type of metric to be evaluated and determine whether a threshold of trustworthiness is satisfied based on the implementation of the Al agent (e.g., UE implementation) , similar to above. For example, the metric may be a confidence measure, e.g., a low, medium or high level of trustworthiness, or a probability (or probability distribution parameter) for an inferencing error of the AI/ML model.

[0050] If the targeted AI/ML model is not yet trained, the Al agent can evaluate whether it is able to obtain a model that can satisfy the one or more pre-configured trustworthy level threshold/metric based on, e.g., the characteristics of its training dataset. If the Al agent determines it can satisfy the metric, the UE may proceed to train the AI/ML model. If the Al agent is configured to train multiple models, the Al agent may determine which model should be trained based on which preconfigured threshold/condition is satisfied. If the Al agent determines it cannot satisfy the metric, the Al agent may choose not to train a model, and/or it can wait until a qualified dataset is collected, and then train the model accordingly.

[0051] Alternatively, even when the trustworthy metric is not satisfied for a model yet to be trained, the UE may still train/report the model and indicate the "achievable" trustworthy level of the trained model based on the evaluation prior to training . [0052] If the targeted AI/ML model is already trained, the UE can evaluate whether the trained model can satisfy the preconfigured trustworthy level threshold (based on, e.g., the characteristics of the dataset that has been used to train the model) . If the UE determines it can satisfy the metric, the UE may proceed to report the trained model. If the UE determines it cannot satisfy the metric, the UE may choose to skip reporting .

[0053] Fig. 3 shows a method 300 for selective AI/ML model training and reporting based on evaluated metrics related to a trustworthiness or quality for the AI/ML model according to various exemplary embodiments. In this example, the Al manager comprises the 5G NR RAN or a network-side functionality instantiated externally to the RAN (e.g., a core network entity or application server) and the Al agent comprises a UE .

[0054] In 305, the UE is enabled as an Al agent for training and reporting an AI/ML model. It should be understood that certain aspects of the Al agent functionalities can be preconfigured, while other aspects of the Al agent functionalities can be indicated to or configured for the UE by the network. In one example, the UE can be hard encoded with features that enable the training of one or more types of AI/ML models. In another example, the UE can download an untrained AI/ML model from the RAN. In still another example, the UE can first exchange capability-related information (and/or a security certificate) with the RAN prior to receiving a configuration from the network that activates one or more AI/ML training techniques . [0055] Depending on the type of AI/ML model to be trained, the UE may receive additional configurations from the network. For example, if the AI/ML model relates to channel estimation, the UE may be configured with a training set of reference signals (RS) to measure and use to train the model. In another example, if the AI/ML model relates to positioning, the UE may be configured with a traditional positioning method (e.g., GNSS or OTDOA) to use to gather positioning data for training the model. Those skilled in the art will understand the types of AI/ML models that can be received and trained by the UE are varied and the Al agent functionalities can be enabled for the UE in any number of different ways depending on the nature of the AI/ML model.

[0056] In some embodiments, the UE receives some additional information from the network prior to collecting data for training the AI/ML model. For example, the UE may receive an indication of one or more types of metrics related to trustworthiness. As described above, the metric can be related to an accuracy of the AI/ML model (e.g., a maximum inferencing error) , a confidence value (e.g., high confidence or low confidence) , etc., to be described in further detail below in step 320. In another example, the UE may receive some assistance information from the network relating to the dataset collection that may inform the UE determination/evaluation of the trustworthiness metric.

[0057] In some embodiments, when the UE receives this additional information from the network prior to collecting the data, the UE may determine from this information that it cannot generate a trustworthy model to report to the network. If this occurs, the UE can determine not to collect data or train the model and the method ends. If the UE determines that it can generate a trustworthy model to report to the network, or if this type of evaluation is not performed, the method proceeds to 310.

[0058] In 310, the UE collects data for training the AI/ML model. As described above, the manner by which the UE collects the training data depends on the type of AI/ML model being trained. In one example, if the AI/ML model relates to channel estimation, the UE may measure a training set of RS to process and use as model input. In another example, if the AI/ML model relates to positioning, the UE may be performing a traditional positioning method (e.g. , GNSS) to gather positioning data to process and use as model input. In still another example, the UE may receive data from an external sensor. Those skilled in the art will understand the types of data collected for training the AI/ML models are varied and can be collected in any number of different ways depending on the nature of the AI/ML model.

[0059] In some embodiments, similar to 305, the UE receives some additional information from the network prior to training the AI/ML model with the collected data, including, e.g., the indication of one or more types of metrics related to trustworthiness, or assistance information.

[0060] The UE can, based on this additional information and the currently collected dataset, determine that it cannot generate a trustworthy model to report to the network. If this occurs, the UE can determine not to train the model and the method ends. Alternatively, the UE can wait until a qualified dataset is collected prior to training the model. If the UE determines that it can generate a trustworthy model to report to the network based on a currently collected dataset, or if this type of evaluation is not performed, the method proceeds to 315.

[0061] In 315, the UE trains the AI/ML model and generates a trained AI/ML model. In some embodiments, if the AI/ML model was trained with a dataset that was previously determined to be a sufficient dataset (e.g., based on additional information received from the network regarding acceptable parameters for the dataset) , the method proceeds to 325 and the UE reports the trained AI/ML model without any further evaluation of the trained AI/ML model. If this type of evaluation is not performed, after training, the method proceeds to 320.

[0062] In 320, the UE evaluates one or more metrics related to the trustworthiness or quality of the trained AI/ML model. As described above, the metric can be related to an accuracy of the AI/ML model (e.g., a maximum inferencing error) , a confidence value (e.g., high confidence or low confidence) , or qualities of the dataset used to train the model.

[0063] The UE can make various determinations based on the evaluated metrics. In one example, the UE can determine the trustworthy level of the trained AI/ML model meets or fails to meet a minimum threshold. In another example, the UE can determine, based on the trained model meeting or failing to meet the minimum threshold, that the model should or should not be reported. In another example, the UE can determine that the AI/ML model does not meet the required quality metric but should still be reported (in association with the quality metric) . In still another example, no determinations are made by the UE based on the evaluated metrics, and both the trained AI/ML model and the associated metrics are reported automatically. [0064] If the UE determines not to report the model, the method can end. Alternatively, the UE can collect additional data and retrain the AI/ML model in an attempt to improve the quality to a level sufficient for reporting. If the UE determines to report the model, the method proceeds to 325.

[0065] In 325, the UE reports the trained AI/ML model to the network. In some embodiments, the UE can include the trustworthy metric when reporting the trained AI/ML. In other embodiments, e.g. , when the AI/ML model is determined to be trustworthy, the UE skips the reporting of such metrics.

[0066] It should be understood that similar techniques may be used regardless of whether the Al agent is a UE, the RAN, or a network-side node such as a core network function or an application server and regardless of whether the Al manager is a UE, the RAN, or a network-side node. Those skilled in art will ascertain that the methods by which the UE, RAN or network-side node are enabled with Al agent or Al manager functionalities are varied and can depend on any combination of preconfigured functionalities, RAN conf iguration/indication, CN entity conf iguration/ indication, UE indication, etc. Thus, although the method 300 of Fig. 3 is described with respect to a UE being enabled for Al agent functionalities and a RAN (or network-side) node being enabled for Al manager functionalities, any one of the aforementioned entities can serve as the Al manager (e.g., providing one or more types of metrics, assistance information, etc. ) or as the Al agent (e.g. , evaluating the metrics and reporting the trained model) in various types of AI/ML operations/ applications . [ 0067 ] In another aspect of these exemplary embodiments , the Al agent can be provided with criteria for a valid dataset that is considered suitable for training a trustworthy AI /ML model . For example , the criteria can relate to the si ze or age of the dataset used for training . In another example , the criteria can relate to a method used for collecting the training data, a source of the training data, or the type of algorithm used for AI /ML model training . I f the criteria are not met, the Al agent may refrain from training the AI/ML model . In still another aspect , the Al agent can report these criteria for a trained model and the Al manager or consumer can determine, based on the reported criteria, whether the trained model is trustworthy . In a related aspect, the Al agent can report these criteria prior to training the model and based on the evaluation by the Al manager/consumer , the Al manager/consumer can provide a response (positive or negative ) to the Al agent regarding whether to train the AI /ML model .

[ 0068 ] In these aspects , the criteria relate to parameters or gualities of the dataset used to train the model and/or the method for training the model . The Al manager first provides to the Al agent information regarding the criteria for a valid dataset .

[ 0069] In some embodiments , the criteria may include the minimum size or the maximum age of the dataset used for training the model . A small dataset (below the minimum size indicated by the Al manager ) or an old dataset ( above the maximum age indicated by the Al manager) may be considered by the Al manager as not trustable, while a larger dataset ( above the minimum si ze ) or a newer dataset (below the maximum age ) may be considered trustable . [0070] In another embodiment, the criteria may include the method(s) used for dataset collection. Multiple types of methods for data collection may be enabled (or potentially enabled) for the UE, but only some of these methods may be acceptable to the network. For example, if the AI/ML model is for UE positioning, only the UE positions estimated by certain methods (e.g., GNSS) can be considered as trustable. In still another embodiment, the criteria may include the algorithm used for AI/ML model training. The dataset may be considered trustable only if certain algorithms (e.g., deep learning) were used while other algorithms (e.g., decision tree) may be considered not trustable

[0071] In still another embodiment, the criteria may include the source of the dataset. For some AI/ML models, the Al agent, e.g., the UE, may gather data from sources external to the UE . For example, in industrial settings, the Al agent may be a robot that is coupled to various types of sensors that may not be authenticated by the network. In these scenarios, where the source of the dataset is from a not trustworthy device, the AI/ML models trained by such a dataset cannot be considered as trustable .

[0072] Based on the criteria received from the Al manager, the Al agent may have the following behavior. The Al agent can first check if it is able to train an AI/ML model based on the criteria (e.g., it has a qualified dataset) . If the Al agent determines the dataset is valid, the Al agent may proceed to train the AI/ML model. If the Al agent determines the dataset is not valid, the Al agent may refrain from training the AI/ML model. The Al agent may proceed to accumulate additional data in an attempt to satisfy the criteria and, if the criteria are eventually satisfied, the Al agent can train the model.

Optionally, the Al agent may directly notify the Al manager that it is unable to perform this AI/ML model training tasks.

[0073] In a related aspect, the Al agent can provide the Al manager with some context information relating to the dataset acquired by the Al agent, prior to training the model. Based on the context information received from the Al agent, the Al manger can determine if the UE can obtain a trustable AI/ML. This context information may be similar to the criteria discussed above, e.g., the size of the dataset to be used to train the model; the age of the dataset to be used to train the model; the methods used for collection of the dataset to be used to train the model; the algorithm to be used for training the model; and the source of the dataset. Additionally, the context information can include a trustworthy level metric determined from at least one preceding AI/ML model.

[0074] Based on the context information received from the Al agent, the Al manager may determine if the Al agent can obtain an AI/ML model that is considered trustable. If the Al manager determines that the context is trustable, the Al manager may provide a positive response to the Al agent, instructing the Al agent to train the AI/ML model based on the context. If the Al manager determines the context is not trustable, the Al manager may provide a negative response to the Al agent, and the Al agent may refrain from training the AI/ML model. In one option, the Al manager may further provide information for how the context/dataset can be improved to provide a trustworthy context. For example, the Al manager can indicate to the Al agent that the size of the dataset should be increased. [0075] In still another related aspect, the Al agent may already possess a previously trained AI/ML model that it has not yet reported to the Al manager. In these aspects, the Al agent can provide the Al manager some context information relating to how this AI/ML model has been trained. This context information may be similar to the context information discussed above, e.g., the size of the dataset used to train the model; the age of the dataset used to train the model; the methods used for collection of the dataset used to train the model; the algorithm used for training the model; and the source of the dataset. Additionally, the context information can include a trustworthy level metric determined from at least one preceding AI/ML model.

[0076] Based on the information received from the Al agent, the Al manager may determine if the AI/ML model trained based on such context could be considered trustable. If the Al manager determines that the context is trustable, the Al manager may provide a positive response to the Al agent, instructing the Al agent to report the trained AI/ML model. If the Al manager determines the context is not trustable, the Al manager may provide a negative response to the Al agent, and the Al agent may refrain from reporting the trained AI/ML model. In one option, the Al agent may discard the trained AI/ML model. In another option, the Al agent may store the trained AI/ML model for a certain period of time, as it could be used for future training/ updating .

[0077] In one embodiment, the Al manager may also instruct the Al agent regarding what to do with the trained AI/ML model. For example, the Al manager can include such instructions in the response message including the negative response for reporting the model. In another embodiment, the Al agent can determine what to do with the trained AI/ML model based on how many times the context checking has failed. For example, if the context checking is failed only one time, the Al agent may store the model for future use. If the context checking fails multiple times, the Al agent may discard the model.

[0078] Fig. 4 shows a method 400 for selective AI/ML model training and reporting based on criteria related to the validity of a training dataset and/or training methods for the AI/ML model according to various exemplary embodiments. In this example, similar to the method 300 of Fig. 3, the Al manager comprises the 5G NR RAN or a network-side functionality instantiated externally to the RAN (e.g. , a core network entity or application server) and the Al agent comprises a UE .

[0079] In 405, the UE is enabled as an Al agent for training and reporting an AI/ML model. Similar to 305, the Al agent functionalities can be enabled for the UE in a variety of ways. Depending on the type of AI/ML model to be trained, the UE may receive additional conf igurations/indications from the network.

[0080] In some embodiments, the UE receives some additional information from the network prior to collecting data for training the AI/ML model. For example, the UE may receive information on criteria for a valid dataset, including a type of context information for the dataset and/or thresholds to be met regarding the context information for the dataset. As described above, the criteria can be related to a minimum size or maximum age of the dataset, the method to be used for dataset collection, the algorithm to be used for training the AI/ML model, or the source of the data to be gathered (e.g., whether the data is from an untrusted device remote to the UE) .

[0081] In some embodiments, when the UE receives these criteria from the network prior to collecting the data, the UE may determine from this information that it cannot generate a trustworthy model to report to the network. For example, the UE may be unable to meet one or more of the criteria based on UE capabilities. If this occurs, the UE can determine not to collect data or train the model and the method ends. If the UE determines that it can generate a trustworthy model to report to the network, or if this type of evaluation is not performed, the method proceeds to 410.

[0082] In 410, the UE collects data for training the AI/ML model. As described above, and similar to step 310 of Fig. 3, the manner by which the UE collects the training data depends on the type of AI/ML model being trained. Those skilled in the art will understand the types of data collected for training the AI/ML models are varied and can be collected in any number of different ways depending on the nature of the AI/ML model.

[0083] In some embodiments, similar to 405, the UE receives some additional information from the network prior to training the AI/ML model with the collected data, including, e.g., the criteria described above. The UE can determine context information for its dataset including, e.g., the size or age of the dataset, etc. In other embodiments, the UE determines the context information for the dataset, including, e.g., its size, its age, etc., based on UE implementation (e.g., without a network instruction or additional information) . In some embodiments, prior to training the model, the UE can report this context information to the network.

[0084] In 415, the UE transmits its context information for the dataset to the network. If the network determines the UE can obtain a trustable model from the context information, the network can transmit a positive response to the UE instructing the UE to train the model based on the reported context. In 420, the UE receives the positive network response and the method proceeds to 430. If the network determines the UE cannot obtain a trustable model from the context information, the network can transmit a negative response to the UE instructing the UE not to train the model based on the reported context. In 425, the UE receives the negative network response. In some embodiments, the method can end after the negative network response is received. In other embodiments, the UE may attempt to improve the dataset and the method can return to 410, where the UE collects additional data. In some embodiments, in the negative response, the network can further provide information for improving the context, e.g., instructions to increase the size of the dataset.

[0085] Returning to 410, if the UE has not yet determined any context information and/or the context information satisfies previously received criteria, the UE can determine to train the AI/ML model and the method proceeds to 430.

[0086] In 430, the UE trains the AI/ML model and generates a trained AI/ML model. In some embodiments, if the AI/ML model was trained with a dataset that was previously determined to be a sufficient dataset (e.g., based on the criteria / context information received from the network regarding acceptable parameters for the dataset) , the method proceeds to 450 and the UE reports the trained AI/ML model without any further evaluation of the trained AI/ML model. In other embodiments, the UE determines the context information for the dataset, including, e.g., its size, its age, etc., based on either network instruction or UE implementation (e.g., without a network instruction or additional information) . In some embodiments, prior to reporting the model, the UE can report this context information for the trained model to the network.

[0087] In 435, the UE transmits its context information for the trained model to the network. If the network determines the UE can obtain a trustable model from the context information, the network can transmit a positive response to the UE instructing the UE to report the model based on the reported context. In 440, the UE receives the positive network response and the method proceeds to 450. If the network determines the UE cannot obtain a trustable model from the context information, the network can transmit a negative response to the UE instructing the UE not to report the model based on the reported context. In 445, the UE receives the negative network response. In some embodiments, the method can end after the negative network response is received. In other embodiments, the UE may attempt to improve the dataset and the method can return to 410, where the UE collects additional data. In some embodiments, in the negative response, the network can further provide information for improving the context, e.g., instructions to increase the size of the dataset.

[0088] Returning to 430, if the trustworthiness of the AI/ML model was previously established from the characteristics of the dataset, the UE can report the trained AI/ML model. In 450, the UE reports the trained AI/ML model to the network.

[0089] Similar to the method 300 of Fig. 3, it should be understood that similar techniques may be used regardless of whether the Al agent is a UE, the RAN, or a network-side node such as a core network function or an application server and regardless of whether the Al manager is a UE, the RAN, or a network-side node. Those skilled in art will ascertain that the methods by which the UE, RAN or network-side node are enabled with Al agent or Al manager functionalities are varied and can depend on any combination of preconfigured functionalities, RAN conf iguration/indication, CN entity conf iguration/indication, UE indication, etc. Thus, although the method 400 of Fig. 4 is described with respect to a UE being enabled for Al agent functionalities and a RAN (or network-side) node being enabled for Al manager functionalities, any one of the aforementioned entities can serve as the Al manager (e.g., providing one or more types of metrics/criteria, evaluating the criteria, etc.) or as the Al agent (e.g., reporting the context information, evaluating the criteria, etc.) in various types of AI/ML operations/ applications .

[0090] In still another aspect of these exemplary embodiments, the Al manager can evaluate the performance of an AI/ML model reported by the Al agent. The Al manager can evaluate the model in various ways, e.g., for accuracy, robustness, stability, etc., as described above. In these embodiments, it is assumed that the trained AI/ML model previously reported by the Al agent was considered trustworthy by the Al manager (or such a trustworthy level check was not performed) . [0091] Aft er a period of time during which one or more models trained by the Al agent has been reported and used by the Al manager, the Al manager evaluates the performance of AI/ML models reported by the UE . The performance may be characterized by, e.g. , an accuracy level of the reported model (s) ; a percentage of correct inference based on the reported models; or a performance index of the functionalities that have used the reported models. For example, if the AI/ML model relates to air interface operations, the block error rate (BLER) of transmission/reception based on the air interface operations controlled using the reported model can be evaluated.

[0092] The Al manager may provide feedback about the AI/ML model performance to the Al agent. In one embodiment, the Al manager may directly provide the performance result. In another embodiment, the Al manager may directly indicate whether the Al agent should improve the context of AI/ML model, e.g. , if the Al agent should further expand its dataset for AI/ML model training. In still other embodiment, the Al manager may instruct the Al agent to pause AI/ML model training until the Al agent has an improved context for AI/ML model training, and/or may instruct the Al agent to guit from AI/ML model training tasks .

[0093] Based on the feedback, the Al agent may determine whether/how it should adapt and improve the trustworthy level of the AI/ML model it can train.

[0094] Fig. 5 shows a method 500 for AI/ML model training adaptation based on performance feedback according to various exemplary embodiments. In this example, similar to the methods 300 and 400 of Figs. 3-4, the Al manager comprises the 5G NR RAN or a network-side functionality instantiated externally to the

RAN (e.g., a core network entity or application server) and the Al agent comprises a UE .

[0095] In 505, the UE trains and reports an AI/ML model to the network. Similar to above, the Al agent functionalities can be enabled for the UE in a variety of ways. Depending on the type of AI/ML model to be trained, the UE may receive additional conf igurations/indications from the network prior to training and reporting the model.

[0096] In this example, it is assumed that the AI/ML model reported by the UE is considered trustworthy. That is, the method 300 and/or the method 400 described above can be performed, in whole or in part, prior to reporting the AI/ML model of the method 500. Alternatively, the method 500 can be performed without any previous analysis of the trustworthiness of the model (e.g., trustworthiness can be established in a different manner or not established) .

[0097] After some duration of time during which the AI/ML model is used by the network (or the network-side entity) , the network can evaluate the performance of the reported model. The performance can be characterized by accuracy, e.g., an accuracy level or a percentage of correct inference, or by a performance index for network functionalities that use the model, e.g., an air interface performance.

[0098] In addition, in some embodiments, the network can evaluate further actions that the UE should take. For example, the network can determine how the UE can improve the AI/ML model (e.g., by expanding the dataset used to train the model) or whether the UE should pause or quit training the model.

[0099] In 510, the UE receives feedback from the network regarding the performance of the AI/ML model reported by the UE . In some embodiments, the UE may receive only a performance result. In other embodiments, the UE may receive further information for improving the model. In still other embodiments, the UE may receive instructions from the network regarding further actions to take regarding the AI/ML model, e.g., to retrain the model, to pause the training, or to quit from the AI/ML model training tasks. It should be understood that multiple types of information may be provided in the feedback

[00100] In 515, based on the feedback, the UE determines whether and how to adapt its training tasks. In some embodiments, the UE may follow the network instructions (e.g., to retrain the model or to pause/quit the training) . In other embodiments, the UE may perform its own evaluation regarding how to improve the model. For example, based on the performance result, the UE can determine that the AI/ML model should be retrained with a new dataset or that the current dataset should be improved.

[00101] If the UE determines (or is instructed) to retrain the model, the UE can perform the new training task and report the new model to the network. Further feedback can be provided to the UE in a similar manner as described above.

[00102] Similar to the method 300 of Fig. 3 and the method 400 of Fig. 4, it should be understood that similar techniques may be used regardless of whether the Al agent is a UE, the RAN, or a network-side node such as a core network function or an application server and regardless of whether the Al manager is a UE, the RAN, or a network-side node. Those skilled in art will ascertain that the methods by which the UE, RAN or network-side node are enabled with Al agent or Al manager functionalities are varied and can depend on any combination of preconfigured functionalities, RAN conf iguration/indication, CN entity conf iguration/ indication, UE indication, etc. Thus, although the method 500 of Fig. 5 is described with respect to a UE being enabled for Al agent functionalities and a RAN (or network-side) node being enabled for Al manager functionalities, any one of the aforementioned entities can serve as the Al manager (e.g., evaluating the trained model, providing feedback, etc.) or as the Al agent (e.g., receiving the feedback, improving the model, etc.) in various types of AI/ML operations/applications .

[00103] In still another aspect of these exemplary embodiments, the Al manager can control the training of an aggregate model in multiple stages. In this aspect, the Al manager is an entity hosting a model aggregator, e.g., for federated learning (FL) operations. As described above, the Al manager in FL operations can be a network-side entity, e.g., a core network function or an application server, instructing multiple Al agents, e.g., UEs, to train and report respective partial models for fusion into a global model (e.g., an additional training stage from multiple partial trained models) . However, these aspects are not limited to FL operations and any type of AI/ML model and/or Al manager/agent entities can be used . [00104] In these aspects, it is assumed that the Al manager already has some knowledge about the context of certain Al agents and knows which Al agents are able to provide more trustworthy AI/ML models. The Al manager can first select a (relatively small) group of "trustworthy"AI agents and instruct these Al agents to train (partial) AI/ML models. Once the models are collected from this group of trustworthy Al agents, the Al manager aggregates these partial models to produce a first version of the global model.

[00105] The Al manager may verif y/evaluate the first version of the global model to ensure that it is actually trustworthy. If the model is evaluated to be not trustworthy, the Al manager may discard it, and select another group of Al agents to generate partial models for aggregation into another global model .

[00106] If the model is verified to be trustworthy, the Al manager may determine that global model has a strong, quality core, and proceed to instruct further Al agents (e.g., a larger set of Al agents) to be involved in the model refinement to generate a second version of the global model. Even if some of the further Al agents are "less trustworthy" than the initial set of Al agents, the strength of the first version of the global model will prevent additional (poor quality) models from substantially affecting the performance of the second version of the global model.

[00107] Fig. 6 shows a method 600 for multi-stage training of a global AI/ML model from multiple partial models according to various exemplary embodiments. In this example, the Al manager comprises the 5G NR RAN or a network-side functionality instantiated externally to the RAN (e.g., a core network entity or application server) and the Al agents comprise UEs.

[00108] In this example, it is assumed that the Al manager has some knowledge of the AI/ML training capabilities, or previously performed training operations (e.g., context information) , of certain UEs enabled as Al agents.

[00109] In 605, the Al manager selects a first group of UEs to perform a first round of AI/ML model training. Each UE from this first group can be determined by the Al manager to be trustworthy. This can be determined in various ways, e.g., based on the performance of previously reported AI/ML models, based on context information received from the UE (e.g., in accordance with the method 400 of Fig. 5) , or in other ways. The first group of UEs may be relatively small compared to the total number of UEs to be used to train the model (in later step 7XX) .

[00110] In 610, the Al manager instructs each of the selected UEs from the first group to train and report respective AI/ML models .

[00111] In 615, the Al manager receives partial AI/ML models from the UEs of the first group and aggregates these partial models into a first version of a global model.

[00112] In 620, the Al manager evaluates the first version of the global model to determine whether the first version is trustworthy. For example, the Al manager can evaluate the accuracy, robustness, stability, etc., of the first version of the global model. If the first version of the global model is evaluated to be not trustworthy, the Al manager can discard the first version of the model and select a new group UEs as the "first group" of UEs (e.g., a new group of "trustworthy" UEs) . In this scenario, the method can return to 610 and the Al manager can instruct this new group of UEs to train and report partial models.

[00113] If the first version of the global model is evaluated to be trustworthy, the Al manager can determine to refine the model and the method proceeds to 625.

[00114] In 625, the Al manager selects a second group of UEs to perform a second round of AI/ML model training. In some embodiments, e.g., in FL operations, the second group of UEs may be significantly larger than the first group selected in 610. Similar to 610, the Al manager may have some context information for the UEs from the second group and may select the UEs based on this context. The UEs from the second group may be associated with a trustworthy level (e.g., a trustworthy level less than that of the first group but still meeting minimum trustworthy requirements) , or may not be associated with a trustworthy level.

[00115] In 630, the Al manager instructs each of the selected UEs from the second group to train and report respective AI/ML models .

[00116] In 635, the Al manager receives partial AI/ML models from the UEs of the second group and aggregates these partial models into a second version of a global model. [ 00117 ] It should be understood that any number of stages of model training/ref inement can be used by the Al manager . In one example , the second version of the global model described above can become the new "core" of the global model , and further versions of the global model can be interactively generated based on further partial models received from the UEs of further selected groups . It should be further understood that the method 600 of Fig . 6 can relate to federated learning operations .

Examples

[ 00118 ] In a first example , a method performed by an artificial intelligence (Al ) agent , comprising receiving, from an Al manager, criteria for determining whether a trustworthy Al or machine learning (ML ) (AI /ML ) model can be generated from a collected dataset , collecting a dataset for training an AI /ML model , determining context information for the collected dataset or an AI /ML training method to be used and determining whether to train the AI/ML model with the collected dataset based on whether the context information satis fies the criteria .

[ 00119 ] In a second example , the method of the first example , wherein the criteria relate to a minimum size or a maximum age of the collected dataset and the context information determined by the Al agent includes a si ze or age of the collected dataset .

[ 00120 ] In a third example , the method of the first example, wherein the criteria relate to a method used for collection of the dataset and the context information determined by the Al agent includes the method used for collection of the dataset . [00121] In a fourth example, the method of the first example, wherein the criteria relate to an algorithm to be used for training the AI/ML model and the context information determined by the Al agent includes the algorithm to be used for training the AI/ML model.

[00122] In a fifth example, the method of the first example, wherein the criteria relate to a source of the dataset and the context information determined by the Al agent includes the source of the dataset.

[00123] In a sixth example, the method of the first example, wherein the Al agent determines to train the AI/ML model when the criteria are satisfied.

[00124] In a seventh example, the method of the first example, wherein the Al agent determines not to train the AI/ML model when the criteria are not satisfied.

[00125] In an eighth example, the method of the seventh example, further comprising collecting additional data to add to the dataset or to replace the dataset and determining to train the AI/ML model when the criteria are satisfied.

[00126] In a ninth example, the method of the seventh example, further comprising notifying the Al manager that the training of the AI/ML model cannot be performed based on the context information .

[00127] In a tenth example, the method of the first example, wherein the Al agent is a user equipment (UE) and the Al manager is a network node or network-side entity. [ 00128 ] In an eleventh seventh example , the method of the first example , wherein the Al agent is a network node or network-side entity and the Al manager is a user equipment (UE ) .

[ 00129 ] In a twel fth example , a processor configured to perform any of the methods of the first through eleventh examples .

[ 00130 ] In an thirteenth example , a user equipment (UE ) comprising a transceiver configured to communicate with a network and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the first through eleventh examples .

[ 00131 ] In a fourteenth example, a network node comprising a transceiver configured to communicate with a user equipment (UE ) and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the first through eleventh examples .

[ 00132 ] In a fi fteenth example, a method performed by an artificial intelligence (Al ) manager, comprising providing, to at least one Al agent, an indication of criteria for determining whether a trustworthy Al or machine learning (ML ) (AI /ML) model can be generated from a dataset collected by the Al agent and receiving, from the Al agent , a trained AI /ML model when the Al agent determines , based on context information for the collected dataset or an AI/ML training method to be used satis fying the criteria, that the trustworthy AI/ML model can be generated . [00133] In a sixteenth example, the method of the fifteenth example, wherein the criteria relate to a minimum size or a maximum age of the collected dataset and the context information determined by the Al agent includes a size or age of the collected dataset.

[00134] In a seventeenth example, the method of the fifteenth example, wherein the criteria relate to a method used for collection of the dataset and the context information determined by the Al agent includes the method used for collection of the dataset .

[00135] In an eighteenth example, the method of the fifteenth example, wherein the criteria relate to an algorithm to be used for training the AI/ML model and the context information determined by the Al agent includes the algorithm to be used for training the AI/ML model.

[00136] In a nineteenth example, the method of the fifteenth example, wherein the criteria relate to a source of the dataset and the context information determined by the Al agent includes the source of the dataset.

[00137] In a twentieth example, the method of the fifteenth example, further comprising receiving a notification from the Al agent that the training of the AI/ML model cannot be performed based on the context information.

[00138] In a twenty first example, the method of the fifteenth example, wherein the Al agent is a user equipment (UE) and the Al manager is a network node or network-side entity. [00139] In a twenty second example, the method of the fifteenth example, wherein the Al agent is a network node or network-side entity and the Al manager is a user equipment (UE) .

[00140] In a twenty third example, a processor configured to perform any of the methods of the fifteenth through twenty second examples.

[00141] In an twenty fourth example, a user equipment (UE) comprising a transceiver configured to communicate with a network and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the fifteenth through twenty second examples.

[00142] In a twenty fifth example, a network node comprising a transceiver configured to communicate with a user equipment (UE) and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the fifteenth through twenty second examples.

[00143] In a twenty sixth example, a method performed by an artificial intelligence (Al) agent, comprising collecting a dataset for training an Al or machine learning (ML) (AI/ML) model, determining context information for the collected dataset, an AI/ML training method to be used, or a metric related to a trustworthiness of a previously trained AI/ML model and, prior to either training the AI/ML model or reporting a trained AI/ML model, reporting the context information to an Al manager . [00144] In a twenty seventh example, the method of the twenty sixth example, wherein the context information determined by the

Al agent includes a size or age of the collected dataset.

[00145] In a twenty eighth example, the method of the twenty sixth example, wherein the context information determined by the Al agent includes the method used for collection of the dataset.

[00146] In a twenty ninth example, the method of the twenty sixth example, wherein the context information determined by the Al agent includes the algorithm to be used for training the Al /ML model.

[00147] In a thirtieth example, the method of the twenty sixth example, wherein the context information determined by the Al agent includes the source of the dataset.

[00148] In a thirty first example, the method of the twenty sixth example, further comprising receiving a positive response from the Al manager instructing the Al agent to report the trained AI/ML model when the Al manager determines, from the context information, that one or more criteria are satisfied and reporting the trained AI/ML model.

[00149] In a thirty second example, the method of the twenty sixth example, further comprising receiving a negative response from the Al manager instructing the Al agent not to report the trained AI/ML model when the Al manager determines, from the context information, that one or more criteria are not satisfied . [00150] In a thirty third example, the method of the thirty second example, further comprising discarding the trained AI/ML model .

[00151] In a thirty fourth example, the method of the thirty second example, further comprising storing the trained AI/ML model and collecting additional data to retrain the AI/ML model.

[00152] In a thirty fifth example, the method of the thirty second example, further comprising receiving additional instructions from the Al manager regarding discarding or retraining the AI/ML model.

[00153] In a thirty sixth example, the method of the thirty second example, further comprising determining whether to discard or retrain the AI/ML model based on a number of negative responses received from the Al manager.

[00154] In a thirty seventh example, the method of the twenty sixth example, further comprising receiving a positive response from the Al manager instructing the Al agent to train the AI/ML model based on the context information when the Al manager determines, from the context information, that one or more criteria are satisfied and reporting the trained AI/ML model.

[00155] In a thirty eighth example, the method of the twenty sixth example, further comprising receiving a negative response from the Al manager instructing the Al agent not to train the AI/ML model based on the context information when the Al manager determines, from the context information, that one or more criteria are not satisfied. [00156] In a thirty ninth example, the method of the thirty eighth example, further comprising receiving additional instructions from the Al manager regarding how to improve the context information for the dataset.

[00157] In a fortieth example, the method of the twenty sixth example, wherein the Al agent is a user equipment (UE) and the Al manager is a network node or network-side entity.

[00158] In a forty first example, the method of the twenty sixth example, wherein the Al agent is a network node or network-side entity and the Al manager is a user equipment (UE) .

[00159] In a forty second example, a processor configured to perform any of the methods of the twenty sixth through forty first examples.

[00160] In an forty third example, a user equipment (UE) comprising a transceiver configured to communicate with a network and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the twenty sixth through forty first examples.

[00161] In a forty fourth example, a network node comprising a transceiver configured to communicate with a user equipment (UE) and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the twenty sixth through forty first examples.

[00162] In a forty fifth example, a method performed by an artificial intelligence (Al) manager, comprising receiving, from an Al agent, context information for a dataset collected by the Al agent to train an Al or machine learning (ML) (AI/ML) model, an AI/ML training method to be used, or a metric related to a trustworthiness of a previously trained AI/ML model, determining, based on the context information, whether one or more criteria are satisfied and based on whether the criteria are satisfied, transmitting a positive response or a negative response to the Al agent regarding whether to train the AI/ML model or report a trained AI/ML model.

[00163] In a forty sixth example, the method of the forty fifth example, wherein the context information reported by the Al agent includes a size or age of the collected dataset.

[00164] In a forty seventh example, the method of the forty fifth example, wherein the context information reported by the Al agent includes the method used for collection of the dataset.

[00165] In a forty eighth example, the method of the forty fifth example, wherein the context information reported by the Al agent includes the algorithm to be used for training the AI/ML model.

[00166] In a forty ninth example, the method of the forty fifth example, wherein the context information reported by the Al agent includes the source of the dataset.

[00167] In a fiftieth example, the method of the forty fifth example, further comprising, when the criteria are not satisfied, transmitting additional instructions to the Al agent regarding discarding or retraining the trained AI/ML model. [00168] In a fifty first example, the method of the forty fifth example, further comprising, when the criteria are not satisfied, transmitting additional instructions to the Al agent regarding how to improve the context information for the dataset .

[00169] In a fifty second example, the method of the forty fifth example, wherein the Al agent is a user eguipment (UE) and the Al manager is a network node or network-side entity.

[00170] In a fifty third example, the method of the forty fifth example, wherein the Al agent is a network node or network-side entity and the Al manager is a user equipment (UE) .

[00171] In a fifty fourth example, a processor configured to perform any of the methods of the forty fifth through fifty third examples.

[00172] In an fifty fifth example, a user equipment (UE) comprising a transceiver configured to communicate with a network and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the forty fifth through fifty third examples.

[00173] In a fifty sixth example, a network node comprising a transceiver configured to communicate with a user equipment (UE) and a processor communicatively coupled to the transceiver and configured to perform any of the methods of the forty fifth through fifty third examples.

[00174] Those skilled in the art will understand that the above-described exemplary embodiments may be implemented in any suitable software or hardware configuration or combination thereof . An exemplary hardware platform for implementing the exemplary embodiments may include, for example , an Intel x86 based platform with compatible operating system, a Windows OS , a Mac platform and MAC OS , a mobile device having an operating system such as iOS , Android, etc . The exemplary embodiments of the above described method may be embodied as a program containing lines of code stored on a non-transitory computer readable storage medium that , when compiled, may be executed on a processor or microprocessor .

[ 00175 ] Although this application described various embodiments each having different features in various combinations , those skilled in the art will understand that any of the features of one embodiment may be combined with the features of the other embodiments in any manner not speci fically disclaimed or which is not functionally or logically inconsistent with the operation of the device or the stated functions of the disclosed embodiments .

[ 00176 ] It is well understood that the use of personally identi fiable information should follow privacy policies and practices that are generally recogni zed as meeting or exceeding industry or governmental requirements for maintaining the privacy of users . In particular, personally identifiable information data should be managed and handled so as to minimi ze risks of unintentional or unauthori zed access or use , and the nature of authori zed use should be clearly indicated to users .

[ 00177 ] It will be apparent to those skilled in the art that various modi fications may be made in the present disclosure , without departing from the spirit or the scope of the disclosure . Thus , it is intended that the present disclosure cover modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalent .