Performance related management of artificial intelligence or machine learning pipelines in cross-domain scenarios

Field

Various example embodiments relate to performance related management of artificial intelligence or machine learning pipelines in cross-domain scenarios. More specifically, various example embodiments exemplarily relate to measures (including methods, apparatuses and computer program products) for realizing performance related management of artificial intelligence or machine learning pipelines in cross-domain scenarios.

Background

The present specification generally relates to artificial intelligence (Al) / machine learning (ML) pipelines in cross-domain scenarios and the management thereof in particular for interoperable and multi-vendor environments.

An Al or ML pipeline helps to automate AI/ML workflows by splitting them into independent, reusable and modular components that can then be pipelined together to create a (trained) (AI/ML) model. An AI/ML pipeline is not a one-way flow, i.e., it is iterative, and every step is repeated to continuously improve the accuracy of the model and achieve a successful algorithm.

Figure 8 shows a schematic diagram of an example of an AI/ML pipeline.

An AI/ML workflow might consist of at least the following three components illustrated in Figure 8, namely, a data stage (e.g., data collection, data preparation/preprocessing), a training stage (e.g., hyperparameter tuning), and an inference stage (e.g., model evaluation).

With AI/ML pipelining and the recent push for microservices architectures (e.g., containers or container virtualization), each AI/ML workflow component is abstracted into an independent service that relevant stakeholders (e.g., data engineers, data scientists) can independently work on.

Besides, an AI/ML pipeline orchestrator shown in Figure 8 can manage the AI/ML pipelines' lifecycle (e.g., commissioning, scaling, decommissioning).

Subsequently, some basics of trustworthy artificial intelligence are explained.

For AI/ML systems to be widely accepted, they should be trustworthy in addition to meeting performance requirements (e.g., accuracy). The High- level Expert Group (HLEG) on Al has developed the European Commission's Trustworthy Al (TAI) strategy.

In April 2021, the European Commission presented the EU Artificial Intelligence Act or the regulatory framework for Al by setting out horizontal rules for the development, commodification and use of Al-driven products, services and systems within the territory of the EU. The Act seeks to codify the high standards of the EU Trustworthy Al paradigm, which requires Al to be legally, ethically and technically robust, while respecting democratic values, human rights and the rule of law. The draft regulation provides seven critical Trustworthy Al requirements for high-risk Al systems that apply to all industries:

1. Transparency: Include traceability, explainability and communication.

2. Diversity, non-discrimination and fairness: Include the avoidance of unfair bias, accessibility and universal design, and stakeholder participation. 3. Technical robustness and safety: Include resilience to attack and security, fall back plan and general safety, accuracy, reliability and reproducibility.

4. Privacy and data governance: Include respect for privacy, quality and integrity of data, and access to data.

5. Accountability: Include auditability, minimization and reporting of negative impact, trade-offs and redress.

6. Human agency and oversight: Include fundamental rights, human agency and human oversight.

7. Societal and environmental wellbeing: Include sustainability and environmental friendliness, social impact, society and democracy.

Additionally, International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) has also published a technical report on 'Overview of trustworthiness in artificial intelligence'. Early efforts in the open-source community are also visible towards developing TAI frameworks/tools/libraries such as IBM AI360, Google Explainable Al and TensorFlow Responsible Al.

However, while such knowledge in relation to trustworthiness of AI/ML exist, no approaches for implementing control and evaluation of performance of AI/ML pipelines in cross-domain management and orchestration architectures are known.

Hence, the problem arises that control and evaluation of performance of AI/ML pipelines in cross-domain scenarios in particular for interoperable and multi-vendor environments is to be provided. Hence, there is a need to provide for performance related management of artificial intelligence or machine learning pipelines in cross-domain scenarios.

Various example embodiments aim at addressing at least part of the above issues and/or problems and drawbacks.

Various aspects of example embodiments are set out in the appended claims.

According to an exemplary aspect, there is provided a method of a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including a first network domain in a network, the method comprising transmitting a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network, and receiving a second artificial intelligence or machine learning performance related message from said second network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided a method of a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network, the method comprising receiving a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting a second artificial intelligence or machine learning performance related message towards said first network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided a method of a third network entity responsible for fulfillment of network operator specifications in a first network domain in a network, the method comprising receiving a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting a fourth artificial intelligence or machine learning performance related message towards said first network entity, wherein said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter, said third artificial intelligence or machine learning performance related message is a crossdomain performance configuration request, and said fourth artificial intelligence or machine learning performance related message is a crossdomain performance configuration response.

According to an exemplary aspect, there is provided an apparatus of a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including a first network domain in a network, the apparatus comprising transmitting circuitry configured to transmit a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network, and receiving circuitry configured to receive a second artificial intelligence or machine learning performance related message from said second network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided an apparatus of a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network, the apparatus comprising receiving circuitry configured to receive a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting circuitry configured to transmit a second artificial intelligence or machine learning performance related message towards said first network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided an apparatus of a third network entity responsible for fulfillment of network operator specifications in a first network domain in a network, the apparatus comprising receiving circuitry configured to receive a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting circuitry configured to transmit a fourth artificial intelligence or machine learning performance related message towards said first network entity, wherein said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter, said third artificial intelligence or machine learning performance related message is a cross-domain performance configuration request, and said fourth artificial intelligence or machine learning performance related message is a cross-domain performance configuration response.

According to an exemplary aspect, there is provided an apparatus of a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including a first network domain in a network, the apparatus comprising at least one processor, at least one memory including computer program code, and at least one interface configured for communication with at least another apparatus, the at least one processor, with the at least one memory and the computer program code, being configured to cause the apparatus to perform transmitting a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network, and receiving a second artificial intelligence or machine learning performance related message from said second network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided an apparatus of a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network, the apparatus comprising at least one processor, at least one memory including computer program code, and at least one interface configured for communication with at least another apparatus, the at least one processor, with the at least one memory and the computer program code, being configured to cause the apparatus to perform receiving a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting a second artificial intelligence or machine learning performance related message towards said first network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to an exemplary aspect, there is provided an apparatus of a third network entity responsible for fulfillment of network operator specifications in a first network domain in a network, the apparatus comprising at least one processor, at least one memory including computer program code, and at least one interface configured for communication with at least another apparatus, the at least one processor, with the at least one memory and the computer program code, being configured to cause the apparatus to perform receiving a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and transmitting a fourth artificial intelligence or machine learning performance related message towards said first network entity, wherein said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter, said third artificial intelligence or machine learning performance related message is a cross-domain performance configuration request, and said fourth artificial intelligence or machine learning performance related message is a crossdomain performance configuration response.

According to an exemplary aspect, there is provided a computer program product comprising computer-executable computer program code which, when the program is run on a computer (e.g. a computer of an apparatus according to any one of the aforementioned apparatus-related exemplary aspects of the present disclosure), is configured to cause the computer to carry out the method according to any one of the aforementioned method- related exemplary aspects of the present disclosure.

Such computer program product may comprise (or be embodied) a (tangible) computer-readable (storage) medium or the like on which the computerexecutable computer program code is stored, and/or the program may be directly loadable into an internal memory of the computer or a processor thereof.

Any one of the above aspects enables an efficient control and evaluation of performance of AI/ML pipelines in cross-domain scenarios in particular for interoperable and multi-vendor environments to thereby solve at least part of the problems and drawbacks identified in relation to the prior art.

By way of example embodiments, there is provided performance related management of artificial intelligence or machine learning pipelines in crossdomain scenarios. More specifically, by way of example embodiments, there are provided measures and mechanisms for realizing performance related management of artificial intelligence or machine learning pipelines in crossdomain scenarios.

Thus, improvement is achieved by methods, apparatuses and computer program products enabling/realizing performance related management of artificial intelligence or machine learning pipelines in cross-domain scenarios.

Brief description of the drawings

In the following, the present disclosure will be described in greater detail by way of non-limiting examples with reference to the accompanying drawings, in which

Figure 1 is a block diagram illustrating an apparatus according to example embodiments, Figure 2 is a block diagram illustrating an apparatus according to example embodiments,

Figure 3 is a block diagram illustrating an apparatus according to example embodiments,

Figure 4 is a block diagram illustrating an apparatus according to example embodiments,

Figure 5 is a schematic diagram of a procedure according to example embodiments,

Figure 6 is a schematic diagram of a procedure according to example embodiments,

Figure 7 is a schematic diagram of a procedure according to example embodiments,

Figure 8 shows a schematic diagram of an example of an AI/ML pipeline,

Figure 9 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments,

Figure 10 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments,

Figure 11 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments,

Figure 12 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments, Figure 13 shows a schematic diagram of signaling sequences according to example embodiments, and

Figure 14 is a block diagram alternatively illustrating apparatuses according to example embodiments.

Detailed description

The present disclosure is described herein with reference to particular nonlimiting examples and to what are presently considered to be conceivable embodiments. A person skilled in the art will appreciate that the disclosure is by no means limited to these examples, and may be more broadly applied.

It is to be noted that the following description of the present disclosure and its embodiments mainly refers to specifications being used as non-limiting examples for certain exemplary network configurations and deployments. Namely, the present disclosure and its embodiments are mainly described in relation to 3GPP specifications being used as non-limiting examples for certain exemplary network configurations and deployments. As such, the description of example embodiments given herein specifically refers to terminology which is directly related thereto. Such terminology is only used in the context of the presented non-limiting examples, and does naturally not limit the disclosure in any way. Rather, any other communication or communication related system deployment, etc. may also be utilized as long as compliant with the features described herein.

Hereinafter, various embodiments and implementations of the present disclosure and its aspects or embodiments are described using several variants and/or alternatives. It is generally noted that, according to certain needs and constraints, all of the described variants and/or alternatives may be provided alone or in any conceivable combination (also including combinations of individual features of the various variants and/or alternatives). According to example embodiments, in general terms, there are provided measures and mechanisms for (enabling/realizing) performance related management of artificial intelligence or machine learning pipelines in crossdomain scenarios.

A framework for TAI in cognitive autonomous networks (CAN) underlies example embodiments.

Figure 9 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments, and in particular illustrates such trustworthy AI/ML framework for CANs (framework for TAI in CANs)) underlying example embodiments.

As shown in Figure 9, according to an introduced trustworthy AI/ML framework for cognitive autonomous networks, an intent/policy manager translates the customer intent into network quality of service (QoS) and network quality of trustworthiness (QoT) (e.g., service level agreement (SLA)), Al QoS (e.g., accuracy) and Al QoT (e.g., explainability, fairness, robustness) requirements and sends them to the service management and orchestration (SMO), the Al pipeline orchestrator, and the Al trust engine, respectively. Alternatively, the SMO may translate the network QoS and network QoT requirements into Al QoS and Al QoT requirements and may send them to the Al pipeline orchestrator and to the Al trust engine, respectively. The Al pipeline orchestrator and the Al trust engine may exchange information about Al QoS and Al QoT requirements with each other.

Considering that the Al pipelines deployed in the network may belong to multiple vendors, according to example embodiments, application programming interfaces (API) are exposed by the vendor-specific Al pipelines (without compromising the vendor's intellectual property rights) towards the Al pipeline orchestrator and the Al trust engine to discover the performance and trust capabilities of the Al pipeline, configure the Al pipeline according to the required Al QoS and Al QoT requirements, and to monitor/collect Al performance and Al trust related metrics from the Al pipeline.

Heretofore, according to the trustworthy AI/ML framework for CANs underlying example embodiments, APIs required for the Al trust engine to discover the Al trustworthiness capabilities via the Al trust manager of the Al pipeline, to configure the Al pipeline according to the required Al QoT via the Al trust manager, and to monitor/collect Al trustworthiness metrics and/or Al explanations related to the Al pipeline via the Al trust manager may be provided. Further, according to according to the trustworthy AI/ML framework for CANs underlying example embodiments, APIs required for the Al pipeline orchestrator to discover the performance capabilities of the Al pipeline via the Al performance manager of the Al pipeline, to (re)configure the Al pipeline according to the required Al QoS via the Al trust manager, and to monitor/collect Al performance metrics related to the Al pipeline via the Al performance manager may be provided.

Figure 10 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments, and in particular illustrates a cross-domain management and orchestration architecture leveraging a (the) domain-specific TAI framework.

Example embodiments are outlined considering such cross-domain management and orchestration architecture as shown in Figure 10. It is noted that a cross-domain end-to-end (E2E) network service scenario is utilized here as an example use case. However, example embodiments are not limited to this example use case. Instead, other cross-domain non-E2E scenarios (i.e., within each domain) are possible, e.g., a core domain can recursively embed 3GPP defined network function (NF) domain and virtualization domain, a radio access network (RAN) domain can include centralized unit (CU), distributed unit (DU), remote radio unit (RRU), midhaul and fronthaul domains provided by different vendors. In the illustrates cross-domain E2E network service example scenario, the cross-domain service management domain (CDSMD) (e.g., E2E service management domain) is responsible for decomposing the cross-domain E2E network service request (as per the service level agreement (SLA)), received from the network operator or the customer (via cross-domain policy/intent manager), into domain-specific (e.g., RAN, transport, core) network resource/service requirements, and for communicating them to the corresponding individual management domains (MD). Then, the individual MDs are responsible for ensuring that the domain-specific resource/service requirements are fulfilled, within their corresponding domains, by continuously monitoring the resource/service related key performance indicators (KPI) and reporting them to the CDSMD.

In the illustrated cross-domain E2E network service example scenario, the requested/instantiated cross-domain E2E network service, e.g. covering RAN, transport and core domains, may be managed by their corresponding Al pipelines (or cognitive network functions (CNF)) in the respective MDs. It is to be noted that, depending on the use case, the Al pipeline may be instantiated either in the domain-specific MDs (e.g., for proactive resource autoscaling) or within the domain itself (e.g., for proactive mobility handover in RAN domain).

Leveraging the domain-specific Al pipeline orchestrator and the Al pipelinespecific Al performance manager of the trustworthy AI/ML framework for CANs underlying example embodiments, the Al pipeline performance for the domain-specific Al pipelines may be defined, configured, measured and reported within the corresponding MD.

However, there is no way for the cross-domain Al pipeline orchestrator (within the CDSMD) to receive the desired cross-domain Al QoS (i.e., defined by the cross-domain policy/intent manager). Consequently, there is no way for the CDSMD to

- translate the cross-domain Al QoS into domain-specific Al QoS,

- discover the Al performance capability information from the domainspecific Al pipeline orchestrator(s),

- communicate the translated domain-specific Al QoS to the domainspecific Al pipeline orchestrator(s), and to

- collect/ request the cross-domain Al performance metrics from the domain-specific Al pipeline orchestrator(s).

In addition thereto, there was no way for the CDSMD to address (e.g., performing root-cause analysis) the Al performance related escalations, belonging to a cross-domain E2E network service, potentially received from the domain-specific Al pipeline orchestrator(s), and there was no way to delegate the relevant Al performance related escalation information potentially received from the domain-specific Al pipeline orchestrator of one MD to another MD so that the other MD may take preventive measures to avoid cross-domain E2E network service SLA violations (in the considered case: cross-domain Al QoS). Moreover, there was also no way for the CDSMD to aggregate the Al performance related escalation metrics potentially received from the Al pipeline orchestrator(s) of individual MDs to provide a global view of an issue (in the considered case: cross-domain Al QoS violations) to the network operator or the customer.

Figure 11 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments, and in particular illustrates a cross-domain management and orchestration architecture with a cross-domain Al trust engine.

Even such cross-domain management and orchestration architecture with a cross-domain Al trust engine (potentially providing cross-domain trust APIs between domain-specific Al trust engine and cross-domain Al trust engine) does not foresee cross-domain performance related APIs between a domainspecific Al pipeline orchestrator and a cross-domain Al pipeline orchestrator. In view of the above, in brief, according to example embodiments, crossdomain performance related APIs are provided between a domain-specific Al pipeline orchestrator and a cross-domain Al pipeline orchestrator in order to allow for control and evaluation of performance of AI/ML pipelines in crossdomain scenarios in particular for interoperable and multi-vendor environments.

Figure 12 shows a schematic diagram of an example of a system environment with interfaces and signaling variants according to example embodiments, and in particular illustrates a cross-domain management and orchestration architecture with a cross-domain Al pipeline orchestrator.

According to example embodiments, the cross-domain trustworthy AI/ML framework for cognitive autonomous networks underlying example embodiments is extended in order to facilitate the discovery, configuration, monitoring and reporting of cross-domain network service-related Al pipelines performance for interoperable and multi-vendor environments. A customer intent corresponding to a network service may include crossdomain Al QoS requirements in addition to the cross-domain QoT requirements, and the cross-domain TAI framework is used to ensure the fulfilment of desired cross-domain Al QoS requirements.

As shown in Figure 12, the cross-domain TAI framework according to example embodiments introduces a new interface (named, e.g., PCD-1) that supports interactions between a cross-domain Al pipeline orchestrator and (a) domainspecific Al pipeline orchestrator(s). Alternatively, or in addition, the crossdomain TAI framework according to example embodiments introduces another new interface (named, e.g., PCD-2) between the cross-domain Al pipeline orchestrator and (a) domain-specific policy/intent manager(s) (to support alternative implementation). According to example embodiments, the cross-domain Al pipeline orchestrator may consequently support the following operations:

- Configuring/delegating the desired/updated Al QoS (derived from the cross-domain Al QoS) that the domain-specific Al pipeline orchestrator is required to meet in the domain-specific Al pipeline belonging to the cross-domain network service,

- Discovering information concerning the performance capabilities (e.g., supported performance metrics, (re)configurable options such as model retraining, model reselection, model termination) of the domainspecific Al pipeline that the Al pipeline orchestrator is capable to configure in the domain-specific Al pipeline belonging to the crossdomain network service,

- Requesting the domain-specific Al pipeline orchestrator to (re)configure (e.g., retrain the model, reselect the model, terminate the model) the domain-specific Al pipeline belonging to the crossdomain network service and/or to configure the Al performance metrics to be measured in the domain-specific Al pipeline belonging to the cross-domain network service,

- Requesting/querying Al performance report for the domain-specific Al pipeline belonging to the cross-domain network service from the domain-specific Al pipeline orchestrator(s),

- Verifying whether the cross-domain Al QoS and/or the domain-specific Al QoS requirements for the cross-domain network service are satisfied,

- Performing root-cause analysis of the Al performance reports received from the domain-specific Al pipeline orchestrator(s); if needed, updating the domain-specific Al QoS requirements based on the Al performance reports,

- Providing a global view of the problem/escalation with respect to the cross-domain network service (e.g., aggregated cross-domain network service-related Al performance report) (in the considered case: crossdomain Al QoS violations) to the network operator, and - Delegating relevant Al performance escalation-related information received from the domain-specific Al pipeline orchestrator of one MD to another MD so that the other MD may take preventive measures to avoid cross-domain E2E network service SLA violations (in the considered case: cross-domain Al QoS).

To facilitate these functionalities, according to example embodiments, the following cross-domain APIs (produced by domain-specific Al pipeline orchestrator(s) and consumed by cross-domain Al pipeline orchestrator) are provided:

1. Cross-Domain Al Performance Capability Discovery API (Request/Response) - It allows the cross-domain Al pipeline orchestrator (entity), via (e.g.) PCD-1 interface, to discover Al reconfiguration methods and/or Al performance metrics that the domain-specific Al pipeline orchestrator (entity) is capable of configuring in the domain-specific Al pipeline belonging to the cross-domain network service.

2. Cross-Domain Al Performance Configuration API or Cross-Domain Al Performance Delegation API (Request/Response) - It allows the cross-domain Al pipeline orchestrator (entity), via (e.g.) PCD-1 interface, to configure/delegate the desired/updated Al QoS (derived from the crossdomain Al QoS) that the domain-specific Al pipeline orchestrator (entity) is required to meet in the domain-specific Al pipeline belonging to the crossdomain network service. Additionally, it allows the cross-domain Al pipeline orchestrator (entity) to request the domain-specific Al pipeline orchestrator (entity) to (re)configure (e.g., retrain the model, reselect the model, terminate the model) the domain-specific Al pipeline belonging to the crossdomain network service and/or to configure the Al performance metrics to be measured in the domain-specific Al pipeline belonging to the cross-domain network service. Alternatively, it allows the cross-domain Al pipeline orchestrator (entity), via (e.g.) PCD-2 interface, to notify the desired/updated Al QoS (derived from the cross-domain Al QoS) that the domain-specific policy/i ntent manager (entity) (via domain-specific Al pipeline orchestrator (entity)) is required to configure in the domain-specific Al pipeline belonging to the cross-domain network service.

3. Cross-Domain Al Performance Reporting API or Cross-Domain Al Performance Escalation API (Request/Response or Subscribe/Notify) - It allows the cross-domain Al pipeline orchestrator (entity), via (e.g.) PCD-1 interface, to request/subscribe for Al performance metrics that the domainspecific Al pipeline orchestrator (entity) is capable of measuring/reporting/escalating in the domain-specific Al pipeline belonging to the cross-domain network service.

Example embodiments are specified below in more detail.

Figure 1 is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a first network node or entity 10 such as a cross-domain artificial intelligence pipeline orchestrator entity (e.g. managing artificial intelligence or machine learning pipelines in a plurality of network domains including a first network domain in a network) comprising a transmitting circuitry 11 and a receiving circuitry 12. The transmitting circuitry 11 transmits a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network. The receiving circuitry 12 receives a second artificial intelligence or machine learning performance related message from said second network entity. Here, said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

Figure 5 is a schematic diagram of a procedure according to example embodiments. The apparatus according to Figure 1 may perform the method of Figure 5 but is not limited to this method. The method of Figure 5 may be performed by the apparatus of Figure 1 but is not limited to being performed by this apparatus.

As shown in Figure 5, a procedure according to example embodiments comprises an operation of transmitting (S51) a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network, and an operation of receiving (S52) a second artificial intelligence or machine learning performance related message from said second network entity. Here, said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

Figure 2 is a block diagram illustrating an apparatus according to example embodiments. In particular, Figure 2 illustrates a variation of the apparatus shown in Figure 1. The apparatus according to Figure 2 may thus further comprise a generating circuitry 21, a creating circuitry 22, and/or a verifying circuitry 23.

In an embodiment at least some of the functionalities of the apparatus shown in Figure 1 (or 2) may be shared between two physically separate devices forming one operational entity. Therefore, the apparatus may be seen to depict the operational entity comprising one or more physically separate devices for executing at least some of the described processes.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance capability information request, said second artificial intelligence or machine learning performance related message is a cross-domain performance capability information response, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of first domain scope information indicative of said first network domain, first scope information indicative of at least one artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance capability information request relates, first phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance capability information request relates, and customer information indicative of a customer or a category of said customer for which said at least one artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance capability information request relates is to be envisaged.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one capability information entry, wherein each respective capability information entry of said at least one capability information entry includes at least one of second domain scope information indicative of said first network domain, second scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective capability information entry relates, second phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said respective capability information entry relates, configuration information indicative of at least one configuration option supported for said artificial intelligence or machine learning pipeline to which said respective capability information entry relates, and performance metrics information indicative of at least one performance metric supported for said at least one artificial intelligence or machine learning pipeline phase of said artificial intelligence or machine learning pipeline to which said respective capability information entry relates.

According to a variation of the procedure shown in Figure 5, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of receiving crossdomain related artificial intelligence or machine learning quality of service requirements, an operation of generating domain-specific artificial intelligence or machine learning quality of service requirements for said first network domain based on said cross-domain related artificial intelligence or machine learning quality of service requirements, and an operation of creating said at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter based on said domain-specific artificial intelligence or machine learning quality of service requirements.

According to a variation of the procedure shown in Figure 5, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of verifying, based on content of said second artificial intelligence or machine learning performance related message, whether said cross-domain related artificial intelligence or machine learning quality of service requirements can be satisfied. According to such variation, an exemplary method according to example embodiments may additionally comprise an operation of transmitting, if, as a result of said verifying, said cross-domain related artificial intelligence or machine learning quality of service requirements cannot be satisfied, a cross-domain related artificial intelligence or machine learning quality of service nonacknowledgement message towards a third network entity responsible for fulfillment of network operator specifications in said first network domain in said network. According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance configuration request, and said second artificial intelligence or machine learning performance related message is a cross-domain performance configuration response.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one configuration entry, wherein each respective configuration entry of said at least one configuration entry includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective configuration entry relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said respective configuration entry relates, at least one of said domain-specific artificial intelligence or machine learning quality of service requirements, method trigger information indicative of at least one to-be- triggered configurable method of said artificial intelligence or machine learning pipeline to which said respective configuration entry relates, and performance metrics configuration information indicative of at least one to- be-configured performance metric for said at least one artificial intelligence or machine learning pipeline phase of said artificial intelligence or machine learning pipeline to which said respective configuration entry relates.

According to a variation of the procedure shown in Figure 5, exemplary additional operations are given, which are inherently independent from each other as such. According to such variation, an exemplary method according to example embodiments may comprise an operation of transmitting a third artificial intelligence or machine learning performance related message towards a third network entity responsible for fulfillment of network operator specifications in said first network domain in said network, and an operation of receiving a fourth artificial intelligence or machine learning performance related message from said third network entity. Here, said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter. Further, said third artificial intelligence or machine learning performance related message is a cross-domain performance configuration request. Still further, said fourth artificial intelligence or machine learning performance related message is a crossdomain performance configuration response.

According to further example embodiments, said at least one third crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one configuration entry, wherein each respective configuration entry of said at least one configuration entry includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective configuration entry relates, and at least one of said domain-specific artificial intelligence or machine learning quality of service requirements.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance report request, said second artificial intelligence or machine learning performance related message is a cross-domain performance report response, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance report request relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance report request relates, a list indicative of performance metrics demanded to be reported, start time information indicative of a begin of a timeframe for which reporting is demanded with said cross-domain performance report request, stop time information indicative of an end of said timeframe for which reporting is demanded with said cross-domain performance report request, and periodicity information indicative of a periodicity interval with which reporting is demanded with said cross-domain performance report request.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of demanded performance metrics.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance subscription, said second artificial intelligence or machine learning performance related message is a cross-domain performance notification, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance subscription relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance subscription relates, a list indicative of performance metrics demanded to be reported, and at least one reporting threshold corresponding to at least one of said performance metrics demanded to be reported.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes demanded performance metrics.

Figure 3 is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a second network node or entity 30 such as a domain-specific artificial intelligence pipeline orchestrator entity (e.g. managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network) comprising a receiving circuitry 31 and a transmitting circuitry 32. The receiving circuitry 31 receives a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network. The transmitting circuitry 32 transmits a second artificial intelligence or machine learning performance related message towards said first network entity. Here, said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

Figure 6 is a schematic diagram of a procedure according to example embodiments. The apparatus according to Figure 3 may perform the method of Figure 6 but is not limited to this method. The method of Figure 6 may be performed by the apparatus of Figure 3 but is not limited to being performed by this apparatus.

As shown in Figure 6, a procedure according to example embodiments comprises an operation of receiving (S61) a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and an operation of transmitting (S62) a second artificial intelligence or machine learning performance related message towards said first network entity. Here, said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

In an embodiment at least some of the functionalities of the apparatus shown in Figure 3 may be shared between two physically separate devices forming one operational entity. Therefore, the apparatus may be seen to depict the operational entity comprising one or more physically separate devices for executing at least some of the described processes.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance capability information request, said second artificial intelligence or machine learning performance related message is a cross-domain performance capability information response, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of first domain scope information indicative of said first network domain, first scope information indicative of at least one artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance capability information request relates, first phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance capability information request relates, and customer information indicative of a customer or a category of said customer for which said at least one artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance capability information request relates is to be envisaged.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one capability information entry, wherein each respective capability information entry of said at least one capability information entry includes at least one of second domain scope information indicative of said first network domain, second scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective capability information entry relates, second phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said respective capability information entry relates, configuration information indicative of at least one configuration option supported for said artificial intelligence or machine learning pipeline to which said respective capability information entry relates, and performance metrics information indicative of at least one performance metric supported for said at least one artificial intelligence or machine learning pipeline phase of said artificial intelligence or machine learning pipeline to which said respective capability information entry relates.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance configuration request, and said second artificial intelligence or machine learning performance related message is a cross-domain performance configuration response.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one configuration entry, wherein each respective configuration entry of said at least one configuration entry includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective configuration entry relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said respective configuration entry relates, at least one of domain-specific artificial intelligence or machine learning quality of service requirements, method trigger information indicative of at least one to-be- triggered configurable method of said artificial intelligence or machine learning pipeline to which said respective configuration entry relates, and performance metrics configuration information indicative of at least one to- be-configured performance metric for said at least one artificial intelligence or machine learning pipeline phase of said artificial intelligence or machine learning pipeline to which said respective configuration entry relates.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance report request, said second artificial intelligence or machine learning performance related message is a cross-domain performance report response, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter. According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance report request relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance report request relates, a list indicative of performance metrics demanded to be reported, start time information indicative of a begin of a timeframe for which reporting is demanded with said cross-domain performance report request, stop time information indicative of an end of said timeframe for which reporting is demanded with said cross-domain performance report request, and periodicity information indicative of a periodicity interval with which reporting is demanded with said cross-domain performance report request.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of demanded performance metrics.

According to further example embodiments, said first artificial intelligence or machine learning performance related message is a cross-domain performance subscription, said second artificial intelligence or machine learning performance related message is a cross-domain performance notification, and said second artificial intelligence or machine learning performance related message comprises a second information element including at least one second cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

According to further example embodiments, said at least one first crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said cross-domain performance subscription relates, phase information indicative of at least one artificial intelligence or machine learning pipeline phase to which said cross-domain performance subscription relates, a list indicative of performance metrics demanded to be reported, and at least one reporting threshold corresponding to at least one of said performance metrics demanded to be reported.

According to further example embodiments, said at least one second crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes demanded performance metrics.

Figure 4 is a block diagram illustrating an apparatus according to example embodiments. The apparatus may be a third network node or entity 40 such as a domain-specific intent/policy manager entity (e.g. responsible for fulfillment of network operator specifications in a first network domain in a network) comprising a receiving circuitry 41 and a transmitting circuitry 42. The receiving circuitry 41 receives a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network. The transmitting circuitry 42 transmits a fourth artificial intelligence or machine learning performance related message towards said first network entity. Here, said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter. Further, said third artificial intelligence or machine learning performance related message is a crossdomain performance configuration request. Still further, said fourth artificial intelligence or machine learning performance related message is a crossdomain performance configuration response.

Figure 7 is a schematic diagram of a procedure according to example embodiments. The apparatus according to Figure 4 may perform the method of Figure 7 but is not limited to this method. The method of Figure 7 may be performed by the apparatus of Figure 4 but is not limited to being performed by this apparatus.

As shown in Figure 7, a procedure according to example embodiments comprises an operation of receiving (S71) a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network, and an operation of transmitting (S72) a fourth artificial intelligence or machine learning performance related message towards said first network entity. Here, said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter. Further, said third artificial intelligence or machine learning performance related message is a crossdomain performance configuration request. Still further, said fourth artificial intelligence or machine learning performance related message is a crossdomain performance configuration response.

In an embodiment at least some of the functionalities of the apparatus shown in Figure 4 may be shared between two physically separate devices forming one operational entity. Therefore, the apparatus may be seen to depict the operational entity comprising one or more physically separate devices for executing at least some of the described processes.

According to further example embodiments, said at least one third crossdomain network service involved artificial intelligence or machine learning pipeline performance related parameter includes at least one configuration entry, wherein each respective configuration entry of said at least one configuration entry includes at least one of domain scope information indicative of said first network domain, scope information indicative of an artificial intelligence or machine learning pipeline in said first network domain to which said respective configuration entry relates, and at least one of domain-specific artificial intelligence or machine learning quality of service requirements.

Example embodiments outlined and specified above are explained below in more specific terms.

Figure 13 shows a schematic diagram of signaling sequences according to example embodiments, and in particular illustrates details of the crossdomain Al performance management APIs offered by the Al pipeline orchestrator (entity) to the cross-domain Al pipeline orchestrator (entity).

In Figure 13, a sequence diagram is shown, illustrating on how the crossdomain Al pipeline orchestrator (entity) can use the new APIs offered by the domain-specific Al pipeline orchestrator(s) (entity/entities) over (e.g.) the PCD-1 interface to discover the performance capabilities of the domainspecific Al pipeline, to (re)configure the domain-specific Al pipeline according to the required Al QoS, and to monitor/collect Al performance metrics from the domain-specific Al pipeline belonging to the cross-domain network service, according to example embodiments. The sequence diagram also illustrates interaction between the cross-domain Al pipeline orchestrator (entity) and the domain-specific policy/intent manager (entity), over (e.g.) the PCD-2 interface, according to example embodiments (representing alternative example embodiments).

In a step 1 of Figure 13, according to example embodiments, a network operator informs the cross-domain policy/intent manager about the intent for the cross-domain network service. In steps 2 to 6 of Figure 13, according to example embodiments, the crossdomain intent/policy manager translates the customer intent into crossdomain network QoS and network QoT (e.g., SLA), cross-domain Al QoS (e.g., accuracy, computational complexity, delay) and cross-domain Al QoT (e.g., explainability) requirements, and sends them to the cross-domain SMO, the cross-domain Al pipeline orchestrator, and the cross-domain Al trust engine, respectively. Alternatively, the cross-domain SMO may translate the cross-domain network QoS and cross-domain network QoT requirements into cross-domain Al QoS and Cross-domain Al QoT requirements and may send them to the cross-domain Al pipeline orchestrator and the cross-domain Al trust engine, respectively.

In a step 7 of Figure 13, according to example embodiments, the domainspecific Al trust engine exposes Al trustworthiness APIs towards the crossdomain Al trust engine to discover the Al trustworthiness capabilities of the domain-specific Al pipeline, to configure the domain-specific Al pipeline according to the required cross-domain Al QoT, and/or to collect Al trustworthiness metrics or explanations from the domain-specific Al pipeline.

In a step 8 of Figure 13, according to example embodiments, the crossdomain Al pipeline orchestrator translates the cross-domain Al QoS requirements into domain-specific Al QoS requirements (i.e., RAN domain Al QoS, transport domain Al QoS, and core domain Al QoS) depending on the customer intent of the cross-domain network service. The translation/mapping logic may take into account the SLA requirements (e.g., service type, service priority, KPI metrics) for the cross-domain network service, and, optionally, also the domain-specific Al performance capability information (i.e., in such case, according to example embodiments, the translation may be even performed after steps 9 to 11 of Figure).

Steps 9 to 13 of Figure 13 particularly illustrate the Cross-domain Al Performance Capability Discovery API according to example embodiments. In a step 9 of Figure 13, according to example embodiments, the crossdomain Al pipeline orchestrator sends the Cross-Domain Al Performance Capability Information Request to the domain-specific Al pipeline orchestrator requesting information concerning the performance capabilities (e.g., supported performance metrics, (re)configurable options such as model retraining, model reselection, model termination) of the domain-specific Al pipeline(s) in data stage and/or training stage and/or inference stage. The Cross-Domain Al Performance Capability Information Request may consist of parameters illustrated in the following table exemplifying content of a Crossdomain Al Performance Capability Information Request according to example embodiments.

In a step 10 of Figure 13, according to example embodiments, the domainspecific Al pipeline orchestrator(s) determines all the information requested in the Cross-Domain Al Performance Capability Information Request by interacting with the Al performance manager of domain-specific Al pipelines belonging to the cross-domain network service.

In a step 11 of Figure 13, according to example embodiments, the domainspecific Al pipeline orchestrator sends the Cross-Domain Al Performance Capability Information Response consisting of all the information about the domain-specific Al pipeline(s) (belonging to the cross-domain network service) on the supported performance capabilities (e.g., supported performance metrics, (re)configurable options such as model retraining, model reselection, model termination) to the cross-domain Al pipeline orchestrator. The Cross-Domain Al Performance Capability Information Response may consist of parameters illustrated in the following table exemplifying content of a Cross-Domain Al Performance Capability Information Response according to example embodiments.

In steps 12 and 13 of Figure 13, according to example embodiments, based on the Cross-Domain Al Performance Capability Information Response, the cross-domain Al pipeline orchestrator may determine whether the crossdomain Al QoS is satisfiable. If not satisfiable, the cross-domain Al pipeline orchestrator may send a cross-domain Al QoS non-acknowledgement (NACK) to the cross-domain intent/policy manager.

Steps 14 to 21 of Figure 13 particularly illustrate the Cross-Domain Al Performance Configuration API or Cross-Domain Al Performance Delegation API according to example embodiments.

Here, steps 14 to 16 of Figure 13 illustrate the Cross-Domain Al Performance Configuration API or Cross-Domain Al Performance Delegation API according to a first alternative of example embodiments. Further, steps 17 to 21 of Figure 13 illustrate the Cross-Domain Al Performance Configuration API or Cross-Domain Al Performance Delegation API according to a second alternative of example embodiments.

In a step 14 of Figure 13, according to example embodiments, the crossdomain Al pipeline orchestrator sends the Cross-Domain Al Performance Configuration/Delegation Request to the domain-specific Al pipeline orchestrator(s) for (re)configuring appropriate methods/options on the domain-specific Al pipeline belonging to the cross-domain network service and/or (re)configuring Al performance metrics to be measured from the domain-specific Al pipeline belonging to the cross-domain network service. Additionally, the Cross-Domain Al Performance Configuration/Delegation Request may also include information on the translated domain-specific Al QoS required to be met in the domain-specific Al pipeline belonging to the cross-domain network service. The Cross-Domain Al Performance Configuration/Delegation Request may consist of parameters illustrated in the following table exemplifying content of a Cross-Domain Al Performance Configuration Request according to example embodiments.

In a step 15 of Figure 13, according to example embodiments, based on the Cross-Domain Al Performance Configuration/Delegation Request, the domain-specific Al pipeline orchestrator may configure the requested (i.e., based on the desired domain-specific Al QoS) methods/options on the domain-specific Al pipeline and/or may configure the Al performance metrics in the domain-specific Al pipeline belonging to the cross-domain network service by interacting with the Al performance manager of the domainspecific Al pipeline.

In a step 16 of Figure 13, according to example embodiments, depending on whether the configuration process in the previous step was successful or not, the domain-specific Al pipeline orchestrator responds to the cross-domain Al pipeline orchestrator with the Cross-Domain Al Performance Configuration/Delegation Response containing an ACK/NACK (ACK: acknowledgement; NOACK: non-acknowledgement) for satisfying the domain-specific Al QoS in the domain-specific Al pipeline belonging to the cross-domain network service.

As mentioned above, steps 17 to 21 of Figure 13 illustrate the Cross-Domain Al Performance Configuration API or Cross-Domain Al Performance Delegation API according to a second alternative of example embodiments alternatively to example embodiments explained with reference to steps 14 to 16 of Figure 13. In a step 17 of Figure 13, according to example embodiments, a CrossDomain Al Performance Configuration/Delegation Request is sent from the cross-domain Al pipeline orchestrator to the domain-specific intent/policy manager to notify about the translated domain-specific Al QoS required to be met in the domain-specific Al pipeline belonging to the cross-domain network service. The Cross-Domain Al Performance Configuration/Delegation Request may consist of parameters illustrated in the following table exemplifying content of a Cross-Domain Al Performance Configuration/Delegation Request according to example embodiments.

In a step 18 of Figure 13, according to example embodiments, the domainspecific intent/policy manager sends the desired Al QoS information to the domain-specific Al pipeline orchestrator.

In a step 19 of Figure 13, according to example embodiments, based on the Cross-Domain Al Performance Configuration/Delegation Request, the domain-specific Al pipeline orchestrator may determine (i.e., based on the desired domain-specific Al QoS) and (re)configure suitable methods/options on the domain-specific Al pipeline and/or may configure the Al performance metrics in the domain-specific Al pipeline belonging to the cross-domain network service by interacting with the Al performance manager of the domain-specific Al pipeline. In a step 20 of Figure 13, according to example embodiments, the domainspecific Al pipeline orchestrator sends the ACK/NACK for satisfying the desired Al QoS in the domain-specific Al pipeline belonging to the crossdomain network service to the domain-specific intent/policy manager.

In a step 21 of Figure 13, according to example embodiments, depending on whether the domain-specific Al QoS was satisfied or not, the domain-specific intent/policy manager responds to the cross-domain Al pipeline orchestrator with the Cross-Domain Al Performance Configuration/Delegation Response containing an ACK/NACK for satisfying the domain-specific Al QoS in the domain-specific Al pipeline belonging to the cross-domain network service.

Steps 22 to 24 of Figure 13 particularly illustrate the Cross-Domain Al Performance Reporting API or Cross-Domain Al Escalation API according to example embodiments.

In a step 22 of Figure 13, according to example embodiments, the crossdomain Al pipeline orchestrator sends the Cross-Domain Al Performance Report Request to the domain-specific Al pipeline orchestrator containing the reporting configuration. The Cross-Domain Al Performance Report Request may consist of parameters illustrated in the following table exemplifying content of a Cross-Domain Al Performance Report Request according to example embodiments.

Alternatively, in the step 22 of Figure 13, according to example embodiments, the cross-domain Al pipeline orchestrator may subscribe to notifications/reports from the domain-specific Al pipeline orchestrator (i.e., Subscribe-Notify model) via a Cross-Domain Al Performance Report Subscribe message. The Cross-Domain Al Performance Report Subscribe may consist of parameters illustrated in the following table exemplifying content of a Cross-Domain Al Performance Report Subscribe according to example embodiments.

In a step 23 of Figure 13, according to example embodiments, the domainspecific Al pipeline orchestrator(s) collects all relevant performance metrics specified in the Cross-Domain Al Performance Report Request or CrossDomain Al Performance Report Subscribe by interacting with the Al performance manager of domain-specific Al pipelines belonging to the crossdomain network service.

In a step 24 of Figure 13, supposing that one or more reporting characteristics (i.e., periodic or on-demand) is met, in that case, according to example embodiments, the domain-specific Al pipeline orchestrator sends the CrossDomain Al Performance Report Response to the cross-domain Al pipeline orchestrator as per the reporting configuration specified in the Cross-Domain Al Performance Report Request.

Alternatively, in the step 24 of Figure 13, supposing that one or more reporting thresholds are met for the applicable Al performance metrics, in that case, according to example embodiments, the domain-specific Al pipeline orchestrator sends the Cross-Domain Al Performance Report Notify message to the cross-domain Al pipeline orchestrator consisting of the actual Al performance reports. The above-described procedures and functions may be implemented by respective functional elements, processors, or the like, as described below.

In the foregoing exemplary description of the network entity, only the units that are relevant for understanding the principles of the disclosure have been described using functional blocks. The network entity may comprise further units that are necessary for its respective operation. However, a description of these units is omitted in this specification. The arrangement of the functional blocks of the devices is not construed to limit the disclosure, and the functions may be performed by one block or further split into sub-blocks.

When in the foregoing description it is stated that the apparatus, i.e. network node or entity (or some other means) is configured to perform some function, this is to be construed to be equivalent to a description stating that a (i.e. at least one) processor or corresponding circuitry, potentially in cooperation with computer program code stored in the memory of the respective apparatus, is configured to cause the apparatus to perform at least the thus mentioned function. Also, such function is to be construed to be equivalently implementable by specifically configured circuitry or means for performing the respective function (i.e. the expression "unit configured to" is construed to be equivalent to an expression such as "means for").

In Figure 14, an alternative illustration of apparatuses according to example embodiments is depicted. As indicated in Figure 14, according to example embodiments, the apparatus (first network entity) 10' (corresponding to the first network entity 10) comprises a processor 1411, a memory 1412 and an interface 1413, which are connected by a bus 1414 or the like. Further, according to example embodiments, the apparatus (second network entity) 30' (corresponding to the second network entity 30) comprises a processor 1431, a memory 1432 and an interface 1433, which are connected by a bus 1434 or the like. Further, according to example embodiments, the apparatus (third network entity) 40' (corresponding to the third network entity 40) comprises a processor 1441, a memory 1442 and an interface 1443, which are connected by a bus 1444 or the like. The apparatuses may be connected via link 141a, 141b, respectively.

The processor 1411/1431/1441 and/or the interface 1413/1433/1443 may also include a modem or the like to facilitate communication over a (hardwire or wireless) link, respectively. The interface 1413/1433/1443 may include a suitable transceiver coupled to one or more antennas or communication means for (hardwire or wireless) communications with the linked or connected device(s), respectively. The interface 1413/1433/1443 is generally configured to communicate with at least one other apparatus, i.e. the interface thereof.

The memory 1412/1432/1442 may store respective programs assumed to include program instructions or computer program code that, when executed by the respective processor, enables the respective electronic device or apparatus to operate in accordance with the example embodiments.

In general terms, the respective devices/apparatuses (and/or parts thereof) may represent means for performing respective operations and/or exhibiting respective functionalities, and/or the respective devices (and/or parts thereof) may have functions for performing respective operations and/or exhibiting respective functionalities.

When in the subsequent description it is stated that the processor (or some other means) is configured to perform some function, this is to be construed to be equivalent to a description stating that at least one processor, potentially in cooperation with computer program code stored in the memory of the respective apparatus, is configured to cause the apparatus to perform at least the thus mentioned function. Also, such function is to be construed to be equivalently implementable by specifically configured means for performing the respective function (i.e. the expression "processor configured to [cause the apparatus to] perform xxx-ing" is construed to be equivalent to an expression such as "means for xxx-ing"). According to example embodiments, an apparatus representing the network node or entity 10 (e.g. managing artificial intelligence or machine learning pipelines in a plurality of network domains including a first network domain in a network) comprises at least one processor 1411, at least one memory 1412 including computer program code, and at least one interface 1413 configured for communication with at least another apparatus. The processor (i.e. the at least one processor 1411, with the at least one memory 1412 and the computer program code) is configured to perform transmitting a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in said first network domain in said network (thus the apparatus comprising corresponding means for transmitting), and to perform receiving a second artificial intelligence or machine learning performance related message from said second network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter (thus the apparatus comprising corresponding means for receiving).

According to example embodiments, an apparatus representing the network node or entity 30 (e.g. ma managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network) comprises at least one processor 1431, at least one memory 1432 including computer program code, and at least one interface 1433 configured for communication with at least another apparatus. The processor (i.e. the at least one processor 1431, with the at least one memory 1432 and the computer program code) is configured to perform receiving a first artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network (thus the apparatus comprising corresponding means for receiving), and to perform transmitting a second artificial intelligence or machine learning performance related message towards said first network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter (thus the apparatus comprising corresponding means for transmitting).

According to example embodiments, an apparatus representing the network node or entity 40 (e.g. responsible for fulfillment of network operator specifications in a first network domain in a network) comprises at least one processor 1441, at least one memory 1442 including computer program code, and at least one interface 1443 configured for communication with at least another apparatus. The processor (i.e. the at least one processor 1441, with the at least one memory 1442 and the computer program code) is configured to perform receiving a third artificial intelligence or machine learning performance related message from a first network entity managing artificial intelligence or machine learning pipelines in a plurality of network domains including said first network domain in said network (thus the apparatus comprising corresponding means for receiving), and to perform transmitting a fourth artificial intelligence or machine learning performance related message towards said first network entity, wherein said third artificial intelligence or machine learning performance related message comprises a third information element including at least one third cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter, said third artificial intelligence or machine learning performance related message is a cross-domain performance configuration request, and said fourth artificial intelligence or machine learning performance related message is a cross-domain performance configuration response (thus the apparatus comprising corresponding means for transmitting). For further details regarding the operability/functionality of the individual apparatuses, reference is made to the above description in connection with any one of Figures 1 to 13, respectively.

For the purpose of the present disclosure as described herein above, it should be noted that

- method steps likely to be implemented as software code portions and being run using a processor at a network server or network entity (as examples of devices, apparatuses and/or modules thereof, or as examples of entities including apparatuses and/or modules therefore), are software code independent and can be specified using any known or future developed programming language as long as the functionality defined by the method steps is preserved;

- generally, any method step is suitable to be implemented as software or by hardware without changing the idea of the embodiments and its modification in terms of the functionality implemented;

- method steps and/or devices, units or means likely to be implemented as hardware components at the above-defined apparatuses, or any module(s) thereof, (e.g., devices carrying out the functions of the apparatuses according to the embodiments as described above) are hardware independent and can be implemented using any known or future developed hardware technology or any hybrids of these, such as MOS (Metal Oxide Semiconductor), CMOS (Complementary MOS), BiMOS (Bipolar MOS), BiCMOS (Bipolar CMOS), ECL (Emitter Coupled Logic), TTL (Transistor-Transistor Logic), etc., using for example ASIC (Application Specific IC (Integrated Circuit)) components, FPGA (Field-programmable Gate Arrays) components, CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components;

- devices, units or means (e.g. the above-defined network entity or network register, or any one of their respective units/means) can be implemented as individual devices, units or means, but this does not exclude that they are implemented in a distributed fashion throughout the system, as long as the functionality of the device, unit or means is preserved; - an apparatus like the user equipment and the network entity /network register may be represented by a semiconductor chip, a chipset, or a (hardware) module comprising such chip or chipset; this, however, does not exclude the possibility that a functionality of an apparatus or module, instead of being hardware implemented, be implemented as software in a (software) module such as a computer program or a computer program product comprising executable software code portions for execution/being run on a processor;

- a device may be regarded as an apparatus or as an assembly of more than one apparatus, whether functionally in cooperation with each other or functionally independently of each other but in a same device housing, for example.

In general, it is to be noted that respective functional blocks or elements according to above-described aspects can be implemented by any known means, either in hardware and/or software, respectively, if it is only adapted to perform the described functions of the respective parts. The mentioned method steps can be realized in individual functional blocks or by individual devices, or one or more of the method steps can be realized in a single functional block or by a single device.

Generally, any method step is suitable to be implemented as software or by hardware without changing the idea of the present disclosure. Devices and means can be implemented as individual devices, but this does not exclude that they are implemented in a distributed fashion throughout the system, as long as the functionality of the device is preserved. Such and similar principles are to be considered as known to a skilled person.

Software in the sense of the present description comprises software code as such comprising code means or portions or a computer program or a computer program product for performing the respective functions, as well as software (or a computer program or a computer program product) embodied on a tangible medium such as a computer-readable (storage) medium having stored thereon a respective data structure or code means/portions or embodied in a signal or in a chip, potentially during processing thereof.

The present disclosure also covers any conceivable combination of method steps and operations described above, and any conceivable combination of nodes, apparatuses, modules or elements described above, as long as the above-described concepts of methodology and structural arrangement are applicable.

In view of the above, there are provided measures for performance related management of artificial intelligence or machine learning pipelines in crossdomain scenarios. Such measures exemplarily comprise transmitting a first artificial intelligence or machine learning performance related message towards a second network entity managing lifecycles of artificial intelligence or machine learning pipelines in a first network domain in a network, and receiving a second artificial intelligence or machine learning performance related message from said second network entity, wherein said first artificial intelligence or machine learning performance related message comprises a first information element including at least one first cross-domain network service involved artificial intelligence or machine learning pipeline performance related parameter.

Even though the disclosure is described above with reference to the examples according to the accompanying drawings, it is to be understood that the disclosure is not restricted thereto. Rather, it is apparent to those skilled in the art that the present disclosure can be modified in many ways without departing from the scope of the inventive idea as disclosed herein.

List of acronyms and abbreviations

3GPP Third Generation Partnership Project

ACK acknowledgement Al artificial intelligence

API application programming interface

AV autonomous vehicle

CAN cognitive autonomous network

CDSMD cross-domain service management domain

CNF cognitive network function

CU centralized unit

DU distributed unit

E2E end-to-end

HLEG High-level Expert Group

IEC International Electrotechnical Commission

ISO International Organization for Standardization

KPI key performance indicator

MANO management and orchestration

MD management domain

ML machine learning

NACK non-acknowledgement

NF network function

QCI QoS class identifier

QoE quality of experience

QoS quality of service

QoT quality of trustworthiness

RAN radio access network

RRU remote radio unit

SLA service level agreement

SMO service management and orchestration

TAI trustworthy Al

TAIF TAI framework