Large Language Model(LLM) assisted End-to-End Network Health Management based on Multi-Scale Semanticization (2024)

2.1 Related Work

The traditional anomaly detection algorithm detects anomalies by monitoring wireless measurement data and comparing it with established norms [29]. However, this approach overly depends on expert annotations and proves both time-consuming and labor-intensive. Concurrently, researchers also attempt to validate their findings using both simulated and actual data. Yet, these studies typically rely on a single KPI, such as the call drop rate, to classify anomalies, thereby constraining diagnostic precision to a degree [14]. The Bayesian-based classification method, extensively explored in [2][3], uses probability and graph theories to correlate network anomalies with their root causes. Despite its widespread application, the efficacy of this method significantly hinges on a substantial corpus of historical anomaly data since the causal graphs it generates demand extensive prior knowledge. Moreover, the Bayesian approach faces challenges in scalability and adaptability, struggling to perform well in dynamic, heterogeneous wireless network environments.

Machine learning, recognized as a powerful analytical tool, can effectively mine and perceive potential information in data and sharply detect subtle changes in network status and KPIs, thus enabling faster and more precise network anomaly detection[23]. Researchers propose a diagnostic method based on a supervised genetic fuzzy algorithm[15]. This method employs a genetic algorithm to learn a fuzzy rule base from a combined dataset of simulated and real data containing 72 records. Its accuracy heavily relies on the labeled training set. The Deep Transformer-based temporal anomaly detection model, TranAD[32], incorporates an attention sequence encoder and leverages broader temporal trend knowledge to swiftly conduct anomaly detection. DCdetector[35] masters the representation of abnormal samples using a dual attention mechanism and contrastive learning. While machine learning methods have advanced in feature learning and enhanced their generalization capabilities, they face challenges in wireless networks. Abnormalities are sporadic, and scarce abnormal samples make the models prone to overfitting. Moreover, modeling only the entire network fails to adapt to dynamic DHNs.

Although research on distributed anomaly detection solutions is extensive[5], practical applications suffer due to inconsistent network entity feature representation, weakening detection capabilities[25]. Additionally, using machine learning to model each device alone is both time-consuming and labor-intensive. The models also struggle to capture the interactive information of communication devices. Additionally, existing distributed fault detection methods often consider abnormal situations as a whole, which neglects the specific abnormal representation of individual communication entities, thereby complicating the rapid detection of abnormal nodes by NMs.

2.2 Problem Statement and Our Objectives

Within DHNs, the diverse range of communication devices poses challenges for domain experts in gathering data encompassing all device anomaly types for model training. Furthermore, these models typically lack autonomous learning capabilities. Consequently, the emergence of new communication devices or technologies within the network often detracts from the detection efficacy of the model, leading to performance degradation.

In addition to the aforementioned shortcomings, existing anomaly detection research often emphasizes enhancing detection accuracy or model interpretability. However, the comprehensive coverage of the entire health management life cycle is seldom taken into account. For anomalies detected by the model, the prevalent approach involves NMs extracting information and experience from satisfactorily resolved and archived cases (i.e. marked cases) to alleviate the anomalies[24]. Undoubtedly, this significantly diminishes the efficiency of anomaly mitigation.

We incorporate LLM into the health management life cycle, leveraging its reasoning capabilities to identify the root causes of abnormal situations, thereby furnishing NMs with end-to-end anomaly resolutions. Moreover, LLM’s learning capability enables rapid adaptation to new abnormal information from communication entities. To facilitate LLM in gathering anomaly information, we devised MSADM, deployed on communication entities to execute anomaly detection and information collection. Given the distributed deployment of MSADM, our scheme offers entity-level visibility, contrasting with prior distributed anomaly detection models. In the subsequent section, we will elaborate on our solution scheme in detail.

3.1 Construction of Rule Base

In this section, we present the packet loss rate (PLR) as an example to illustrate the shortcomings of existing distributed approaches. We compute a positively distributed interval for the average PLR over $T$ for all devices. Next, we insert the average value of each device into the interval, and its distribution appears in Fig.4. The distribution of PLR varies significantly across different devices, and if such a dataset is used for model training, the model will struggle to adapt to this scenario of anomalous performance with multi-scale devices. Fig.5 shows the change in anomaly detection accuracy for different devices before and after using the rule base. Next, we will provide a detailed description of the process for designing and using the rule base.

We analyze the KPIs[16] common to multiple devices within the simulated network and construct the rule base accordingly. A comprehensive list of KPI types and contents is detailed in appendixA. For each device type, we analyze the collected data to ascertain the distribution of each KPI across various dimensions. Subsequently, we compare the actual KPI changes for these devices against their respective distributions to pinpoint anomalous statuses.

We represent the network background information within $T$ under normal conditions as $\mathcal{N}_{normal}=(N_{f},~{}E_{f},~{}T)$.$N_{f}$ denotes the attributes of the node itself, expressed as $N_{f}=\{~{}f_{N1},~{}f_{N2},~{}\ldots,~{}f_{Nn}\}$, while $E_{f}$ represents the attributes of the communication link, similar to the node, and is given by $E_{f}=\{~{}f_{E1},~{}f_{E2},~{}\ldots,~{}f_{En}\}$. $T$ indicates the period for recording network information.

We collected a substantial number of $\mathcal{N}_{normal}$ for hom*ogeneous entities to enhance our analysis. For each KPI, we calculate its average value (Avg,$F_{a}$), fluctuation value (Jitter,$F_{j}$), variance (Variance,$F_{v}$), and trend (Trend,$F_{t}$). The average represents the center or average of the dataset and aids in understanding the general performance level. The fluctuation value represents the dispersion or range of values in the dataset, calculated as the average of the differences between adjacent data points. Variance, the average of the squared differences of each data point from the mean, measures the extent to which individual data points deviate from the mean. Data trends describe the changes in data over time.

We can readily compute the numerical distribution diagram of the first three dimensions, thereby getting a set of intervals $Dist$ that depicts the abnormality of the performance indicator. According to its distribution, the interval closer to the peak indicates that the dimensional data aligns more closely with normal data and should be considered more normal. As trend falls into categories such as rise, fall, fluctuation, etc. Its calculation is different. We assess the instantaneous performance and overall trend of the network based on the number of extreme points obtained. The data within $T$ is subdivided into n small periods $t$. By obtaining the average value within each $t$, the continuous time data is converted into discrete data values $v=\{~{}v_{1},~{}v_{2},~{}\ldots,~{}v_{n}\}$.

To mitigate noise interference and facilitate smoother data processing, we increase the threshold $h$ during the identification of maximum and minimum values. If a value and its adjacent value differ by no more than one $h$, we do not classify it as an extreme point. The presence of multiple maximum and minimum values signifies a fluctuating trend. Conversely, a single minimum value suggests a sudden drop, whereas a single maximum value indicates a sudden rise. Regarding the threshold definition, we derive it from the distribution of fluctuation values among $n$ discrete data points under normal conditions. Utilizing this methodology enables us to ascertain the trend status of performance indicators.

As demonstrated in formula2, when the absolute value of the difference between point $t_{i}$ and its two neighboring points exceeds $h$, we classify the point as an extreme value point. The determination of the minima is shown in formula3.

The algorithm1 outlines the procedure for computing the four evaluation dimensions from our rule base and obtaining the KPIs status list:

We have also explored the possibility of using a machine learning-based classification model to categorize data trends. However, suppose new features or wireless access technologies emerge in the future, affecting the performance evaluation data of KPIs. In that case, we will need to recollect and relabel the dataset to train the model. In contrast, with the rule-based method, we only need to gather sufficient data and update the threshold using the built-in script to refresh the rule base. Therefore, the rule base demonstrates superior scalability and adaptability.

3.2 Anomaly Information Learning and Detection

We have designed an anomaly detection architecture for KPIs time series data in MSADM. Fig.6 illustrates the structure of the anomaly detection model. In this framework, the time-series data initially passes through a convolutional layer that captures time-series features within a specific segment, followed by a two-layer converter to fully perceive changes in the KPIs. To enhance the model’s robustness, we have embedded a rule-filtered states list prior to the model entering the fully connected layer. Because our goal is for MSADM to recognize the anomaly type while performing anomaly detection, a four-layer fully connected network is employed. The first two layers sense the data association, while the latter two layers handle the detection and classification tasks. The remainder of this section details specific model design concepts.

For anomaly detection tasks, certain element fragments often harbor more anomaly information features. Convolutional Neural Networks (CNN) improve classification accuracy by extracting local features from time series[10]. However, the sequence of elements and their interdependencies are essential for time series analysis. While CNNs excel at focusing on local features, their capability to model global dependencies is comparatively limited[22]. In time series classification tasks that require a global perspective, this limitation may lead to a decrease in model accuracy.

The Transformer, via its self-attention mechanism, can process sequences of any length[21]. This feature efficiently captures global dependencies within sequences, effectively overcoming CNN’s limitations in global modeling.

The fully connected layer gradually transforms the extracted features into classification probabilities that identify anomalies. Simultaneously, the model goes beyond merely outputting these probabilities; it also specifies the type of anomaly detection identifying the abnormal entity. Consequently, we have separated the fully connected layer at the end to acquire both anomaly detection results and anomaly types through distinct linear layers.

3.3 Semantic Rule Tree Structure

In the initial section, we obtain a list of statuses $S$ for the KPIs of the anomaly network entities, filtered according to predefined rules. Utilizing these status lists, MSADM generates detailed anomaly information reports for anomalous network entities via a semantic rule tree.

We explored logical semantics, distributed semantics, hybrid semantics for the NLG model, and a Knowledge Graph-based replication mechanism for sentence generation[4][18]. These models necessitate a large amount of high-quality textual training datasets. However, since our method generates sentences from a list of statuses, training becomes highly inefficient following a significant number of events, and the utterances produced are overly slow and filled with superfluous information. Moreover, the dataset requires expansion to train the model whenever a new description of an anomaly manifestation arises.

Our goal is to generate timely, accurate, and concise sentences. Therefore, we opted, after careful consideration, to employ a template-like approach to sentence generation. Given the limited variety of statuses in the status list, we chose to select words that correspond to the number of statuses for each KPIs evaluation dimension. Unlike traditional template-based approaches, we use a tree structure with a unique one-to-many configuration that effectively captures the abnormal statuses of KPIs under various evaluation metrics. This structure is not only highly flexible and extensible but also facilitates the future integration of new evaluation metrics and statuses. We employ this tree structure to generate sentences for each KPI, which are then compiled into the comprehensive anomaly reports.

As shown in Fig.7, we maintain a vocabulary describing KPIs performance metrics and KPIs status levels and a lexicalized tree adjoining grammar (LTAG) representing the lexicality of words. MSADM can utilize the evaluation dimensions of arbitrary KPIs as the root, connect syntactic trees to form the syntactic part of a sentence and construct a sentence tree by positioning fixed vocabulary in the leaves. Meanwhile, to further speed up the sentence generation, we added the pruning operation of words and LTAGs before sentence generation and tried to keep only the words related to the current KPIs.

The specific build process is as follows: MSADM traverses the sentence tree starting from the root, categorized by a KPIs type with a list of evaluated dimensions and statuses. Each traversal from the root to the leaves yields a semanticized description corresponding to the current KPIs statuses. Considering that actual KPIs data may be more precise than the status description, we incorporate a judgment call in the sentence generation process. When a KPI exhibits significant abnormalities, we add its actual values, such as mean, variance, and jitter, within the timeframe $T$ to enrich the information content of the sentence. The process is shown by algorithm 2.

After compiling all abnormal sentence expressions from a node and considering the input constraints of the LLM, we strike a balance between the simplicity of the report and the completeness of the information. We then assess the need to further refine the entity information collected in the sentences based on the report’s length and the severity of the KPIs anomalies. We use regular expressions to optimize the report content while ensuring that essential and critical anomaly information is retained.

3.4 Information Integration

The LLM’s powerful natural language processing capabilities allow it to deeply understand semantic information and derive meaningful features and patterns[6]. Simultaneously, LLM’s continuous learning ability enables it to adapt and respond effectively to evolving event types, showcasing remarkable scalability and rapid adaptability in complex scenarios[28].

In the information integration phase, we compile the abnormal reports of communication entities within the DHNs and generate prompt text language that the LLM can understand, and tailor.

LLMs often struggle with complex and in-depth reasoning due to their reliance on patterns in data rather than true understanding, leading to difficulties in consistently generating accurate, contextually appropriate responses that require deep domain knowledge or logical consistency[34]. In our integration process, we have bootstrapped the LLM to assist in generating anomaly reports that better align with the requirements of NMs, based on the life cycle of health management.

The structure of the prompt is illustrated in Fig.8. We provide the model with context, questions, and options. The context enables the LLM to comprehend network anomaly information. The question addresses the needs of NMs, specifically the types of abnormalities that may occur and the associated mitigation plans. The option constrains the LLM’s inference results to the specified types of anomalies, thereby enhancing the accuracy of the inferences. Naturally, the options also include others.

Given that large models face input length limitations, the anomaly context must encompass all relevant information of abnormal entities within the local network at the time of the anomaly, a requirement that significantly exceeds the input capacity of the existing model. Consequently, the anomaly context cannot be directly embedded within the prompt text. We collate the collected contextual information regarding entity anomalies, utilize the abnormal status to pinpoint KPIs exhibiting significant abnormalities within network entities and provide a detailed description of such KPIs. Conversely, KPIs exhibiting minor abnormalities are summarized in a consolidated manner. Furthermore, we incorporate the abnormal detection results obtained in section3.2 into the report, thereby enriching the LLM with additional dimensions of information focus.

4.1 MSADM Evaluations

We surveyed several popular time series classification models that utilize various technologies. CL-MPPCA employs both neural networks and probabilistic clustering to enhance anomaly detection performance[31]. SR-CNN integrates SR and CNN models to boost the accuracy of time series anomaly detection[26]. AnomalyBERT, built on the Transformer architecture, is designed to discern temporal contexts and identify unnatural sequences [12]. LSTM-transformer introduces a novel hybrid architecture combining LSTM and Transformer, tailored for multi-task real-time prediction[7]. We compare these models with the anomaly detection module of MSADM. We will train these models using the same equipment and conduct a comprehensive comparison.

In Fig.9, the model’s evolution in classification accuracy, detection accuracy, and cross-entropy loss function is depicted over increasing iterations. Notably, our model consistently achieves the highest accuracy, ultimately converging to 91.3%. This figure marks an approximately 3% lead over the runner-up model, LSTM-transformer. Additionally, the Cross-Entropy loss of our model substantially surpasses that of other models upon final convergence.

In Table1, we conducted a comparative analysis between MSADM and various other models concerning fault detection accuracy, anomaly detection accuracy, detection recall rate (Recall), detection false negative rate (FNR), and detection false positive rate (FPR). We meticulously assessed performance across these metrics. Notably, we highlight the superior performance of MSADM, as indicated by the bold data for each metric. The conclusive findings demonstrate that MSADM surpasses other models across most performance indicators. It’s worth mentioning that the detection time, while marginally lower than the LSTM method lacking rule embedding, is attributed to the initial requirement of rule filtering.

The ROC curves represent the true positive rate (TPR) and false positive rate (FPR) under different threshold settings[2]. To compare the robustness and reliability of the models. We plotted the ROC curve. As shown in Fig.10, the ROC curve of the MSADM model is higher than other models most of the time, while the AUC of MSADM is 0.1 higher than the current hottest LSTM-transformer structure.

Due to the anomaly’s limited range of influence, enlarging the network size might result in overlooking the anomaly. Fig.11 illustrates the variation in model accuracy corresponding to changes in network size. In both scenarios with a small and large number of nodes, the MSADM model outperforms other models in both anomaly detection and classification accuracy.

Fig.12 illustrates the confusion matrix analysis of the anomaly detection results produced by our MSADM model on the test set. Fig.12 (a) primarily assesses the model’s accuracy in identifying various anomalies. The results underscore the model’s high accuracy across most anomaly-type classification tasks. Fig.12 (b) depicts the accuracy of anomaly detection. Our identification accuracy for abnormal samples reaches as high as 95%, implying that we can analyze and collect information from almost all abnormal network entities within the network structure. For normal samples that are incorrectly detected, because we gauge the degree of abnormality in the generation of abnormal reports, a large amount of minor abnormal information will not excessively consume abnormal reporting resources.

4.2 Semanticization Evaluations

In this section, we present the text generation component of MSADM to showcase the quality of our semantic generation. We will also highlight segments of the LLM output to underscore the benefits of our thought prompts in guiding LLM reasoning. Due to space constraints, we display only a portion of the anomaly report and LLM output, with the complete textual content available as chapters in the appendix.

In the event of a node application crash, the current node becomes unable to request and respond to data packets due to application anomaly, yet it retains its functionality as a packet forwarding relay. We use this scenario as an example to demonstrate the practicality of our generated statements.

The results are depicted in Fig.13. We show a partial anomaly report generated by a single network entity when an anomaly occurs. This section includes descriptions of packet rates, bit error rates, and latencies, while also providing anomalies diagnosed by the model. It is evident from the report that the PLRs and the bit error rate of the nodes are notably high, whereas the PLR and the bit error rate of the communication link remain relatively unaffected, aligning with the observed real-world scenario. See the appendixD for the complete report.

We input the analyzed data from the collected reports into the LLM to generate relevant reports and conclusions. The solution produced by the LLM appears in Fig.14. By incorporating chain-of-thought-based prompts, the LLM assesses various factors that may have contributed to the anomaly, including software and hardware issues, as well as troubleshooting and resolution strategies. This exception report, enhanced by LLM’s insights, significantly surpasses traditional operations and maintenance documentation by reducing empiricism that leads to incorrect exception handling. At the same time, the anomaly solution enables NMs to rapidly mitigate anomalies and maintain network health. The comprehensive exception analysis report is detailed in the appendixE.

E.1 Node Analysis

Based on the provided network context, here’s the detailed analysis:

Node0: Status: High packet loss rate averaging 44.43Communication Link Info: Indicates a possible fault for the app down.Analysis: The high packet loss rate and the indication of an appdown fault suggest a significant issue with the application layer on Node0. This could be due to software errors, resource limitations, or misconfigurations.Recommended Action: Investigate the application running on Node0, check for any software updates, review configurations, and allocate additional resources if necessary to mitigate packet loss.

Node1: Status: The number of neighboring nodes and routing table caches are significantly above average, with minor fluctuations and an upward trend.Communication Link Info: Possible fault for malicious activity.Analysis: The increasing number of neighboring nodes and routing table caches, along with the indication of malicious activity suggests a potential security breach or an attack targeting Node1.Recommended Action: Conduct a thorough security audit, implement intrusion detection systems, and strengthen network security measures to detect and mitigate malicious activity targeting Node1.

Node2: Status: Similar to Node1, with an increasing number of neighboring nodes and routing table caches showing minor fluctuations and an upward trend.Communication Link Info: Also indicates a possible fault for appdown.Analysis: The pattern of increasing neighboring nodes and routing table caches, along with the appdown fault indication, points to a potential issue similar to Node0, possibly related to application failures or disruptions.Recommended Action: Conduct a detailed investigation into the application running on Node2, review configurations, and address any software issues or resource constraints to prevent further disruptions.

E.2 Overall Network Analysis

The network is experiencing multiple issues across different nodes, including high packet loss rates, indications of malicious activity, and potential application failures.There might be underlying systemic issues affecting the stability and security of the network, such as insufficient resources, misconfigurations, or vulnerabilities.Addressing these issues promptly is crucial to prevent service disruptions, security breaches, and network performance degradation.