An online parsing framework for semi-structured streaming system logs of Internet of Things systems

This paper presents a novel log abstraction framework based on Neural Open Information Extraction (OpenIE) and dynamic word embedding principles. Though various log parsing frameworks are proposed in the literature, the existing frameworks are modelled on predefined heuristics or auto-regressive methodologies that work well in offline scenarios. However, these frameworks are less suitable for dynamic self-adaptive systems, such as the Internet of Things (IoT), where the log outputs have diverse contextual variations and disparate time irregularities. Therefore, it is essential to move away from these traditional approaches and develop a systematic model that can effectively analyse log outputs in real-time and increase the system up-time of IoT networks so that they are almost always available. To address these needs, the proposed framework used OpenIE along with Term Frequency/Inverse Document Frequency (TF/IDF) vectorisation for constructing a set of relational triples (aka triple-sets). Additionally, a dynamic pre-trained encoder-decoder architecture is utilised to imbibe the positional and contextualised information in its resultant outputs. The adopted methodology has enabled the proposed framework to extract richer word representations with dynamic contextualisation of time-sensitive event logs to enhance further downstream activities such as failure prediction and prognostic analysis of IoT networks. The proposed framework is evaluated on the system event log traces accumulated from a LoRaWAN (Long Range Wide Area Network) Internet of Things (IoT) gateway to proactively determine the probable causes of its various failure scenarios. Additionally, the study also provided a comparative analysis of its mathematical representations with that of the current state-of-the-art (SOTA) approaches to project the advantages and benefits of the proposed model, particularly from its data analytics standpoint.