An online parsing framework for semi-structured streaming system logs of internet of Things systems.pdf (7.34 MB)

An online parsing framework for semi-structured streaming system logs of Internet of Things systems

Download (7.34 MB)
This paper presents a novel log abstraction framework based on Neural Open Information Extraction (OpenIE) and dynamic word embedding principles. Though various log parsing frameworks are proposed in the literature, the existing frameworks are modelled on predefined heuristics or auto-regressive methodologies that work well in offline scenarios. However, these frameworks are less suitable for dynamic self-adaptive systems, such as the Internet of Things (IoT), where the log outputs have diverse contextual variations and disparate time irregularities. Therefore, it is essential to move away from these traditional approaches and develop a systematic model that can effectively analyse log outputs in real-time and increase the system up-time of IoT networks so that they are almost always available. To address these needs, the proposed framework used OpenIE along with Term Frequency/Inverse Document Frequency (TF/IDF) vectorisation for constructing a set of relational triples (aka triple-sets). Additionally, a dynamic pre-trained encoder-decoder architecture is utilised to imbibe the positional and contextualised information in its resultant outputs. The adopted methodology has enabled the proposed framework to extract richer word representations with dynamic contextualisation of time-sensitive event logs to enhance further downstream activities such as failure prediction and prognostic analysis of IoT networks. The proposed framework is evaluated on the system event log traces accumulated from a LoRaWAN (Long Range Wide Area Network) Internet of Things (IoT) gateway to proactively determine the probable causes of its various failure scenarios. Additionally, the study also provided a comparative analysis of its mathematical representations with that of the current state-of-the-art (SOTA) approaches to project the advantages and benefits of the proposed model, particularly from its data analytics standpoint.






Start Page


End Page


Number of Pages





Institute of Electrical and Electronics Engineers (IEEE)

Additional Rights

CC BY 4.0

Peer Reviewed

  • Yes

Open Access

  • Yes

Acceptance Date


Author Research Institute

  • Centre for Intelligent Systems

Era Eligible

  • Yes


IEEE Open Journal of Instrumentation and Measurement

Article Number