Big Data Evolution in Distributed Intelligent Systems

Abstract

The massive quantity of data available in digital form is increasing exponentially. Corporations, government agencies, and even medium-sized companies having huge datasets. These datasets are referred as Big Data but they can’t be processed and analyzed through traditional techniques. A Big Data can be described through its velocity, volume, variety, value, and veracity. A data processing mechanism should be developed considering all these characteristics. Distributed Artificial Intelligence (DAI) provides an efficient way to process and analyze Big Data. DAI is basically a subset of Artificial Intelligence (AI) but it distributed nodes or agents to draw preliminary results which are then combined to develop a final solution. This paper identify the history and challenges of Big Data and DAI. The challenges of big data includes capturing, storing, searching, updating, privacy, visualizing, transferring, and analyzing data. Several DAI frameworks are available which address these challenges. Some of the most common frameworks are batch-only framework, stream-only framework, and hybrid framework. Finally identify these frameworks to find out the best option for a certain database.