Enabling HPC-Big Data Convergence for Intelligent Extreme-Scale Analytics

Big Data Analytics (BDA) refers to the process of examining and extracting relevant knowledge from sets of data which are so huge, exhibit such a high format variety and are generated at such a high speed that traditional systems for data storage and processing cannot be used any longer in an efficient way to extract knowledge in an acceptable time. Potentially coming from a very large variety of sources (e.g., sensors from the Internet of Things, social networks, business applications), such data are curated, (sometimes partially) stored, processed and fed into analysis engines that build representations through data-driven models that further enable descriptive, predictive and prescriptive analytics to get valuable insights for decision making.

High-Performance Computing (HPC) refers to the use of parallel processing techniques on high-end machines (supercomputers) to solve complex problems from science and industry which require extreme amounts of computation. The major focus is on performance, therefore HPC typically relies on extreme-scale aggregation of the fastest available hardware and software technologies for processing, communication and storage. Examples of representative HPC application areas include physics, biochemistry, materials science, environment and industrial design (e.g., for car manufacturing). Such applications typically rely on modelling and simulation of the evolution of a complex system, thanks to mathematical models expressing physics-based laws underlying that system.

Why HPC and BDA need each other. As HPC supercomputers evolve towards what is called the exascale, they enable simulations at an increasingly high precision, which results in overwhelming amounts of data generated faster and faster. The analysis of HPC-generated data is thus becoming a Big Data Analytics problem, which makes it perfectly relevant to envision the application of Big Data Analytics techniques to get relevant insights from that data. At the same time, Big Data Analytics exhibits an increasing use of high-performance computational capacities in order to allow extremely-fast knowledge extraction to be performed (e.g., real-time high-frequency analysis), to enable timely and precise decisions. Thus HPC and BDA exhibit clearer and clearer dependencies, which has recently motivated intense efforts to identify how the two areas could leverage each other in a converged way.

Why Artificial Intelligence is a catalyst for HPC-Big Data convergence. Big Data Analytics increasingly relies on Machine Learning (ML), a subfield of Artificial Intelligence typically used for data classification and feature extraction. While traditional ML deals with tractable feature extraction, Deep Learning recently attracted a very high interest as a particularly efficient approach when classical machine learning is intractable. DL relies on neural network representations with a high number of layers, able to learn very complex representations and subsequently use them for predictions (inference). It uses dense linear algebra kernels and allows for lower-precision representation and arithmetic, for which general-purpose GPU (GPGPU) accelerators (increasingly available on HPC systems) are a relevant infrastructure. Consequently, DL-based Big Data Analytics generates workloads that naturally fit HPC systems, thereby acting as a catalyst for HPC-Big Data convergence.

The overall challenge: overcome diverged cultures, methodologies and tools. HPC’s initial focus was on computational performance for tightly-coupled workloads requiring fast computation and frequent communication. The associated software stacks and programming models (e.g., MPI, OpenMP) were therefore developed and optimized accordingly. In contrast, traditional Big Data workloads are loosely coupled and can typically be divided into a huge number of independent jobs. BDA frameworks followed a scale-out model where a very high number of standard-performance processing units can be aggregated and used together without substantial communication. This favored the usage of cloud systems as reference infrastructures for BDA. Map-Reduce (consisting of two stages – map and reduce – each of which can be executed efficiently by a large set of highly parallel tasks) emerged as the dominant programming model. It was further generalized to other operators – not only map and reduce – and to multi-stage processing – not only two stages – in frameworks like Spark and Flink.

HPC and BDA thus underwent divergent evolutions motivated by different optimization goals. The major challenge posed by the convergence of HPC-Big Data comes precisely from the difficulty to ”put together” the inherited methodologies and tools as such, as they developed following diverging targets. For instance, preliminary experiments have shown that Big Data Analytics frameworks perform inefficiently on HPC systems and are totally ignorant of the huge optimization potential allowed by the high-performance underlying hardware.

Focus of the thesis: the data processing level. In the high-performance computing area (HPC), the need to get fast and relevant insights from massive amounts of data generated by extreme-scale computations led to the emergence of in situ and in transit processing approaches. They allow data to be visualized and processed in real-time, in an interactive way, as they are produced, as opposed to traditional approach consisting of transferring data off-site after the end of the computation, for offline analysis. In the Big Data area, the search for real-time, fast analysis was materialized through a different approach: stream-based processing, in support to intelligent, ML-based data analytics.