Our research aims to address high-level challenges for data management and processing in the context of the HPC-Big Data – AI convergence.
- First, to allow scientists to obtain fast, real-time insight from complex workflows combining extreme- scale computations with data analytics, it becomes necessary to explore how recently emerged Big Data processing techniques (e.g., based on stream processing) can be leveraged with modern in situ/in transit processing approaches used in HPC environments.
- Second, a unified management of the disparate data produced over the lifetime of a scientific activity could dramatically enhance scientists ability to glean knowledge from this data.
- Finally, the explosion of learning and AI provides new tools that can enable much more adaptable data services than available today, which can further optimize such data processing workflows.
