Speaker
Description
Modern astronomy and astrophysics produce massively large data volumes (in the order of petabytes) coming from observations or simulation codes executed on high performance supercomputers. Such data volumes pose significant challenges for storage, access and data analysis. Visual exploration of big datasets poses some critical challenges that must drive the development of a new generation of graphical software tools, specifically: (i) Interactivity to deal with datasets exceeding the local machine’s memory capacity, so for complex visualizations the relevant computations should be performed close to the data to avoid time consuming streaming of large data volumes; ii) Integration to be ideally fully integrated within the scientists’ toolkit for seamless usage, abstracting from technical details related to the underlying high performance computing (HPC) resources, freeing scientists to concentrate in doing science; iii) Collaboration to facilitate visualization, processing and analysis of big data in a collaborative manner within e.g. science gateway technologies to allow collaborative activity between users and provide customization and scalability of data analysis/processing workflows, hiding underlying technicalities.
INAF-OACT has been developing and maintaining the Visualization Interface for the Virtual Observatory (VisIVO) and recently extended it with the ViaLactea Visual Analytic modules. VisIVO is developed adopting the Virtual Observatory standards and its main objective is to perform 3D and multi-dimensional data analysis and knowledge discovery of a-priori unknown relationships between multi-variate and complex astrophysical datasets. VisIVO has been already deployed using Science Gateways to access DCIs (including clusters, grids and clouds) using containerization and virtualization technologies, it has also been selected as one of the pilot applications deployed on EOSCpilot infrastructure demonstrating that the tools can be accessed using gateways and cloud platforms and it has been deployed on the European Open Science Cloud (EOSC), efficiently exploiting Cloud infrastructures and interactive notebooks applications.
During this period, and thanks to the collaboration within the SPACE EU Centre of Excellence, the H2020 EUPEX Project and the ICSC National Research Centre for High Performance Computing, Big Data and Quantum Computing, we are planning to adapt VisIVO solutions for high performance visualisation, remote, in-situ, in-transit visualisation of data generated on the (pre-)Exascale systems by HPC applications in Astrophysics and Cosmology (A&C) including GADGET (GAlaxies with Dark matter and Gas intEracT) simulated data and astrophysical fluid dynamics (PLUTO) simulations.
In this talk I will present the evolution direction and the related implementation activities tailored to pursue the following objectives: 1) Enhance the portability of the VisIVO modular applications and their resource requirements. 2) Foster reproducibility and maintainability; 3) Take advantage of a more flexible resource exploitation over heterogeneous HPC facilities (including also mixed HPC-Cloud resources); 4) Minimize data-movement overheads and improve I/O performances.
Acknowledgements
This work is funded by the European High Performance Computing Joint Undertaking (JU) and Belgium, Czech Republic, France, Germany, Greece, Italy, Norway, and Spain under grant agreement No 101093441 and it is supported by the spoke "FutureHPC & BigData” of the ICSC – Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing – and hosting entity, funded by European Union – NextGenerationEU”
