Published July 2, 2018 | Version v1
Presentation Open

Data validation beyond Big Data

  • 1. Kapteyn Astronomical Institute

Description

From KiDs  to Euclid OU-Ext   to Euclid  data validation.
For the OmegaCAM@VST  datahandling we have build  and operated the distributed information system Astro-WISE. Astro-WISE was successfully used for the processing of KiDS data and particularly its built in extreme data-lineage facilitated the quality control and re-processing of the data with improved calibrations  and improved code.
Many of the aspects of the Astro-WISE approach will be applied in the data centric information system  being build for the data processing for the Euclid satellite. However, the large amounts of data from Euclid in combination with the required much higher accuracies and danger of plural hidden systematics and biases   forces to anticipate  a new era  beyond the Big data hype: data validation.  In popular terms discriminating facts and fakes.
I will discuss some new steps towards advanced data validation, such as   build in dynamical reference systems in the  OU-Ext approach,  the validation of and by machine learning,  and applying extreme data lineage to trace the roots and dependencies  of data products.

Files

Valentijn.pdf

Files (3.5 MB)

Name Size Download all
md5:53c599429649ae14b944a3d4c5c89dd3
3.5 MB Preview Download