Jorge Carretero - CosmoHub on Hadoop: a web portal to analyze and distribute massive cosmological data

Europe/Madrid
Seminar (IFAE)

Seminar

IFAE

Jelena Aleksic (IFAE), Joern Lange (IFAE Barcelona), John E Ward (IFAE)
Description
We present CosmoHub, a web platform to perform interactive analysis of massive cosmological data without any SQL knowledge. CosmoHub is built on top of Apache Hive, which is an Apache Hadoop ecosystem component, which facilitates reading, writing, and managing large datasets. CosmoHub is hosted at the Port de Informació Científica (PIC) and currently provides support to several international cosmology projects such as the Euclid space ESA mission, the Dark Energy Survey (DES), the Physics of the Accelerated Universe (PAU) and the Marenostrum Institut de Ciències de l'Espai Simulations (MICE). More than two billion objects, from public and private data, as well as observed and simulated data, are available among all projects. In the last three an a half years more than 400 users have produced about 1500 custom catalogs occupying 2TB in compressed format. CosmoHub allows users to access value-added data, to load and explore pre-built datasets and to create their own custom datasets through a guided process. All those datasets can be interactively explored using an integrated visualization tool which includes 1D histogram and 2D heatmap plots. In our current implementation, online analysis of datasets of a billion objects can be done in less than 25 seconds. Finally, all those datasets can be downloaded in three different formats: CSV.BZ2, FITS and ASDF.
Participants
  • Alícia Labián
  • Antonio Pineda
  • Bruno Bourguille
  • Chihaya Anzai
  • Daniel Guberman
  • Daniel Moreno
  • Delgado Jordi
  • Dirk Hornung
  • Elena Planas
  • Enrique Fernandez
  • Federico Sanchez
  • Francesc Torradeflot
  • gloria de la rosa
  • Isaac Esparbé
  • Jelena Aleksić
  • Joaquim Palacio Navarro
  • Jordi Casals
  • José R. Espinosa
  • Juli Mundet
  • Laia Cardiel
  • Leyre Nogués
  • Lluïsa-Maria Mir
  • Machiel Kolstein
  • Manel Martinez
  • Manuel Delfino
  • Mari Carmen Porto
  • Mario Martinez
  • Matthias Jamin
  • Nadia Tonello
  • Oscar Blanch Bigas
  • Oscar Martinez
  • Otger Ballester
  • Paolo Cumani
  • Pilar Casado
  • Rafel Escribano
  • Raimon Casanova
  • Ramon Miquel
  • Sara Strauch
  • Vanessa Acín
  • Xabier Llobregat
    • 13:00 13:05
      Introduction and General Announcements 5m
      Speakers: Joern Lange (IFAE Barcelona), Dr John E Ward (IFAE), Dr Stefano Terzo (IFAE)
    • 13:05 13:25
      Presentation of the Week 20m
      Speaker: Dr Jorge Carretero
      Cosmohub Website
      Slides
    • 13:25 13:35
      Discussion 10m
    • 13:35 14:00
      Pizza 25m
Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×