Keywords: Analytics, Big Data, Cloud, Scala, R, Python, Space research, fluid dynamic, Automation, solar physics
Scientist in all domains are struggling with the same challenge, they have more data than can be analyzed. With the eve of Internet of Things, powerful sensors measuring all sorts of phenomena, super computers running vast simulations, data production is huge. Scientist are increasingly spending time as data engineers, building systems that can extract, transform and refine data rather than performing actual analysis for interpreting and detecting new ideas. To overcome this challenge we have designed, built a platform and applied it here on a user-case that studies one of the outstanding scientific questions in solar physics, “why the solar corona is hotter than the lower levels of the solar atmosphere?” In this work we used data generated from simulations of a possible solution to this question that looked at how the magnetic energy in the ionized gas is converted into thermal energy and thereby increased the temperature of the corona beyond that of the sun’s surface This work demonstrates how we can use commonly available industrial automation methodologies with latest big data technologies to create an automated, scalable, fast and easy to use “Platform As A Service” system that caters to all of these needs. This resource enables the scientist to be a scientist instead of a data engineer. We will look at implementations of different operations common in analyzing data from fluid dynamics calculations combining the power of Scala, R and Python together with Apache Spark.