Like many other government agencies, NASA faces a number of Big Data challenges from how to store it to how to make it useable. But there’s a general consensus that NASA, along with a handful of other agencies, like HHS, has a more difficult challenge given the vast amount of information the agency collects, the complexity of the data itself, and the high demand from outside organizations wanting to use the data. With a wide range of uses from predicting the Earth’s weather to identifying new planets in the “Goldie Locks” bands of far off solar systems, Big Data from NASA is coveted by scientists and astronomers around the world.
In a recent interview, Eric De Jong, principal investigator for NASA’s Solar System Visualization Project, discussed the Big Data challenge that he and other researchers face on a daily basis. NASA’s researchers face three main challenges – data storage, data processing, and access – when dealing with the data generated not only by space missions, but also experiments being conducted back on earth. To meet these challenges and provide end-users with the information they need, NASA has embraced some creative approaches to speeding up data processing as well as how data is housed and organized to maximize its value to end users.
Instead of looking at additional expensive and proprietary repositories and analytics options, NASA has turned to the cloud to house information. Tom Soderstrom, IT CTO of the Jet Propulsion Lab, has discussed at length why the cloud is vital to NASA’s mission. By using cloud technologies for storage and processing, NASA is able to ramp up server use when large amounts of data are returned from a mission and need to be processed and reduce server capacity once tasks have been completed. This elasticity in capacity afforded by the cloud allows not only enables concurrent processing and analysis for faster results, but reduces overall costs for the agency since it is not a capital expense. As Soderstrom points out in an interview with NBC, his team at NASA also created a cloud commodity trading board to identify the best available pricing for capacity across vendors.
NASA’s innovation with cloud technologies is well recognized across both the private and public sector. Lewis Carr, Senior Director for Market Research at HP Software, in discussing NASA’s need for the cloud commented that “ while estimates of future data collection capacity for next-generation telescopes sound daunting, ranging into the multi-petabyte range, scientist could easily find a community cloud based on inexpensive COTS hardware and Hadoop as the data lake, and the latest in embedded analytics running in columnar databases, will support access by a far flung research community.” He cautioned, however, that “ensuring the integrity of the research and data would mean additional security for diagnostics and monitoring of the data would also be required – a bit of a Big Data challenge unto itself – yet met through equivalent COTS log management and network security software.”