There’s been a lot of talk about big data and all the benefits it could bring to government agencies.   But it seems that much of that data simply sits on networks, creating security risk, rather than actually being used for projects to benefit citizens.

Slowly, but surely, that balance is changing, and it may be that 2013 is the year big data fulfills its promise.  Recently, several agencies announced projects that leverage big data.  The Department of Defense says it will develop a “Cyber Targeted Attack Analyzer” to analyze and organize information from across many sources to help highlight abnormalities.

This is a smart move.  As we all know, the government is shorthanded in the IT Security sector.  Automating the first level of analysis provides increased efficiencies, allowing analysts to concentrate on the relationships between valid data, rather than sift through millions of bytes of potentially meaningless data to find the few that matter.

Also moving into the next phase of a big data project is the Transportation Security Administration.  In a similar vein to the Department of Defense’s analyzer, the TSA is evaluating a tool that would mine data from disparate sources such as email, cloud, and hard drives to derive patterns of activity that are worthy of additional, human-level analysis.  This will help the TSA to not only prevent breaches, but build stronger cases for conviction should an event occur.

What other big data projects do you think the federal agencies should pursue?