Hi, Thanks for creating this community, it will definitely help us communicate in a lot better way and share our thoughts on Big data management. Pitching in, i would like to share my thoughts here to get started and the current scenario of big data -
Big data is nothing new to large organizations, however, it’s also becoming popular among smaller and medium sized firms due to cost reduction and provided ease to manage data. Big Data is the result of practically everything in the world being monitored and measured, creating data faster than the available technologies can store, process or manage it.
Since it is a lot more intuitive to represent information as a “file” than a relational object, there has been a surge of unstructured data, making up as much as 80% of new data we must manage.
Organizations are struggling to manage Big Data. According to IDC, the amount of information created, captured or replicated has exceeded available storage for the first time since 2007. The size of the digital universe this year will be tenfold what it was just five years earlier.
Therefore, organizations must find smarter data management approaches that enable them to effectively corral and optimize their data.
Too many organizations think they can manage Big Data by throwing increasing amounts of storage at the problem.
They often buy additional storage capacity every 6-to-12 months, which not only results in exorbitant costs but forces their frazzled IT teams to spend more time on data management rather than more strategic IT initiatives.
The lack of a real solution for managing Big Data simply causes tremendous inefficiencies all across the organization.
At the same time, Big Data just keeps growing and growing,
-The average organization will grow their data by 50 percent in the coming year.
--Overall corporate data will grow by a staggering 94 percent.
--Database systems will grow by 97 percent.
--Server backups for disaster recovery and continuity will expand by 89 percent.
Big Data results in three basic challenges: storing, processing and managing it efficiently.
What most people don’t know is that the vast majority of Big Data is either duplicated data or synthesized data.
So how should it be managed?
The first step is to bring the data down to its unique set and reduce the amount of data to be manage
Leverage the power of virtualization technology.
Organizations must virtualize this unique data set so that not only multiple applications can reuse the same data footprint, but also the smaller data footprint can be stored on any vendor-independent storage device.
Virtualization is the secret weapon that organizations can wield to battle the Big Data management challenge.
A smarter data management approach not only allows Big Data to be backed up far more effectively but also makes it more easily recoverable and accessible with a whopping 90% cost savings - while freeing IT staff to drive more strategic technology initiatives that drive corporate growth instead of engaging in a futile battle with an out-of-control Big Data beast.
Big Data Security Issues are Surfacing
Big data brings with it new security concerns.
Although our data capacity is growing exponentially, we have imperfect solutions for the many security issues that affect even local, self-contained data.
Hacking technology outstrips defensive technologies, and there are demons that haunt large organizations in the area of website security.
Cloud-based storage has facilitated data mining and collection. However, this big data and cloud storage integration has caused a challenge to privacy and security threats.
The reason for such breaches may also be that security applications or websites that are designed to store certain amounts of data cannot the big volumes of data that the data sets have. Also, these security technologies are inefficient to manage dynamic data and can control static data only.
Therefore, just a regular security check can not detect security patches for continuous streaming data. For this purpose, you need full-time privacy while data streaming and big data analysis.
Organizations must ensure that all big data bases are immune to security threats and vulnerabilities. During data collection, all the necessary security protections such as real-time management should be fulfilled.
Keeping in mind the huge size
of big data, organizations should remember the fact that managing
such data could be difficult and requires extraordinary efforts.
However, taking steps would help maintain consumer
Big data increases the risk. For one thing, big data breaches will be big breaches. For another, the more information you have, the more likely it is that it includes personal or sensitive information.
Building big data infrastructure in-house is a major investment of time and money for research, hardware, software, and countless other details, so most organizations will not install their own big data infrastructure. T
Thus, big data in the cloud and its security should be considered.Technologies for dealing with some of these issues have become more robust in response to big data demands.
Encryption is a crucial part of maintaining confidentiality and integrity of data. The problem is not an absence of technologies, but the absence of an all-inclusive systemic approach or framework. Big data analytics itself can be used to identify threats, including differentiating real threats from false positives.
Logs can be mined for anomalous connections to the cluster. Improved analytics can help separate out the false positives
It is worth noting that an emerging area of security uses big data analytics to detect threats at an early stage, applying pattern analysis to identify anomalies that may indicate a breach or a threat. It indicates that big data analytics is an effective technique with extensible applications.
Website Security Advisor/Architect @WP Hacked Help