Skip to main content

Historical Developments of Big data

 Big data dates back to the 60/70s when computers were first introduced for data processing. However, in the 90s the term big data was used to describe the growing value of data (volume, variety and velocity) 

In the early 2000s, the introduction of the internet and increase of people having devices meant a massive increase in the amount of data being generated and collected, in turn created new tools and technologies to collect and analyse the data.

In 2004, google introduced MapReduce which allowed large scale data processing on distributed systems using commodity hardware. This tech became the starting point for Hadoop which is an open source platforms for data storage and processing which released in 2006. 

Comments

Post a Comment

Popular posts from this blog

Types of Visualisation

Big data visualisation refers to the techniques and tools used to present the large and complex data into such a way that it’s easy to read and understand. Some examples include: Heat maps which is used to show the amount of data points or activities across different regions and categories.   Network diagrams helps to visualise relationships and interactions in data for example social connections/data flows etc.  Geospatial maps show the mass amount of geographical data with traditional data sets to provide spread apart analysis.  Stream graphs show trends and patterns across loads of different categories and regions.   Same for the parallel coordinates which also show patterns but also correlations across numerous different variable.  Chord diagrams help to identify clusters, patterns, trends etc to help with making intense decisions and also helps with analysis etc.  https://www.geeksforgeeks.org/what-is-big-data-visualization/#what-is-big-data-visualizat...
Post a Comment
Read more

Data Mining Methods

Data mining is the process of using statistical analysis and machine learning to reveal hidden patterns or odd things in large datasets because of this you can help make important decisions and predict what’s going to happen. it’s where you take data such as structured data, an image, video, text etc and train it, deploy and serve it then you can get actionable insights and application events.  The techniques are classification which is used to organise data into different classes or categories so it trains a model on labelled data and uses it to predict the class. Regression is used to predict numbers or continuous values based on relationships so it finds the function or model that best fits the data to make accurate predictions.  Clustering is used to group similar data uncover patterns or structures in the data without any classes or labels.  There is also association rule, anomaly detention, time series analysis, neural networks, decision trees, ensemble methods and ...
Post a Comment
Read more

Types of Problem suited to Big Data Analysis

 One of the greatest challenges is storage with insane amounts of data generated everyday. Also because Unstructured data can’t be stored in traditional databases.  Processing big data is also a problem which refers to reading, analysing etc or useful information from raw information because of this the changing from all this data to finding all the useful parts is very challenging.  Security is another problem because non-encrypted info is more likely to be stolen or damaged which makes it such a large concern for organisations.   https://www.simplilearn.com/challenges-of-big-data-article#:~:text=Storage,be%20stored%20in%20traditional%20databases.
Post a Comment
Read more