Some key events in evolution of Data Warehouse-
1960- Dartmouth and General Mills in a joint research project, develop the
terms dimensions and facts.
1970- A Nielsen and IRI introduces dimensional data marts for retail
sales.
1983- Tera Data Corporation introduces a database management system
which is specifically designed for decision support
# The idea of data warehousing came to the late 1980's when IBM
researchers Barry Devlin and Paul Murphy established the "Business
Data Warehouse.".
However, the real concept was given by Inmon Bill. He was considered as
a father of data warehouse. He had written about a variety of topics for
building, usage, and maintenance of the warehouse & the Corporate
Information Factory.
# The term "Data Warehouse" was first coined by Bill Inmon in 1990.
According to Inmon, a data warehouse is a subject oriented, integrated, time-
variant, and non-volatile collection of data. This data helps analysts to take
informed decisions in an organization.
In short, the data warehousing idea was planned to support an architectural model
for the flow of information from the operational system to decisional support
environments. The concept attempt to address the various problems associated
with the flow, mainly the high costs associated with it.
In the absence of data warehousing architecture, a vast amount of space was
required to support multiple decision support environments. In large
corporations, it was ordinary for various decision support environments to operate
independently.
Goals of Data Warehousing
o To help reporting as well as analysis
o Maintain the organization's historical information
o Be the foundation for decision making.
Data Warehouse is needed for the following reasons:
1. Business User: Business users require a data warehouse to view summarized
data from the past. Since these people are non-technical, the data may be
presented to them in an elementary form.
2. Store historical data: Data Warehouse is required to store the time variable
data from the past. This input is made to be used for various purposes.
3. Make strategic decisions: Some strategies may be depending upon the data
in the data warehouse. So, data warehouse contributes to making strategic
decisions.
4. For data consistency and quality: Bringing the data from different sources at
a commonplace, the user can effectively undertake to bring the uniformity and
consistency in data.
5. High response time: Data warehouse has to be ready for somewhat
unexpected loads and types of queries, which demands a significant degree of
flexibility and quick response time.
Data warehousing is a method of organizing and compiling data into one
database, whereas data mining deals with fetching important data from databases.
Data mining attempts to depict meaningful patterns through a dependency on the
data that is compiled in the data warehouse.
DATA WAREHOUSE:
Data warehousing is a unique technique that helps collect and manage data from
various sources. A data warehouse is where data can be collected for mining
purposes, usually with large storage capacity. Various organizations’ systems are
in the data warehouse, where it can be fetched as per usage.
Source Extract Transform Load Target.
(Data warehouse process)
Data warehouses collaborate data from several sources and ensure data accuracy,
quality, and consistency. In a data warehouse, data is sorted into a formatted pattern
by type and as needed. The data is examined by query tools using several patterns.
Data warehouses store historical data and handle requests faster, helping in online
analytical processing, whereas a database is used to store current transactions in a
business process that is called online transaction processing.
# A Data Warehouse provides integrated, enterprise-wide, historical data and
focuses on providing support for decision-makers for data modelling and analysis.
FEATURES OF DATA WAREHOUSES:
Time - Variant
The data collected in a data warehouse is identified with a specific period. It means
data warehouse has to contain historical data, not just current values.
1.) Allow the analysis of past.
2.) Relate information to the present
3.) Enables forecast to future
Integrated:
Different sources are put together to build a data warehouse, such as level documents
or social databases. Data in Data warehouse comes from several operational system.
Before data integration some steps are follows:--
1.) Remove inconsistency
2.) Transformation
3.) Integration of source data
Non-volatile:
This means the earlier data is not deleted when new data is added to the data
warehouse. Data granularity The operational database and data warehouse are kept
separate and thus continuous changes in the operational database are not shown in
the data warehouse.
Subject Oriented
It provides you with important data about a specific subject like suppliers, products,
promotion, customers, etc. Data warehousing usually handles the analysis and
modelling of data that assist any organization to make data-driven decisions.
Applications of Data Warehouses:
Banking Services
Consumer Goods
Manufacturing
Financial Services
Retail Sectors
Benefits of Data Warehousing
The most compelling advantages of data warehousing are:
Improved performance and productivity
Cost-effective
Consistent and accurate data access
What is Data Mining?
In this process, data is extracted and analysed to fetch useful information. In data
mining hidden patterns are researched from the dataset to predict future behaviour.
Data mining is used to indicate and discover relationships through the data.
Data mining uses statistics, artificial intelligence, machine learning systems, and
some databases to find hidden patterns in the data. It supports business-related
queries that are time-consuming to resolve.
Features of Data Mining
Some of the unique features of data mining are:
It is capable of predicting future results
It can efficiently handle large datasets and databases
It can seamlessly utilise the automated discovery of patterns
It has the potential to create actionable insights, etc.
Applications of Data Mining
Research
Education Sector
Transportation
Market Basket Analysis
Business Transactions
Intrusion Detection
Scientific Analysis
Finance and Banking Sector
Insurance and Healthcare
Benefits of Data Mining
Analysing trends within the existing marketplace.
Detecting frauds in phone calls, insurance claims, debit or credit
purchases, etc.
It can make easy predictions within the market before making business
decisions.
Difference Between Data Mining and Data Warehousing
Data Mining Data Warehousing
This procedure involves analysing It is exquisitely designed for analytical
data patterns analysis
Regular data analysis Periodical data storage
Uses pattern recognition logic to Extracts and stores data to enable easy
identify patterns reporting
It is carried out by business users with
It is carried out by engineers
the help of engineers
It helps in extracting data from large
It pools all the relevant data together
data sets
Statistics, AI, Machine Learning, and Integrated, subject-oriented, non-volatile,
Databases are used in data mining and time-varying constitute data
technologies warehouses
Pattern recognition logic is used for It involves extracting and storing data in
determining patterns perfect order to make efficient reporting
Extracts are stores data in an orderly
Employs pattern recognition tools to
format, thereby making reporting faster
help identify the access patterns
and easier
When connected with operational
It helps in creating suggestive patterns
business systems like CRM, it deliberately
of key parameters
adds value to it
Common Tools and Software Used in Data Warehousing and Data Mining
Let’s look at the common tools and software used in data warehousing and data
mining: Some of the popular data warehouse tools are:
Amazon Redshift
Microsoft Azure
Google BigQuery
Snowflake
Micro Focus Vertica
Amazon DynamoDB
Some of the popular data mining tools are:
RapidMiner
MonkeyLearn
IBM SPSS Modeler
Oracle Data Mining
Knime
Weka
Orange
H2O
Apache Mahout
SAS Enterprise Miner
Common Data Mining and Data Warehousing Techniques
Let’s look at the common techniques used in data warehousing vs data mining:
The most common techniques of data mining are:
Association
Clustering
Data Visualisation
Data Cleaning
Machine Learning
Classification
Neural Networks
Prediction
Data Warehousing
Outlier Detection
Common Data Warehousing Techniques
Some of the most common data warehousing techniques are:
Database Compression
Columnar Data Storage
In-Memory Processing
Massive Parallel Processing (MPP)
Scope of Data Mining & Data Warehouse
The scope of data mining vs data warehousing is different from each other. Data
mining involves sorting enormous data sets to identify relationships and patterns
that can easily solve business problems through data analysis. The scope and
techniques of data mining enable enterprises to predict future trends and make
informed business decisions.
On the other hand, the scope of data warehousing lies within any domain that has
something to do with analytics. Now, let us discuss the differences between data
mining and data warehousing challenges faced.
Challenges of Data Mining & Data Warehousing
Some of the most common challenges of data mining vs data warehouse
challenges:
Some of the most common challenges experienced by data mining are:
Incomplete and noisy data
Social and security challenges
Complex data
Distributed data
Efficiency and scalability of algorithms
Performance
Incorporating background knowledge
Improving mining algorithms, etc.
Some of the most common challenges experienced by data warehousing are:
Manual data processing
Data quality
Data Accuracy
Testing
Performance, etc.
