Showing posts with label big data analytics. Show all posts

Tuesday, 17 December 2019

big data analytics 12/17/2019 12:30:00 pm

AWS Data Lake and Analytics Solutions

Overview of AWS Data Lake

Amazon Web Services (AWS) data lake is a place to store data on the cloud when data is ready for the cloud. It can immediately locate the data in Data lake with Amazon Glue that maintains the catalog of the data. AWS Data Lake has the capability of storing almost unlimited data. Backup and Archive operations are optimized through Amazon Glacier. S3 object storage is the place where data is situated, and it is the cheapest of its kind on the cloud. AWS Data Lake can be optimized with various AWS tools that can save costs up to 80% and can process jobs effectively on the scale. You can also explore Azure Data Lake Analytics capabilities in this. Some of the essential components that AWS data lake has been -

S3 object storage

Amazon Simple Storage Service (or, only S3) is object storage that can store any amount of data, any number of files on the cloud. S3 storage can store enterprise data, IoT data, transactional or operational data and so on. Once data is loaded to S3 then this data can be used anytime and anywhere for all kinds of needs. The data in the Data lake may or may not be curated. Amazon S3 has a wide range of S3 classes to choose from for Data storage. Each of them has its capabilities and securities. We can query in-place by using Amazon Athena and Redshift for data processing.

Glacier for Backup and Archive

Amazon Glacier is a service on S3 than enables support for secure Archiving of data and managing backups. Retrievals of data form current Archive stores are very fast as they can access and retrieve data within 5 minutes. It archives the data across three availability zones within a region. The glacier is best suitable for use cases like asset delivery, healthcare information archiving and scientific data storage.

Glue for Data Catalog Operation

Amazon Glue is a Catalog management service that helps to find and catalog the metadata for faster queries and searches over data. Once we point Glue to the data stored in S3 Storage, Glue then sees all the data and loads its metadata such as schema that will help to query and search among that data faster. The purpose of Glue is performing ETL operations on data. Glue is serverless; hence there is no infrastructure set up for it. This feature makes AWS glue is more efficient and beneficial.

AWS Analytics and Its Capabilities

Amazon Web services have the capability of Analytics based on various market trends. AWS analytics is one of the broad and cost-effective services of its kind. It offers multiple services on the cloud such as Interactive Analytics, Operational Analytics, data warehousing, real-time analytics and many more. Every service offered by AWS analytics is best of its kind and is highly optimized to be deployed on Cloud.

Continue Reading: XenonStack/Blog

XenonStack 0

Wednesday, 26 April 2017

streaming analytics 4/26/2017 10:43:00 am

Enabling Real Time Analytics For IoT

What is Fast Data?

A few years ago, we remember the time when it was just impossible to analyze petabytes of data. Then emergence of Hadoop made it possible to run analytical queries on our huge amount of historical data.

As we know Big Data is a buzz from last few years, but Modern Data Pipelines are constantly receiving data at a high ingestion rate. So this constant flow of data at high velocity is termed as Fast Data.

So Fast data is not about just volume of data like Data Warehouses in which data is measured in GigaBytes, TeraBytes or PetaBytes.

Instead, we measure volume but with respect to its incoming rate like MB per second, GB per hour, TB per day. So Volume and Velocity both are considered while talking about Fast Data.

What is Streaming and Real-Time Data

Nowadays, there are a lot of Data Processing platforms available to process data from our ingestion platforms. Some support streaming of data and other supports true streaming of data which is also called Real-Time data.

Streaming means when we are able to process the data at the instant as it arrives and then processing and analyzing it at ingestion time. But in streaming, we can consider some amount of delay in streaming data from ingestion layer.

But Real-time data needs to have tight deadlines in the terms of time. So we normally consider that if our platform is able to capture any event within 1 ms, then we call it as real-time data or true streaming.

But When we talk about taking business decisions, detecting frauds and analyzing real-time logs and predicting errors in real-time, all these scenarios comes to streaming. So Data received instantly as it arrives is termed as Real-time data.

Stream & Real Time Processing Frameworks

So in the market, there are a lot of open sources technologies available like Apache Kafka in which we can ingest data at millions of messages per sec. Also Analyzing Constant Streams of data is also made possible by Apache Spark Streaming, Apache Flink, Apache Storm.

Apache Spark Streaming is the tool in which we specify the time-based window to stream data from our message queue. So it does not process every message individually.

We can call it as the processing of real streams in micro batches.

Whereas Apache Storm and Flink have the ability to stream data in real-time.

Why Real-Time Streaming

As we know that Hadoop, S3 and other distributed file systems are supporting data processing in huge volumes and also we are able to query them using their different frameworks like Hive which uses MapReduce as their execution engine.

Why we Need Real-Time Streaming?

A lot of organizations are trying to collect as much data as they can regarding their products, services or even their organizational activities like tracking employees activities through various methods used like log tracking, taking screenshots at regular intervals.

So Data Engineering allows us to convert this data into structural formats and Data Analysts then turn this data into useful results which can help the organization to improve their customer experiences and also boost their employee's productivity.

But when we talk about log analytics, fraud detection or real-time analytics, this is not the way we want our data to be processed.The actual value data is in processing or acting upon it at the instant it receives.

Imagine we have a data warehouse like hive having petabytes of data in it. But it allows us to just analyze our historical data and predict future.

So processing of huge volumes of data is not enough. We need to process them in real-time so that any organization can take business decisions immediately whenever any important event occurs. This is required in Intelligence and surveillance systems, fraud detection etc.

Earlier handling of these constant streams of data at high ingestion rate is managed by firstly storing the data and then running analytics on it.

But organizations are looking for the platforms where they can look into business insights in real-time and act upon them in real-time.

Alerting platforms are also built on the top of these real-time streams. But Effectiveness of these platform lies in the fact that how truly we are processing the data in real-time.

Use Of Reactive Programming & Functional Programming

Now when we are thinking of building our alerting platforms, anomaly detection engines etc on the top of our real-time data, it is very important to consider the style of programming you are following.

Nowadays, Reactive Programming and Functional Programming are at their boom.

So, we can consider Reactive Programming as subscriber and publisher pattern. Often, we see the column on almost every website where we can subscribe to their newsletter and whenever the newsletter is posted by the publisher, whosoever have got subscription will get the newsletter via email or some other way.

So the difference between Reactive and Traditional Programming is that the data is available to the subscriber as soon as it receives. And it is made possible by using Reactive Programming model.

In Reactive Programming, whenever any events occur, there are certain components (classes) that had registered to that event. So instead of invoking target components by event generator, all targets automatically get triggered whenever any event occurs.

Now when we are processing data at high rate, concurrency is the point of concern. So the performance of our analytics job highly depends upon memory allocation/deallocation. So in Functional Programming, we don’t need to initialize loops/iterators on our own.

We will be using Functional Programming styles to iterate over the data in which CPU itself takes care of allocation and deallocation of data and also makes the best use of memory which results in better concurrency or parallelism.

Streaming Architecture Matters

While Streaming and Analyzing the real-time data, there are chances that some messages can be missed or in short, the problem is how we can handle data errors.

So, there are two types of architectures which are used while building real-time pipelines.

Lambda Architecture:

This architecture was introduced by Nathan Marz in which we have three layers to provide real-time streaming and compensate any data error occurs if any. The three layers are Batch Layer, Speed layer, and Serving Layer.

Continue Reading the full Article At - XenonStack.com/Blog

XenonStack 0

Tuesday, 14 March 2017

iot 3/14/2017 12:48:00 pm

Why We Need Modern Big Data Integration Platform

Data is everywhere and we are generating data from different Sources like Social Media, Sensors, API’s, Databases.

Healthcare, Insurance, Finance, Banking, Energy, Telecom, Manufacturing, Retail, IoT, M2M are the leading domains/areas for Data Generation. The Government is using BigData to improve their efficiency and distribution of the services to the people.

The Biggest Challenge for the Enterprises is to create the Business Value from the data coming from the existing system and from new sources. Enterprises are looking for a Modern Data Integration platform for Aggregation, Migration, Broadcast, Correlation, Data Management, and Security.

Traditional ETL is having a paradigm shift for Business Agility and need of Modern Data Integration Platform is arising. Enterprises need Modern Data Integration for agility and for an end to end operations and decision-making which involves Data Integration from different sources, Processing Batch Streaming Real Time with BigData Management, BigData Governance, and Security.

BigData Type Includes:

What type of data it is

Format of content of data required

Whether data is transactional data, historical data or master data

The Speed or Frequency at which data made to be available

How to process the data i.e. whether in real time or in batch mode

5 V’s to Define BigData

Additional 5V’s to Define BigData

Data Ingestion and Data Transformation

Data Ingestion comprises of integrating Structured/unstructured data from where it is originated into a system, where it can be stored and analyzed for making business decisions. Data Ingestion may be continuous or asynchronous, real-time or batched or both.

Defining the BigData Characteristics: Using Different BigData types, helps us to define the BigData Characteristics i.e how the BigData is Collected, Processed, Analyzed and how we deploy that data On-Premises or Public or Hybrid Cloud.

Data type: Type of data

Transactional

Historical

Master Data and others

Data Content Format: Format of data

Structured (RDBMS)

Unstructured (audio, video, and images)

Semi-Structured

Data Sizes: Data size like Small, Medium, Large and Extra Large which means we can receive data having sizes in Bytes, KBs, MBs or even in GBs.

Data Throughput and Latency: How much data is expected and at what frequency does it arrive. Data throughput and latency depend on data sources:

On demand, as with Social Media Data

Continuous feed, Real-Time (Weather Data, Transactional Data)

Time series (Time-Based Data)

Processing Methodology: The type of technique to be applied for processing data (e.g. Predictive Analytics, Ad-Hoc Query and Reporting).

Data Sources: Data generated Sources

The Web and Social Media

Machine-Generated

Human-Generated etc

Data Consumers: A list of all possible consumers of the processed data:

Business processes

Business users

Enterprise applications

Individual people in various business roles

Part of the process flows

Other data repositories or enterprise applications

Major Industries Impacted with BigData

What is Data Integration?

Data Integration is the process of Data Ingestion - integrating data from different sources i.e. RDBMS, Social Media, Sensors, M2M etc, then using Data Mapping, Schema Definition, Data transformation to build a Data platform for analytics and further Reporting. You need to deliver the right data in the right format at the right timeframe.

BigData integration provides a unified view of data for Business Agility and Decision Making and it involves:

Discovering the Data

Profiling the Data

Understanding the Data

Improving the Data

Transforming the Data

A Data Integration project usually involves the following steps:

Ingest Data from different sources where data resides in multiple formats.

Transform Data means converting data into a single format so that one can easily be able to manage his problem with that unified data records. Data Pipeline is the main component used for Integration or Transformation.

MetaData Management: Centralized Data Collection.

Store Transform Data so that analyst can exactly get when the business needs it, whether it is in batch or real time.

Why Data Integration is required

Make Data Records Centralized: As data is stored in different formats like in Tabular, Graphical, Hierarchical, Structured, Unstructured form. For making the business decision, a user has to go through all these formats before reaching a conclusion. That’s why a single image is the combination of different format helpful in better decision making.

Format Selecting Freedom: Every user has different way or style to solve a problem. User are flexible to use data in whatever system and in whatever format they feel better.

Reduce Data Complexity: When data resides in different formats, so by increasing data size, complexity also increases that degrade decision making capability and one will consume much more time in understanding how one should proceed with data.

Prioritize the Data: When one have a single image of all the data records, then prioritizing the data what's very much useful and what's not required for business can easily find out.

Better Understanding of Information: A single image of data helps non-technical user also to understand how effectively one can utilize data records. While solving any problem one can win the game only if a non-technical person is able to understand what he is saying.

Keeping Information Up to Date: As data keeps on increasing on daily basis. So many new things come that become necessary to add on with existing data, so Data Integration makes easy to keep the information up to date.

Continue Reading The Full Article At - XenonStack.com/Blog

XenonStack 0