Artificial Intelligence Overview

AI refers to ‘Artificial Intelligence’ which means making machines capable of performing intelligent tasks like human beings. AI performs automated tasks using intelligence.

The term Artificial Intelligence has two key components -

• Automation  
• Intelligence

Goals of Artificial Intelligence

Stages of Artificial Intelligence

Stage 1 - Machine Learning - It is a set of algorithms used by intelligent systems to learn from experience.

Stage 2 - Machine Intelligence - These are the advanced set of algorithms used by machines to learn from experience. Eg - Deep Neural Networks.

ArtificiaI Intelligence technology is currently at this stage.

Stage 3 - Machine Consciousness - It is self-learning from experience without the need of external data.

Types of Artificial Intelligence

ANI - Artificial Narrow Intelligence - It comprises of basic/role tasks such as those performed by chatbots, personal assistants like SIRI by Apple and Alexa by Amazon.

AGI - Artificial General Intelligence - Artificial General Intelligence comprises of human-level tasks such as performed by self-driving cars by Uber, Autopilot by Tesla. It involves continual learning by the machines.

ASI - Artificial Super Intelligence - Artificial Super Intelligence refers to intelligence way smarter than humans.

What Makes System AI Enabled

Difference Between NLP, AI, ML, DL & NN

AI or Artificial Intelligence - Building systems that can do intelligent things.

NLP or Natural Language Processing - Building systems that can understand language. It is a subset of Artificial Intelligence.

ML or Machine Learning - Building systems that can learn from experience. It is also a subset of Artificial Intelligence.

NN or Neural Network - Biologically inspired network of Artificial Neurons.

DL or Deep Learning - Building systems that use Deep Neural Network on a large set of data. It is a subset of Machine Learning.

What is Natural Language Processing?

Natural Language Processing (NLP) is “ability of machines to understand and interpret human language the way it is written or spoken”.

The objective of NLP is to make computer/machines as intelligent as human beings in understanding language.

The ultimate goal of NLP is to the fill the gap how the humans communicate(natural language) and what the computer understands(machine language).

There are three different levels of linguistic analysis done before performing NLP -

Syntax - What part of given text is grammatically true.

Semantics - What is the meaning of given text?

Pragmatics - What is the purpose of the text?

NLP deal with different aspects of language such as

• Phonology - It is systematic organization of sounds in language.

• Morphology - It is a study of words formation and their relationship with each other.

Approaches of NLP for understanding semantic analysis

• Distributional - It employs large-scale statistical tactics of Machine Learning and Deep Learning.
• Frame - Based - The sentences which are syntactically different but semantically same are represented inside data structure (frame) for the stereotyped situation.
• Theoretical - This approach is based on the idea that sentences refer to the real word (the sky is blue) and parts of the sentence can be combined to represent whole meaning.
• Interactive Learning - It involves pragmatic approach and user is responsible for teaching the computer to learn the language step by step in an interactive learning environment. 

The true success of NLP lies in the fact that humans deceive into believing that they are talking to humans instead of computers.

Why Do We Need NLP?

With NLP, it is possible to perform certain tasks like Automated Speech and Automated Text Writing in less time.

Due to the presence of large data (text) around, why not we use the computers untiring willingness and ability to run several algorithms to perform tasks in no time.

These tasks include other NLP applications like Automatic Summarization (to generate summary of given text) and Machine Translation (translation of one language into another)

Process of NLP

In case the text is composed of speech, speech-to-text conversion is performed.

The mechanism of Natural Language Processing involves two processes:

• Natural Language Understanding
• Natural Language Generation

Natural Language Understanding

NLU or Natural Language Understanding tries to understand the meaning of given text. The nature and structure of each word inside text must be understood for NLU. For understanding structure, NLU tries to resolve following ambiguity present in natural language:

• Lexical Ambiguity - Words have multiple meanings
• Syntactic Ambiguity - Sentence having multiple parse trees.
• Semantic Ambiguity - Sentence having multiple meanings
• Anaphoric Ambiguity - Phrase or word which is previously mentioned but has a different meaning.

Next, the meaning of each word is understood by using lexicons (vocabulary) and set of grammatical rules.

However, there are certain different words having similar meaning (synonyms) and words having more than one meaning (polysemy).

Natural Language Generation

It is the process of automatically producing text from structured data in a readable format with meaningful phrases and sentences. The problem of natural language generation is hard to deal with. It is subset of NLP

Natural language generation divided into three proposed stages:-

1. Text Planning - Ordering of the basic content in structured data is done.

2. Sentence Planning - The sentences are combined from structured data to represent the flow of information.

3. Realization - Grammatically correct sentences are produced finally to represent text.

Difference Between NLP and Text Mining or Text Analytics

Natural language processing is responsible for understanding meaning and structure of given text.

Text Mining or Text Analytics is a process of extracting hidden information inside text data through pattern recognition.

Natural language processing is used to understand the meaning (semantics) of given text data, while text mining is used to understand structure (syntax) of given text data.

As an example - I found my wallet near the bank. The task of NLP is to understand in the end that ‘bank’ refers to financial institute or ‘river bank'.

What is Big Data?

According to the Author Dr. Kirk Borne, Principal Data Scientist, Big Data Definition is described as big data is everything, quantified, and tracked.

For More Details on Big Data, Please Read - Ingestion And Processing of Data For Big Data and IoT Solutions

NLP for Big Data is the Next Big Thing

Today around 80 % of total data is available in the raw form. Big Data comes from information stored in big organizations as well as enterprises. Examples include information of employees, company purchase, sale records, business transactions, the previous record of organizations, social media etc.

Though human uses language, which is ambiguous and unstructured to be interpreted by computers, yet with the help of NLP, this huge unstructured data can be harnessed for evolving patterns inside data to better know the information contained in data.

NLP can solve big problems of the business world by using Big Data. Be it any business like retail, healthcare, business, financial institutions.

What is Chatbot?

Chatbots or Automated Intelligent Agents

• These are the computer program you can talk to through messaging apps, chat windows or through voice calling apps
• These are intelligent digital assistants used to resolve customer queries in a cost-effective, quick, and consistent manner.

Importance of Chatbots

Chatbots are important to understanding changes in digital customer care services provided and in many routine queries that are most frequently enquired.

Chatbots are useful in a certain scenario when the customer service requests are specific in the area and highly predictable, managing a high volume of similar requests, automated responses.

Working of Chatbot

Knowledge Base - It contains the database of information that is used to equip chatbots with the information needed to respond to queries of customers request.

Data Store - It contains interaction history of chatbot with users.

Continue Reading About AI & NLP  At: XenonStack.com/Blog

What is Log Analytics

Understanding Log Data

Log Analysis Process

• Collection and Cleaning of data
• Structuring of Data
• Analysis of Data

Collection and Cleaning of data

• Crashing of disk where log data is stored
• Applications are terminated abnormally
• Disturbance in the configuration of input/output
• Presence of virus in the system and much more

Structuring of Data

• Clarity about the usage of collected log data
• Same assets involve across the data so that values of log data are consistent. This means that naming conventions can be used
• Correlation between the objects is created automatically due to the presence of nested files in the log data. It’s better to avoid nested files from the log data.

Importance of Log Analysis

Knowledge Discovery and Data Mining

Log Mining

• Day by day volume of log data is increasing from megabytes to gigabytes or even petabytes. Therefore, there is a need of advanced tools for log analysis.
• The essential information is missing from the log data. So, more efforts are needed to extract useful data.
• The different number of logs are analyzed from different sources to move deep into the knowledge. So, logs in different formats have to be analyzed.
• The presence of different logs creates the problem of redundancy of data without any identification. This leads to the problem of synchronization between the sources of log data.

Define Anomaly

• Point Anomalies

  A single instance of a point is considered as an anomaly when it is farthest from the rest of the data.

• Contextual Anomalies

  This type of anomaly related to the abnormal behavior of the particular type of context within data. It is commonly observed in time series problems.

• Collective anomalies

  When the collected instance of data is help for detecting anomalies is considered as collective anomalies.

Network Intrusion Detection using Data Mining

What is Fast Data?

What is Streaming and Real-Time Data

Stream & Real Time Processing Frameworks

Why Real-Time Streaming

Why we Need Real-Time  Streaming?

Use Of Reactive Programming & Functional Programming

Streaming Architecture Matters

• Lambda Architecture:

  This architecture was introduced by Nathan Marz in which we have three layers to provide real-time streaming and compensate any data error occurs if any. The three layers are Batch Layer, Speed layer, and Serving Layer.
  

What is PostgreSQL?

• It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, SGI IRIX, MacOS, Solaris, Tru64), and Windows.

• It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages)

• It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP.

• It also supports storage of binary large objects, including pictures, sounds, or video.

• It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

Prerequisites

• Kubernetes Cluster

• GlusterFS Cluster

FROM ubuntu:latest
MAINTAINER XenonStack

RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8

RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" > /etc/apt/sources.list.d/pgdg.list

RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6

RUN /etc/init.d/postgresql start &&\
 psql --command "CREATE USER root WITH SUPERUSER PASSWORD 'xenonstack';" &&\
 createdb -O root xenonstack

RUN echo "host all  all 0.0.0.0/0  md5" >> /etc/postgresql/9.6/main/pg_hba.conf

RUN echo "listen_addresses='*'" >> /etc/postgresql/9.6/main/postgresql.conf

# Expose the PostgreSQL port
EXPOSE 5432

# Add VOLUMEs to allow backup of databases
VOLUME  ["/var/lib/postgresql"]

# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.6/bin/postgres", "-D", "/var/lib/postgresql", "-c", "config_file=/etc/postgresql/9.6/main/postgresql.conf"]

$ docker build -t dr.xenonstack.com:5050/postgres:v9.6

$ gluster volume create postgres-disk replica 2 transport tcp k8-master:/mnt/brick1/postgres-disk  k8-1:/mnt/brick1/postgres-disk
$ gluster volume start postgres-disk
$ gluster volume info postgres-disk

• Docker Image: We have created a Docker Image for Postgres in Step 2

• Persistent Shared Storage Volume: We have created a Persistent Shared Storage Volume in Step 3

• Deployment & Service Files: Next, we will create Deployment & Service Files

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
  namespace: production
spec:
  replicas: 1
  template:
 metadata:
   labels:
    k8s-app: postgres
 spec:
   containers:
   - name: postgres
     image: dr.xenonstack.com:5050/postgres:v9.6
     imagePullPolicy: "IfNotPresent"
     ports:
     - containerPort: 5432
     env:
     - name: POSTGRES_USER
       value: postgres
     - name: POSTGRES_PASSWORD
       value: superpostgres
     - name: PGDATA
       value: /var/lib/postgresql/data/pgdata
     volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgredb
   volumes:
     - name: postgredb
       glusterfs:
         endpoints: glusterfs-cluster
         path: postgres-disk
         readOnly: false

• What type of data it is

• Format of content of data required

• Whether data is transactional data, historical data or master data

• The Speed or Frequency at which data made to be available

• How to process the data i.e. whether in real time or in batch mode

5 V’s to Define BigData

Additional 5V’s to Define BigData

Data Ingestion and Data Transformation

• Data type: Type of data

• Transactional

• Historical

• Master Data and others

• Data Content Format: Format of data

• Structured (RDBMS)

• Unstructured (audio, video, and images)

• Semi-Structured

• Data Sizes: Data size like Small, Medium, Large and Extra Large which means we can receive data having sizes in Bytes, KBs, MBs or even in GBs.

• Data Throughput and Latency: How much data is expected and at what frequency does it arrive. Data throughput and latency depend on data sources:

• On demand, as with Social Media Data

• Continuous feed, Real-Time (Weather Data, Transactional Data)

• Time series (Time-Based Data)

• Processing Methodology: The type of technique to be applied for processing data (e.g. Predictive Analytics, Ad-Hoc Query and Reporting).

• Data Sources: Data generated Sources

• The Web and Social Media

• Machine-Generated

• Human-Generated etc

• Data Consumers: A list of all possible consumers of the processed data:

• Business processes

• Business users

• Enterprise applications

• Individual people in various business roles

• Part of the process flows

• Other data repositories or enterprise applications

Major Industries Impacted with BigData

What is Data Integration?

• Discovering the Data

• Profiling the Data

• Understanding the Data

• Improving the Data

• Transforming the Data

• Ingest Data from different sources where data resides in multiple formats.

• Transform Data means converting data into a single format so that one can easily be able to manage his problem with that unified data records. Data Pipeline is the main component used for Integration or Transformation.

• MetaData Management: Centralized Data Collection.

• Store Transform Data so that analyst can exactly get when the business needs it, whether it is in batch or real time.



Why Data Integration is required

• Make Data Records Centralized: As data is stored in different formats like in Tabular, Graphical, Hierarchical, Structured, Unstructured form. For making the business decision, a user has to go through all these formats before reaching a conclusion. That’s why a single image is the combination of different format helpful in better decision making.

• Format Selecting Freedom: Every user has different way or style to solve a problem. User are flexible to use data in whatever system and in whatever format they feel better.

• Reduce Data Complexity: When data resides in different formats, so by increasing data size, complexity also increases that degrade decision making capability and one will consume much more time in understanding how one should proceed with data.

• Prioritize the Data: When one have a single image of all the data records, then prioritizing the data what's very much useful and what's not required for business can easily find out.

• Better Understanding of Information: A single image of data helps non-technical user also to understand how effectively one can utilize data records. While solving any problem one can win the game only if a non-technical person is able to understand what he is saying.

• Keeping Information Up to Date: As data keeps on increasing on daily basis. So many new things come that become necessary to add on with existing data, so Data Integration makes easy to keep the information up to date.