Chatbot Development and Platform with Machine Learning

Overview of Building ChatBots with Deep Learning

A ChatBot is an implementation of Conversational Interface Intelligently comprising of Machine Learning, Deep Learning as their backbone. ChatBots hold variety including be Textual, Voice and Image-based interactions.

The growth of chatbots has opened up new areas of customer engagement and new methods of fulfilling business in the form of conversational commerce. It is the most useful technology that businesses can rely on, possibly following the old models and producing apps and websites redundant. A chatbot is a computer program that copies human communications in its natural format including text or spoken language using AI procedures such as Natural Language Processing, image, video processing, and audio analysis. The most impressive characteristic of the bots is that they learn from past interactions and become intelligent and more intelligent over time. Chatbots works in two ways- rule-based and smart machine-based. Rule-based chatbots give predefined responses from a database, based on the keywords used for the research. However, smart machine based chatbots receive their capabilities from Artificial Intelligence and Cognitive Computing and adapt their operation based on customer interactions.

The chatbot can be of two types: Goal-oriented (such as Siri, Alexa, Cortana, etc.) and General Conversation (Microsoft Tay bot).

Conversation framework of ChatBot acts in three stages:

Business Challenge for Building Chatbots

• A fixed set of answers
• Integration of ChatBot
• Understanding of Problem
• Security Issues
• Lack of Human Behaviour and Intentions
• Managing a ChatBot
• NLP limitations

Solution Offered for Implementing ChatBots with Deep Learning

Deep Learning which is galvanized by the functioning of the human brain, has composite engineering and used for the imitation of the data. Neural Network acts as the elementary brick of Deep Learning. A Neural Network is an Artificial Model of the human brain network modeled using hardware and software.

ChatBots Implementation Techniques

• Streaming of incoming data through the backend
• Create a model using Natural Language Processing
• Create a Natural Conversational Flow
• Add features to automate the process
• Invite Customers to Join

 

Source: XenonStack/Use-Cases

Predictive Maintenance Using Machine learning Techniques

Understanding Predictive Maintenance Applications

Predictive maintenance nowadays gaining popularity among enterprises that predict failure of the system, and the actions could include corrective actions, the replacement of the system, or even planned failure. This helps enterprises to cost savings, greater predictability, and the improved availability of the systems. Predictive maintenance sidesteps both the limits and maximizes the use of its sources. Predictive maintenance also detects the irregularities and failure patterns and provide real-time alerts. These signals can facilitate the efficient maintenance of those components. AI-enabled Predictive Maintenance is uncommon in that instead of just predicting impending failure, it also attempts to provide outcome-focused instructions for operations and maintenance from analytics. Let’s explore the areas where predictive maintenance can be used:

Predictive maintenance covers diverse application areas, such as -

• Manufacturing industry
• Information and technology
• Aerospace
• Heavy-Machinery sector
• Predicting the future performance of a subsystem or a component to make RUL (Remaining Useful Life) estimation.

In this use case, we will guide you through how to build a machine learning platform for predictive maintenance.

Business Challenge for Enabling Predictive Maintenance

• Monitoring of Assets in Real-Time via sensor data patterns to predict the breakdown of Assets.
• Production systems deteriorate with time and need maintenance.
• The regular way to keep the system good is to apply preventive maintenance practices, in the case of clearly detected malfunctions or equipment breakdowns. All this affects the quality, cost and in general, productivity.

Other than this, the uncertainty of machine reliability at any given time also impacts on product/production delivery times.

Predictive Maintenance Analytics Pipeline

Collecting targeted data

The targeted data reside in remote locations and get into the analysis pipeline including sensors, meters, supervisory control, etc. Collect data from all of the remote data sources to learn and continually make better, more informed business decisions.

Determining Analytics Pipeline

Establish an Advanced Analytics Pipeline based on the specific operation. Cloud analytics should be balanced to reduce the burden of streaming perishable PdM data on Cloud Deployment. Follow a distributed approach to detect and respond to local events at Cloud dataflow consumer step, take immediate action on Streaming data, while simultaneously integrating additional data sources in the Cloud.

Technology Stack -

• Python
• Flask
• Cloud IoT Core
• Cloud Pub/Sub

Source: XenonStack/Use-Cases

Advanced Threat Analytics and Intelligence

Overview of Advanced Threat Analytics and Intelligence

• Probe
• Denial of Service (DoS)
• User to Root
• Remote to User

Apache Metron Overview

Apache Spot Architecture for Cyber Security

Threat Detection Using Deep Learning

Threat Detection Using Machine Learning

Challenges to Real-Time Cyber Threat Intelligence

• To perform Real-Time Threat Intelligence on trillions of messages per year.
• Storing and Processing the unstructured security data.
• Combine Machine Learning and Predictive Analytics to perform Real-Time Threat Analytics.

Solution Offerings for Threat Detection and Cyber Security

• Network Dataset
• Pre-Processing of Data
• Feature Extraction
• Reduce Data Amount
• Improve Accuracy
• Avoid Overfitting

Training and Testing of Data Using Classification Models

• Decision Tree
• Random Forest
• Naive Bayes
• KNN
• Result Analysis

Sentiment Analytics Solutions with Deep Learning

What is SEMANTIC ANALYSIS?

Need for Semantic Analysis

• Getting the appropriate documents
• Listing them in the ranked order
• Giving the answer to the query if it is specific
  

Difference between Keyword-based Search and Semantic Search

Semantic Search based on Domain Ontology

What is Ontology?

How do we create Ontology?

Storing the Unstructured Text Data in RDF Form

Architecture for Real-Time Semantic Search Engine

• Crawl the documents (DOC, PDF, XML, HTML etc) from various universities and classify faculty profiles
• Convert the unstructured text present in various formats to structured RDF form as described in earlier sections.
• Build Ontology for Computer Science Domain
• Store the data in Apache Jena triple store (Both Ontology and RDF's)
• Use SparQL query language on the data

Data Ingestion using our Web Crawler Service

Introduction to Time Series Data

• Stock Series - It is a measure of attributes at a particular point in time and taken as a stock takes.
• Flow Series - It is a measure of activity at a specific interval of time. It contains effects related to the calendar.

Components of Time Series Data

• Trend
• Cyclical
• Seasonal
• Irregular

1. Deterministic - In this case, the effects of the shocks present in the time series are eliminated i.e. revert to the trend in long run.
2. Stochastic - It is the process in which the effects of shocks are never eliminated as they have permanently changed the level of the time series.

• Climate
• Institutions
• Social habits and practices
• Calendar

• Additive Model - It is the model in which the seasonal component is added with the trend component.
• Multiplicative Model - In this model seasonal component is multiplied with the intercept if trend component is not present in the time series. But, if time series have trend component, sum of intercept and trend is multiplied with the seasonal component.

Difference between Time Series Data and Cross-Section Data

What is Time Series Analysis?

• In the time domain
• In the frequency domain

• Decomposing the time series
• Identifying and modeling the time-based dependencies
• Forecasting
• Identifying and model the system variation

Need of Time Series Analysis

• Descriptive - In this case, patterns are identified in correlated data. In other words, the variations in trends and seasonality in the time series are identified.
• Explanation - In this understanding and modeling of data is performed.
• Forecasting - Here, the prediction from previous observations is performed for short term trends.
• Invention Analysis - In this case, effect performed by any event in time series data is analyzed.
• Quality Control - When the specific size deviates it provides an alert.

Applications of Time Series Analysis

• CrateDB
• Graphite
• InfluxDB
• Informix TimeSeries
• Kx kdb+
• Riak-TS
• RRDtool
• OpenTSDB

What is Anomaly?

Different Types of Anomalies

• Point Anomalies - If the specific value within the dataset is anomalous with respect to the complete data then it is known as Point Anomalies. The above mentioned example of bank transaction is an example of point anomalies.
• Contextual Anomalies - If the occurrence of data is anomalous for specific circumstances, then it is known as Contextual Anomalies. For example, the anomaly occurs at a specific interval of period.
• Collective Anomalies - If the collection of occurrence of data is anomalous with respect to the rest of dataset then it is known as Collective Anomalies. For example, breaking the trend observed in ECG.

Introduction

• Data Preprocessing
• Data Wrangling

Data Preprocessing

Data Wrangling

Need of Data Preprocessing

Need of Data Wrangling

Why is Data Preprocessing used?

• Inaccurate data (missing data) - There are many reasons for missing data such as data is not continuously collected, a mistake in data entry, technical problems with biometrics and much more.

• The presence of noisy data (erroneous data and outliers) - The reasons for the presence of noisy data could be a technological problem of gadget that gathers data, a human mistake during data entry and much more.

• Inconsistent data - The presence of inconsistencies is due to the reasons such that existence of duplication within data, human data entry, containing mistakes in codes or names i.e. violation of data constraints and much more.

Why is Data Wrangling used?

What is Data Leakage in Machine Learning/Deep Learning?

• Leakage of data from test dataset to training dataset.
• Leakage of computed correct prediction to the training dataset.
• Leakage of future data into the past data.
• Usage of data outside the scope of applied algorithm

How to check the presence of Data Leakage within the applied model?

• At the time of dividing time series dataset into training and test, the dataset is a complex problem.
• Implementation of sampling in a graphical problem is a complex task.
• Storage of analog observations in the form of audios and images in separate files having a defined size and timestamp.

How is Data Preprocessing performed?

• Ignoring the missing record - It is the simplest and effective method for handling the missing data. But, this method should not be performed at the time when the number of missing values are huge or when the pattern of data is related to the unrecognized basic root of the cause of statement problem.

• Filling the missing values manually - This is one of the best-chosen methods. But there is one limitation that when there is large dataset and missing values are large then, this method is not efficient as it becomes a time-consuming task.

• Filling using computed values - The missing values can also be filled by computing mean, mode or median of the observed given values. Another method could be the predictive values that are computed by using any Machine Learning or Deep Learning algorithm. But one drawback of this method is that it can generate bias within the data as the computed values are not accurate with respect to the observed values.

• Binning method - In this method sorting of data is performed with respect to the values of the neighborhood. This method is also known as local smoothing.
• Clustering method - In the approach, the outliers may be detected by grouping the similar data in the same group i.e. in the same cluster.
• Machine Learning - A Machine Learning algorithm can be executed for smoothing of data. For example, regression algorithm can be used for smoothing of data using a specified linear function.
• Removing manually - The noisy data can be removed manually by the human being but it is a time-consuming process so mostly this method is not given priority.

How is Data Wrangling performed?

Difference Between Data Preprocessing and Data Wrangling

Tasks of Data Preprocessing

• Data Cleaning - This is the first step which is implemented in Data Preprocessing. In this step, the main focus is on handling missing data, noisy data, detection, and removal of outliers, minimizing duplication and computed biases within the data.

• Data Integration - This process is used when data is gathered from various data sources and data is combined together to form consistent data. This consistent data after performing data cleaning is used for analysis.

• Data Transformation - This step is used to convert the raw data into a specified format according to the need of the model. The options used for transformation of data are given below:
  • Normalization - In this method, numerical data is converted into specified range i.e. between 0 and 1 so that scaling of data can be performed.
  • Aggregation - The concept can be derived from the word itself, this method is used to combine the features into one. For example combining two categories can be used to form a new category.
  • Generalization - In this case, lower level attributes are converted into a higher level.

• Data Reduction - After the transformation and scaling of data duplication i.e. redundancy within the data is removed and organize the data in an efficient manner.

Tasks of Data Wrangling

• Discovering - Firstly, data should be understood thoroughly and examine which approach will best suit. For example: if have a weather data when we examine the data it is observed that data is from one area and so main focus is on determining patterns.

• Structuring - As the data is gathered from different sources, the data will be present in different shapes and sizes. Therefore, there is a need of structuring the data in proper format.

• Cleaning - Cleaning or removing of data should be performed that can degrade the performance of analysis.

• Enrichment - Extract new features or data from the given dataset in order to optimize the performance of the applied model.

• Validating - This approach is used for improving the quality of data and consistency rules so that transformations that are applied to the data could be verified.

• Publishing - After completing the steps of Data Wrangling, the steps can be documented so that similar steps can be performed for similar kind of data to save time.