Chatbot Development and Platform with Machine Learning

Overview of Building ChatBots with Deep Learning

A ChatBot is an implementation of Conversational Interface Intelligently comprising of Machine Learning, Deep Learning as their backbone. ChatBots hold variety including be Textual, Voice and Image-based interactions.

The growth of chatbots has opened up new areas of customer engagement and new methods of fulfilling business in the form of conversational commerce. It is the most useful technology that businesses can rely on, possibly following the old models and producing apps and websites redundant. A chatbot is a computer program that copies human communications in its natural format including text or spoken language using AI procedures such as Natural Language Processing, image, video processing, and audio analysis. The most impressive characteristic of the bots is that they learn from past interactions and become intelligent and more intelligent over time. Chatbots works in two ways- rule-based and smart machine-based. Rule-based chatbots give predefined responses from a database, based on the keywords used for the research. However, smart machine based chatbots receive their capabilities from Artificial Intelligence and Cognitive Computing and adapt their operation based on customer interactions.

The chatbot can be of two types: Goal-oriented (such as Siri, Alexa, Cortana, etc.) and General Conversation (Microsoft Tay bot).

Conversation framework of ChatBot acts in three stages:

Business Challenge for Building Chatbots

• A fixed set of answers
• Integration of ChatBot
• Understanding of Problem
• Security Issues
• Lack of Human Behaviour and Intentions
• Managing a ChatBot
• NLP limitations

Solution Offered for Implementing ChatBots with Deep Learning

Deep Learning which is galvanized by the functioning of the human brain, has composite engineering and used for the imitation of the data. Neural Network acts as the elementary brick of Deep Learning. A Neural Network is an Artificial Model of the human brain network modeled using hardware and software.

ChatBots Implementation Techniques

• Streaming of incoming data through the backend
• Create a model using Natural Language Processing
• Create a Natural Conversational Flow
• Add features to automate the process
• Invite Customers to Join

 

Source: XenonStack/Use-Cases

Advanced Threat Analytics and Intelligence

Overview of Advanced Threat Analytics and Intelligence

• Probe
• Denial of Service (DoS)
• User to Root
• Remote to User

Apache Metron Overview

Apache Spot Architecture for Cyber Security

Threat Detection Using Deep Learning

Threat Detection Using Machine Learning

Challenges to Real-Time Cyber Threat Intelligence

• To perform Real-Time Threat Intelligence on trillions of messages per year.
• Storing and Processing the unstructured security data.
• Combine Machine Learning and Predictive Analytics to perform Real-Time Threat Analytics.

Solution Offerings for Threat Detection and Cyber Security

• Network Dataset
• Pre-Processing of Data
• Feature Extraction
• Reduce Data Amount
• Improve Accuracy
• Avoid Overfitting

Training and Testing of Data Using Classification Models

• Decision Tree
• Random Forest
• Naive Bayes
• KNN
• Result Analysis

Applications of Natural Language Processing (NLP)

What is Natural Language Processing (NLP)

Natural Language Processing techniques, a subset of Artificial Intelligence increasing its necessity with the improvement of its sub-technologies day by day. Language is the prime source of communication and interactions. Without language, communication is not possible, and without communication, process completion is not possible. This is also another reason for the increment in the involvement of Natural Language Processing in different domains. The involvement of Natural Language Processing (NLP) is also increasing the dimensions of different areas. Some of these domains are -

These domains can be considered as use cases of Natural Language Processing but also have their separate use cases. The implementation of these use cases can be generalized to an extent, But these domains also demand diversities in the different levels of implementation as well as in the expertise required to implement them. Let’s understand briefly about these domains from the mirror of Natural Language Processing. One point is to be noted here that Natural Language Processing also requires some integration with other technologies such as Machine Learning, Deep Learning, and Big Data Analytics.

Natural Language Processing (NLP) Applications

Business Applications for Natural Language Processing

Let’s start with the domain of commercialization. It is pretty evident that the business domain is consists of some interesting and necessary use cases and problems that can be addressed by the use of Natural Language Processing. Some use cases of Natural Language Processing which are used in the Business domain are -

Sentiment Analysis — It is widely used in social media analytics and web monitoring which allows knowing the insights of the customers concerning particular products or services. It can be advantageous for any company to know about the thinking of the customers about a product so that they can know about the scope of improvement and how to achieve robustness. Natural language processing can not solely handle this task; it requires integration with highly computational methods such as Machine learning and deep learning to do the back end computation and Big data analytics to digest the data at an enormous scale.

Email Filters — Emails are adopted as a medium of communication officially now. Even the government considers it is official to communicate with the help of Email. But this medium is also vulnerable to spamming of the content. Companies that provide Email domains such as Google, Zoho or Yahoo are researching in the field of making it Full-proof by using different measures. Email filtering is an everyday use case of Natural Language Processing by applying various text analytics measures. It is a task of spam detection which is also in sentiment analysis as a pre-processing technique.

Voice Recognition — These are techniques that are powered by Natural Language Processing that allow companies to develop smart voice-driven services and interfaces for any product and service. To narrow the communication gap between the machines and humans is the most critical and necessary step to increase the grip on Artificial Intelligence. It can be achieved by only and only Voice Recognition which is possible by Natural Language Understanding a sub-process of Natural Language Processing.

Information Extraction — Information is the new fuel, it is a well-known fact now. But the data which is received at any receiving end mostly consist of unstructured format. The emergence of the advanced statistical algorithms results in the rise of predictive analytics and prescriptive analytics which made the prediction system more accurate. But these algorithms demand more and more information for finding the patterns and Matching them. Of course, Machine learning and Deep learning methods are doing an impressively great job, but without Natural Language Processing these things are not possible.

Continue Reading: XenonStack/Blogs

DevOps for Machine Learning, Tensor Flow and PyTorch

Why Continuous Integration and Deployment?

Like in Modern Web Applications, it needs Agile systems because of the ever-changing requirements of the clients and consumers. In Machine Learning the challenge is to make the system that works well with the real world, and the real-world scenarios change continuously. The system needs continuous learning and training from the real world. The solution is DevOps for Machine learning and deep learning. Which continuously trains the model on the new data after some time and then validates and tests the model accuracy to make sure it will work well with the current real-world scenarios.

DevOps for Machine Learning, Tensor Flow and PyTorch

TensorFlow and PyTorch are open source tools that help for DevOps for machine learning. Google develops TensorFlow and based on Theano Whereas PyTorch is developed by Facebook and based on Torch. Both frameworks define computational graphs. In TensorFlow, it needs to determine the entire computational graph and then run the ML algorithms. PyTorch uses dynamic graphs and creates it on the go.

TensorFlow has Tensorboard for visualization and enables directly on the browser. PyTorch doesn’t have a tool like that, but Matplotlib can be used with it. TensorFlow has more community support and online solutions than PyTorch.Whichever framework is used to build the Machine Learning Model, the CI/CD is a much-needed thing. Developers or Data Scientists spend most of their time in managing and Deploying their model to production manually, and this makes a lot of human errors. This needs to be an automated process with a well-defined pipeline and Model Versioning.

The skills needed by a data scientist is changing now, its less visualization and statistics-based and moving closer to engineering. Continuous Integration and Deployment of Machine Learning Models is the real challenge in the Data Science world, Productionizing the models requires proper Integration and Deployment pipeline. As the real world changes continuously so the system should have the capability to learn with time. Continuous Integration and Deployment Pipeline make this happen.

Currently, if a modern application needed to be developed a continuous pipeline then tools like Git, Bitbucket, GitCI, Jenkins were used for versioning and management of the code. As in the case of Modern application, only the codebase is required to be managed and versioned but in Machine Learning and AI Applications things are more iterative and complex. The data is another thing to accomplish here. A system is required which can version and manage the data, models and intermediate data.

Continuous Development life cycle

Git is a source code management and also a version control management tool. Versioning the code is much more critical for product releases, and there is history for every file for exploring the changes and reviewing the code.

Git for versioning and managing code

In Machine learning and AI systems, the model code needs to manage for releases and changes tracking. Git is the most used source code management tool used. In Continuous Integration and Continuous Deployment, Git manages the versions by tagging branches, and git-flow can be used for feature branches.

Dvc for versioning models and data

Unlike source code, the size of the model and data is much larger, and Git is not suitable for this kind of cases where the data is, and models files are large. Dvc is a Data science version control system that provides the end to end support for managing the Training data, intermediate data and model data.

Version control

DVC provides the commands like Git to add commit and push models and data to S3, Azure, GCP, Minio, SSH. It also includes data provenance for tracking the evolution of Machine Learning Models. Dvc helps in reproducibility if need to get back to a particular experiment.

Experiment management

Metric tracking is easy to use using DVC. It provides a Metric tracking feature that lists all the branches along with metrics values and picks the best version of the experiment.

Deployment and Collaboration

Dvc push-pull command is available to push the changes to production or staging. It also has a built-in way to create DAG using ML steps. DVC run command is used to create the deployment pipeline. It streamlines the work into a single, reproducible environment and also makes it easy to share the environment.

Packaging Models

There are a vast number of ways with models that can be packaged but the most convenient and automated using Docker on Kubernetes. Docker is not only applied to packaging but also as a development environment. It also handles version dependency management. It provides more reliability than running Flask on a Virtual Machine.

In this approach, Nginx, Gunicorn and Docker Compose will be used to create a scalable, repeatable template for making it easy to run with continuous integration and deployment.

Directory Structure

├── README.md
├── nginx/
├ ├── Dockerfile
├ └── nginx.conf
├── api/
├ ├── Dockerfile
├ ├── app.py
├ ├── __init__.py
├ └── models/
├── docker-compose.yml
└── run_docker.sh

How to Perform Continuous Model Testing for PyTorch and TensorFlow?

Feature Test

• value of features lies between the threshold values
• feature importance changed concerning previous
• Feature a relationship with the outcome variable in terms of correlation coefficients.
• Feature unsuitability by testing RAM usage, inference latency, etc.
• generated feature violates the data compliance-related issues
• code coverage of the code generating functions
• static code analysis outcome of code generating features

Continue Reading: XenonStack/Blogs

Sentiment Analytics Solutions with Deep Learning

Introduction to Time Series Data

• Stock Series - It is a measure of attributes at a particular point in time and taken as a stock takes.
• Flow Series - It is a measure of activity at a specific interval of time. It contains effects related to the calendar.

Components of Time Series Data

• Trend
• Cyclical
• Seasonal
• Irregular

1. Deterministic - In this case, the effects of the shocks present in the time series are eliminated i.e. revert to the trend in long run.
2. Stochastic - It is the process in which the effects of shocks are never eliminated as they have permanently changed the level of the time series.

• Climate
• Institutions
• Social habits and practices
• Calendar

• Additive Model - It is the model in which the seasonal component is added with the trend component.
• Multiplicative Model - In this model seasonal component is multiplied with the intercept if trend component is not present in the time series. But, if time series have trend component, sum of intercept and trend is multiplied with the seasonal component.

Difference between Time Series Data and Cross-Section Data

What is Time Series Analysis?

• In the time domain
• In the frequency domain

• Decomposing the time series
• Identifying and modeling the time-based dependencies
• Forecasting
• Identifying and model the system variation

Need of Time Series Analysis

• Descriptive - In this case, patterns are identified in correlated data. In other words, the variations in trends and seasonality in the time series are identified.
• Explanation - In this understanding and modeling of data is performed.
• Forecasting - Here, the prediction from previous observations is performed for short term trends.
• Invention Analysis - In this case, effect performed by any event in time series data is analyzed.
• Quality Control - When the specific size deviates it provides an alert.

Applications of Time Series Analysis

• CrateDB
• Graphite
• InfluxDB
• Informix TimeSeries
• Kx kdb+
• Riak-TS
• RRDtool
• OpenTSDB

What is Anomaly?

Different Types of Anomalies

• Point Anomalies - If the specific value within the dataset is anomalous with respect to the complete data then it is known as Point Anomalies. The above mentioned example of bank transaction is an example of point anomalies.
• Contextual Anomalies - If the occurrence of data is anomalous for specific circumstances, then it is known as Contextual Anomalies. For example, the anomaly occurs at a specific interval of period.
• Collective Anomalies - If the collection of occurrence of data is anomalous with respect to the rest of dataset then it is known as Collective Anomalies. For example, breaking the trend observed in ECG.

Introduction

• Data Preprocessing
• Data Wrangling

Data Preprocessing

Data Wrangling

Need of Data Preprocessing

Need of Data Wrangling

Why is Data Preprocessing used?

• Inaccurate data (missing data) - There are many reasons for missing data such as data is not continuously collected, a mistake in data entry, technical problems with biometrics and much more.

• The presence of noisy data (erroneous data and outliers) - The reasons for the presence of noisy data could be a technological problem of gadget that gathers data, a human mistake during data entry and much more.

• Inconsistent data - The presence of inconsistencies is due to the reasons such that existence of duplication within data, human data entry, containing mistakes in codes or names i.e. violation of data constraints and much more.

Why is Data Wrangling used?

What is Data Leakage in Machine Learning/Deep Learning?

• Leakage of data from test dataset to training dataset.
• Leakage of computed correct prediction to the training dataset.
• Leakage of future data into the past data.
• Usage of data outside the scope of applied algorithm

How to check the presence of Data Leakage within the applied model?

• At the time of dividing time series dataset into training and test, the dataset is a complex problem.
• Implementation of sampling in a graphical problem is a complex task.
• Storage of analog observations in the form of audios and images in separate files having a defined size and timestamp.

How is Data Preprocessing performed?

• Ignoring the missing record - It is the simplest and effective method for handling the missing data. But, this method should not be performed at the time when the number of missing values are huge or when the pattern of data is related to the unrecognized basic root of the cause of statement problem.

• Filling the missing values manually - This is one of the best-chosen methods. But there is one limitation that when there is large dataset and missing values are large then, this method is not efficient as it becomes a time-consuming task.

• Filling using computed values - The missing values can also be filled by computing mean, mode or median of the observed given values. Another method could be the predictive values that are computed by using any Machine Learning or Deep Learning algorithm. But one drawback of this method is that it can generate bias within the data as the computed values are not accurate with respect to the observed values.

• Binning method - In this method sorting of data is performed with respect to the values of the neighborhood. This method is also known as local smoothing.
• Clustering method - In the approach, the outliers may be detected by grouping the similar data in the same group i.e. in the same cluster.
• Machine Learning - A Machine Learning algorithm can be executed for smoothing of data. For example, regression algorithm can be used for smoothing of data using a specified linear function.
• Removing manually - The noisy data can be removed manually by the human being but it is a time-consuming process so mostly this method is not given priority.

How is Data Wrangling performed?

Difference Between Data Preprocessing and Data Wrangling

Tasks of Data Preprocessing

• Data Cleaning - This is the first step which is implemented in Data Preprocessing. In this step, the main focus is on handling missing data, noisy data, detection, and removal of outliers, minimizing duplication and computed biases within the data.

• Data Integration - This process is used when data is gathered from various data sources and data is combined together to form consistent data. This consistent data after performing data cleaning is used for analysis.

• Data Transformation - This step is used to convert the raw data into a specified format according to the need of the model. The options used for transformation of data are given below:
  • Normalization - In this method, numerical data is converted into specified range i.e. between 0 and 1 so that scaling of data can be performed.
  • Aggregation - The concept can be derived from the word itself, this method is used to combine the features into one. For example combining two categories can be used to form a new category.
  • Generalization - In this case, lower level attributes are converted into a higher level.

• Data Reduction - After the transformation and scaling of data duplication i.e. redundancy within the data is removed and organize the data in an efficient manner.

Tasks of Data Wrangling

• Discovering - Firstly, data should be understood thoroughly and examine which approach will best suit. For example: if have a weather data when we examine the data it is observed that data is from one area and so main focus is on determining patterns.

• Structuring - As the data is gathered from different sources, the data will be present in different shapes and sizes. Therefore, there is a need of structuring the data in proper format.

• Cleaning - Cleaning or removing of data should be performed that can degrade the performance of analysis.

• Enrichment - Extract new features or data from the given dataset in order to optimize the performance of the applied model.

• Validating - This approach is used for improving the quality of data and consistency rules so that transformations that are applied to the data could be verified.

• Publishing - After completing the steps of Data Wrangling, the steps can be documented so that similar steps can be performed for similar kind of data to save time.