Overview

Prerequisites for Rocket Chat Deployment on Kubernetes

• Kubernetes
• Shared Persistent Storage - Option are GlusterFS, Ceph FS, AWS EBS, AzureDisk etc.
• Rocket Chat Application
• Dockerfile
• Container-Registry
• Kubectl

Step 1 - Create a Rocket Chat Container Custom Image

Step 2 - Build Rocket Chat Docker Custom Image

Step 3 - Create a MongoDB Container Custom Image

Step 4 - Building a MongoDB Docker Custom Image

Step 5 - Adding Container Registry to Docker Daemon

Step 6 - Pushing Custom PostgreSQL Container Image to Container Registry

Step 7 - Create a Storage Volume (Using GlusterFS)

Step 8 - Deploy MongoDB on Kubernetes

• Docker Image: We have created a Docker Image for MongoDB in Step 4 and pushed to docker hub or private docker registry.

Introduction

Workspace

$GOPATH

• All the import declaration in GO code references to a package through import path, and directories inside $GOPATH/src where go tool can compute all the imported packages.
• 
  To store the dependencies retrieved by go get.

Simplicity of GO GET

GO GET Flags

Shortcomings in GO GET

• We won't be able to determine which version of package we need unless it is hosted in complete different repository as go get always fetches from the latest version of a package. This leads to problem when working as team we might end up fetching different version of a package.
• 
  And since go get installs package at $GOPATH/src directory so we will have to use the same version of a package for different projects as all our projects are under a single directory. So each of the projects will not have different versions of dependencies with them.

Package Management Using Glide

What is Glide?

Installing Glide

• At first, we will install glide, using shell script curl https://glide.sh/get | sh

Introduction

What are Microservices?

Benefits of Microservices

Deployed Independently

Parallelize Development

Need of Testing in Microservices

Challenges of Microservices

Testing Strategies

Unit Testing

• Sociable Unit Testing
• Solitary Unit Testing

What is SEMANTIC ANALYSIS?

Need for Semantic Analysis

• Getting the appropriate documents
• Listing them in the ranked order
• Giving the answer to the query if it is specific
  

Difference between Keyword-based Search and Semantic Search

Semantic Search based on Domain Ontology

What is Ontology?

How do we create Ontology?

Storing the Unstructured Text Data in RDF Form

Architecture for Real-Time Semantic Search Engine

• Crawl the documents (DOC, PDF, XML, HTML etc) from various universities and classify faculty profiles
• Convert the unstructured text present in various formats to structured RDF form as described in earlier sections.
• Build Ontology for Computer Science Domain
• Store the data in Apache Jena triple store (Both Ontology and RDF's)
• Use SparQL query language on the data

Data Ingestion using our Web Crawler Service

Introduction to Time Series Data

• Stock Series - It is a measure of attributes at a particular point in time and taken as a stock takes.
• Flow Series - It is a measure of activity at a specific interval of time. It contains effects related to the calendar.

Components of Time Series Data

• Trend
• Cyclical
• Seasonal
• Irregular

1. Deterministic - In this case, the effects of the shocks present in the time series are eliminated i.e. revert to the trend in long run.
2. Stochastic - It is the process in which the effects of shocks are never eliminated as they have permanently changed the level of the time series.

• Climate
• Institutions
• Social habits and practices
• Calendar

• Additive Model - It is the model in which the seasonal component is added with the trend component.
• Multiplicative Model - In this model seasonal component is multiplied with the intercept if trend component is not present in the time series. But, if time series have trend component, sum of intercept and trend is multiplied with the seasonal component.

Difference between Time Series Data and Cross-Section Data

What is Time Series Analysis?

• In the time domain
• In the frequency domain

• Decomposing the time series
• Identifying and modeling the time-based dependencies
• Forecasting
• Identifying and model the system variation

Need of Time Series Analysis

• Descriptive - In this case, patterns are identified in correlated data. In other words, the variations in trends and seasonality in the time series are identified.
• Explanation - In this understanding and modeling of data is performed.
• Forecasting - Here, the prediction from previous observations is performed for short term trends.
• Invention Analysis - In this case, effect performed by any event in time series data is analyzed.
• Quality Control - When the specific size deviates it provides an alert.

Applications of Time Series Analysis

• CrateDB
• Graphite
• InfluxDB
• Informix TimeSeries
• Kx kdb+
• Riak-TS
• RRDtool
• OpenTSDB

What is Anomaly?

Different Types of Anomalies

• Point Anomalies - If the specific value within the dataset is anomalous with respect to the complete data then it is known as Point Anomalies. The above mentioned example of bank transaction is an example of point anomalies.
• Contextual Anomalies - If the occurrence of data is anomalous for specific circumstances, then it is known as Contextual Anomalies. For example, the anomaly occurs at a specific interval of period.
• Collective Anomalies - If the collection of occurrence of data is anomalous with respect to the rest of dataset then it is known as Collective Anomalies. For example, breaking the trend observed in ECG.

• Real-time Streaming
• Batch Streaming

• Data ingestion From Logs
• Data Ingestion from IoT Devices
• Data Collection and Ingestion from RDBMS (e.g., MySQL)
• Data Collection and Ingestion from ZiP Files
• Data Collection and Ingestion from Text/CSV Files

 
Objectives for the Data Lake

• A Central Repository for Big Data Management
• Reduce costs by offloading analytical systems and archiving cold data
• Testing Setup for experimenting with new technologies and data
• Automation of Data pipelines
• MetaData Management and Catalog
• Tracking measurements with alerts on failure or violations
• Data Governance with clear distinction of roles and responsibilities
• Data Discovery, Prototyping, and experimentation
  
  

Apache NiFi

Benefits of Apache NiFi

• Real-time/Batch Streaming
• Support both Standalone and Cluster mode
• Extremely Scalable, extensible platform
• Visual Command and Control
• Better Error handling

Core Features of Apache NiFi

• Guaranteed Delivery - A core philosophy of NiFi has been that even at very high scale, guaranteed delivery is a must. It is achievable through efficient use of a purpose-built persistent write-ahead log and content repository.

• Data Buffering / Back Pressure AND Pressure Release - NiFi supports buffering of all queued data as well as the ability to provide back pressure as those lines reach specified limits or to an age of data as it reaches a specified age (its value has perished).

• Prioritized Queuing - NiFi allows the setting of one or more prioritization schemes for how data from a queue is retrieved. The default is oldest first, but it can be configured to pull newest first, largest first, or some other custom scheme.

• Flow Specific QoS - There are points of a data flow where the data is critical, and it is less intolerant. There are also times when it must be processed and delivered within seconds to be of any value. NiFi enables the fine-grained flow particular configuration of these concerns.

• Data Provenance - NiFi automatically records, indexes, and makes available provenance data as objects flow through the system even across fan-in, fan-out, transformations, and more. This information becomes extremely critical in supporting compliance, troubleshooting, optimization, and other scenarios.

• Recovery / Recording a rolling buffer of fine-grained history - NiFi’s content repository is designed to act as a rolling buffer of history. As Data ages off, it is removed from the content repository or as space is needed.

• Visual Command and Control - NiFi enables the visual establishment of data flows in real-time. And provide UI based approach to build different data flow.

• Flow Templates - It also allows us to create templates of frequently used data streams. It can also help in migrating the data flows from one machine to another.

• Security - NiFi supports Multi-tenant Authorization. The authority level of a given data flow applies to each component, allowing the admin user to have a fine grained level of access control. It means each NiFi cluster is capable of handling the requirements of one or more organizations.
  
  

• Parallel Stream to Multiple Destination - With NiFi we can move data to multiple destinations at one time. After processing the data stream, we can route the flow to the various destinations using NiFi’s processor. It can be helpful when we need to back our data on multiple destinations.

NiFi Clustering

Data Collection and Ingestion from Twitter using Apache NiFi to Build Data Lake

• Amazon S3
• Amazon Redshift
• Apache Kafka
• Apache Spark
• Apache Flink
• ElasticSearch
• Apache Hive

Fetching Tweets with NiFi’s Processor

• Twitter Endpoint - Here we can set the endpoint from which data should get pulled. Available parameters -

• Consumer Key - Consumer key provided by Twitter.
• Consumer Secret - Consumer Secret provided by Twitter.
• Access Token - Access Token provided by Twitter.
• Access Token Secret - Access Token Secret provided by Twitter.
• Languages - Languages for which tweets should fetch out.
• Terms to Filter - Hashtags for which tweets should fetch out.
• IDs to follow - Twitter user IDs that should be followed.

Data Integration Using Apache NiFi and Apache Kafka

Data Integration Using Apache NiFi to Amazon RedShift with Amazon Kinesis Firehose Stream

Data Integration Using Apache NiFi to Amazon S3

• Introduction to Twelve-Factor App
• Kubernetes Architecture
• Kubernetes Components
• Kubernetes Security
• Monitoring of Kubernetes
• Open Source Tools For Kubernetes
• Applications of Kubernetes
• What's New in Kubernetes 1.7

Introduction To The Twelve-Factor App For Microservices

• Can minimize time and cost for new developers joining the project.
• Offering maximum portability between execution environments.
• Are suitable for deployment on modern cloud platforms.
• Obviating the need for servers and systems administration.
• Minimize divergence between development and production.
• Enabling continuous deployment for maximum agility.
• And can scale up without significant changes to tooling and architecture.

The Twelve Factors

• A twelve factor app should have only one codebase per app, but there will be many deploys of the app. A deploy is a running instance of an app.
• A twelve-factor app is always tracked in a version control system. A copy of the revision tracking database is known as a code repository.

• A twelve factor app never relies on implicit existence of system wide packages. It declares all dependencies, completely and exactly.

• A twelve factor app should store config in the environment. An app’s config is everything that is likely to vary between deploys (staging, production, developer environments etc).

• A backing service is any service the app consumes over the network as part of its normal operation. The code for a twelve-factor app makes no distinction between local and third party services.
• A deploy of the twelve-factor app should be able to swap out a local MySQL database with one managed by a third party without any changes to the app’s code.

• The twelve-factor app uses strict separation between the build, release, and run stages. Transformation of code repo into an executable bundle is known as a build. Combination of build stage and current config makes release stage and the run stage runs the app in the execution environment.

• The app is executed in the execution environment as one or more processes. Twelve-factor processes are stateless and share nothing. Any data that needs to persist must be stored in a stateful backing service typically a database.

• The twelve-factor app is completely self-contained and does not rely on runtime injection of a webserver into the execution environment to create a web-facing service. The web app exports HTTP as a service by binding to a port, and listening to requests coming in on that port.

• Processes in the twelve-factor app will be scale out via a process model. Using this model, the developer can architect their app to handle diverse workloads by assigning each type of work to a process type.

• The twelve-factor app’s processes are disposable, meaning they can be started or stopped at a moment’s notice. This facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.

• We have to keep development, staging and production as similar as possible. There should be less time gap, personnel gap and tool gap in development and production. It will help in continuous deployment.

• Treat logs as event streams. A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream.

• Run admin or management tasks as one off processes. Admin tasks are such as running one time scripts, running database migrations, running a console to run arbitrary code.

Kubernetes Architecture 

• Master Node
• Worker Node

Kubernetes Components

Master Node

• Kube-apiserver
• Etcd
• controller-manager
• kube-scheduler

Kube API Server

Etcd

Kube Controller Manager

• Replication controller
• Endpoints controller
• Namespace controller
• Service accounts controller
• DaemonSet Controller
• Job Controller

Kube Scheduler

Worker Node

• Kubelet
• Kube-proxy
• Docker
• Rkt
• Supervisord
• fluentd

Kubelet

Kube Proxy

• It handles DNS entry for service and pods.
• It provides the hostname, IP address to pods.
• It also forwards traffic from Cluster/Service IP address to specified set of pods.
• Alter IPtables on all nodes so that different pods can talk to each other or outside world.

Docker

• Lightweight
• Open-Source
• Most Popular

RKT

• Open-Source
• Pod-native approach
• Pluggable execution environment

Supervisor

Fluentd

Terminology

• Nodes

• Containers

• Pods

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

• Deployment

• Create new resources
• Update existing resources

align="justify">apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
-  name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

• Service

align="justify">Example: - service.yml
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376