Wednesday, 26 April 2017

streaming analytics 4/26/2017 10:43:00 am

Enabling Real Time Analytics For IoT

What is Fast Data?

A few years ago, we remember the time when it was just impossible to analyze petabytes of data. Then emergence of Hadoop made it possible to run analytical queries on our huge amount of historical data.

As we know Big Data is a buzz from last few years, but Modern Data Pipelines are constantly receiving data at a high ingestion rate. So this constant flow of data at high velocity is termed as Fast Data.

So Fast data is not about just volume of data like Data Warehouses in which data is measured in GigaBytes, TeraBytes or PetaBytes.

Instead, we measure volume but with respect to its incoming rate like MB per second, GB per hour, TB per day. So Volume and Velocity both are considered while talking about Fast Data.

What is Streaming and Real-Time Data

Nowadays, there are a lot of Data Processing platforms available to process data from our ingestion platforms. Some support streaming of data and other supports true streaming of data which is also called Real-Time data.

Streaming means when we are able to process the data at the instant as it arrives and then processing and analyzing it at ingestion time. But in streaming, we can consider some amount of delay in streaming data from ingestion layer.

But Real-time data needs to have tight deadlines in the terms of time. So we normally consider that if our platform is able to capture any event within 1 ms, then we call it as real-time data or true streaming.

But When we talk about taking business decisions, detecting frauds and analyzing real-time logs and predicting errors in real-time, all these scenarios comes to streaming. So Data received instantly as it arrives is termed as Real-time data.

Stream & Real Time Processing Frameworks

So in the market, there are a lot of open sources technologies available like Apache Kafka in which we can ingest data at millions of messages per sec. Also Analyzing Constant Streams of data is also made possible by Apache Spark Streaming, Apache Flink, Apache Storm.

Apache Spark Streaming is the tool in which we specify the time-based window to stream data from our message queue. So it does not process every message individually.

We can call it as the processing of real streams in micro batches.

Whereas Apache Storm and Flink have the ability to stream data in real-time.

Why Real-Time Streaming

As we know that Hadoop, S3 and other distributed file systems are supporting data processing in huge volumes and also we are able to query them using their different frameworks like Hive which uses MapReduce as their execution engine.

Why we Need Real-Time Streaming?

A lot of organizations are trying to collect as much data as they can regarding their products, services or even their organizational activities like tracking employees activities through various methods used like log tracking, taking screenshots at regular intervals.

So Data Engineering allows us to convert this data into structural formats and Data Analysts then turn this data into useful results which can help the organization to improve their customer experiences and also boost their employee's productivity.

But when we talk about log analytics, fraud detection or real-time analytics, this is not the way we want our data to be processed.The actual value data is in processing or acting upon it at the instant it receives.

Imagine we have a data warehouse like hive having petabytes of data in it. But it allows us to just analyze our historical data and predict future.

So processing of huge volumes of data is not enough. We need to process them in real-time so that any organization can take business decisions immediately whenever any important event occurs. This is required in Intelligence and surveillance systems, fraud detection etc.

Earlier handling of these constant streams of data at high ingestion rate is managed by firstly storing the data and then running analytics on it.

But organizations are looking for the platforms where they can look into business insights in real-time and act upon them in real-time.

Alerting platforms are also built on the top of these real-time streams. But Effectiveness of these platform lies in the fact that how truly we are processing the data in real-time.

Use Of Reactive Programming & Functional Programming

Now when we are thinking of building our alerting platforms, anomaly detection engines etc on the top of our real-time data, it is very important to consider the style of programming you are following.

Nowadays, Reactive Programming and Functional Programming are at their boom.

So, we can consider Reactive Programming as subscriber and publisher pattern. Often, we see the column on almost every website where we can subscribe to their newsletter and whenever the newsletter is posted by the publisher, whosoever have got subscription will get the newsletter via email or some other way.

So the difference between Reactive and Traditional Programming is that the data is available to the subscriber as soon as it receives. And it is made possible by using Reactive Programming model.

In Reactive Programming, whenever any events occur, there are certain components (classes) that had registered to that event. So instead of invoking target components by event generator, all targets automatically get triggered whenever any event occurs.

Now when we are processing data at high rate, concurrency is the point of concern. So the performance of our analytics job highly depends upon memory allocation/deallocation. So in Functional Programming, we don’t need to initialize loops/iterators on our own.

We will be using Functional Programming styles to iterate over the data in which CPU itself takes care of allocation and deallocation of data and also makes the best use of memory which results in better concurrency or parallelism.

Streaming Architecture Matters

While Streaming and Analyzing the real-time data, there are chances that some messages can be missed or in short, the problem is how we can handle data errors.

So, there are two types of architectures which are used while building real-time pipelines.

Lambda Architecture:

This architecture was introduced by Nathan Marz in which we have three layers to provide real-time streaming and compensate any data error occurs if any. The three layers are Batch Layer, Speed layer, and Serving Layer.

Continue Reading the full Article At - XenonStack.com/Blog

XenonStack 0

Wednesday, 22 March 2017

postgresql 3/22/2017 03:04:00 pm

How To Deploy PostgreSQL on Kubernetes

What is PostgreSQL?

PostgreSQL is a powerful, open source Relational Database Management System.

PostgreSQL is not controlled by any organization or any individual. Its source code is available free of charge. It is pronounced as "post-gress-Q-L".

PostgreSQL has earned a strong reputation for its reliability, data integrity, and correctness.

It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, SGI IRIX, MacOS, Solaris, Tru64), and Windows.

It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages)

It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP.

It also supports storage of binary large objects, including pictures, sounds, or video.

It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

Prerequisites

To follow this guide you need -

Kubernetes Cluster

GlusterFS Cluster

Step 1 - Create a PostgreSQL Container Image

Create a file name “Dockerfile” for PostgreSQL. This image contains our custom config dockerfile which will look like -

FROM ubuntu:latest
MAINTAINER XenonStack

RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8

RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" > /etc/apt/sources.list.d/pgdg.list

RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6

RUN /etc/init.d/postgresql start &&\
 psql --command "CREATE USER root WITH SUPERUSER PASSWORD 'xenonstack';" &&\
 createdb -O root xenonstack

RUN echo "host all  all 0.0.0.0/0  md5" >> /etc/postgresql/9.6/main/pg_hba.conf

RUN echo "listen_addresses='*'" >> /etc/postgresql/9.6/main/postgresql.conf

# Expose the PostgreSQL port
EXPOSE 5432

# Add VOLUMEs to allow backup of databases
VOLUME  ["/var/lib/postgresql"]

# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.6/bin/postgres", "-D", "/var/lib/postgresql", "-c", "config_file=/etc/postgresql/9.6/main/postgresql.conf"]

This Postgres image has a base image of ubuntu xenial. After that, we create Super User and default databases. Exposing 5432 port will help external system to connect the PostgreSQL server.

Step 2 - Build PostgreSQL Docker Image

$ docker build -t dr.xenonstack.com:5050/postgres:v9.6

Step 3 - Create a Storage Volume (Using GlusterFS)

Using below-mentioned command create a volume in GlusterFS for PostgreSQL and start it.

As we don’t want to lose our PostgreSQL Database data just because a Gluster server dies in the cluster, so we put replica 2 or more for higher availability of data.

$ gluster volume create postgres-disk replica 2 transport tcp k8-master:/mnt/brick1/postgres-disk  k8-1:/mnt/brick1/postgres-disk
$ gluster volume start postgres-disk
$ gluster volume info postgres-disk

Step 4 - Deploy PostgreSQL on Kubernetes

Deploying PostgreSQL on Kubernetes have following prerequisites -

Docker Image: We have created a Docker Image for Postgres in Step 2

Persistent Shared Storage Volume: We have created a Persistent Shared Storage Volume in Step 3

Deployment & Service Files: Next, we will create Deployment & Service Files

Create a file name “deployment.yml” for PostgreSQL. This deployment file will look like -

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
  namespace: production
spec:
  replicas: 1
  template:
 metadata:
   labels:
    k8s-app: postgres
 spec:
   containers:
   - name: postgres
     image: dr.xenonstack.com:5050/postgres:v9.6
     imagePullPolicy: "IfNotPresent"
     ports:
     - containerPort: 5432
     env:
     - name: POSTGRES_USER
       value: postgres
     - name: POSTGRES_PASSWORD
       value: superpostgres
     - name: PGDATA
       value: /var/lib/postgresql/data/pgdata
     volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgredb
   volumes:
     - name: postgredb
       glusterfs:
         endpoints: glusterfs-cluster
         path: postgres-disk
         readOnly: false

Continue Reading The Full Article At - XenonStack.com/Blog

XenonStack 0

Tuesday, 14 March 2017

iot 3/14/2017 12:48:00 pm

Why We Need Modern Big Data Integration Platform

Data is everywhere and we are generating data from different Sources like Social Media, Sensors, API’s, Databases.

Healthcare, Insurance, Finance, Banking, Energy, Telecom, Manufacturing, Retail, IoT, M2M are the leading domains/areas for Data Generation. The Government is using BigData to improve their efficiency and distribution of the services to the people.

The Biggest Challenge for the Enterprises is to create the Business Value from the data coming from the existing system and from new sources. Enterprises are looking for a Modern Data Integration platform for Aggregation, Migration, Broadcast, Correlation, Data Management, and Security.

Traditional ETL is having a paradigm shift for Business Agility and need of Modern Data Integration Platform is arising. Enterprises need Modern Data Integration for agility and for an end to end operations and decision-making which involves Data Integration from different sources, Processing Batch Streaming Real Time with BigData Management, BigData Governance, and Security.

BigData Type Includes:

What type of data it is

Format of content of data required

Whether data is transactional data, historical data or master data

The Speed or Frequency at which data made to be available

How to process the data i.e. whether in real time or in batch mode

5 V’s to Define BigData

Additional 5V’s to Define BigData

Data Ingestion and Data Transformation

Data Ingestion comprises of integrating Structured/unstructured data from where it is originated into a system, where it can be stored and analyzed for making business decisions. Data Ingestion may be continuous or asynchronous, real-time or batched or both.

Defining the BigData Characteristics: Using Different BigData types, helps us to define the BigData Characteristics i.e how the BigData is Collected, Processed, Analyzed and how we deploy that data On-Premises or Public or Hybrid Cloud.

Data type: Type of data

Transactional

Historical

Master Data and others

Data Content Format: Format of data

Structured (RDBMS)

Unstructured (audio, video, and images)

Semi-Structured

Data Sizes: Data size like Small, Medium, Large and Extra Large which means we can receive data having sizes in Bytes, KBs, MBs or even in GBs.

Data Throughput and Latency: How much data is expected and at what frequency does it arrive. Data throughput and latency depend on data sources:

On demand, as with Social Media Data

Continuous feed, Real-Time (Weather Data, Transactional Data)

Time series (Time-Based Data)

Processing Methodology: The type of technique to be applied for processing data (e.g. Predictive Analytics, Ad-Hoc Query and Reporting).

Data Sources: Data generated Sources

The Web and Social Media

Machine-Generated

Human-Generated etc

Data Consumers: A list of all possible consumers of the processed data:

Business processes

Business users

Enterprise applications

Individual people in various business roles

Part of the process flows

Other data repositories or enterprise applications

Major Industries Impacted with BigData

What is Data Integration?

Data Integration is the process of Data Ingestion - integrating data from different sources i.e. RDBMS, Social Media, Sensors, M2M etc, then using Data Mapping, Schema Definition, Data transformation to build a Data platform for analytics and further Reporting. You need to deliver the right data in the right format at the right timeframe.

BigData integration provides a unified view of data for Business Agility and Decision Making and it involves:

Discovering the Data

Profiling the Data

Understanding the Data

Improving the Data

Transforming the Data

A Data Integration project usually involves the following steps:

Ingest Data from different sources where data resides in multiple formats.

Transform Data means converting data into a single format so that one can easily be able to manage his problem with that unified data records. Data Pipeline is the main component used for Integration or Transformation.

MetaData Management: Centralized Data Collection.

Store Transform Data so that analyst can exactly get when the business needs it, whether it is in batch or real time.

Why Data Integration is required

Make Data Records Centralized: As data is stored in different formats like in Tabular, Graphical, Hierarchical, Structured, Unstructured form. For making the business decision, a user has to go through all these formats before reaching a conclusion. That’s why a single image is the combination of different format helpful in better decision making.

Format Selecting Freedom: Every user has different way or style to solve a problem. User are flexible to use data in whatever system and in whatever format they feel better.

Reduce Data Complexity: When data resides in different formats, so by increasing data size, complexity also increases that degrade decision making capability and one will consume much more time in understanding how one should proceed with data.

Prioritize the Data: When one have a single image of all the data records, then prioritizing the data what's very much useful and what's not required for business can easily find out.

Better Understanding of Information: A single image of data helps non-technical user also to understand how effectively one can utilize data records. While solving any problem one can win the game only if a non-technical person is able to understand what he is saying.

Keeping Information Up to Date: As data keeps on increasing on daily basis. So many new things come that become necessary to add on with existing data, so Data Integration makes easy to keep the information up to date.

Continue Reading The Full Article At - XenonStack.com/Blog

XenonStack 0

Thursday, 9 March 2017

healthcare technology 3/09/2017 11:23:00 am

Healthcare is Drowning in Data, Thirst For Knowledge

The Amount of Data in Healthcare is increasing at an astonishing rate. However, in general, the industry has not deployed the level of data management and analysis necessary to make use of those data.

As a result, healthcare executives face the risk of being overwhelmed by a flood of unusable data.

Consider the many sources of data. Current medical technology makes it possible to scan a single organ in 1 second and complete a full-body scan in roughly 60 seconds. The result is nearly 10 GB of raw image data delivered to a hospital’s Picture Archive and Communications System (PACS).

Clinical areas in their digital infancy, such as pathology, proteomics, and genomics, which are the key to personalized medicine, can generate over 2TB of data per patient.

Add to that the research and development of advanced medical compounds and devices, which generate terabytes over their lengthy development, testing and approval processes.

Doctors Are Drowning In Data

Technology isn't enough to improve healthcare. Doctors must be able to distinguish between valuable data and information overload.

One of the hopes of Electronic Health Records (EHRs) is that they will revolutionize medicine by collecting information that can be used to improve how we provide care. Getting good data from EHRs can occur if good data is input.

This doesn't always happen. To see patients, document encounters, enter smoking status, create coded problems lists, update medication lists, e-prescribe medications, order tests, find, open, and review multiple prior notes, schedule follow-up appointments, search for SNOWMED codes, search for ICD-9 codes, and find CPT codes to bill encounters(tasks previously delegated to a number of people) and compassionately interact with patients, providers have to take shortcuts.

But We have to Say HealthCare Drowning in Data Elements not yet interoperable onto one Platform.

First, the Data Exchange and Interoperability between EMRs, HIEs, Hospitals, Nursing Homes, Home care, ERs, portals, etc., must be addressed and industry standards need to emerge on the technology, but also the costs need to be defined. Who is going to pay for what and when?

It seems like the deepest pockets in the industry – pharmaceuticals and insurance – have put a dime into technology solutions or Big Data. Yet they have the most to gain. This is a huge disconnect because physicians and hospitals cannot afford to capitalize this start up by ourselves.

I believe that they will need to be influenced to contribute to this effort, in kind or with cash, for this system to be made whole and meaningful.

HIT industry leaders need to sit down with busy clinicians to create a workflow of automated Big Data in a way that provides all the stakeholders with the data to improve all levels of efficiencies and outcomes.

Decisions Through Data-Small data, Predictive modeling expansion, and real-time analytics are three forms of data analytics.

Healthcare data will continue to accumulate rapidly. If practices, hospitals, and healthcare systems do not actively respond to the flood of unstructured data, they risk forgoing the opportunity to use these data in managing their operations.

Small data and Real-Time Analytics are two methods of data analytics that allow practices, hospitals, and healthcare organizations to extract meaningful information.

Predictive Modeling is best suited for organizations managing large patient populations. With all three methods, the applicable information mined from raw data supports improvements in the quality of care and cost efficiency.

The use of Small Data, Real-Time Analytics, and Predictive Modeling will revolutionize the healthcare field by increasing those opportunities beyond reacting to emerging problems.

About RayCare:

RayCare is an Integrated HealthCare Platform Starting From Connecting Doctors, Labs, Medicine, Dieticians and Get Healthy Life Tips to Creation of Health Profile, Medical Reports, Daily Health Tracking to Predictive Diagnostic Analytics and Second Option Consultation & Recommendations. Know More!

For More, Visit - XenonStack.com/Blog

XenonStack 0

Thursday, 16 February 2017

kubernetes 2/16/2017 11:21:00 am

BlockChain App Deployment Using Microservices With Kubernetes

What is a BlockChain?

BlockChain is a distributed database that maintains a continuously-growing list of ordered records called blocks. This technology underlying Bitcoin and other cryptocurrencies.

It is a public ledger of all Bitcoin transaction. These blocks are added in a chronological order. In order to deploy a blockchain application, you need a distributed Hyperledger blockchain on your choice of infrastructure (on-premise or cloud).

In this article, we will deploy a Hyperledger Fabric cluster using Kubernetes.

Prerequisites

To follow this guide you need a system with working kubernetes cluster on it. We will use Fabric which is an implementation of blockchain technology written in Golang so go version go1.6.2 or above is required.

Before proceeding further let’s have a look on Hyperledger Fabric.

The Hyperledger Project

Hyperledger is an open source project with collaborative effort created to advance blockchain technology.

It helps in cross-industry distributed ledgers which support transaction system, property transaction, and other services.

Hyperledger Fabric

The Fabric is an implementation of blockchain technology. It provides a modular architecture allowing pluggable implementations of the various function.

Setting Hyperledger Cluster on Kubernetes

Hyperledger Kubernetes Replication Controller

We will launch hyperledger on kubernetes as a Replication Controller it will ensure us the high - availability of hyperledger pods.

Create a file named membersrvc-rc.yml.

 
apiVersion: v1
kind: ReplicationController
metadata:
  creationTimestamp: null
  labels:
    service: membersrvc
  name: membersrvc
  namespace: default
spec:
  replicas: 1
  selector:
    service: membersrvc
  template:
    metadata:
      creationTimestamp: null
      labels:
        service: membersrvc
    spec:
      containers:
      - command:
        - membersrvc
        image: hyperledger/fabric-membersrvc
        imagePullPolicy: ""
        name: membersrvc
        ports:
        - containerPort: 7054
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
      volumes: null
status:
  replicas: 0

In the same way, create another file vp0-rc.yml


apiVersion: v1
kind: ReplicationController
metadata:
  creationTimestamp: null
  labels:
    service: vp0
  name: vp0
  namespace: ${NAMESPACE}
spec:
  replicas: 1
  selector:
    service: vp0
  template:
    metadata:
      creationTimestamp: null
      labels:
        service: vp0
    spec:
      containers:
      - command:
        - sh
        - -c
        - sleep 5; peer node start --peer-chaincodedev
        env:
        - name: CORE_PEER_ADDRESSAUTODETECT
          value: "true"
        - name: CORE_VM_ENDPOINT
          value: unix:///var/run/docker.sock
        - name: CORE_LOGGING_LEVEL
          value: DEBUG
        - name: CORE_PEER_ID
          value: vp0
        - name: CORE_PEER_PKI_ECA_PADDR
          value: membersrvc:7054
        - name: CORE_PEER_PKI_TCA_PADDR
          value: membersrvc:7054
        - name: CORE_PEER_PKI_TLSCA_PADDR
          value: membersrvc:7054
        - name: CORE_SECURITY_ENABLED
          value: "false"
        - name: CORE_SECURITY_ENROLLID
          value: test_vp0
        - name: CORE_SECURITY_ENROLLSECRET
          value: MwYpmSRjupbT
        image: hyperledger/fabric-peer
        imagePullPolicy: ""
        name: vp0
        ports:
        - containerPort: 7050
        - containerPort: 7051
        - containerPort: 7053
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
      volumes: null
status:
  replicas: 0

That’s enough with replication controller. Now our next target is to deploy services for the Replication Controller.

Create a file called membersrvc-srv.yml


apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  name: membersrvc
  namespace: default
spec:
  ports:
  - name: ""
    nodePort: 0
    port: 7054
    protocol: ""
    targetPort: 0
  selector:
    service: membersrvc
status:
  loadBalancer: {}

Create another file vp0-srv.yml


apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  name: vp0
  namespace: default
spec:
  type: NodePort
  ports:
  - name: "port1"
    port: 7050
    protocol: ""
    targetPort: 0
  - name: "port2"
    nodePort: 0
    port: 7051
    protocol: ""
    targetPort: 0
  - name: "port3"
    nodePort: 0
    port: 7053
    protocol: ""
    targetPort: 0
  selector:
    service: vp0
status:
  loadBalancer: {}

Running Hyperledger Pods

After creating all the necessary file, next step is to start these rc pods

$ kubectl create -f membersrvc-rc.yml

$ kubectl create -f vp0-rc.yml

Continue Reading the Full Article At - XenonStack.com/Blog

XenonStack 0

Friday, 10 February 2017

serverless microservices 2/10/2017 03:45:00 pm

Building Serverless Microservices With Python

Serverless Computing is Exploding

As we move to the different models of production, distribution, and management when it comes to applications, it only makes sense that abstracting out the, behind the scenes processes should be handled by third parties, in a move towards further decentralization.

And that’s exactly what serverless computing does – and startups and big companies are adopting this new way of running applications.

In this post, we will discover answers to questions:

What Serverless is all about and how does this new trend affect the way people write and deploy applications?

Serverless Computing

"Serverless” denotes a special kind of software architecture in which application logic is executed in an environment without visible processes, operating systems, servers or virtual machines.

It’s worth mentioning that such an environment is actually running on the top of an operating system and use physical servers or virtual machines, but the responsibility for provisioning and managing the infrastructure entirely belongs to the service provider.

Therefore, a software developer focus more on writing code.

Serverless Computing Advances the way Applications are Developed

Serverless applications will change the way we develop applications. Traditionally a lot of business rules, boundary conditions, complex integrations are built into applications and this prolongs the completion of the system as well as introduces a lot of defects and in effect, we are hard wiring the system for certain set of functional requirements.

The serverless application concept moves us away from dealing with complex system requirements and evolves the application with time. It is also easy to deploy these microservices without intruding the system.

Below figure shows how the way of application development changed with time.

Monolith- A monolith application puts all its functionality into a single process and scale by replicating the monolith on multiple servers.

Microservice- A microservice architecture puts each functionality into a separate service and scale by distributing these services across servers, replicating as needed.

FaaS- Distributing Microservices further into functions which are triggered based on events.

Monolith => Microservice => FaaS

building serverless microservices with python

Let’s get started with the deployment of a Serverless Application on NexaStack.To create a function, you first package your code and dependencies in a deployment package.

Then, you upload the deployment package on our environment to create your function.

Creating a Deployment Package

Uploading a Deployment Package

You May also Like: Building Serverless Microservices With Java

Database Integration For Your Application

Install MongoDB and configure it to get started.

Create Database EmployeeDB

Create table Employee

Insert some records into the table for the demo.

Write a file “config.py” to setup configuration on the serverless architecture as shown below.

Continue Reading the full Article at: XenonStack.com/Blog

XenonStack 0

Monday, 6 February 2017

serverless microservices 2/06/2017 11:16:00 am

Building Serverless Microservices With JAVA

Serverless Architecture

The phrase “serverless” doesn’t mean servers are no longer required. It solely proposes that developers no longer have to think that much about them.

Going serverless lets developers shift their focus from the server level to the task level which is writing codes.

What it means to have servers?

First, let’s talk about what it means to have servers (virtual servers) providing the computing power required by your application. Owning servers comes with responsibilities -

Managing how the primitives (functions in the case of applications, or objects when it comes to storage) map to server primitives (CPU, memory, disk etc.).

Own provisioning (and therefore paying) for the capacity to handle your application’s projected traffic, independent of whether there’s actual traffic or not.

Own managing reliability and availability constructs like redundancy, failover, retries etc.

Advantages of going Serverless

Why should one move to serverless architecture can be adequately described through its benefits.

PaaS and Serverless - A user of traditional PaaS have to specify the amount of resources—such as dynos for Heroku or gears for OpenShift—for the application. The Serverless platform will take care of finding a server where the code is to run and to scale up when necessary.

Lower operational and development costs - The containers used to run these functions are decommissioned as soon as the execution ends. And the execution is metered in units of 100 ms, You don't pay anything when your code isn't running.

Fits with microservices, which can be implemented as functions.

Serverless architectures refer to applications that significantly depend on third-party services (knows as Backend as a Service or "BaaS") or on custom code that's run in ephemeral containers (Function as a Service or "FaaS").

But there are cons related to moving your application to FaaS which is discussed in our next post: Building Serverless Microservices with Python

Simplest way of thinking about FaaS is that it changes thinking from "build a framework to sit on a server to react to multiple events to "build/use micro-functionality to react to a single event."

How to migrate to a Microservices Architecture?

In a simple definition, Microservices are independently scalable, independently deployable systems that communicate over some protocols HTTP (XML, JSON), Thrift, Protocol Buffers etc.

Microservices are Single Responsibility Principle at code base level.

Below are some of the factors that can be followed to build Microservices:

One code per app/service: There is always a one-to-one correlation between the codebase and the service.

Explicitly declare and isolate dependencies: This can be done by using packaging systems.

Use environment variables to store configurations.

Strictly separate build, release and run stages.

Treat logs as event streams. Route log event stream to analysis system such as Splunk for log analysis.

Keep development, staging, and production as similar as possible.

Microservices Architecture: Benefits

Microservices Architectures have lots of very real and significant benefits:

Systems built in this way are inherently loosely coupled

The services themselves are very simple, focusing on doing one thing well

Multiple developers and teams can deliver independently under this model

They are a great enabler for continuous delivery, allowing frequent releases whilst keeping the rest of the system available and stable

In this post, we will implement a Nexastack function which integrates with a database(MongoDB used here).

We are going to implement this new function in Java using Spring Framework. So, Let’s get started -

Employee Service

We are going to build an Employee Service consisting of a function to show Employees information from the database.

For Demo purpose we are here implementing one function “GetEmployee”.

1. Setting up MongoDB Instance

Install MongoDB and configure it to get started.

Create Database EmployeeDB

Create table Employee

Insert some records into the table for demo.

Write a file “config.properties” to setup configuration on the serverless architecture

Continue Reading the full article at: XenonStack.com/Blog

XenonStack 0

Wednesday, 1 February 2017

2/01/2017 01:34:00 pm

BlockChain Apps Deployment Using Microservices With Dockers

BlockChain on Docker

What is a BlockChain?

Blockchain is a distributed database that maintains a continuously-growing list of ordered records called blocks. This technology underlying Bitcoin and other cryptocurrencies.

It is a public ledger of all Bitcoin transaction. These blocks are added in a chronological order.

In order to deploy a Blockchain applicatiIn order to deploy a Blockchain application, you need a distributed Hyperledger Blockchain on your choice of infrastructure (on-premise or cloud).

In this article we will deploy a Hyperledger Fabric cluster using Docker.

Prerequisites

To follow this guide you need a system with working Docker engine and docker-compose on it. We will use Fabric which is an implementation of Blockchain technology written in Golang, so go version go1.6.2 or above is required. Before proceeding further let’s have a look on Hyperledger Fabric.

The HyperLedger Project

Hyperledger is an open source project with collaborative effort created to advance Blockchain technology.

It helps in cross-industry distributed ledgers which support transaction system, property transaction, and other services.

HyperLedger Fabric

Fabric is an implementation of blockchain technology. It provides a modular architecture allowing pluggable implementations of the various function.

Setting HyperLedger Cluster

Pulling Images

First, pull the latest images published by the Hyperledger fabric project from DockerHub.

docker pull hyperledger/fabric-peer:latest

docker pull hyperledger/fabric-membersrvc:latest

Now in order to run these images. Create a docker-compose file which will launch both of these services.


membersrvc:
image: hyperledger/fabric-membersrvc
ports:
- "7054:7054"
command: membersrvc
vp0:
image: hyperledger/fabric-peer
ports:
- "7050:7050"
- "7051:7051"
- "7053:7053"
environment:
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_VM_ENDPOINT=unix:///var/run/docker.sock
- CORE_LOGGING_LEVEL=DEBUG
- CORE_PEER_ID=vp0
- CORE_PEER_PKI_ECA_PADDR=membersrvc:7054
- CORE_PEER_PKI_TCA_PADDR=membersrvc:7054
- CORE_PEER_PKI_TLSCA_PADDR=membersrvc:7054
- CORE_SECURITY_ENABLED=false
- CORE_SECURITY_ENROLLID=test_vp0
- CORE_SECURITY_ENROLLSECRET=MwYpmSRjupbT
links:
- membersrvc
command: sh -c "sleep 5; peer node start --peer-chaincodedev"

That’s it now we are ready to launch these service by simply running docker-compose up

Running the ChainCode

Before running chaincode you need to set your $GOPATH and then make a directory to download the sample chaincode in the src directory.


mkdir -p $GOPATH/src/github.com/chaincode_example02/
cd $GOPATH/src/github.com/chaincode_example02
Curl --request GET https://raw.githubusercontent.com/hyperledger/fabric/master/examples/chaincode/go/chaincode_example02/chaincode_example02.go > chaincode_example02.go

Next, you’ll need to download the Hyperledger fabric to your local $GOPATH, after that you have to build the chaincode.


mkdir -p $GOPATH/src/github.com/hyperledger
cd $GOPATH/src/github.com/hyperledger
git clone http://gerrit.hyperledger.org/r/fabric

Go to chaincode_example02 directory and build the code


cd $GOPATH/src/github.com/chaincode_example02
go build

Starting And Registering The ChainCode

Run the following command to start the chaincode.


CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=0.0.0.0:7051 ./chaincode_example02

After that chaincode console will display the message “Received REGISTERED, ready for invocations” which shows that chaincode is ready for use.

Running Rest API

To log in with the help of REST API, send a POST request to the /registrar endpoint, with the enrollment ID and enrollment PW. These parameters are listed in the eca.users section of the membersrvc.yaml file.

REST Request:


POST localhost:7050/registrar

{
"enrollId": "jim",
"enrollSecret": "6avZQLwcUe9b"
}

REST Response:

200 OK
{
"OK": "Login successful for user 'jim'."
}

Read The Full Article At: XenonStack.com/blog

XenonStack 0

Friday, 27 January 2017

1/27/2017 01:24:00 pm

Everything You Need to Know About NexaStack - DevOps & Serverless Computing Platform

What is NexaStack

NexaStack is a DevOps Platform to Automate | Monitor | Analyse your Application Delivery. NexaStack DevOps Platform Accelerates your Data Driven Application Delivery with proper visibility and security.

NexaStack was started as internal project of XenonStack in the year 2014, developed as DevOps Platform to serve the internal requirements of our team.

Why Should You Adopt DevOps Culture

DevOps is a process and balanced organisation approach to improve collaboration, communication among development and operation. Redesign and find new ways for faster and Reliable Delivery.

Why NexaStack

NexaStack For Developers
- Self Provision Resources:
  
  NexaStack empowers Developers to provision Infrastructure on their own and build environment to start development rapidly. It helps in breaking wall of confusion between Developers and Sysmins.
- Continuous Delivery:
  
  Leaner and flexible by providing fast release cycles with the help of Continuous Delivery. Deploy from SCM to Production within minutes with fully automated build pipeline.

NexaStack For Operations
- Multi-Cloud Support:
  
  Integration with multiple cloud platforms at one place by providing abstraction layer between Multi Cloud platform APIs.
- Runbook Automation:
  
  NexaStack's Auto-Config Management Engine helps reducing operations complexity by providing one place for all runbook execution with support to leading Configuration Management Softwares.
- Define Topologies:
  
  For a complex software running on multiple techonologies, NexaStack helps defining topologies for applications running on MultiTier as well Microservices.

Features Of NexaStack

Adaptive And Secure(DevOps Delivery Pipeline):

NexaStack DevOps Infrastructue is agile and flexible to support different workflows and modern data centers with technologies from different decades.

MicroServices (Serverless Computing):

Provision, Configure and Manage Infrastructure to enable collaboration and automation for more faster and reliable software delivery.

Integrations:

NexaStack Plug N Play Integrations support

Cloud Integration: Use NexaStack for 'On Premises' Infrastructure with support of Hybrid Cloud Infrastructure with Leading Public Cloud Providers like AWS, Google Cloud, Microsoft Azure, OpenStack and much more.

Database Integration: Supported Runbooks to provision Databases in your technology Stack like Cassandra, MongoDB, SQL Server, Elastic Search, MySQL, Oracle and much more.

Languages and Framworks: Multiple languages and Industry's Leading Framwork Support like Go, Python, C++, JavaScript, .Net, JAVA, C and much more.

NexaStack is a DevOps Platform to Automate | Monitor | Analyse your Application Delivery. NexaStack DevOps Platform Accelerates your Data Driven Application Delivery with proper visibility and security.

nexastack - devops and serverless computing platform

NexaStack DevOps Technology Solutions

NexaStack Essentials (For Startups): In NexaStack Essentials we take care of all basic needs of a Startup for hassle free IT and Application Lifecycle Operations.

NexaStack Productivity (For DevOps Team): NexaStack Productivity is best suited for workloads which are planning or already on production and looking forward to adopt DevOps Approach.

NexaStack Enterprise (Enterprise Stack): Create a single file for your project to describe the type of machine you want, the software that needs to be installed and the way you want to acccess the machine.

Summary

By incorporating DevOps Culture into your organization, you can deploy your code 30 times more frequently than your competitor with less than 50% of deployment fails.

By adopting DevOps culture you can collaborate your team which is composed of cross-functional team members including developers, QA, DBAs, Business Analysts, Operation Engineers and so forth which delivers many benefits such as continuous software delivery, less complex problems to fix, faster resolution of problems and more stable operating environment.

NexaStack is a DevOps Platform used to Automate, Monitor and Analyse your Application Delivery.

Now Accelerate your Data Driven Application Delivery with the Visibility and Security of NexaStack DevOps Platform.

Unleash the Power of MicroServices.Start Your Free Trial Today!

XenonStack 0

lean start to devops 1/27/2017 01:17:00 pm

How To Adopt DevOps in your Organization

While Scaling up the Business and working with remote teams with different Skillset and culture, I realize the need of processes and automation to improve the productivity and collaboration.

At growth Stage, with 3+ Years experience of delivering more than 55 projects in various domains for Startups and Enterprises including:

Mobility

BigData

Internet of Things

Private Cloud and Hybrid Cloud

Problems Faced By Developers & Operations Team

Ownership Issues during deployment

Fewer and Slow Releases

Flat Access Control and Security

Revision Control

Scaling up resources for application Stack

Manual processes involved in Delivery pipeline

Isolated Declaration of Dependencies

Single Configuration for Multiple Deployments

Manual Testing results into Slower release

Shared backing Services

Lean Start To Adopt DevOps

We Started transformation towards DevOps Strategy by adopting processes like Integration of DevOps Tools, Processes and Data into our work Culture.

Parallely, We Started adopting different Infrastructure architectures, Building Private Cloud, Docker, Apache Mesos and Kubernetes.

Steps We taken to adopt DevOps

Enforcing Rules with the help of right tools - Agile board integration with SCM, Build Tool and Deployment Tool

Collaboration Tools - Rocket Chat Integration with Taiga, GitLab, Jenkins

Continuous Integration and Delivery

Explicit Dependency Management

Automated Testing

Hands on Training

We started by creating two separate teams from existing pool of developers to adopt DevOps culture for new Projects in BigData and Mobile Applications.

After Initial hurdles in adaptation to Collaboration Tools and new delivery pipeline, results came out were extraordinary.

Results After Initial Phase

Improved Performance & productivity

Less Manual work

Better Collaboration and Communication

Developers getting more Empowered and Involved in Delivery

Proper Dependency and Configuration Management

Challenges In First Phase

Cultural Shift in the way Things were being developed

Changing Mindset for Adaptation

Support for Legacy Environments

Integrating Security and Compliance on new Setup

No support for Overlay Networks

Overall Results

Deployed Solution in Healthcare Startup

Then we implemented this approach in our HealthCare Startup RayCare where we were having multiple work flows and Big Data loads.

The Technology Stack for the startup was latest and cutting edge and tall enough to leverage Microservices.

Stood up Development, Staging and Production Environment with almost Zero Parity.

Jenkins Jobs for Android, iOS, Angular and Backend.

Highly Available and Distributed Cassandra Cluster.

Ansible Playbooks for turning up and down environment with one command.

Using Docker for Development and Staging Environment.

Deployed Solution in Analytics Startup

Introducing DevOps practices in an Analytics Startup with well experienced Database Administrators was altogether a great experience.

The aim was to fasten data loading, database provisioning and isolating environments for a team working on 3 distinct location.

The main highlights of the implementation are:

Bridging gap between 3 different development location by using relevant tools to improve collaboration.

Writing scripts to automate the process of Data Operations as much as possible.

Application Delivery Pipeline of an app with multiple versions running for different clients at same time.

Deployed and collaborated with Oracle Partner for Service Now Integration & DevOps

The aim was to integrate complex Oracle Enterprise Manager (OEM) with Service Now, a popular IT Services Management solution for easing provisioning of Oracle based resources through Service Now. The solution provided capable of:

Automating Oracle DB Binaries Provisioning.

Running Chef Recipes from Oracle Enterprise Manager.

A Restful Service to trigger DB Provisioning Recipe.

Item listing in Service Now to trigger Restful Service on successful transaction of item.

Admin approval process for infrastructure process.

Deployed Solution in WebRTC based Calling Platform –Web & Mobile App

The company is fully dedicated to free calling App for mobile iOS and Android platform like iOS, Android to phone number with facility to call all around the globe.

The technologies used are VMware, Couchbase, PostgreSQL, CentOS 7, Zabbix and many more

Hardware: From OVH dedicated server

Virtualization: Using Vmware bare metal

All services: on VMware Virtual Machines

The system had more than 10 NodeJS Apps, Highly Available and Distributed CouchDB Cluster which was changed to MongoDB in later stages.

We made the application delivery pipeline fully automated using Ansible and Jenkins. The infrastrucutre was also made mutable using Ansible to scale up and down according to load.

Deployed Solution For Microsoft Technology Stack in HealthCare Wellness

Migrating applications powered by .NET Ecosystem to DevOps powered application lifecycle pipeline is a challenging job. The application was deployed on private cloud powered by Openstack with the following features:

Openstack Images for .NET Ecosystem.

Jenkins Jobs for Continuous Integration

Integration of Openstack with Object Storage powered by Ceph.

Visual Studio Integration with popular DevOps Tools.

Summary

During Transformation Towards Agile & DevOps we realised that DevOps needs a platform where we can define workflow with different Integrations -

And the result which came out was NexaStack – Auotmate, Orchestrate and Monitor the Delivery pipeline

XenonStack 0

Post Top Ad

Wednesday, 26 April 2017

What is Fast Data?

What is Streaming and Real-Time Data

Stream & Real Time Processing Frameworks

Why Real-Time Streaming

Why we Need Real-Time Streaming?

Use Of Reactive Programming & Functional Programming

Streaming Architecture Matters

Lambda Architecture:

Wednesday, 22 March 2017

What is PostgreSQL?

Prerequisites

Tuesday, 14 March 2017

5 V’s to Define BigData

Additional 5V’s to Define BigData

Data Ingestion and Data Transformation

Major Industries Impacted with BigData

What is Data Integration?

Why Data Integration is required

Thursday, 9 March 2017

Doctors Are Drowning In Data

Thursday, 16 February 2017

What is a BlockChain?

Prerequisites

The Hyperledger Project

Hyperledger Fabric

Setting Hyperledger Cluster on Kubernetes

Hyperledger Kubernetes Replication Controller

Running Hyperledger Pods

Friday, 10 February 2017

Serverless Computing is Exploding

Serverless Computing

Serverless Computing Advances the way Applications are Developed

Monolith => Microservice => FaaS

Database Integration For Your Application

Continue Reading the full Article at: XenonStack.com/Blog

Monday, 6 February 2017

Serverless Architecture

What it means to have servers?

Advantages of going Serverless

How to migrate to a Microservices Architecture?

Microservices Architecture: Benefits

Employee Service

1. Setting up MongoDB Instance

Continue Reading the full article at: XenonStack.com/Blog

Wednesday, 1 February 2017

BlockChain on Docker

What is a BlockChain?

Prerequisites

The HyperLedger Project

HyperLedger Fabric

Setting HyperLedger Cluster

Pulling Images

Running the ChainCode

Starting And Registering The ChainCode

Read The Full Article At: XenonStack.com/blog

Friday, 27 January 2017

What is NexaStack

Why Should You Adopt DevOps Culture

Why NexaStack

NexaStack For Developers​

Self Provision Resources: ​

Continuous Delivery:

NexaStack For Operations

Multi-Cloud Support:

Runbook Automation:

Define Topologies:

Features Of NexaStack

NexaStack DevOps Technology Solutions

Summary

NexaStack For Developers

Self Provision Resources: