What is PostgreSQL?

• It runs on all major operating systems, including Linux, UNIX (AIX, BSD, HP-UX, SGI IRIX, MacOS, Solaris, Tru64), and Windows.

• It is fully ACID compliant, has full support for foreign keys, joins, views, triggers, and stored procedures (in multiple languages)

• It includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP.

• It also supports storage of binary large objects, including pictures, sounds, or video.

• It has native programming interfaces for C/C++, Java, .Net, Perl, Python, Ruby, Tcl, ODBC, among others, and exceptional documentation.

Prerequisites

• Kubernetes Cluster

• GlusterFS Cluster

FROM ubuntu:latest
MAINTAINER XenonStack

RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8

RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main" > /etc/apt/sources.list.d/pgdg.list

RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6

RUN /etc/init.d/postgresql start &&\
 psql --command "CREATE USER root WITH SUPERUSER PASSWORD 'xenonstack';" &&\
 createdb -O root xenonstack

RUN echo "host all  all 0.0.0.0/0  md5" >> /etc/postgresql/9.6/main/pg_hba.conf

RUN echo "listen_addresses='*'" >> /etc/postgresql/9.6/main/postgresql.conf

# Expose the PostgreSQL port
EXPOSE 5432

# Add VOLUMEs to allow backup of databases
VOLUME  ["/var/lib/postgresql"]

# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.6/bin/postgres", "-D", "/var/lib/postgresql", "-c", "config_file=/etc/postgresql/9.6/main/postgresql.conf"]

$ docker build -t dr.xenonstack.com:5050/postgres:v9.6

$ gluster volume create postgres-disk replica 2 transport tcp k8-master:/mnt/brick1/postgres-disk  k8-1:/mnt/brick1/postgres-disk
$ gluster volume start postgres-disk
$ gluster volume info postgres-disk

• Docker Image: We have created a Docker Image for Postgres in Step 2

• Persistent Shared Storage Volume: We have created a Persistent Shared Storage Volume in Step 3

• Deployment & Service Files: Next, we will create Deployment & Service Files

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
  namespace: production
spec:
  replicas: 1
  template:
 metadata:
   labels:
    k8s-app: postgres
 spec:
   containers:
   - name: postgres
     image: dr.xenonstack.com:5050/postgres:v9.6
     imagePullPolicy: "IfNotPresent"
     ports:
     - containerPort: 5432
     env:
     - name: POSTGRES_USER
       value: postgres
     - name: POSTGRES_PASSWORD
       value: superpostgres
     - name: PGDATA
       value: /var/lib/postgresql/data/pgdata
     volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: postgredb
   volumes:
     - name: postgredb
       glusterfs:
         endpoints: glusterfs-cluster
         path: postgres-disk
         readOnly: false

• What type of data it is

• Format of content of data required

• Whether data is transactional data, historical data or master data

• The Speed or Frequency at which data made to be available

• How to process the data i.e. whether in real time or in batch mode

5 V’s to Define BigData

Additional 5V’s to Define BigData

Data Ingestion and Data Transformation

• Data type: Type of data

• Transactional

• Historical

• Master Data and others

• Data Content Format: Format of data

• Structured (RDBMS)

• Unstructured (audio, video, and images)

• Semi-Structured

• Data Sizes: Data size like Small, Medium, Large and Extra Large which means we can receive data having sizes in Bytes, KBs, MBs or even in GBs.

• Data Throughput and Latency: How much data is expected and at what frequency does it arrive. Data throughput and latency depend on data sources:

• On demand, as with Social Media Data

• Continuous feed, Real-Time (Weather Data, Transactional Data)

• Time series (Time-Based Data)

• Processing Methodology: The type of technique to be applied for processing data (e.g. Predictive Analytics, Ad-Hoc Query and Reporting).

• Data Sources: Data generated Sources

• The Web and Social Media

• Machine-Generated

• Human-Generated etc

• Data Consumers: A list of all possible consumers of the processed data:

• Business processes

• Business users

• Enterprise applications

• Individual people in various business roles

• Part of the process flows

• Other data repositories or enterprise applications

Major Industries Impacted with BigData

What is Data Integration?

• Discovering the Data

• Profiling the Data

• Understanding the Data

• Improving the Data

• Transforming the Data

• Ingest Data from different sources where data resides in multiple formats.

• Transform Data means converting data into a single format so that one can easily be able to manage his problem with that unified data records. Data Pipeline is the main component used for Integration or Transformation.

• MetaData Management: Centralized Data Collection.

• Store Transform Data so that analyst can exactly get when the business needs it, whether it is in batch or real time.



Why Data Integration is required

• Make Data Records Centralized: As data is stored in different formats like in Tabular, Graphical, Hierarchical, Structured, Unstructured form. For making the business decision, a user has to go through all these formats before reaching a conclusion. That’s why a single image is the combination of different format helpful in better decision making.

• Format Selecting Freedom: Every user has different way or style to solve a problem. User are flexible to use data in whatever system and in whatever format they feel better.

• Reduce Data Complexity: When data resides in different formats, so by increasing data size, complexity also increases that degrade decision making capability and one will consume much more time in understanding how one should proceed with data.

• Prioritize the Data: When one have a single image of all the data records, then prioritizing the data what's very much useful and what's not required for business can easily find out.

• Better Understanding of Information: A single image of data helps non-technical user also to understand how effectively one can utilize data records. While solving any problem one can win the game only if a non-technical person is able to understand what he is saying.

• Keeping Information Up to Date: As data keeps on increasing on daily basis. So many new things come that become necessary to add on with existing data, so Data Integration makes easy to keep the information up to date.