Introduction

• Data Preprocessing
• Data Wrangling

Data Preprocessing

Data Wrangling

Need of Data Preprocessing

Need of Data Wrangling

Why is Data Preprocessing used?

• Inaccurate data (missing data) - There are many reasons for missing data such as data is not continuously collected, a mistake in data entry, technical problems with biometrics and much more.

• The presence of noisy data (erroneous data and outliers) - The reasons for the presence of noisy data could be a technological problem of gadget that gathers data, a human mistake during data entry and much more.

• Inconsistent data - The presence of inconsistencies is due to the reasons such that existence of duplication within data, human data entry, containing mistakes in codes or names i.e. violation of data constraints and much more.

Why is Data Wrangling used?

What is Data Leakage in Machine Learning/Deep Learning?

• Leakage of data from test dataset to training dataset.
• Leakage of computed correct prediction to the training dataset.
• Leakage of future data into the past data.
• Usage of data outside the scope of applied algorithm

How to check the presence of Data Leakage within the applied model?

• At the time of dividing time series dataset into training and test, the dataset is a complex problem.
• Implementation of sampling in a graphical problem is a complex task.
• Storage of analog observations in the form of audios and images in separate files having a defined size and timestamp.

How is Data Preprocessing performed?

• Ignoring the missing record - It is the simplest and effective method for handling the missing data. But, this method should not be performed at the time when the number of missing values are huge or when the pattern of data is related to the unrecognized basic root of the cause of statement problem.

• Filling the missing values manually - This is one of the best-chosen methods. But there is one limitation that when there is large dataset and missing values are large then, this method is not efficient as it becomes a time-consuming task.

• Filling using computed values - The missing values can also be filled by computing mean, mode or median of the observed given values. Another method could be the predictive values that are computed by using any Machine Learning or Deep Learning algorithm. But one drawback of this method is that it can generate bias within the data as the computed values are not accurate with respect to the observed values.

• Binning method - In this method sorting of data is performed with respect to the values of the neighborhood. This method is also known as local smoothing.
• Clustering method - In the approach, the outliers may be detected by grouping the similar data in the same group i.e. in the same cluster.
• Machine Learning - A Machine Learning algorithm can be executed for smoothing of data. For example, regression algorithm can be used for smoothing of data using a specified linear function.
• Removing manually - The noisy data can be removed manually by the human being but it is a time-consuming process so mostly this method is not given priority.

How is Data Wrangling performed?

Difference Between Data Preprocessing and Data Wrangling

Tasks of Data Preprocessing

• Data Cleaning - This is the first step which is implemented in Data Preprocessing. In this step, the main focus is on handling missing data, noisy data, detection, and removal of outliers, minimizing duplication and computed biases within the data.

• Data Integration - This process is used when data is gathered from various data sources and data is combined together to form consistent data. This consistent data after performing data cleaning is used for analysis.

• Data Transformation - This step is used to convert the raw data into a specified format according to the need of the model. The options used for transformation of data are given below:
  • Normalization - In this method, numerical data is converted into specified range i.e. between 0 and 1 so that scaling of data can be performed.
  • Aggregation - The concept can be derived from the word itself, this method is used to combine the features into one. For example combining two categories can be used to form a new category.
  • Generalization - In this case, lower level attributes are converted into a higher level.

• Data Reduction - After the transformation and scaling of data duplication i.e. redundancy within the data is removed and organize the data in an efficient manner.

Tasks of Data Wrangling

• Discovering - Firstly, data should be understood thoroughly and examine which approach will best suit. For example: if have a weather data when we examine the data it is observed that data is from one area and so main focus is on determining patterns.

• Structuring - As the data is gathered from different sources, the data will be present in different shapes and sizes. Therefore, there is a need of structuring the data in proper format.

• Cleaning - Cleaning or removing of data should be performed that can degrade the performance of analysis.

• Enrichment - Extract new features or data from the given dataset in order to optimize the performance of the applied model.

• Validating - This approach is used for improving the quality of data and consistency rules so that transformations that are applied to the data could be verified.

• Publishing - After completing the steps of Data Wrangling, the steps can be documented so that similar steps can be performed for similar kind of data to save time.

Overview

Prerequisites  

• Kubernetes - Kubernetes is an open source platform that automates container operations and Minikube is best for testing Kubernetes.
  
  

• Kubectl - Kubectl is command line interface to manage Kubernetes cluster either remotely or locally. To configure kubectl in your machine follow this link.

• Shared Persistent Storage - Shared Persistent Storage is permanent storage that we can attach to the Kubernetes container so that we don`t lose our data even container died. We will be using GlusterFS as a persistent data store for Kubernetes container applications.

• .NET Application Source Code - Application Source Code is source code that we want to run inside a kubernetes container.

• Dockerfile - Dockerfile contains a bunch of commands to build .NET application.

• Container-Registry - The Container Registry is an online image store for container images.

• Private Docker Hub
• AWS ECR
• Docker Store
• Google Container Registry

Create a Dockerfile

FROM microsoft/dotnet:1.1-sdk
# Setting Home Directory for application 
WORKDIR /app 

# copy csproj and restore as distinct layers
COPY dotnetapp.csproj .
RUN dotnet restore

# copy and build everything else
COPY . .
RUN dotnet publish -c Release -o out

EXPOSE 2223

ENTRYPOINT ["dotnet", "out/main.dll"]

Building .NET Application Image

$ docker build -t <name of your application>:<version of application> .

Publishing Container Image

Creating Deployment Files for Kubernetes

• Deployment File

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <name of application>
  namespace: <namespace of Kubernetes>
spec:
  replicas: <number of application pods>
  template:
 metadata:
   labels:
    k8s-app: <name of application>
 spec:
   containers:
   - name: <name of application>
     image: <image name >:<version tag>
     imagePullPolicy: "IfNotPresent"
     ports:
      - containerPort: 2223

• Service File

apiVersion: v1
kind: Service
metadata:
  labels:
 k8s-app: <name of application>
  name: <name of application>
  namespace: <namespace of Kubernetes>
spec:
  type: NodePort
  ports:
  - port: 2223
  selector:
 k8s-app: <name of application>

Running .NET Application on Kubernetes

$ kubectl create -f <name of application>.deployment.yml
$ kubectl create -f <name of application>.service.yml

Verification

$ kubectl get po --namespace=<namespace of kubernetes> | grep <application name>

Result of above command

What is Kotlin Language?

/* Java Code */
static int num1, num2; //explicit declaration
num1 = 20; //use the variables anywhere
num2 = 30;

/* Kotlin Code*/
val a: Int
val b: Int
a=5
b=10

/* Kotlin Code*/
/* Simple Hello Word Example*/

//optional package header
package hello 

//package level function, which return Unit and takes an array of string as parameter
fun main(args: Array < String > ) { 
 val scope = “world”
 println(“Hello, $scope!”) //semicolons are optional, have you noticed that? :)
}

Filename extensions of the Java are .java, .class, .jar but on the other hand filename extensions of the Kotlin are .kt and .kts.

Benefits of Kotlin Language

• Kotlin compiles to JVM bytecode or JavaScript - Like Java, Bytecode is the compiled format for Kotlin programs also. Bytecode means Programming code that, once compiled, is run through a virtual machine instead of the computer’s processor. By using this approach, source code can be run on any platform once it has been compiled and run through the virtual machine. Once a kotlin program has been converted to bytecode, it can be transferred across a network and executed by JVM(Java Virtual Machine).

• Kotlin programs can use all existing Java Frameworks and Libraries - Yes, it's true that Kotlin programs can use all existing java frameworks and libraries, even advanced frameworks that rely on annotation processing. The main important thing about kotlin language is that it can easily integrate with Maven, Gradle and other build systems.

• Kotlin can be learned easily and it is approachable. It can be learned easily by simply reading the language reference.The syntax is clean and intuitive(easy to use and understand). Kotlin looks a lot like Scala but is simpler.

• Kotlin is Open Source and it costs nothing to adopt.

• Automatic conversion of Java to Kotlin - JetBrains integrated a new feature into IntelliJ which converts Java to Kotlin and saves a huge amount of time. And it also saves us to retype mundane code.

• Kotlin’s null-safety is great - Now get rid of NullPointerExceptions. This type of system helps us to avoid null pointer exceptions. In Kotlin the system simply refuses to compile code that tries to assign or return null. Consider the following example -

val  name: String = null  // tries to assign null, won’t  compile.
 
fun  getName() : String = null  // tries to return null, won’t  compile.

• Code reviews are not a problem - Kotlin is much more focuses on readable syntax so code reviews are not a problem, they can still be done by those team members who are not familiar with the language.

Features of Kotlin Language

• The billion dollar mistake made right. As already mentioned above that Kotlin avoids the null pointer exception. If we try to assign or return null to a variable or function respectively, then it won’t compile.

val name: String? = null    //assigned  null and it will compile  also.
 
fun getName() : String? = null //returned null and it  will compile too.

/* won’t compile */ 
  val name: String? = null
  val len = name.length 

/* correct way */ 
  val name: String? = null 
  val len = name?.length

• Versatile
  
  
  
  
  

• Lean Syntax and Concise - One liner functions take one line, simple structs/JavaBeans can also be declared in one line. Real properties generate getters and setters behind the scenes for Java interop. And Adding the data annotation to a class triggers autogeneration of boilerplate like equals, hashCode, toString and much more.

/*  Java program */ 

public class Address {
  
   private String street;

   private int streetNumber;

   private String postCode;

   private String city;

   private Country country;

   public Address(String street, int streetNumber, String postCode, String city, Country country) {

       this.street = street;

       this.streetNumber = streetNumber;

       this.postCode = postCode;

       this.city = city;

       this.country = country;

   }

   @Override
   public boolean equals(Object o) {
       if (this == o) return true;

       if (o == null || getClass() != o.getClass()) return false;

       Address address = (Address) o;

       if (streetNumber != address.streetNumber) return false;

       if (!street.equals(address.street)) return false;

       if (!postCode.equals(address.postCode)) return false;

       if (!city.equals(address.city)) return false;

       return country == address.country;
   }

   @Override
   public int hashCode() {
       int result = street.hashCode();

       result = 31 * result + streetNumber;

       result = 31 * result + postCode.hashCode();

       result = 31 * result + city.hashCode();

       result = 31 * result + (country != null ? country.hashCode() : 0);

       return result;

   }

   @Override

   public String toString() {

       return "Address{" +

               "street='" + street + '\'' +

               ",     streetNumber=" + streetNumber +

               ",     postCode='" + postCode + '\'' +

               ",     city='" + city + '\'' +

               ",     country=" + country +

               '}';

   }

   public String getStreet() {

       return street;

   }

   public void setStreet(String street) {

       this.street = street;

   }

   public int getStreetNumber() {

       return streetNumber;

   }

   public void setStreetNumber(int streetNumber) {

       this.streetNumber = streetNumber;

   }

   public String getPostCode() {

       return postCode;

   }

   public void setPostCode(String postCode) {

       this.postCode = postCode;

   }

   public String getCity() {

       return city;

   }

   public void setCity(String city) {

       this.city = city;

   }

   public Country getCountry() {

       return country;

   }

   public void setCountry(Country country) {

       this.country = country;

   }

}

/*  Kotlin Program */

data class Address(var street:String,

       var streetNumber:Int,

       var postCode:String,

       var city:String,

       var country:Country)

Difference Between Kotlin And Java

• Null Safety - As already mentioned in above section that Kotlin avoids NullPointerException. Kotlin fails at compile-time whenever a NullPointerException may be thrown.

• Data Classes - In Kotlin there are Data Classes which leads to autogeneration of boilerplate like equals, hashCode, toString, getters/setters and much more...
  
  
  Continue Reading About Kotlin V/s Java At: XenonStack.com/Blog