My Useful Documents

One of my biggest problems is that I tend to forget the technologies that I learned so fast! therefore, I spent sometime documenting some of them. Here is the link to some of them:

Python and Bash tips:
https://docs.google.com/document/d/1bTDROhpg7YMwpW0PbM_K0I9VEl9k3JnvFw6DS_ZLupE/edit#heading=h.fgq7wdx48rqg

PyTest:
https://docs.google.com/document/d/1SmzZCnmrELgW9DksIJTaR6ECBy7fUk0Raa6jg1NmsG0/edit

GitHub and Git:
https://docs.google.com/document/d/10E5abLpL0lAnfPO9TcsbbnAElYMUK9rMNn8pCD6nhRk/edit?usp=sharing

Bash and Linux:
https://docs.google.com/document/d/1ABg9GMk-PTDGHLIU4Y48kOc-Zs8wvehGOzyn8XpFBYs/edit?usp=sharing

Docker:
https://docs.google.com/document/d/1uV78_eww4FQmNzW5cynJJJ8noM_VwKY6oPPQ0-FH1zc/edit?usp=sharing

NumPy and Matplotlib:
https://docs.google.com/document/d/1VPvQpXDywo6r142EThhRhC6WvXC51HcD_pKR8L3VqHs/edit?usp=sharing

Pandas:
https://docs.google.com/document/d/1H0ztba0e2n8hCCMxTEtq4uUTPGObq9NJLOtUX86TAuI/edit?usp=sharing

Ansible:

https://docs.google.com/document/d/1qZzsrLD0jGY28CrfEU6u8nXrLfoIOzbWBvYxwz7Ues8/edit?usp=sharing

Open Source programs cheat sheets and guides:

https://drive.google.com/drive/folders/15fMkW4BFcEw7F_5VwBq6suXqmm3s7Q92?usp=share_link

Introduction to BAZEL:

https://docs.google.com/document/d/1gC9LtrWSiZfjRH_kFYqy3IkPWMwx5u1zZNx_IDTh_3Y/edit?usp=sharing

Start using VIM at intermediate level

This text document (which is a .c file format only for fancy markup by default) is a quick intro to VIM editor that could be very useful both as a mini cheat sheet and a guide to start using it at an intermediate level without any previous knowledge.

BONUS: Also a very quick intro for TMUX is available under the "miniTMUX.txt" file.


Check it out here!

My Article About LizardFS

Thanks to Mark Mulrainey, my experience as a DevOps engineer that worked with LizardFS (an open-source distributed storage) for a scalable distributed infrastructure, is now published as a community article on its weblog. Check it out from the link below! 




Check it out here

Netbill Transaction Protocol

NetBill Protocol in Theory

What is NetBill? [From Original Paper]

NetBill is a system for micropayments for information goods (digital commodities) on the Internet. A customer, represented by a client computer, wishes to buy information from a merchant’s server. An account server (the NetBill server), maintains accounts for both customers and merchants, linked to conventional financial institutions.

The NetBill Transaction Model [From Original Paper]

The NetBill transaction model involves three parties: the customer, the merchant and the NetBill transaction server. A transaction involves three phases: price negotiation, goods delivery, and payment.

Download Implementation From Github With Detailed Explanations

Implementation Of NetBill Protocol

In this project me (Navid Malek) and my fellow friend Reza Rahimi implemented most of NetBill transaction protocol, including:

• Transaction Protocol

Paper Sections

3.2. The Price Request Phase

3.3. The Goods Delivery Phase

3.4. The Payment Phase

• Error recovery (Not enough balance, Courruption, No access, etc.)

• Pseudonyms Protocol

Paper Sections

4.2. Pseudonyms

• Access Control Mechanism

Mini Access contol app not According to paper

Approach

Our main focus was to implement the protocol, so the approach we take was to use intermediary files that act as Sockets; hence, for various steps of protocol istead of writing data into socket and read from it, we have used files. In the next section, I have provided more details about the files and codes presented.

How To Run

1. run the following command in terminal: git clone https://github.com/navidpadid/NetBill_Transaction_Protocol/

2. run the codes

Here are various scenarios which I've ran the code from a fresh clone of repository.

Some scenarios include: with/without pseudonyms, with/without access to buy, with/wihtout NetBill account, with/wihtout enough credits to buy a commodity.

Download Implementation From Github With Detailed Explanations

Limited DevOps Documents

Download From Github With Explanations

Various limited documents from my work tools when I used to work as DevOps Engineer, these are basic usage and limited testing results ( because I don't have the premission to publicize the full documents that I have written for my work )

Description:

ETCD_CLUSTER.pdf ==> Setting up an ETCD cluster

objective:

• setup an etcd cluster on 3 servers

• write appropriate service to be sure ETCD will be always running

• ETCD version ==> 3.3.9

• server OS ==> CentOS7

The related ansible code are in the mycontrolansible directory.

LizardFS.pdf ==> Setting up LizardFS and testing it

objective:

• setup simple LizardFS master on one server

• test LizardFS performance on 3,5,7 chunk servers (HDD)

• test LizardFS performance on SSD

• LizardFS version ==> 3.12

ProxySQL.pdf ==> Set up and testing ProxySQL

objectives:

• setup ProxySQL load balancer on one server

• configuring ProxySQL

• test the performance and load balancing of ProxySQL with SYSBENCH

• setup ProxySQL architecture with no single point of failure

telegrafLogparser.pdf ==> Using Telegraf to Parse Custom Logs

objective:

• read custom logs with telegraf

• parse custom logs with telegraf

• generate an output from custom logs to influxdb

The TIG stack [ telegraf, influxdb, grafana ]

DatabaseTestDoc.pdf ==> Testing Database response time

We will test 4 databases ( this doc is very limited and acts as a road map for more professional tests ):

RDB: -mysql -postgres

No-SQL: -mongo -cassandra

GrafanaPrometheus.pdf ==> Setup Prometheus and Grafana for monitoring (very basic)

objectives:

• setup Prometheus server

• setup Prometheus exporters

• setup Grafana server

• setup Grafana dashboard

GalleraCluster.pdf ==> Setting up a MySQL Gallera cluster (very basic)

objective:

• setup an MySQL Gallera cluster on 3 servers

• write apporopiate service to be sure MySQL Gallera cluster will be always running

• MySQL version ==> 5.7

• MySQL-wsrep ==> 5.7

• server OS ==> CentOS7

TCP_tune.pdf ==> Some TCP tuning parameters that i have gathered form internet ( this is a messy doc, just gathered information from Internet, for more info on TCP_TUNING refer to my blog! )

 

Download From Github With Explanations

Transparently Proxy All Traffic Through TOR

In this post we will learn how to proxy all the network's traffic (TCP and UDP) transparently through TOR with least difficulty.

Download full tutorial from here

Requirements:

• Linux OS (tested on alpine and ubuntu)

• iptables (Linux firewall)

• RedSocks

What is RedSocks?

Reference: https://github.com/darkk/redsocks

Redsocks is the tool that allows you to proxify(redirect) network traffic through a SOCKS4, SOCKS5 or HTTPs proxy server. It works on the lowest level, the kernel level (iptables). The other possible way is to use application level proxy, when the proxy client is implemented in the same language as an application is written in. Redsocks operates on the lowest system level, that’s why all running application don’t even have an idea that network traffic is sent through a proxy server, as a result it is called a transparent proxy redirector.

System’s Architecture and Setup for TCP Connections

So this is the big image, almost every tcp packet will be redirected to port 12345 which redsocks service listens for incoming packets; after that, redsocks will redirect the received traffic to another ip and port in socks protocol format.

Also have in mind that for iptables in docker, you have to use docker run --privileged flag

Download full tutorial from here

Easiest Way to Run Multimedia (GUI) Applications Inside Docker

In this post I will show you not only how to run any multimedia application inside docker, but also in efficient and easy way.

Download full tutorial from here

Requirements:

• Linux OS and Docker (tested on ubuntu)

• X or Wayland (Linux Display Servers)

  • Ensure that the packages for an X or Wayland server are present on the Docker host. Please consult your distribution's documentation if you're not sure what to install. A display server does not need to be running ahead of time.

• X11docker

  • x11docker allows Docker-based applications to utilize X and/or Wayland on the host. Please follow the x11docker installation instructions and ensure that you have a working setup on the Docker host.

What is X11Docker?

Reference: https://github.com/mviereck/x11docker/

x11docker allows to run graphical applications (or entire desktops) in Docker Linux containers.

• Docker allows to run applications in an isolated container environment. Containers need much less resources than virtual machines for similar tasks.

• Docker does not provide a display server that would allow to run applications with a graphical user interface.

• x11docker fills the gap. It runs an X display server on the host system and provides it to Docker containers.

• Additionally x11docker does some security setup to enhance container isolation and to avoid X security leaks. This allows a sandbox environment that fairly well protects the host system from possibly malicious or buggy software.

Download full tutorial from here

Operating Systems Project - Linux Packet Capturing

The main project that me and my colleague designed for Operating Systems course ( Spring 2018 ) - TA of OS

Project Description ( summary ):

The goal of this project is to gather information about incoming and outgoing packets in system. ( some kind of packet capturing )

The desired informations are :

• Lenght of packet
• Protocols of packet ( in all available layers of network except application layer, for example Ethernet,IP,TCP )
• Hash value of packet
• Total Processing time of packet

Phase one :

1. implement a systemcall with a single integer input indicating what information you desire from packets and single output buffer to copy the data from kernel space to user space.
2. an interactive user space program that talks to the user and systemcall call above ( clean input and output for user ).
   
   
   Phase two :

   1. implement a kernel module with a single proc entry file for input indicating what information you desire from packets and single proc entry file for output and to copy the data from kernel space to user space.
   2. an interactive user space program that talks to the user and kernel module and proc entry files above ( clean input and output for user ).

   Phase three :

   Performance comparsion of kernel module and system call ( the first two phases ).
   
   
   DOWNLOAD PROJECT DESCRIPTIONS IN DETAIL
   

   Approach Taken for answer:
   
   All of the desired informations are in sk_buff data structure.
   my approach was to clone sk_buff from driver, right before the driver ( here : e1000 ) wants to pass the packet to the next network layer handler ( application or the NIC ). with this approach i will have a clone for each packet.
   now that i have the information the rest is easy, just copy the desired information from the sk_buff to the output buffer and copy from there to user space.
   
   
   
   
   note: for the LKM you should use extern and export symbol.
   
   
   
   the idea for this approach was from here

   
   
   DOWNLOAD THE IMPLEMENTATION OF THIS APPROACH WITH DOCUMENTATION IN PERSIAN FROM STUDENTS ( WITH MY HELP )
   
   
   Second approach:
   
   second approach i believe is based on this idea from IBM.
   
   Download the Implementation of Second Approach From a Student With Document

Getting Started With Linux

This is a very straight forward guide for someone who wants to get started with linux in a matter of hours.

the Guide consists of three main steps:

• gathering some basic knowledge
• gathering some system knowledge
• gathering some command-line knowledge
  
  

------------------------------

Download the Guide here

Download Sample Answer From Students ( there are some mistakes )

Linux 2.3 - Part 2.3 Implementation of desired kernel module and user program

Part 2.3

Implementation of desired kernel module and user program

Download full tutorial from here

Implement the module

?Now What will this module do and how
.tops_init is the first routin that will be runned , the module is well commented so just read the comments to see what happens
 

Implement user program

Just do the writing and reading from file automatically!

 

Download full tutorial from here

Linux - 2.1 Implementation of Simple “hello world” Module and Run

Part 2.1

Implementation of Simple “hello world” Module and Run

 

TO GET THE FULL TUTORIAL CLICK HERE

Every module structure have two major functions:

1. init

• executed when the module is loaded into kernel

2. cleanup

• executed when the module is removed from kernel

the simple module implementation is here:



TO GET THE FULL TUTORIAL CLICK HERE

Linux - Having fun with Pipes

1) what is pipe?

Pipe man page :

«pipe()  creates  a pipe, a unidirectional data channel that can be used for interprocess communication. The array pipefd is used to return two file  descriptors  referring to the ends of the pipe. pipefd[0]  refers  to the read end of the pipe. pipefd[1] refers to the write end of the pipe.  Data written to the write end of  the pipe is buffered by the kernel until it is read from the read end of the pipe. For further details, see pipe(7).»

 

2)Creating half-duplex communication between parent and child:

 

the code and output:

 

description:

The child closes it’s write (fd[1]) and then reads from fd[0] into readForChild buffer and prints the output.

The parent closest it’s read(fd[0]) and then writes into writeForParent buffer and then passes the address to the fd[1].

 

 

More complicated Half-Duplex communication:

 

read comments for better undrstanding.

Linux - 1.2 Finding the important functions in kernel which gather information about some hardware's

Part 1.2

finding the important functions which gather information about some hardware's

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE

 

list of hardwares:

• CPU

• Memory

• network interface

• sound card

• power supplier

inorder to implement our information on system we should find :

1. what are the type of variables we want?

2. What is implementing them?

3. Where are they defined?

4. How to implement from number 2, to number 1?

so in this part we have to answer these 4 questions.

 

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE

 

Linux - 1.1 Building kernel from source with adding a simple system call

Part 1.1

Adding simple system call to kernel from scratch

 

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE

First create a directory named linuxSRC to download linux source, and Change directory to the folder:

console:

mkdir /linuxSRC
cd linuxSRC

 

after that you should first download the linux source, inorder to download every source on the package repository of ubuntu you should go to your package list ,You need to uncomment the deb-src lines in the main file /etc/apt/sources.list not the extra files in/etc/apt/sources.list.d:
open software & updates
go to ubuntu software tab
check source code
save and exit

 

after that you should get your current version linux kernel and build dependencies:

console:

 

mkdir newKernel

cd newKernel

 

now download the linux source and decompress it:

console:

 

sudo apt-get install linux source

tar xjvf /usr/src/linux-source-<yourversion>tar.bz2

now go to the specified folder bellow :

 

/home/nubuntu/linuxSRC/linux-source-4.4.0/arch/x86/entry/syscalls

now open syscall_64.tbl or syscall_32.tbl (based on your system)(i use 64)

at the end add your hello syscall after the last number (here 333), with the following format, save and exit.
 

from here the tutorial have images, so I've created a nice PDF document with screen shots to download.

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE