Reinforcement Learning Algorithms Implementation: From Zero To Hero

What is Reinforcement Learning about?

In contrast to supervised learning where machines learn from examples that include the correct decision and unsupervised learning where machines discover patterns in the data, reinforcement learning allows machines to learn from partial, implicit and delayed feedback. This is particularly useful in sequential decision making tasks where a machine repeatedly interacts with the environment or users. Applications of reinforcement learning include robotic control, autonomous vehicles, game playing, conversational agents, assistive technologies, computational finance, operations research, etc..

Disclaimer!

This repository mainly contains my assignments for this Reinforcement Learning course, which was offered in Fall 2021 at UWaterloo by Professor Pascal Poupart. Because of the academic integrity, I don't have the permission to post this repository publicly online; therefore, this repository is only accessible upon explicit request to me as defined in this document.

Download From Github With Explanations [PRIVATE REPO, ONLY ACCESSIBLE BY EXPLICIT REQUEST]

Part 1

Summary:

• Markov Decision Process [from scratch in Python]
  
  • value iteration
  • policy iteration
  • modified policy iteration
• Maze problem to test above algorithms
• Compare the performance of each algorithm
• Q-Learning [from scratch in Python]
• Use matplotlib to compare the effect of the Q-Learning parameters on the cumulative discounted rewards per episode
• deep Q-network to solve the CartPole problem from Open AI Gym
  • Using Agents library from TensorFlow
• Use matplotlib to compare the effect of the deep Q-network parameters on the average cumulative discounted rewards [also averaged across several runs to reduce stochasity]
• More details: https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-fall21/assignments.html assignment 1 section

Part 2

Summary:

• Bandit algorithms from scratch in Python
  
  • epsilon-greedy
  • Thompson sampling
  • UCB
• REINFORCE algorithm from scratch in Python
• model-based RL algorithm from scratch in Python
  
• Soft Q-Learning in Pytorch
• Soft Actor Critic in Pytorch
• Discussion over the properties of each algorithms and their effect over the performance
• More details: https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-fall21/assignments.html assignment 2 section

Part 3

• Partially Observable RL
  • Deep Recurrent Q learning (DRQN) algorithm in Pytorch
    • Using LSTM and MLP
    • Compare to Deep Q Network's performance
• Generative Adversarial Imitation Learning (GAIL) algorithm in Pytorch
  • Using deterministic policy gradient update technique
  • Compare to Behavior Cloning's (BC) performance
• Categorical (C51) distributional RL algorithm
  • Compare to DQN on the Cartpole domain with epsilon greedy exploration
• More details: https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-fall21/assignments.html assignment 3 section

Download From Github With Explanations [PRIVATE REPO, ONLY ACCESSIBLE BY EXPLICIT REQUEST]

Linux - 2.1 Implementation of Simple “hello world” Module and Run

Part 2.1

Implementation of Simple “hello world” Module and Run

 

TO GET THE FULL TUTORIAL CLICK HERE

Every module structure have two major functions:

1. init

• executed when the module is loaded into kernel

2. cleanup

• executed when the module is removed from kernel

the simple module implementation is here:



TO GET THE FULL TUTORIAL CLICK HERE

Linux - 1.1 Building kernel from source with adding a simple system call

Part 1.1

Adding simple system call to kernel from scratch

 

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE

First create a directory named linuxSRC to download linux source, and Change directory to the folder:

console:

mkdir /linuxSRC
cd linuxSRC

 

after that you should first download the linux source, inorder to download every source on the package repository of ubuntu you should go to your package list ,You need to uncomment the deb-src lines in the main file /etc/apt/sources.list not the extra files in/etc/apt/sources.list.d:
open software & updates
go to ubuntu software tab
check source code
save and exit

 

after that you should get your current version linux kernel and build dependencies:

console:

 

mkdir newKernel

cd newKernel

 

now download the linux source and decompress it:

console:

 

sudo apt-get install linux source

tar xjvf /usr/src/linux-source-<yourversion>tar.bz2

now go to the specified folder bellow :

 

/home/nubuntu/linuxSRC/linux-source-4.4.0/arch/x86/entry/syscalls

now open syscall_64.tbl or syscall_32.tbl (based on your system)(i use 64)

at the end add your hello syscall after the last number (here 333), with the following format, save and exit.
 

from here the tutorial have images, so I've created a nice PDF document with screen shots to download.

DOWNLOAD FULL TUTORIAL WITH IMAGES FROM HERE

 

Operating System - Linux

in these series of tutorials i want to play with Linux kernel and Loadable Kernel Modules (LKM) along with getting basic information about kernel architecture and system calls (part 1) , proc file system and processes information's (part 2), these tutorials are adopted from my operating system's class project.

there are two major parts as i mentioned before:

 

1. Linux kernel (focusing on system calls)

• Building kernel from source with adding a simple system call
• Finding specified devices information and necessary functions to implement them
• How to implement the system call and user space program to get info on specified drivers from kernel

     2. Loadable Kernel Modules (focusing on Proc File system)

• Implementation of simple “hello world” module and run
• Finding specified information and necessary functions to implement desired module
• Implementation of desired kernel module and user program

requirements:

• Ubuntu version: 16.04.2 (recommended not required)

• Linux kernel : 4.x.x

• Internet connection

• about 30 or 40 gigabytes of space (first part)

• lots of time for compile ! (first part)

i will start from the simple system call in the next post.

navidx