ARTH Task 7

By Team ARTH2020.1.3

Task 7.1: Elasticity Task

🔅Integrating LVM with Hadoop and providing Elasticity to Data Node Storage

🔅Increase or Decrease the Size of Static Partition in Linux.

🔅Automating LVM Partition using Python-Script.

Solution (7.1)

Pre-requisites — Set up a Hadoop cluster with one name node, at least 2 data nodes, and 1 client node.

1. Creating physical volume using pvcreate command

Fig: Creating a new PV of size 1 GiB

2. Creating a volume group by allocating the new PV

Fig: Creating the new VG

3. Create a logical volume of the desired size (Here 700 MiB) using lvcreate command

Fig: Creating the new LV

4. Format the new LV and mount it on the directory through data node will contribute storage

Fig: Formatting and mounting the new LV

Fig: Checking the list of mounted partitions and their filesystems

5. Connect all the data nodes and check the dfsadmin report for the contribution of each data node

IP addresses to be noted: Name Node — 192.168.1.39/32, Data node 1–192.168.1.40/32, Data Node 2–192.168.1.52/32

Fig: IP address of name node

Fig: IP address of data node 1

Fig: IP address of data node 2

Fig: dfsadmin report — Part 1

Fig: dfsadmin report — Part 2

As you can see in the images above, Data node 1 is contributing roughly 700 MiB as only a particular partition is mounted on the directory through which the node contributes storage.

6. Using LV’s nature of dynamic partitions to increase the storage contributed by data node 1 and format it partially

Fig: Increasing LV size and partially formatting it

Fig: Changes seen in the Hadoop cluster report

7. Automating LVM using Python script — Please refer to https://github.com/devanshu06/ArthTeamTasks-Repo for the code

Task 7.2: Docker Task

🔅Configuring HTTPD Server on Docker Container

🔅Setting up Python Interpreter and running Python Code on Docker Container

Solution (7.2)

Pre-requisites — Docker must be set up in the system. (Here, the entire task is done on an RHEL8 VM)

1. Configuring HTTPD server on Docker

To configure the HTTPD server on Docker, we can use any official Docker images from Docker Hub. We will be using CentOS for this.

While launching the Docker container with CentOS, we can also expose port 80 of the container so that the webserver can be contacted by the base OS on which the VM is running.

After launching the container, we can use the pre-configured yum reps to directly install httpd, as well as net-tools, which is helpful in many networking-related operations.

Fig: Installing httpd and net-tools on CentOS container — Part 1

Fig: Installing httpd and net-tools on CentOS container — Part 2

Now, to start the service, we cannot use systemctl like we typically use in RHEL8. Instead, we need the command /usr/bin/httpd, which is actually the command that starts httpd service in RHEL8 in the background, when we use the systemctl start httpd command.

Fig: Starting and enabling the httpd server

To check if the server is actually accessible, we can try to access the webserver through the port we have exposed (8080) on the host to the container’s port 80.

Fig: Checking for accessibility of webserver

2. Setting up Python in the container

To work with Python in a container using CentOS image, we can once again use the pre-configured yum repository to install Python version 3.

Fig: Setting up Python version 3 in CentOS container

To test it, we can simply type python3 to access the Python REPL, which is also a confirmation that Python programs can be executed in this container.

Fig: Confirming Python is installed