Linux - Damchey's Blog

Tutorial: Installing Proxmox VE 6 on Software RAID1 (mdraid)

Proxmox VE is an amazing piece of server hypervisor software that lets you create and manage both VMs and LXC containers through a very user friendly web based GUI. However if you want any form RAID with your drive, the default solution is ZFS RAID and it discourages the use of Linux Software RAID. And with NVMe drives becoming affordable and more commonplace, it offers much better performance compared to SATA or SATA SSDs, but as of now I haven’t found any hardware RAID cards that support NVMe, so hardware RAID is out of the question if you want to use NVMe drives.

I have no experience with ZFS, but from what others say, it requires a lot of RAM and it is suited more towards enterprise (expensive) NVMe drives as it wears out consumer grade NVMe drives very fast.

So, in this guide we will install Proxmox on top of two NVMe drives in linux RAID1 (mdraid) configuration. We will be using UEFI bootloader instead of legacy BIOS.

Debian Installation & MD RAID setup:

We need to install Debian 10 (Buster) first and setup the RAID during the installation process. We cannot use the ISO provided by proxmox, it will not provide you with any options to create a software RAID.

Create a 500MB partition on each disk
Set the partitions to ESP (EFI system partition). We cannot raid these two partitions. We will later on learn how to load the efi bootloader on both partitions and keep them in sync, so that the system will boot even if one disk fails.
Create a partition on each of the disk with on the remaining space and choose “physical volume for raid” as the partition type.
Then you need to “Configure software RAID” and select “Create MD device“. We are setting up a mirrored raid (RAID1), so enter “2” for the number of devices and then choose /dev/nvme1n1p1 and /dev/nvme0n1p1

Fix: Failed to access DBM file when using mod_ruid2 and mod_security

If you are using mod_ruid2 and mod_security on your cpanel server, you may come across a problem if one or more of your mod_security rules use the persistent storage (tracking user IP addresses, etc.). This can be seen in your apache error log file at /var/log/apache2/error_log containing the following lines:


ModSecurity: collection_store: Failed to access DBM file "/var/cpanel/secdatadir/ip": Permission denied

This happens when running apache under mod_ruid2, apache cannot access the ip.pag and ip.dir files located in /var/cpanel/secdatadir/ as the apache process runs as the user and not as nobody.

So far, there’s no proper fix for it, with cpanel saying that the fix has to come from mod_security itself.

Meanwhile to get your rules to work, you can change permissions of the the files, so its writeable by everyone.


# chmod 777 /var/cpanel/secdatadir/ip*

Another problem: Cpanel reverting the above changes

During the daily cpanel update process (upcp), the above files are flushed and regenerated causing the permissions to be reset to the original one. To fix this, you will need to create a cpanel hook that runs right after the daily cpanel update process.

Create a script called resetmodsecperm and store it in the /scripts/ folder.


#!/bin/bash
chmod 777 /var/cpanel/secdatadir/ip.*

Make the script executable


# chmod a+x /scripts/resetmodsecperm

Create a cpanel hook


# /usr/local/cpanel/bin/manage_hooks add script /scripts/resetmodsecperm --manual --category System --event upcp --stage post

What the above does is, you are creating a hook that runs the resetmodsecperm script after (–stage post) the cpanel update process (–event upcp).

So with that done, your mod_sec rules will run properly and not throw errors. I know this is a hacky way and chmodding the files 777 could be risky, but this seems to be the only way to make all mod_sec rules work.

UPDATE: With a recent cPanel update, I have noticed a new script has been added and is made to run every two hours. The script basically shrinks the ip.pag and ip.dir files, but this also reverts the permissions of the files and you get the same error as before.

New entry found in crontab:


0 */2 * * * /usr/local/cpanel/scripts/shrink_modsec_ip_database -x 2&amp;gt;&amp;amp;1

So, in your cron tab, have your script run after the above script.


2 */2 * * * /scripts/resetmodsecperm

This will run your script 2 minutes after every time the ip.pag and ip.dir files are recreated.

May/June 2019 UPDATE:

With the recent EasyApache update from cPanel (29/5/2019, 5/6/2019), modsecurity has been updated from 2.9.2 to 2.9.3. Although modsecurity mentions this new update has fixed compatibility issues with modruid2, some rules do not seem to work.

cPanel has also raised an issue regarding the same problem.

So, until the issue is fixed, a work around is to downgrade modsecurity to 2.9.2 and prevent cpanel from upgrading it:


yum remove ea-apache24-mod_security2

yum install ea-apache24-mod_security2-2.9.2-11.11.7.cpanel.x86_64

Append ea-apache24-mod_security2* to the exclude line in /etc/yum.conf

How to log real visitor IP when using Engintron on Cpanel

If you are using nginx as reverse proxy on your cpanel server through the Engintron plugin, you will need to make some changes to the apache configuration to log the real visitor’s IP instead of the server’s own IP address.

Engintron will install the mod_remoteip during installation and properly make the changes to the apache LogFormat. But the changes don’t seem to persist.

Engintron also provides a “Restore Nginx IP forwarding in Apache” option to restore apache log files through the WHM panel, but the settings are lost when apache rebuilds its configuration at a later time. I have noticed this happening quite often.

For cPanel 74.0.4 and later, you can now make the changes through the WHM GUI. Refer instructions at the end of the post.

So, to make your changes permanent, you’ll have to do the following:

Assuming you are using EasyApache 4, copy the ea4_main.default file to a new file named ea4_main.local. Next time cpanel rebuilds apache configuration, it will use this file instead. This has precedence over the .default file.


# cd /var/cpanel/templates/apache2_4

# cp ea4_main.default ea4_main.local

Make the following changes to the ea4_main.local file

Look for the lines:


LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

And replace it with


LogFormat "%a %l %u %t \"%r\" %>s %b" common
LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Basically, you are replacing %h with %a. Then do a rebuild of the apache config and restart the apache service.


# /scripts/rebuildhttpconfig

# /scripts/restartsrv_apache

Now, your changes will be saved permanently and you’ll get the real visitor’s IP on your log files instead of the server’s IP.

Update (August 2018) : With the release of cPanel 74.0.4 (EA4), you can directly change the Apache LogFormat from within cPanel/WHM.

To change your LogFormat, you can go to Apache Configuration > Global Configuration and replace %h with %a as shown below:

Once you have saved the changes, you can click on the Rebuild Configuration and Restart Apache button to have the changes come into effect.

You can then delete the /var/cpanel/templates/apache2_4/ea4_main.local file if you have created it before the update.

I have tested that the changes remain after Apache and Engintron updates. Do let me know in the comments if it doesn’t work.

Find and delete duplicate files in Bash

For the times that I need to find duplicate files on my linux server and delete them, I go through the following procedure.

md5sum

This command calculates and outputs the md5 checksum of a file. If two files are the same, they will have the same hash.

To, get the md5sum of a file, simply do the following:


# md5sum example.php

# 312e9f7d1d6600989f0d1ac8c72f1de7 example.php

In the above, 312e9f7d1d6600989f0d1ac8c72f1de7 is the md5 hash of the example.php file.

Now, find all files with the exact md5 hash, and store their filenames in a file.


# find /home/ -type f -exec md5sum {} + | grep 312e9f7d1d6600989f0d1ac8c72f1de7 | awk '{ print $2 }' > duplicates.txt

With the above code, we are finding all files that have the md5 hash 312e9f7d1d6600989f0d1ac8c72f1de7, and outputting the second column (which is the filename) to a file called duplicates.txt

Now we loop through the duplicates.txt file, and delete each file one by one.


for f in $(cat duplicates.txt);
do rm -f $f;
done;

How to setup cPanel WHM DNS clusters

Its a good idea to have different physical name servers for your websites for better redundancy. If you are using cPanel to host your websites, it is plain simple to setup a fully working DNS cluster.

In this tutorial, we’ll be setting up a DNS cluster in cPanel.

Besides your main cPanel server, you will need two additional servers or VPS. To lower the risk even more, we can setup these servers from different hosts or locations. As with cPanel, these should be running the CentOS operating system.
You can get cheap $5 per month ~~512MB~~ 1GB droplets from Digitalocean and install CentOS 6 on it.

Install cPanel DNSONLY

After you have setup your server, its time to install cPanel’s DNSONLY software. Its a watered down version of the normal cPanel/WHM software. You do not even get a DNS zone editor, but that’s fine. You won’t be editing any zones on these servers manually.

cd /root
mkdir cpanel-dnsonly
cd cpanel-dnsonly
curl -o latest-dnsonly -L https://securedownloads.cpanel.net/latest-dnsonly

cPanel states that you need a minimum of 768MB ram to install the software and if your server does not have that required ram, installation will quit with an error. However my DNS servers never use more than 200MB. Maybe it is required for the installation/compiling. Anyway, you can hack the installation process to accept your 512MB ram.

Run the bash script with the keep switch.


sh latest-dnsonly --keep

After running the above command, a new installd directory will be created inside the /root/cpanel-dnsonly directory. You need to open the install file inside and change the following to as below.

my $min_memory_rules = {
    default =&amp;gt; 256, # changed this to 256 from 768

After that, you need to run the below file in the same folder (/root/cpanel-dnsonly/installd)

./bootstrap-dnsonly

Setup DNS Cluster

Once the cpanel DNSONLY software has finished installing, you open your browser and go to

https://server-ip:2087

The username will be root and the password will be your server’s root password.

Once logged in, go to Clusters > Remote Access Key and generate a new key. This key will be used by your main server to access the DNS server.

Now login to your main cpanel server and go to Clusters > DNS Cluster. You should now click on Enable DNS clustering and add a new server to the cluster. Choose cPanel as the backend type. You need to enter your DNS server’s IP address and the remote access key that you copied earlier from your DNS server.

Tick Setup Reverse Trust Relationship and choose Synchronize Changes as the DNS role. With this setting, your main server will push changes to your DNS servers.

Repeat the process to add the second DNS server. Once that’s done, you can now disable BIND on the main cpanel server by going to Service Configuration > Nameserver Selection. It will not be used as a DNS server any more, but only push DNS records to the separate name servers in the cluster. This way, you will free up resources used by BIND.

You now have multiple DNS servers which automatically stays in sync with your accounts in the cpanel server.

OpenVZ Tips & Commands

Once you have finished setting up openvz virtualization on your server, below are few commands to control and manage your containers.

Show all containers on the node


#vzlist

CTID NPROC STATUS IP_ADDR HOSTNAME
120 203 running 123.123.123.12 vps1.nodedomain.com
110 101 running 123.123.123.13 vps2.nodedomain.com
150 79 running 123.123.123.15 vps3.nodedomain.com

Start container 110

#vzctl start 110

Stop container 110

#vzctl stop 110

Restart container 110

#vzctl restart 110

Enter container 120


#vzctl enter 120

Show all IPs assigned to containers

#cat /proc/vz/veinfo

120 0 11  123.123.123.12
110 0 182 123.123.123.13 123.123.123.14
150 0 51  123.123.123.15

Harden PHP on Cpanel Server with suPHP and open_basedir

If you run a cpanel server and provide hosting services to customers, you really need to take care that one customer should not be able to access another customer’s files. Not doing this will introduce you to a world of pain. Hackers, script kiddies and spammers are the scum of the internet and they do not care how much trouble they cause to others. Your customers may not be malicious themselves but they can also be unaware of their actions in regards to security.

With the popularity of CMS’s like WordPress, Joomla, its very easy to get a website/blog up and running in minutes. For the unaware customer, its also very easy to forget to upgrade their installations and install insecure themes and plugins. These are the main sources for hackers to gain access to their files and account on the server, and if the server is not properly secured, it only takes one customer getting his account compromised to enabling the hacker to access all other files on the server.

Fortunately there’s a way to prevent or at least make it harder for them to do that. We’ll be making use of suPHP and open_basedir.

suPHP

This executes php scripts with the permissions of the script’s owner instead of the user nobody.

To enable this, apache needs to compiled with mod_suphp. This can be done via easy apache in cpanel.

Inside WHM:

– Go to Software >> EasyApache (Apache Update)

– Click on a previously saved profile and customize it, choose your apache and php version. On the next page, make sure Mod suPHP is enabled/ticked.

– Click on save and build. This should take about 5 minutes. Apache is now compiled with suPHP.

easy-apache

open_basedir

This is a PHP directive found in the php.ini file that restricts the running php script to only certain file directories specified in the directive. For cpanel users, this can be set to the /home/, /usr/local/lib, and /tmp directories. To set the open_basedir, you can do it via WHM or from the command line:

Command/Shell:


# vi /usr/local/lib/php.ini

# open_basedir = "/home/:/tmp:/usr/local/lib/"

From WHM:

Go to Service Configuration >> PHP Configuration Editor
Check “Advanced Mode”, find the open_basedir directive and set the values

open_basedir

/usr/local/lib – This area stores the php.ini files and other configurations which would be needed by the running PHP script.

/tmp – This is default area where PHP stores temporary files like sessions, file uploads, etc. So the script would need access to it.

/home – This is normally where a user’s account is stored. If we want to restrict user to its own user directory (/home/username) then each user should have an independent php.ini file specifying the directory in the open_basedir directive.

Disable user’s custom php.ini

suPHP by default loads a user’s custom php.ini stored in their directory. This enables the user to override the global php.ini settings. A malicious user/script could override the global open_basedir value and the above protection will be lost. To prevent this, we have to ask suPHP to load only the main php.ini file:

Open the suPHP configuration file:


# vi /opt/suphp/etc/suphp.conf

And uncomment the following lines:


;application/x-httpd-php=/usr/local/lib/
;application/x-httpd-php4=/usr/local/php4/lib/
;application/x-httpd-php5=/usr/local/lib/

The above lines should read as below:


application/x-httpd-php=/usr/local/lib/
application/x-httpd-php4=/usr/local/php4/lib/
application/x-httpd-php5=/usr/local/lib/

Now, save the file and restart apache:


# service httpd restart

From what I have noticed, with the above settings, a user can still access another user’s files (/home/user2/), but since cpanel sets a 750 (No permission for others) permission on the public_html folder, it cannot access the other user’s files under public_html (where most of their files will be anyway). So, as long as users don’t change their public_html permission and store most of their files inside public_html folder, its an easier way than creating php.ini files for every single user. For me, it works, so its a decision you have to make yourself.

Setup OpenVZ on Centos 6 and create VPS containers

If you have a server and want to divide it into smaller servers (VPS) that can be used for different purposes, this guide will show you how to do that with OpenVZ.

OpenVZ is one of many virtualization systems (Xen, KVM, etc.) that can be used to partition your server into many Linux containers. It is free and very easy to setup. It can only be used to create Linux containers though.

Requirements:

SSH access to a physical server
At least 2 IP addresses (public if you want it to be accessible across the internet)

This tutorial assumes that you have already installed CentOS 6 on your main server and know how to edit files using vi. This will be called the node.

First, we have to add the OpenVZ yum repo so we can install the kernel using yum


# wget -O /etc/yum.repos.d/openvz.repo http://download.openvz.org/openvz.repo
# rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ

Install the openvz kernel using yum install.


# yum install vzkernel

Once the openvz kernel is installed, we have to boot into that kernel and not the centos 6 kernel. Select the boot order, set it to the openvz kernel by changing the default number to the one corresponding to the installed kernel.


# vi /etc/grub/grub.conf

On my system, I have the following contents in the /etc/grub/grun.conf file:


default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title OpenVZ (2.6.32-042stab108.2)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-042stab108.2 ro root=UUID=ae3594f2-6283-4171-a706-9ecb965fcfd9 rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 nodmraid SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_MD_UUID=dba406c1:4bde3b09:0272bd39:a5f39fcc rd_NO_LVM rd_NO_DM rhgb quiet
initrd /boot/initramfs-2.6.32-042stab108.2.img
title CentOS (2.6.32-504.16.2.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-504.16.2.el6.x86_64 ro root=UUID=ae3594f2-6283-4171-a706-9ecb965fcfd9 rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 nodmraid SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_MD_UUID=dba406c1:4bde3b09:0272bd39:a5f39fcc rd_NO_LVM rd_NO_DM rhgb quiet
initrd /boot/initramfs-2.6.32-504.16.2.el6.x86_64.img
title CentOS 6 (2.6.32-504.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-504.el6.x86_64 ro root=UUID=ae3594f2-6283-4171-a706-9ecb965fcfd9 rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 nodmraid SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_MD_UUID=dba406c1:4bde3b09:0272bd39:a5f39fcc rd_NO_LVM rd_NO_DM rhgb quiet
initrd /boot/initramfs-2.6.32-504.el6.x86_64.img

The default is set to 0, which is the first one in the list.

Next, install the openvz tools and utilities that we will use to control and manage our containers.


# yum install vzctl vzquota vzploop

Make changes to the /etc/sysctl.conf file


# vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.default.forwarding=1

With OpenVZ we have the option of two filesystems for our containers.

simfs – psuedo filesystem that maps the hosts files to the containers
ploop – One large file containing all container files

The default filesystem is set to ploop, but for ease of use, we’ll set it to simfs


# vi /etc/vz/vz.conf
VE_LAYOUT=simfs

Save the file and reboot into the new kernel


# reboot

Creating Containers

Now that we have installed the openvz kernel and rebooted into it, we will create our first container.

Select and download a pre-created template from the openvz template website and save it to the /vz/template/cache directory.

We will create a container with the following specs:

Container ID: 110
OS: Centos 6
Hostname: myfirst.vps.com
IP Address: 123.123.123.12
Nameserver (Google’s): 8.8.8.8 and 8.8.4.4
Total CPUs: 2
RAM: 4GB
Disk space: 50GB
SWAP (vswap): 2GB


# vzctl create 110 --ostemplate centos-6-x86_64
# vzctl set 110 --onboot yes --save
# vzctl set 110 --hostname myfirst.vps.com --save
# vzctl set 110 --ipadd 123.123.123.123 --save
# vzctl set 110 --nameserver 8.8.8.8 --nameserver 8.8.4.4 --save
# vzctl set 110 --cpus 2 --save
# vzctl set 110 --ram 4G --save
# vzctl set 110 --diskspace 50G --save
# vzctl set 110 --swap 2G --save
# vzctl start 110

Setting onboot to yes will make sure that the container is started automatically right after the host node is started.

We then set a root password for the newly created container.


# vzctl exec 110 passwd

Now, you can login to your container via SSH or by from the host node by


# vzctl enter 110

You can now enter your container and install anything and do anything as you please. All isolated from your main server.

With OpenVZ you can even change the resource limits of your container without rebooting it using the vzctl command. This can be very handy if you need to avoid any downtime when changing limits.

Below are the file locations of a default configuration:

/vz – main folder/partition storing all your container files
/vz/private/<Container ID>/ – the files of a particular container, identified by its Container ID
/etc/vz/vz.conf – the main configuration file for openvz
/etc/vz/conf/<Container ID>.conf – the configuration file of a particular container

Enjoy your new virtual machines!!