Linux-VServer Errors and Solutions

Published 8 February 2020

A collection of problems I encountered while running Linux-VServer guests, together with the workarounds and solutions that worked in my installations.

Contents

• Some VServers cannot access the Internet
• SSH or su login is slow
• Errors when starting a 32-bit guest on XFS
• Subversion clients report strange errors
• /proc gets unmounted and the host breaks

Some VServers cannot access the Internet

Assume that several public IP addresses have been assigned by the ISP and are used by different VServer guests. In some situations one or more guests, and sometimes even the host, may suddenly be unable to reach the Internet.

One possible cause is a stale ARP entry in the upstream gateway. This is especially likely after moving a guest from one physical host to another, because the IP address remains the same while the MAC address changes.

If the server is behind a Ubiquiti firewall

Log in to the firewall using SSH and send an ARP announcement:

sudo -s
arping -s <vserver-guest-ip> <isp-gateway-ip>

If the host is directly connected to the Internet

On a Debian host, I used:

arping -S <vserver-guest-ip> -B

This announces the IP address again so that upstream equipment can update its ARP cache.

SSH or su login is slow

I encountered long delays when logging in through SSH or when using su inside some guests.

The problem could be fixed in the guest with:

pam-auth-update

and then disabling:

elogind Session Management

Errors when starting a 32-bit VServer on XFS

I encountered a particularly confusing problem after setting up a new host with a large mechanical RAID array and an internal microSD card. Debian Buster itself was installed on the microSD card, while most write-heavy data was moved to the RAID array.

The disk layout looked approximately like:

Device        Start         End        Sectors    Size Type
/dev/sda1      2560      1953279       1950720  952.5M Linux swap
/dev/sda2   1953280     22924799      20971520     10G Linux filesystem
/dev/sda3  22924800  11720797326   11697872527    5.5T Linux filesystem

Moving /var away from the flash device

To reduce writes to the microSD card, I moved /var to the 10 GB ext4 partition:

init 1

mkfs -t ext4 /dev/sda2
mount /dev/sda2 /mnt/var

cp -apx /var/* /mnt/var

echo "/dev/sda2 /var/ ext4 defaults 0 0" >> /etc/fstab

umount /mnt/var
exit

The XFS problem

I initially formatted the large partition as XFS and mounted it as /vservers:

# This was the problematic configuration:
echo "/dev/sda3 /vservers xfs defaults 0 1" >> /etc/fstab

modprobe -v xfs
mkfs.xfs -f /dev/sda3
mount /dev/sda3 /vservers

I then transferred existing guests using rsync:

apt-get install rsync

cd /vservers

rsync -aPxvze "ssh -c aes128-ctr" \
      --whole-file \
      --stats \
      --numeric-ids \
      remote-vserver:/srv/* .

When starting an old 32-bit guest, several applications failed in apparently unrelated ways.

For example, cron reported:

cron: '/var/spool/cron' is not a directory, bailing out.

Apache also failed, and apt-get update complained that directories did not exist even though they clearly did.

Finding the cause with strace

With help from Bertl_oO and Guy on IRC, I traced the problem using strace.

A call to stat64() returned an inode number such as:

st_ino=4832143994

That inode number requires more than 32 bits:

log2(4832143994) = approximately 32.17 bits

In other words, 33 bits are required to represent it. The 32-bit guest was not handling the large inode value correctly.

As a quick test I moved the guest to an ext4 filesystem. It then started and operated normally.

Solution

One solution for XFS is to force inode allocation into the 32-bit range using the inode32 mount option:

/dev/sda3 /vservers xfs defaults,inode32 0 1

Alternatively, using ext4 for old 32-bit VServer guests avoids this particular problem.

Subversion clients report strange errors

I did not document this problem as thoroughly, so this section should be regarded mainly as a diagnostic hint.

After installing a newer Buster VServer kernel, I started an old guest that hosted a Subversion server through Apache, SSL and mod_dav_svn.

Subversion clients then produced unusual errors when committing. I suspected that this might also have been related to running an old 32-bit guest on a 64-bit XFS filesystem.

Rather than continue debugging the old guest, I rebuilt the SVN server using a fresh Buster guest. The problem disappeared.

Basic SVN server installation

apt-get install subversion libsvn-dev subversion-tools

mkdir -p /var/www/svn
cd /var/www/svn

svnadmin create --fs-type fsfs /var/www/svn/test-repo

chown -R www-data:www-data /var/www/svn
chmod -R 755 /var/www/svn

apt-get install apache2 apache2-utils libapache2-mod-svn

a2enmod dav
a2enmod dav_svn

I then edited:

/etc/apache2/mods-enabled/dav_svn.conf

and added:

<Location /svn>
    DAV svn
    SVNParentPath /var/www/svn

    AuthType Basic
    AuthName "SVN test Repo"
    AuthUserFile /etc/svnusers

    Require valid-user
</Location>

Then:

systemctl restart apache2

htpasswd -cm /etc/svnusers svn1

Adding SSL

For this test system I generated a self-signed certificate:

openssl req -x509 -nodes -days 3650 \
  -newkey rsa:2048 \
  -keyout /etc/ssl/private/apache-selfsigned.key \
  -out /etc/ssl/certs/apache-selfsigned.crt

I used an Apache SSL configuration along these lines:

SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On

Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff

SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
SSLSessionTickets Off

The SSL virtual host used:

<IfModule mod_ssl.c>
    <VirtualHost _default_:443>

        ServerAdmin your_email@example.com
        ServerName server_domain_or_IP

        DocumentRoot /var/www/html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        SSLEngine on

        SSLCertificateFile \
            /etc/ssl/certs/apache-selfsigned.crt

        SSLCertificateKeyFile \
            /etc/ssl/private/apache-selfsigned.key

    </VirtualHost>
</IfModule>

Finally:

a2enmod ssl
a2enmod headers
a2ensite default-ssl
a2enconf ssl-params
apache2ctl configtest

/proc gets unmounted and the host goes bananas

This was one of the more serious Linux-VServer problems I encountered on Debian Buster.

With the wrong util-vserver version installed, starting a guest could produce output such as:

vserver buildbuster642 start

Using makefile-style concurrent boot in runlevel 3.

ERROR: could not open /proc/stat: No such file or directory

rsyslogd: imklog: cannot open kernel log (/proc/kmsg):
No such file or directory.

vshelper.init: can not determine xid of vserver 'buildbuster642'

An error occurred after executing the vserver startup sequence.

At this point /proc on the host had actually been unmounted.

A temporary recovery is:

mount -t proc none /proc

Unfortunately, /dev and /sys could also be affected.

Check /run and /var symlinks

The following links should exist:

/var/run  -> /run
/var/lock -> /run/lock

and this directory should exist:

/run/lock

Guest context remains alive

After the failed startup, attempting to start the guest again may result in:

vcontext: vc_ctx_create(): File exists

This happens because processes from the failed guest context are still running.

First inspect the contexts:

vserver-stat

For example:

CTX   PROC    VSZ    RSS   userTIME   sysTIME   UPTIME
2000     4  157.9M     0    0m00s00   0m00s00   21m16s13

Note the context ID, in this example 2000.

Processes within that context can be listed with:

chcontext --xid 2000 -- ps -ax

For example:

PID   TTY   STAT   TIME   COMMAND
1     ?     Ss     0:01   /sbin/init
9326  ?     Ssl    0:00   /usr/sbin/rsyslogd
9335  ?     Ss     0:00   /usr/sbin/cron

I terminated the remaining guest services with:

chcontext --xid 2000 -- kill 9335 9326

After that, vserver-stat no longer showed the guest.

I then rebooted the host to ensure that /dev, /sys and the other system mounts were restored correctly.

The permanent fix was to install the working util-vserver packages and patch described in:

Installing Linux-VServer on Debian Buster .

Related articles

• Installing Linux-VServer on Debian Buster
• Installing Pi-hole as a Linux-VServer guest on Debian 10