After beating my head against the wall over the course of many hours I have finally figured out how to get Proxmox working quite well on my cheap KimSufi server… with IPv6.
The goal of this article is to document (with varying levels of detail) how to go from a fresh KimSufi, OVH, or SoYouStart server to a running standalone Proxmox node with both IPv6 public networking as well as an internal VM/Container network.
Before we get going, here are my server specs if you are looking to get a comparable system…
It is the equivalent of a KS-4C hosted in France:
CPU: Intel(R) Core(TM) i5-3570S CPU @ 3.10GHz – Quad-Core – No HT
RAM: 16 Gb
Bandwidth: 100 mbit – No Transfer Limit
Address Space: 1xIPv4 and an IPv6/128**
Storage: 1 x 2 TB SATA
You can definitely run Proxmox on a lower spec system but I wouldn’t recommend anything less than a Core-i3 w/ 8 GB of ram.
Why is IPv6 such a big deal? Simply put they tell you that you have a /128 (1 address) but in fact you have an entire /64 subnet available to you. That is equal to 18,446,744,073,709,551,616 addresses… With a server as small as what I have got, I will be happy if I use more than 5… Having the extra IP space available means we can host multiple VM’s or containers and they can each have one or more public IP addresses.
Why ProxMox? Simply put, it lets you (somewhat) easily manage Linux Containers (LXC) side by side with full-fledged VM’s running on KVM. That means you can virtualize anything, including Windows, and also opt for Containers when your guest machine is going to be Linux for maximum utilization of your hardware and the best guest performance. All that from one interface.
KimSufi provides Proxmox 4.1 as one of their preinstall options out the gate and it is almost completely up-to-date. From here on out I am going to be discussing how to get going with a fresh install of Proxmox from KimSufi. Let’s go!
Getting Started – Server Housekeeping
1. KimSufi sets up SSH for you from the outset which is a blessing. We need to make some configuration changes immediately, so SSH into your server using putty and login as root. Kimsufi should have provided you credentials via email. Proxmox 4 runs on Debian Jessie so things are a little bit different from Ubuntu Server but not much.
2. Once logged in, the first thing we need to do is update our repositories (assuming you are using ProxMox for free and not paying for a support license). I also wanted to change my root password and do some other housekeeping.
passwd root #if you want to change your root password
apt-get install vim (if you want to use VIM to edit files, it is my preferred editor and I will be using it throughout)
mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak
echo "deb http://download.proxmox.com/debian jessie pve-no-subscription" > /etc/apt/sources.list.d/pve-public-repo.list
What this did is make sure that when we run apt-get update the system doesn’t try to pull from the enterprise repos which are only available if your box has a paid subscription. It is perfectly legal to use the box however you want without a subscription, the only difference is the community repos are advertised as not being as good as (stable? up-to-date?) the enterprise ones. Additionally, above, I also ran apt-get commands to update the repo info and upgrade they system. If you don’t run a dist-upgrade there is a bug present that won’t allow you to create containers.
Finally, the “pveam update” command just updates your list of downloadable container templates with the latest from Proxmox. This occurs automatically once a day but the command can be used to manually trigger it at any time.
Next, one of the things I find highly useful is internal networking between guest systems. The easiest way to do this (bonus round section at the end) is with a Proxmox package called openvswitch. It is fully supported but you need to install it. IF you want to do this, I recommend installing it now…
Login to your Proxmox Panel
Time to get into the Proxmox GUI. Open a browser and go to:
Substitute your public IP in place of the fake one provided above. You can login with the user “root” and whatever password kimsufi provided or whatever you set it to if you changed it earlier.
If you are familiar at all with just about any other virtualization platform, getting your sea legs in Proxmox is pretty easy. Proxmox organizes your install into three levels: the Datacenter (top level), Nodes (second level, server(s) that run VM’s and containers), and the guest systems (VM’s and containers) running on each node (third level). You can navigate to the different tiers using the area on the left. For this tutorial we are dealing with only a single-node setup and I will have us create a single VM.
Rant About Lack of Documentation
Proxmox documentation on the GUI is sparse, documentation on networking and firewall I found to be confusing and poor, and documentation on using IPv6 is the worst of all. A large part of why I am writing this article is to condense all of my research regarding the use of IPv6 with Proxmox into one location. It was a LOT of research work that I don’t want to repeat in the future. To be fair, I don’t take issue with the lack of documentation, after all it IS an enterprise-grade product that is completely and totally FREE.
Getting Started with Proxmox
The first thing you probably want to do is create a container to play with. However before we get ahead of ourselves we need to do all the prep work. That starts with creating a “pool” for the container to sit in…
Create a pool…
1. On the left, click “datacenter” at the top.
2. Click the “Pools” tab
3. Click “Create”
4. Enter a name (ex. defaultpool)
5. Click “create”
Create a storage location for container backups…
1. SSH into your server again then:
2. In the GUI, Go to Datacenter –> Storage Tab –> Add –> Directory
3. ID: Backups, Directory: /pve/backups, Content: select only “VZdump backup file” and deselect everything else, increase “max backups” to like 10 or more depending on how many backups you want to store for each VM/container –> click “add”
Download a container template
In the GUI, on the left, select “local” (which is the default storage object already created on your node) –> content tab –> Templates button –> select a template (I went with “Ubuntu 14.04-standard”) –> click Download
Container templates are the base install template for a new container. You need at least one to deploy a new container.
Configure Networking – Round 1 – About Proxmox Networking
Networking configuration and firewall configuration are DANGEROUS if you are working on a remote server with no direct KVM (Keyboard Video Mouse) way to console into the box. Make the wrong move and you will lose all remote connectivity to the server and have to possibly start all over again after reinstalling the OS (KimSufi offers a rescue boot that could save your tail but you don’t want to go down that route if you can avoid it…). As a somewhat humorous and perhaps sad aside, I had to rebuild my initial box like four times on another host. At least one of those times was related to making a mistake either with the network or the firewall…
Okay, that disclaimer is out of the way…
About Proxmox Networking and Firewall
To state it simply, I found neither the network nor the firewall to be simple. They actually aren’t bad once you figure out the idiosyncrasies but figuring those out is well… bad (see rant about lack of documentation above). I am neither a network admin nor a Cisco guy but I am also not a complete ignoramus as I have worked with enterprise firewalls and switches for over 5 years now. In short I have had to do configuration on all manner of firewalls, security appliances, switches, etc. and I still found this to be a bit of a challenge. On the bright side I sharpened my *nix networking skills substantially.
Anyhow, what you should KNOW is that Proxmox IS NOW your firewall. You will not be configuring a firewall in any of your VMs or containers and doing so within a container could have unexpected consequences. Proxmox is ALSO now your NETWORK configuration tool. Aside from the occasional small change in a guest system, you should not be modifying a container’s network configuration directly.
I will only be talking about setting up networking in this article and will leave the firewall and some other security bits and pieces for the next write-up.
Okay, Proxmox networking… When you install Proxmox it takes over your physical NIC(s) and uses it to create what I will call VNICs. If you have worked in Hyper-V and/or many other virtualization platforms you are familiar with this concept. The default result is a single “bridged” VNIC called either vmbr0 or vmbr1.
Bridged networking keeps things simple in that when you create a container and connect it to your bridged interface it can have an IP address in the same range as the Proxmox host server. On a public server that means you can assign a public IP in the same block as your physical server. This makes life easy because we don’t have to bother with NAT, routing, etc. This makes life difficult because cheap hosts (like KimSufi) only give you one IP address to work with and that is already taken by your host machine.
We are going to start by configuring vmbr0 and because this is cheap hosting and we only have one IPv4 address, we are going to be adding that large IPv6 range for our guest machines to use.
In the Proxmox Web Panel:
1. On the left click the icon for your Proxmox server node (ex. NS32958) –> Network Tab
2. You should see at least two network connections listed, eth0 and vmbr0 (or vmbr1, I can’t remember what the system defaults to). If you system has multiple physical nics you will see additional ethX nics listed. I don’t think Proxmox creates additional vmbrX interfaces for each but I may be wrong.
3. Double-Click on vmbr0
4. You should see an IPv4 address assigned already (your server’s public IP) along with a subnet mask and gateway. This is all for IPv4. We want to add an IPv6 address to this interface as well.
Configure Networking – Round 2 – Add an IPv6 Address
You should still have the dialog box open for vmbr0 and the goal is to enter the correct info into IPv6 Address, Prefix Length, and Gateway (the empty one at the bottom).
If you have a server with KimSufi, login to your KimSufi Dashboard and click “IP”. On this page you should see an IPv6 address that has been assigned to your server. This is actually the first IP in a very large range, all of which you have access to use. For my example, let’s say my IPv6 address is 2001:41d0:1:4462::1. That goes in the IPv6 Address field for vmbr0 in your Proxmox Panel.
Next, the Prefix Length for Kimsufi is “64”
Finally, you need to figure out your gateway which is very easy.
1. Take your IPv6 address – 2001:41d0:1:4462::1
2. Strip off everything from and including the last two characters before ::1 at the end. Which gives you this – 2001:41d0:1:44
3. Add FF:FF:FF:FF:FF (that is FIVE sets of FF) which gives you this – 2001:41d0:1:44FF:FF:FF:FF:FF
Enter your gateway address into the bottom empty Gateway field for vmbr0 in Proxmox. Click “OK” which will close the dialog box. You should see your changes reflected on the network tab for the node. However those changes aren’t active until after a reboot. With the node still selected on the left, click “restart” in the upper-right hand corner to reboot the server.
Configure Networking – Round 3 – Add a Permanent Network Route to Your IPv6 Gateway
Okay, we aren’t quite done yet. Proxmox did most of the work HOWEVER the Gateway IP for KimSufi is in a different subnet from your IPv6 IP address. That means that when packets try to leave your system they go for the gateway but they don’t know how to get there. So we need to tell them how by adding an IPv6 route on your server.
1. SSH into your Proxmox host server again.
2. Now that you are in we are going to write a quick script that will run everytime your vmbr0 network interface becomes active.
This creates a new blank file to put our script in and opens it in VIM for editing. Here is the script to put in the file (you will need to change the IPv6 gateway address to the one you figured out above):
# Check for specific interface if desired
[ "$IFACE" != "vmbr0" ] || exit 0
# Adding additional routes on connection
ip -6 r a 2001:41d0:1:44FF:FF:FF:FF:FF dev vmbr0
ip -6 r a default via 2001:41d0:1:44FF:FF:FF:FF:FF
UPDATE 3.17.2016 – I started noticing that my containers using IPv6 would suddenly lose IPv6 connectivity after a few hours. After some digging it became apparent that the default route to the gateway just disappears from the IPv6 routing table. After manually running the following command inside each container “ip -6 r a default via 2001:41d0:1:44FF:FF:FF:FF:FF” connectivity would be restored and it would stay… I am not sure if that will survive a reboot of the container though so I would recommend placing the script above in your containers as well and modify “vmbr0” to “eth0”. That way when you do reboot the system should run those route-add commands from the start. I am not sure without some more testing if this is a good fix or not. The other option might be to set a cron job to just run the script every hour otherwise…
3. Save and close the file. Then do the following to make it executable and finally restart the network stack:
NOTE: You can use ‘ip -6 route show’ to see if the routes were added to the IPv6 routing table.
4. Now you can try to ping your server from your home system to see if it responds. If so, you are golden. ex. ping 2001:41d0:1:4462::1
Configure Networking – Round 4 – Deploy a Container and Configure IPv6 networking
Almost through… By this point, you should have created a resource pool, downloaded a template to work from, and the IPv6 part of your network is now primed for action…
1. In the Proxmox Web Panel, click on “Create CT” in the upper-right-hand corner.
2. Give your container a hostname (ex. mycoolsite.com)
3. Select the resource pool you created earlier (ex. defaultpool)
4. type in a root password for the container, twice.
5. Click next.
6. Select the template from the list (ex. Ubuntu-14.04-standard)
7. Click next.
8. Give it some disk space (20 GB?)
9. Tell it how many cores worth of CPU it gets. The CPU Units field is fine at default. It is completely relative to every other VM/Container you create. A higher CPU Unit means the VM/Container has higher priority than other VM/containers with lower CPU unit value.
10. Click next.
11. Assign it some memory and at least the same amount of SWAP space as memory (2 GB?)
12. Click next
13. Finally the networking tab, I am going to break this into a separate list of steps…
Networking the Container:
1. You can leave the ID as net0,
2. The name eth0 can also be left alone (this is the name of the interface inside of the container/vm)
3. Make sure Bridge is set to vmbr0
4. Leave VLAN TAG empty
5. Leave the firewall unticked (for this article at least)
6. Okay, IPv4 – I am working off the assumption you want to stick with IPv6, so leave IPv4 set to static and the fields for it blank.
7. Leave IPv6 set to static
8. For IPv6/CIDR: you need to use the NEXT IPv6 address in your range. Going off of our IPv6 above, that means I would fill in 2001:41d0:1:4462::2/64 (I could also use, 2001:41d0:1:4462::3/64 or 2001:41d0:1:4462::4/64 etc.)
9. For Gateway (IPv6): you will use the same gateway you figured out above, for our example that is: 2001:41d0:1:44FF:FF:FF:FF:FF
10. Click Next.
11. For DNS domain, you can your server name if your want or something random. (ex. mycoolsite.com)
12. For DNS servers we can use Google’s Public IPv6 DNS server
12a. DNS Server 1: 2001:4860:4860::8888
12b. DNS Server 2: 2001:4860:4860::8844
12c. You can leave the third entry blank, then hit next
13. Confirm everything and then hit “finish” and wait while your container builds. Close the dialog box once the task completes.
Start the container…
1. In the Proxmox webgui, on the left, select the new container that should have popped up once the task completed.
2. In the upper-right-hand corner click “start” to start the container and wait a minute while it fires up.
3. From your home desktop computer, try pinging the IP of your new container and see if it responds. ex. ping 2001:41d0:1:4462::2
If you get a response you have accomplished much!
Configure Networking – Bonus Round – Setup and Internal VM network with OpenVswitch
This is the bonus round. While not required, it is very nice to have an internal private network if you have multiple guest systems. Especially if you want to do interesting things, like running MySQL in one container and Apache in another. Here is how to quickly setup a private network…
Create the network:
1. In the Proxmox WebGUI, Select your node on the left –> Go to the Network Tab –> Create –> OVS Bridge
2. Name: vmbr1, IP address: 192.168.1.100, Subnet Mask: 255.255.255.0
3. Make sure “AutoStart” is ticked and leave everything else blank.
4. Click “create” – The changes should reflect and then you need to click “restart” to restart the node/server.
Connect a container:
1. After the reboot is finished, in the Proxmox WebGUI, select one of your containers on the left.
2. Click “Shutdown” to turn the container off. Wait for it to finish shutting down.
3. With the container still selected, click the “network” tab –>t Add
4. ID: Net1, Name: eth1, Bridge: vmbr1, IPv4/Cidr: 192.168.1.10/24, Gateway (IPv4): LEAVE BLANK
5. Including the IPv4 gateway, leave everything else blank.
6. Start your container.
You can follow the above steps on another container and just increment the IPv4/CIDR address to 192.168.1.11/24 and now these systems can talk to each other on this private network!
Hopefully this is a huge help to everyone aspiring to use Proxmox and figure out the networking piece, particularly the IPv6 bit of it.