On Thursday I released an article detailing how to get Proxmox setup and also how to configure networking with IPv6. However that article got long and I just said I would address the firewall in the future. Well, that’s today because I need to get the configuration stuff written down before I forget. In addition to the firewall there are some other security house keeping items for a new proxmox install, that includes disabling the root account and using sudo and changing the default SSH port. So let’s go.

The base OS under Proxmox is Debian. Debian is great and it is lighter-weight than Ubuntu so I am all for using it.


If you are already somewhat comfortable with Proxmox and Debian configuration and just prefer I get to the point then click here to expand this section of text and skip reading the rest.

Here is the list of things to do to secure your ProxMox + Debian Node.

Setup Sudo and a New User in Debian.
1. Install the “Sudo” package in Debian.

2. In Debian, Add a new user account and then add the account to the sudo group. Test the account with SSH.

3. In the ProxMox Web Panel, create a new “admins” user group and give the group the “Administrator” permission.

4. In the ProxMox Web Panel, add the new user account you created in Debian to the admins group in the ProxMox panel. The logout and log back in to ProxMox and make sure the account works.

5. Finally, in Debian, disable the root account.


A bit of SSH Security
Change the default SSH port from 22 to something else.


Configure the ProxMox Firewall

Quirks and General Practice that Works
1. Proxmox Cascading Firewall rules don’t work as described.

2. Blank fields in Rules = “Any” or “All” in traditional firewall parlance.

3. Guest systems must have the firewall enabled on each guest nic interface in addition to enabling it generally.

4. The default policy (set in firewall options tab for Datacenter Tier and Guest Systems) should be set to “Accept” and a default drop rule should be added to every Rule chain on all tiers. Be sure to set the default input/output polices to “Accept” BEFORE enabling the firewall and be sure to all “Accept” rules for ports and services BEFORE adding the drop rule to each chain!

5. Treat each Rule Chain as if the order of the rules does matter because it seemingly does (which makes sense) on some of the chains, particularly the guest machine chains. In short, rules higher in the chain have precedence over those below them.

6. Firewall rules generally apply to BOTH IPv4 and IPv6 – you do not need separate rules for each.

Secure Config of the Firewall
Before enabling the firewalls or doing anything else.
1. Starting at the Datacenter tier, Create a Security Group called “proxmox” and add to it inbound accept rules for the following ports. TCP 3128, TCP 22, TCP 111, TCP 85, TCP 5900:5999, TCP 8006, UDP 5404:5505. Make sure all of these rules are set to “enabled” in the group.

2. Add the new “proxmox” security group to the rule chains for the Datacenter and Node Rule Chains. Make sure it is enabled.

3. Additionally, in the Datacenter and Node firewall chains, add an inbound rule for the port you configured for SSH and make sure it is enabled.

4. Add a “drop all in” rule to all Firewall Chains (Datacenter, Nodes, Guests) and do NOT set it to enabled. Drag it to the bottom of the chain first, then tick the box to enabled it.

5. On the Datacenter Firewall, go to the options tab, set the default input policy to “accept”, then enable the firewall.

6. On each Node Firewall, go to the options tab, enable the firewall.

7. On each Guest NETWORK tab, enable the firewall on every interface for the guest you want to protect.

8. On each Guest FIREWALL –> Options tab, set the default input policy to “accept”, then enable the firewall.

9. Add additional rules to the Guest Firewall chains to allow other ports IN.


IPv6
See notes at the bottom of this article if you are having issues with IPv6 and the firewall. This article assumes you have already read the previous article and successfully configured IPv6 networking for your Proxmox node(s).


That’s it! If you are struggling with any of the above, read the fuller article. This was the shorthand version for those wanting to save time.
————————————————–



However something introduced as a default into Ubuntu was the use of “sudo” to elevate a regular user account to root priviliges as needed. Additionally, this allowed for disabling the built-in root account by default (on Ubuntu systems). If you have a public server and you have SSH running on the default port of 22, check your auth.log sometime… The “root” account gets hammered by brute-force attempts all day long. So disabling it all together is a wonderful idea as is switching the default SSH port to something else to avoid script-kiddies roaming the internet for vulnerable boxes with which to play.

This isn’t just “debian” though and the more I work with Proxmox the more I am learning THIS IS PROXMOX. What do I mean by that? Well, disabling the root account has some implications, namely losing admin access to your Proxmox portal. So, before you disable the root account you need to not only setup a new account with SUDO privileges, but you also need to add that user as an admin in Proxmox. Let’s do all of that now…


Setting up a New User and Giving them Sudo Privileges

I will be setting up and account for fictional Jane Doe, eg. jdoe.

SSH to your Debian Install, then…

su
apt-get install sudo
adduser jdoe

Fill in all the prompts, you can just ignore full name and room number, etc… Once the user is created add them to the sudoers group.

adduser jdoe sudo

Then try to SSH to the server and make sure you can login as jdoe and once logged in make sure you can “sudo su” – i.e. elevate yourself to root. If that all works, continue…


Make the User a Proxmox Admin
In the ProxMox Web GUI.

1. On the left, click DataCenter –> Groups tab –> create –> Name: “Admins” –> create
2. Datacenter –> Permissions Tab –> Add –> Group Permission –> Path: / , Group: Admins, Role: Administrator –> Add
3. Datacenter –> Users Tab –> Add –> User name: jdoe, Realm: Linux PAM, Group: Admins –> Add

Now logout of the ProxMox Web-GUI and then log back-in as jdoe using the password you set in the shell earlier. If that all works and Jdoe has permission to do stuff across your node(s) and datacenter you are good to go.

Back in your SSH session… let’s go ahead and disable the root account on your server… Login to SSH using the jdoe account you setup.

sudo su
passwd -dl root

Great, now if you try to login with your root account it just plain won’t work. That goes for the proxmox web console as well.


Change Default SSH Port
Next, lets change the default port of SSH. DISCLAIMER – THIS ASSUMES YOU HAVE NOT ENABLED THE PROXMOX FIREWALL YET!!

I will be using the port 9022 as an example…

SSH into your server, then…

sudo su
vim /etc/ssh/sshd_config

This should open up your SSH configuration file in VIM for editing. Find the line that says “Port 22” and underneath that add another line “Port 9022” then save and close the file. The two lines should look something like this:

Port 22
Port 9022

After you have saved the file, restart the SSH service on the server.

service sshd restart

DO NOT close your SSH session window yet. Rather, open up a new SSH shell on the new port to ensure it works. If it does, great, from the new session window:

sudo su
vim /etc/ssh/sshd_config

This reopens the sshd config file for editing. This time, comment out the line “Port 22”. So the two lines should look like this:

#Port 22
Port 9022

Save and close and then restart the SSH service on the server again.

service sshd restart

Leave your current SSH session open and make sure you can still establish a new session on the new port and conversely cannot establish a connection on port 22.

Once that is all said and done and working well you should be good to go.

Configuring the ProxMox Firewall

As stated in a previous article, if you are using ProxMox, it IS your firewall. Don’t mess with UFW/iptables on either your proxmox host server or any of your guest systems. Everything most people need from a firewall can be configured via the GUI.

“Cascading Zones”
ProxMox divides the firewall into three “cascading” zones. The top-level is the data center. The middle tier is each Proxmox node in your cluster, and the final tier are the guest systems themselves. I call them “Cascading” tiers because rules in a higher-level tier (**are supposed to**) “cascade” down to all the tiers below… As long as those “tiers” have the firewall enabled. However this described behavior (in the official proxmox documentation) isn’t inline with my experience… See “quirks” below…

Unique Settings, Options, Etc. Per Zone
Configuration across the three different zones differs a bit so I wanted to talk quickly about what is unique in each “zone”

Datacenter – Define “security groups” (groups of ports/services) that can be used in all rules in all zones, define aliases in this zone, define IPSets in this zone. Define default input and output policies.

Node – Notable for NOT being able to define security groups, aliases, or IPsets.

Guest – Define Aliases and IPsets and default input and output policies. Must enable the firewall on each interface in addition to enabling generally.

Quirks
There are several of what I will term “quirks” regarding the Proxmox firewall that might throw you off if you are used to configuring other enterprise firewalls.

1. If a field in a rule is left blank that is the same as saying “any” or “all” except for the “Macro” field which is special…

2. Proxmox comes with predefined “Macros” – Basically instead of saying you want to ACCEPT ALL IN TCP 443, you say you want to ACCEPT IN and then select HTTPS from the Macro list and don’t fill in anything else. Macros aren’t really a quirk per se’. I just wasn’t sure how they worked or what they did without a bit of digging. Basically they provide a way to quickly apply rules, especially for lesser known service ports, without having to look up what you need.

3. Because leaving a field blank = “any” or “all” you can pretty much just define a direction, an action (ex. “allow”), and a Destination Port (ex. 80) and that makes a rule.

4. You will have all kinds of issues getting rules to work on Guest systems if you don’t enable the firewall on all the interfaces on the guest for which you want rules to apply. This really threw me for a loop because I had the firewall “enabled” on the guest but hadn’t explicitly enabled it on any interfaces which led to odd behavior. This ONLY applies to guest machines and not to nodes.

5. The default Input/Output policy REALLY confused me because I am used to just defining this in the rule chain. Default policies are configured under Guest –> Firewall Tab (up top) –> Options Tab (down below) –> Input Policy, Output Policy. I am understanding these to be a “default drop” or “default accept” unless there is a rule saying otherwise for a specific service.

5. While we are on the topic of the default Input Policy,Output Policy settings, I will also note that the ONLY thing that can supersede these defaults are rules on same Guest chain.

6. Which leads to point 6 – When you think all of the above through, it means cascading doesn’t seem to matter. If the default policy on a host is set to “Accept” then it overrides what I would assume are policies that would say for example “deny 22” but the guests default is to allow and that wins. Hence cascading doesn’t actually exist. Feel free to straighten me out if I am wrong on this.

7. The order of rules in an individual chain DOES MATTER on the guest machines. However for the Datacenter level firewall it is hit or miss. I am used to working with firewalls where a rule that is higher in a chain supersedes all rules below it. So, for example, if you you a policy to “DROP” all traffic to your host, every ACCEPT rule below it is null and void. That isn’t always the case with Proxmox based on my testing. Solution? Treat every chain like the order DOES matter and in doing so hopefully avoid unhappy surprises…

A Basic Secure Configuration
All that to say, at the end of the day we mostly just want a “basic” secure configuration so I will attempt to walk through that setup. Before I start though, a disclaimer (yay…)

I am not responsible for you jacking up access to your proxmox server. If you follow these instructions carefully and in order you should be fine. However I can’t emphasize enough the need for care. If you, for example, turn the firewall on and don’t adjust the default rules you will immediately lose all access to your box. So take care!

First, Configure the DataCenter level… (be sure to do these steps IN ORDER). In the Proxmox Web Console

1. Select Datacenter –> Firewall Tab (up top) –> Security Group Tab (down below) –> Create –> Name: proxmox –> create
2. You should now see a new security group called “proxmox” – click it.
3. Now start adding rules with the Rules button. Each rule consists of the following: Direction: In, Action: “Accept”, Tick enable, Protocol: TCP, and a destination port.
3A. There are a total of SIX TCP rules for this security group. Here is what should go in the destination port box for each: 3128, 22, 111, 85, 5900:5999, 8006
4. Still in the security group called “proxmox” add one more rule consisting of the following: Direction: In, Action: “Accept”, Tick enable, Protocol: UDP, and Dest.Port: 5404:5405.

You should now have a total of 7 rules in the proxmox group.

1. Datacenter –> Firewall Tab (up top) –> Rules Tab (down below) –> “Insert:Security Group” –> select “proxmox” and tick “enable” –> add

2. Still on the Datacenter-Firewall-Rules Tab –> Add (to create a new rule) –> Direction: In, Action: Accept, Tick Enable, Protocol: TCP, Dest.Port: 9022 (or whatever you set your SSH port to earlier) –> Add

3. Still on the Datacenter-Firewall-Rules tab, create a new policy with “add” –> Action: Drop, DO NOT TICK ENABLE –> hit “add” –> drag this rule to the bottom of the chain and do not enable it yet.

4. Still on Datacenter –> Firewall Tab –> Options Tab (down below) –> Set input policy to “Accept” —> Set Enable Firewall to “Yes”

5. Still on Datacenter –> Firewall Tab –> Rules Tab (down below) –> tick the box to enable the Drop policy we put at the bottom of the chain in step 3.

Okay, so your Datacenter firewall is now configured. Move on to your Node.

1. Node –> Firewall Tab (up top) –> Rules tab (down below) –> insert: security group –> choose proxmox and make sure it is enabled –> add in a rule for your SSH port that you set earlier and enable it. –> add in a drop rule but don’t enable it –> drag the drop rule to the bottom of the chain, then enable it. Once you are done your node rule chain should match datacenter rule chain.

2. Still on the Node –> Firewall Tab (up top) –> Options tab (down below) –> Set Enable Firewall to “yes”

Your node is now protected.

Finally your guest firewall(s).

1. Guest (100 is the first default guest) –> Firewall Tab (up top) –> Options Tab (down below) –> Set enable firewall to “yes”, Input Policy: Accept, Output Policy: Accept.

2. Guest –> Firewall Tab (up top) –> Rules Tab (down below) –> Add a default drop policy and enable it. –> Add in any other rules you need (ssh port, port 80, 443, etc.) and enable them. Make sure they are all above the drop policy in the chain.

3. Guest –> Network tab (up top) –> double-click on the network connection you want to protect –> ensure “Firewall” is ticked –> hit ok.

Your guest is now protected.

What is the thinking? By setting the default input policies in the “options” tabs at the Datacenter and Guest tiers to “accept” and then manually adding a “drop all” policy to the bottom of each we pretty much are now managing the firewalls all individually. I went this route because I can see what is going on and Cascading just didn’t seem to actually happen correctly (or, if ever) and seemed best to just avoid it.

The documentation for the Proxmox firewall, imho, really just sucks. I burned a lot of hours and brain cells only to come to the conclusion that Cascading doesn’t really happen. Going this route allows me to feel comfortable because it dependably works. I know the ports I want to be closed are closed and the ones I want open are open.

What about IPv6

If you followed my previous article and setup IPv6 correctly then you should be all set. This was all done on Proxmox 4.1 which has full support for IPv6 and most of the kinks seem to be worked out. There is one caveat if you are having trouble with the firewall and think it is related to IPv6, namely you need to have a loopback address for IPv6 defined on your node. It should be defined by default if you setup IPv6 properly. If you want to quickly check then SSH into your node and:

ifconfig lo

If you see “inet6 addr: ::1/128” then you have an IPv6 loopback address setup.

The firewall TCP and drop rules all apply to both IPv4 and IPv6, you do not need to setup separate rules to apply to one or the other.

1 of 1

19 comments on: Secure Proxmox Install – Sudo, Firewall with IPv6, and more – How to Configure from Start to Finish

  1. luisv
    Reply

    Thanks nbeam, I’m reading yours posts to applicate this on my first OVH Proxmox server.

  2. Benni
    Reply

    Hi, changing default SSH port to nodes in a cluster makes issues with migration.


    # /usr/bin/ssh -o ‘BatchMode=yes’ [email protected] /bin/true
    ssh: connect to host 192.168.10.10 port 22: Connection refused
    ERROR: migration aborted (duration 00:00:00): Can’t connect to destination address using public key
    TASK ERROR: migration aborted

    For a cluster, I thing it’s better to install Fail2Ban and keep default SSH settings.

    • nbeam
      Reply

      Absolutely correct 🙂 – If you are clustering, keep the SSH port the default and use either an ACL or software like Fail2BAN which I am a huge fan of.

      • BadAss

        Or add -p to your command like: ssh -o -p 9022 ‘BatchMode=yes’ [email protected] /bin/true

      • nbeam

        I am unfamiliar with this. What exactly does setting it up like this do? I appreciate the tip as I am always learning more about Proxmox and linux in general. -Thanks!

    • Gilberto
      Reply

      Yep! But in this case, you can run ssh only in one IP, that one in private network, where all cluster traffic run…
      Perhaps something like a second NIC with different IP rather your local network!
      So, now you can leave the SSH unchanged, and connect to SSH of your proxmox server, with noVNC inside web interface…

  3. Flowz
    Reply

    Hey NBEAM, I don’t understand your tutorial about firewall in proxmox. You don’t say which port is for what, or why we should do what you do in the examples so I can’t follow it but that’s the ONLY tutorial available 🙁

    Seriously, I’m thinking about sticking with my 5 lines iptables rules which work perfectly.

    • nbeam
      Reply

      Port numbers are fairly standard. Port numbers specific to proxmox are also documented. I changed from default ports to non-default so my systems don’t get hit by roving bots on the internet. If you leave port 22 exposed publicly for example and then check your systems auth log you will see all kinds of brute-force attempts against the system. Hence changing it to a non-standard port tends to help because a lot of those automated systems are looking for standard ports and aren’t fingerprinting your entire system.

      As far as some of the less common ports mentioned, they were mostly taken directly from Proxmox documentation:
      3128 – Is listed as the SPICE Proxy/Console and I am guessing this has to do with Console connections through the web GUI to guest systems
      22 – SSH – This is for SSH access to the box and if you use a non-standard port you can change it
      111 – RPCbind – Not sure what this is for but it is mentioned in the Proxmox documentation
      85 – Has to do with the PVEdaemon service
      5900 – 5999 – These are important for me but this is the default VNC ports for remote desktop connections to some of my VM’s/containers
      8006 – Default Web Management GUI port for Proxmox

      Firewall management can be a headache but it is worth the time and effort. If your 5 line IPtables rules essentially opens the server wide up (I don’t know if it does), yes, everything will work, but your security is seriously compromised. The idea is to “deny all” by default and then only open up very specific ports.

      • Flowz

        No, my rules were blocking everything but ssh and apache and two custom ports.

        Thanks for the list of ports. I managed to make the PVE firewall works but that was not easy (so I dumped my iptables rules).

        I can now read the logs (which is awesome) and I realize how many requests on all my ports are done, it’s crazy. I had no firewall for few days, but I’m confident.

        By the way, I’d like to ban these tards scanning my server with fail2ban or something but apparently the only thing we can find about fail2ban and proxmox is about protecting the webgui port. Nothing for apache or to ban all the guys scanning ports (dunno if that would be very useful though).

        PS: sorry double post

      • Tommy Peanuts

        Great guide! It helped me finally to work out my PVE setup. BTW I created an at job to run “pve-firewall stop” 10 minutes after I started changing my firewall settings. That way, if I screwed up, I could get back in!

        Also BTW, since PVE 4, VNC Web console has been retired, so you don’t need 5900-5999 open any more. (https://pve.proxmox.com/wiki/Ports). And although it says portmapper on 111 is needed, it seems from the forums that you can block that.

      • nbeam

        My “real” IT job has been keeping me slammed lately 🙂 so I apologize for replying so late. Thank you for the updates. I am glad they dropped the VNC console. I never had much luck with that. I like what you did with the firewall job that just runs 10 minutes after making changes and kills the FW. Tricks like that are smart/handy when working with remote servers you will never have physical access to. Cheers

  4. Flowz
    Reply

    No, my rules were blocking everything but ssh and apache and two custom ports.

    Thanks for the list of ports. I managed to make the PVE firewall works but that was not easy (so I dumped my iptables rules).

    I can now read the logs (which is awesome) and I realize how many requests on all my ports are done, it’s crazy. I had no firewall for few days, but I’m confident.

    By the way, I’d like to ban these tards scanning my server with fail2ban or something but apparently the only thing we can find about fail2ban and proxmox is about protecting the webgui port. Nothing for apache or to ban all the guys scanning ports (dunno if that would be very useful though).

  5. Will
    Reply

    time for a proxmox 5 guide….

  6. Gilberto
    Reply

    Hi! I think that the ‘cascading’ effect is more related to Cluster. I meant, when u set up a rule into servers that composer a Cluster, that rules cascade to entire Cluster!
    Nice job with this article…

    • nbeam
      Reply

      In my case I have primarily only worked with standalone boxes and that is where I witnessed the behavior.

  7. Barry Docherty
    Reply

    This is actually quite good. I never used to use PVE-Firewall instead opting for iptables on host and UFW/Firewalld on guest systems manually (Double the trouble but worked fine).

    However, setting the pve-firewall to a NAT guest system cuts it off from internet and internal networking. So I am thinking that the host system is protecting the NAT guest. Would I be correct in thinking so?

  8. Thomas Kelly
    Reply

    To put it simply: thank you! I completely agree with the firewall docs – terrible. Sparse. Not entirely accurate? This little write-up was precisely what I needed. Much appreciated.

  9. Don McC
    Reply

    I’ll second all of the positive response to this write up – I have been trying to wrap my wits around the PVE Firewall for years without success – until I read this – so THANK YOU !!!
    One note is that After I secured the GUI and started logging in as a real user rather than root I lost the ability to run updates via the “Node/Updates” page – is there something I missed with that ? (I can always run it from the CLI – after running “sudo su -” but it would be nice to be able to use that GUI functionality to run my updates.)

  10. Isrial Pritchard
    Reply

    Hi Kiloroot,

    Love this tutorial and so happy that my system is secure now. I feel much safer especially as I have Ubiquiti devices and they had security issues at the beginning of the year!

    Can you suggest what to do about this?

    I have rebooted my Proxmox and getting the following:

    Reloading system manager configuration
    Starting default target
    [49664.495677] FAT-FS (sdb)” bogus number of reserved sectors
    You are in emergency mode. After logging in, type “journalctl -xb” to view sysmte logs, “systemctl reboot” to reboot, “systemctl default” or “exit” to boot into default mode.

    Cannot open access to console, the root account is locked.
    See sulogoin(8) man page for more details

    Press enter to continue.

    And when I do I get exactly the same output.

    I have tried to ssh in remotely to with the Admin account we made in the new step

    ssh [email protected]
    ssh: connect to host 192.168.10.19 : Connection refused

Leave a Reply to Barry Docherty Cancel reply

Your email address will not be published. Required fields are marked *