Just finished Part 3 and I'm exhausted collecting all the information you put in the 3 videos. Really fabulous work, you must have been 10 times more tired than I am after creating them. Now the fun part is building the Proxmox OPNsense VM's and getting them configured. I have a TP-Link ER605 router that is going to be my ISP router interface that will connect to my cable modem which I've have already tested with an OPNsense trial VM so it looks like I have all the pieces to move forward tomorrow. Thanks again for all the hard work you invested to make my project successful. Jim
From one Jim to another, you're welcome! You might want to check out my latest OPNSense video (check the playlist). It does HA with only a single machine, it's much simpler!! Unless you're set on having 2 discrete devices.
I had a cluster with 3 nodes which did a nice job moving VM's so I decided to try removing a node and I was intent on re-establishing the node as an experiment just to see how involved it was. Following some documentation I used the kill command which removed the node but after the PVE reboot all that restarted was Debian and Proxmox wouldn't restart. So I had to reinstall PVE and of course lost the entire configuration. Also the other two nodes lost their NFS share to a bare metal TrueNAS server and I had trouble connecting to the nodes individually. Because of time limitations I'm using two nodes for now and went with your OPNsense video to get HA on two nodes. I figured I'd try three nodes in a cluster when some projects get completed this fall. Ceph really interests me and I see you have a video on the subject. I'm sure after watching it I'll get the bug to create a cluster again. Thanks again for your hard work.
Once again I have to say thanks. After crabbing about how I trashed my cluster you got me thinking so I went back into my notes and found my mistake. My age is affecting my memory but you suggestion got me motivated to readdress doing a cluster again. Good thing I'm retired you have so many interesting videos to watch and learn.
Really cool Jim, every single one of your videos is relevant to different things I'm implementing in my homelab. Keep it coming! I've had a lot of issues getting things to work reliably, but that's thanks to overcomplicating everything :) Nice to have clear guidance on exactly how to get things working. I find you explain all the caveats well and any question I have usually gets answered during the video.
@@Jims-Garage It's interesting that my day job is Network Engineering so I'm super familiar with (Fortigate) HA setup and operation. Yet wanting to set it up for my home lab is very different! As the saying goes "If someone can't explain it simply, then they don't understand it" (I always have this in my head when explaing things to people) and you 100% have nailed that 😎
Loving all of these videos as I'm working to rebuild my homelab. Would love to see a deeper dive on how you have your Ubiquiti kit setup. Keep up the great videos!
Hi 🙂 Just found this video and it is everything that I need in a house where your family needs CONSTANT Internet connectivity... 😀 I was just wondering how you would proceed with an existing firewall OPNsense setup with VPN, a lot of rules and interfaces? I think the ground principle is to migrate the current interfaces IPs to be transformed in VIPs, but some questions about it: - Would you prepare a pair of firewalls instances (VMs staged with edited config files and new interfaces IPs) in parallel with some temporary VIP and the you shut down the single FW and switch all VIPs? - Can be keep/modify existing VPN tunnels to keep everything running (I have a IPSEc tunnel and an OpenVPN system)? If yes, how? But your video could not come at any better time, thanks a lot!
I believe the recommended approach is to start with 2 blank (new firewalls) and create the HA. Then configure the primary from scratch. In your case you will likely keep the current, and copy the rules over to the new HA pair.
@@Jims-Garage Hi again, I'm in the starting block with 2 minisforum for my HA setup. I'll go with recommanded approach : start blank and import things I need. After looking at video again, I'm wondering why you are not configuring the HA earlier, wouldn't it spare a few configuration steps (like the firewall rules & CARP VIPs) as master would push them to the slave? I might test if you don't know 😉
In order for the firewalls to know which is the Master, Skew must be configured to for a lower value indicating the Master (I use Skew of 0 for Master and 100 for Backup). Interfaces > Virtual IPs > Settings and when you edit the interface it won't show up until you click Advanced Mode in the top left. 20:13 in the video, though it isn't shown here NOR in the documentation! Also, Jim appears to be repeating himself over and over. This is a great speaking strategy with a live audience. Though, when making a youtube video, be more like Adam Ragusea; say things clearly once and move on. Remember we have the ability to rewind!
@@aaronbreault thanks, good point that I should have elaborated on, albeit it's not strictly required from my testing and research (but definitely recommended). I do repeat key items several times, the avg view times is around 4min so there's less chance someone will miss it. Saves endless comments.
@@Jims-Garage Thanks for all your videos man, they have helped me along my path to sobriety and being all the computer guy I can be. My initial reaction is to say that comments drive viewership, in case that is something you care about. Also, I'm not sure someone watching the video for 4 minutes is going through the process of setting anything up, lol. So like, what is the goal of the video? Are you trying to make a perfect guide? Are you trying to drive viewership? Other things? The answer is always somewhere in between right? So perhaps my criticism was incorrect, depending on what your goals are. I now have multiple virtualized firewalls running in a setup that will soon include more pieces of the puzzle you have helped show us all. So thanks again!
@@aaronbreault that's great, glad you have it configured. I don't really have a masterplan, it's certainly not to make money as it's almost impossible with this kind of content. More just to help people out if they want a starting point. Congrats on sobriety, that's a big win.
Greetings Jim, As always nice videos. I watching Part 3 after looking at Part 1 & half of Part 2. I chose that route because I wanted to get an idea how you did HA and then I was going to go back and watch the rest of Part 2's configurations. Hopefully I'm not asking a question covered in the section of Part 2 I haven't looked at yet. So 8 minutes into Part 3 I thought you said net0 which is vmbr7 is the WAN and net1 which is vmbr6 is the LAN. In Part 1 about 16 minutes into the video I heard vmbr7 is the LAN and vmbr6 is the WAN. Did I miss something somewhere and the LAN and WAN got switched?
Hey, thanks for your comment. I might have misquoted, not sure. Either way, keep them the same throughout. For HA, you might want to watch a later video "HA the wrong way" - it's a lot simpler and only requires 1 VM.
I thought that might be the case but just checking. The video you mentioned, if your referring ti this video - OPNSense High Availability - 1 VM, 1 IP!, that uses a cluster and I originally was going that way but during some experimenting I crashed a node on purpose and then used the recommended CLI command called 'kill' and I had issues with my two remaining nodes so I moved away from clustering for the time being. Thanks for the fast response. Have a good weekend.
Great work again Jim. When you check boxes to sync from master to slave firewall, and not other way around which will help in inital config sync but if a failover to happen and you make changes to config on 2nd firewall "slave" and if 1st firewall come up would that config copy over? Appreciate you taking the time to do this video on opnsense.
No, it doesn't work that way (you can check the link to the docs). There needs to be a master. This at least gives you the opportunity to get the master back up and running as your network is still available.
The prophecy is true, Jim told me I should be up and running by the end of the video, the video is 28 minutes long yet I am 3 hours into it. Life is a simulation.....
Thanks, probably the quickest answer is that I'm not changing from Sophos XG, I'm keeping it. Nothing wrong with OpnSense, I think it's great, just isn't giving me anything I don't already have (plus I find OpnSense a little trickier to use).
@@altimeterlabs correct. A lot of my subscribers use it (70%) and it was a common request. I went into it with a trial in mind, after using it for a couple of weeks I was impressed, but ultimately prefer Sophos. Some only want opensource so Sophos isn't an option.
Really great video @Jims-Garage , really appreciate the help you give us with these tutorials, i'm trying to figure out how to setup HA but with 2 isp and 2 opnsense firewall, honestly the diagrams found on the internet seems far too complicated, i found in an office a master opnsens and a backup working with 2 isp but the configuration only had a single switch, do you have simpler way to make ha with 2 isp ? Thank you in advance for your time!
Should be quite simple. Essentially, copy my video but replicate switch vLAN part. You'll need to do that twice with separate vLANs, and then add 2 wan NICs to each firewall.
Thanks, your guide was helpful. Setup bit different with single LAN CARP and regular DHCP WAN with CARP WAN INT toggle script. (Inet terminates into Opnsense, no upstream router) I don't understand why you setup the outbound NAT of LAN net on pfsync INT. Why would lANt net need to flow this way? In my environment I have transparent fortigate 60F downstream so I setup outbound LAN INT NAT of lan net --> LAN CARP IP. This way downstream MGT traffic of DNS, LDAP, Zabbix, etc is seen as CARP IP rather than each Opnsense separate LAN IP. 3 things hung me up for a bit. 1. Make sure all OPT#s match. 2. Make sure admin listen interfaces on all recommended. 3. Make sure define custom Webui port # in HA config.
I do believe you need to permit all traffic between the pfsync interfaces. I only allowed CARP and ICMP at first, only noticing that the master was communicating to HTTPS port on the slave when viewing the logs. Edit: at least permit port 443 I mean
Hi, I am thinking to set it up the same way. The only thing you didn't show was to create the sync interface in proxmox. Did you just create an empty linux bridge and pass iton to the opnsense VM on both nodes? thanks for an amazing video as always
Hi Jim, I've been following your channel for months. Thanks for the great content. I'm currently setting up my OpnSense HA following your tutorial. I noticed a little discrepancy at minute 22:00 when you are comparing the Outband NAT rules between the 2 OpnSense instances. Looks like the second rule on the WAN interface is showing LAN net when the other instance is showing LAN address on the same rule. Which is the correct setup?
@@Jims-Garage Hi. Just sharing my findings. LAN Address does not work, but LAN net does. I am fortunate enough to have public IPs and if I use LAN Address, it doesn't use the VIP WAN IP, but rather the IP of the WAN Interface and my failover didn't work. I had to set both to LAN net and now it is working as expected. Thank you for the video.
Hi Jim! After syncing master and backup firewall, is there going to be same configuration on backup automagically or do I have to manually configure interfaces, dhcp leases etc...?
@@Jims-Garage Thank you. I will have to find out why they are not syncing 😔 Although everything else seems fine. It shows checkboxes after I click on "restart all services" on master fw.
And now I found out in OpnSense documentation that combination of physical machine and virtual machine will not work because of the different interface names 😔😔😔
@@Sejl I got mine working with a physical machine and a virtual machine. I actually tested my machines virtual first then once I was happy with the configuration, downloaded the configuration and downloaded it on my old firewall appliance to work as a backup.Personally, I don't think you can beat the reliability of OPNsense on bare metal. As much as I like proxmox, I feel like it's not as reliable. I've had my machine reboot randomly. It was likely due to something I did, but that's the thing. I can't afford to have a simple mistake bring down my entire internet. As far as how I got DHCP working on my setup. I actually cloned the mac address from the master to the backup. Not 100% sure why this works, but seems the backup just picks up the exact same IP address as the master. I guess it tricks your ISP into thinking it's the same machine. The only caveat I noticed is that while the machine appears to have a WAN IP address, it doesn't seem to have any internet. So, it was basically impossible to download any plugins while the master was active. The nice thing about having a virtualized firewall is that you can simply pause the master to make the backup active. And if you happen to need to reboot your backup, your master kicks back in almost instantly after you unpause the VM. It's pretty magical, honestly.
@@Jims-Garage i have a modem from my isp which has one ethernet cable to a dual port nic which is pcie passthrough to my opnsense vm on proxmox, lan is the other port which goes to a managed switch which then splits to my network. I mainly dont want to do double nat or have to manage a third router.
Are you using UniFi Access Points on your network? Do you run into any problem with those using this setup? I've seen where UniFi APs run into problems with ForcePoint firewalls in hot-standby mode.
@@Jims-Garage I just dug out my notes I made at a client of mines months ago and I have to partially correct myself. The issue I saw is when Forcepoint Firewalls are in a cluster of 2 or more and in Active/Active load balancing and failover mode. The load-balancing active/active mode of Forcepoint clusters borks the UniFi APs. The client worked around their issue by making it active/standby & never found a solution to the problem. My guess is the Unifi APs updates to it's MAC table don't happen fast enough for the FP FW's load balancing? But no, not applicable to your setup.
@@cryptodendrum thanks for updating. Yeah active active can get funky due to what you described. Mine is essentially a single instance that migrates on demand.
@@Jims-Garage I did some more (a lot more) digging and found that ForcePoint is aware of the problem in both 6.11 and 7.0 releases of their firmware and do NOT identify it as specific to UniFi. The problem is listed with a reference # NGFW-46050 on their website. For the base price of their NGFW-330 series appliances, one could better buy 6 X MS-01's and get better IDS / IPS performance running OPNsense or pfSense on Proxmox or XCP-NG. I know enterprises like their vendor supported products, but if there's no fixes for this problem after 2 major releases - maybe buying clusters to load-balance underpowered hardware appliances ain't the way to go. lol
Just about 7 minuits man im loving it! one thought is most ISP routers don't allow to have multiple ip's for the same firewall rule/port. How could this be tackled?
@@Jims-Garage I get that, We are talking about the wan side of opnsense and lan side of my isp router. Most ISP routers are not able to switch between opnsense wan ips in port forward rule to expose services/sites. That still has to be done manually for me. For keeping internet connection its not a real worry. Hope that made a bit more clearer. (reposting as I accidentally deleted my comment)
@@sebasdt2103 That's a good point. If you're double NAT and you have open ports, any port forwards will be setup to only one of the LAN IPs. I usually turn UPnP off, but I wonder if it could be used to manage the open ports between onpsense and your ISP router. Wouldn't be much of a risk since the traffic all hits opnsense anyway.
@@SurfSailKayak maybe somehow create a vip on the wan side... Not sure How that would work. I usually use KeepaliveD to put both of my piholes between a vip. maybe we can do something like this with opnsense on wan side? but its still an interesting point.
very helpful video. Question: What do you suggest for users who do not have a ISP router? I have fiber to my house that goes to an ONT. The ONT provides ethernet that is plugged in directly into my Opensense Router on my WAN port. They did provide a Eero but it is limited to 1G and my service is 2G.
What would you suggest when -in my case- the fiber can be plugged directly (via sfp or media converted utp) into a (aggregation) switch or in the proxmox server? The internet is on its own VLAN from the isp. So my thoughts are to buy an unifi aggregation switch, put the fiber from ISP in port 1, make port 2&3 WAN and 4/5 LAN and 6 to the network switch (all sfp+). But now I am struggeling to translate this to your concept.
@@jellevanburen9427 That sounds similar to what I'm doing. I guess you'd plug into switch, make a vlan group that matches the vlan id of the ISP, and then plug both respective WANs of your firewall into the switch on the same vlan
Very easy to understand even for people like me that are want to learn and doesn't speak a very good english. Thanks! Back on topic, is there a way to avoid the double nat (bad for online gaming) without having paying for a second line? My modem/router has a 4 port switch, it set as bridge and it uses pppoe to connect (from opensense). Do you know if it is possible to turn on the pppoe connection on the backup when the master is down? Ciao Roberto
Will everything you did here work if I opt for your first diagram? The one where I use double switch (one for splitting my wan coming from my modem to both proxmox nodes and one for my LAN)? The reality of it is that I don't wanna use my ISP box as router, I want to keep it only as modem in bridge mode, and I wanna use Proxmox as my sole routing solution. Also on a side note I never would've expected that Opnsense supports HA inside of it. When I was thinking to do HA for Opnsense in Proxmox I thought it more along the lines of Proxmox spawning my Opnsense VM in the next available Node if the current Node were to shut down. Did I have the complete wrong idea about it?
When creating the second OPNsense VM for the Backup Firewall The install creates the same 192.168.1.1 IP for the LAN and also creates the same DHCP scope range. So the Backup firewall LAN IP needs to be changed to 192.168.1.2 . What I haven't been able to find out in searches is when the Backup VM is in Backup mode does it's DHCP scope interfere with the Master's DHCP scope so it has to be a different range?
I have accidentaly set virtualIp same as wan ip, and locked myselft out of everything 😂 Now I have to wait for my vacation to be over, to fix it _from inside the house_
Thanks again, it was great especially that you used the diagram to simplify the roadmap.
Thanks 👍
Just finished Part 3 and I'm exhausted collecting all the information you put in the 3 videos. Really fabulous work, you must have been 10 times more tired than I am after creating them. Now the fun part is building the Proxmox OPNsense VM's and getting them configured. I have a TP-Link ER605 router that is going to be my ISP router interface that will connect to my cable modem which I've have already tested with an OPNsense trial VM so it looks like I have all the pieces to move forward tomorrow. Thanks again for all the hard work you invested to make my project successful.
Jim
From one Jim to another, you're welcome! You might want to check out my latest OPNSense video (check the playlist). It does HA with only a single machine, it's much simpler!! Unless you're set on having 2 discrete devices.
I had a cluster with 3 nodes which did a nice job moving VM's so I decided to try removing a node and I was intent on re-establishing the node as an experiment just to see how involved it was. Following some documentation I used the kill command which removed the node but after the PVE reboot all that restarted was Debian and Proxmox wouldn't restart. So I had to reinstall PVE and of course lost the entire configuration. Also the other two nodes lost their NFS share to a bare metal TrueNAS server and I had trouble connecting to the nodes individually. Because of time limitations I'm using two nodes for now and went with your OPNsense video to get HA on two nodes. I figured I'd try three nodes in a cluster when some projects get completed this fall. Ceph really interests me and I see you have a video on the subject. I'm sure after watching it I'll get the bug to create a cluster again. Thanks again for your hard work.
Once again I have to say thanks. After crabbing about how I trashed my cluster you got me thinking so I went back into my notes and found my mistake. My age is affecting my memory but you suggestion got me motivated to readdress doing a cluster again. Good thing I'm retired you have so many interesting videos to watch and learn.
Really cool Jim, every single one of your videos is relevant to different things I'm implementing in my homelab. Keep it coming! I've had a lot of issues getting things to work reliably, but that's thanks to overcomplicating everything :) Nice to have clear guidance on exactly how to get things working. I find you explain all the caveats well and any question I have usually gets answered during the video.
That's great. I appreciate the feedback. Nice work!
Thanks for the demo and info, have a great day
Thanks, you too!
Thank you!
@@ElTebe you're welcome
Only recently discoverd your channel. Thanks for all the great content 👍
You're welcome, appreciate the feedback
@@Jims-Garage It's interesting that my day job is Network Engineering so I'm super familiar with (Fortigate) HA setup and operation. Yet wanting to set it up for my home lab is very different! As the saying goes "If someone can't explain it simply, then they don't understand it" (I always have this in my head when explaing things to people) and you 100% have nailed that 😎
@@TheDervMan thanks 👍
Loving all of these videos as I'm working to rebuild my homelab. Would love to see a deeper dive on how you have your Ubiquiti kit setup. Keep up the great videos!
Thanks, it's on the list 😁
OMG.. this video is gold!
Thanks 🥇
Hi 🙂
Just found this video and it is everything that I need in a house where your family needs CONSTANT Internet connectivity... 😀
I was just wondering how you would proceed with an existing firewall OPNsense setup with VPN, a lot of rules and interfaces?
I think the ground principle is to migrate the current interfaces IPs to be transformed in VIPs, but some questions about it:
- Would you prepare a pair of firewalls instances (VMs staged with edited config files and new interfaces IPs) in parallel with some temporary VIP and the you shut down the single FW and switch all VIPs?
- Can be keep/modify existing VPN tunnels to keep everything running (I have a IPSEc tunnel and an OpenVPN system)? If yes, how?
But your video could not come at any better time, thanks a lot!
I believe the recommended approach is to start with 2 blank (new firewalls) and create the HA. Then configure the primary from scratch. In your case you will likely keep the current, and copy the rules over to the new HA pair.
@@Jims-Garage Hi again, I'm in the starting block with 2 minisforum for my HA setup. I'll go with recommanded approach : start blank and import things I need.
After looking at video again, I'm wondering why you are not configuring the HA earlier, wouldn't it spare a few configuration steps (like the firewall rules & CARP VIPs) as master would push them to the slave?
I might test if you don't know 😉
In order for the firewalls to know which is the Master, Skew must be configured to for a lower value indicating the Master (I use Skew of 0 for Master and 100 for Backup). Interfaces > Virtual IPs > Settings and when you edit the interface it won't show up until you click Advanced Mode in the top left. 20:13 in the video, though it isn't shown here NOR in the documentation!
Also, Jim appears to be repeating himself over and over. This is a great speaking strategy with a live audience. Though, when making a youtube video, be more like Adam Ragusea; say things clearly once and move on.
Remember we have the ability to rewind!
@@aaronbreault thanks, good point that I should have elaborated on, albeit it's not strictly required from my testing and research (but definitely recommended).
I do repeat key items several times, the avg view times is around 4min so there's less chance someone will miss it. Saves endless comments.
@@Jims-Garage Thanks for all your videos man, they have helped me along my path to sobriety and being all the computer guy I can be.
My initial reaction is to say that comments drive viewership, in case that is something you care about. Also, I'm not sure someone watching the video for 4 minutes is going through the process of setting anything up, lol. So like, what is the goal of the video? Are you trying to make a perfect guide? Are you trying to drive viewership? Other things? The answer is always somewhere in between right? So perhaps my criticism was incorrect, depending on what your goals are.
I now have multiple virtualized firewalls running in a setup that will soon include more pieces of the puzzle you have helped show us all. So thanks again!
@@aaronbreault that's great, glad you have it configured.
I don't really have a masterplan, it's certainly not to make money as it's almost impossible with this kind of content. More just to help people out if they want a starting point.
Congrats on sobriety, that's a big win.
Greetings Jim, As always nice videos. I watching Part 3 after looking at Part 1 & half of Part 2. I chose that route because I wanted to get an idea how you did HA and then I was going to go back and watch the rest of Part 2's configurations. Hopefully I'm not asking a question covered in the section of Part 2 I haven't looked at yet. So 8 minutes into Part 3 I thought you said net0 which is vmbr7 is the WAN and net1 which is vmbr6 is the LAN. In Part 1 about 16 minutes into the video I heard vmbr7 is the LAN and vmbr6 is the WAN. Did I miss something somewhere and the LAN and WAN got switched?
Hey, thanks for your comment. I might have misquoted, not sure. Either way, keep them the same throughout. For HA, you might want to watch a later video "HA the wrong way" - it's a lot simpler and only requires 1 VM.
I thought that might be the case but just checking. The video you mentioned, if your referring ti this video - OPNSense High Availability - 1 VM, 1 IP!, that uses a cluster and I originally was going that way but during some experimenting I crashed a node on purpose and then used the recommended CLI command called 'kill' and I had issues with my two remaining nodes so I moved away from clustering for the time being. Thanks for the fast response. Have a good weekend.
Great work again Jim. When you check boxes to sync from master to slave firewall, and not other way around which will help in inital config sync but if a failover to happen and you make changes to config on 2nd firewall "slave" and if 1st firewall come up would that config copy over?
Appreciate you taking the time to do this video on opnsense.
No, it doesn't work that way (you can check the link to the docs). There needs to be a master. This at least gives you the opportunity to get the master back up and running as your network is still available.
The prophecy is true, Jim told me I should be up and running by the end of the video, the video is 28 minutes long yet I am 3 hours into it. Life is a simulation.....
Haha 😂
H i Jim - I'd love an explanation on why you decided to swithc from Sophos to OPNSense and how you chose OPNSense vs PFSense. Thanks!
Thanks, probably the quickest answer is that I'm not changing from Sophos XG, I'm keeping it. Nothing wrong with OpnSense, I think it's great, just isn't giving me anything I don't already have (plus I find OpnSense a little trickier to use).
Users are abandoning pfSense (and switching to OPNsense) because Netgate disregards the community.
@@Jims-Garage I see, so this is just a series on how to use OonSense for those that use it, but you are not implementing it yoursef
@@altimeterlabs correct. A lot of my subscribers use it (70%) and it was a common request. I went into it with a trial in mind, after using it for a couple of weeks I was impressed, but ultimately prefer Sophos. Some only want opensource so Sophos isn't an option.
Really great video @Jims-Garage , really appreciate the help you give us with these tutorials, i'm trying to figure out how to setup HA but with 2 isp and 2 opnsense firewall, honestly the diagrams found on the internet seems far too complicated, i found in an office a master opnsens and a backup working with 2 isp but the configuration only had a single switch, do you have simpler way to make ha with 2 isp ? Thank you in advance for your time!
Should be quite simple. Essentially, copy my video but replicate switch vLAN part. You'll need to do that twice with separate vLANs, and then add 2 wan NICs to each firewall.
Thanks, your guide was helpful. Setup bit different with single LAN CARP and regular DHCP WAN with CARP WAN INT toggle script. (Inet terminates into Opnsense, no upstream router)
I don't understand why you setup the outbound NAT of LAN net on pfsync INT. Why would lANt net need to flow this way? In my environment I have transparent fortigate 60F downstream so I setup outbound LAN INT NAT of lan net --> LAN CARP IP. This way downstream MGT traffic of DNS, LDAP, Zabbix, etc is seen as CARP IP rather than each Opnsense separate LAN IP.
3 things hung me up for a bit. 1. Make sure all OPT#s match. 2. Make sure admin listen interfaces on all recommended. 3. Make sure define custom Webui port # in HA config.
I do believe you need to permit all traffic between the pfsync interfaces. I only allowed CARP and ICMP at first, only noticing that the master was communicating to HTTPS port on the slave when viewing the logs.
Edit: at least permit port 443 I mean
Solve my problem, thanks
Hi, I am thinking to set it up the same way. The only thing you didn't show was to create the sync interface in proxmox. Did you just create an empty linux bridge and pass iton to the opnsense VM on both nodes? thanks for an amazing video as always
Thanks for the feedback. This is part 3, I created the 3 NICs in part 1, please refer to that and then reach out if you have further questions.
Hi Jim,
I've been following your channel for months. Thanks for the great content.
I'm currently setting up my OpnSense HA following your tutorial.
I noticed a little discrepancy at minute 22:00 when you are comparing the Outband NAT rules between the 2 OpnSense instances. Looks like the second rule on the WAN interface is showing LAN net when the other instance is showing LAN address on the same rule. Which is the correct setup?
Thanks, and well spotted. It should be net. Both should work though.
@@Jims-Garage Hi. Just sharing my findings. LAN Address does not work, but LAN net does. I am fortunate enough to have public IPs and if I use LAN Address, it doesn't use the VIP WAN IP, but rather the IP of the WAN Interface and my failover didn't work. I had to set both to LAN net and now it is working as expected. Thank you for the video.
Hi Jim! After syncing master and backup firewall, is there going to be same configuration on backup automagically or do I have to manually configure interfaces, dhcp leases etc...?
The configurations should sync after HA.
@@Jims-Garage Thank you. I will have to find out why they are not syncing 😔 Although everything else seems fine. It shows checkboxes after I click on "restart all services" on master fw.
And now I found out in OpnSense documentation that combination of physical machine and virtual machine will not work because of the different interface names 😔😔😔
@@Sejl I got mine working with a physical machine and a virtual machine. I actually tested my machines virtual first then once I was happy with the configuration, downloaded the configuration and downloaded it on my old firewall appliance to work as a backup.Personally, I don't think you can beat the reliability of OPNsense on bare metal. As much as I like proxmox, I feel like it's not as reliable. I've had my machine reboot randomly. It was likely due to something I did, but that's the thing. I can't afford to have a simple mistake bring down my entire internet.
As far as how I got DHCP working on my setup. I actually cloned the mac address from the master to the backup. Not 100% sure why this works, but seems the backup just picks up the exact same IP address as the master. I guess it tricks your ISP into thinking it's the same machine. The only caveat I noticed is that while the machine appears to have a WAN IP address, it doesn't seem to have any internet. So, it was basically impossible to download any plugins while the master was active. The nice thing about having a virtualized firewall is that you can simply pause the master to make the backup active. And if you happen to need to reboot your backup, your master kicks back in almost instantly after you unpause the VM. It's pretty magical, honestly.
Could you do a video on how to do this with one IP without the ISP-Router infront?
Can you explain your setup?
@@Jims-Garage i have a modem from my isp which has one ethernet cable to a dual port nic which is pcie passthrough to my opnsense vm on proxmox, lan is the other port which goes to a managed switch which then splits to my network. I mainly dont want to do double nat or have to manage a third router.
Are you using UniFi Access Points on your network? Do you run into any problem with those using this setup? I've seen where UniFi APs run into problems with ForcePoint firewalls in hot-standby mode.
I do, I haven't noticed any issues so far.
@@Jims-Garage I just dug out my notes I made at a client of mines months ago and I have to partially correct myself.
The issue I saw is when Forcepoint Firewalls are in a cluster of 2 or more and in Active/Active load balancing and failover mode. The load-balancing active/active mode of Forcepoint clusters borks the UniFi APs. The client worked around their issue by making it active/standby & never found a solution to the problem.
My guess is the Unifi APs updates to it's MAC table don't happen fast enough for the FP FW's load balancing? But no, not applicable to your setup.
@@cryptodendrum thanks for updating. Yeah active active can get funky due to what you described. Mine is essentially a single instance that migrates on demand.
@@Jims-Garage I did some more (a lot more) digging and found that ForcePoint is aware of the problem in both 6.11 and 7.0 releases of their firmware and do NOT identify it as specific to UniFi. The problem is listed with a reference # NGFW-46050 on their website.
For the base price of their NGFW-330 series appliances, one could better buy 6 X MS-01's and get better IDS / IPS performance running OPNsense or pfSense on Proxmox or XCP-NG. I know enterprises like their vendor supported products, but if there's no fixes for this problem after 2 major releases - maybe buying clusters to load-balance underpowered hardware appliances ain't the way to go. lol
Just about 7 minuits man im loving it! one thought is most ISP routers don't allow to have multiple ip's for the same firewall rule/port. How could this be tackled?
Not sure what you mean. If you attach the WAN port of OpnSense(s) to the LAN port of the ISP router it'll pick up different DHCP LAN addresses.
@@sebasdt2103 Yes, you're right. That is a problem with respect to availability. Best best is to use the master IMO.
@@Jims-Garage I get that, We are talking about the wan side of opnsense and lan side of my isp router.
Most ISP routers are not able to switch between opnsense wan ips in port forward rule to expose services/sites.
That still has to be done manually for me.
For keeping internet connection its not a real worry.
Hope that made a bit more clearer.
(reposting as I accidentally deleted my comment)
@@sebasdt2103 That's a good point. If you're double NAT and you have open ports, any port forwards will be setup to only one of the LAN IPs. I usually turn UPnP off, but I wonder if it could be used to manage the open ports between onpsense and your ISP router. Wouldn't be much of a risk since the traffic all hits opnsense anyway.
@@SurfSailKayak maybe somehow create a vip on the wan side... Not sure How that would work.
I usually use KeepaliveD to put both of my piholes between a vip. maybe we can do something like this with opnsense on wan side? but its still an interesting point.
very helpful video. Question: What do you suggest for users who do not have a ISP router? I have fiber to my house that goes to an ONT. The ONT provides ethernet that is plugged in directly into my Opensense Router on my WAN port. They did provide a Eero but it is limited to 1G and my service is 2G.
I'd recommend a firewall that doesn't require CARP. I use Sophos XG partly for this reason. I can use my single IP and split it across both.
What would you suggest when -in my case- the fiber can be plugged directly (via sfp or media converted utp) into a (aggregation) switch or in the proxmox server? The internet is on its own VLAN from the isp. So my thoughts are to buy an unifi aggregation switch, put the fiber from ISP in port 1, make port 2&3 WAN and 4/5 LAN and 6 to the network switch (all sfp+). But now I am struggeling to translate this to your concept.
@@jellevanburen9427 That sounds similar to what I'm doing. I guess you'd plug into switch, make a vlan group that matches the vlan id of the ISP, and then plug both respective WANs of your firewall into the switch on the same vlan
Very easy to understand even for people like me that are want to learn and doesn't speak a very good english.
Thanks!
Back on topic, is there a way to avoid the double nat (bad for online gaming) without having paying for a second line?
My modem/router has a 4 port switch, it set as bridge and it uses pppoe to connect (from opensense). Do you know if it is possible to turn on the pppoe connection on the backup when the master is down?
Ciao Roberto
No, it cannot be done with OpnSense due to how carp works. Sophos XG will do HA with a single IP and no doubt nat. I have guides on that as well.
@@Jims-Garage thank you! I will give it a look!
@@crc-error-7968 it's what I'm using if that gives you any further comfort.
Will everything you did here work if I opt for your first diagram? The one where I use double switch (one for splitting my wan coming from my modem to both proxmox nodes and one for my LAN)? The reality of it is that I don't wanna use my ISP box as router, I want to keep it only as modem in bridge mode, and I wanna use Proxmox as my sole routing solution.
Also on a side note I never would've expected that Opnsense supports HA inside of it. When I was thinking to do HA for Opnsense in Proxmox I thought it more along the lines of Proxmox spawning my Opnsense VM in the next available Node if the current Node were to shut down. Did I have the complete wrong idea about it?
When creating the second OPNsense VM for the Backup Firewall The install creates the same 192.168.1.1 IP for the LAN and also creates the same DHCP scope range. So the Backup firewall LAN IP needs to be changed to 192.168.1.2 . What I haven't been able to find out in searches is when the Backup VM is in Backup mode does it's DHCP scope interfere with the Master's DHCP scope so it has to be a different range?
I have accidentaly set virtualIp same as wan ip, and locked myselft out of everything 😂 Now I have to wait for my vacation to be over, to fix it _from inside the house_
I feel your pain, I've been there 😭
I thought high availiability in the context of firewall meant 2 ISP no?
Full HA does, and in enterprise you'd have 2 wan. I simply want 2 firewalls to enable me to reboot certain nodes.
Hi Jim, can you confirm that youtube has deleted some comments in this video?!?
It was "held for review", I've approved it.
@@Jims-Garage There were other comments...
@@Glatze603 I've approved a long one that you wrote (which was very helpful). Not sure why it isn't showing.
0:51 Lizard blink
@@nemac23 the peril of a smooth transition 🦎
@@Jims-Garage 🤣