Hey there, I’ve been on a networking journey that has, over a few years, taken me from simple unmanaged networking, to managed networking, to advanced VLAN management. It’s all been self taught, but mostly successful. However, I’ve gotten myself into a bit of a pickle and I’m hitting a wall in troubleshooting. Apologies for the length of the post, however I want to provide as much detail as possible.

High level, I have several /16 vlans for things. VLAN 99 is networking, 2, is servers, 4 is clients, 6 is wireguard clients, and there are some others. They’re all 10.99.0.0/16 with a gateway at 10.99.1.254, etc.

I have had a very old Netgear Layer3 switch for some time. I’ve replaced it with a Brocade ICX6610, mostly so I can move my storage infrastructure to 10G fiber (I have a small hypervisor cluster). I had done a ton of preparatory work to configure the new L3 switch so that it could just be dropped in place of the old one; this was MOSTLY successful…

…However, in doing that I broke the connection to my opnsense firewall and sort of had to redo that piece from scratch. During my planning, I didn’t realize some of the config changes I’d made would require changes on the firewall, and after the cut over I was locked out of the firewall. This is all my fault; that’s the piece of this I understand the least, and I had followed dodgy guides when getting it to initially work. I have a backup in xml format, but even having that I’m realizing what I had been doing didn’t make sense. Previously, I had a firewall interface on all of my vlans and the trunk going to it was carrying all the VLANS. Now, I set this up with only 2 vlans going to the firewall, the networking vlan and the wireguard vlan, as it seems to make more sense with my understanding of how Layer 3 routing works. All routing should happen on the Brocade L3 switch. The firewall itself has 4 physical ports, 1 going to my comcast gateway, and 2 in an LACP lagg going to my L3 switch. (I have a single interface right now going to the L3 switch separately for troubleshooting, removing the LACP lag as a complexity source).

So, in recovering this, I had to get into the firewall at the console and re-define the interfaces and IP’s. I got this to work, but at this point I had tons of connection problems which I didn’t understand fully. I have found some of opnsense’s configuration to be a bit obfuscating, which I think is making my learning more difficult. The following were put in place:

  • The “LAN” interface was given a static 10.99.1.40/16 IP, and an upstream gateway was defined at 10.99.1.254.
  • The “WAN” interface was given DHCP, and is up and works

Once I recovered the connection to the web interface I had to make the following changes:

  • Under the “Firewall” sidebar, under “Aliases”, I defined each of my VLANS/Subnets with a CIDR notation and a name.
  • Under the “Firewall” sidebar, under “NAT” and then under “Outbound” I switched the mode to “hybrid” and added a rule for each of my vlans on the “LAN” interface, with the “Source” being the aliases defined above, and the target (NAT Address) being the “WAN address”
  • Under the “Firewall” sidebar, under “NAT” and then under “Port Forward” I added some port forward rules.
  • While it’s outside the scope of my immediate troubleshooting, I had a working WireGuard setup. I have an interface defined for it on that VLAN, and a second gateway defined at 10.6.1.254. It’s all set up according to the opnsense documentation, and I can connect from the WAN and can access any resources on the LAN.

So onto the problem…I can access the internet from almost all of my LAN clients. I can access LAN clients via the port forward rules from the WAN. The firewall itself CANNOT access the WAN; for example, I can’t check for updates. I can access the firewall web interface from anywhere on the LAN, I can ssh to the firewall from anywhere on the LAN, but once I’m ssh’d in, I can’t ping back to the client I’m connecting from. The firewall CAN ping things like 8.8.8.8, but as my DNS resolver is on the LAN, DNS queries from the firewall fail. I believe in a related note, my WireGuard clients can access anything on the LAN, but cannot connect to anything on the WAN.

I believe this has to do with outbound routes from the firewall, but any time I mess with it I end up locking myself out and having to reset interfaces from the console. I tried defining some static routes in “System” -> “Routes” -> “Configuration” but that isn’t working. I’m kind of stumped and have been looking at it so long that I don’t think more reading and configuring is going to help me anymore. I’ll post some screenshots of rules and routes as well (you’ll be able to see various things enabled/disabled for experimentation), but I’m kind of in over my head and need some help.

  • @tuxedEnglish
    1•3 months ago

    If the firewall cant reach the LAN, either because of a firewall rule or bad routing, it will not be able to access the DNS server even if it works well for the rest of the LAN. I’m assuming that the rest of the LAN talks to the DNS server directly and not through the firewall.

    It sounds like you would benefit from reading a bit about how routing and gateways work, as it seems like you’re mostly trying stuff without really knowing what it does. Please save yourself some sanity and make some proper planning on your different subnets, their vlans and how they should route their traffic, ideally in a diagram of some sort.

    Without knowing your exact setup I’m getting a feeling that your current configuration is both overly permissive and overly restrictive, meaning you cant access the things you want but any potential attackers can probably get around just fine.

    I would seriously consider tearing it down and starting over with a more cohesive plan, but I know that might not be possible for you time-wise. On the other hand, having a well planned network that you understand would almost certainly save you time in the long run, especially if you want to keep doing more advanced and unorthodox stuff to it.

    • surfrock66OPEnglish
      1•3 months ago

      I probably need to burn it down and restart, but I need to find a time the family will tolerate an extended outage. I did share some things on the opnsense forum though which might be useful here.

      1. My diagram. At the bottom you will see why I have /16; in truth, it’s from back when I only had a single subnet, and I made it /16 so I could use the third octet to form DHCP scopes. That’s how the network worked in my head and I knew the IP scheme, so when it came time to add VLANS much later, I just made those the 2nd octet, and that’s how we are here today. Maybe one day I’ll re-do that, but it’s not in scope right now: https://nextcloud.surfrock66.com/s/txnZdzxHaiA5t65
      2. I did an experiment with static routes last night. I have the static route in, so I untagged the “LAN_GW” as an upstream gateway, and tagged “WAN_GW” as an upstream gateway. No change in the ability for opnsense to ping anything (it can ping WAN, not LAN), however all my LAN clients lost internet. In this state, from opnsense, I ran a “ping -S 10.99.1.40 10.2.2.213” (that’s my DNS server). This failed, but interestingly enough I was looking at the live logs, and even though the interface is LAN, the source IP was the WAN IP. I’m very confused; I’ve confirmed the LAN and WAN interfaces are correct and they have correctly assigned default gateways. See the attached picture. This would make sense; is opnsense doing something to switch the LAN and WAN somehow? I’m blown away how this is the case; that being said, it makes sense that tagging the LAN interface as upstream allows traffic out.

      It feels like somehow opnsense is treating LAN like WAN or something? I don’t know the obfuscation feels like it’s hiding things. A “ping -S 10.99.1.40 10.2.2.213” shouldn’t show in the logs with a source of the WAN address, right???

      • @tuxedEnglish
        1•3 months ago

        Okay, I think I know (at least one of) the problem(s).

        It is sending the ping from the WLAN interface because that is your default route, and you either don’t have a route specified for your 10.2.x.x network or you’re overwriting it with a different route (I’m guessing the first option).

        E.g. you need to tell your firewall “if you want to reach an ip-address in 10.2.x.x you need to go through here”, with “here” probably being either your managed switch if it works as a gateway (10.6.1.254?) or an interface on your router if it works as a switch (10.6.1.41?).

        • surfrock66OPEnglish
          1•3 months ago

          I’m totally with you…and I have that, which is why I think I’m hitting some sort of bug, or a firewall rule that is somehow breaking this:

          • @tuxedEnglish
            1•3 months ago

            Have you tried setting the gateway to one of your LAN interfaces? And what happens if you ping 10.99.1.254 from the firewall?

            • surfrock66OPEnglish
              1•3 months ago

              If I go to my LAN interface and set the gateway to “LAN_GW” at 10.99.1.254, everything works (but I can’t ping anything on the LAN from the firewall itself, including the client I’m ssh’d from). If I set that to Auto, all LAN clients lose WAN access.

              I’ve got a backup, but I think I’m gonna try to rebuild from scratch :/ I just worry I’m gonna end up in the same spot since I don’t understand how it all got here and don’t know what to avoid.

              • @tuxedEnglish
                1•3 months ago

                Probably a good idea sadly… There can be a lot of different things wrong, so will probably be faster doing that either way.

                When rebuilding, try to verify each that each step works so you find the problem eventually, Im guessing it will be easier to find that way

                • surfrock66OPEnglish
                  1•3 months ago

                  Ok, good news, I re-imaged and after about an hour of tinkering it’s working. (My wife is a doctor who does tele-medicine from home so it was tricky to get a downtime, even riskier if I couldn’t get back to working; usually she works when kids are in bed and that usually my window for these kind of projects). I still have my old config backup; I have a lot of firewall rules and services to put back in (I had redirects for google trying to reach their dns from chromecasts to my pihole, I had a zabbix client pointing to my zabbix server, I had wireguard working and want to see if I can restore existing key exchanges, it was tied to my LDAP server, etc). I really want to compare my old backup with a new one when this is done and see if I can’t figure out what was broken. I want to document that because I found a bunch of people with similar questions that only had incomplete answers:

                  1. From the CLI, the WAN interface was DHCP, I set up the lagg between my 2 ports (lagg0), created a vlan 99 interface off of it (lagg0_vlan99) and made that the LAN interface with a static IP and no gateway.
                  2. I made a gateway for my 10.99.1.254 LAN gateway, had to assign it to the LAN interface when I made it. It is not tagged as upstream. One thing I noticed, WAN_GW is priority 255; it was 254 before. Just a difference I noticed.
                  3. I made an alias for each of my VLANS that might need internet access
                  4. In Outbound NAT, I switched it to Hybrid and made rules to allow traffic through to each VLAN.
                  5. Under Firewall->Rules->LAN I created a pass rule for each VLAN (This will get tuned later)

                  With this, LAN clients access the WAN, after putting in a port forward WAN clients can access things on the LAN, the firewall can ping both LAN and WAN.

                  • @tuxedEnglish
                    2•3 months ago

                    Glad to hear it seems to be working! Hoping you find the issue in the backups, would be interesting to know what went wrong haha