BUG [URGENT] Reply-to does not working in latest release 26.1.2 with multi-wan

[Oriann]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

OPNsense does not add reply-to multi-wan rules, traffic is routing asymetric (eg.access to website or cloud is going through WAN1, download is initiating from WAN2). Manually setting up reply-to gateway or gateway policy does not help. I tried new rules and old rules too without success.

To Reproduce

Steps to reproduce the behavior:

1. Set up multiwan
2. Set up PF and manual rules same for both WAN (no extra settings needed)
3. Try to access website and then download something from it
4. See asymetric routing (best to test with downloading something big)

Expected behavior

With default PF and rules, reply-to should be added to all rules IF not disabled globally or locally by user himself

Describe alternatives you considered

I tried alternative from which is OPNsense forked, it worked on first try, no asymetric routing no special settings in PF or rules.

Relevant log files

I am not experienced in logging so I tell me what to download I will do that.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 26.1.2 (amd64).
Intel(R) Celeron(R) J4125 CPU @ 2.00GHz
Network Intel

[fichtner]

Steps are unclear. Did you set up a new box using 26.1 or did you upgrade? Old rules or new rules? Wizard used or not?

And is there a forum reference for it?

Cheers,
Franco

[Oriann]

I have tried:

1. Factory reset my box
2. Set up a new box
3. Tried alternative pfSense

In my first post I have already wrote -

I tried new rules and old rules too without success.

I am always setting up box without wizard. There is no forum reference as I have searched for it if somebody have same problem. I did not made a new thread because everytime on forum is first answer to make github ticket. So I did it first.

[fichtner]

I still don't understand. It was working on 25.7.x and stopped working on 26.1.2 or where is your conclusion coming from?

[Oriann]

I have moved to new house and got another ISP finally so I tried to set up same multiwan config for my services. Just to clarify I already had this setup working earlier in times when I was using pfSense and it worked on first try.
So I set up multiwan same but on OPNsense version 26.1.2 but came into issues also in previous version 26.1.1 and also in 25.7.11_9

How did I find out its not working as intended:
I am selfhosting my nextcloud instance and other 15 services. My flow with services are different based on what need more upload and download. So for example I have:

Backend (Wordpress) > reverse proxy (Nginx) > WAN1
Bacekend (Nextcloud) > reverse proxy (still the same one) > WAN2

WAN1 with WAN2 is in failover group (Tier 1 WAN1 and Tier2 WAN2). Default WAN in GUI is WAN1.

When I access my Nextcloud instance (from remote location) nslookup and network inspector showing the correct WAN2 interface, BUT when I start downloading from my nextcloud (also from remote location) stream is coming from WAN1. And this is not desirable for me.

So to your question I have not used this configuration on OPNsense till now, I have just discovered it 3 days earlier and found out via my testing that reply-to maybe not working correctly.

[Oriann]

Here is screenshots of what I did to make multiwan work without asymetric routing (because thats standard config I suppose for most users).

Gateways

DNAT

Firewall>Advanced

[Oriann]

Another insight:
Disabling reply-to on both WAN1 and WAN2 rules will result in the default gateway (WAN1) functioning normally and services being available through it. While services going through WAN2 will stop working completely (ERR_CONNECTION_TIMED_OUT).

Setting Bind states to interface will result in only working default WAN1 services, WAN2 services does not accept any connections (ERR_CONNECTION_TIMED_OUT)

Looks like, default gateway takes the precedence no matter what you set up in FW rules.

[fichtner]

We've just released a hotfix for 26.1.2 which includes one known reply-to bug but I'm unsure if this is relevant here.

[Oriann]

Just upgraded to hotfix 26.1.2 but I have still same problems. Tested all again with resetting states everytime.

[fichtner]

I was expecting as much. Unfortunately we can't help much with setup issues in community support and there's no clear picture of the bug that was according to you also present in 25.7.11_9 which makes this rather strange.

[Oriann]

I made forum post so maybe somebody will test and help to reproduce. I will try older version if I have time I have invested 2 days to testing this...

https://forum.opnsense.org/index.php?topic=50956.0