Let's assume that a firewall is connected via two separate Ethernet interfaces to two different ISPs.[1] as in the following diagram.

Each of these providers is described in an entry in the file /etc/shorewall/providers.

Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in /etc/shorewall/tcrules and /etc/shorewall/rtrules can be used to direct particular outgoing connections to one ISP or the other. Use of /etc/shorewall/tcrules is not required for /etc/shorewall/providers to work, but in most cases, you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you.

When you use the track option in /etc/shorewall/providers, connections from the Internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.

Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names and numbers of the tables that it adds.

The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation for an example.

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value.

Adding another entry in the providers file simply creates an alternate routing table for you (see the LARTC Howto). The table will usually contain two routes:

Note that the first route is omitted if "-" is specified as the GATEWAY; in that case, the default route does not specify a gateway (point-to-point link).

If the DUPLICATE column is non-empty, then routes from the table named in that column are copied into the new table. By default, all routes (except default routes) are copied. The set of routes copied can be restricted using the COPY column which lists the interfaces whose routes you want copied. You will generally want to include all local interfaces in this list. You should exclude the loopback interface (lo) and any interfaces that do not have an IP configuration. You should also omit interfaces like tun interfaces that are created dynamically. Traffic to networks handled by those interfaces should be routed through the main table using entries in /etc/shorewall/rtrules (see Example 2 below) or by using USE_DEFAULT_RT=Yes.

In addition:

That's all that these entries do. You still have to follow the principle stated in the Shorewall Routing documentation:

The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in /etc/shorewall/tcrules and you must do that marking in the PREROUTING chain; or, you must provide the appropriate rules in /etc/shorewall/rtrules.

Shorewall itself provides no mechanism for dealing with provider links that are in the up state but not responsive. If you want transparent failover when a link is unresponsive, you must configure all provider interfaces as optional (shorewall-interfaces(5)) then install and configure LSM.

Shorewall-init provides for handling links that go hard down and are later brought back up.

If you masquerade a local network, you will need to add masquerade rules for both external interfaces. Referring to the diagram above, if each of the interfaces has only a single IP address and you have no systems with public IP addresses behind your firewall, then I suggest the following simple entries:

#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27

If you have a public subnet (for example 206.124.146.176/30) behind your firewall, then use exclusion:

#INTERFACE       SOURCE               ADDRESS
eth0             !206.124.146.176/29  206.124.146.176
eth1             0.0.0.0/0            130.252.99.27

Note that exclusion is only used on the interface corresponding to internal subnetwork.

If you have multiple IP addresses on one of your interfaces, you can use a similar technique -- simplY exclude the smallest network that contains all of those addresses from being masqueraded.

One problem that often arises with Multi-ISP configuration is 'Martians'. If your Internet interfaces are configured with the routefilter option in /etc/shorewall/interfaces (remember that if you set that option, you should also select logmartians), then things may not work correctly and you will see messages like this:

Feb  9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 
Feb  9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00

The above message is somewhat awkwardly phrased. The source IP in this incoming packet was 64.86.88.116 and the destination IP address was 206.124.146.176. Another gotcha is that the incoming packet has already had the destination IP address changed for DNAT or because the original outgoing connection was altered by an entry in /etc/shorewall/masq (SNAT or Masquerade). So the destination IP address (206.124.146.176) may not have been the destination IP address in the packet as it was initially received.

There a couple of common causes for these problems:

If all else fails, remove the routefilter option from your external interfaces. If you do this, you may wish to add rules to log and drop packets from the Internet that have source addresses in your local networks. For example, if the local LAN in the above diagram is 192.168.1.0/24, then you would add this rule:

#ACTION          SOURCE                     DEST
DROP:info        net:192.168.1.0/24         all

Be sure the above rule is added before any other rules with net in the SOURCE column.

This section describes the legacy method of configuring multiple uplinks. It is deprecated in favor of the USE_DEFAULT_RT=Yes configuration described below.

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows.

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2

Other configuration files go something like this:

/etc/shorewall/interfaces:

#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …

/etc/shorewall/policy:

#SOURCE    DESTINATION    POLICY     LIMIT:BURST
net        net            DROP

/etc/shorewall/masq:

#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27

This continues the example in the preceding section.

Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and if you are running a version of Shorewall earlier than 3.0.0, you would set TC_ENABLED=Yes in /etc/shorewall/shorewall.conf).

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2:P             <local network> 0.0.0.0/0       tcp     25

Note that traffic from the firewall itself must be handled in a different rule:

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2               $FW             0.0.0.0/0       tcp     25

Shorewall provides considerable flexibility for port forwarding in a multi-ISP environment.

Normal port forwarding rules such as the following will forward from both providers.

/etc/shorewall/rules:

#ACTION        SOURCE             DEST              PROTO     DEST PORT(S)     SOURCE       ORIGINAL
#                                                                              PORTS(S)     DEST
DNAT           net                loc:192.168.1.3   tcp       25

Continuing the above example, to forward only connection requests from ISP 1, you can either:

When there are more than two providers, you need to extend the two-provider case in the expected way:

If we extend the above example to add eth3 with IP address 16.105.78.4 with gateway 16.105.78.254, then:

/etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2
ISP3    3       3       main            eth3            16.105.78.254   track,balance    eth2

/etc/shorewall/masq:

#INTERFACE       SUBNET            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27
eth3             0.0.0.0/0         16.105.78.4

As noted above, separate entries in /etc/shorewall/tcrules are required for traffic originating from the firewall.

Experience has shown that in some cases, problems occur with applications running on the firewall itself. This is especially true when you have specified routefilter on your external interfaces in /etc/shorewall/interfaces (see above). When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.

Examples:

Note that some traffic originating on the firewall doesn't have a SOURCE IP address before routing. At least one Shorewall user reports that an entry in /etc/shorewall/rtrules with 'lo' in the SOURCE column seems to be the most reliable way to direct such traffic to a particular ISP.

Example:

#SOURCE     DEST      PROVIDER        PRIORITY
lo          -         shorewall       1000

The rtrules file allows assigning certain traffic to a particular provider just as entries in the tcrules file. The difference between the two files is that entries in rtrules are independent of Netfilter.

gateway:~ # ip rule ls
0:      from all lookup local                <=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     <=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    <=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 <=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              <=== This table is usually empty
gateway:~ #

#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000

gateway:~ # ip rule ls
0:      from all lookup local
1000:   from all iif eth1 lookup Comcast
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #

#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000

Beginning with Shorewall 4.4.15, additional routes can be added to the provider routing tables using the /etc/shorewall/routes file.

The columns in the file are as follows.

Assume the following entry in /etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
Comcast      1     -    xxx             eth2            ....     

The following table gives some example entries in the file and the ip route command which results.

#PROVIDER     DEST             GATEWAY         DEVICE        |              Generated Command
Comcast       172.20.1.0/24    -               eth0          | ip -4 route add 172.20.1.0/24 dev eth0 table 1
Comcast       192.168.4.0/24   172.20.1.1                    | ip -4 route add 192.168.1.0/24 via 172.20.1.1 table 1
Comcast       192.168.4.0/24                                 | ip -4 route add 192.168.4.0/24 dev eth2 table 1 

Null routing is a type of routing which discards a given packet instead of directing it through a specific predefined route. Generally speaking, there are 3 different types of Null routing as indicated below: