U-Turn NAT is a method for accessing an internal server by making it possible to connect to its external IP address, which is also part of the internal network.

Understanding how to configure a connection with U-Turn NAT is one aspect, while comprehending the underlying processes is another. To gain a deeper insight, I delved into the network-level processes and the logic of the firewall.

For this post it’s important to know that every 172.16.x.x address is considered external and not RFC 1918. All networks configured on the firewall are /24.

Situation

What’s working

Working nat

As you can see in the diagram we have a webserver that has the internal IP address 10.10.100.10. From outside of the company the server should be reachable via the public IP address 172.16.30.10. For this we have a destination NAT policy.

Everyone from within the z-muenster-server zone can reach the internet with the use of a source NAT policy. An external server only sees 172.16.30.2 for this traffic.

It’s also possible that 10.10.100.101 can reach 10.10.100.10 directly and vice versa. They are in the same layer 2 network and this traffic wouldn’t traverse the firewall in this example (they are on the same switch).

What’s not working

Not working

Here you can see a diagram for the traffic that’s not working: The server with the IP 10.10.100.10 is also reachable via 172.16.30.10 externally, but if we try to establish a connection to the external address it doesn’t work.

I don’t want to spoil the fun, so let’s explore together…

What’s happening

For troubleshooting purposes PAN-OS, which is the operating system of Palo Alto Networks Firewalls, has a feature called flow basic logs. The way you do this is by specifying a filter for which traffic you want to log, enable the flow basic logs and then replicate the traffic you want to test. Usually you want to do a packet capture at the same time.

For the packet capture you have different stages which can be stored in different files:

  • receive
  • firewall
  • transmit
  • drop

More about stages later.

The traffic

As seen before, traffic originating from outside has no problem reaching the server. Someone external with IP address 1.2.3.4 can access our webserver with the public IP address 172.16.30.10. Destination NAT gets applied on the firewall.

Back to the traffic flow in the last diagram, which is not working: For capturing the traffic and logs I’ll send one single ICMP packet from 10.10.100.101 to 172.16.30.10.

Destination NAT policy to server

Here you can see what the original packet for the NAT policy looks like

And here that there is a destination NAT (same policy)

But we can’t seem to establish a connection from within the same zone to the webserver while clients from outside the company can reach it perfectly fine.

Packet capture

So let’s first look at the packet capture, before digging deeper:

Receive stage

For the receive stage we see that the firewall got the ICMP packet we send before. The receive stage shows the traffic as it is received by the ingress port.

Firewall stage

The receive and firewall stage captures are 1:1 identical. This is the stage when a new session is created or if there is an existing session already. More on that later.

Drop stage

Now we see that there is traffic in the drop stage, which means that the firewall dropped the traffic.

But why is that? You might have an idea already. But let’s confirm with the basic flow logs feature.

Basic flow logs

Info: I slightly highlighted the lines that I think are interesting

Ingress stage

Looking at the ingress stage it shows the received packet.

 1Packet received at ingress stage, tag 0, type ORDERED
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 68035 packet 0x0xe01d27c100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 53538, frag_off 0x0000, ttl 64, checksum 64880(0x70fd)
 9ICMP:   type 8, code 0, checksum 64825, id 8913, seq 1
10Flow lookup, msgtype 0, wp.sport 8,wp.dport 0, wp.l4info 524288 key word0 0x1000722d10001 word1 0  word2 0x65640a0affff0000 word3 0x0 word4 0xa1e10acffff0000
11Session setup: vsys 1
12No active flow found, enqueue to create session

In case you’re interested what ORDERED and ATOMIC means, here is a KB article on that.

Slowpath stage

Because there was no active session the next stage is slowpath.

 1Packet received at slowpath stage, tag 435579907, type ATOMIC
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 68035 packet 0x0xe01d27c100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 53538, frag_off 0x0000, ttl 64, checksum 64880(0x70fd)
 9ICMP:   type 8, code 0, checksum 64825, id 8913, seq 1
10Session setup: vsys 1
11PBF lookup (vsys 1) with application ping
12Session setup: ingress interface ethernet1/7.20 egress interface ethernet1/1 (zone 2)
132023-08-20 10:38:58.674 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(0) uappid(0)
142023-08-20 10:38:58.674 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 2,0 for app 0 uapp 0 proto 1 sport 8 dport 0
152023-08-20 10:38:58.674 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(0) uappid(0)
162023-08-20 10:38:58.674 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 3,0 for app 0 uapp 0 proto 1 sport 8 dport 0
17NAT policy lookup, matched rule index 2
182023-08-20 10:38:58.674 +0200 quarantine lookup: src IP 10.10.100.101 src_quarantine 0
192023-08-20 10:38:58.674 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(318) uappid(0)
202023-08-20 10:38:58.674 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 3,0 for app 318 uapp 0 proto 1 sport 8 dport 0
21Policy lookup, matched rule index 3, 
22TCI_INSPECT: Do TCI lookup policy - appid 318
23TCI_DECAP: policy not available for vsys 1
24Allocated new session 24114.
25set exclude_video in session 24114 0xe02f29fd80 0 from work 0xe01b8f5600 0
26Rule: index=2 name=SNAT outside, cfg_pool_idx=2 cfg_fallback_pool_idx=0
27NAT Rule: name=SNAT outside, cfg_pool_idx=2; Session: index=24114, nat_pool_idx=2
28Packet matched vsys 1 NAT rule 'SNAT outside' (index 3),
29source translation 10.10.100.101/8913 => 172.16.30.2/8913
30no NPB policy 
31Created session, enqueue to install. work 0xe01b8f5600 exclude_video 0,session 24114 0xe02f29fd80 exclude_video 0

We see that a session is set up and the first weird thing is on line 12. It shows the egress interface ethernet1/1 which is the z-outside zone. But my server isn’t outside. It’s internal, in the same zone z-muenster-server.

Then we see a lookup for the NAT policy with the name SNAT outside. This is our main source NAT policy which all zones use to connect to the internet.

Then in line 29 we see that the source will later be translated from 10.10.100.101 to 172.16.30.2

Fastpath stage

 1Packet received at fastpath stage, tag 24114, type ATOMIC
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 68035 packet 0x0xe01d27c100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 53538, frag_off 0x0000, ttl 64, checksum 64880(0x70fd)
 9ICMP:   type 8, code 0, checksum 64825, id 8913, seq 1
10Flow fastpath, session 24114 c2s (set work 0xe01b8f5600 exclude_video 0 from sp 0xe02f29fd80 exclude_video 0)
11IP checksum valid
122023-08-20 10:38:58.675 +0200  pan_flow_process_fastpath(src/pan_flow_proc.c:4672): SESSION-DSCP: set session DSCP: 0x00
13NAT session, run address/port translation
142023-08-20 10:38:58.675 +0200 quarantine lookup: src IP 10.10.100.101 src_quarantine 0
15Forwarding lookup, ingress interface 260
16L3 mode, router 2
17Route lookup in router 2, IP 172.16.30.10
18Route found, interface ethernet1/1, zone 2
19Resolve ARP for IP 172.16.30.10 on interface ethernet1/1
20Resolve ARP
21resolve ARP on interface ethernet1/1 for IP 172.16.30.10
22Packet dropped, no ARP

Since we now have a session ID (32011) we can enter the fastpath stage: NAT gets applied. Which means we have a packet with

Source: 172.16.30.2 Destination: 172.16.30.10

But then on line 21-22 we see that resolving ARP fails on ethernet1/1 and the packet is dropped 🤔

Let’s show the ARP table for ethernet1/1 at that moment:

Oh, this failed, what if we try…

Hm. That’s weird. Checking our DNAT policy from before:

We explicitly specified that the original packet should come from the the zone z-outside.

Okay, then there might be an easy solution: Just add z-muenster-server to the source zone of the NAT policy, right?

After adding the zone to the NAT policy we notice that indeed the ICMP packet went through successfully, it reached the server interface… But, let’s take a closer look

When the packet reaches the server it looks exactly as if we had pinged the server internally, which usually works. However, what’s wrong? Why didn’t we receive a response?

Let’s take a closer look at the packets we captured:

server packet capture

From the servers perspective it looked like the echo request came from 10.10.100.10, so it sends an echo reply back. If I’d see this isolated without context there is nothing suspicious.

client packet capture

But we pinged 172.16.30.10 initially from the client. So we’re not expecting a reply from 10.10.100.10 for that ping request. We’re waiting for 172.16.30.10 😤

That’s why the ping is not successful.

Solution: U-Turn NAT

To make this kind of traffic possible we need a new policy that handles what happens when an internal IP from the same zone sends traffic to the public IP address 172.16.30.10

That’s basically the idea.

If traffic is send from the server zone to 172.16.30.10 give it a source IP from the same zone. In this example it’s the firewall gateway IP 10.10.100.1.

Info: We could just add other internal zones to the policy and the traffic would work if security policies exist. But it would be confusing, because this traffic doesn’t need source translation then. We can clone the policy without the source translation for other zones.

If you’re interested to know how the same zone U-Turn NAT looks like on a flow basic log level, keep on reading.

Ingress stage

 1Packet received at ingress stage, tag 0, type ORDERED
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 50839 packet 0x0xe01d191100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 63179, frag_off 0x0000, ttl 64, checksum 21579(0x4b54)
 9ICMP:   type 8, code 0, checksum 21524, id 52214, seq 1
10Flow lookup, msgtype 0, wp.sport 8,wp.dport 0, wp.l4info 524288 key word0 0x10007cbf60001 word1 0  word2 0x65640a0affff0000 word3 0x0 word4 0xa1e10acffff0000
11Session setup: vsys 1
12No active flow found, enqueue to create session

Packet arrives at the firewall and gets send to the next stage

Slowpath stage

 1Packet received at slowpath stage, tag 3158222251, type ATOMIC
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 50839 packet 0x0xe01d191100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 63179, frag_off 0x0000, ttl 64, checksum 21579(0x4b54)
 9ICMP:   type 8, code 0, checksum 21524, id 52214, seq 1
10Session setup: vsys 1
11PBF lookup (vsys 1) with application ping
12Session setup: ingress interface ethernet1/7.20 egress interface ethernet1/1 (zone 2)
132023-08-20 13:19:39.293 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(0) uappid(0)
142023-08-20 13:19:39.293 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 0,0 for app 0 uapp 0 proto 1 sport 8 dport 0
152023-08-20 13:19:39.293 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(0) uappid(0)
162023-08-20 13:19:39.293 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 1,0 for app 0 uapp 0 proto 1 sport 8 dport 0
172023-08-20 13:19:39.293 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(0) uappid(0)
182023-08-20 13:19:39.293 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 4,0 for app 0 uapp 0 proto 1 sport 8 dport 0
19NAT policy lookup, matched rule index 0
20Destination NAT, translated IP 10.10.100.10
21PBF lookup (vsys 1) with application ping
22Session setup: egress zone 7 for natted IP
23Translated IP in zone 7, egress id 260
242023-08-20 13:19:39.295 +0200 quarantine lookup: src IP 10.10.100.101 src_quarantine 0
252023-08-20 13:19:39.295 +0200 debug: pan_policy_lookup(pan_policy.c:2402): [ACE] Trigger slow match for appid(318) uappid(0)
262023-08-20 13:19:39.295 +0200 debug: pan_policy_match_service(pan_policy.c:1613): match 3,0 for app 318 uapp 0 proto 1 sport 8 dport 0
27Policy lookup, matched rule index 3, 
28TCI_INSPECT: Do TCI lookup policy - appid 318
29TCI_DECAP: policy not available for vsys 1
30Allocated new session 32011.
31set exclude_video in session 32011 0xe02f67af80 0 from work 0xe01b4c2a00 0
32no swg configured
33Rule: index=0 name=U-Turn Server Zone, cfg_pool_idx=4 cfg_fallback_pool_idx=0
34NAT Rule: name=U-Turn Server Zone, cfg_pool_idx=4; Session: index=32011, nat_pool_idx=4
35Packet matched vsys 1 NAT rule 'U-Turn Server Zone' (index 1),
36source translation 10.10.100.101/52214 => 10.10.100.1/52214
37destination translation 172.16.30.10/1 => 10.10.100.10/1
38no NPB policy 
39Created session, enqueue to install. work 0xe01b4c2a00 exclude_video 0,session 32011 0xe02f67af80 exclude_video 0

Because there was no matching traffic the packet goes to the slowpath stage. There is a NAT policy lookup with the info that source and destination NAT will be applied later and that the corresponding policy is named U-Turn Server Zone.

Remember that the last time, we had the default SNAT policy here. And instead of just destination translation we also see source translation.

Fastpath stage

 1Packet received at fastpath stage, tag 32011, type ATOMIC
 2Packet info: len 102 port 22 interface 260 vsys 1
 3  wqe index 50839 packet 0x0xe01d191100, HA: 0, IC: 0
 4Packet decoded dump:
 5L2:     00:50:79:66:68:04->0c:81:9b:e4:00:07, VLAN 20 (0x8100 0x0014), type 0x0800
 6IP:     10.10.100.101->172.16.30.10, protocol 1
 7        version 4, ihl 5, tos 0x00, len 84,
 8        id 63179, frag_off 0x0000, ttl 64, checksum 21579(0x4b54)
 9ICMP:   type 8, code 0, checksum 21524, id 52214, seq 1
10Flow fastpath, session 32011 c2s (set work 0xe01b4c2a00 exclude_video 0 from sp 0xe02f67af80 exclude_video 0)
11IP checksum valid
122023-08-20 13:19:39.298 +0200  pan_flow_process_fastpath(src/pan_flow_proc.c:4672): SESSION-DSCP: set session DSCP: 0x00
13NAT session, run address/port translation
142023-08-20 13:19:39.299 +0200 quarantine lookup: src IP 10.10.100.101 src_quarantine 0
15Forwarding lookup, ingress interface 260
16L3 mode, router 2
17Route lookup in router 2, IP 10.10.100.10
18Route found, interface ethernet1/7.20, zone 7
19Resolve ARP for IP 10.10.100.10 on interface ethernet1/7.20
20ARP entry found on interface 260
21Transmit packet size 84 on port 22

A session now exists, NAT gets applied, route lookup shows an interface that belongs to the z-muenster-server zone, there is an ARP entry for our server 10.10.100.10 and the packet is send.

Packet capture for successful traffic

I think at this point I have a feeling for what’s going on and don’t need to follow up on the ICMP reply for basic flow logs. But in the packet capture you can see the entire traffic flow.

client traffic

server traffic

Conclusion

I think I now have a more in depths picture on what happens under the hood when I configure a U-Turn NAT policy. This might be helpful when troubleshooting firewalls in production with a lot more traffic flow and where the log and capture files are bigger.

While the output can be quite verbose at times, the firewall is not a black box. I appreciate that you can closely observe what’s happening on the firewall.