Especially for new users understanding how NAT and the traffic processing works under the hood with PAN-OS might take some time. I always wanted to thoroughly follow a “simple” NAT policy and see all the steps in action. Maybe it helps someone new to the topic, but also writing about this helps me to internalize and refresh how this works.
Resources: Packet Flow Sequence in PAN-OS (Diagram is helpful)
The setup
The setup is a HA-pair PAN-OS VM firewalls. For this example the HA setup is not relevant. Let me try to explain what we have here:
There is a firewall PA1-n1 which has 5 dataplane ports utilized:
ethernet1/1: ISP1 (Zone: z-outside)ethernet1/2: ISP2 (Zone: z-outside)ethernet1/3: Zone: z-DMZ1ethernet1/4: Zone: z-SERVERS1ethernet1/5: Zone: z-CLIENTS1
The default route with the lowest metric (highest priority) goes via ISP1 - The router has the IP address 203.0.113.1 the firewall has 203.0.113.2 assigned.
The firewall interfaces look like this
web1 10.3.3.10
This is our internal webserver listening on port 80.
ext-inc 192.0.2.123
This is considered an external client coming from the internet into our network. Our public IP-address range is 203.0.113.0/24 for ISP1.
What has to be done so that ext-inc can reach web1 from the internet?
We already setup the default route. Our outside interface ethernet1/1 has the IP address 203.0.113.2 assigned. We want to use a new public IP address for the webserver and decide to use 203.0.113.10. It is not required to add that IP to our outside interface, a NAT policy suffices.
Do we need a NAT policy for the return traffic?
No, if the traffic gets initiated from the outside and a DNAT policy hits, the firewall will implicitly apply the correct NAT for the return traffic for that session.
Excursus: What does the Bi-directional checkbox do?
Since NAT is applied automatically for the same session when using a DNAT policy, enabling Bi-directional would only make sense if you want to initiate traffic from both ends.
If you enable Bi-directional traffic PAN-OS will create two NAT policies in the background, that you can verify with show running nat-policy via CLI.
In my opinion it’s better to create two policies if you need SNAT and DNAT for the following reasons:
- The second implicit policy that gets created isn’t shown in the web UI. You always have to remember that it exists.
- No fine grained policy for the automatically created policy. (for example zones)
- No possibility to filter all policies for zones, because the implicitly created policy isn’t shown
- For example the filter
(to/member eq 'z-outside')shows all NAT policies with original packets heading to thez-outsidezone. This might be useful if you want to check what could potentially be forwarded to your internal network. If you have aBi-directionalpolicy enabled on a SNAT policy, you won’t see it there.
- For example the filter
This might be worth a blogpost in the future (to check what really happens and where it can go wrong.)
NAT policy
We’d like to match traffic coming from the internet to one of our public IPs.
Now, the “Original Packet” part here might already be confusing if you just start with networking or PAN-OS, I remember it was for me.
You might notice that the zone that the traffic eventually lands in (z-DMZ1) is not referenced in our NAT policy at all. This is because the original packet comes from the z-outside zone and is directed to an IP address that is also in the z-outside zone. We want to decide later what we do with traffic that looks like this. The policy is evaluated from left to right.
IF
- traffic comes from zone
z-outsideAND - traffic is directed to the zone
z-outsideAND - traffic is directed to the IP address
203.0.113.10AND - traffic is directed to the port
80AND - there is no policy before this that matches the traffic
THEN
- translate the destination to
10.3.3.10(the firewall will determine later which zone that IP has)
Security policy
Now buckle up, if you’re new. Let’s discuss the required security policy.
What do you think: Which destination zone and destination IP has to be allowed?
. . .
. .
.
Are you surprised?
We have to allow the public (before NAT) IP and our (after NAT) internal zone.
Security Policy: Pre-NAT IP, Post-NAT zone.
The reason for that is that the security policy is enforced after the NAT policy has been evaluated, but before NAT is applied.
What does evaluated mean? The firewall does the evaluation a little bit like the bullet points from the last step and then determines that the target zone of 10.3.3.10 is z-DMZ1.
Similar to the NAT policy the return traffic for that session is automatically allowed without having to add another policy (content inspection might intervene though).
Policy setup for the test
With a packet capture and flow basic logs we can have deeper insights about what happens. For this I have a very minimal setup. The default route is set and we have only one NAT and one security policy.
I added dummy policies that should never hit, because the order and index might be interesting later.
NAT
Security
Here the same diagram from the top again:
The client from outside uses curl to do a web request via CLI
curl http://203.0.113.10
the request should find its way to our internal server web1 that has IP address 10.3.3.10 eventually.
After the test: What the web-GUI shows
The entry from the traffic monitor looks like this
Checking the session browser looks like this. You can see the name of the NAT policy in the session monitor, but not in the traffic logs.
After the test: What the traffic capture shows
For this I merged the receive and transmit capture, so that you can see traffic pre and post NAT. There is an info in the Comment column. Receive captures are pre-NAT, transmit captures are post-NAT.
If you just check the firewall stage alone it looks like the following screenshot, which gives a better impression about the traffic between both systems. Firewall captures are pre-NAT.
After the test: What do the flow basic logs show?
With flow basic logs you can get deeper insights in the decision making of the firewall. We “just” use a simple example without any security profiles for further inspection, but there is already a lot to see.
first packet (SYN)
This is the first packet initiated by the external IP 192.0.2.123. The screenshot shows the packet before NAT has been applied (receive) the second one where NAT got applied. (transmit)
Step 1: ingress stage
== 2024-08-31 12:50:31.008 +0200 ==
Packet received at ingress stage, tag 0, type ORDERED
Packet info: len 74 port 16 interface 16 vsys 1 TC 0
wqe index 27803 packet 0x0xd01e1fe900, HA: 0, IC: 0
Packet decoded dump:
L2: 0c:1f:37:cd:00:00->0c:3b:46:ea:00:01, type 0x0800
IP: 192.0.2.123->203.0.113.10, protocol 6
version 4, ihl 5, tos 0x00, len 60,
id 54228, frag_off 0x4000, ttl 63, checksum 24937(0x6961)
TCP: sport 35790, dport 80, seq 1158301534, ack 0,
reserved 0, offset 10, window 64240, checksum 54310,
flags 0x02 ( SYN), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 b4 04 02 08 0a 09 ac 58 2f 00 00 00 00 ........ ..X/....
00000010: 01 03 03 07 ....
Flow lookup, msgtype 0, wp.sport 35790,wp.dport 80, wp.l4info 2345533520 key word0 0x600018bce0050 word1 0 word2 0x7b0200c0ffff0000 word3 0x0 word4 0xa7100cbffff0000
* Dos Profile NULL (NO) Index (0/0) *
Session setup: vsys 1
No active flow found, enqueue to create session
The packet is received at the ingress stage.
No active flow found, enqueue to create session
No session is found, so it gets forwarded to the stage that sets up the session known as “slowpath”.
Step 2: slowpath stage
== 2024-08-31 12:50:31.008 +0200 ==
Packet received at slowpath stage, tag 434973284, type ATOMIC
Packet info: len 74 port 16 interface 16 vsys 1 TC 0
wqe index 27803 packet 0x0xd01e1fe900, HA: 0, IC: 0
Packet decoded dump:
L2: 0c:1f:37:cd:00:00->0c:3b:46:ea:00:01, type 0x0800
IP: 192.0.2.123->203.0.113.10, protocol 6
version 4, ihl 5, tos 0x00, len 60,
id 54228, frag_off 0x4000, ttl 63, checksum 24937(0x6961)
TCP: sport 35790, dport 80, seq 1158301534, ack 0,
reserved 0, offset 10, window 64240, checksum 54310,
flags 0x02 ( SYN), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 b4 04 02 08 0a 09 ac 58 2f 00 00 00 00 ........ ..X/....
00000010: 01 03 03 07 ....
Session setup: vsys 1
PBF lookup (vsys 1) with application none
Session setup: ingress interface ethernet1/1 egress interface ethernet1/1 (zone 1)
2024-08-31 12:50:31.008 +0200 debug: pan_policy_lookup(pan_policy.c:2413): [ACE] Trigger slow match for appid(0) uappid(0)
2024-08-31 12:50:31.008 +0200 debug: pan_policy_match_service(pan_policy.c:1614): match 0,0 for app 0 uapp 0 proto 6 sport 35790 dport 80
NAT policy lookup, matched rule index 2
Destination NAT, translated IP 10.3.3.10
PBF lookup (vsys 1) with application none
Session setup: egress zone 2 for natted IP
Translated IP in zone 2, egress id 18
2024-08-31 12:50:31.008 +0200 No netconfig available for souce zone to read prenat options
2024-08-31 12:50:31.008 +0200 debug: pan_policy_lookup(pan_policy.c:2413): [ACE] Trigger slow match for appid(0) uappid(0)
2024-08-31 12:50:31.008 +0200 debug: pan_policy_match_service(pan_policy.c:1614): match 0,0 for app 0 uapp 0 proto 6 sport 35790 dport 80
Policy lookup, matched rule index 4,
TCI_INSPECT: Do TCI lookup policy - appid 0
Allocated new session 19528.
set exclude_video in session 19528 0xd02d655280 0 from work 0xd01c543000 0
no swg configured
Packet matched vsys 1 NAT rule '' (index 32771),
destination translation 203.0.113.10/80 => 10.3.3.10/80
no NPB policy
Created session, enqueue to install. work 0xd01c543000 exclude_video 0,session 19528 0xd02d655280 exclude_video 0
PBF lookup (vsys 1) with application none
shows that there is no Policy Based Forwarding policy.
NAT policy lookup, matched rule index 2
A NAT policy for this traffic pattern exists with the index 2. Remember we had some dummy policies in front of the real one? Let’s look at it again.
In the GUI it shows as policy 3. What does the CLI output show?
admin@PA1-n1(active)> show running nat-policy
Nat policy configured on vsys1:
"Dummy DNAT1; index: 1" {
nat-type ipv4;
from z-outside;
source any;
source-region none;
to z-outside;
to-interface ;
destination 203.0.113.201;
destination-region none;
service 0:any/any/any;
translate-to "dst: 10.3.3.10";
terminal no;
}
"Dummy DNAT2; index: 2" {
nat-type ipv4;
from z-outside;
source any;
source-region none;
to z-outside;
to-interface ;
destination 203.0.113.202;
destination-region none;
service 0:any/any/any;
translate-to "dst: 10.3.3.10";
terminal no;
}
"DNAT web1 http; index: 3" {
nat-type ipv4;
from z-outside;
source any;
source-region none;
to z-outside;
to-interface ;
destination 203.0.113.10;
destination-region none;
service 0:tcp/any/80;
translate-to "dst: 10.3.3.10";
terminal no;
}
Okay, index 3, the same. That’s weird. Turns out that the basic flow logs start counting with 0. But only for policies. Zones for example match what we see on the firewall. For example with
> debug device-server dump idmgr type zone all
ID Version Peer Version Name
---------- ---------- --------------------
1 7 7 vsys1+z-outside
2 7 7 vsys1+z-dmz1
3 7 7 vsys1+z-servers1
4 7 7 vsys1+z-clients1
Type: 12 Last id: 5 Current Version: 7 Current Peer Version: 7 Mismatch cnt: 0
So, we just have to remember that the flow logs start counting with 0 and the default GUI and CLI output not.
That’s a little bit confusing for my taste, let’s all start counting with 0 everywhere, like normal people. :)
Policy lookup, matched rule index 4
This is the same for security policies.
Step 3: fastpath stage
== 2024-08-31 12:50:31.008 +0200 ==
Packet received at fastpath stage, tag 19528, type ATOMIC
Packet info: len 74 port 16 interface 16 vsys 1 TC 4
wqe index 27803 packet 0x0xd01e1fe900, HA: 0, IC: 0
Packet decoded dump:
L2: 0c:1f:37:cd:00:00->0c:3b:46:ea:00:01, type 0x0800
IP: 192.0.2.123->203.0.113.10, protocol 6
version 4, ihl 5, tos 0x00, len 60,
id 54228, frag_off 0x4000, ttl 63, checksum 24937(0x6961)
TCP: sport 35790, dport 80, seq 1158301534, ack 0,
reserved 0, offset 10, window 64240, checksum 54310,
flags 0x02 ( SYN), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 b4 04 02 08 0a 09 ac 58 2f 00 00 00 00 ........ ..X/....
00000010: 01 03 03 07 ....
Flow fastpath, session 19528 c2s (set work 0xd01c543000 exclude_video 0 from sp 0xd02d655280 exclude_video 0)
IP checksum valid
* Dos Profile NULL (NO) Index (0/0) *
* Dos Profile NULL (NO) Index (0/0) *
2024-08-31 12:50:31.010 +0200 pan_flow_process_fastpath(src/pan_flow_proc.c:4777): SESSION-DSCP: set session DSCP: 0x00
NAT session, run address/port translation
Syn Cookie: pan_reass(Init statete): c2s:0 c2s:nxtseq 1158301535 c2s:startseq 1158301535 c2s:win 0 c2s:st 3 c2s:newsyn 0 :: s2c:nxtseq 0 s2c:startseq 0 s2c:win 64240 s2c:st 0 s2c:newsyn 0 ack 0 nosyn 0 plen 0
CP-DENY TCP non data packet getting through
Forwarding lookup, ingress interface 16
L3 mode, router 2
Route lookup in router 2, IP 10.3.3.10
Route found, interface ethernet1/3, zone 2
Resolve ARP for IP 10.3.3.10 on interface ethernet1/3
ARP entry found on interface 18
Transmit packet size 60 on port 18
NAT session, run address/port translation
NAT gets applied here.
Resolve ARP for IP 10.3.3.10 on interface ethernet1/3
ARP for the destination IP gets resolved, so we know the target IP exists.
Port numbers
References to the port numbers can be checked with show interfaces all (or the specific interface) via CLI
For our experiment we see that the ingress interface was 16 -> ethernet1/1 and the egress interface 18 -> ethernet1/3
second packet (SYN-ACK)
The first packet was SYN, it was processed and forwarded by the firewall. Now the webserver replies with a SYN-ACK.
Step 1: Ingress stage
== 2024-08-31 12:50:31.024 +0200 ==
Packet received at ingress stage, tag 0, type ORDERED
Packet info: len 66 port 18 interface 18 vsys 1 TC 0
wqe index 27796 packet 0x0xd01e2d8900, HA: 0, IC: 0
Packet decoded dump:
L2: 0c:7e:25:e6:00:00->0c:3b:46:ea:00:03, type 0x0800
IP: 10.3.3.10->192.0.2.123, protocol 6
version 4, ihl 5, tos 0x00, len 52,
id 26110, frag_off 0x4000, ttl 64, checksum 15877(0x53e)
TCP: sport 80, dport 35790, seq 1693018326, ack 1158301610,
reserved 0, offset 8, window 1017, checksum 54787,
flags 0x10 ( ACK), urgent data 0, l4 data len 0
TCP option:
00000000: 01 01 08 0a b6 5a ce 62 09 ac 58 3c .....Z.b ..X<
Flow lookup, msgtype 0, wp.sport 80,wp.dport 35790, wp.l4info 5278670 key word0 0x6000200508bce word1 0 word2 0xa03030affff0000 word3 0x0 word4 0x7b0200c0ffff0000
Flow 39057 found, msgtype 0, state 2, HA 0
Active flow, enqueue to fastpath process, type 0
Active flow, enqueue to fastpath process, type 0
In the ingress stage for the first packet we got No active flow found, enqueue to create session and so it was forwarded to the slowpath stage. This time it’s different. We get:
Flow 39057 found, msgtype 0, state 2, HA 0
Active flow, enqueue to fastpath process, type 0
This means we need no “session setup” / slowpath stage here and can directly continue to the fastpath stage.
Step 2: Fastpath stage
== 2024-08-31 12:50:31.024 +0200 ==
Packet received at fastpath stage, tag 19528, type ATOMIC
Packet info: len 66 port 18 interface 18 vsys 1 TC 0
wqe index 27796 packet 0x0xd01e2d8900, HA: 0, IC: 0
Packet decoded dump:
L2: 0c:7e:25:e6:00:00->0c:3b:46:ea:00:03, type 0x0800
IP: 10.3.3.10->192.0.2.123, protocol 6
version 4, ihl 5, tos 0x00, len 52,
id 26110, frag_off 0x4000, ttl 64, checksum 15877(0x53e)
TCP: sport 80, dport 35790, seq 1693018326, ack 1158301610,
reserved 0, offset 8, window 1017, checksum 54787,
flags 0x10 ( ACK), urgent data 0, l4 data len 0
TCP option:
00000000: 01 01 08 0a b6 5a ce 62 09 ac 58 3c .....Z.b ..X<
Flow fastpath, session 19528 s2c (set work 0xd01c542900 exclude_video 0 from sp 0xd02d655280 exclude_video 0)
IP checksum valid
NAT session, run address/port translation
CP-DENY TCP non data packet getting through
Forwarding lookup, ingress interface 18
L3 mode, router 2
Route lookup in router 2, IP 192.0.2.123
Route found, interface ethernet1/1, zone 1, nexthop 203.0.113.1
Resolve ARP for IP 203.0.113.1 on interface ethernet1/1
ARP entry found on interface 16
Transmit packet size 52 on port 16
NAT session, run address/port translation
NAT for the return traffic gets applied.
We could go on for a long time with all those packets we captured. But the processing is very similar, with the exception that the firewall checks the “App-ID” at one point: We allowed web-browsing only in the security policy and there is a check if this is really HTTP traffic.
Significant fields
Session ID
With a packet capture under production conditions you will most likely receive a huge flow output. Within the flow basic log the session ID is a good value to match the relevant traffic. The session ID is also shown in the web-UI.
Sequence number
Good for comparing with packet capture output.
Header Checks
Also good for comparing with packet capture output.
Files
If you want to check some of this for yourself, here are the raw capture files from the test.
Download: PAN-OS_DNAT-test_raw-files.zip
contains
- firewall.pcap
- receive.pcap
- transmit.pcap
- receive-transmit-merged.pcapng
- pan_packet_diag.log (basic flow log)
All this for a little web request
All this just happened because I made one little web request. For me it’s still fascinating and humbling that all this and much more is constantly happening on our personal computers and servers. And most of the time it even works!