Heartbeat

Each node in the cluster sends out a multicast heart beat that tells the other member of the cluster that it is alive and healthy. By default a cluster node will consider another node dead if it misses the heartbeat from that node for 10 seconds.

The interface used for heartbeats is configured in the cluster.conf file (see configuration section for more details ). When discussion cluster configuration with Redhat support they highly recommended that a cross-connect between the two nodes is not used and that an interface connected to a switch and a private VLAN be used for heartbeats. They also recommended that this be the same interface used to initiate fencing (See below).

Quorum

One of the most dangerous situations that can happen in clusters is that both nodes become active at the same time. This is especially true for clusters that share storage resources. In this case both cluster nodes could be writing to the data on shared storage which will quickly cause data corruption.

When both nodes becoming active it is called “split brain” and can happen when a cluster stops receiving heartbeats from its partner node. Since the two nodes are no longer communicating they do not know if the problem is with the other node or if the problem is with itself.

For example say that the passive node stops receiving heartbeats from the active node due to a network failure of the heartbeat network. In this case if the passive node starts the cluster services then you would have a split-brain situation.

Most clusters use a Quorum Disk to prevent this from happening. The Quorum Disk is a small shared disk that both nodes can access at the same time. Whichever node is currently the active node writes to the disk periodically (usually every couple of seconds) and the passive node checks the disk to make sure the active node is keeping it up to date.

When a node stops receiving heartbeats from its partner node it looks at the Quorum Disk to see if it has been updated. If the other node is still updating the Quorum Disk then the passive node knows that the active node is still alive and does not start the cluster services.

Redhat clusters support Quorum Disks, but Redhat support had recommended not to use one since they are difficult to configure and can become problematic. Instead they recommend to relying on Fencing to prevent split brain.

Fencing

One of the strategies used by Redhat clusters to prevent split brain is a concept called fencing.

While there are several different types of fencing, fencing via the HP iLO devices (or similar) built into the servers is the recommended method. With this type of Fencing when the passive node stops receiving heartbeats from the active node it will connect to the iLO of the active node and reboot the active node. Once the passive node reboots (i.e. fences) the active node it will then start the cluster services.

By rebooting the active node the passive node can be sure that the active node is no longer running the cluster services and it is safe to start them.

From a design point of view the NIC used to connect to the iLO of a node’s partner server is the NIC that should also be used for heartbeat. This ensures that the node that lost its connection to the heartbeat network cannot fence its partner

ASA 8.3/8.4 NAT Migration Lab Guide – Lab 1.1

This lab is part of the series of LAB which details how migrate NAT configurations from Pre ASA 8.2 version to ASA 8.3/8.4

Lab1.1 Setup

Dynamic Policy NAT / PAT

Building on what we had before lets add one more router to the picture and consider this as DMZ. We will also configure policy NAT for DMZ subnet such that it chooses a different IP address based on the destination.

The device configurations and GNS3 Topology can be downloaded from the the following link if you want it to import it for yourself.

NAT Policy

1. Configure ASA for DMZ network such that when subnet 11.11.11.0/24 tries to ping ISP router it uses a public address 192.168.100.202.
2. Configure ASA for DMZ network such that when subnet 11.11.11.0/24 tries to ping Inside router it uses IP address 192.168.0.202

Pre ASA 8.3 Configuration

access-list POLICY-NAT-ACL-11 permit ip 11.11.11.0 255.255.255.0 192.168.0. 255.255.255.0
access-list POLICY-NAT-ACL-11 permit ip 11.11.11.0 255.255.255.0 192.168.100.0 255.255.255.0

nat (dmz) 1 access-list POLICY-NAT-ACL-11
global (outside) 1 192.168.100.202
global (inside) 1 192.168.0.202

ASA 8.3/8.4 Configuration

1. First we need to create network objects for source subnet. Then we need to create object for destination subnet and finally we need to create to objects for addresses which will be used for translation. So, lets start with 11.11.11.0/24

object network DMZ-Source-11.11.11.0
subnet 11.11.11.0 255.255.255.0

object network DMZ-Destination-192.168.100.0
subnet 192.168.100.0 255.255.255.0

object network obj-192.168.100.202
host 192.168.100.202

2. We use the following NAT statement such that NAT is performed if subnet 11.11.11.0 tries to access ISP.

nat (dmz,outside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.100.202 destination static DMZ-Destination-192.168.100.0 DMZ-Destination-192.168.100.0

3. Now becuase the source subnet is going to remain the remain, we just need a new object to define the NAT address and destination subnet

object network DMZ-Destination-192.168.0.0
subnet 192.168.0.0 255.255.255.0

object network obj-192.168.0.202
host 192.168.0.202

4. Here is the NAT statement for subnet 11.11.11.0/24 when it tried to access subnets on the Inside router.

nat (dmz,inside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.0.202 destination static DMZ-Destination-192.168.0.0 DMZ-Destination-192.168.0.0

Verification

1. Use 'show run object' will show the objects that we created in step 1 & 3
Output:
ASA1# sh run object
object network Inernal-10.10.10.0
subnet 10.10.10.0 255.255.255.0
object network Inernal-10.10.11.0
subnet 10.10.11.0 255.255.255.0
object network Inernal-0.0.0.0
subnet 0.0.0.0 0.0.0.0
object network DMZ-Source-11.11.11.0
subnet 11.11.11.0 255.255.255.0
object network DMZ-Destination-192.168.100.0
subnet 192.168.100.0 255.255.255.0
object network obj-192.168.100.202
host 192.168.100.202
object network DMZ-Destination-192.168.0.0
subnet 192.168.0.0 255.255.255.0
object network obj-192.168.0.202
host 192.168.0.202

2. Use 'show run nat' to get the NAT statements used in the running config
ASA1# sh run nat
nat (DMZ,outside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.100.202 destination static DMZ-Destination-192.168.100.0 DMZ-Destination-192.168.100.0
nat (DMZ,inside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.0.202 destination static DMZ-Destination-192.168.0.0 DMZ-Destination-192.168.0.0
!
object network Inernal-10.10.10.0
nat (inside,outside) dynamic interface
object network Inernal-10.10.11.0
nat (inside,outside) dynamic 192.168.100.200
object network Inernal-0.0.0.0
nat (inside,outside) dynamic 192.168.100.201

3. Use 'show nat' to see the translations performed
ASA1# sh nat
Manual NAT Policies (Section 1)
1 (DMZ) to (outside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.100.202 destination static DMZ-Destination-192.168.100.0 DMZ-Destination-192.168.100.0
translate_hits = 0, untranslate_hits = 0
2 (DMZ) to (inside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.0.202 destination static DMZ-Destination-192.168.0.0 DMZ-Destination-192.168.0.0
translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (inside) to (outside) source dynamic Inernal-10.10.10.0 interface
translate_hits = 0, untranslate_hits = 0
2 (inside) to (outside) source dynamic Inernal-10.10.11.0 192.168.100.200
translate_hits = 0, untranslate_hits = 0
3 (inside) to (outside) source dynamic Inernal-0.0.0.0 192.168.100.201
translate_hits = 0, untranslate_hits = 0

Now we have verified that components are in place let start verifciation on the devices

4. On ISP router use command 'debug ip packets' to get an idea what IP address ISP sees when a packet hits its interface
5. Use extended ping from DMZ router as following to verify the first rule. Now ISP router should see the IP (192.168.100.202) as a source of ping. However, as we configured and we know that the real source is the Loopback0 Interface IP on ther DMZ Router.

DMZ#ping
Protocol [ip]:
Target IP address: 192.168.100.2
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 11.11.11.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.100.2, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/36/72 ms

ISP Router IP Packet Debug Output

*Mar 1 08:50:44.227: IP: tableid=0, s=192.168.100.202 (FastEthernet1/0), d=192.168.100.2 (FastEthernet1/0), routed via RIB
*Mar 1 08:50:44.227: IP: s=192.168.100.202 (FastEthernet1/0), d=192.168.100.2 (FastEthernet1/0), len 100, rcvd 3
*Mar 1 08:50:44.231: IP: s=192.168.100.202 (FastEthernet1/0), d=192.168.100.2, len 100, stop process pak for forus packet
*Mar 1 08:50:44.231: IP: s=192.168.100.2 (local), d=192.168.100.202 (FastEthernet1/0), len 100, sending
*Mar 1 08:50:44.231: IP: s=192.168.100.2 (local)
ISP#, d=192.168.100.202 (FastEthernet1/0), len 100, sending full packet

6. Now lets try again but this time with different destinatio IP. This time we will use Loopback0 to ping Inside Router interface and it should be translated with IP 192.168.0.2002. Let try that and see what Iinside router thinks the packets coming from

DMZ#ping
Protocol [ip]:
Target IP address: 192.168.0.2
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 11.11.11.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/32/72 ms

Inside Router IP Packet Debug Output
*Mar 1 08:53:05.867: IP: tableid=0, s=192.168.0.202 (FastEthernet1/0), d=192.168.0.2 (FastEthernet1/0), routed via RIB
*Mar 1 08:53:05.867: IP: s=192.168.0.202 (FastEthernet1/0), d=192.168.0.2 (FastEthernet1/0), len 100, rcvd 3
*Mar 1 08:53:05.867: IP: s=192.168.0.202 (FastEthernet1/0), d=192.168.0.2, len 100, stop process pak for

8. Let use the 'show nat' command to see if the hit count is still is same or increased

Manual NAT Policies (Section 1)
1 (DMZ) to (outside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.100.202 destination static DMZ-Destination-192.168.100.0 DMZ-Destination-192.168.100.0
translate_hits = 2, untranslate_hits = 2
2 (DMZ) to (inside) source dynamic DMZ-Source-11.11.11.0 obj-192.168.0.202 destination static DMZ-Destination-192.168.0.0 DMZ-Destination-192.168.0.0
translate_hits = 1, untranslate_hits = 1

So, this lab ends with successfull implementation of policy NAT where we demonstrated that NAT is performed based on the source and destination IP address.

Knowledge is Power

Tuesday, January 21, 2014

Cluster Terms

ASA 8.3/8.4 NAT Migration Lab Guide – Lab 1.1 - Dynamic Policy NAT / PAT

ASA 8.3/8.4 NAT Migration Lab Guide – Lab 1.1