Archive | August 2013

Add a Local Controller to an Existing Master

Once a network has scaled up beyond a single controller capacity, a master/local architecture is recommended.  This allows the master controller to handle configuration and correlation while the locals can be responsible for handling the user traffic. 

Steps to Add a local controller to an existing Master:

From the Master GUI:

Configuration -> Network -> Controller -> System Settings

–          Click New Local Controller IPSec Keys

–          Enter the IP Address of the local controller that you are adding

–          Enter an IPSec key

–          Re-Enter the IPSec key

–          Click Add

–          Click Apply (don’t forget)

–          Save Configuration

Or From the Master CLI:

localip <local ip address> ipsec <ipsec key>


From the Local GUI:

Configuration -> Network -> Controller -> System Settings

–          Select Controller Role Drop-Down and select Local

–          Enter the Master Controller IP (VRRP address if using master redundancy)

–          Enter IPSec Key  (Same key you entered on Master)

–          Click Apply

–          Reboot Controller

Or From the Local CLI:

masterip <master ip address> ipsec <ipsec key>


AP Troubleshooting

In order to make the WLAN function, the access points need connectivity to the controller.  Let’s review what an AP does during the boot process

Acquire IP address (can be static or acquired from DHCP)

  • IP address is required for communication with the controller using PAPI and GRE
  • To verify the access point properly acquires an IP you will need console access to the AP

Discover Controller

  • The AP goes through the following process in trying to discover a controller 
  1. Statically assigned
  2.  DHCP Vendor Option 43
  3. ADP Multicast: Group Address (requires multicast routing to be enabled on infrastructure)
  4. ADP L2 Broadcast
  5. DNS (aruba-master.<localsuffix>
  • The AP will follow the sequence exactly as above.  Once the AP learns a controller address, it terminates the discovery process and attempts to communicate with the learned address.  If the AP doesn’t receive a response from the learned address, then the AP will initiate a full reboot and start the process again

Update Code if necessary

  • AP Compares the code level to the controller’s code level
  • If the code revision matches the AP will continue the boot process
  • If the code revision does not match the AP will obtain the new code from the controller using FTP (TFTP is used on the initial join or if the AP is purged
  • The AP will automatically reboot after the code upgrade/downgrade
  • “show ap database” shows the current status of each AP (will list if upgrading, rebooting, etc)

Obtain Configuration Information

  • Once an AP connects to a controller and has compatible code, it will receive its configuration over PAPI
  • “show ap config ap-name <ap-name>” will show the AP configuration being pushed to the AP

Build GRE Tunnel

  • GRE is used to carry all of the wireless traffic between the AP and the local controller
  • A GRE tunnel is created per SSID per AP
  • The AP System Profile Controller LMS-IP setting tells the AP which controller the AP should terminate with
  • Be sure to allow Protocol 47 between the controllers and APs
  • “show ap debug system-status ap-name <ap-name>” – shows the communication status between the controller and AP
  • “show datapath tunnel table”shows the GRE tunnels established with the controller (look for prt 47)
  • “show ap debug counters”shows how many times an AP has rebooted or bootstrapped

Enable Radio

  • Once the GRE tunnel has been established the Radios will become enabled
  • “show profile-errors” – shows the list of invalid user created profiles.  An invalid user profile could cause the AP not to broadcast its assigned SSIDs.  

Aruba – Network Ports Needed

Aruba - Network Ports Needed

Aruba – Network Ports Needed

VRRP Troubleshooting

When you experience VRRP issues they are typically caused by incorrect provisioning or misconfiguration.  Here are some common failure scenarios:

  • Are the two controllers L2 adjacent? (VRRP uses L2 multicast exchanges between VRRP participants)
  •  Did you us a globally unique VRRP instance?
  • Did you assign the correct VLAN for the VRRP instance?
  • Did you set priority values correctly to determine which controller should be primary?
  • Did you enable passwords?  If so do they match?
  • Are the controllers running the same code revision?
  • Are the WMS and local user databases being synched (master redundancy only)?

Now that we know what the common VRRP issues are, let’s figure out the best ways to track down each issue.

VLAN Mismatch: The best command to determine network connectivity between the VRRP instanaces is: “show vrrp <VRRP ID> statistics”.  If the VLAN is not properly provisioned or configured then you will not see any advertisements being sent or received.  You can also look at the network log which will show the VRID is not configured “show log network 5”

Passphrase Mismatch:  The “show vrrp <VRRP ID> statistics” command can also be used to track down passphrase mismatches.  If you did not enter matching passphrases on both controllers you will authentication failures.

For a quick summary of how you have your VRRP configured you can use the show vrrp | include <VRRP ID>” command.  This will show the VIP, MAC, VLAN, priority, and preemption status

To verify whether or not the databases are synching properly use the “show database synchronization” command.  This command will tell the last time that the databases were synched.  It will also show you if there have been any sync failures.

Master Controller Redundancy – Configuration

I’m going to walk through the steps for configuring master redundancy.  Here is my scenario:

Primary Master – (should always be the master if up)

Backup Master –


VRRP ID – 10

VLAN – 10

First let’s start with the VRRP configuration on the Primary Master. 

config t

vrrp 10

vlan 10

ip address

priority 110


description Preferred-Master

tracking master-up-time 30 add 20

no shut

Now let’s configure the Backup-Master

config t

vrrp 10

vlan 10

ip address

priority 100


description Backup-Master

tracking master-up-time 30 add 20

no shut

Once VRRP is up I need to associate the VRRP instance with master controller redundancy:

On the Primary:

config t


master-vrrp 10

peer-ip-address ipsec aruba123

On the Backup:

config t


master-vrrp 10

peer-ip-address ipsec aruba123

Once the VRRP instance is associated with master redundancy then I need to synchronize the WMS and local user database between the two controllers:

config t

database synchronize period <minutes> – defines the scheduled time to sync the databases (minimum should be 20 minutes)

Controller Redundancy

For my first technical deep dive let’s get into controller redundancy. During this post I will define the different types of redundancy in the Aruba system.  Please no controller vs controller-less rants!

Let’s begin by defining redundancy.  According to Wikipedia, redundancy is the duplication of critical components or functions of a system with the intention of increasing reliability of the system.  Unfortunately redundancy is left off a lot of wireless network designs due to cost.  In today’s mobility first environments redundancy needs to be implemented properly to ensure the reliability of the mission critical WLAN.

In Aruba world we have four levels of controller redundancy:

1)      Fully redundant – includes both master and local redundancy

2)      Redundancy aggregation – local redundancy

3)      Hot Standby – Local access points fail-over to Master

4)      No Redundancy – self-explanatory (far too common)

Master Redundancy:

The first controller redundancy model we will look at is Master redundancy.  The master controller is the control plane of the centralized WLAN.  The master controller is responsible for handling the global configuration of the WLAN system, location tracking, IDS event correlation and alerting.  The first question we should ask ourselves is what happens if the master controller is unavailable?  If the master becomes unavailable all master functions are lost (configuration, location tracking, and IDS) but the WLAN itself will continue to function.  New and existing clients will still be able to access the WLAN while the master controller is down.

To provide redundancy for the master controller we will setup a master/standby relationship with two controllers.  The Standby Master is a hot standby controller.  The Standby Master will not terminate AP sessions while it is the backup unit.  Updates on the state of the network are sent from the active Master to the Backup.  The two controllers sync the databases (WMS and local user) at a configured interval (typically 30 minutes).

VRRP (Virtual Router Redundancy Protocol) is used as the redundancy mechanism between the two controllers.  VRRP requires Layer2 adjacency.  The two master controllers will use a shared VRRP interface address.  The VRRP address is used by local controllers, access points, and mobility access switches to discover the master controller on the network.  The VRRP address can also be used by network administrators to access the management interface for the current master controller.

Local Redundancy:

Next we will look at Local Controller Redundancy.  The Local Controllers in an Aruba WLAN are responsible for AP termination, user authentication, and policy enforcement.  If a local controller fails and there is no backup the WLAN will become unavailable.

Local controllers have three methods for redundancy


  • two locals share a set of APs,  divide the load, acts as a backup for each other
  • if the two controllers are L2 adjacent, run two instances of VRRP with each controller acting as a primary for one instance and backup for the other instance
  • if the two controllers are not L2 adjacent then you will need to setup a LMS/Backup LMS IP address in the AP System Profile
  • You can also combine VRRP and LMS/Backup LMS for a more robust redundancy design, the VRRP addresses can be used as the LMS/Backup LMS IP addresses


  • similar to Active-Active except one controller sits idle while the primary controller supports the full loads of APs and users
  • this model has a larger failure domain (increases latency because the full load must failover to the backup
  • typically this model utilizes the LMS/Backup LMS configuration
  • you could also use a single VRRP instance if the controllers are L2 adjacent

Many to One

  • typically used in remote networks where branch offices have local mobility controllers but redundancy onsite is not feasible
  • a large controller is deployed as the +1 controller at the data center
  • failure typically occurs across a WAN link
  • preemption should be enabled in this scenario due to the possible delay introduced by failing over to a remote site

No Redundancy

  • if the local goes down, no users can connect
  • Any AMs associated go down

Now that we know the different types of redundancy options we need to be aware of a few rules to ensure our network stays up according to plan.  There are four major rules in dealing with controller redundancy:

  1. Make sure the redundant controller can support the additional AP load during a failover event
  2. Make sure the same VLANs exist on both controllers and that named VLANs are mapped on the redundant controller
  3. Make sure the controllers are running the same OS version
  4. Make sure the redundant controller has the same license features enable and ensure you have enough license capacity to support the additional AP load during a failover event (AOS 6.3 will address this previous limitation)

In my next post I will begin configuring each of the different redundancy methods.