ADFS 2012 R2 Web Application Proxy servers in Load Balanced Configuration loses trust with ADFS farm (Event ID 422).

TL;DR: If you have a load balanced ADFS farm, make sure you have the June 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2.

_________________________________________________________ 

We recently implemented ADFS 2012 R2 (aka ADFS 3.0) in our production environment to allow our internal domain credentials to be used with an outside application provider.

We setup two Web Application Proxy servers in the DMZ, with a load balancer in front of them.  We have another load balancer on the internal network that the traffic from the Web Application Proxy servers flows through to load balance the internal ADFS servers as well.

With this configuration, we can easily reboot any single server in the system for patches and maintenance, without having to schedule a downtime.  We are monitoring port 443 with the load balancers so as soon as 443 doesn't answer on a particular server, it is taken out of the load balancer config automatically.

After configuring all of this, we were pretty sure we had setup everything correctly to have this be a highly redundant system.  Now we just needed to test.

I was testing the load balancer configuration by rebooting different ADFS servers in our farm and next thing I know I can no longer reach the ADFS login page.  At first we thought this was related to the load balancers not recognizing the servers had come back up after a reboot, but from looking at the event logs on the web application proxy servers it was evident that we had other issues.

This error was occuring on both Web Application Proxy servers on a 1 minute interval.

The first thing we did to get things back up and running quickly was to run the following command (as one line) from each of the ADFS proxy servers to reestablish the trust.

 Install-WebApplicationProxy –CertificateThumbprint 'abcd1234' -FederationServiceName adfs.yourdomain.com

You can find your thumbprint in the details of your ADFS certificate.  This is not our ADFS thumbprint of course, but an example.



What this command should do it re-establish the ADFS trust and create self signed certs with whatever internal ADFS server it is currently communicating with.  After that, a process is supposed to run to copy the newly created trust config to the other ADFS servers in the farm.

This way you don't have to create a trust with each and every ADFS server.  You make a trust with one and the config is stored in the database and copied to the others ADFS servers in the farm.  Very cool design.  However, this is where it went wrong.  Without the device registration feature enabled, which was not needed for what we were trying to accomplish, this feature of distributing the proxy trust config to the other servers in the farm never happens.  So you have a trust with only one of the internal ADFS servers and this is a problem in a load balanced environment.

Researching this further, we found the following KB detailing our issue:

https://support.microsoft.com/kb/2964735

A fix was in the July 2014 Rollup located here:

https://support.microsoft.com/kb/2962409

Once we applied the roll up and rebooted, the issue never cropped back up.  I've been monitoring the event logs since then and everything is working as expected at this point, so looks like the July 2014 roll up took care of this issue.

VMware's VMotion Technologies

I wanted to write my blog post to expand on the topic of VMware’s VMotion technology.  Being an IT Professional, I have found this technology to be one of the most important advances in availability and redundancy.  To a system administrator, using VMware’s VMotion technology would’ve been thought of as a dream 10 years ago.  Coming from a 24/7 computing environment, getting agreement from users to allow system downtimes has always been a pain.  Whether you need to maintenance a server, like applying firmware or patches, or if you need to swap out to newer hardware, it always used to mean middle of the night or weekend work, which nobody likes if they can avoid.  What VMware’s VMotion technology has brought to the table is the ability to move the workload from one VMware server to another.  That is, if you have a server that is virtualized with VMware, it can actually move from one VMware “host” server to another without ever having a downtime.  I went to a VMware conference where I attended what they call a deep dive session on the subject and wanted to go more into depth on how it actually works.  I won’t go into the high level details about VMware guests and hosts as those topics were covered in the video lessons.

The two types of VMotion VMware has currently available is VMotion where it moves the workload from one server to another allowing you to free up the host server for maintenance, etc.  The other type of VMotion is called “Storage VMotion” that allows the administrator to move the actual data location of where the virtual server is located to another storage device or drive.  This is useful in many of the same ways that regular VMotion is useful in because it allows the administrators to do maintenance on the storage, or move to a different class of storage, etc.

The first technology VMotion technology I will talk about is regular VMotion (not Storage VMotion).  When you ask the system to VMotion a virtual server off of a host or physical server, the key is that the virtual machines, also known as VM’s, are stored on shared storage and shared network space.  This means that every host in your VMware farm can see the same data storage and network.  When a VMotion is performed, the active memory and execution state of the virtual machine is transferred to another host in your VMware farm.  A copy of the data in memory is moved to the host the moving guest will become live on, and then a delta copy of data is done until it reaches a small enough amounts of data where it can halt the virtual machine for literally a few milliseconds and copy the tiny delta left and then bring the system up on the new VMware host.  The user and operating system never notices that it is now on a new piece of physical hardware.  Once you VMotion all of the “guests” off of a VMware server, it is no longer hosting a workload and you can take it down for maintenance, replace it, or do whatever you need to do to the system.  I included a link to YouTube video that will give you some graphical representation of what I am talking about.

Video created by HPC Systems on Youtube.

The second VMotion technology I will talk about is Storage VMotion, and it works in a similar manner to the above concept.  When you need to move the location of the VMware files to new storage, you can use storage VMotion to accomplish this.  It copies the virtual machine files to new storage, and then does a delta copy, and finally when it has such a small amount to copy left it momentarily halts the execution of code and brings the VMware guest up on the new storage.  This is different than regular VMotion because during a regular VMotion, the data stays on the same storage, but just moves to another physical server.  I included another link to a YouTube video that further explains the technology I’ve been discussing.

Video created by Varrow on Youtube.

These technologies have really changed the lives of system administrators as I mentioned above, giving much more flexibility in how we do our work.  Sometimes when using VMware at work, I almost feel like I am cheating because I’m doing things that used to be so difficult and time consuming and VMware has made those same tasks so easy and manageable.

I’ve included some links so you can do some more reading on  the VMotion technologies and other amazing technologies by VMware.  They are really an innovative company and I always look forward to what new technologies they are working on releasing in the future.

VMotion PDF From VMware's Site

Storage VMotion PDF From VMware's Site

Wikipedia Article on VMware



IPv4 vs IPv6 basic overview

I’d like to expand on the different IP models, IP version 4 (IPv4) vs. IP version 6 (IPv6).  Many people are aware that each computer or internet connected device must have a unique IP address that provides connectivity to the internet.  Normally this happens by way of a DHCP server, which is a server that handles the delivery of IP addresses to end devices.  For example, when you connect to the internet, your internet provider has a DHCP server that automatically issues an IP address based on a pool of IP’s it has.  Once your computer has been assigned the IP, you can then connect to and browse the internet.

I took a screenshot of my computer’s IP address; you can see it under the section IPv4 address.

Right now, IPv4 addresses are made up of four sections and each section is called an octet.  This is because each section represents a group of 8 bits which when combined together create an IP address.  The IPv4 protocol can have as many as 4.3 billion possible address combinations.  This sounds like a lot of addresses, but now that the world is increasingly relying on the internet for business, entertainment, and day to day activities, the number of IP addresses will eventually run out.  This is one of the reasons IPv6 will be implemented.  IPv6 will be able to support 2128 billion IP addresses.  This is a huge jump in the number of available IP addresses between the two protocols.

Here is a great video that goes over some IP basics and talks about IPv4 vs. IP v6

Luckily, most of the IP information needed by your computer is automatically negotiated (via DHCP server discussed before) between your computer and your ISP (Internet service provider), so you really don’t  need to know the specifics between the two protocols for your computer to work, but it’s really interesting to see how something so complex functions. 

Another really great feature that we will get when IPv6 is implemented as the mainstream IP delivery technology will be what is called IPsec (Internet security protocol).  IPsec can be used in IPv4, but is mandatory in the IPv6 protocol.  This means that traffic will be encrypted by default where with IPv4 traffic; you actually had to configure encryption to work.  This will greatly increase the security of internet communications.

I’ve included some links below to read more about IPv6 if anyone is interested:

Wiki on IPv6

IPv6 Website

Microsoft Site on IPv6

I hope you have enjoyed learning more about a techonolgy that we will be moving to in the future.  Although, if implemented correctly, you will never even notice a change when you are moved to the new system.

RAID Array Basics



Computer hard drives that are combined together to setup a RAID configuration is something we learned about in the videos on computer hardware (Weeks 2 and 3).  The acronym RAID stands for "Redundant Array of Inexpensive Disks".

An example of a hard drive that could be used to create a RAID array.  You would need at least two of these drives.  Photo credit: Public domain from (http://publicphoto.org/)



The ability to use multiple hard drives that act like one drive to a computer operating system is a topic I wanted to explore further.  Setting up a RAID array on your computer can offer performance or redundancy, or in some configurations can offer both advantages.

The videos in week 2 and 3 taught us about RAID 0 (Striping) and RAID 1 (Mirroring).  Sometimes I get the types of RAID configurations mixed up, and one easy way that I use to remember the difference between RAID 0 and RAID 1 is to think of the zero in RAID 0 to mean you get zero redundancy protection when using that configuration.  Although you have the most performance with RAID 0 because the data being written has the combined horsepower of two hard drives, if one of the hard drives in the array were to stop working, you would lose all the data on both drives since the data is striped across the set of drives.  Striping means some of the data is written to one drive and some of the data written to the other, which is why you get the speed increase.   In other words, make sure you have good backups of your data.  You could equate the performance increase of RAID 0 to a car that has two engines.  The car is going to be able to drive much faster with two engines; however, the engines in this car rely on each other to work so if one fails you have complete engine failure.

Alternatively, RAID 1 mirrors the data across both drives so if one drive were to fail, you would still be able to access your data that is an exact copy on the other working drive.  When a drive failure occurs, you would want to replace the failed drive as soon as possible to let the data replicate from the working drive to the replaced drive so you are once again in a “mirrored” configuration.  This method of RAID doesn’t offer any performance increase because the same data is being written to both hard drives at one time for redundancy purposes.

In the below video, I show a business class server (a computer that would go into a datacenter) that is setup in a RAID 1 configuration.

My HP DL360 Server with a RAID 1 configuration

There are other types of RAID configurations than just RAID 0 and RAID 1.  Since the server in the video only has two drive bays, it would be limited in the types of RAID configurations you can setup with it.   Other RAID configurations, such as RAID 5 require a minimum of 3 hard drives.  RAID 5 is one of the types of RAID configurations that offer both performance and redundancy as mentioned above.

A good overview of the most common types of RAID configurations can be found here.  I liked this site because it limits its scope to only the most common RAID configurations and also describes the advantages and disadvantages of each one.  You can read more about RAID 5 here.

http://gcn.com/Articles/2009/08/31/GCN-Lab-Review-NAS-sidebar-RAID.aspx

I hope my blog about RAID explained why you would want to use this technology and the different options to boost performance and reliability of your computer system.