Pages

Friday, January 15, 2010

Windows Server 2008 Freezes -- Finally Solved!


In our environment at work, we have a Citrix farm that users connect to that is running on Windows Server 2008 x64.  During our testing phase while we still had a relatively light load of users on the farm, things went pretty smoothly.  As we added more and more users to the Citrix environment, different issues cropped up here and there, but none as horribly evil as our servers freezing to the point of becoming completely unresponsive.  At that point, all sessions that users were in would lock up, forcing them to lose any unsaved data and restart their sessions again.  As you can imagine, management did not see this as an enhancement to their productivity.

So, for the last several months we have been troubleshooting this issue.  There was no pattern in regards to when servers would freeze.  At any given time, any of the four servers we have in production would freeze.  There was also no consistent user base on the server that would freeze (the only consistency being that they weren't too happy when it would happen).  After bringing in several consultants that helped set up this environment initially, we took our case to Citrix.  Several log and memory dump files later, they came to the conclusion that Internet Explorer was causing our servers to lock up.  Naturally, I then presented this information all to Microsoft support.  Upon further analysis, they discovered that we were experiencing a bug that has been resolved by a hotfix:
http://support.microsoft.com/kb/976674

Basically, the hotfix resolves an issue that occurs when Server 2008 or Windows Vista is under a heavy load and there are a lot of network share accesses going on.  Well, in our case, the user profile is a network share, plus their Outlook PST files were out on a network share, plus their other file shares were network shares, and the list goes on.  After applying this hotfix (which was a little over two weeks ago) we have not experienced any freezes.  Good news for everyone.

If anyone is interested in the detailed symptoms:

  • Users sessions (terminal services/Citrix) would become completely unresponsive
  • The server would become unresponsive even at the console level
  • The server would respond to pings
  • Apparently anything in memory at the time of the freeze would continue to function -- as soon as you tried to access something else, the session would freeze
  • The only workaround when this occurred was to hard reboot the server

23 comments:

Anonymous said...

Ugh! Solving problems like these reminds us (and often only us techs) why the world needs us. It's a good thing we have a reward waiting for us in heaven, but it's sure not likely to be dished out sufficiently here on earth!

Matt Augustine said...

Thanks for the stopping by and leaving a comment! My hope is definitely in Heaven... because technology is an ever-changing, ever-challenging beast :)

Anonymous said...

Just wondering if the fix is still solving the problem. We have been having similiar issues on our servers

Matt Augustine said...

Yep -- as long as we had 2008 installed it was what solved the problem for us. We actually recently moved to R2, which ended up creating the freeze problem all over again, but applied a different hotfix for that one :)

Brett said...

We're seeing similar behaviour here. Do you know if there is any way to confirm whether or not this is the issue? I'm hesitant to apply that hotfix without some sort of solid proof that this is what is occurring.

Thanks for the post, this is definitely the most hopeful I've been about solving this thing in a while!

Matt Augustine said...

Hey Brett!
I'm glad that you found this to be useful. Unfortunately, I don't think there is any way to know with 100% certainty without opening a case with Microsoft and sending them a memory dump.

If your symptoms (and environment) are the same as mine were, is there a server you could use to test on and then roll back if necessary? That's what I would recommend if possible.

Anonymous said...

MAtt, I'm having that pbm with my 2008 r2 ! What hotfix solved your problem ?

Matt Augustine said...

With R2 there is a whole different Hotfix that may or may not help you. The one that ultimately helped us was: KB 2465772. If you still have problems after installing that hotfix, I would also try: KB 2265716 and KB 2383928. Good luck!

Anonymous said...

we still have the issue. tryed the hotfix a year ago , and it went away a couple mounth , but it came back, now im pretty damn tierd of my job..

Matt Augustine said...

I completely understand the exhaustion and frustration that comes along with troubleshooting a Terminal Services/Citrix environment. If you have any kind of support agreement with Microsoft, I would suggest, if nothing else, opening up an email case with them. It provides 24-hour response times, but I've had good success with them!

phunktional said...

Hi Matt,

We are experiencing similar situation as well but terminal servers are Windows 2003 based. Any ideas to point me in right direction?

Matt Augustine said...

Hi Punktional,

Unfortunately, we just started using Citrix, with our initial installation being with Server 2008, so I really don't have any experience with 2003.

That said -- there are a few things you can do: If you have support (either Microsoft or Citrix) you should be able to open a case and they're usually pretty good about getting to the bottom of an issue (that has been the case for me anyway). If you don't have any support, then Google is your best friend. Between searching, and checking out the Citrix support forums, you'll want to look for the exact symptoms that you're seeing.

If you're not alone (which you probably aren't) with the problem, something is bound to show up in a search result.

Best of luck!

Anonymous said...

Great stuff, This has worked for us too. XenApp server used to freez and need hard reboot but after this patch its more than a week and its fine.
Thanks fo

Anonymous said...

Matt,

Were there any errors in the event log or any other possible warning signs that you saw with this? We have a similar random issue.

Matt Augustine said...

Hi, Unfortunately I can't remember if there were any specific error messages in the logs -- but I do remember the servers becoming unresponsive, even at the console level. Basically, when I attached to them via KVM, I didn't even get video out. The odd thing though was that they were still ping-able. The best suggestion I can make is to open a case with Microsoft and collect a memory dump for them while in the frozen state (assuming you've already tried the hotfix).

Anonymous said...

Hi Matt,
Thanks for this fix, We have a 2008 with the same problem. I have apply this patch a week ago and we don't have any freeze for the moment.
I just want to clarify, We have 2008 server with TSE but no XenApp service and the server was randomly freeze but ping-able with events error with wsearch and Outlook process.
Thanks from France !!!

David said...

The typical error in the event log would in my experience be: A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Service service.

Anonymous said...

Well, I just installed this hotfix. Tonight all of my 7 rig's will reboot and the fix is active. I'm suffering from this problem since christmas 2011!! I really hope this is the one! I'll report back :)

Anonymous said...

has anyone tried this one?

http://support.microsoft.com/kb/980568

Anonymous said...

I'm having issues with 2 servers, one we installed Aug 2012 and now the new one Jan 2013, The first 2008 r8 "Stops" responding and locks up. I was debugging asp on the new server and it stopped serving pages, I could log in and had full access. I moved a test asp up in the Default Doc's and nothing. We rebooted and now everything is fine. Both servers are 2008 R2. Is this the same problem from 2010's "Hotfix" or something new?

Anonymous said...

I have a Windows 2008 R2 SP1 Terminal Service farm (RDS) with nine servers. We also run a product called RES Workspace Manager that controls the user desktop. We are getting this issue!! Our servers freeze and users cannot connect to their sessions, we have to reboot the boxes to allow them to reconnect. I have applied the various fixes mentioned in this document but we are still seeing the issue. The servers have been rebuilt a few times and they seem fine for a while but then the issue comes back. Does anybody else still get this issue have there been any recent MS update that are known to resolve the problem.
Thanks,
LB

Pwilliams WallstreetSystems said...

I have several Windows 2008 R2 terminal servers not in a server farm at this time. Most End users use with one application with RDP. I have had both servers lock up without reason at the exact same time twice now. I now suspect a problem with redirected folder server, network, or license server. I am exporting system and application event logs to investigate. I will deploy hot fix and report back my status next time this happens.

IT.teleconnsinc said...

Hi guys, just wanna ask if the hotfix still works by this time and what hotfix should i use for Windows Server 2008R2. Also , may I ask how long will it take for the fix to complete?Gladly appreciate your help and fast reply :D