So, for the last several months we have been troubleshooting this issue. There was no pattern in regards to when servers would freeze. At any given time, any of the four servers we have in production would freeze. There was also no consistent user base on the server that would freeze (the only consistency being that they weren't too happy when it would happen). After bringing in several consultants that helped set up this environment initially, we took our case to Citrix. Several log and memory dump files later, they came to the conclusion that Internet Explorer was causing our servers to lock up. Naturally, I then presented this information all to Microsoft support. Upon further analysis, they discovered that we were experiencing a bug that has been resolved by a hotfix:
http://support.microsoft.com/kb/976674
Basically, the hotfix resolves an issue that occurs when Server 2008 or Windows Vista is under a heavy load and there are a lot of network share accesses going on. Well, in our case, the user profile is a network share, plus their Outlook PST files were out on a network share, plus their other file shares were network shares, and the list goes on. After applying this hotfix (which was a little over two weeks ago) we have not experienced any freezes. Good news for everyone.
If anyone is interested in the detailed symptoms:
- Users sessions (terminal services/Citrix) would become completely unresponsive
- The server would become unresponsive even at the console level
- The server would respond to pings
- Apparently anything in memory at the time of the freeze would continue to function -- as soon as you tried to access something else, the session would freeze
- The only workaround when this occurred was to hard reboot the server
23 comments:
Ugh! Solving problems like these reminds us (and often only us techs) why the world needs us. It's a good thing we have a reward waiting for us in heaven, but it's sure not likely to be dished out sufficiently here on earth!
Thanks for the stopping by and leaving a comment! My hope is definitely in Heaven... because technology is an ever-changing, ever-challenging beast :)
Just wondering if the fix is still solving the problem. We have been having similiar issues on our servers
Yep -- as long as we had 2008 installed it was what solved the problem for us. We actually recently moved to R2, which ended up creating the freeze problem all over again, but applied a different hotfix for that one :)
Hey Brett!
I'm glad that you found this to be useful. Unfortunately, I don't think there is any way to know with 100% certainty without opening a case with Microsoft and sending them a memory dump.
If your symptoms (and environment) are the same as mine were, is there a server you could use to test on and then roll back if necessary? That's what I would recommend if possible.
MAtt, I'm having that pbm with my 2008 r2 ! What hotfix solved your problem ?
With R2 there is a whole different Hotfix that may or may not help you. The one that ultimately helped us was: KB 2465772. If you still have problems after installing that hotfix, I would also try: KB 2265716 and KB 2383928. Good luck!
we still have the issue. tryed the hotfix a year ago , and it went away a couple mounth , but it came back, now im pretty damn tierd of my job..
I completely understand the exhaustion and frustration that comes along with troubleshooting a Terminal Services/Citrix environment. If you have any kind of support agreement with Microsoft, I would suggest, if nothing else, opening up an email case with them. It provides 24-hour response times, but I've had good success with them!
Hi Matt,
We are experiencing similar situation as well but terminal servers are Windows 2003 based. Any ideas to point me in right direction?
Hi Punktional,
Unfortunately, we just started using Citrix, with our initial installation being with Server 2008, so I really don't have any experience with 2003.
That said -- there are a few things you can do: If you have support (either Microsoft or Citrix) you should be able to open a case and they're usually pretty good about getting to the bottom of an issue (that has been the case for me anyway). If you don't have any support, then Google is your best friend. Between searching, and checking out the Citrix support forums, you'll want to look for the exact symptoms that you're seeing.
If you're not alone (which you probably aren't) with the problem, something is bound to show up in a search result.
Best of luck!
Great stuff, This has worked for us too. XenApp server used to freez and need hard reboot but after this patch its more than a week and its fine.
Thanks fo
Matt,
Were there any errors in the event log or any other possible warning signs that you saw with this? We have a similar random issue.
Hi, Unfortunately I can't remember if there were any specific error messages in the logs -- but I do remember the servers becoming unresponsive, even at the console level. Basically, when I attached to them via KVM, I didn't even get video out. The odd thing though was that they were still ping-able. The best suggestion I can make is to open a case with Microsoft and collect a memory dump for them while in the frozen state (assuming you've already tried the hotfix).
Hi Matt,
Thanks for this fix, We have a 2008 with the same problem. I have apply this patch a week ago and we don't have any freeze for the moment.
I just want to clarify, We have 2008 server with TSE but no XenApp service and the server was randomly freeze but ping-able with events error with wsearch and Outlook process.
Thanks from France !!!
The typical error in the event log would in my experience be: A timeout (30000 milliseconds) was reached while waiting for a transaction response from the Service service.
Well, I just installed this hotfix. Tonight all of my 7 rig's will reboot and the fix is active. I'm suffering from this problem since christmas 2011!! I really hope this is the one! I'll report back :)
has anyone tried this one?
http://support.microsoft.com/kb/980568
I'm having issues with 2 servers, one we installed Aug 2012 and now the new one Jan 2013, The first 2008 r8 "Stops" responding and locks up. I was debugging asp on the new server and it stopped serving pages, I could log in and had full access. I moved a test asp up in the Default Doc's and nothing. We rebooted and now everything is fine. Both servers are 2008 R2. Is this the same problem from 2010's "Hotfix" or something new?
I have a Windows 2008 R2 SP1 Terminal Service farm (RDS) with nine servers. We also run a product called RES Workspace Manager that controls the user desktop. We are getting this issue!! Our servers freeze and users cannot connect to their sessions, we have to reboot the boxes to allow them to reconnect. I have applied the various fixes mentioned in this document but we are still seeing the issue. The servers have been rebuilt a few times and they seem fine for a while but then the issue comes back. Does anybody else still get this issue have there been any recent MS update that are known to resolve the problem.
Thanks,
LB
I have several Windows 2008 R2 terminal servers not in a server farm at this time. Most End users use with one application with RDP. I have had both servers lock up without reason at the exact same time twice now. I now suspect a problem with redirected folder server, network, or license server. I am exporting system and application event logs to investigate. I will deploy hot fix and report back my status next time this happens.
Hi guys, just wanna ask if the hotfix still works by this time and what hotfix should i use for Windows Server 2008R2. Also , may I ask how long will it take for the fix to complete?Gladly appreciate your help and fast reply :D
Same situation at time of writing. No idea which update has caused my DC to misbehave.
https://support.microsoft.com/en-gb/help/972685/fix-the-explorer-exe-process-stops-responding-hangs-when-you-use-windo
Post a Comment