So, for the last several months we have been troubleshooting this issue. There was no pattern in regards to when servers would freeze. At any given time, any of the four servers we have in production would freeze. There was also no consistent user base on the server that would freeze (the only consistency being that they weren't too happy when it would happen). After bringing in several consultants that helped set up this environment initially, we took our case to Citrix. Several log and memory dump files later, they came to the conclusion that Internet Explorer was causing our servers to lock up. Naturally, I then presented this information all to Microsoft support. Upon further analysis, they discovered that we were experiencing a bug that has been resolved by a hotfix:
http://support.microsoft.com/kb/976674
Basically, the hotfix resolves an issue that occurs when Server 2008 or Windows Vista is under a heavy load and there are a lot of network share accesses going on. Well, in our case, the user profile is a network share, plus their Outlook PST files were out on a network share, plus their other file shares were network shares, and the list goes on. After applying this hotfix (which was a little over two weeks ago) we have not experienced any freezes. Good news for everyone.
If anyone is interested in the detailed symptoms:
- Users sessions (terminal services/Citrix) would become completely unresponsive
- The server would become unresponsive even at the console level
- The server would respond to pings
- Apparently anything in memory at the time of the freeze would continue to function -- as soon as you tried to access something else, the session would freeze
- The only workaround when this occurred was to hard reboot the server
16 comments:
Ugh! Solving problems like these reminds us (and often only us techs) why the world needs us. It's a good thing we have a reward waiting for us in heaven, but it's sure not likely to be dished out sufficiently here on earth!
Thanks for the stopping by and leaving a comment! My hope is definitely in Heaven... because technology is an ever-changing, ever-challenging beast :)
Just wondering if the fix is still solving the problem. We have been having similiar issues on our servers
Yep -- as long as we had 2008 installed it was what solved the problem for us. We actually recently moved to R2, which ended up creating the freeze problem all over again, but applied a different hotfix for that one :)
We're seeing similar behaviour here. Do you know if there is any way to confirm whether or not this is the issue? I'm hesitant to apply that hotfix without some sort of solid proof that this is what is occurring.
Thanks for the post, this is definitely the most hopeful I've been about solving this thing in a while!
Hey Brett!
I'm glad that you found this to be useful. Unfortunately, I don't think there is any way to know with 100% certainty without opening a case with Microsoft and sending them a memory dump.
If your symptoms (and environment) are the same as mine were, is there a server you could use to test on and then roll back if necessary? That's what I would recommend if possible.
MAtt, I'm having that pbm with my 2008 r2 ! What hotfix solved your problem ?
With R2 there is a whole different Hotfix that may or may not help you. The one that ultimately helped us was: KB 2465772. If you still have problems after installing that hotfix, I would also try: KB 2265716 and KB 2383928. Good luck!
we still have the issue. tryed the hotfix a year ago , and it went away a couple mounth , but it came back, now im pretty damn tierd of my job..
I completely understand the exhaustion and frustration that comes along with troubleshooting a Terminal Services/Citrix environment. If you have any kind of support agreement with Microsoft, I would suggest, if nothing else, opening up an email case with them. It provides 24-hour response times, but I've had good success with them!
Hi Matt,
We are experiencing similar situation as well but terminal servers are Windows 2003 based. Any ideas to point me in right direction?
Hi Punktional,
Unfortunately, we just started using Citrix, with our initial installation being with Server 2008, so I really don't have any experience with 2003.
That said -- there are a few things you can do: If you have support (either Microsoft or Citrix) you should be able to open a case and they're usually pretty good about getting to the bottom of an issue (that has been the case for me anyway). If you don't have any support, then Google is your best friend. Between searching, and checking out the Citrix support forums, you'll want to look for the exact symptoms that you're seeing.
If you're not alone (which you probably aren't) with the problem, something is bound to show up in a search result.
Best of luck!
Great stuff, This has worked for us too. XenApp server used to freez and need hard reboot but after this patch its more than a week and its fine.
Thanks fo
Matt,
Were there any errors in the event log or any other possible warning signs that you saw with this? We have a similar random issue.
Hi, Unfortunately I can't remember if there were any specific error messages in the logs -- but I do remember the servers becoming unresponsive, even at the console level. Basically, when I attached to them via KVM, I didn't even get video out. The odd thing though was that they were still ping-able. The best suggestion I can make is to open a case with Microsoft and collect a memory dump for them while in the frozen state (assuming you've already tried the hotfix).
Hi Matt,
Thanks for this fix, We have a 2008 with the same problem. I have apply this patch a week ago and we don't have any freeze for the moment.
I just want to clarify, We have 2008 server with TSE but no XenApp service and the server was randomly freeze but ping-able with events error with wsearch and Outlook process.
Thanks from France !!!
Post a Comment