Xenapp 6.0: "A device attached to the system is not functioning."

We had a user call our help desk stating that they weren't able to log into their XenApp 6.0 published desktop.  It accepted their credentials, started going through the motions of applying group policy, etc., but before the desktop actually appeared, the session disappeared completely.  After doing some testing, I noticed that an error message was popping up, for less than a second, right before the session closed.  With my trigger finger, I was able to snap a screen shot to discover the following error: "A device attached to the system is not functioning."

After poking around the Citrix Delivery Services Console, I discovered the that the user had a Disconnected session out there, that was obviously in a completely cheddared state.  So, when the user was logging in, they were being reconnected to their broken session, and were never able to log in.  After logging off the disconnected session, the user was able to start a brand new session and log in successfully.

This is the XenApp 6.0 Hotfix you are looking for

Basically, this is just a quick update to my previous post on Citrix servers freezing.  In looking at the dates, I now realize that I have been waiting over two months for this hotfix!

Anyway -- the hotfix to resolve a LOT of stability issues was released this morning (XA600W2K8R2X64046) and can be obtained at the following site:
http://support.citrix.com/article/CTX128342

Happy patching!

Citrix XenApp 6.0 Servers Freezing

When we finally made our way through our XenApp 5.0 farm freezes, I never thought that I would revisit such a problem so quickly.  When we made our upgrade to XenApp 6.0 and Windows Server 2008 R2 I thought for sure we would be jumping into a much more stable environment.  Unfortunately, I was wrong.

Shortly after making the upgrade we noticed that our Citrix servers became unresponsive at random times throughout the day: you could ping them, but that's about it.  Any attempt to RDP or log in even at the console level was an absolute failure.  If you are experiencing this problem, I would recommend upgrading your servers to Server 2008 R2 SP1, and applying the following Microsoft Hotfix: KB 2465772.  In addition, you're going to want to install at LEAST this Citrix Hotix as well: CTX127023.  But really, I would install as many Public Hotfixes for XenApp 6.0 as possible that apply to your environment.

If that would have been the end of the freezing saga, I would have been content.  However, the freezing dragon decided to rear its ugly head again: whenever we tried to shut down a Citrix server after having a decent amount of load on it, it froze.  Basically, it would sit on the Windows "Shutting down..." screen with the spinning circle to never fully shut down or recover without a hard power off.

After working with Microsoft and Citrix, it turns out that there is a Citrix Hotfix that's working its way toward becoming public (supported by Citrix), that should be available in the coming weeks.  If you are experiencing this same issue (freezing when shutting down) you're going to want to keep your eye out for Hotfix 46 (full name: XA600W2K8R2X64046).

UPDATE: I've added a new post that provides links to this hotfix.

In the meantime, there are a couple pages you should be keeping an eye on:

1. The list of Public Hotfixes currently available for XenApp 6.0
2. A list of recommended Microsoft Citrix Hotfixes for XenApp 6.0 and Server 2008 R2

Both of these get updated from time to time, so you'll want to bookmark them and see what's new whenever you get a chance.

As always, this is what worked for me, in our environment, so your mileage may vary and I won't be responsible for any problems this post may cause :)

Setting up KMS for Windows Activation on Server 2008 x64

We initially set up all of our Windows Servers using MAK keys.  For those not familiar, MAK keys are very simple: you copy them from your licensing agreement with Microsoft and paste them into Windows activation, and then you're done.  Simple right?  The only trouble with using MAK keys is that you can only use them a limited number of times before you need to call Microsoft to increase the number of times that you can use them.  In our Citrix environment, I regularly image about 10 servers at a time when we make changes -- so you can see how I'd chew up the limited number of activations very quickly.

Enter KMS.  With KMS, you need to set up one server on your network as a KMS host, and then all of your Windows machines can activate against it an unlimited number of times as KMS clients.  The only potential "gotcha" is that each of these KMS clients need to periodically check in with the KMS host to remain activated.  Since our Citrix farm is all on the same network and never gets shut down, this would not be a problem for us.  To understand more about KMS please check out this article.

Our current Citrix farm is installed on Server 2008 R2; however, the domain controller that I made to be the KMS host, is running Server 2008 x64.  Setting up the KMS host was pretty straightforward: Change the current license key to our KMS B license key (B is for Standard and Enterprise edition servers), and then restart the Software Licensing Service.  By default, if you have Dynamic DNS enabled, this should create some new records on your DNS server to let clients know where the KMS host is listening.  If you don't have Dynamic DNS enabled, you'll need to create these records manually.  Plenty of information on that and other fun KMS topics can be found in this article.

Now for the client side of things.  By default, Windows installations are KMS clients (I didn't know this before, so that's a fun fact) -- they will check in with the DNS server to see if there is a KMS host out there, and then try to activate against it (note: for Windows Server, the KMS host must receive at least 5 activation requests before it will start activating).  I had a little more work to do since I had previously entered our MAK key for activation.  In order to switch the server back to a KMS client, I had to type in the following command from an elevated command prompt:
slmgr.vbs /ipk KMSSetupKey


where KMSSetupKey is a key that is provided by Microsoft that is not part of a licensing agreement -- it's just a generic key.  I pulled the one I needed from this site.  Then, to attempt an activation, you can type in the following command from an elevated command prompt:
slmgr.vbs /ato


Unfortunately, my attempt was not very successful.  Instead, I was prompted with the error "The key management service is unavailable."  This can mean a number of different things, but to get to the bottom of it, I pulled up the Event Log on the KMS host and noticed the following error number: 0xC004F042, which according to this site of common activation errors, means that there is a mismatch of some kind between our KMS host and our KMS client.  After a bit of digging, I discovered that I needed to apply an update to our KMS host running Server 2008 x64 to allow KMS clients running 2008 R2 to activate against it.  If you are in need of this update, it is currently available here.


Now, that's not the end of the story.  After applying the update and rebooting the KMS host, I was still unable to activate our 2008 R2 Servers.  In order to complete the update process, you must install the Server 2008 R2 KMS host key using the same command (slmgr.vbs /ipk KMSSetupKey) as mentioned above, except you'll need to use the KMS key from your license agreement with Microsoft.  Then, I just needed to activate it (slmgr.vbs /ato) and restart the Software Licensing Service one more time.  Now we can activate R2 servers against our KMS host.


Next stop will be determining how this all affects my Altiris imaging process...

Held Hostage By Symantec

In our Citrix environment, we needed good Anti-Virus/Anti-Malware protection since there will be a huge wave of users inhabiting each server on a daily basis. Our first choice was to deploy Symantec Endpoint Protection since we've already had pretty decent success with the product across our PCs. So, we dove in to adding it to our Citrix Environment.

A few weeks in we noticed some very odd things happening: Users access to network shares was excruciatingly slow. It would literally take minutes to enumerate all of the items in a given folder. Since our Start Menus were also redirected on the network, this meant their start menu would not show up for quite some time. Between this, slow log-on and log-off times and slow access to network shares (a main part of our business), this performance was unacceptable.

Naturally I started down the path of getting support. Now, Symantec support is pretty bad in my opinion in terms of wait times, turn-around-times, etc., but that's a different story. To make a very long story (approximately 3 months in time) shorter, it boiled down to the fact that there is a defect in their code that causes network scanning to stay turned on (in File-system auto-protect) even when you uncheck the box in the policy settings. So, in the end, Symantec acknowledged the defect, but as of now, still is unwilling to create the fix for it, or even provide a timeline for when this fix may show up in a future version. Since we've already had several hold ups on this project, this kind of indefinite wait was just unacceptable.

So -- now we're on to looking at our options. At the moment, Kaspersky is looking pretty good. How about any of you? Any experience deploying anti-virus in a Xenapp 5 environment running on Windows Server 2008 x64?

Altiris Agent Prevents Roaming Profile Removal

When I logged into one of our Citrix Servers, I noticed something interesting:  The user directory folder (C:\Users on Windows 2008), contained multiple copies of users' cached roaming profiles.  So, for example, there would be folders maugustine.domain, maugustine.domain.000, maugustine.domain.001, etc.  I thought for sure that I had set the Group Policy to remove locally cached roaming profiles upon log out to avoid this sort of mess, and when I double checked, my memory was correct.

Upon further scrutiny, I discovered that everything was being deleted from the user's profiles except for one empty directory: C:\Users\username.domain\AppData\LocalLow\Microsoft\CryptnetUrlCache, which is related to Security Certificates when using Internet Explorer.  In a few tests, I was able to reproduce the problem by logging in, opening IE, browsing to an https site (like www.bankofamerica.com), and then trying to log off.

After going through the services on the system and the startup items under msconfig one by one, I learned that it was the Altris DAgent service that caused this.  If I stopped the service and then ran the previously mentioned test, the user's profile would be deleted just fine.

So, I went straight to Symantec support, who has acknowledged the problem, has been able to reproduce it, and now considers it a "known issue."  If you are sitting on Windows Server 2008 and are experiencing the same problem, unfortunately there is no permanent fix available yet.  Your best bet is to disable the Altiris DAgent services and startup items and subscribe to the following KB article for updates:
https://kb.altiris.com/article.asp?article=51228&p=1

However, if you happen to not be on Vista or 2008, you also have the option of going back to the Altiris AClient until they release a new version of the DAgent.  Unfortunately, the AClient is not compatible with Vista or Server 2008.

Sorry for not providing a permanent solution at this point in time, but at least you have a workaround and a KB article to follow!

Windows Server 2008 Freezes -- Finally Solved!

In our environment at work, we have a Citrix farm that users connect to that is running on Windows Server 2008 x64.  During our testing phase while we still had a relatively light load of users on the farm, things went pretty smoothly.  As we added more and more users to the Citrix environment, different issues cropped up here and there, but none as horribly evil as our servers freezing to the point of becoming completely unresponsive.  At that point, all sessions that users were in would lock up, forcing them to lose any unsaved data and restart their sessions again.  As you can imagine, management did not see this as an enhancement to their productivity.

So, for the last several months we have been troubleshooting this issue.  There was no pattern in regards to when servers would freeze.  At any given time, any of the four servers we have in production would freeze.  There was also no consistent user base on the server that would freeze (the only consistency being that they weren't too happy when it would happen).  After bringing in several consultants that helped set up this environment initially, we took our case to Citrix.  Several log and memory dump files later, they came to the conclusion that Internet Explorer was causing our servers to lock up.  Naturally, I then presented this information all to Microsoft support.  Upon further analysis, they discovered that we were experiencing a bug that has been resolved by a hotfix:
http://support.microsoft.com/kb/976674

Basically, the hotfix resolves an issue that occurs when Server 2008 or Windows Vista is under a heavy load and there are a lot of network share accesses going on.  Well, in our case, the user profile is a network share, plus their Outlook PST files were out on a network share, plus their other file shares were network shares, and the list goes on.  After applying this hotfix (which was a little over two weeks ago) we have not experienced any freezes.  Good news for everyone.

If anyone is interested in the detailed symptoms:

• Users sessions (terminal services/Citrix) would become completely unresponsive
• The server would become unresponsive even at the console level
• The server would respond to pings
• Apparently anything in memory at the time of the freeze would continue to function -- as soon as you tried to access something else, the session would freeze
• The only workaround when this occurred was to hard reboot the server

Random Empty Print Jobs Sent to Network Printers

I recently came across an odd problem having to do with empty print jobs being sent to printers across our network.  Since our printers are all configured to first print a cover sheet with the user's username before each job, this would result in dozens of coversheets being printed on various printers throughout our company.  After a closer inspection, the jobs being spooled were all named "Remote Desktop Redirected Printer Doc" -- so I knew that Remote Desktop had something to do with it at least.

At some point in the troubleshooting process I found that I could recreate the problem at will (which is gold in the technical troubleshooting world).  All I had to do was start a remote desktop session from a computer with network printers installed, to an XP computer on our network.  I also had to choose the option to redirect local printers to the remote session for it to occur as well.  Whichever network printers were installed on the local machine, and were redirected to the remote session, would receive a random amount of these empty print jobs.

After scouring the internet for awhile, and involving Microsoft support, I discovered that our problem was related to a little 3rd party application called Scan2PC.exe -- which was installed with some drivers to a Dell multifunction printer that all of our executives have.  So, if we started a remote desktop session to any XP computer that had this application installed, the problem happened.  As soon as I removed the Dell drivers, the problem went away.  Since our executives aren't planning getting rid of their brand new multifunction devices anytime soon, our current work around is to uncheck the box in the RDP session to redirect local printers to the remote session.

It's crazy how such a small thing can cause so many weird things to happen!

Users Unable to Change Expired Passwords on Windows Server 2008

In our environment, we have Citrix XenApp 5.0 publishing desktops from Windows Server 2008.  Our users connect to these published desktops via thin client or through the web interface.  We recently decided to put our password policy into effect, which included expiring user passwords once a month.

When the first user experienced the password expiration interface in Server 2008 after coming back to a locked workstation, they received the following message:
"The password for this account has expired. To change the password, click Cancel, click Switch User, and then log on."

However, there was no cancel button to click on, and no apparent way for them to either change their password or log off and log on again to do so.  The only way we could get around this was for them to call the help desk, we'd manually reset their passwords in Active Directory and then they could log in again using that new password.  An unacceptable solution in my opinion :)

So, I started to do some digging and found the following Microsoft KB article:
http://support.microsoft.com/kb/958900
which has an associated hotfix, that we applied and users were able to happily go on changing their passwords when they expired.

But then, we noticed something else.  Users that had two monitors set up at their station were experiencing an interesting symptom now: When the Server 2008 login screen came up, the dialog was now centered in between the two monitors (instead of only being in the primary), and it only showed the left half of the dialog in the primary monitor.  The secondary monitor was just completely black.

Oddly enough, the solution was to upgrade to Server 2008 SP2.  The service pack includes the previously mentioned hotfix, but for some reason did not have the same affect on dual monitors that the hotfix alone had.

I spent several hours scouring the web for a solution and didn't find anything -- so hopefully this will help you!

How to add a DNS suffix from a command prompt

We recently moved our datacenter, which involved adding our servers to our new domain and related DNS server (let's call it mydomain.com).  Previously, users connected to a telnet server by hostname (let's call it server1).  Well, after moving our datacenter, users could no longer connect to the server unless they used its FQDN of server1.mydomain.com. 

At this point we knew we'd have to hit every single computer (hundreds) on our network to update their preset connections.  We had an option -- we could either join each machine to the domain, or we could add a DNS suffix of mydomain.com to their local area connection tcp/ip properties.  We opted for the latter, since joining each machine to the domain would take longer, and is unnecessary considering we're in the process of rolling out thin clients to replace each of these machines.

In order to add a DNS suffix to a TCP/IP connection remotely, all you need is a list of IP addresses and the following command:
wmic /USER:administrator /PASSWORD:adminpassword /node:@c:\iplist.txt nicconfig call SetDNSSuffixSearchOrder (mydomain.com)

Where C:\iplist.txt contains a list of IP addresses, line separated.

After running this command for all of the IP addresses, users could then resolve server1 without needing to type out the whole FQDN.  Of course, this command could also be put in a script if you wanted to use it in such a way as well. 

Happy Networking!

How to "forget" network share credentials so you can authenticate as another user

UPDATE (10/26/09): I recently stumbled across a case where I was getting the "multiple connections" error and the steps below did not resolve the issue. There is one more place to look (especially if you checked the box to "Remember my password" when connecting to the network share) to remove stored credentials. Go to Control Panel -> User Accounts. Then go to the Advanced tab and under "Passwords and .NET Passports" click on "Manage Passwords". If the server you're trying to connect to is listed there, you're in luck. Simply remove the entry, and log off and then back in again. You should then be prompted for your new credentials.

This is something that has saved me a lot of time by removing the need to log out from my current Windows session. Have you ever connected to a network share and then wanted to authenticate as another user and received the following error:
"Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed."?

For the longest time, I thought the solution was to log out and then back in again to authenticate as the other user. It turns out there is a much easier way to get your computer to "forget" your current credentials so that you can use alternative ones. Simply bring up a command prompt and type:
net use * /d /y
This will effectively disconnect all remote connections as well as their associated credentials.
If you have more than one network connection open and you'd rather just delete a specific connection, type in:
net use
in a command prompt window. This will list all of your current remote connections. To delete a specific connection (let's say to \\server\share), type in the following in the command prompt:
net use \\server\share /d /y

If you're more of a GUI fan, you can go to My Computer and then select:
Tools -> Disconnect Network Drive
Then just select the specific connection that you'd like to disconnect and click the OK button.

I hope that this saves you some time!

"The specified port is unknown" When Adding a Network Printer

We currently have a Windows 2008 x64 Server with print services installed and we are sharing several network printers through that server. Every time a user logs in, a vbs script is run that maps certain printers for them depending on which groups they belong to in Active Directory. All was well and good until randomly we started getting the error message "The specified port is unknown" on one of our servers when trying to add the printers. So, instead of mapping the specified printers, the logon script failed leaving the user printerless. I then went on to test adding the printer manually through the control panel, and to my surprise, I got the exact same error. What gives?

I tried several things on the server hosting the printer, but it made absolutely no difference whatsoever. Finally, I thought, maybe if I just restart the Print Spooler service on the server the user was logging into. As soon as I did that, everything worked again! So, I'm not exactly sure what caused the problem to occur in the first place, but I'm glad that I know how to fix it now.

Bonus: We were also having issues with the wrong username/job owner showing up on the printer separator sheets. Instead of printing the username of the person who printed, it would print the username of the last Domain Admin to print something. We searched Google near and far and came back with nothing. After running out of ideas, we pinged Microsoft support on the issue to discover that this is a known problem with Server 2008 and Vista SP1 and there's actually a hotfix available for it: http://support.microsoft.com/kb/958741 I hope that someone finds this useful, since I had such a hard time finding the solution.

Getting around the Windows Rearm Limit with Sysprep and Altiris

We are currently in the process of deploying 9 IBM Blade servers as a Citrix XenApp farm for our employees. After evaluating the options for maintaining the servers, and their images, we settled on using Altiris. The past several weeks has been consumed with coming up with a "Golden" image that we can use across all 9 servers so that each user's experience will be consistent and predictable (well, as much as possible anyway). Coming up with this "Golden" image required going through a few iterations of a Server 2008 build, installing applications, customizing options and updates, etc. We finally came to the point yesterday when we were ready to make one last change and take a final image -- and this is when I ran into a problem. For the life of me, I could not get Altiris to take the image properly from the "Golden" server.

For the record, we've never used Altiris before, so for the most part we kept most of the default settings: we chose a path to save the image to, we chose to sysprep (using the default answer file) the server, and then take the image. I was finally able to narrow down the problem to sysprep not running correctly. So, at that point, I went on to the server to try and manually run sysprep and got a fatal error! (A full description of the problem can be found in this MS knowledgebase article).

Basically, it boils down to these facts: When Altiris syspreps a server it generalizes it (as it should since this strips out all of the uniqueness of the server, so it can be applied to other servers), resetting the licensing information (or rearming it), and whatever else it does to prepare the image. Now, this is all well and good except for the fact that Microsoft limits the number of rearms to 3 for any given Vista/Server2008 image. So, not really being exposed to this world before, I burned up these 3 rearms very quickly :)

At that point I scoured the internet for workarounds for this problem, since I had a golden image, but could not sysprep it. I tried the answer file solution proposed in the knowledgebase article above, but could not get it to work for whatever reason. Finally after much searching I came across this article, which describes how to get around this limitation.

So, to solve this problem (if you've read the entire post until this point, well done -- but if you just skipped down here to the solution, I don't blame you), I just went into the registry on the "Golden" server and set the following key to a value of "1":
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurentVersion\SL\SkipRearm (For Windows 7: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\SoftwareProtectionPlatform\SkipRearm, thanks Mike!)
Without a restart, I was able to run the Altiris imaging job again successfully. It cost me a day of work, so I hope someone finds this helpful. If you have any questions, feel free to leave a comment below.

Note: I also noticed that I had to do this registry change EVERY time before taking an image. The image does not retain this value, it gets reset after doing the sysprep.