Slowness with RPC as root on AIX

I was responsible for writing a little RPC client (and server) program that talks to our Unidata database to retrieve some arbitrary values.  The client has been running fine on Linux, but when we moved it to our AIX server, we noticed something interesting: in a loop of 1000 calls, it would take about 13 seconds on Linux, and 45 seconds on AIX.  It was absolutely maddening, and made no sense since it was the exact same code, just recompiled (and our AIX box has FAR superior hardware specs).

Then I made a discovery: when we ran the program as a non-root user, we got comparable times to the Linux box.  So, what is it about root on AIX that was causing the hold up?  Well, apparently, in AIX there is something built in that uses reserved, or privileged, ports for that kind of communication when running as root.  So, instead of having thousands of ports the client program could communicate on, it had a very limited subset of ports, which resulted in waiting for one of those ports to be available before completing the request.

Long story short, the result was to set the sticky bit on the client application (which has owner and group as non-root), to force it to always run in non-user space.  In case you come across a similar issue, here are the steps to fix it:
1) Use chmod and chgrp to make sure that the permissions on the application are non-root for ownership/group
2) Enable the setuid sticky bit on the application with the following syntax:
chmod u+s AppNameHere

This caused quite the headache and resulted in a support call to IBM, so hopefully this will help someone out there!

New Look and Feel

I've never really liked the default template I've been using for this blog, so I figured it was time for a change.  Welcome to the new look!

Setting up a Web Server on AIX 6.1

There have been several occasions when I have needed to set up web servers on either Windows or Linux.  Pretty much every time it has been a simple process of installing binaries through a GUI and making sure everything plays nice together.  Well, for the first time I needed to set up Apache, PHP and MySQL on an AIX machine that is completely GUI-less (Note: the GUI-less portion isn't the hard part).  In going through this process I learned a few things, so I thought that I would share them here.

Step One: Trying to find binaries for AIX
This step in itself is a little difficult.  Previously I have used Bull Freeware (http://www.bullfreeware.com/) for AIX binaries.  However, in this case, most of the binaries available are fairly outdated and I really wanted to put something together that was a little more recent.  This drove me to the discovery of PmWiki (http://www.perzl.org/aix/index.php) maintained by Michael Perzl.  This repository of RPMs built for AIX is very thorough, and very recent!  I highly recommend this site for anyone looking for binaries for AIX in general.  I quickly snatched up binaries for Apache and PHP, but no MySQL binaries were available.  Have no fear, MySQL actually maintains their own, so I was able to download them from their website (http://dev.mysql.com/downloads/mysql/).

Step Two: Trying to install the binaries
This quickly turned into a dependency hunt for both Apache and PHP, but the PmWiki site had every package that I needed, so although tiring, it was simple work.... especially since each RPM install tells you exactly which packages it's missing.  Installing MySQL was as simple as unzipping the directory to /usr/local/mysql and following a few quick steps found in the install readme included with the binaries.  At this point, Apache was up and running, PHP was configured and MySQL was up and running.  In order to get MySQL to run at system startup I copied the mysql.server script to /etc/rc.d/init.d and then created symbolic links to it under rc2.d, rc3.d and rc5.d.  (Note: If for some reason, you are following this like a tutorial, do not install the PHP binaries just yet.... you'll see why).

Step Three: Trying to get PHP to connect to a MySQL database
This is when things started going downhill a little bit.  In a quick test, I got an error in my PHP script noting that the mysqli_connect (and mysql_connect) functions were undefined... this is not a good sign.  In going through the PHP info, I discovered that the PHP binary I downloaded was not compiled with MySQL support.  Bummer -- this meant that I now needed to compile PHP on my own on AIX.  Fun stuff!

Step Four: Compiling and Installing PHP on AIX
This part was the most painful of this entire process.  I don't have too much experience building and compiling from source (especially on AIX), so it probably took a little bit longer than it should have.  I did discover a few interesting things though:

• You must use the 32-bit version of MySQL if you want PHP to compile using it
• You must use gmake/GNU make instead of the AIX make command for compilation to complete
• I had to install the apxs (from the httpd-devel package) to reference in my configure command.  The AIX one (under /opt/pware64) made a libphp5.so library that caused Apache to fail to start up.
• The --enable-maintainer-zts configure option is required to make php thread safe.  Without it, Apache complains and won't load the library.
• I also came across a very odd problem when I did the "make install".  The first part of the script deletes libphp5.so and then subsequently tries to call chmod on it, at which point it bombs out since it can't find the file.  Nice.  I came across a hack that also worked for me: Have two windows open, one ready to do the make install, and one ready to copy libphp5.so to the directory the script is expecting it to be in.  Start the make install, and then as soon as it has tried to delete libphp5.so, copy it where the script expects it to be.  By doing this, the file will be there when it's looking for it.  It requires a little bit of timing, so it may take a few tries to get it right.

For anyone interested, this is the configure line I used: 

configure '--cache-file=../config.cache' '--prefix=/opt/freeware' '--with-config-file-path=/opt/freeware/etc' '--enable-shared' '--enable-static' '--without-pear' '--with-gd=/opt/freeware' '--with-openssl=/opt/freeware' '--with-zlib' '--with-bz2' '--with-curl=/opt/freeware' '--with-t1lib=/opt/freeware' '--with-freetype-dir=/opt/freeware' '--with-jpeg-dir=/opt/freeware' '--with-png-dir=/opt/freeware' '--with-xpm-dir=/opt/freeware' '--with-zlib-dir=/opt/freeware' '--enable-soap' '--enable-bcmath' '--enable-ftp' '--with-iconv' '--enable-dom' '--enable-json' '--with-pcre-regex=/opt/freeware' '--with-apxs2=/opt/freeware/sbin/apxs' '--with-mysql=/usr/local/mysql' '--with-mysqli=/usr/local/mysql/bin/mysql_config' '--enable-maintainer-zts'

Step 5: Testing and Completion

At this point, I now had Apache running with PHP with support for MySQL.  I happily connected to the database and then handed the server over to another developer who would actually be using it.  This was definitely a learning experience, so I hope someone finds this information useful.

UPDATE: I also discovered that by default, cgi scripts were not executing.  This was due to the file permissions set on the httpd log file directory.  So, if you also run into this problem, just run a chmod 775 on your httpd log file directory (mine was located at /var/log/httpd).

Installing Jolicloud 1.0 on a Macbook

As of late, I have gained an interest in Cloud OSes.  If you have no idea what I'm talking about, Cloud OSes, from my understanding, deliver a very thin desktop OS that largely lives and acts in the Could (AKA the Internet).  Two examples of such an OS are Google Chrome OS and jolicloud.  For a good introduction video on the topic, you can watch this video. 

 Now, for the time being, Chrome OS is in its very early stages, so much so that it doesn't really have a formal release just yet.  So, I decided to take jolicloud for a test drive.  Of course, I don't have a netbook, which is what it's designed for, so I worked my way through installing this thing on my 3-year-old white Macbook.

So, for anyone else interested in installing jolicloud on a Macbook, these are the steps I took:
1) Use Bootcamp Assistant (located under Applications -> Utilities) to create a partition for jolicloud.  I made mine about 8GB, but you could get by with much less.
2) Install rEFIt (available here).  This is basically a boot menu that makes installing (and subsequently launching) jolicloud a bit easier.  Make sure you run the enable script mentioned in the install documentation, or you'll never see the boot menu.  If you've done everything up to this point, you should see the rERIt boot menu when you reboot your Mac.
3) Burn the jolicloud ISO (available here) to a CD.  (Note: I tried using a USB thumb drive for this portion, but I couldn't get my Mac to detect it).
4) Insert the CD and reboot your Mac.
5) You should see a little penguin icon next to your Mac partition in the Boot Menu -- go for the penguin!
6) At that point you should be able to follow the installer -- it's basically your typical Linux installer.

Once installed, you'll probably want to change a couple of things immediately:
1) Go into the device settings/control panel and update your keyboard settings.  You'll want to make sure that it's set to the appropriate Mac keyboard
2) In that same area, update your mouse/accessibility settings so that long-clicking on something = right-clicking.  Unfortunately, with only one mouse button, this was the only way I could figure out how to get right-clicking to function.

I haven't had much time to play around with the OS itself yet -- but if you have any tips or suggestions, I'd be glad to hear them.

TCPing: Add it to your toolbelt

I've been using this tool for several months now, thanks to the introduction provided by one of our Citrix consultants.  TCPing is a simple utility which expands upon the functionality provided by the "ping" command that I'm sure you're all very familiar with.

Most notably, it adds the ability to "ping" a specific port on a host to check whether or not it is open for business.  There are many situations where you could use something like this, such as making sure a service is up and running when you expect it to be.  However, my favorite use of this tool is when I'm waiting for a host to reboot and I want to RDP into it.  There have been so many times, before I had this utility, that I would let ping run until there was a response, and then I would open my RDP client and periodically try and connect until I got through.

Well, now with tcping, I can run a command like "tcping -t hostname 3389" and just sit back and wait for the port to be open, at which time I know I'm good to go.

I know this is not a major discovery, or anything new, but it's the little things that count, right?

If you would like to download this utility, you can get it from here.

My Next Android App - Group to Voicemail

I have been hard at work (in my spare time... whatever that means) putting together my second Android application called Group to Voicemail. It is available on the Android Market for all Android devices running 2.0 and above.

Here are the features:

• Create and manage Contact groups that can sync with your Google account automatically
• There are currently three action types:
  • Send to Voicemail – Send all Contacts in that group to voicemail
  • Don’t Send to Voicemail – Don’t send all Contacts in that group to voicemail
  • Everyone Else to Voicemail – Send all Contacts but those in the specified group(s) to voicemail
• Create schedules to automatically apply actions to groups
• Schedules can be created for a variety of scenarios:
  • Span multiple days (ex: from Friday at 5:00pm until Monday at 8:00am)
  • A time range on one day (ex: from 8:00am until 5:00pm on Friday)
  • All day (ex: All day on Thursday — will be in effect from 12:00am until 12:00am the following day)
  • Repeat days (ex: Every Monday, Wednesday and Friday from 4:00pm to 6:00pm)
• Color-coded visual cues to specify which groups are currently being sent to voicemail and which schedules are currently active
• Option to receive a notification anytime a schedule or group status changes. This provides a quick glance at how many groups are currently being sent to voicemail and how many schedules are currently active.
• Temporarily disable schedules if you’re doing something different that day/week.

For screenshots and more information, you can also visit the website I put together for all of my Android-related projects: ForTheGamer.com.

Held Hostage By Symantec

In our Citrix environment, we needed good Anti-Virus/Anti-Malware protection since there will be a huge wave of users inhabiting each server on a daily basis. Our first choice was to deploy Symantec Endpoint Protection since we've already had pretty decent success with the product across our PCs. So, we dove in to adding it to our Citrix Environment.

A few weeks in we noticed some very odd things happening: Users access to network shares was excruciatingly slow. It would literally take minutes to enumerate all of the items in a given folder. Since our Start Menus were also redirected on the network, this meant their start menu would not show up for quite some time. Between this, slow log-on and log-off times and slow access to network shares (a main part of our business), this performance was unacceptable.

Naturally I started down the path of getting support. Now, Symantec support is pretty bad in my opinion in terms of wait times, turn-around-times, etc., but that's a different story. To make a very long story (approximately 3 months in time) shorter, it boiled down to the fact that there is a defect in their code that causes network scanning to stay turned on (in File-system auto-protect) even when you uncheck the box in the policy settings. So, in the end, Symantec acknowledged the defect, but as of now, still is unwilling to create the fix for it, or even provide a timeline for when this fix may show up in a future version. Since we've already had several hold ups on this project, this kind of indefinite wait was just unacceptable.

So -- now we're on to looking at our options. At the moment, Kaspersky is looking pretty good. How about any of you? Any experience deploying anti-virus in a Xenapp 5 environment running on Windows Server 2008 x64?

My First Android App - Game Encyclopedia

I have decided to join the onslaught of Android developers and build my first app: Game Encyclopedia. Essentially, it is a news and reference application that provides a front end to news and video game data hosted by GamesRadar.com.

Here are the current list of features:
- Up to the minute video game News Features
- A video game Browser that allows you to navigate through video games past, present and future by Platform, Genre or Name with the ability to sort results
- View related information such as a description, release date, ESRB Rating and review score for games
- A video game search that allows you to find the game you are looking for quickly

If you have an Android phone, you can now download it from the market by searching for "Game Encyclopedia." If you want to help me out, you could go install it, give it a nice rating, and leave a nice comment in the market :)

For more information, you can also visit the website I put together for all of my Android-related projects: ForTheGamer.com.

As always, I welcome all questions and suggestions!

Altiris Agent Prevents Roaming Profile Removal

When I logged into one of our Citrix Servers, I noticed something interesting:  The user directory folder (C:\Users on Windows 2008), contained multiple copies of users' cached roaming profiles.  So, for example, there would be folders maugustine.domain, maugustine.domain.000, maugustine.domain.001, etc.  I thought for sure that I had set the Group Policy to remove locally cached roaming profiles upon log out to avoid this sort of mess, and when I double checked, my memory was correct.

Upon further scrutiny, I discovered that everything was being deleted from the user's profiles except for one empty directory: C:\Users\username.domain\AppData\LocalLow\Microsoft\CryptnetUrlCache, which is related to Security Certificates when using Internet Explorer.  In a few tests, I was able to reproduce the problem by logging in, opening IE, browsing to an https site (like www.bankofamerica.com), and then trying to log off.

After going through the services on the system and the startup items under msconfig one by one, I learned that it was the Altris DAgent service that caused this.  If I stopped the service and then ran the previously mentioned test, the user's profile would be deleted just fine.

So, I went straight to Symantec support, who has acknowledged the problem, has been able to reproduce it, and now considers it a "known issue."  If you are sitting on Windows Server 2008 and are experiencing the same problem, unfortunately there is no permanent fix available yet.  Your best bet is to disable the Altiris DAgent services and startup items and subscribe to the following KB article for updates:
https://kb.altiris.com/article.asp?article=51228&p=1

However, if you happen to not be on Vista or 2008, you also have the option of going back to the Altiris AClient until they release a new version of the DAgent.  Unfortunately, the AClient is not compatible with Vista or Server 2008.

Sorry for not providing a permanent solution at this point in time, but at least you have a workaround and a KB article to follow!

Windows Server 2008 Freezes -- Finally Solved!

In our environment at work, we have a Citrix farm that users connect to that is running on Windows Server 2008 x64.  During our testing phase while we still had a relatively light load of users on the farm, things went pretty smoothly.  As we added more and more users to the Citrix environment, different issues cropped up here and there, but none as horribly evil as our servers freezing to the point of becoming completely unresponsive.  At that point, all sessions that users were in would lock up, forcing them to lose any unsaved data and restart their sessions again.  As you can imagine, management did not see this as an enhancement to their productivity.

So, for the last several months we have been troubleshooting this issue.  There was no pattern in regards to when servers would freeze.  At any given time, any of the four servers we have in production would freeze.  There was also no consistent user base on the server that would freeze (the only consistency being that they weren't too happy when it would happen).  After bringing in several consultants that helped set up this environment initially, we took our case to Citrix.  Several log and memory dump files later, they came to the conclusion that Internet Explorer was causing our servers to lock up.  Naturally, I then presented this information all to Microsoft support.  Upon further analysis, they discovered that we were experiencing a bug that has been resolved by a hotfix:
http://support.microsoft.com/kb/976674

Basically, the hotfix resolves an issue that occurs when Server 2008 or Windows Vista is under a heavy load and there are a lot of network share accesses going on.  Well, in our case, the user profile is a network share, plus their Outlook PST files were out on a network share, plus their other file shares were network shares, and the list goes on.  After applying this hotfix (which was a little over two weeks ago) we have not experienced any freezes.  Good news for everyone.

If anyone is interested in the detailed symptoms:

• Users sessions (terminal services/Citrix) would become completely unresponsive
• The server would become unresponsive even at the console level
• The server would respond to pings
• Apparently anything in memory at the time of the freeze would continue to function -- as soon as you tried to access something else, the session would freeze
• The only workaround when this occurred was to hard reboot the server