techrobatics.com

Sysadmin Ex Machina

For you python enthusiasts, here is a script that will recurse through directories, from the location where you run this, and show you the top disk space consumers. It’s handy for finding a runaway process, or indulgent user…

#!/usr/bin/env python
#dutop.py
import os
import sys
import operator

cutoff_percentage=5

#The following matches "du -h" output
def human(num, power="K"):
    powers=["K","M","G","T"]
    while num >= 1000:
        num /= 1024.0
        power=powers[powers.index(power)+1]
        human(num,power)
    return "%.1f%s" % (num,power)

def print_du(path):
    dufile=os.popen('find \'' + path + '\' -maxdepth 1 -mindepth 1 \( -type f -o -type l -o -type d \) -print0 | xargs -r0 du --max-depth 0 -k | grep -v "^0"')
    dulist=dufile.readlines()
    if len(dulist) == 0: #file specified
        sys.exit(0)
    filedict={}
    for line in dulist:
        size,name = line[:-1].split('\t',2)
        stat_val = os.stat(name)
        filedict.setdefault((int(size), stat_val.st_ino),[]).append(name)
    dulist=None

    keys = filedict.keys()
    keys.sort() #dict randomizes keys
    keys.reverse()
    total=reduce(operator.add, [item[0] for item in keys])
    found_one=0
    for key in keys:
        size=key[0]
        percentage = size*100/total
        if percentage < cutoff_percentage:
            continue
        names = filedict[key][:]
        name = names[0]
        isdir=os.path.isdir(name)
        if isdir and (percentage > (100-cutoff_percentage)): #nothing of interest @ this level
            filedict=None;keys=None
            if print_du(name):
                return 1
        found_one=1
        print "%2d%%%10s    " % (percentage, human(size)),
        for name in names:
            if isdir:
                sys.stdout.write("\033[01;34m")
            print name,
        print "\033[00m"
    return found_one

if len(sys.argv) == 1:
    sys.argv.append('.')

print_du(sys.argv[1])

Place a copy of this block at the bottom of your apache2.conf. Reproduce it for EACH directory, on which you want to enable the rewrite rules (each wordpress root instance).

< Directory /var/www/wordpress>
  Options Indexes FollowSymlinks MultiViews
  < IfModule mod_rewrite.c>
    RewriteEngine On
    #RewriteBase /wordpress/              <-- If you want to point somewhere else
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    #RewriteRule ^index\.php$ - [L]    <-- Alternative lookup
  < /IfModule>
< /Directory>

Under the covers, Mac OSX is a not quite clone of freeBSD, known as “Darwin”.

The mascot is appropriately a platypus, both because the platypus is a nod to the famous Charles Darwin, and because the folks at Apple have outfitted Darwin with all sorts of customizations cannibalized from the old NeXT project that confound the typical *Nix user, in the same way that the platypus confounded early biologists.

In spite of it’s hybrid nature, Darwin still manages to retain a SUSv3 compliance stamp from the folks at the Austin Group. This is little comfort to the habituated Debian, Ubuntu, or worse yet, Solaris admin.

Recently, a friend of mine decided to learn PHP, and I triumphantly exclaimed to him that his Mac mini is actually a lovely little apache2 server platform by default, if he wants to use it for learning the language. Luckily, I happened to be mostly aware of all the freaky little differences between your generic open-source apache installation, and the Mac “edition”, so his foray into the world of web development will be a little less painful than it might have been.

First things first

For the uninitiated, you’ll have to know a few things. First, how to find and use your “Terminal” application, and second, how to get “root” control over your system.

The mac dev center has a magnificent tutorial on using the terminal here, so I won’t reinvent that wheel. To get to root, you merely need to do this:

Welcome to Darwin!
Last login: Mon Feb 22 00:56:50 on ttys005
linus:~ gmgauthi$ sudo -i
Password:
linus:~ root#

Note that I supplied my own personal user password at the prompt. In theory, when you first set up your OS X system, it should have automatically added the userid you created to the “sudoers” list.

Where Is My Apache?

If you’re used to modern generic apache2 installations, the first instinct is to go looking in /etc/init.d for an apache start and stop script. Dont. Darwin doesn’t have one.

If you want to know the gory details of the OSX (Darwin) startup sequence, and service management model, you could try reading this, or this. But suffice it to say here, that you can pretty much throw out everything you know about unix rc.

So, where is the http startup then? Embedded in some System Properties plist file, that I can’t seem to locate at the moment. In any case, it’s not going to help you. To manage the daemon, you either have to use the System Preferences GUI, or apachectl. I’m a command-line geek, so here’s what it would look like:

sh-3.2# apachectl start
sh-3.2# apachectl -v
Server version: Apache/2.2.13 (Unix)
Server built:   Oct 16 2009 02:12:22

You’ll still want to double-check the GUI Preferences Pane, to make sure it stays up (I’m not sure if that sets the check-box or not). In any case, if you’re used to finding the apache2 process after starting your server, you’ll be surprised to find there isn’t one:

sh-3.2# ps -ef|vgrep apache2
sh-3.2#

That’s because it’s been renamed. I have no idea why. Anyway, you can find it here:

sh-3.2# ps -ef|vgrep httpd
    0 87769     1   0   0:00.03 ??         0:00.29 /usr/sbin/httpd -D FOREGROUND
   70 87770 87769   0   0:00.00 ??         0:00.00 /usr/sbin/httpd -D FOREGROUND
sh-3.2#

How Do I Know It Works?

Well, that’s easy. Go to your web browser and type this in to the address bar:

http://localhost/

What you should see is this:


It works!


If you don’t, well, something’s really wrong then. Give me a call.

Nuts-n-Bolts

In any case, it’s not really working. What do I mean? Well, these days, unless you’re nostalgic for 1997, then you’re building your site using CSS stylesheets, and mature web development languages like PHP, Ruby and Python.

Luckily, the OS X apache installation includes the load module for PHP5 already, but for some unknown reason, your OS X installation comments out PHP by default, in apache’s startup configuration, so we need to enable it. To do this, we need to modify your httpd.conf file, and then restart the server.

sh-3.2# cd /private/etc/apache2
sh-3.2# ls -la|grep -i conf
-r--r--r--    1 root  wheel  19548 Feb 22 00:59 httpd.conf
-rw-r--r--    1 root  wheel  17727 May 18  2009 httpd.conf~orig
sh-3.2# cat httpd.conf|grep -i php
#LoadModule php5_module        libexec/apache2/libphp5.so
sh-3.2#

The hash (pound-sign) that you see at the beginning of the “LoadModule” line echoed back from the cat command, tells you that php is disabled on your system. You need to edit the file to remove it. To be able to edit the file, we first need to change the permissions, to allow write access. here’s how I do it:

sh-3.2# chmod 664 httpd.conf
sh-3.2# ls -la|grep -i conf
-rw-rw-r--    1 root  wheel  19548 Feb 22 00:59 httpd.conf
-rw-r--r--    1 root  wheel  17727 May 18  2009 httpd.conf~orig

You can see there are a couple extra “w” in the attributes. That means the owner, and the group both now have authority to write to this file. That’s a good first step. Now, to edit the file. If you don’t like using tools like vi or emacs, then editing this file requires yet another step.

You see, the folks at Apple really would rather you didn’t know these files existed at all. So, they’ve added some of their own special attributes to keep you from finding them at the graphical desktop. An invisibility flag. Fortunately, their own developers included a utility to unmask them. Here’s how you do that:

sh-3.2# SetFile -a v /private/etc/apache2
sh-3.2# SetFile -a v /private/etc/apache2/httpd.conf

Now, when you look in Finder, you should see a “private” folder, just off the hard drive:

If you open that folder, you should be able to drill down to apache2, and then to the httpd.conf file. Open it there, with your editor of choice. Find the LoadModule line referenced above, and remove the hash. Save the file. Now, when you go back to your Terminal, you should see the hash is gone:

sh-3.2# ls -la|grep -i conf
-rw-rw-r--    1 root  wheel  19548 Feb 22 00:59 httpd.conf
-rw-r--r--    1 root  wheel  17727 May 18  2009 httpd.conf~orig
sh-3.2# cat httpd.conf|grep -i php
LoadModule php5_module        libexec/apache2/libphp5.so
sh-3.2#

Now, we can restart apache:

sh-3.2# apachectl stop
sh-3.2# ps -ef|vgrep httpd
sh-3.2#
sh-3.2# apachectl -k start -e alert
sh-3.2# ps -ef|vgrep httpd
    0 97511     1   0   0:00.01 ??         0:00.10 /usr/sbin/httpd -k start -e alert
   70 97512 97511   0   0:00.00 ??         0:00.00 /usr/sbin/httpd -k start -e alert
sh-3.2# apachectl -v
Server version: Apache/2.2.13 (Unix)
Server built:   Oct 16 2009 02:12:22

Fantastic!

Your First Web Page

Now, how can you tell if PHP is actually working? Well, we need to do a bit more editing. Navigate to your webserver’s “document root” folder. On a unix system, this would normally be /var/www by default. But on OS X, the default is:

Library : WebServer : Documents

Now, rename “index.html.en” to “index_old”. That will effectively delete the “It Works” you saw earlier. Instead, we’re going to replace it with our own php script. Create a text file in this directory called “index.php”, and put the following code in it:

< ?php  	/* 	 * PHP Environment Information Generator 	 */ 	phpinfo()  ?>

Then save the file.

Now, go back to your browser, and enter the url you entered before. What you should see, is a long panel of PHP environment information, that looks something akin to this:

If you do, it means everything is ready to go! The PHP engine is working, and you’re ready to start building your website!

If you don’t see this, well, then something is really wrong, and you should give me a call.

You can replace that index.php with anything you like, now. All your website files will go into the same directory as that file, or into subfolders under it. If you were to want to install mediawiki, or wordpress, this is where you’d drop the unzipped distribution files. However, if you were to want to properly install one of those two things, then you’re going to need an instance of MySQL installed, as well.

But that’s a topic for another post.