tech.agilitynerd.com

When I wrote my plugins for tracking the visits to each page (hitcounter), presenting the counts by popularity (favorites), and displaying the most recently read posts (lastread) I used Perl's Storable module as a simple way to serialize the data structure to and from disk. At the time I wasn't too concerned with performance, my blog agilitynerd.com didn't suffer under the cost of the disk reads/writes.

But periodically I get DOS'd and my website becomes unresponsive. Not too suprising since Blosxom has to run as a plain CGI script; but I'm certain the disk writes of these plugins weren't helping. Occasionally the Storable on disk would become corrupted which would take down the site (not enough exception handling in the plugins).

I decided to go with Mongo as the backing data store to remove the disk reads/write from the cgi script. It supports an increment operation (ala memcached) so multiple threads can update the hit count and MongoDB will "do the right thing". It also has a query API that made rewriting favorites and lastread trivial.

I haven't tested the performance but my website, especially the favorites page, seems faster. Once I write a JavaScript client to update the hitcounts I'll be able to move the website to a fully cached deployment while still tracking hits which should make the load of running the agilitynerd blog very low.

I've hosted the plugins on github: http://github.com/saschwarz/blosxom-mongodb-plugins The mongohitcounter plugin can be configured to automatically import existing count data from the hitcounter plugin's Storable. So conversion to the new plugins just requires updating the template's variables to the new plugin names.

Loading mentions Retweet

I've always wanted my AgilityNerd blog to serve as a reference for interesting dog agility subjects. Consequently, I'm interested in making it as easy as possible for readers to locate, explore, and learn more about the sport. So improving navigation is a strong interest to me.

I have been reading Designing Web Navigation by James Kalbach. In the section discussing Browser Mechanisms Kalbach discusses the back and forward buttons and the Session History drop down. The session history is usually a menu that drops down from the back button (or near it). It lists the pages the user has visited in reverse order showing their page titles. Kalbach states: "Session history is a good reason to supply meaningful browser titles". That reminded me that for my blog the session history has always just displayed "AgilityNerd" for each page visited on my site. I was leaving out a valuable aid to my readers who use the session history to find where they had been previously.

Another benefit of setting the title in pages is in tracking site statistics. It had always bothered me that GoogleAnalytics never showed statistics for my pages by page title.

So these two things finally moved me to take some action. I had been thinking about writing a Blosxom plugin to set the title element in the HTML head section but I've always been too busy. Luckily I didn't have to.

I found out about the storytitle plugin from Nick Leverton's Serendipity blog. He had made some changes and released version 0.7 of the plugin. It was trivial to setup, I just followed the directions in the plugin and had this site and agilitynerd.com/blog working within minutes.

So if you are a Blosxom user just grab this plugin. Otherwise, take the time to set your page title sections so you can help your readers and help analyzing your own site's statistics

Loading mentions Retweet

Welcome to the new look of the AgilityNerd website. I haven't significantly changed the look of this website since I created it back in August of 2004 and it was certainly overdue! There have been a number of enhancements I've wanted to make that I just didn't have the time to implement.

It's All About Navigation

I like to consider AgilityNerd an "informational blog". It isn't really about how our dogs do in dog sports (competition or practice) or even much about our dogs, although I do indulge myself sometimes. So to me this site is only useful if readers can find what they are looking for. Before this update folks could use these mechanisms:

• The Categories listing in the sidebar
• The Search box in the sidebar
• The [[navigate Navigation page]] shows the full category tree view
• The Favorites page showing the most popular articles and some of the less well known articles that I like

I also wanted to let readers know which articles not on the front page were being read most recently and also what articles had recent comments. So I wrote two Blosxom plugins, [[lastcommented_lastread lastcommented and lastread]] to show the last ten articles of each type to readers.

So to make this all fit I went to a "three column" layout. Fixed sized left and right menu sidebars surrounding a resizable center section. This design is based on Alan Pierce's Multi-Column Layout with a modification to how the left side sidebar is positioned. This design allows readers with large monitors to control the size of the page as they see fit. According to my web host logs 98% of my readers view my pages at sizes larger than 800x600 pixels so this layout should work well but it also looks OK on smaller screens.

Ads and Affiliations

As I mentioned earlier this year I'm moving to a bigger, faster, and more expensive web hosting service and to offset the costs I've entered into three advertising/affiliate programs. All three are now active on my site.

You'll notice a [[CleanRun Clean Run]] icon in the left hand menu. I've become a Clean Run Affiliate so if you purchase anything from Clean Run through any links on my site or through clicking that icon I get a small percentage of the sale. So your Clean Run purchases will help pay my web hosting fees. I really like all the things Clean Run does for our sport, in addition to publishing their must-read magazine, so I am happy to be an affiliate.

I've also become an [[Amazon Amazon.com Affiliate]]. Amazon is the de facto site for book links on the web. Currently I'm using their recommendation system for dog books; but it sometimes comes up with some books that I don't like. So I'll be replacing it soon with only links to books I personally recommend. Again if you purchase a book through this link I'll get a small "kickback" from Amazon.

Lastly, I'd like to thank everyone who saw an interesting Google link on my website and clicked on it. So far this year I've raised $ 40 from Google ads, one click at at time. Please only click on ads that really interest you, unlike some dishonest websites, I'm not looking to make money on these ads through fraud.

I appreciate everyone's support. Especially the kind comments and emails!

Minor Changes

In addition to the new layout I've made some small enhancements.

The writeback (commenting) feature has been tweaked to try to correctly format (optional) websites or email addresses entered by commentors. The date and time of all new comments will now be stored and displayed. If the person leaving a comment doesn't include their name I now default it to "Anonymous" (I thought the Anonymous Coward moniker used on other websites was too hostile for my readers!).

I also spent far too much time tweaking the [[CSS]] to change fonts, and spacing. Overall fonts are one or two pixels smaller but line spacings are a little bigger. I tried to standardize the spacing of elements too. I cleaned up the printing style sheet so it is tiny now (and downloads faster) - as before if you print a page the side menus and extraneous elements aren't printed. I also fixed the fonts so Linux browsers would use a sans-serif font instead of failing back to a serif font.

I enhanced my [[hitcounter]] plugin to not count pages loaded by bots so the page visitor counts only reflect actual humans reading the pages.

I updated Site Plugin page with modifications to the various plugins

Still To Come

There are still some changes I'm planning on making and hope to roll out over the course of the summer...

• I ran the [[Python]] webchecker script that ships with Python 2.4 over my site and found about 50 internal and external bad links (most are due to the demise of www.blosxom.com) and need to fix those.
• I need to fix and correct links and email address formats in some old comments.
• I'll be replacing the Amazon generic recommendations with a rotating selection of my specific recommendations. I'll also probably include recommendations for books and DVDs from Clean Run into the mix. I'd rather folks bought from Clean Run since they are a smaller company that needs the income.
• I need to update my site page to include these changes.
• Convert all Clean Run links to include my affiliate id. Add images for all Clean Run magazines in my Clean Run search. Add link to both electronic and paper clean run magazines. Add links to the 2007 magazines from the index file that was just uploaded.
• Change my agility glossary to launch articles in a separate window.
• Update my shortcut icon shown in browser tabs to match my new logo.

I guess I should be doing agility training sometime too...

Loading mentions Retweet

I had a comment entered on my [[AgilityNerd dog agility blog]] that I subsequently deleted and of course my lastcommented plugin recorded the comment as the most recent comment. Since the plugin uses Storable to store the data I couldn't manually edit the file to remove the entry.

So I modified my lastcommented plugin to allow you to specify the, base zero, index of the entry on the URL and delete the entry. For example to delete the second entry in the list (index 1):

http://agilitynerd.com/blog/?lastcommentremove=1

To keep mischief makers from deleting all your comments the plugin has a variable in its configuration section called lastcommentremove that you set to a non zero value to enable this feature. Since this is probably a relatively infrequent activity (if you have blacklisting or wbcaptcha comment spam protection enabled) I figured this was a sufficient mechanism.

You can download the latest version of this plugin here.

Loading mentions Retweet

I had previously extended Jason Clark's moreentries plugin for Blosxom to allow adding images to links to the previous and next group of articles/entries in the head or foot of a Blosxom weblog. While attempting to have valid HTML on my blogs I found that I had left an img tag unclosed. I also found that it wasn't as easy to change the styling of the links as I had originally planned.

So I've closed the open img tags and added the configuration variable $selfstyle into which you can place any CSS style information you'd like added to the td containing each page link. For example:

# Set to the CSS style for $moreentries::links
$selfstyle = 'style="padding: 0 3px;"'; # set to '' for no style

You can download the modified plugin (version 0.3) here.

Loading mentions Retweet

After fixing a bug in these plugins I decided to add some new features:

• The links generated by the favorites plugin used to display the path and filename of each popular article. It now displays the article title and the filename is displayed when hovering over the link. This makes for a much nicer looking and more useful display. Thanks to Doug for the suggestion.
• The hitcounter plugin now tests for existence of articles before adding them to its hash of counts.
• Requesting a page which no longer exists deletes the page and its counts from the hash of counts. Since previous versions of this plugin would record non-existent pages they could appear in the favorites listing. So now you can delete them by clicking on the link in the favorites page.
• As pages previously stored in the hitcounter hash of counts are visited their titles will be added so they will appear in the favorites' generated links.

Both plugins must be upgraded together and will upgrade the existing hitcounter database as pages are accessed. The new versions can be downloaded here:

• favorites
• hitcounter

Loading mentions Retweet

I found a bug in my hitcounter, favorites, lastread and lastcommented plugins due to my not using the correct Perl Storable functions. For some reason I didn't use the locking versions of retrieve() and nstore(). So when my old webhost had some problem causing long page load times (and many simultaneous requests) I ended up having my favorites data file written as zero sized.

So I changed to Storable::lock_retrieve() and Storable::lock_nstore() for all four plugins. The new versions are here:

• favorites
• hitcounter
• lastcommented
• lastread

I never heard any bug reports about these plugins, but I'm sorry if anyone ran into any difficulties due to these bugs.

Loading mentions Retweet

After my recent Referer Spam attacks I've been checking my logs and AWStats reports daily. I noticed that occasionally I'd get people requesting pages with incorrect suffixes. i.e. foo.ht or bar.|id| (some of these are probably spammers too). A "feature" of Blosxom is that it will still serve the page with the default layout or flavour. This feature causes a not too helpful and ugly error message to be displayed on the top of the page:

Error: I'm afraid this is the first I've heard of an "ht" flavoured Blosxom. Try dropping the "/+ht" bit from the end of the URL.

Of course removing the suffix from the URL doesn't usually work; the client is served with an empty (but flavoured) page. So I though a better solution would be to have Blosxom serve my default_flavour whenever the flavour couldn't be identified. A little Googling later turned up this Blosxom mod by James Vasille. James explicitly loads a flavour file named "default" whenever the requested flavour can't be found.

For my site it is more appropriate to serve the $default_flavour when the requested flavour can't be found. That way I didn't have copy/create head.default, foot.default, etc files. So my code change is almost the same as James':

$template = 
  sub {
    my ($path, $chunk, $flavour) = @_;

    do {
      return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$flavour");
    } while ($path =~ s/(\/*[^\/]*)$// and $1);
# Begin added code
    do {
      return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$default_flavour");
    } while ($path =~ s/(\/*[^\/]*)$// and $1);
# End added code
    return join '', ($template{$flavour}{$chunk} || $template{error}{$chunk} || '');
  };

So now a less ugly page will be provided when an unknown suffix/flavour is requested.

Update 19-Apr-2005 - this algorithm has a bug that has been fixed by the defaultflavour plugin.

Loading mentions Retweet

I added a Favorites page to the side menu of my AgilityNerd site. Since then I've been updating it manually about once a month based on the AWStats reports from my web hosting provider (HostMagix). This task is unnecessarily complicated because AWStats keeps page counts independently for each URL, which includes the Blosxom flavour (filename extension). For my purposes I don't want to distinguish between say HitCounterFavorites.html and HitCounterFavorites.htm

So I decided to write a plugin to track the hits per page and "lump" together counts independently from the file extension. This plugin allows me to display the hit count of visitors for each page on each page. I wrote a second plugin to use the hit count data to automatically generate my Favorites page.

Hitcounter Plugin

I based the hitcounter plugin on the categories plugin written by Todd Larason. The plugin has a couple features of interest:

• The $reset_count flag within the plugin lets you provide a starting count value for any page. For example to set a page's count to 10 append "?count=10" to the page's URL. Disable this flag once you've set your counts to avoid mischevious count setting by outsiders.
• The $retrieve_only_flavour flag within the plugin can be set to a flavour you want to use for retrieving counts without incrementing the count. Use this flavour to view the counts for URLs of interest.
• You can add filters to the start() subroutine to exclude certain requests from updating your counters. I exclude RSS and Atom feed requests from my counts.
• As of version 0.5 you can filter out loading and incrementing page counts for specific user agents via the ignore_agents array.

The hitcounter plugin stores the $blosxom::path as the key in a hash whose value is the count of hits. The hash is stored in a file in the data directory. The same hash is used for the entire site. As each page is "hit" the value is incremented and stored in the $hitcounter::count variable. I put this variable in the footer of my pages.

Favorites Plugin

My favorites plugin uses the data file containing the hash of paths and counts as its input. This plugin generates an HTML unordered list of the most visited URLs with the following configuration options:

• The $num_entries variable controls how many URLs are listed.
• The $include_counts variable controls if directories will be included in the list of entries.
• The $include_counts variable controls whether or not the number of counts is appended to each list entry.
• The $anchor_format variable controls if the full path or only the filename is displayed in the list entry.
• The $anchor_link variable controls whether or not the entry is wrapped in an anchor <a>.
• The $excludes variable controls which pages are excluded from consideration. I exclude the main URL for my site from showing up in the list.
• The $groups array holds regexps that will generate separate unordered lists grouping together pages matching each regexp of $num_entries list elements. See the configuration section for an example. I use this feature for my Favorites page to group pages for agility and tech categories into separate lists.
• The $url_flavour variable lets you specify the flavour to use for anchors to non-category URLs. It defaults to to $$blosxom::default_flavour.
• You can add filters to the start() subroutine to exclude certain requests from doing the work to generate the HTML. For example, I only run this plugin for my Favorites page.

When this plugin runs the $favorites::count variable is populated with the HTML. I then put this variable in my Favorites.txt page.

Download version 0.5 of the hitcounter plugin here.

Download version 0.1 of the favorites plugin here.

Comments/bug reports are welcome.

Loading mentions Retweet

I came up with two improvements to the first release of my refererblock plugin:

• If the referer string matches the site's URL it passed immediately and isn't checked against the blacklist.
• The blacklist.txt file was being read even if the referer string was empty. Now it is only read if the referer string is not empty and it isn't for the site's URL.

These optimizations do improve the performance of the plugin. My testing on a PIII 800MHz running Fedora Core 3 Linux with Apache 2.0 showed the following average latencies:

• 1.5 ms - Empty referer string or current domain.
• 2.0 ms - Referer string matching the first regex of the example blacklist file.
• 3.0 ms - Referer string matching the final regex of the example blacklist file.

I was kind of surprised at how little additional time was required to load the blacklist file and process the regular expressions. This is probably due to the file remaining in the disk cache for subsequent requests. Of course your mileage may vary.

Download version 0.2 of the plugin here.

See my original plugin description for installation, configuration, and testing information. Please let me know if you use this plugin or if you have comments or suggestions for improving it.

Loading mentions Retweet