One

In the article about why we don't use more data structures in web applications I showed an example of PHP code. I copied the code below. The example shows how we start to write web applications. I will rewrite the code so it says one thing: insert a person into the database.

<?php
    $name    = $_POST['name'];
    $address = $_POST['street'];
    $city    = $_POST['city'];
    db_person_insert($db, $name, $address, $city);
?>

Two things happen in this example. First we get the variables from the $_POST variable. And then we insert the values into the database. First I'll add the 'code' of the db_person_insert function as well.

<?php
function db_person_insert($db, $name, $address, $city) {
    // some SQL code...
}
?>

The function accepts four parameters: one database connection, a name, an address and a city. The last three arguments are related because they talk about the same person. The code however doesn't show us that. Let's make a change so it does.

<?php
function db_person_insert($db, $person) {
    // some SQL code...
    // use $person['name'], $person['address'] and $person['city']
}
?>

The interface of the function is much cleaner now and doesn't have to change when we change what data we want to know about people. In the previous article I showed what happened when we added the phone number. Now only the implementation of the function needs to change.

Let's go back to the first example. This code has to change to work with the new db_person_insert function.

<?php
    $name    = $_POST['name'];
    $address = $_POST['street'];
    $city    = $_POST['city'];
    db_person_insert($db, 
        array('name' => $name, 'address' => $address, 'city' => $city)
    );
?>

This code is already better, because it shows that there is only one argument that is used as a whole. But there is another change we can make, the three variables at the top refer to the same person. Why don't we have one variable representing that person?

<?php
    $name    = $_POST['name'];
    $address = $_POST['street'];
    $city    = $_POST['city'];
    $person = array(    
      'name'    => $name,
      'address' => $address,
      'city'    => $city
    )
    db_person_insert($db, $person);
?>

The structure of the program becomes a bit clearer again. Let's take it one step further and create a function that gets the values from the post variable.

<?php
    $person = person_from_postarray($_POST);
    db_person_insert($db, $person);

    function person_from_postarray($arr) {
        $name    = $arr['name'];
        $address = $arr['street'];
        $city    = $arr['city'];
        $person = array(    
          'name'    => $name,
          'address' => $address,
          'city'    => $city
        )
        return $person;
    }
?>

This still isn't the best version of the code, because there are still a few pieces of duplication. The person_from_postarray function is a bit sad, because it only makes a copy of particalur fields from $arr to the new array $person. So lets rename the function.

function person_copy($arr) {
    $name    = $arr['name'];
    $address = $arr['street'];
    $city    = $arr['city'];
    $person = array(    
      'name'    => $name,
      'address' => $address,
      'city'    => $city
    )
    return $person;
}

This piece of code is more general than the previous version, because it can be used for more problems, while it still makes sense. Now we make the change to make it more like a function that copies a person. The first step.

function person_copy($arr) {
    $person = array();
    $person['name'] = $arr['name'];
    $person['address'] = $arr['street'];
    $person['city'] = $arr['city'];
    return $person;
}

This makes the structure very obvious, but there is one field that's problematic, the street or address field. The nice thing about where we're going what this, is that we can make our lives simpler by making the fields of the person the same everywhere. So one thing we could do would be to change the street field to the address field, or the other way around. The solution depends on what you can change. Let's say we name the field 'address' and change the web interface.

function person_copy($arr) {
    $person = array();
    foreach (array('name', 'address', 'city') as $field) {
        $person[$field] = $arr[$field];
    }
    return $person;
}

Done! This function now creates a new person based on values from another array. By representing a person as one thing, we can simplify the code that works with it.

Template 'table' filter

After these two changes I got into an object-oriented (with a splash of functional and generic programming) mindset. So I created a table filter in my template code. A filter is a function that transforms a data structure to an HTML string.

This filter transforms a template parameter value from an arrayref to an HTML table. Currently it only supports lists of items, but it could be expanded to single items. I'm not sure how useful that would be, however. The question is: "How can we produce a nice HTML table from an arrayref?" The answer lies in the use of objects.

A few months ago I created a few classes that allow me to easily transform arrayrefs to HTML. I wrote one Model class and one Render class. The Model class models the information in a list. It's not always required, because the Render class also renders simple arrays, without a model.

The Model classes (there are three, at the moment), have at least three methods: update, length, item. The three classes all implement these methods.

The Render class renders these list Models. The important method on this class is the render method. It renders a HTML version of the table. The set_header method accompanies the render method. The set_header method sets the description of the table structure. This description is an arrayref with an item for each column in the table. Each item consists of, at least, a field and a title. It's also possible to add a class or a link.

These two classes are combined in the template code in the template_table function.

sub template_table {
    my ($type, $value) = @_;
    my $table = Render::Table->new();
    $table->set_header(...);
    $table->set_model(Model::Table::List->new($value));
    return $table->render;
}

This function renders the table, but notice that the set_header function lacks an argument. What argument should we use?

In a explicitly typed language we would have known the type of the $value argument. In Perl we only know that it's an arrayref. However, we can ignore the type of the value inside. The render function only needs a description of the table to be able to render the HTML. The best way to do this is, is by calling a method on the $value. In this case that's a call to the first value: $value->[0]->column_descriptions.

This change adds two preconditions to the function. Ignoring these preconditions, introduces an error. This function only works for non-empty arrays, where the first value has a method, that returns a table description.

What should we do? Can we declare these preconditions? How do we cope with these?

The first precondition — handling empty arrays — can be removed when we add code that returns a table when an empty array is passed as an argument. To solve it we return an empty string. To return an empty table based on the structure of the first value element would be really hard.

The second precondition — the value should contain the method — is already handled by Perl itself; it will throw an exception when the method missing. It could be a problem that we have to test the code, to find this error.

It like how this chunk of code enables us to remove the simpler table generating pieces of template code. These guidelines enabled me to write simpler code.

• Write code for arrays, ignorant of the type inside.
• Use objects instead of primitive types. Call methods on those objects.
• Allow the value to tell you what to do with it.
• Look for preconditions in your code and know how these are handled.
• Make your code more general by moving specific code to specific classes.

Middleware in Plack

I've been using Plack for a few days now, and I have to say, I really like it. I rewrote the base of my webshop platform to use it. The change wasn't really difficult and the code is a lot cleaner now.

Especially with the Middleware structure I was able to remove a lot of code from the main application. Middleware classes are classes that are called between the webserver and the main application. These classes have the opportunity to rewrite the request and the response classes for each request.

Because I use the same code for each webshop, but different databases for each customer, so I need a way to select the right database for a request. This is a great use for a Middleware class. The software selects and connects to the database based on the hostname in the request. The connection is then added to the environment hash. And the way back the database connection gets closed.

Furthermore there are already a few nice Middleware classes for logging and debugging. That is code you don't have to write. I really like it.

Use perl 5.10: Given / when

The feature I'll talk about here is the given/when construct, which was added in perl 5.10. It works like switch/case in other programming languages, but is much more powerful. The matching is based on smart matching, which is another feature added in 5.10;

I will start with a simple example to give you an idea of the syntax that is used.

use 5.010;

my $x = <>;
chomp $x;

given ($x) {
    when ([0..99]) {
        say "Looking good";
    }
    when ([100..199]) {
        say "That's a bit much";
    }
    default {
        say "This could be a problem";
    }
}

This code compare the value of $x with the array's in the when statements. If $x is between 0 and 99 (inclusive) it will the text Looking good. If it's between 100 and 199 then it will say That's a bit much. The default block will be called when the value isn't matched by the when blocks.

Next I will give a more useful example, but not much more.

use 5.010;

my ($x, $y) = (0,0);

LINE: while (<>) {
    my @parts = split /\s+/;

    for (@parts) {
        when (/^x(\d+)/) {
            $x = $1;
        }
        when (/^y(\d+)/) {
            $y = $1;
        }
        when (/^p/) {
            say $x + $y;
        }
        when (/^q/) {
            last LINE;
        }
    }
}

This example reads lists of tokens from STDIN and matches them and executes code based on the input. In effect it's a small programming language. Notice that this code doesn't use the given statement. It's not needed here, because the for already assigns each element of @parts to $_.

It's also possible to use simple expressions like you would use in an if statement. For example:

use 5.010;

my $age = <>;
chomp $age;

given ($age) {
    when (!/^\d+$/) {
        say "Not a number";
    }
    when ($_ > 100) {
        say "That's quite old";
    }
    when (18) {
        say "Now your life begins...";
    }
    when (0) {
        say "Just born, and already using the computer.";
    }
    default {
        say "I have nothing useful to say about '$age'";
    }
}

As you can see when is quite smart about what to do with different expressions. The first when clause contains a negated regular expression. This will be matched using $age !~ m/REGEX/. The second one do what you expect. The 18 and 0 clauses will match using $age == 18 and $age == 0. You should watch out with comparing to 0 because this will also match empty strings or just strings. For example if $age = 'hello', when(0) will match.

Smartmatching is really powerful. With given and when it's easy to use this power for deciding what to do with the value that you've been given. You should take a look at the manual for more information about the possible smart matches and the things you can with given and when.

Testing for improvement

When I'm programming, I sometimes need to use a problem solving pattern, that Kent Beck called Parallel. The pattern tells us that when your making a change you should also leave the old way of doing things in the program, so you can gradually move to the new solution. This helps in situations, where you can't just flip a switch to migrate.

The problem with this however is, that you do have to migrate all the way to the new solution, because otherwise you will have twice the code and data in various states.

When I'm using Parallel, I like to keep an eye on the progress and I don't like to go back to the old way. To help me with this I created a unit test that monitors the progress that I make, while changing the program and the data.

It would be nice to have a test module that keeps track of this, but at the moment I only have a small test program. It does three things:

1. Read the previous count (of old way occurences)
2. Check if the current count is smaller or equal to the previous count.
3. If true, the test succeeds and then writes the count to a file. If false, the test will fail.

This way the tests will only succceed if I make improvements or if the code stays the same. This way it can only improve the code.

Sending email with Perl in 2010

Example of how you should send email in 2010 with Perl.

#!/usr/bin/perl -w
use strict;

use Email::Sender::Simple qw(sendmail);
use Email::Simple;
use Email::Simple::Creator;

my $email = Email::Simple->create(
    header => [
        To      => '"Peter Foo" <foo@example.com>',
        From    => '"Peter Bar" <bar@example.org>',
        Subject => "What's up",
    ],
    body => "Hey, how are you doing?",
);

sendmail($email);

You can see that this is simple enough. First you create an Email::Simple object. This object will format the message that wil be sent.

After that sendmail will send this message using it's default transport.

The nice thing about using these modules is that you can, while testing, replace the default transport with testing modules.

use Test::More;
BEGIN { $ENV{EMAIL_SENDER_TRANSPORT} = 'Test' }
use YourModule;

YourModule->run;
my @deliveries = Email::Sender::Simple->default_transport->deliveries;
is(@deliveries, 1);

This way you can check how many emails were send using in the code your testing. By picking apart @deliviries you can check out the text of the emails that were send. See Email::Sender::Transport::Test for more information about this.

Using user-generated HTML on webpages

Jeff on Coding Horror writes:

Among programmers of any experience, it is generally regarded as A Bad Idea (tm) to attempt to parse HTML with regular expressions.

You should read the rest of the article. I'm mostly of the same opinion about this as Jeff. Additionally I think that parsing HTML is a bad idea in the first place.

One of the examples Jeff gives for parsing HTML is sanitizing user input. The user input will be used on the website as comments for example. I think allowing people to write HTML and putting it on a webpage is the wrong way to go.

If you want to include user input in a web page there are only two ways to do it:

1. Encode all special HTML characters. You can start with <, >, &, " and '. If you encode these then people can send all the HTML they want, but it's encoded and will not affect your web page. For Perl I recommend the HTML::Entities module.

2. The other way to allow HTML is, on a page, where it is their own. If they want to break their own web page, they should be allowed to do that. This doesn't mean profiles on websites, but actual websites that are completly their own.. So you say it is HTML and then use the input verbatim.

If you want to include some kind of formatting for the input, you can use a markup language. Use a markup language that allows to specify as much formatting as you like, this could be a HTML-like language, that only transforms the tags that you want. Therest of the text should be HTML encoded.

I would have lost interest somewhere between...

Adam Turoff wrote:

I could have written my taxicab number search in C or Java, but I would have lost interest somewhere between #include <stdio.h> or public static void main(String args[]) and firing up the compiler.

It's a shame this happens so often. It should be easier to just write the part your idea is about and then insert it into some thing, that will make the rest work.

Before being able to try and research your ideas, you need to have the basic code working. When people ask: why would you write your own X software? Where X is one of the pieces of software you have written, that looks like something that already exists. This would be the answer.

If there is a program or system that works for 95 percent or even 90, that would take a lot of time write. Then there is almost no incentive to create (or test) a similar tool. An example of this would be Google. Or a computer game that uses a 3D engine.

These all take many years to create, and there is not much to gain from the work that is done to get to the same level. To innovate you need to have a basis; but you will be bankrupt before you even get near.

Simple tool for finding an external ip address

I just released a small program that gets the ip address or your computer. The nice thing is, this service is REST-based.

You can find it at Stuifzand Software Tools.

Programmers and the iPad

It seems a few people wrote that they wouldn't have learned to program as a kid on an iPad instead of a real computer.

The things is that no one thought this was a problem with the iPhone or the iPod Touch. People think of those as a computer, but not, as a real one. The conclusion could be this: for a device to be a computer, it needs to be programmable on the device itself.

Using a webhook to GET information

This week I'm working on an application that needs to get a list of email addresses from another application. It has the following requirements.

Requirement #1: Only HTTP+URL+HTML. No SOAP, SQL or other technologies.

Requirement #2: The list can't be cached. It needs to be updated on the moment that the information is used.

Requirement #3: Just one URL.

Requirement #4: Authentication is very important. Only programs, that are authenticated can read this information.

So, how can we create this, while keeping these requirements in mind. First I will explain what I did. Then I will show how these requirements are met.

The first program has an interface that starts a process. This process needs to have the information of the other program. So I created a place to configure an url. This url can be anything you want. The moment the first process starts, a GET request for this url is started. It parses the information on moves on to its main task.

The second program is a web application that responds on the url with a carefully selected list of information and sends it in XML format to the requester. The other thing this program does, is authenticate the request with the HTTP authentication.

Requirement #1 is the only one that I didn't quite follow, but I'm not yet happy with how this works. It could just as easy be an HTML file.

The other requirements are all met without problems. The easiest way to authenticate the client is with HTTP authentication. This is also session-less, which is great for integrating two applications.

The only thing I will probably change is the format of the response.

Cool Websocket, Twitter and XFactor demo

Pushing the boundary of Real Time Web with Twitter and XFactor:

I am glad that I was able to show how exciting node.js and WebSockets are to make real time web application.

This is a really cool realtime web application demo. Watch the video.

Find your local IP address in Linux

Today I created this simple script that will find your local IP address or you Linux machine.

#!/bin/sh
# Shows ip address of eth0
/sbin/ifconfig | awk '/^eth0/,/^$/' | awk '/inet addr/ { print $2 }' | cut -d: -f2

It uses the output of ifconfig. First it finds the part that contains the information for eth0. Then it find the line with the inet addr, which contains your IP address. At the end it cuts the line in two parts and only prints the second part.

If you want to print the IP address of another interface, then you need to change the name eth0 to that interface name.

Spreadsheet import

osfameron asked:

I've been prototyping ways for customers to import data into our system. [...] How would you tackle this task? Is there even an elegant way to do it imperatively?

He is trying to import data into his system, but his information is not all in a flat structure. It looks like this:

Name        Price   Tag
Brie        2.00    Dairy
                    Cheese
                    Food
Chablis     5.00    Wine
                    Alcohol
                    Drink

As you can see the there is a start row and a few extra data rows after that.

The program that I would write to import this data looks like this:

use strict;
use warnings;
use Product;
use Data::Dumper;

my @products;

while (<>) {
    if (my ($name, $price, $tag) = m/^(\w+)\s+(\d\.\d{2})\s+(\w+)$/) {
        my $product = Product->new(product => $name, price => $price);
        $product->add_tag($tag);
        push @products, $product;
    }
    elsif (($tag) = m/^\s+(\w+)$/) {
        my $product = $products[$#products];
        $product->add_tag($tag);
    }
}

print Dumper(\@products);

In the way that I have written this code it doesn't need two pushes. Pushing it on the array is the first thing that I do after creating the product. The other parts of the code that need to refer to the product that was added last use the last product in the array.

My program is different to osfamerons code by using regexes, but this allowed my to actually test my code.

I want a programmable everything

As a programmer I want a programmable world. The web has been a big step forward in making things programmable. Especially in the form of REST. HTTP clients can send HTTP requests to urls and proxy can filter or change what is passed through.

In the future more real world things will become programmable. For example I can program my domain name provider. As long as I supply enough money to them, I can register as many domain names as I like.

The same thing could happen with webshops and supermarkets. Each week I will send an electronic grocery list to the webshop via the web using REST and a person will deliver it the same day to my house.

The biggest problem with this approach is that people will make it harder for automated processes to interact with websites. The webs nature is to be programmable. Google (the search engine) couldn't exist without the web. They use the programmable web since the start of their company. The way they use the programmable web is the simplest way in which the web can be used. People do it every day with their browsers.

The sad thing is, that other more complicated uses of the web are made harder to do than they should be.

I can't imagine that I would write that code in the first place

I'm afraid that's a failure of your imagination. -- Rush, SG:U.

When I'm looking at code on Stackoverflow (or some other place with bad code), I'll sometimes see people asking questions about pieces of code, that I can't imagine I would have written like that in the first place.

Why is that? I just noticed it a few minutes ago, but it looks like I have to think about that some more...

Bytecode interpreter

A bytecode interpreter is basically a loop and a lookup table. The interpreter starts at the beginning of an array of bytes. Each byte in this array is an index into the lookup table. For each key in this table there is a piece of code that needs to be executed. Code for that may look like this:

typedef unsigned char byte;

// the simplest program
byte program[] = {
    0,   
};

byte* ip = program;                   // instruction pointer

while ((ip=(lookup_table[*ip])(ip))!=0) {
    // does nothing else, but could
}

The expression in the while loop is a bit complex, but we can divide it in parts. First there is ip. This is the instruction pointer. It points to the instruction that needs to be executed.

The lookup_table is an array with the functions that correspond with the bytecode. The typedef for that looks like this:

typedef byte* (*bytecode_function)(byte* ip);

byte* func_end(byte* ip) {
    return 0;
}

bytecode_function lookup_table[] = {
    func_end,
};

The two lines define the lookup_table. The size of the lookup table is defined to be 256, because that's the size of bytes. The bytecode_function returns the next ip. This way a function can change the ip and jump to other places. If the function returns a NULL pointer, it will end the loop.

The instruction pointer ip is dereferenced to give the current byte at that place in the program. This byte is the bytecode that is used to lookup the function that needs to be executed.

The last thing is the ip argument to the function. This argument lets the function look at the bytes around the function. These bytes are the arguments to the function.

Use Perl 5.10: say

This is the first article in a series about new Perl 5.10 features. I'll try to write one article each week, but you'll never know.

This first article is about the new builtin function say. The say function is similar to print in how it works. Whenever you want to use say or other features from Perl 5.10, you need to declare that you want to use these features.

use 5.010;

After you have done that you can use the function like this.

say 'Hello world';

This will print the text 'Hello, world' to STDOUT and add a newline "\n" after it, equal to:

print "Hello, world\n";

This doesn't seem like a big feature, but it will also help with the following.

say for @lines;

This is instead of

print "$_\n" for @lines;

The say function removes some complexity in this example. You don't have to add the newline anymore.

Programming notebook

Using a small notebook or legal pad helps me to focus on the problem at hand. If I notice a problem or bug, that I don't have to fix to solve the current problem, I will write it down on the list of todo items for the program.

The other nice thing about writing down a todo list this way, is that it will grow and shrink over time. At the end of the day I have two lists: one with the things that I did that day, and one list with the things that I can work on the next time.

It's still a good idea to create tickets in the bug tracker for the big problems.

My explanation of WebHooks

HTTP requests are send by clients to servers. Most of the time the client is a webbrowser. The only thing that WebHooks says is this: servers are clients, too.

WebHooks have nothing to do with realtime updates or with sending HTTP requests to browsers. It's about servers sending HTTP requests to other servers when something happens.

Programming pattern: Temporary data

Temporary data is a simple pattern that I use while I'm trying to get some code working. The inverse of Temporary data is that you get the data, for example, from a database.

The code you have to write to get the data from the database is often many times as long as the code you need to specify the data structure.

my $languages = [
    { code => 'nl', name => 'Nederlands' },
    { code => 'en', name => 'English' },
];

If you would like to get this same data structure from your database you have to do a lot of work before this works. You need to design and create a table, fill the table with example data and write code to retrieve the values from that table.

By using this small data structure you can easily start prototyping the interface and the design of page you're working on.

After you're done prototyping, designing and coding you can refactor these four lines and replace them with:

my $languages = $db->get_languages();

Related to the simplest thing that could possibly work.

Use MySQL Fulltext for search on your website

This week I have been working on a search engine for my webshop platform. And while a big part of this work is on the layout of the result page, I also have to work on the backend.

The backend that I built is using MySQL fulltext indexing. The default MySQL was not optimal for my goals. Some queries don't return any results. This can can fixed by changing a few config variables in my.cnf.

The two variables that I changed are

1. ft\_stopword\_file
2. ft\_min\_word\_len

The first variable tells MySQL which stopword file it should use while creating an index. It will remove the stopwords from that file from the input. Stopwords are common words, that will not be used in many queries, but will increase the size of index. However, this list can contain words that are useful for your problem domain. I set the variable to '' and re-created the index. Now all words can be found, except short words.

The second variable is used to filter out shorter words. The default minimum word length is set to 4. This is to big for my website. So I have set it to 2.

With these two changes the search results show the expected results. These changes will also change the size of the index.

Logging HTTP requests with Apache

I created two simple mod_perl handlers, that help me find the response time of my web application. See it on github apache2-logging-handler.

The code is so simple that you can recreate them easily yourself. Note the use of the pnotes function.

$r->pnotes('start_time');

With pnotes you can share information between handlers. I had to search for a bit and this works great.

Try and experiment

When I am programming I will try things out. When I don't know if or how a feature of a programming language works, the way to find out is just trying it. The nice thing about computers and programming languages is that they won't work or will complain, when something isn't the way they expect it.

For Trying and experimenting to work, you'll need two things: feedback and reasoning skills. I could write here that you also need creativity, but I can't write anything useful about that yet.

Feedback is the way that the computer will tell you that the thing you just did, didn't work. Feedback exists in many forms: log files, error messages, CPU usage, load times, process lists, files and blank screens, all give you some information about your program. Every thing that gives some notion of the state of the working of the program, can help you find out what's going on.

Most of the time, more feedback is better. Of course with more feedback there comes more noise. Knowing where to look for certain kinds of problems will help you find your problem faster. Grepping and tailing can also help a lot when examining logfiles.

For websites there is a extension to Firefox called Firebug, which will let you inspect the HTML, Layout, CSS and DOM tree of you webpage. You can also try out if a part of your website will look better if you change margin or other parts of the CSS file without actually making the change to your websites.

Reasoning skills will help you help compare your model of the program with the actual reality of your program. Having such a model of the program in your head is the most useful thing. By trying and experimenting you will build this model. When your model is not good enough, you reasoning skills will help you find out where a problem is occuring. It is useful to add more feedback. Adding a 'print' statement before, between and after lines of code, will show which lines are executed and which aren't.

This way you can also see what the values of variables are. By comparing the values of the variables with the values that you expect, you can see where your model differs from reality.

Walking trough your code line by line is useful if you want to find out which line makes the output differ from the expected output.

Creativity will help you find new solutions or new theories for the problems you find. That's all I have to say about this.

Trying and experimenting is a safe and useful way to find problems and their solutions.

Question of the day:

Which kinds of feedback do you use while coding?

jQuery and postJSON

The last few days I've been using jQuery to write some REST based javascript and HTML. While I was doing that I needed a function that looked like getJSON, that did a POST instead of a GET. The difference is that I send a bit of JSON and receive a bit of JSON, instead of sending form-like data. So I created the following bit of code:

$.postJSON = function (url, data, callback) {
    $.ajax({
        'url': url,
        'type': 'post',
        'processData': false,
        'data': JSON.stringify(data),
        contentType: 'application/json',
        success: callback,
    });
};

If you know jQuery a bit, you will know that this sends an Ajax request to url. Later when I used this code to do a POST a followed by a GET, I didn't receive the thing that I wanted: a Javascript object. I did however get a string that looked like one. That confused me for a bit. Then I remembered a FAQ or a weblog post about another implementation of postJSON, that was written like this.

$.postJSON = function(url, data, callback) {
    jQuery.post( url, data, callback, "json") ;
};

It seemed my problems where solved. When I clicked around my website, everything had stopped working. I checked what this code was doing and it did something different than what I expected. It doesn't send the data variable as JSON, but as application/x-form-encoded.

First the problem was that the code didn't parse the return value. Then the problem became that the code didn't encode the arguments as JSON. When I found the problem the change was easy: change the callback to parse the return value. The final code became this:

$.postJSON = function (url, data, callback) {
    $.ajax({
        'url': url,
        'type': 'post',
        'processData': false,
        'data': JSON.stringify(data),
        contentType: 'application/json',
        success: function (data) { callback(JSON.parse(data)); },
    });
};

The current problem is that this method has a different interface than the normal postJSON. So I have to choose a different name.

Using ack to check coding conventions

Do you know Ack? It's a grep-like program. That uses perl regular expressions instead of the normal Posix ones. You can find it on the CPAN.

The following Ack call will check your perl code for problem with a space missing behind a controlstatement keyword.

ack --perl '(?\@<!\w)(if|while|elsif|return)('

If you use Vim you can also use the following piece of vimscript in your .vimrc file:

highlight WHITE_ON_RED ctermfg=white ctermbg=red

function! BadNonInvocations ()
    2match WHITE_ON_RED /\w\@<!(if\|elsif\|while\|return\|for)(/
endfunction
call BadNonInvocations()

I released a HTML parser

I created a HTML parser written in C++. It's based on the HTML5 specification. The code can be downloaded with git:

git clone http://code.peterstuifzand.nl/git/htmlparser.git/

or:

http://code.peterstuifzand.nl/cgi-bin/gitweb.cgi

I use this HTML parser in a internal searchengine. It parsed everything the spider could find.

If you like to use this parser, look for the Parser class. This class expects an Emitter object. This can be a ListEmitter or a subclass of Emitter. I wrote a simple Emitter that finds all <a href=""> tags and inserts them into a MySQL table.

Other examples of Emitter that you could write are a tag remover.

class TagRemover : public Emitter {
    public:
        virtual void emit_char(char c) {
            std::cout << c;
        }
        virtual void emit_multichar(std::string s) {
            std::cout << s;
        }

        virtual void emit_tag(const Tag& tag) {}
        virtual void emit_comment(std::string comment) {}
};

This will show all characters in a HTML page (including whitespace). Removing consecutive whitespace is left as an exercise for the reader.

Ternary operator

The ternary operator can be found in many programming languages. In some popular ones it is written as bool ? expr1 : expr2, which will evaluate to expr1 if bool is true or expr2 if bool is false.

The nice thing about the ternary operator is that one can write an otherwise big if block in one line. For example:

$var = null;
if ($x > 10) {
    $var = 102;
}
else {
    $var = 1044;
}

becomes

$var = $x > 10 ? 102 : 1044;

I hope it's obvious what the pros are of this approach.

But then I read an article which tries to explain the usage of the ternary operator in Java. At first I liked the simple design of the page. But the code examples are probably some of the worst that I have seen.

boolean giveTicket;
giveTicket = speed > speedLimit ? true : false;
if (giveTicket)
    pullEmOver();   // nab the offender!

The biggest problem with this example is that second and third part of the expression are true and false. The first part of the expression itself will already evaluate to that, so that's redundant. This example can be written without the ternary operator.

if (speed > speedLimit) {
    pullEmOver();
}

This is cleaner and contains less words. To have the same obvious code, you could create a method for that comparison.

public boolean giveTicket(int speed) {
        return speed > speedLimit;
}

...

if (giveTicket(speed)) {
    pullEmOver();
}

It won't get better than this. The nice thing about this is that you can change the code in giveTicket if there are changes to the ideas about what is illegal.

I programmed my first attiny2313

I programmed my first Atmel ATTiny2313 today. Of course this was a blinking led program.

I used the following code:

#define F_CPU 8000000UL
#include <avr/io.h>
#include <util/delay.h>

int main(void) {
    DDRB |= 1<<PB0; /* set PB0 to output */
    while(1) {
        PORTB &= ~(1<<PB0); /* LED on */
        _delay_ms(100);
        PORTB |= 1<<PB0; /* LED off */
        _delay_ms(900);
    }
    return 0;
}

This will blink a led if it's connected to PB0 (or pin 12 on the microcontroller).

To program the chip I used the avrdude program. I use the following commandline:

avrdude -p t2313 -c avr910 -P /dev/ttyUSB0 -e -U flash:w:blink.hex

The programmer that I used is the MySmartUSB programmer. This works on Linux, the drivers were already compiled in my kernel. When I connected the programmer, dmesg showed me that it worked.

The MySmartUSB programmer has a 10 pin avr connector for programming microcontroller. This means you need an extra board that has the pins for the chip. I created one myself. This is not that hard. The only thing you need is a pinheader (10 pins) a IC foot (I used 20 pins, as I was programming a ATTiny2313) and a few wires.

A visual explaination of SQL joins

The visual explaination of SQL joins contains the explaination of SQL joins that I never knew how to give. If you don't understand joins completely, this will give you a visual model to help you understand SQL joins.

Sanitize your database inputs

I just saw this over on xkcd.

Slowchat

Hmm slowchat. Why would anyone want it? I created this little piece of slowchat goodness on the right side here.

Post some nice messages. The winter is coming and it could be cold. Show me some love :).

My very own ticket system

I created my first Ticket System. It may look a bit like RT, but that doesn't matter to me. The code was written from scratch in about 12 hours.

I used DBIx::DWIW, HTML::Template, CGI::Session and Web::Controller. This last module was created by myself, to make it easier to create web applications. The other modules weren't written by me, but did make it easier to create this application.

I created a demo version of the ticket system. You can login using: demo/demo. So give it a try and send me your opinions.

Deploying web applications

Deploying stuff is hard enough already. And so it shouldn't be made harder by code that isn't in source code repositories. Read my latest article about these kinda things and some ways to make that whole thing a bit better.

The two things that I forgot to say are:

1. make it repeatable,
2. make in one command.

Plagger: a platform for feeds

While I was reading the Perl planet I found a link to a project called Plagger. It's system for creating, changing and publishing feeds.

To create a new 'program', you need to make a config file. The example directory includes a bloglines2gmail config and a config for a 'planet' aggregator. Both are less than twenty lines of configuration. It's really easy to create your own feeds with this.

Be sure to check out the presentations on the website, for an explaination of Plagger (the documentation is a bit light).

A clipboard webservice

This is my first test with my new webbased clipboard system, everybody can copy a piece text to the server. When you want to paste the text again, you can do that from any computer that has an internet connection. The last thing you need is the code that you got when you copied the text.

One thing left to say: don't send in stuff that you don't want anyone else to read. The text is saved for about one hour (actually between one and one-and-a-half hour). Alsa anyone with enough time on his hands could go and guess the codes; so don't do it, or don't care; both will work. I'm not responsible and warned you.

Added comments to the weblog

Ok, I think it's time for this. I added the comments feature to my website software. It works really great now.

It was a problem when I couldn't add comments, because the pages are static. It can be made, so I did.

A new website: Imba-nator

Yesterday I created a new website. Everyone can type statements. After that people can vote about the Imba-ness of the statement.

Check it out: imba.peterstuifzand.nl.

A ftp deployment tool: ftpup

This weekend I wrote a ftp deployment tool. It helps me deploy my changes faster to my concert website.

The program is written in ruby. It will search for the file .ftpup in this or a higher directory. In this file you can specify some rules that let the program know where to upload the files to.

The rules use regex to specify where the file should go. To let this approach work as it should, it will first find out where the file is, relative to the project 'basedir'. The basedir is the directory where the .ftpup file is.

When all is set, you can call ftpup [filenames...] to upload those files. The nice thing is that this can be done from any directory below the basedir.

The sourcecode can be found in my repository and you'll need darcs to get it. It has a GPL license so I would like it if patches come back to me.

Using ruby-xosd to show a timer

After the articles on 43folders about using timers for specific things, I thought I would create a timer program. Now I have a few programs that I 'integrated' in my window manager. With Alt-F4 ion3 will show a query. I can type the number of seconds the timer should run for. A timer is shown in the lower right corner of the screen (which is already evil enough).

A next version will probably make it possible to show and hide with a push of the button.

The program can be found in my darcs repository. The ruby-xosd extension can be found on the ruby-xosd homepage and not in my repository.

My new repository uses Darcs

Darcs is a distributed revision control system. It can be used to keep revision of documents (which includes sourcecode). I tried to use it to keep some projects. The repository is online at http://darcs.peterstuifzand.nl and is read-only.

The nice thing about darcs is that anyone can be get a version of the repository on his own computer. That repository is a complete repository and can be used to keep revisions locally. Changes can be push-ed to or pull-ed from the remote repository. This makes it easy to work on a local copy, while keeping different revisions (actually patches).

More information can be found on the darcs homepage and on the darcs wiki.

Also feel free to check out the projects in the repository and send patches.

Time in a packet

I was so nice to create a package for osdtime.

Source

osdtime-0.1.tar.gz.

On screen time display

Today I liked to know the time. I don't have a clock and on my computer it's to much work to show the time. I use ion3 and it can show the time in a statusbar, but I don't like that, because it takes too much of my screen real estate.

So I created a little program that shows the time for a few seconds in the right bottom corner of my screen. It's called (how original) osdtime.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <locale.h>
#include <xosd.h>
#include <time.h>

/* seconds to display time */
int seconds_to_display = 3;

/* display font */
const char* display_font = "-adobe-helvetica-bold-r-normal-*-*-320-*-*-p-*-*";

/* time format string */
const char* time_fmt = "%H:%M:%S";

/* text-colour */
const char* text_colour = "white";

/* outline-colour */
const char* outline_colour = "black";

int main(int argc, char *argv[])
{
  xosd *osd;
  setlocale(LC_ALL, "");
  osd = xosd_create(2);
  if (osd == NULL) {
    perror("Could not create \"osd\"");
    exit(1);
  }
  /* Set the position of the display. */
  xosd_set_pos(osd, XOSD_bottom);
  xosd_set_align(osd, XOSD_right);

  /* Set the font and the colours. */
  xosd_set_font(osd, display_font);
  xosd_set_colour(osd, text_colour);

  xosd_set_outline_offset(osd, 2);
  xosd_set_outline_colour(osd, outline_colour);

  /* Display the time for some seconds. */
  int i;
  for (i = 0; i < seconds_to_display; i++) {
    time_t tm;
    struct tm *localtm;
    time(&tm);
    localtm = localtime(&tm);

    char buf[9];
    strftime(buf, 9, time_fmt, localtm);

    xosd_display(osd, 0, XOSD_string, buf);
    sleep(1);
  }

  xosd_destroy(osd);
  exit(0);
}

It uses libxosd to show the text. As always, it comes without any warranty. Use at your own risk. It can be compiled with the following command line.

gcc `xosd-config --libs --cflags` osdtime.c -o osdtime

At the beginning of the program it's possible to set the variables to nicer values if you like.

At the ion3 side of this hack I changed the ~/.ion3/cfg_bindings.lua file. The time will be displayed when I press F11. This can be done by putting the following lines in the bindings file after the line that uses F9.

bdoc("Display the time."),
kpress(MOD2.."F11", "ioncore.exec_on(_, 'osdtime')"),

Using a script to update Mediawiki

I created a script that allows me to update a page on a media wiki. It uses WWW::Mechanize, IO::All and Config::Std. The last two modules aren't needed probably. But it makes it all a lot simpler.

#!/usr/bin/perl -w

use IO::All;
use WWW::Mechanize;
use Config::Std;

read_config "$ENV{HOME}/.mediawiki" => my %config;

my $username = $config{login}{username};
my $password = $config{login}{password};
my $hostname = $config{login}{hostname};

my $title = $ARGV[0] or die "usage:   mediawiki [page title with under scores]\n\n";
my $url = "http://$hostname/mediawiki/index.php?title=$title&action=edit";

my $ua = WWW::Mechanize->new;

my $login_url = "http://$hostname/mediawiki/index.php?title=Special:Userlogin&returnto=Main_Page";
$ua->get($login_url);

$ua->field('wpName', $username);
$ua->field('wpPassword', $password);
$ua->click('wpLoginattempt');

$ua->get($url);

$ua->form_number(1);

my $form = $ua->current_form();

my $tmpfile = io('/tmp/mediawiki-1');
$form->value('wpTextbox1') > $tmpfile;

system('vi /tmp/mediawiki-1');

$tmpfile->open('r');

my $content = $tmpfile->slurp;
$ua->field('wpTextbox1', $content);
$ua->click('wpSave');

print "Done!\n\n";

The configuration is read from /home/$USER/.mediawiki. It should look something like this.

[login]
username=your name
password=your password
hostname=your wiki hostname

Where hostname should be something like 'localhost' or 'www.somedomain.org'.

Oh, this code comes without any warranty. Use at your own risk.

Plugins with perl

This post is not about perl plugins (what would that be?). But about programs that use plugins. One of those programs is Qpsmtpd, a perl mail server. They use a special and very nice way to use plugins.

Moveable Type is another perl program, that uses plugins. These plugins are a little bit simpler than the qpsmtpd plugins, but just as powerful.

And now there is another perl program that uses plugins, my webshop. It is really nice to see how easy it is to create a new type of shop on top of the current code. I can't yet show it, but within a few days it will be online.

Another music player

The day after I posted the previous post, I became aware of a serious flaw in the design of the player. It couldn't play music from on artist or one album. I really liked this functionality, so I wrote it.

MUSICDIR=~/music
cd $MUSICDIR
find . -type f -name "*.mp3" -o -name "*.ogg" | grep "$1" > playlist
mplayer -quiet -shuffle -playlist playlist | grep Playing

Also it tried to play the cover images that are in some of the directories. This was solved by specifying the extensions of the files to play.

My new music player tool

Today I created a new music tool quiet like the Apple Shuffle. It will play music files randomly from my 'collection'. Here follows the code.

#!/bin/sh
cd ~/music
find . -type f > playlist
mplayer -quiet -shuffle -playlist playlist | grep Playing

It can be started in it's own terminal and be controlled with all the normal mplayer combinations. PgUp and PgDn will skip songs. Left and Right will skip in a song. + and - will change the volume. Q let's you quit.

It's a little crude at the edges, but it's probably the best tool I've written in a while. No stuff that will keep you from getting your job done.

A code generator

At the company, where I'm doing my internship, I needed a code generator for some C code. There was some C code needed. So I started of writing it. After some lines, there started to show some patterns, which could be easily handled by some description language.

The first description language was a line based format. First a line with the name of the object. And after that the names of the functions. This format is easy to parse and easy to use. The output was done with print statements.

For code generation I always used this method. But why should the webapplications have all the template fun? So I downloaded Template (which is the Template Toolkit), from the CPAN and I tried some things with it. It was really easy to generate code like this. The code was also much cleaner and shorter.

All nice and easy, but not good enough, because it should be possible to add C code to the functions. I almost started to try and create regular expressions for this task, but then the laziness started to kick in again. I remembered something about a Text::Balanced, which is written by Damian Conway. I can extract delimeted text from strings. Which was exactly what I wanted to do.

The code is again much smaller, and writing the regexen would be the wrong way to go. In this case Text::Balanced is the middle ground between regexen and some parser written with something like Parse::RecDescent. Laziness + CPAN is a good thing.

Weblog now contains monthly archive pages

Today I added monthly archive pages on this weblog. The design is not clean yet, but it works, like you can see at the archives. The archive page now contains a list of links to the pages of the months with posts.

The archive first contained a pages with all the posts, which uses a little little too much bandwidth. This new approach uses just a little.

Two new articles

I have written two new articles. First is about using time based movement in a computergame. The other is about creating a Java/Swing GUI with RiverLayout. I will also try to create more articles about GUI programming in java and maybe about using invariants and contracts in object oriented programming.

My first game programming article

Yesterday I have written my first article about game programming. I have ideas for some other articles that I will write in some days. It's really fun to write, but I'm not yet sure if there are people who think the article is helpful.

I would appreciate comments, ideas, suggestions, feedback about the article.

Webwinkel Reloaded

Hobby-Koopjes.nl has been upgraded multiple times. I've added all kinds of functionality.

• New products
• Links to other pages
• Ability to specify if a product should be put into a packet.
• Some cleanups of the order database code.

All in all not that many upgrades, nut at least the structure of the code is better suited now to be used with the order/billing code.

A Webmail To RSS tool

At my school we now have to use a webmail client. And I don't like it. Because it is a lot of work to open up the page, enter a password, look over the messages and find nothing new.

I use Liferea as my RSS Feed Reader and it now uses a program to get a feed of the messages. The program downloads the webpage with the messages and then generates a simple rss2.0 feed from that.

If I have cleaned up the sourcecode I will put it on my webpage as free software. This will solve some real problems some students have with the e-mail. It will at least solve some of the problems.

Notemanager Webapp

Today I started with the creation of a webapp, in which you can make notes. It's really simple, but already it has some nice features, like categories. It is written in ruby and uses YAML to save the notes. This makes it easier to save and load the notes.

Nightly Build

I'm thinking of creating a nightly build of the webshop I'm working on. But what is a nightly build or what should it do?

1. get newest files from CVS
2. prepare/compile files
3. install files to some directory that's visible through the webserver
4. run all tests
5. create all kinds of reports

Possibly it should do more, but this is at least some of it.

Bugfixing and rss feed

After some bugfixing, I now have the permalinks working right. I also added an RSS2.0 feed. I'm not yet sure if it's the way I like it, or that's correct. But it's a test so.

Website generation software

The website generation software I use has a few little flaws.

• It uses the filename for the date. (e.g. 20040927174819.tp)

  This actually goes two ways. On the one hand it's a unique name for the file. On the other hand, it has metadata on the outside of the file.

• There's no easy way to add more data to an entry. (e.g. I can't add category's)

  Actually I can but then I've to change the entry format.

• By the generation of the permalinked pages, the pages get a wrong link to themselves.

• There is no RSS file. This one is really important.

New Weblog Software

I created some new weblog software for myself. It's written in ruby, uses BlueCloth, and is very simple.

Actually it's a part of a system that's a little bigger. That system generates my webpages. To create a website you only have to write the content, all the stuff is automatically created.

Weblog gui tweaks

Today I also made some tweaks to the layout of my weblog. It's really nice like this. Some extra space between post of another day and earlier posts.

SpringLayout is really nice

Today I was creating some panels for webshop manager program I'm creating. I used SpringLayout for this. It's really nice, also because I found a class called SpringUtils on java.sun.com somewhere in a tutorial. With this class you can connect the components on a page. And they will be shown in a table-like fashion. At last it's possible, for me, to create nice and professionally looking interfaces using java.

But the strange thing is that when I tried to create the same interface using gtk, I was done in a about five minutes.

GridBagLayout is evil!!

Yesterday I found an animation on the web. It will show you why gridbags are evil. It's briljant. Especially because I coding a GUI myself, yesterday. An I was about to use the gridbag and the memory came back to me...

Totally Gridbag

Forum improvements

It's still needed to create a better forum. I've have installed a forum on the roadtrip website, but their is nobody that posts a message.

I would be nice if someone would post a message, but if you need to register to read the forum, a lot of people will not try it. Some people have done research on this. And I think it's true. There was also a problem with some cookie stuff, so people couldn't login.

Now I want to make the login code optional for everything. So that you don't have to login even for posting messages. Maybe then people are going to post some messages.

Changes to the blog

As you can see my blog is powered by Bryar. It can be found on CPAN. I made some changes to Bryar that I liked. I'm not sure what to do with these changes. Should I give them back to the owner, or not? It's more like a it works for me hack.

I added the following items to the software.

• Wiki formatting using Text::KwikiFormatish
• Separated datadir and entrydir.
• Simple attr_accessor, attr_reader (like ruby) functions

These items are really nice to have I think. There is some other stuff that would be nice.

• Use of links to the baseurl of your webserver. (like: /index.html).
• That the ordering of the entries doesn't change, when you change it.
• Because Bryar uses the modification time of an entry.
• It would be better to use the filename of the entry, especially of you use the my elisp bryar entry maker. It's nice that this software can be changed so easily.

Emacs to the rescue

Now I've a blog, but it's not easy to add an item. To make it somewhat easy, I have created an elisp functions in emacs to create an entry.

the elisp file