January 31, 2018

Writing a WebSub hub

This week I've been writing a WebSub hub. It's server program that receives subscription and publish requests and sends notications to the subscribers when a website changes. The website itself sends a request when the website changes. This way the subscribers now when to update their feeds.

Implementing the server quote simple, epecially with the helps of a website that checks your implementation, websub.rocks.

WebSub Rocks! is a validator to help you test your WebSub implementation. Several kinds of tests are available on the site.

By reading the spec and following the steps from websub.rocks. It was quite simple. The problem is that not all websites implement this exact specification. FeedBin, for example, implements a feature from an older version of the spec and won't validate the subscription, when it otherwise would be good. A small change makes me support it, but I also send a commit to the FeedBin software on Github. That removes that use of the feature. As expected, this change leads to more changes, so we will see how this pans out.

Just fixed a small problem with loading and saving subscriptions in the hub. It again shows that it's a good idea to check all error's from function calls in Go.

#

Sending the websub notification seemed to have worked. The previous was added to FeedBin.

#

Trying WebSub with this weblog. I have a subscription from Feedbin. Now I will try publishing from the feed.

#

January 16, 2018

January 14, 2018

May 27, 2017

Collection and select

Sometimes you need to get a lot of information from the database and you need it fast. The way to get this information normally would be to use the Magento collection for the model. It would look like this:

$collection = Mage::getModel('customer/customer')->getCollection()
    ->addNameToSelect()
    ->joinAttribute('billing_city', 'customer/city', 'default_billing', null, 'left');

foreach ($collection as $customer) {
    // Use $customer
}

The problem with this way of getting the information is that it loads all customer information into memory, which can take a long time when you have many customers.

The solution to this performance problem is that you read the customers one at a time by using the select that was created.

$select = $collection->getSelect();
$stmt = $select->query();

while ($row = $stmt->fetch()) {
    // Use $row
}

This is often many times faster and won't fill your memory.

March 31, 2016

XML-lister

XML lister is a small program that lists the elements, values and attributes of XML files.

<?xml version="1.0"?>
<layout>
    <default>
        <reference name="root">
            <reference name="content">
                <block name="product.info">
                    <action method="test">
                        <param>test</param>
                        <param2>test2</param2>
                    </action>
                </block>
            </reference>
        </reference>
    </default>
</layout>

XML lister will print the following list:

layout  
layout/default  
layout/default/reference    
layout/default/reference@name   root
layout/default/reference/reference  
layout/default/reference/reference@name content
layout/default/reference/reference/block    
layout/default/reference/reference/block@name   product.info
layout/default/reference/reference/block/action 
layout/default/reference/reference/block/action@method  test
layout/default/reference/reference/block/action/param   test
layout/default/reference/reference/block/action 
layout/default/reference/reference/block/action/param2  test2
layout/default/reference/reference/block/action 
layout/default/reference/reference/block    
layout/default/reference/reference  
layout/default/reference    
layout/default  
layout  

A few days ago I wanted to find out if there are places in Magento layout XML files where reference elements are used inside other reference elements. With XML lister it becomes really easy to find places like this.

find app/design -name '*.xml' | xargs xml-lister --filename | grep reference/reference

With the list it becomes really easy to find the files where a certain construct is used.

March 16, 2016

Rewrite XML with xmlstarlet

XML Starlet is a program that makes transforming XML documents really simple.

The command that you need to transform an XML document is the following.

xmlstarlet tr [<options>] <xsl-file> {-p|-s <name>=<value>} [ <xml-file-or-uri> ... ]

The XSL file that transforms a document to itself is this. It copies all elements, attributes and texts. We can extend this file to remove, add, or transform elements.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

To remove an element, we match an element and don't call apply-elements inside it. This example will remove the remove me element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <!-- Remove the element -->
    <xsl:template match="remove-me">
    </xsl:template>
</xsl:stylesheet>

To add an element, we just add the part we would like to add. If it needs to be in a certain place, we can add an extra xsl:template rule. The this-will-get-added element will be inserted after the element element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>

        <this-will-get-added></this-will-get-added>
    </xsl:template>
</xsl:stylesheet>

To transform an element we remove xsl:copy and replace it with the elements we need. If we keep the xsl:apply-elements tag in the xsl:template tag, we copy everything inside the element in it's place.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <my-new-element>
            <xsl:apply-templates select="@* | node()" />
        </my-new-element>
    </xsl:template>
</xsl:stylesheet>

XSL used in this way becomes quite simple to use. With these three patterns we can extend the identity transformation and get quite a bit of power when we transform XML documents.

November 6, 2014

September 4, 2014

Sorting DNS records

I don't update my DNS records often, but when I do I like these records sorted.

A DNS record consists of the following fields: domain name, type and prio (for some types). I use these fields to determine the order of the records. That is also the order in which these fields should be ordered.

The first insight I had about sorting domains was that the domain names consist of parts separated by a period. The domain names may be strings. But a domain name is actually a reversed array of strings. We map the domain name from "sub.example.com" to [ "com", "example", "sub" ]. This puts it in an order that makes lexicographical compare work. To compare domain names is to compare these arrays of parts with a lexicographical compare function.

sub lexicographical_compare(\@\@) {
    my ($a, $b) = @_;
    my $i = 0;
    while ($i < @$a && $i < @$b) {
        my $c = $a->[$i] cmp $b->[$i];
        return $c if $c; # $c != 0;
        $i++;
    }
    return @$a <=> @$b;
}

sub compare_dns_record {
    my ($a, $b) = @_;

    my $c = lexicographical_compare($a->{domain_parts}, $b->{domain_parts});
    return $c if $c;

    $c = $a->{type} cmp $b->{type};
    return $c if $c;

    $c = $a->{prio} <=> $b->{prio};
    return $c;
}

The second insight here is that the record itself can also be compared lexicographically. But this is harder to write in Perl. In Haskell, for example, this is the default for tuples.

August 18, 2014

Rotate one to the left

Last week I wrote a small function in Perl that rotates and array to left. The function takes the item from the front and appends it to the back.

# precondition: @list >= 1;
sub rotate_left_by_one {
    my $ar = shift;
    my $item = shift @$ar;
    push @$ar, $item;
    return;
}

After calling this function as many times as the length of the array, it leaves the items in the same order they started in. Every call to this function puts a different item in the first location. I use it to compare this item to the rest of the items.

You would use it like this:

while (1 .. @list) {
    # your code
    rotate_left_by_one(\@list);
}

This function works great when the code doesn't depend on the order of the items in @list. The items in @list will be in a different order than what it started with for most of the iterations of the loop. This means that the code in the loop body needs to have the commutative property.

May 23, 2014

Low level code

Google released the PDF code of Chrome just a few days. I took a look at it. This file (core/src/fpdfdoc/doc_vt.cpp) is an example of how the code looks.

Sometime later it dawned on me that this code is built out of only low level pieces. There are almost no algorithms extracted to functions.

An example would binary search. There are five copies of binary search hidden in just that file. No wonder this file is 1864 lines of code.

What else would be hidden in there?

Welcome

My name is Peter Stuifzand. You're reading my personal website.

Profiles

Peter Stuifzand
peter@peterstuifzand.nl
Zwolle, The Netherlands