August 15, 2018

A tour of my Indieweb projects

Earlier this year I started to rebuild this website to work better with the Indieweb. At the same I build a new website P83 with Indieweb support from the start.

This website is generated from static files with a system that I created the first time when I started writing in the website. Not much has changed, except that I some point a few years ago I rewrote the whole system in Perl.

The only actual changes to this website, was adding some support for smaller posts, and having a better view of multiple posts on the same day. I also added microformats to the posts.

I still think there is a lot to like in a static website. It's even possible to combine the static nature of pre-generated pages with the dynamic properties of the Indieweb like webmentions and comments. Quite a few people support that usecase. Some even use Micropub to write new posts for their blog.

This blog doesn't support any of that, except webmentions.

P83 is the new name of the other blog. It started out from an old code base, that I was working on a few years ago. The software running it is called publog (from something like publishing log). It's quite a bit more dynamic software. It supports webmention and microsub. It supports projects, different channels, syndication to other websites (like Indieweb.xyz, Twitter and Github), Bridgy and all kinds of small experiments.

In the last few months, I have build out the functionality for connecting with the Indieweb. I improved the presentation of the posts, the microformats and the parsing of microformats. The microformats organization on Github gathered many parsers together and have a repository with tests. This helps a lot when you want to work with microformats in your own projects. I have written my own version of a microformats parser for Perl. But I'm sure it's not very complete compared to the parsers for the other languages. I should add the test suite to this parser and see how far it gets. Perhaps I should even rewrite it, as it could contain a lot of unused code. The microformats parser is the weak link in the website. I should try to improve it.

When I found that there was no Android client for Micropub, I dusted of my Android developer account and started working on the client. Android Studio helps a lot. Today the client works pretty well, and I hope that it can be moved to a production release in the coming weeks. I only still need some confirmation from people using it with some of the more popular systems (like Wordpress).

I'll mention the Microsub server (Ekster) in my next post.

January 31, 2018

Writing a WebSub hub

This week I've been writing a WebSub hub. It's server program that receives subscription and publish requests and sends notications to the subscribers when a website changes. The website itself sends a request when the website changes. This way the subscribers know when to update their feeds.

Implementing the server is quite simple, especially with the help of a website that checks your implementation, websub.rocks.

WebSub Rocks! is a validator to help you test your WebSub implementation. Several kinds of tests are available on the site.

By reading the spec and following the steps from websub.rocks, it was quite simple. The problem is that not all websites implement this exact specification. FeedBin, for example, implements a feature from an older version of the spec and won't validate the subscription, when it otherwise would be good. A small change makes me support it, but I also send a commit to the FeedBin software on Github. That removes that use of the feature. As expected, this change leads to more changes, so we will see how this pans out.

Just fixed a small problem with loading and saving subscriptions in the hub. It again shows that it's a good idea to check all error's from function calls in Go.

#

Sending the websub notification seemed to have worked. The previous was added to FeedBin.

#

Trying WebSub with this weblog. I have a subscription from Feedbin. Now I will try publishing from the feed.

#

January 16, 2018

January 14, 2018

May 27, 2017

Collection and select

Sometimes you need to get a lot of information from the database and you need it fast. The way to get this information normally would be to use the Magento collection for the model. It would look like this:

$collection = Mage::getModel('customer/customer')->getCollection()
    ->addNameToSelect()
    ->joinAttribute('billing_city', 'customer/city', 'default_billing', null, 'left');

foreach ($collection as $customer) {
    // Use $customer
}

The problem with this way of getting the information is that it loads all customer information into memory, which can take a long time when you have many customers.

The solution to this performance problem is that you read the customers one at a time by using the select that was created.

$select = $collection->getSelect();
$stmt = $select->query();

while ($row = $stmt->fetch()) {
    // Use $row
}

This is often many times faster and won't fill your memory.

March 31, 2016

XML-lister

XML lister is a small program that lists the elements, values and attributes of XML files.

<?xml version="1.0"?>
<layout>
    <default>
        <reference name="root">
            <reference name="content">
                <block name="product.info">
                    <action method="test">
                        <param>test</param>
                        <param2>test2</param2>
                    </action>
                </block>
            </reference>
        </reference>
    </default>
</layout>

XML lister will print the following list:

layout  
layout/default  
layout/default/reference    
layout/default/reference@name   root
layout/default/reference/reference  
layout/default/reference/reference@name content
layout/default/reference/reference/block    
layout/default/reference/reference/block@name   product.info
layout/default/reference/reference/block/action 
layout/default/reference/reference/block/action@method  test
layout/default/reference/reference/block/action/param   test
layout/default/reference/reference/block/action 
layout/default/reference/reference/block/action/param2  test2
layout/default/reference/reference/block/action 
layout/default/reference/reference/block    
layout/default/reference/reference  
layout/default/reference    
layout/default  
layout  

A few days ago I wanted to find out if there are places in Magento layout XML files where reference elements are used inside other reference elements. With XML lister it becomes really easy to find places like this.

find app/design -name '*.xml' | xargs xml-lister --filename | grep reference/reference

With the list it becomes really easy to find the files where a certain construct is used.

March 16, 2016

Rewrite XML with xmlstarlet

XML Starlet is a program that makes transforming XML documents really simple.

The command that you need to transform an XML document is the following.

xmlstarlet tr [<options>] <xsl-file> {-p|-s <name>=<value>} [ <xml-file-or-uri> ... ]

The XSL file that transforms a document to itself is this. It copies all elements, attributes and texts. We can extend this file to remove, add, or transform elements.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

To remove an element, we match an element and don't call apply-elements inside it. This example will remove the remove me element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <!-- Remove the element -->
    <xsl:template match="remove-me">
    </xsl:template>
</xsl:stylesheet>

To add an element, we just add the part we would like to add. If it needs to be in a certain place, we can add an extra xsl:template rule. The this-will-get-added element will be inserted after the element element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>

        <this-will-get-added></this-will-get-added>
    </xsl:template>
</xsl:stylesheet>

To transform an element we remove xsl:copy and replace it with the elements we need. If we keep the xsl:apply-elements tag in the xsl:template tag, we copy everything inside the element in it's place.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <my-new-element>
            <xsl:apply-templates select="@* | node()" />
        </my-new-element>
    </xsl:template>
</xsl:stylesheet>

XSL used in this way becomes quite simple to use. With these three patterns we can extend the identity transformation and get quite a bit of power when we transform XML documents.

November 6, 2014

September 4, 2014

Sorting DNS records

I don't update my DNS records often, but when I do I like these records sorted.

A DNS record consists of the following fields: domain name, type and prio (for some types). I use these fields to determine the order of the records. That is also the order in which these fields should be ordered.

The first insight I had about sorting domains was that the domain names consist of parts separated by a period. The domain names may be strings. But a domain name is actually a reversed array of strings. We map the domain name from "sub.example.com" to [ "com", "example", "sub" ]. This puts it in an order that makes lexicographical compare work. To compare domain names is to compare these arrays of parts with a lexicographical compare function.

sub lexicographical_compare(\@\@) {
    my ($a, $b) = @_;
    my $i = 0;
    while ($i < @$a && $i < @$b) {
        my $c = $a->[$i] cmp $b->[$i];
        return $c if $c; # $c != 0;
        $i++;
    }
    return @$a <=> @$b;
}

sub compare_dns_record {
    my ($a, $b) = @_;

    my $c = lexicographical_compare($a->{domain_parts}, $b->{domain_parts});
    return $c if $c;

    $c = $a->{type} cmp $b->{type};
    return $c if $c;

    $c = $a->{prio} <=> $b->{prio};
    return $c;
}

The second insight here is that the record itself can also be compared lexicographically. But this is harder to write in Perl. In Haskell, for example, this is the default for tuples.

August 18, 2014

Rotate one to the left

Last week I wrote a small function in Perl that rotates and array to left. The function takes the item from the front and appends it to the back.

# precondition: @list >= 1;
sub rotate_left_by_one {
    my $ar = shift;
    my $item = shift @$ar;
    push @$ar, $item;
    return;
}

After calling this function as many times as the length of the array, it leaves the items in the same order they started in. Every call to this function puts a different item in the first location. I use it to compare this item to the rest of the items.

You would use it like this:

while (1 .. @list) {
    # your code
    rotate_left_by_one(\@list);
}

This function works great when the code doesn't depend on the order of the items in @list. The items in @list will be in a different order than what it started with for most of the iterations of the loop. This means that the code in the loop body needs to have the commutative property.

Welcome

My name is Peter Stuifzand. You're reading my personal website.

Profiles

Peter Stuifzand
peter@peterstuifzand.nl
Zwolle, The Netherlands