December 20, 2019

Year in review: 2019

I was always interested in the Year in Review posts from people. It looks like it is something that you can't create after the fact. You need to keep some notes, and perhaps log data about the things you did. Some people keep count of the number of emails sent and received or kilometers traveled. After the fact, these kinds of details are rather hard to come by.

To make this a great post, I will have to dig into the notes I made, the data that I collected, and the posts I wrote on the other blog and combine these into witty graphs and paragraphs of enjoyable words.

You are reading the first Year in Review post I ever wrote on this blog.

Lead developer

This year I got a promotion to Lead Developer. I'm very happy with the opportunity I got this year and let's what we get to do next year.

Route 66

This year my wife and I traveled to the United States. We drove 2869 miles from Chicago to Los Angeles along the historic Route 66. We saw some things of the United States. Not much when you think about it. Just nine states.

Quantified Self

When we returned from the United States, I found that the Fitbit had charged while we were gone. This Fitbit had stopped charging some time ago. I had found out that the problem was with the charger and not with the watch itself. I was even more surprised that the problem was not with the small connector for the Fitbit, but with the USB-A connector on the other end.

This started my exploration of possible Quantified Self tracking that I could do. I started using Exist, with Fitbit, Last.fm and RescueTime.

A few months ago Exist added the ability to track coffees, alcoholic_drinks and keystrokes. The problem is that these attributes there are no programs that track these values. To remedy this I created a new program that sends these values to Exist. The first version just sends these values. A later version will send these values automatically every day.

I already created a small widget on my phone home screen with Tasker and LWGT, that counts the cups of coffee that I drank in a day. At night this will get send to Exist. This doesn't work from beginning to end, but as they say manual until it hurts.

This whole Quantified Self thing will help me make next years review a lot more numerical.

Indieweb

This year I haven't developed much for the Indieweb. I made some improvements to the blog. I added destination support. At some point it was possible to send something to my blog and then let Node-Red handle the post. It could then send to post to the blog.

I added support for Join. It's a way to send stuff to your phone, or other supported devices. It's written by the developer of Tasker.

I did experiment a bit with Server-Sent Events in Ekster. The reader part now supports small posts in the right sidebar. These get sent by the Micropub server when new posts arrive in realtime. It's even faster this way, compared to updating every minutes. But for the most part I only did cleanup of the code.

Home Automation

This year I started automating some parts of my home. This started with the Smart Meter that was installed in the beginning of the year. Around the same time I found the IKEA smart lights. Which led me to Home Assistant and Node-Red.

The automation I'm the most proud of is the I get a notification when the washing machine is done cleaning. This is build with a Wifi socket with energy measurement built-in, ESPHome and Home Assistant.

The sockets themselves are flashed with a firmware created with ESPHome. These sockets send the power measurements with Wifi to Home Assistant. When the measurements reach a certain level of power it changes a state in Home Assitant from Idle to Running. When the power level drops again, the state is changed from Running to Clean and a notification is sent to my phone.

Conclusion

It always hard to know what the new year will bring. We do already a expect a little miracle in april. So I guess that will take most of my time next year.

December 31, 2018

Ekster: a microsub server

Ekster is a Microsub server. As far as I know Ekster was the second implementation of a Microsub server.

Microsub is a protocol for feed reader applications. Ekster fetches web pages and feeds and gathers the entries into channels. A channel contains multiple feeds, these feeds are combined into a timeline.

Ekster can find entries from pages that are marked up with microformats. It uses the go-microformats library to get the MF2 structure and converts these microformat items to JF2. JF2 is a simplified version of MF2, which is easier to display in a Microsub reader.

Ekster also reads RSS and Atom feeds. It does discovery on websites and tries to find the alternate urls. RSS and Atom are simpler feed formats, that don't contain the likes and replies of websites.

To work in the realtime web, Ekster also supports WebSub. WebSub is a protocol that notifies subscribers of feeds. Ekster subscribes to these feeds when it finds the WebSub hub.

Another way to get entries into Ekster is by using Micropub. Micropub is a protocol for clients applications to create new posts and weblogs. It can also be used in Ekster to send notifications for example.

The storage of the microsub server is built on Redis. Redis is a key-value data structure server. On the wiki there is a description how Ekster keeps the state of feeds and items in Redis.

August 15, 2018

A tour of my Indieweb projects

Earlier this year I started to rebuild this website to work better with the Indieweb. At the same I build a new website P83 with Indieweb support from the start.

This website is generated from static files with a system that I created the first time when I started writing in the website. Not much has changed, except that I some point a few years ago I rewrote the whole system in Perl.

The only actual changes to this website, was adding some support for smaller posts, and having a better view of multiple posts on the same day. I also added microformats to the posts.

I still think there is a lot to like in a static website. It's even possible to combine the static nature of pre-generated pages with the dynamic properties of the Indieweb like webmentions and comments. Quite a few people support that usecase. Some even use Micropub to write new posts for their blog.

This blog doesn't support any of that, except webmentions.

P83 is the new name of the other blog. It started out from an old code base, that I was working on a few years ago. The software running it is called publog (from something like publishing log). It's quite a bit more dynamic software. It supports webmention and microsub. It supports projects, different channels, syndication to other websites (like Indieweb.xyz, Twitter and Github), Bridgy and all kinds of small experiments.

In the last few months, I have build out the functionality for connecting with the Indieweb. I improved the presentation of the posts, the microformats and the parsing of microformats. The microformats organization on Github gathered many parsers together and have a repository with tests. This helps a lot when you want to work with microformats in your own projects. I have written my own version of a microformats parser for Perl. But I'm sure it's not very complete compared to the parsers for the other languages. I should add the test suite to this parser and see how far it gets. Perhaps I should even rewrite it, as it could contain a lot of unused code. The microformats parser is the weak link in the website. I should try to improve it.

When I found that there was no Android client for Micropub, I dusted of my Android developer account and started working on the client. Android Studio helps a lot. Today the client works pretty well, and I hope that it can be moved to a production release in the coming weeks. I only still need some confirmation from people using it with some of the more popular systems (like Wordpress).

I'll mention the Microsub server (Ekster) in my next post.

January 31, 2018

Writing a WebSub hub

This week I've been writing a WebSub hub. It's server program that receives subscription and publish requests and sends notications to the subscribers when a website changes. The website itself sends a request when the website changes. This way the subscribers know when to update their feeds.

Implementing the server is quite simple, especially with the help of a website that checks your implementation, websub.rocks.

WebSub Rocks! is a validator to help you test your WebSub implementation. Several kinds of tests are available on the site.

By reading the spec and following the steps from websub.rocks, it was quite simple. The problem is that not all websites implement this exact specification. FeedBin, for example, implements a feature from an older version of the spec and won't validate the subscription, when it otherwise would be good. A small change makes me support it, but I also send a commit to the FeedBin software on Github. That removes that use of the feature. As expected, this change leads to more changes, so we will see how this pans out.

Just fixed a small problem with loading and saving subscriptions in the hub. It again shows that it's a good idea to check all error's from function calls in Go.

#

Sending the websub notification seemed to have worked. The previous was added to FeedBin.

#

Trying WebSub with this weblog. I have a subscription from Feedbin. Now I will try publishing from the feed.

#

January 16, 2018

January 14, 2018

May 27, 2017

Collection and select

Sometimes you need to get a lot of information from the database and you need it fast. The way to get this information normally would be to use the Magento collection for the model. It would look like this:

$collection = Mage::getModel('customer/customer')->getCollection()
    ->addNameToSelect()
    ->joinAttribute('billing_city', 'customer/city', 'default_billing', null, 'left');

foreach ($collection as $customer) {
    // Use $customer
}

The problem with this way of getting the information is that it loads all customer information into memory, which can take a long time when you have many customers.

The solution to this performance problem is that you read the customers one at a time by using the select that was created.

$select = $collection->getSelect();
$stmt = $select->query();

while ($row = $stmt->fetch()) {
    // Use $row
}

This is often many times faster and won't fill your memory.

March 31, 2016

XML-lister

XML lister is a small program that lists the elements, values and attributes of XML files.

<?xml version="1.0"?>
<layout>
    <default>
        <reference name="root">
            <reference name="content">
                <block name="product.info">
                    <action method="test">
                        <param>test</param>
                        <param2>test2</param2>
                    </action>
                </block>
            </reference>
        </reference>
    </default>
</layout>

XML lister will print the following list:

layout
layout/default
layout/default/reference
layout/default/reference@name   root
layout/default/reference/reference
layout/default/reference/reference@name content
layout/default/reference/reference/block
layout/default/reference/reference/block@name   product.info
layout/default/reference/reference/block/action
layout/default/reference/reference/block/action@method  test
layout/default/reference/reference/block/action/param   test
layout/default/reference/reference/block/action
layout/default/reference/reference/block/action/param2  test2
layout/default/reference/reference/block/action
layout/default/reference/reference/block
layout/default/reference/reference
layout/default/reference
layout/default
layout

A few days ago I wanted to find out if there are places in Magento layout XML files where reference elements are used inside other reference elements. With XML lister it becomes really easy to find places like this.

find app/design -name '*.xml' | xargs xml-lister --filename | grep reference/reference

With the list it becomes really easy to find the files where a certain construct is used.

March 16, 2016

Rewrite XML with xmlstarlet

XML Starlet is a program that makes transforming XML documents really simple.

The command that you need to transform an XML document is the following.

xmlstarlet tr [<options>] <xsl-file> {-p|-s <name>=<value>} [ <xml-file-or-uri> ... ]

The XSL file that transforms a document to itself is this. It copies all elements, attributes and texts. We can extend this file to remove, add, or transform elements.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

To remove an element, we match an element and don't call apply-elements inside it. This example will remove the remove me element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <!-- Remove the element -->
    <xsl:template match="remove-me">
    </xsl:template>
</xsl:stylesheet>

To add an element, we just add the part we would like to add. If it needs to be in a certain place, we can add an extra xsl:template rule. The this-will-get-added element will be inserted after the element element.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>

        <this-will-get-added></this-will-get-added>
    </xsl:template>
</xsl:stylesheet>

To transform an element we remove xsl:copy and replace it with the elements we need. If we keep the xsl:apply-elements tag in the xsl:template tag, we copy everything inside the element in it's place.

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <!-- IdentityTransform -->
    <xsl:template match="/ | @* | node()">
        <xsl:copy>
            <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
    </xsl:template>

    <xsl:template match="element">
        <my-new-element>
            <xsl:apply-templates select="@* | node()" />
        </my-new-element>
    </xsl:template>
</xsl:stylesheet>

XSL used in this way becomes quite simple to use. With these three patterns we can extend the identity transformation and get quite a bit of power when we transform XML documents.

November 6, 2014

Welcome

My name is Peter Stuifzand. You're reading my personal website.

Profiles

Peter Stuifzand
peter@peterstuifzand.nl
Zwolle, The Netherlands