Peter Stuifzand weblog

Create your own YouTube playlists

Sun, 09 Jun 2013 14:40:00 +0100

A few days ago Mindcrack Ultra Hardcore season 11 started. In Ultra Hardcore the play by special rules, that are different from vanilla Minecraft. The biggest difference is that players don't regenerate health automatically. They need to eat golden apples to increaes their health. These apples are harder to make, because you need golden ingots instead of golden nuggets.

Each season players record videos of their perspective of the match and post these videos to their YouTube channel. This season these videos are posted every other day around 22:00 UTC. This is mostly to give viewers the opportunity to watch multiple perspectives. The total running time for the first episode was about 7 hours. That's a lot of video to watch.

At the moment the best way to watch these videos is to subscribe to a channel and follow along whenever a video is posted. The problem (for me) is that these players post many other videos of games that I don't like to watch. There also isn't a place where these videos are linked in one place. To remedy this problem I wrote a few programs that create the playlists automatically.

Let's start with the output. The program created a few playlists on YouTube from the videos that I gathered from the channel. I created two views into the videos.

The first view is the list of videos for each player. Each player has a playlist with all the videos in episode order. For example the playlist for Zisteau. There is also a playlist for the other players.

The second view is a list of all videos in each episode. This means that the videos happened around the same 20 minutes of play time. For example all videos from episode 1. The other episodes are also available.

To create these playlists I wrote three Perl scripts. The first one gets the latest episodes for each players. This way I get a list of the last videos of the channel. This list contains all videos, also videos that I don't want to add to the playlist.

In the next step (with the other two scripts) I parse the titles of the videos and find the videos that match the regular expressions for UHC or Ultra Hardcore and Season 11. Here I also try to find the episode number of the video.

Next I create a playlist or find the playlist_id of the episode (or player) and add all the videos that haven't been added yet.

Now we're done. I used the WebService::GData::Youtube module for calling the YouTube API, as it removes most of the dirty work.

Mon, 03 Jun 2013 11:15:00 +0100

Barnaby Walters wrote:

build stuff which kills your own FB usage before trying to kill facebook.

That seems to be a good way to build software that fills a real need.

Load Firefox search in background tab

Mon, 06 May 2013 13:53:00 +0100

Today I noticed something that bothered me. When I search in Firefox through the right-click context menu the search results open in a new tab and Firefox switches automatically to the new page. I remembered there is option that makes Firefox open new links in the background and I thought that maybe this option was disabled. The option was enabled as it should be.

I opened a few links and it behaved as expected. The search results however still opened in a new tab in the foreground. This was unexpected. For me it seems that links and searches should open in a new tab in the background.

So I started a search for this behaviour. I found a bug from the Mozilla Bugzilla database Bug 695482 that seemed to call for the reverse of what I expected. There seem to be some strong arguments for this behaviour. There are even some that tests the suggest that the majority of people click on the new tab within 5 seconds of starting the search. I don't think this a what I do. I am lucky that some people some disagree with this behaviour. These people created Bug 727131 that calls for the about:config option browser.search.context.loadInBackground that makes searches open in a background tab.

I enabled the option now and will try it some.

A universal subscribe button

Mon, 22 Apr 2013 17:24:00 +0100

SubToMe is a universal subscribe button that you can add to your weblog. You just add small bit of HTML that shows a subscribe button. The software is written in Javascript and runs in your browser. It doesn't need a central server. Take a look and add it to your weblog.

Changes to the heredoc parser example

Mon, 22 Apr 2013 16:31:00 +0100

Jeffrey Kegler, the author of Marpa, was kind enough to suggest a few changes to the parser from the previous blog post.

Two of the three regular expressions were replaced with calls to the Marpa recognizer. The first one was replaced with a call to the pause_span method, which returns the start and length of the lexeme that caused the pause. Passing these two values to the literal method which returns the substring from the input string. This removes the need to mess around with the pos() from the input string.

The second regular expression that was replaced is the one that matches the newline. There is no need to match with this regex, because we already know at which lexeme the parser was paused with a call to the pause_lexeme method. This method will return 'newline' when that parser found a newline. We only need to resume the parser in the right position.

These two changes makes the code simpler, because we don't have to change the pos() for the input string every time we want to match with the regexes.

Parse a heredoc with Marpa

Fri, 19 Apr 2013 18:13:00 +0100

I already wrote that I'm quite interested in the things Marpa can do for writing parsers. The latest release of Marpa includes an addition that makes parsing strange languages even more possible. The new functionality allows you to control the parser from your own code while it doesn't stop you from using the internal scanner.

To try out this new functionality in Marpa I tried to write a parser for a small language with just one function and heredocs. I choose this language, because it can't be parsed with just the internal scanner. The scanner needs to find the end marker with the same name as the begin marker of the heredoc. The parser doesn't know about the name of the marker, just that it looks like a marker.

The code is in a repository on github. I will talk about the lines containing the expression, heredoc and lexemes. The comments in the file explain what happens. The tests in the t directory show a few ways in which we can combine the heredocs in statements and expressions.

The moment the parser finds a beginning of a heredoc it pauses and passes control to the parser. Where the parser is paused can be controlled by the :lexeme rule and the pause adverb.

The external scanner first finds the end of the line where the beginning of the heredoc was found. This is the position where the text of the heredoc will begin. Then it will find the name of the marker. If it found the name it will try to find the marker at the beginning of one of the following lines. The marker and the literal will be send to the parser with the two calls to the lexeme_read method. Then it moves the $last_heredoc_end to the end of the line of text and the position of the input string to the position after the beginning of the marker and passes control back to the parser with a call to the resume method.

The program will do this until the parser pauses again at the end of the line. Then it moves the position of the parser to the end of the possibly multiple heredocs. The repeats until the end of the input is reached.

I was impressed with how short the code is and how easy it was to parse heredocs and pass control back and forth between the internal and external scanner. This allows for many new ways to parse text for all kinds of languages.

Apps that share

Mon, 08 Apr 2013 12:27:00 +0100

On my phone I can share stuff between apps. It's like pipes. I write something in one app and send it to another app that posts it to my weblog. The apps in question are Draft and WordPress. It shows how easy it is to send stuff between apps. The apps don't even have to know the other app exist. Why is this easier on my phone than on my PC?

Grok

Mon, 11 Mar 2013 13:02:00 +0100

In this video Steve Yegge talks about a project he's working on called Grok. I seems ambitious and useful.

I worked on a few parsers in the last few months and it seems that most of the work still has to be done even if you've got a parse tree of a language.

Yegge explains that they don't write the parsers themselves, but use the work compiler builders have done. This is a good idea, because writing a parser isn't easy, even if you have a really good tools to build a parser with.

After you have a parse tree you still need to generate code or evaluate the tree to do the actual work.

It's the same problem you have when you're parsing HTML: why would you write that code (badly) when someone else already did it for you?

You have a piece of HTML you want to use some part of. You start with regexes and people start yelling at you for using the wrong tool for the job. So you move on to a better tool.

That's all well and good when you want to parse HTML (I would use Marpa::R2::HTML), but what if you would like to parse C, Javascript or my language of choice, Perl. It would be the same (but better) madness to write a grammar and parser for those languages.

If I need to parse a few lines of C function declarations, I would start with a small regex, fail and move on to a combination of regexes, an ad-hoc state machine and an actual parser. When you have an actual parser in your toolbox you can take on more problems. It's not always the right tool. But I will try it first the next time I need to mangle and transmute text, even before I start with the regexes.

There is a need for parse Javascript and C and general tools to work with them. Parse a Javascript file and list the function names.

use Parse::Javascript;
my $file = Parse::Javascript->parse($js_file);
say $_->name for $file->functions;

Or the same for C.

use Parse::C;
my $file = Parse::C->parse($c_file);
say $_->name for $file->functions;

Or change all function calls the match one thing and change them to another thing.

use Refactor 'refactor';
my $refactored_file = refactor($file, 'compare(@a, @b) == 0', 'equal(@a, @b)');

Or how about this create C code from a Javascript AST?

my $c_file = Serialize::C->serialize($js_ast);

Or as Steve Yegge says creating a parser (with semantics), code generator, tools and all that is a life's work. Maybe we should try to make that easier?

DIY refactoring

Tue, 29 Jan 2013 21:52:00 +0100

Yesterday I released a piece of code the includes a parser, a rewriter and serializer for a small expression language.

The code is an example how you can parse source code and build a tree, rewrite the tree using two expressions and serialize the tree back to a piece of 'code'.

The idea that I used here was to improve the language by adding a way to specify structural variables. The variables start with a colon :, but are otherwise like variables.

The actual source code shouldn't contain these structural variables. You should only use them in the transformation specification. I wrote a function replace(source, from, to) that find and replaces the structure from in source to the structure to. Let's start with an example.

my $source = <<'SOURCE';
print(power(10, 2));
print(power(10, 3))
SOURCE

print replace($source, 'power(:b, 2)', 'twice(:b)') . "\n";

In this example we replace the power function with the twice function, but only if the exponent equals 2. The power call with exponent 3 is not replaced.

The function replace parses all three arguments $source, $from and $to. It searches for the $from structure using the _expr_match function, which returns the matching structural variables in the third parameter.

To play with this you should git clone the gist and run the test.pl program. Changing the $source, $from and $to allows you to try different things.

Generate a list of the same thing

Sun, 20 Jan 2013 12:12:00 +0100

Last year I wrote about how we can build an extended syntax for Marpa on top of the basic interface.

For the counted rule lhs ::= rhs{5} where you want to match the right hand side multiple times, we can just generate the rhs item 5 times.

This isn't the right thing. Jeffrey pointed out that you can build these {min,max} rules up out of blocks.

Then I remembered a piece of code from Elements of Programming which explained that it's possible to generate the Fibonacci series in O(log n) time. The same should be possible for the counted rules.

So I wrote this code that generated these rules, not in O(log n) time, but with more rules and less items in the rules. This scales better for big n.

$ ./counted_rule.pl lhs rhs 5 action
_rhs_block_5  ::= _rhs_block_4 _rhs_block_1    action => do_flatten_list
_rhs_block_4  ::= _rhs_block_2 _rhs_block_2    action => do_flatten_list
_rhs_block_2  ::= rhs rhs                      action => do_flatten_list
_rhs_block_1  ::= rhs                          action => do_flatten_list
lhs           ::= _rhs_block_5                 action => action

$ ./counted_rule.pl lhs rhs 50 action
_rhs_block_50 ::= _rhs_block_48 _rhs_block_2   action => do_flatten_list
_rhs_block_48 ::= _rhs_block_32 _rhs_block_16  action => do_flatten_list
_rhs_block_32 ::= _rhs_block_16 _rhs_block_16  action => do_flatten_list
_rhs_block_16 ::= _rhs_block_8 _rhs_block_8    action => do_flatten_list
_rhs_block_8  ::= _rhs_block_4 _rhs_block_4    action => do_flatten_list
_rhs_block_4  ::= _rhs_block_2 _rhs_block_2    action => do_flatten_list
_rhs_block_2  ::= rhs rhs                      action => do_flatten_list
lhs           ::= _rhs_block_50                action => action

The number of rules scales logarithmically.

The code for counted-rule.pl is on GitHub.