Schwartzian transform in shell and vim

A few days ago I had to create list of email addresses sorted on the domainname. All addresses with the same name should clump together. Being a Perl programmer, I quickly wanted to write a Perl script to sort this list of data. Then I tought, why not use Vim to look at the intermediate steps and do it with filtering. The following is the result.

:%!perl -pe 'm/\@([-\w]+)\./; $_="$1\#$_";'
:%!sort
:%!cut -d\# -f2

The first line parsed the domain from the data and pastes it in front of the orignal, seperating it with a '#'. The second line obviously sorts the new list. The final line removes the domain that I put in front. This leaves a list of the originals.

:%!perl -pe 'm/\@([-\w]+)\./; $_="$1\#$_";' | sort | cut -d\# -f2

This will do it in one line.

Whitespace at end of line

On Coding Horror Jeff wrote about whitespace at the end of lines of code. I hate that as much as the next guy. It seems irrational, but I think it's not. It has no function and only creates problems on times when you don't expect them, especially with source code control tools.

So I wrote a small piece of Vim script code to highlight the spaces at the end of the line in bright red.

highlight OverLength ctermbg=darkred ctermfg=white guibg=#FFD9D9

function! EndOfLineWhitespace()
    3match OverLength /\s\+$/
endfunction

call EndOfLineWhitespace()

This will highlight all the whitespace at the end of a line. If you don't like to use this for all files, you can use the usual ways to do this in Vim.

UPDATE: Since Vim 7.2 it is possible to use another function instead of the match functions. It's called matchadd. The solution I gave above can be written as follows:

highlight OverLength ctermbg=darkred ctermfg=white guibg=#FFD9D9
call matchadd('OverLength', '\s\+$', -1)

See the help documentation for matchadd to see how this works.

Via VimTip810

Autocomplete contexts and people with GTD in Vimoutliner

I wrote an article about GTD and Vimoutliner. This post adds a few simple things you can do to simplify your edits to the todo.otl file.

In Vim it is easy to autocomplete a word or line. You can complete a word by typing the first few characters of the word you want to complete. Then type CTRL-N to complete the word. You can try other words by typing CTRL-N more often. This will try to complete your word with a word that can be found in the same file.

Sometimes you need to complete a full line. This can be a person from the PEOPLE section or a context from the CONTEXT section. To complete a full line, type the first few characters of the line you want to complete and then type CTRL-X CTRL-L and then you can try more lines by typing CTRL-N.

Vim function to replace many spaces with one

At last I have found the function that will replace many spaces with one space. I first wrote a blog post about this in januari of 2007. Then I wrote a blog post about how to enhance this function to move the cursor one position after the inserted space. With an update a day after that, which said that it didn't work in some instances.

Today I'm proud to announce the final and working function that works as it should.

I created this function with help from Al on StackOverflow.

Insert the current date and keep it up to date

You would like to add an automatically updated timestamp in your text files. You can do that by adding 6 lines of code to your .vimrc.

The following command will call the function LastMod() whenever a buffer of file is written.

autocmd BufWritePre,FileWritePre *.html   call LastMod()

The following function will find all files containing 'Last modified:' and replace them with the current date and time.

fun LastMod()
  exe "%g/Last modified: /s/Last modified: .*/Last modified: " . strftime("%Y-%m-%d %T")
endfun

The biggest problem with this function is that using the :substitute function will move the cursor to the beginning of the line. This will even happen when nothing gets replaced. This is extremely annoying.

This can be fixed by remembering the current column and setting the cursor back to that after the command. Remembering the cursor position is easy.

let save_cursor = getpos(".")

And restoring it is not much harder.

call setpos('.', save_cursor)

I added . "/e" to the and of the regex to catch errors. Without the e flag an error will happen when there is no line matching the regex.

With all these changes the code now looks like this:

fun LastMod()
    let save_cursor = getpos(".")
    exe "%s/Last modified: .*/Last modified: " . strftime("%Y-%b-%d %X") . "/e"
    call setpos('.', save_cursor)
endfun

How to use Vimoutliner checkboxes

Vim outliner is an outliner tool. As the name implies you can write outlines with it, but with this particular one you can also create todo lists. A todo list can be created without much effort.

First open your todo file, todo.otl for example. By using the otl extension, Vim outliner will be loaded automatically.

Next, type a task you have to do in the future. This could something like write blog post about Vim outliner. Remember, because this is Vim, you have to type i to start insert mode.

write blog post about Vim outliner

Then type ,,cb this will insert a checkbox in front the current line. Which will make the line look like this:

[_] write blog post about Vim outliner

So now you start writing the blog post. When you're done after a few minutes (about 30min). You can check of the item on the list. Do this by typing ,,cx. Your file will look like this now:

[X] write blog post about Vim outliner

And this concludes this first How To blog post about Vim outliner.

Summary

• Article about How to use GTD with Vim outliner

Write Twitter drafts in Vim

Not everyone writes drafts for his Twitter posts. It seems silly even to do this.

But for the people who don't think writing drafts is silly (and use Vim), I wrote a few lines of Vim script that will help them keep their lines just short enough.

Put these lines in your .vimrc file and start writing. You should change the filename to the name of the file that you write your drafts in. As usual, this code comes without any warranty.

" Call the function that will highlight the lines that are too long.
au BufNewFile,BufRead /home/peter/doc/twitter.txt call TwitterLineLengths()

highlight OverLength ctermbg=darkred ctermfg=white guibg=#FFD9D9

function! TwitterLineLengths()
    3match OverLength /\%141v.\+/
endfunction

Code based on code from Stackoverflow.

Using ack to check coding conventions

Do you know Ack? It's a grep-like program. That uses perl regular expressions instead of the normal Posix ones. You can find it on the CPAN.

The following Ack call will check your perl code for problem with a space missing behind a controlstatement keyword.

ack --perl '(?\@<!\w)(if|while|elsif|return)('

If you use Vim you can also use the following piece of vimscript in your .vimrc file:

highlight WHITE_ON_RED ctermfg=white ctermbg=red

function! BadNonInvocations ()
    2match WHITE_ON_RED /\w\@<!(if\|elsif\|while\|return\|for)(/
endfunction
call BadNonInvocations()

Human readable date in vim outliner calender

I created a small Perl program to convert relative dates to absolute dates in the format that I use for my calendar. My current calendar file looks like this.

2008
    08
        2008-08-12
        2008-08-13
    09
        2008-09-10
        ...

If I want to add a date and I don't know the actual numbers, I can use the following program to convert the date. It will also respect the whitespace in front of the text.

#!/usr/bin/perl -w
use v5.10;

use strict;
use warnings;

use Date::Manip;

Date_Init('Language=Dutch');

my $inp = <>;

if (my ($ws, $date) = $inp =~ m/^(\s*)(.+)$/) {
    say $ws . UnixDate($date, "%Y-%m-%d");
}
else {
    print $inp;
}

I use Date::Manip for parsing the date. It works with human language style dates like thursday. I added Date_Init so it will parse Dutch days like donderdag. Also I use Perl 5.10, because I can. It has some nice features.

To use it put this script in the path. I called it refdate.pl. To use it type a date and type !!refdate.pl<Enter>. In Vim this will call the program on the current line. The date will be formatted in the YYYY-MM-DD format.

Vim: Just once space revisited

At the beginning of last year I wrote a weblog entry about how to write a function that removes all space but one in Vim. It didn't work like I wanted, but now it does.

function JustOneSpace()
    " replace all whitespace around the cursor with a space
    s/\s*\%#\s*/ /e
    " search backwards for a space
    call search(' ', 'be')
    " move to the first character after the space
    normal l
endfunction

nmap <space> :call JustOneSpace()<cr>

I added the call to the search() function to move to the space that was substituted. All this time I wanted to fix this function by using regexes or special vim variables. I couldn't find these. This is quite obvious, but it didn't come to mind at first.

Update

I looks like this function doesn't work when it's used on a line with no space. It will insert a space on the spot it is supposed to, but then moves the cursor to the first space it finds when searching backwards.

GTD and Vim outliner

I wrote an article about Getting Things Done using Vim outliner, an outliner for Vim.

Vim page

Today I created a page about vim on this website. Vim is a useful editor, but sometimes it would be nice to have more articles describing the cool features that it has. The documentation describes every features in vim, but sometimes it's hard to know where to start looking. I'm building this page with links to articles that describe features or general vim goodies. So enjoy!

Just one space

I always wondered why vim didn't have the just-one-space function. It's one of the most useful functions that is available during coding.

So today I tried to write one myself, the newsgroups and searchengines didn't come up with something good. First of course it is a good idea to understand the effect of this operation. The just-one-space function removes all whitespace around the cursor and replaces it with one space character.

How can this be done in vim. The replacement part isn't that hard. It can be done with one :s///, something like: :s/\s+/ /. This will of course remove whitespace from the wrong place, at the start of the line.

The missing piece in this substitution is of course the place of the cursor. So I started looking for it. Most help text about the substitution operator can be found on :s helppage (:help :s). But there is no mentioning of the first part of the substitution. The description of the vim regexp engine can be found at the regexp help page (:help regexp). This contains a description of all the possible meta characters that can be matched. So I the meta character that will match at the cursor position (\%#). This solved the second part of the problem.

:s/\s\+\%#\s\+/ /

This will remove the whitespace surrounding the cursor. The only only problem that still isn't solved is the problem that the cursors moves to the beginning of the line. It would be better if it stays at the same or a the end of line if there is not enough room to go to the last position. Probably the best place to put the cursor is on the first character after the inserted space. I may have to look into that a little more.

The function is finished by adding the e flag at the end of the substitution. With the e flags the :s will be silent when the first part of the substitution doesn't match.

I bound the function to the spacebar in normal mode with the following command:

:nmap <space> :s/\s\+\%#\s\+/ /e

I will try this for some time and look how it works, could be good or better.

Using ruby to create vim extensions

Today a simple vim plugin is featured on the vim homepage: toggle_word.vim. I think the plugin in itself is not that useful, but it seems to be a good starting point for ruby programmers who want to create their own vim plugin and don't know where to start.

Subversion diff inside Vim

I like using vim. It is simple want it to, but has a lot of possibities when you need them. Like Perl in vim simple things are easy and hard things are possible. Like the following piece of code.

I wanted to see a diff of my current version and the version in my svn repository.

function Svndiff()
    let file = system('tempfile')
    execute 'silent !svn cat ' expand("%") ' > ' file
    execute 'silent vertical diffsplit ' file
endfunction

map <F4> :call Svndiff()<CR>

Of course the <F4> binding is like icing on the already delicious cake. The only thing that I'm not happy about is the problem with the tempfile, that will stay open after you've opened the split screen.

External programs in Vim

When you're typing some text in Vim, you will sometimes need a piece of text that is easy to generate on the command line or with a simple perl script. Maybe you want to insert the current date, a simple calendar or a list of day names.

Don't you worry there's a simple vim command that will help you with this. Here are some examples. From simple to complex.

Insert the current date

Type:

date

on a line by itself. Then move the cursor to that line and type !!sh<cr>. This will filter date through the shell (which is equivalent to :r!date. The current date will be replaced into the buffer. If the output isn't correct or different from what you expected, then you can always use u to undo the change.

Insert a simple calendar

Type:

cal

And type the same command as before !!sh<cr>. The nice thing about using the !! command this way, is that you can create a command line in Vim.

SQL queries collecting output

Sometimes it's easier to use the command line when conversing with you're MySQL database instead of using phpmyadmin. This will happen when you need to create queries based on output of other queries. An example:

SELECT `id` 
FROM `entry` 
WHERE `post_date` 
    BETWEEN '2006-10-25' AND '2005-11-05'

This query of a fictitious database will get the id's of the entries in a datarange. This query can be piped into mysql by using the following command:

mysql -u dbuser -pdbpass databasename

As always prefix the command with !!mysql .... This will send the query to the database. The output of the query will be replaced into the current buffer one id on each line.

After this you can use the output together with vim to creates new queries. Assume we got the following output:

10
11
12
13

These are the id's we got. We can translate the id into new queries by using some Vim commands.

UPDATE `entry` SET `visible` = 1 WHERE `id` = 10
UPDATE `entry` SET `visible` = 1 WHERE `id` = 11
UPDATE `entry` SET `visible` = 1 WHERE `id` = 12
UPDATE `entry` SET `visible` = 1 WHERE `id` = 13

Now selected these queries by using V visual line mode. When you press ! now, vim will ask for a program to start, just like in the other examples. When you use mysql, vim will send each query to the database.

A script

To print all the letters of the alphabet you can use the keyboard and type all the letters one after another. If you are a bit more lazy and want to learn some nice vim command and perl code, you can use the next command.

print 'a' .. 'z';

This is a line of perl code that will print all the letters of the alphabet when executed. This line of code can be executed by typing !!perl<cr> on the line of code. The code will expand into abcdefghijklmnopqrstuvwxyz.