Evernote Extraction Part II - Vimwiki Inception

Part 2 of my migration from Evernote to Vimwiki.

Installing vimwiki

I don't use a package or plugin manager for vim and I don't want to install one now. At the recommendation of the vimwiki docs, I unzipped vimwiki into .vim/pack/ and stared vim. Nothing happened. It turns out my version of vim, 7.4.576, is just below what's needed for this installation method to work. According to the docs, I should be able to install vimwiki by unzipping the archive into the .vim/bundle/ folder. That didn't do the trick, either.

After reading How to Install Vim Plugins Without a Plugin Manager, I add set runtimepath^=~/.vim/bundle/vimwiki to my vimrc. I start vim, type <leader>ww, and it asks me if I want to create my vimwiki directory. Success!

The docs say that I should “run :helptags” after unzipping the archive under .vim/bundle/ and then type :help vimwiki to verify that vimwiki is installed. There's a problem with this because :helptags requires an argument. I tried giving helptags a number of arguments, including “ALL” and “.vim” but :help vimwiki never worked.

Extracting Notes

My main notebook has about 400 notes. I would really like to retain the creation date and tags for each of my notes. This metadata helps me contextualize and evaluate the content. I looked at the installed source for ever2simple and I don't think I'll be hacking on that. Back to jq!

Starting with

jq '.[] | .tags,.createdate,.content' dbucklin.json | less -S

The -S option on less chops long lines, making it easier to grasp the linewise format of the output. The tags come back in an array. It's often empty. I need to join the array into a string.

jq '.[] | (.tags|join(", ")),.createdate,.content' dbucklin.json | less -S

That looks better. I'm going to pass this output to awk so that it can split it into separate files, one for each note. I need a to insert a symbol that will signify the end of a note.

jq '.[] | .createdate,(.tags|join(", ")),.content,"%%"' dbucklin.json | less

As before, the content contains lots of encoded newlines and every line has quotes around it. Also, quotes within values are escaped and there are random occurrences of a double-backslash. I want to clean all of that out. I'll use sed to do most of the cleanup.

sed -e 's/^"//;s/"$//s/\\"/"/g;s/\\\\//g;s/\\n/\n/g'

I want to split each note into its own file. I'll pipe the output from jq through sed and into an awk script to split the output.

#!/usr/bin/awk -f
# makenotes
BEGIN {fcount=1}
$0 == "%%" {fcount++}
$0 != "%%" {
  print $0 >> "notes/" fcount ".txt";
}

Looking good! All together, now.

jq '.[] | .createdate,(.tags|join(", ")),.content,"%%"' dbucklin.json \
| sed -e 's/^"//;s/"$//;s/\\"/"/g;s/\\\\//g;s/\\n/\n/g' \
| awk -f makenotes

This puts each of my notes, with creation dates and tags, into its own file in the notes/ subfolder off my working directory. Done! Right?

Migrating Notes to vimwiki

I extracted my notes and attachments from the XML files I exported from Evernote. I've spent some time looking over everything, and I've realized that the overwhelming majority of the content just isn't relevant to my life today. There are recipes, grocery lists, packing lists, and call notes – from seven years ago. At one point, I was forwarding email receipts and shipping confirmations to Evernote. Why? Much of this content doesn't have a future. I'm still keeping a zipped archive of my enex files because I'm a hoarder.

The real challenge is that I used Evernote as an unstructured dumping ground for my notes. The only organizational methods I used in Evernote were the separate notebooks and, in my main notebook, I used tags to categorize notes – most of the time. I relied on Evernote's search to find information that was relevant in the moment. This allowed me to side-step the growing volume of irrelevant information I was gathering. A wiki, on the other hand, demands structure.

I'll have to review my notes and pick out themes that will help me translate them into vimwiki. For example, I have a number of individual notes that just contain links to bike routes I've created in gmap-pedometer or Google Maps. Most of them are tagged with “routes”. I create a Bike Routes page in vimwiki and populate it with

:r !grep -l routes notes/* | xargs cat -s

This dumps the contents of every note containing the word “routes” into this page. There's a little bit of editing to do, but it's manageable.

I went through my remaining notes, one by one. Most of them I left behind. When I found a note that I wanted to keep, I created a node for it in vimwiki and used :r filename to read that file into the page.

As I look for more things to put into vimwiki, I recognize that I have reverted back to storing my notes in textfiles organized in folders rather than use Evernote. I think part of this is that I want to edit using Vim. That's what was nice about Geeknote. I was also avoiding the cognitive overload that comes with opening the Evernote client. This vimwiki solution isn't as feature-rich as Evernote, but I think it's feature-appropriate.