Evernote Extraction

I remember hearing about something called jq that should let me work with JSON. The apt package description for jq starts with, “jq is like sed for JSON…”. Well, I’m sold. Also, no dependencies! What a bonus. The man page is full of explanations and examples, but I’m going to need to experiment with the filters. After some experimentation, I land on

jq '.[] | .createdate,.content' journal.json

This cycles through each top-level element and extracts the createdate and content values. Now I wonder how I can add a separator so that I can dissect the data into discrete files with awk or something. I should be able to add a literal to the list of filters.

jq '.[] | .createdate,.content,"%%"' journal.json

Well, the %% lines include the quotes, but that’s not the end of the world. I wonder what date format I need for jrnl. Each jrnl entry starts with

YYYY-MM-DD HH:MM Title

Evernote gives me dates that look like

Jul 25 2011 HH:MM:SS

date –help to the rescue!

Looking at date handling in jq, I should be able to convert the dates from the format used by Evernote to the format used by jrnl with the filter

strptime("%b %d %Y %H:%M:%S")|strftime("%Y-%m-%d %H:%M")

All together, then.

jq '.[] | (.createdate|strptime("%b %d %Y %H:%M:%S")|strftime("%Y-%m-%d %H:%M")),.content,"%%"' journal.json

I still have some garbage in there, but I’m getting close to being able to just prepend this to my journal.txt file. OK, I’m close enough with this:

jq '.[] | (.createdate|strptime("%b %d %Y %H:%M:%S")|strftime("%Y-%m-%d %H:%M")),.content,"%%"' journal.json | sed -e 's/^"//;s/"$//;s/\\n/\n/g' | sed -e '/^ *$/d' >journal.part

Okay, let’s try the recipes notebook. My recipes notebook should be a little more challenging than my journal entries, but it’s not as massive as my main notebook.

ever2simple -f json -o recipes.json recipes.enex

My journal json file was 5k. This one is 105k. Running the same command as before gives me pretty legible output. I know some of these notes had attachments, but I don’t see them in the JSON. I wonder if they are mime-encoded in the XML file.

Looking back at my recipes.enex file, attachments do appear to be base64 encoded in the XML, but ever2simple doesn’t copy this data into the JSON file it creates. This makes sense since its target is Simplenote. Maybe html-xml-utils can help me get these files out.

hxextract 'resource' recipes.enex

It looks like the files are encapsulated within resource elements. The resource element contains metadata about the attachment and the base64-encoded data itself is inside a data element. I can isolate the data using hxselect.

hxselect -c -s '\n\n' data < recipes.enex > recipes.dat

This gives me all the mime attachments in a single file. Each base64-encoded file is separated by two newlines. This doesn’t preserve my metadata, but I’m anxious to get the data out and see what’s in there. Let’s see if I can pipe the first one into base64 -d to decode it. An awk one-liner should let me terminate output at the first blank line.

awk '/^$/ {exit}{print $0}' recipes.dat | base64 -d > testfile

Now I can use file to find out what kind of file it is.

file testfile

This tells me that it’s an image. A JPEG, to be specific, and it’s 300 dpi and 147x127. That seems seems small. I wonder if Evernote encoded all of the images that were in the html pages I saved. Opening the file in an image viewer, I can see that that’s exactly what it is. How many attachments are in there? Could I…

sed -e '/^./d' recipes.dat | wc

Damn, that’s slick. There are 74 files in there. I’ll bet only a handful of them have any value to me. I think the easiest way to go forward is to copy each base64 attachment into it’s own file. Looking at split(1), it splits on line count, not a delimiter. What if I do something like…

#!/usr/bin/awk -f
BEGIN {fcount=1}
/^$/ {fcount++;}
{ print $0 >> "dump/" fcount ".base64"}

This goes through my recipes.dat file and puts each base64-encoded attachment into its own file. Now I need to decode them and give them an appropriate suffix.

#/bin/bash
for f in dump/*
do
  outfile="${f%.*}.out"
  base64 -d "${f}" > "${outfile}"
  type=$(file ${outfile})
  type="${type#* }"
  type="${type%% *}"
  newout="${outfile%.out}.${type}"
  mv "$outfile" "$newout"
done

Phew! Now I have 74 files to look through. Most of these are garbage from web pages I saved. There’s really only five of these that I want to keep. There are a few problems with this approach:

• I lose the original file name.
• I use the file utility to reconstruct the filename extension.
• I lose the association between the file and the note I saved it in.

Looking at my main notebook, I may revisit ever2simple’s -f