Sunday, September 25, 2011

Land and Waste New Zealand - #mixandmashnz

Land And Waste New Zealand



Here's a behind the scenes write-up for my Mix and Mash NZ mashup.

Check it out here :

http://www.landandwaste.co.nz

This mashup attempts to show land and waste data as it changes, and hopefully illuminates any improvements of our environmental impact as well as regressions.

The journey

It started out as an idea to gather all kerb recycling data from all 75 odd local councils in New Zealand. That meant scraping the data off the websites of various local and unitary councils. I quickly backed away once I found that lots of councils forbade any re-use, and the data that's available is too inconsistent for me to compare and visualize.

Therefore much time and effort was spent looking for suitable data. I was interested in mainly environmental type data, which was to found mainly on the Ministry for the Environment website. I was also interested in regional level data, instead of a single national figure, which wasn't that interesting .

In the end, I found suitable data for land use, rates, farm animal counts, population, and waste. Given more time, I would have hunted for transport data, which would then completely encompass all the major responsibilities of a regional council.

Putting it together
This mashup was made with Ruby on Rails, TileMill (http://mapbox.com/tilemill/), d3 (http://mbostock.github.com/d3/) and polymaps (http://polymaps.org/)

The excellent Koordinates.com put up a CC licensed shapefile of various land types in New Zealand. Instead of reaching for the de-facto choice of Google maps, this time I chose to try my hand at creating custom map tiles from that data. This had the advantage of being able to design the map to be far friendlier for statistical visualizations like cartograms, etc.

TileMill is simply a brilliant piece of software, which allows once to pull data from various sources and create beautiful maps, using any style you want. In addition, there are options for mouseover and click events.

There is an export function which will allow you to host the map tiles online ala Google Maps. I chose to utilise the hosted service at TileStream (http://mapbox.com/#/tilestream). It is completely possible to self-host as well, as the hosting code has been made available athttps://github.com/mapbox/tilestream.

The final map I produced using TileMill can be seen at http://www.landandwaste.co.nz/land.

Importing and processing the tabular data seen on the mashup was just a matter of using a combination of Google Docs, Google Refine, and csv files. Choosing a suitable graphical representation for each set of data was slightly tougher. I used a combination of sparklines, quantile maps, and just plain text. This was done using d3 and polymaps.

Land use area visualization (example here for Bay of Plenty)

Waste visualization for each region in New Zealand


Where to from here
It was a huge learning experience. I learnt that creating something useful is very possible, and takes less time than I expected. Creating something polished requires far more time. I will definitely find a designer or improve my own design skills next time.

Monday, September 13, 2010

Fix home/end keybindings in Terminal.app on OS X

Go all the way to the end of the less buffer! Drove me crazy the way end behaves exactly like page down instead. This tip fixed that for me.


Monday, August 23, 2010

Untitled

Amplify’d from martinfowler.com

Identity Map

Ensures that each object gets loaded only once by keeping
every loaded object in a map. Looks up objects using the map when
referring to them.

An old proverb says that a man with two watches never knows
what time it is. If two watches are confusing, you can get in an even
bigger mess with loading objects from a database. If you aren't
careful you can load the data from the same database record into two
different objects. Then, when you update them both you'll have an
interesting time writing the changes out to the database
correctly.

An Identity Map keeps a record of all objects that have been
read from the database in a single business transaction. Whenever you
want an object, you check the Identity Map first to see if you already
have it.

Read more at martinfowler.com
 

Facebook - greedy defaults and jarring configs

Commited Facebook Suicide for the second time. Where do I start? ...



The first thing that jarred me was that my full birthday is published. This in combination with search engine visibility means that it's spammer time.



Next thing that happened is that the privacy settings had "Everyone", "Friends of Friends", and "Friends". Nice and simple, I thought. But no, I have to exclude my full birthday again from everyone. (I know about how your friend's applications can just mine everyone's data through friends).



Ok, I'll just configure that again.



What do I see next. Social Ads. Ads. Default On. In my name. I hate Ads that purport to be in my name. That's it. Goodbye.



--

Here are some tools below that can help you, but to be honest, I don't think anyone can keep up with the myriad of configurations.



http://www3.untangle.com/saveface

http://www.reclaimprivacy.org/

http://github.com/mjpizz/reclaimprivacy

http://itunes.apple.com/us/app/s-n-a-p/id384773305

Friday, June 18, 2010

Bash hackery to get around old rspec

I have a problem where a project is using an older version of RSpec (currently the lastest version is RSpec 1.3.0). Therefore running
spec spec/model/some_model_spec.rb
gives me an error
/spec/models/../spec_helper.rb:23: undefined method `use_transactional_fixtures='
The solution would be to use the older, vendored spec. Now, the spec binary that is installed as part of the gem installation should do this (some gems do checked for vendor'ed binaries) and my rspec is installed under vendor/plugins, not vendor/gems. Therefore,  I use some bash trickery to dynamically redefine the "spec" command. Put the code below in .bashrc :

vendored_or_path_spec() {
  if [ -x vendor/plugins/rspec/bin/spec ];
  then 
    echo 'vendor/plugins/rspec/bin/spec';
  else
    echo '*spec';
  fi
}
alias spec='$(vendored_or_path_spec) $@'


Some explanations. The hardest thing to figure out was how to dynamically alias something. After such scrumbling and asking on vark.com, I managed to find command substitutions, which is the $(command) part. Combining that with $@ for all arguments meant that I can run any command in the alias.

Because I'm aliasing spec, I use *spec to refer to the original un-aliased spec.

That's it!

Friday, April 23, 2010

How to upgrade to Firefox 3.6 for Karmic (Ubuntu 9.10) - firefox-stable

There are a plethora of install instructions out there for Firefox 3.6 Karmic (which uses Firefox 3.5 by default), however it mostly suggests the mozilla-daily build.

By chance, I found the firefox-stable ppa (https://launchpad.net/~mozillateam/+archive/firefox-stable/)

Here's how (enter each in turn):
sudo add-apt-repository ppa:mozillateam/firefox-stable
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
Here's what it looks like:
~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages have been kept back:
  firefox firefox-3.5 firefox-3.5-branding firefox-gnome-support
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
~ $ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  firefox-branding
The following packages will be upgraded:
  firefox firefox-3.5 firefox-3.5-branding firefox-gnome-support
4 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.5MB of archives.
After this operation, 32.4MB of additional disk space will be used.
Do you want to continue [Y/n]? y
You will now get /usr/binfirefox as Firefox 3.6

Sunday, February 07, 2010

Rails 3 presentation

At the January WellRailed (Wellington Rails User group), I gave a presentation on the upcoming Rails 3.0, with special emphasis on cool new things.

You can view the slides after the jump. There are a huge number of changes, but it should be all good fun :)