In my case, it seems like both the gnome-settings-daemon and the brightness switch for the video driver were both trying to adjust the brightness, so whenever I pressed I button, it adjusted twice.
Now while the proposed solution
worked, it stopped me from adjusting brightness on a tty (ok not a biggie), but especially I've noticed that leaving brightness up to gnome is slow and sluggish.
sudo sh -c 'echo -n 0 > /sys/module/video/parameters/brightness_switch_enabled'
So instead I thought I'd disable gnome instead. There's a lot of old information relating to gnome-power-manager and other deprecated configurations, but I managed to discover that gnome-settings-daemon uses the gsd-backlight-helper (on my system it is installed as /usr/lib/gnome-settings-daemon/gsd-backlight-helper) to do its work.
To disable it, I just did
sudo chmod -x /usr/lib/gnome-settings-daemon/gsd-backlight-helper # For Ubuntu 13.10
You do lose the brightness OSD, but that's a price I'm willing to pay.
sudo chmod -x /usr/lib/unity-settings-daemon/usd-backlight-helper # For Ubuntu 14.04 LTS