Tuesday, February 28, 2012

Metaprogramming in Ruby

Just yesterday I started the what I learned today series and I'm already thinking it should have been a weekly digest. Not that I didn't learn anything new (I, in fact, did learn a few things that I will share in a second), I just figured it would be too much noise to publish a digest daily. So weekly it is.

Today I instead want to talk about metaprogramming in Ruby. I actually not so much want to talk about it as I want to put a summary for my future self. Like a collection of links that finally made me get it. The links are in no particular order:

I am sure there are more decent blog posts out there on what metaprogramming in Ruby is about and how to do it right and when it is right to do it so please add it to my collection.

UPDATE. As I was wrapping up this one I came across two more very interesting and very relevant posts:

  • awesome writeup on include and included. It goes into showing off bizarre examples of massive metaprogramming in ActiveRecord as well as some conscious evolution of rails in doing it more right than it was doing it before
  • A response to the post I just mentioned that brings up an interesting point of namespace pollution. While I am ok protecting the global namespace in JavaScript with the "closure everywhere" (function(...){})(); pattern, I am not sure I see the point in giving up syntax convenience and protecting my class namespace in the same way.

What I Learned Today (02/27/12)

There shouldn't be a day in your life when you don't learn something new.

I believe we all learn every day and just not always think about it. I thought I would start a daily digest of things I learned "today" to help me better remember it and to also share it with you as some may find it useful.

Some of it comes from the things I did while working on a problem in the office, some comes from reading or listening to other people, some comes from proof reading my son's homework. There's clearly different depth to the whole notion of "I learned it" so I by no means say that I am now an expert at the things I learned.

Too much "learned" already so with no further due today I have learned:

  1. Groovy way of grabbing Maven dependencies via Grape and @Grab import annotations isn't searching in your local .m2 repo. You will need to make an explicit configuration in your .groovy/grapeConfig.xml to make it do a trip "home".
  2. Cassandra's secondary index is a hash index so all you can ask it is the equality query. you can't do a range query on it and bitmap secondary indexes are not yet on the roadmap
  3. The way you bulk load data into Cassandra is via building SSTable files directly and pushing them to the cluster. Nothing new really, all "databases" do direct datafiles writes for fast bulk inserts but here you literally create datafiles "increments"
  4. Ruby has a define operator that you can use to find out if super is defined for the method you're in so you can do things like super if defined?(super). Pretty neat
  5. Maven requires you to specify a version of your dependency and would otherwise refuse to build. There's a very popular question on SO why it's this way and how to make it go for the "latest" version.
  6. There are 8 planets in our solar systems. There was 9 when I went to school and Pluto was since downgraded to be a dwarf planet with two more dwarfs discovered so it didn't feel lonely

Friday, February 17, 2012

Net:::HTTP, Encoding, \x8B

Last night coding session, that I kind of waited for the whole evening while putting my kids to bed, quickly turned out to be the "what the heck is going on" session.

I have this small home project where I am building a Ruby library wrapper around a public JSON API. It's half-fun-half-learning exercise so I am trying to stay down to bare metal Ruby without unnecessarily introducing gems that I don't really need. So I am using net/http to interact with the API endpoint and it's been all fine until yesterday.

My code fires up the request, stucks the response right into JSON.parse() and off it goes. As I build this new class to represent a new entity my tests start to fail with :

JSON::ParserError: 743: unexpected token at '▼'

The saying goes that there are two categories of developers when it comes to debugging. Faced with an error, one would fire up the debugger and dive right into it, while another one would sprinkle puts statements and just follow the trails. I am a member of the second group. If you think it's not right then I would encourage you to listed to the Debugging in Ruby episode on the Ruby Rogues podcast. There are cases when debugger is the way to go, but I am convinced that working on your own code with a good test coverage (like it should be!) you're better off finding the problem with your brain and some help from the puts method.

As I was inspecting the response object that JSON parser choked on I got to learn the Ruby 1.9 String#encoding and some other very interesting things. Somehow I remembered early 2000s when I was actively writing Java for the web and dealing with encoding to support multilingual was a black belt art. The database, the IO streams, the XSLT transformations, HTTP headers and HTML meta tags, and what have you. Dejavu. Only it wasn't.

The next thing I did was searching for what Web the almighty had to say. I very quickly figured that if JSON library couldn't figure the encoding then probably no one would. There were receipts suggesting to go through the JSON library only to get the encoding straight! I also came across an enlightening conversation on the ruby lang about whether net/http should set the encoding based on the HTTP headers or not. Good stuff. I still thought it was the encoding issue and for a second I even considered resorting to Mechanize as was suggested in that thread on the ruby lang, even though it reportedly had some similar issues as well

Well, if I did move to Mechanize it wouldn't have helped me. It wasn't the encoding issue. It turned out to be exactly what my code was asking for and all I had to do was to teach my code not to ask for what it couldn't chew on.

Sending JSON requests via net/http I was pretending to be a browser. Not that I needed it, but since I had to send some state in cookie headers anyway I figured I would just pretend I am a full blown browser client. Among other harmless things that I carried over from the Firebug Net log was the following rather important "expectation":

Accept-Encoding=gzip, deflate;

I was telling the server that I am ready to process zipped content and I wasn't. All previous JSON requests that I sent were small enough for the server not to bother zipping it so I had no issues. And then I asked for something big enough and I said I am ok with the things zipped. The server was built by some good guys who knew their stuff and they did just like I asked. Removing that single line and not telling the server I am ok with the things zipped did the trick... Then I thought how silly it was of me not to get puzzled from the byte stream that I saw when I first time dumped the response object to the STDOUT. it must have been too late already :)

Even though I didn't have good time coding the way I planned it, I had a terrific time troubleshooting. It's through exercises like this you learn to be a better software engineer - I got exposed to so many new things thanks to this little trap I got myself into. It's basically like bugs-driven-learning :)