Deep Send: How to wrangle with large object graphs and come out alive

In a perfect world, all of our objects are small, orderly and well defined. They encapsulate data that is semantically obvious and easy to work with. And then… in an obvious homage to Jack Handey, we attack that world with large, complex and ugly object graphs, mostly the result of monolithic integration technologies like SOAP.

Sure, with gems like soap4r, the pain of dealing with SOAP is somewhat minimized, but at the end of the day, you’re still going to have to look deep into the eyes of a huge object graph, and get at the delicate properties you so covet.

What if we were able to reference deeply nested properties from within the bowels of these large object graphs, as easily as we could with neat and orderly objects? How nice would it be not to have to repeat yourself while asking the object graph to cough up two properties buried thirty chained-methods deep?

Deep Send

This is so simple it feels like cheating. The deep_send method could be attached somewhere high up in the SOAP hierarchy if you wish, but I like having it available everywhere.

class Object def deep_send(chained_methods) methods = chained_methods.split(".") methods.inject(self) {|object, method| object.send(method) } end end

As you can see, deep_send works like the regular send method, but takes chained method calls as arguments. Let’s see how this might work in a sample SOAP client:

class HugeNastySoapClient # produces a magnificent cacophony of data elements from all across the globe def get_nasty_graph(secret_key) ... end def get(secret_key, *method_chains) huge_graph = get_nasty_graph(secret_key) results = [] method_chains.each do |method| results << huge_graph.deep_send(method) end rescue NoMethodError ensure return results end end end

In our example, we have a method appropriately called get_nasty_graph. Let’s assume this invokes a SOAP service method and returns all of it’s enormous goodness. We could write a million methods in this class to pull out a few related elements to make a reasonable API. The problem with this is that you’re always wrong. There is always a reason to pull out three of these properties and two of those, and your API just won’t accommodate. If an object needs to use two different API methods to get all that data, it may result in two SOAP calls, which is surely no good. Using that magic get method now allows us to construct our own method to pull out exactly what we want. Nothing more, nothing less.

Encapsulate, you must!

If we stopped here, we’d end up painting ourselves in a corner. Requiring the users of your SOAP client to know about the deep recesses of your nasty object graph is just bad manners. It’s also a leaky abstraction, so stop it!

class HugeNastySoapClient NAME_ENTITY = "some.deep.reference.to.get.to.name" FIRST_NAME = NAME_ENTITY + ".first_name" LAST_NAME = NAME_ENTITY + ".last_name" end

Now that we’ve defined a few constants, the clients of our SOAP client can use our magic get method, without knowing how horrific it is to get at those name properties.

class SimpleElegentPerfectClass def get_full_name(secret_key) now_elegant = HugeNastySoapClient.instance # your soap clients are Singleton's right? first, last = now_elegant.get(secret_key, HugeNastySoapClient::FIRST_NAME, HugeNastySoapClient::LAST_NAME) first + " " + last end end

Gotcha! Order thy calls!

You may have noticed that rescue NoMethodError bit in our magic get method. Choosing to implement that method in this way is not without it’s faults (ha!). If you passed in a method chain that resulted in a NoMethodError being raised, you’ll need to decide what to do. In this example, I’ve chosen to swallow it and return what I got so far. This means that the order of your method chains is significant, if there is a higher likelihood that some of them will occasionally go missing. Isn’t SOAP marshaling awesome? You may decide, of course, to rescue the error, call the authorities and raise the threat level to red instead. Good luck with that.

Things I learned at RailsConf 2008

There were about 2k people at RailsConf 2008, and like most of them, I thought of posting my review of each of the presentations I saw and heard about. Thankfully, the masses have spoken, and there are lots of reviews for you to read elsewhere. Also, if you want to check out some of the presentations themselves, O’Reilly is hosting many of them online.

What I’d like to do instead is to document the things I learned at RailsConf, that stretch across presentations. The first thing that I noticed were the themes that seemed to find their way into almost every session, lighting talk and hallway conversation. There was a lot of focus on testing, deployment, automation, performance benchmarking, design (anti-)patterns and the social aspects of distributed source control management. These are real topics, discussed in great detail by people with real experience to an impassioned audience. It was impossible not to get caught up in the energy and passion, two things that may seem at odds with the typical software conference. Thankfully, RailsConf is anything but typical.

JRuby

WIth all this talk of the various Ruby implementations, my colleagues and myself have been interested to learn more about JRuby. I attended a few JRuby events (the “hackfest” on Thursday was great!) and got a much better understanding of JRuby and what sort of costs/benefits there may be in using it.

The Good:

• I was able to get a small Rails app running on JRuby in a few minutes. It just works.
• The availability of Java types from Ruby is astoundingly simple
• Simple .war and .ear deployments can make for easy integration into a Java environment
• Using JDBC drivers and JNDI is pretty sweet
• For some applications, switching between Ruby and JRuby can be seamless. Thoughtworks does dual-development on some projects (for reasons not explained)

The “bad”:

• Exception handling in Java/JRuby projects is tricky. Exceptions wrap either a Java or JRuby exception and you must unwrap and inspect upon rescuing them.
• It is not uncommon to get Java stack traces in JRuby projects. It would be nice if these were entirely hidden regardless of root cause.
• Windows paths for file URIs is currently broken (I reported this bug during the hackfest, JRUBY-2591)
• The ease of integration for Java libraries can turn an otherwise clean and simple Rails project into a horrific beast of amalgamation. While this is an implementation detail and not a problem of the project, I can see a Java team not wanting to follow “The Rails Way”, and taking away any and all advantages of using Ruby and Rails by overusing Java.
• JRuby is essentially on par with Ruby 1.9 in most performance benchmarks. The JRuby team will be focusing on making Rails perform better on JRuby, but for the time being it does not bring significant performance improvements.

Rails Hosting

With Engine Yard as a major sponsor of the conference, and many startups opting for hosted solutions, this was a well worn topic, with a few key insights.

• The golden ratio is 3 to 4 mongrels (or instances of your app) per CPU core. Don’t debate this, it’s written in copper.
• Cache, cache, cache. Use the same file system across your app instances so they can share page cache. Engine Yard and others spoke highly of Red Hat’s GFS system.
• (Almost) nobody needs database sharding. It is a special use case, with lots of considerations. If you don’t understand it, you probably don’t need it.

Building Software with a dynamic language

Neil Ford had a “Design Patterns” in Ruby presentation, and Obie Fernandez had an interesting talk that showcased awful Rails code, which started to identify some anti-patterns.

• The Gang of Four design patterns are all structural, in that they use objects to construct them. In Ruby, many of these patterns are solved within the standard library, and in an elegant and simple manner.
• Dynamic languages allow you to do things that you can’t conceive of when working in static languages. The mixin/unmix portion of his presentation was good on this point.
• An interesting code smell for refactoring: If code looks visually similar, and differs only in whitespace, it’s likely that you should refactor.
• An obvious notion that should go without saying: Know your tools. Github’s rss feeds on projects (like Rails) makes for an easy time keeping up with the direction of the project, on each commit.
• OOP paradigms are still critical in Ruby and Rails projects. A programmer without OOP experience will likely make for a terrible Rails programmer. Thanks Obie for reiterating this point. Don’t get caught up in the “blog in 20 minutes” vision of what Rails is.
• Beware of Helpers that try to do too much. They can feel anti-OO and are usually the first to break down.

Scalability and Performance Monitoring

• Scalability, on large and high-volume systems, is a never-ending battle of finding and reducing bottlenecks. There is no single solution or architecture. Understand how your app performs, and rely only on data, not hunches.
• Someone else’s benchmarks are as relevant to your application as pickles to ice cream. Sure there is an edge case where they relate, but that’s just gross.
• It’s essentially impossible to benchmark a framework. However, it may be relevant to get lots of specific performance data from lots of Rails applications to determine some relationships. I asked Koz about this during his presentation and he has already talked to New Relic about allowing their customers to share the framework-relevant performance data with the Rails core team.
• Rely solely on real-world performance data in production when making scalability or performance-related adjustments to your system. All else will be either time poorly spent, or will end up hurting you.
• Trending is nice, but the unexpected hockey stick graphs can still happen. Sometimes you just hit a wall (cache sizes, max IO nodes, etc.) that you never saw comming.
• Understand the ratio of capacity to application instances. If you are not getting close to linear growth, find out why.
• New Relic offers some great real-time performance monitoring. Their monitor points are extensible, so you can wrap your own stuff (SOAP calls, etc.) and have it monitored automatically

Testing

There were several talks on testing, and the topic came up in many others. It’s nice to see a community so focused on testing from both TDD and BDD perspectives.

• Josh Susser’s obscure quote that summed up his presentation well: “De gustibus non est disputandum” or, There is no accounting for taste. Josh Susser wrote test suites in all the top testing frameworks for Rails and noted that personal preference and taste may be the only core difference.
• Some testing frameworks (Rspec) influence code design to some degree, although all were capable of helping you evolve your API during the test cycle.
• Each framework had similar levels of test code to test the same app. None stood out as particularly terse or verbose in comparison to one another.
• It is certainly possible and in some cases ideal to mix-and-match testing frameworks. Using test/unit for unit tests and shoulda for functional (behavioral) testing would work well.
• unit_record gem allows you to run unit tests without touching the database. Huge speed improvement, but means you need to plan accordingly when dealing with models.
• fixtures, when used for test data are evil and should be avoided. Instead rely on a Factory pattern or Object Mother pattern which assumes reasonable defaults.
• unit_controller gem allows you to better separate controller tests from the view
• deep_test gem for running tests in parallel. Will sometimes cause problems, decent for individual developers, not for CI

During the conference, I also got a better understanding of Phusion’s Passenger, a.k.a mod_rails. While I didn’t attend either of their sessions, I met with one of their developers at breakfast. They seem to have a great story for reducing the complexity of rails deployment, they leverage some of the great performance of Apache and can handle up to 20 instances of your app per Apache (+mod_rails) instance.

Both David and Kent Beck’s keynote presentations were of an inspirational nature. While David played role of “life coach”, Kent Beck talked a lot about what the best perspective might be on “agile” development, and the various value propositions that make sense when selling ideas to the masses. While I took several lessons away from these great presentations, it’s pretty difficult to sum them up in writing. I’m sure (better quality) videos of them both will be online soon, so you can see them for yourself.

Overall, the conference far exceeded my expectations. It was nice to meet and hang out with a lot of the Rails community, see where our collective heads are at and learn a lot along the way.

Connecting to Oracle from Ruby on Rails

When Rails 2 was initially announced, we heard that commercial database adapters would be maintained outside of rails core.

The commercial database adapters now live in gems that all follow the same naming convention: activerecord-XYZ-adapter. So if you gem install activerecord-oracle-adapter, you’ll instantly have Oracle available as an adapter choice in all the Rails applications on that machine. You won’t have to change a single line in your applications to take use of it.

That also means it’ll be easier for new database adapters to gain traction in the Rails world. As long as you package your adapter according to the published conventions, users just have to install the gem and they’re ready to roll.

While getting ActiveRecord to speak Oracle is available by way of just one gem, connecting to Oracle from rails can still be a touch difficult. Here are some notes from my recent venture into Rails/Oracle land, where I came out victorious, if left feeling just a little dirty.

The Goal
I’ll assume that the desired goal is to have an “application server”, which hosts your Rails app, connect to a remote database server, running some recent version of Oracle (8i+).

Application Server

1. First, make sure you’ve got everything you need to run your Rails app. Ruby, rubygems, Rails, necessary gems (vendor’d or otherwise).
2. Install the Oracle Client (or instant client) software for your hardware architecture.
   Note: On Linux, the client requires over 1Gb of swap space to install. (insert snide remark about how “fat” the Oracle database is here) Here are some tips on adding swap space to a UNIX system.
3. Create a tnsnames.ora file on your app server with the connection information to your trusty, hardworking oracle database.
   (/etc/tnsnames.ora is a nice (and default) spot on UNIX systems.)
4. Install the ruby-oci8 gem.
   On UNIX systems, you’ll need to build the gem, which means you’ll need a fairly complete development stack including make, (g)cc, headers, etc. I’d love to find out if there is an easier way to build this gem so it could be deployed in app/vendor/gems!
5. Install the activerecord-oracle-adapter gem:
   > gem install activerecord-oracle-adapter –source http://gems.rubyonrails.org
6. Configure your rails environment(s) to use the oracle database adapter:
   production:
     adapter: oracle
     database: [your tns name]
     username: [your user name]
     password: [your password]

Gotcha!
Rails migrations specify modifications to your schema in a ruby syntax. The available column types defined in a rails migration will be translated into a database-specific type during the migration. The table below maps the ruby column types to Oracle data types:

Rails migration notation	Oracle data type
:integer	NUMBER(38,0)
:float	NUMBER
:datetime	DATE
:date	DATE
:timestamp	DATE
:time	DATE
:text	CLOB
:string	VARCHAR2(255)
:binary	BLOB
:boolean	NUMBER(1,0)

 

Fixtures
If you’re using fixtures with an Oracle database, make sure the values in your yaml file can be translated approrpiately into the corresponding Oracle data types.

One example may be the translation of a bare Fixnum (eg. 100) into a column set to :text. Since Oracle translates :text columns into CLOBs, the Fixnum in your fixture data will need to be coerced into a String. FAIL! In this case, quote the values in your fixtures (eg. “100″).

Further Reading
Obie Fernandez’s article on Connecting to Oracle from Ruby on Rails (A little dated, thus this post)
Oracle Instant Client
Ruby on Rails with Oracle FAQ
Tips for optimizing Rails with Oracle

Supporting unique headers per SOAP request in soap4r

soap4r is a SOAP implementation in Ruby. If you’re writing a SOAP client in Ruby (with or without Rails), soap4r is great. Until it isn’t. Since this is a rather specific use case that I’m solving for, I’ll assume that you have some experience using soap4r.

Headers
Some SOAP services will utilize a header, which is part of the SOAP spec. A header may include information that is used to process the request, but is not part of the SOAP request message body. It is typical for some services to use headers to send API keys or other authentication mechanisms.

Here is a simple example that ships with soap4r

Headers are a property of the Driver?
Yes. As you can tell, the SOAP driver has a property called headerhandler, which stores the header(s) that will be sent along with each service call. The call looks like this:

driver.headerhandler << ClientAuthHeaderHandler.new

Since << is an alias to add, we’re just adding a header to the driver here.

But what if the service requires certain unique properties in the headers of each service request? Let’s say you have a service that within the header of the SOAP request, requires that you specify the IP address of the remote host, if the service is initiated via a web browser. In this use case, we need a unique header per SOAP request.

Open up the headerhandler
We can see that the headerhandler property of the SOAP driver holds a SOAP::Header::HandlerSet object. Let’s open ‘er up.  We’ll add two new methods that will allow us to set a single header on the driver, regardless of its current state.

class SOAP::Header::HandlerSet def reset @store = XSD::NamedElements.new end def set(header) reset add header end end

Set and send
Now we can do this, to set the header before we make our call:

driver.headerhandler.set( unique_header ) response = driver.getFoo("bar")

Gotta synchronize!
Since the creation of a SOAP driver is expensive (for those of us who create them on-the-fly by reading in the WSDL), it is common practice to cache the driver and reuse it each time. (Remember that this is how we got into this mess in the first place, since the headers live on the driver, and we reuse the same driver for every request.)

It is possible, then, to have two threads interacting with your service client class, which in turn is relying on this same driver for those two requests. The code above is not thread safe, in that the driver’s headerhandler may have been set by thread1, but thread2 makes the first getFoo service call, resulting in the sending of the wrong header. DOH!

To fix this, we’ll need to wrap that call in a synchronize block:

@@guard = Mutex.new ... @@guard.synchronize do driver.headerhandler.set( unique_header ) response = driver.getFoo("bar") end

Sure, it is ugly. Since the design of soap4r is such that the headers belong to the driver, this sort of solution may be our best bet. I look forward to any alternate suggestions.

Note: This work was in response to my original question to the soap4r forum.

Using AJAX to load Rails partials

I had an excuse recently to try something I thought was going to be trivial, what with all the kids using AJAX these days. Unfortunately since I wasn’t trying to create another live search feature, documentation, blog postings and several books on the subject were surprisingly lacking.

See, all I was trying to do was to load a page fragment, or partial in rails speak, in an asynchronous way. While this doesn’t sound very glamorous, its a great way to make your web application feel more responsive when you have a particularly heavy weight partial to deal with.

An Example:
A page within your web application displays your site layout, a menu or two, some paginated flummoxes, topped with a lovely, glass like logo that is showing it’s bright and shiny reflection on some strange surface that shields your website from anything that might fall from the address bar.

Toward the bottom of the page, you display some recommended flummoxes, based on whatever flummoxes the user has visited recently. (As you can surely tell, this example is not only made up, but is quite far from the more complex scenario that led me down this path in the first place. As we know, creating example code is a PITA.) This recommendation flummox feature is a real drag. It can take seemingly forever to render since it involves a few complex queries, and of course a REST call or two. Wouldn’t it be great if you could render the whole page, minus the flummox recommender, and have that feature filled in once it responds? Exactly.

The Call to Action

http://localhost:3000/browse

Note that this request is (likely) handled by a BrowseController. The important thing to realize is that we need to pull out the slow, heavy logic that would normally drive the content in that slow flummox recommender feature and put it elsewhere. In our example, we’ll pull it out into its own, new controller.

The Controller
Here we have our controller, which will coordinate our requests to our model, as well as make a web service call or two:

class SlowController < ApplicationController def super_slow_action_call @query_results = some_model.find(:all) @type = "name_of_real_partial_to_load" @heavy_data = some_slow_method_call(@query_results) render :partial => "load" end end

Notice that render line? We’ll come back to that in a minute.

The View
index.html.erb (This is rendered by the original call to BrowseController, our otherwise fast loading page.)

<% @some_awesome_erb_here %> . . . <% javascript_tag do -%> new Ajax.Request("<%= url_for :controller => :slow, :action => :super_slow_action_call %>", { method:'get'}); <% end -%> <div id="name_of_real_partial_to_load"></div>

Aha! There’s our AJAX! You’ll notice that I have to actually type some JavaScript there. I am alive to talk about it, but I sure don’t like to dirty my hands in it. If this pattern seems useful to folks, I may try to submit a patch to one of the rails helpers (JavaScript or Prototype helper) to make this a bit less dirty.

So what are we doing there? We’re creating a new Ajax.Request, thanks to our beloved Prototype JavaScript library, and feeding it a URL to request via a GET method. We can use the url_for helper to clean that up a bit, so it’s not too nasty.

Our erb partial:
You’ve probably already built the code for the partial that displays those flummox recommendations, right? Notice that @type variable in our controller? We set that to “name_of_real_partial_to_load”, so let’s create app/slow/_name_of_real_partial_to_load.html.erb

Here are some flummoxes you might like:<br> <% @heavy_data.each do |flx|%> <li><%= link_to flx.name, :action => :flx, :id => flx %> <% end %>

Great, but how do we get that to render? We made a call to load “load” from the super_slow_action_call action in our SlowController!

Our RJS View, the glue to load them all!
app/slow/_load.rjs

page.insert_html :bottom, @type, :partial => @type, :object => @heavy_data page.visual_effect :highlight, @type, :duration =>1.5

Here, we’re inserting our partial (@type), into a div by the same name (@type), within the original view serving the request.

Since it is such a slow loading page fragment, we mine as well draw some attention to it once it gets there, so we’ll highlight it using our dear friend script.aculo.us.

So there you have it. Render a slow-loading partial in your Rails application, using AJAX.

Note:
Admittedly this pattern takes a bit of elbow grease to build out. If you have alternative suggestions, aligning more closely with The Rails Way, please let me know. If this pattern seems to mass the muster however, I’ll consider appending to one of the existing helpers to facilitate the Ajax.Request bits, to make this a little smoother.

Next Page →

• 
  
  

  Brian Doll
  Software/Systems Architect
  San Francisco Bay Area, California
  

  brian@emphaticsolutions.com briandoll @ github
  Subscribe in a reader
  
  

  
  
  
• 
  

• Recent Posts

  • New gem: rails_framework_diff
  • Deep Send: How to wrangle with large object graphs and come out alive
  • Things I learned at RailsConf 2008
  • Connecting to Oracle from Ruby on Rails
  • Supporting unique headers per SOAP request in soap4r
  • Using AJAX to load Rails partials
  • Quick Book Thoughts: The Back of the Napkin
  • Google Calendar Sync Doesn’t
  • News travelling around the internets
  • Troubleshooting Tips
  • Quick Book Thoughts: Presentation Zen
  • ActionWebService on Rails 2
  • Blocks, Closures and The Ruby Way
  • Dancing with dynamic features in Ruby
  • Leaky abstractions, seepage of a dangerous kind

• Blogroll

  • /.
  • art and craft of architecture
  • coding the architecture
  • gapingvoid
  • grady booch
  • how to change the world: guy kawasaki
  • interface21 blog
  • jay fields thoughts
  • joel on software
  • lifehack
  • martin fowler
  • meme agora
  • otaku, cedric’s weblog
  • paul graham
  • paul graham’s essays
  • pragmatic craftsman
  • presentation zen
  • schneier on security
  • terse systems
  • the rails way
  • the tao of the mac
  • wide awake developers

• Categories

  • books
  • java
  • mac
  • rails
  • ruby
  • software

Copyright © 2007 Brian Doll (emphatic solutions) • Using Dropshadow theme by Brian Gardner