Python is not Java

calendar Apr 29, 2008 · 2 min read · python  ·
Share on: linkedin copy

I admit it, I’ve been working on my middleware python server.   Here’s the basic design of the app at this point:

PHP <— JSON-RPC —-> Python <–> MySql

The assumption is that as things grow, you’ve already defined a good layer to scale at and have a reasonable Object abstraction from the front end.  At one level it’s turning out to be duplication of work, like duh!  I almost need three versions of the same object instance.

  1. the ORM version (read/write to the DB)
  2. the Python version — manipulation and management
  3. the PHP version — slightly different mix of methods, but same data

That’s the object trials and tribulations.  Now we’re off to RPC land.  First you have to pick a framework, which if you want to use Python and JSON-RPC [lighter weight than XML-RPC] you end up in a world of hurt since the jsonrpc support for the long laundry list of frameworks is pretty piss poor. 

Currently using Twisted as the framework de jour, but it’s json-rpc library (via a code.google.com project) doesn’t quite match the spec!  For instance by default it always returns a list.  Second problem, which turns out to be a fun rant…  This is where one really discovers that Python is not Java, is that if you want to serialize anything out via the Twisted mess of frameworks you’re waiting for pain.

  1. simplejson (as used by Twisted JSON-RPC) is slow
  2. cjson is fast, but suffers even more than simplejson

Biggest problem, I want to have this nice simple return statement in my json handler:

1return { 'data': object }

Looks nice, but wait!  simplejson doesn’t know what to do with an object.  The only “fix” is to subclass simplejson to extend the handlers.  Of course since I’m sitting eight layers down in the stack from where simplejson is instantiated you can’t really do that.  So, what’s the fix?

The Fix: tear apart the Twisted JSON-RPC handler make it use both simplejson and cjson (why not) and add 4 lines of code to cjson encode_object:

1if ((result = PyObject_CallMethod(object, "__getstate__", NULL)) != NULL) {
2    PyObject *r = encode_object(result);
3    Py_XDECREF(result);
4    return r;
5}

Ah, now we’ve duck typed the pickle __getstate__ method to give us a back a dictionary…  The problem of course is that you can’t register “unknown” type handlers for any of the json serializes out there.