It’s a truism that one of the first bottlenecks in scaling an application is the database. After all, its the most obvious shared resources, and is hit by almost every request (assuming of course that we haven’t optimized anything by caching). There’s a variety of well-know techniques for helping with this (caching, mater-slave replication, scaling up the database engine/server), but here I want to consider increasing fine-grained parallelism by making MySQL overlap reads and writes whenever it can. It can do this using a concept called non-locking consistent reads, made possible by a property of the InnoDB storage engine called Multi Version Concurrency Control.

(more…)

Ah, that old chestnut again. It’s a common misconception. How many times have you heard ”Yes, I’d love to use Rails – but it doesn’t scale” in response to some argument or another about how/why someone should use Rails to build their next web app. And, like all simple answers to complex questions, there is a measure of truth in it. But the real answer to “does Rails scale” is that old philosophical answer: it depends.

It depends on what you are asking, and what you mean by “scale”. If you mean ‘Can I build my Rails app in a standard way and expect it to handle hundreds of hits a second’, the answer is ‘No’. If you mean ‘with some extensive work on architecture, caching and database layout, can I get my Rails app to handle 100,00 users and 10,000 hits an hour” the answer is ‘Yes’. So why is that, and how could it be made better?

(more…)

Just got back from LSRC in Austin, TX. Had a good time. There were a couple of great talks, and one or two clunkers, though mostly the content was middling to good. Dave Thomas gave a fun opening talk on why he likes Ruby. He said it ‘thinks the way he thinks’ and that while it’s imperfect – this is a feature, not a bug (’The pursuit of perfection impedes progress’). He gave several examples of wonderfully ambiguous Ruby code, but pointed out that this ambiguity had an upside – Ruby reads beautifully for 99% of the important cases.

Glen Vandenburg talked about programming intuition, what made a good programmer, and why he thought there was such a productivity difference between good and mediocre programmers. He said good programmers think about code as if they can see and feel it. Pointing out that what most of our brains do is deal with the real world, he suggested that this was the way good programmers engaged the best and biggest parts of their brains in the task of programming. If we are oriented to dealing with the real world, then by conceiving of code as ‘real’, we engage most of our brains in the task.

This rings absolutely true with me. For me, building code is akin to sculpture. I’ve always conceived of code as ‘real stuff’, and work with it mostly in terms of shape, relationship and structure, at many levels of scale – from the statement level to the system level. He talked about beauty, and how good programmers are guided by a sense of aesthetics. Again – this is absolutely how I think about code quality.

Following on from this, he then suggested that this might show a way of teaching programmers to be better, emphasizing patterns, relationship to the code, tools and a shortened feedback loop between cause and effect. I couldn’t agree more. One of the current rants in my head is the primitiveness of most of the tools that Ruby programmers use to build software. Can we please do better than Vim and printf for debugging? (Update: I’ve been using RubyMine 2.0 beta, and it’s pretty good. This from a died-in-the-wool Smalltalker!)

Rich Kilmer gave a great final keynote on the encoding of domain knowledge. He described a process he used to sit with a domain expert and create a domain-specific DSL on the fly, in syntactically correct Ruby, as he and the expert sat and explored domain concepts. He then fleshed this out, and used it to drive a UI written in Flash. It’s an incredibly efficient process, and blew the client away.  I’m trying to get the presentation, and will post it when/if I do, but there is more on his blog: http://richkilmer.blogs.com/ether/2008/09/rubys-best-feature.html