i am organizing my notes from cloudcamp nyc 2009. the whole thing was done the unconference style, and i bounced around different groups for a few hours.

taxonomy

it seems that folks were struggling to come up with the taxonomy of the cloud. i see it is a continuum from bare OS instances (amazon’s EC2) to increasingly specialized services (google app engine, S3, salesforce, and even facebook).

bare OS instances are easy – nothing there is new, and what is changing is the elasticity of the capacity – you can spin up and wind down instances much faster (and recently announced amazon’s auto scaling makes it even easier). the only catch there is that you have to design for scalability yourself. but you have all the power and the flexibility, and there is no lock-in.

the latter is when you give up some of the flexibility for scalability out of the box – the design and implementation are taken care of for you – google app engine and facebook just scale. there is a lock-in there, but you might not care. somewhat unrelated, but pmarca had some things to say about it a while ago.

automation

when dealing with OS instances, these developments make it obvious that if you have not embraced automation yet, it is time to do so. the lone admin doing things by hand does not cut it anymore; even barebones shell scripts and basic packaging might not be enough.

the sad thing is that i do not see an easy, thorough, cross-platform management tool that does everything out of the box. cfengine was the closest thing back in the day, but i want something that embraces source control and declarative configuration; something that is transparent and easy to use; something that is cross-platform. chef and puppet look promising.

so there are a lot of new vendors coming out and touting their products for managing the cloud.

monitoring

same thing with monitoring – sscope was a killer app almost a decade ago, but i have not seen a compelling polished replacement (until, possibly, hyperic) – something that could scale down and up, something that can be either agentless for small easy deployments and agent-based to be able to scale beyond that. amazon’s cloud watch is a step in the right direction, but it is just the beginning.

i hope that cloud computing developments will give this industry a push it needs to clean up its act and finally come up with comprehensive convenient set of tools that is sorely overdue.

bottom line is that with clouds there will be more stuff to manage, and the demands for the management tools will become stricter.

i want commodization of these tools – monitoring, deployment, etc. ideally they will be independent of cloud provider and i want them to be simple and transparent and composable and hackable.

usage scenarios

it is almost a no-brainer these days for startups writing green-field apps to rely on amazon for their capacity. it is great, since they are not as concerned with SLAs and lock-in is not that big of a deal.

amazon truly has kick-started the industry that lurked in the shadows for so many years with all the major players trying and failing to launch a successful pay-as-you-go cloud computing service. i guess their secret was extracting a service from their own successful projects, instead of trying to come up with something new for others.

as for enterprises though, i do not see them embracing the cloud platform as it is now, however there are some scenarios possible right now, and i am sure many more will emerge:

• a single batch that needs to be processed (amazon now has hadoop services ; also consider load testing from the cloud, some sort of one-off processing, or anything else that is a spike in demand could be good)

• overflow capacity for something like flash traffic during holidays for retailers – extend your private infrastructure, or certain pieces of it into the cloud and wind down when you are done

• POC, development – it is hard to test on the same hardware as prod, so roll your stuff out on the cloud and then for prod/uat deployment put it back inside

• one-off isolated apps that can deal with current state of SLAs and lack of private clouds, or even pieces of existing apps

there is a chance for internal infrastructure folks to learn something from the cloud experience. in most large companies servers and storage have been increasingly provided as a service and billed back to the projects, but provisioning is still not elastic, and utilization is often the usual low numbers. granted, more and more are using vmware to virtualize (at least dev/uat) to increase utilization, but it is a far cry from the experience of the likes of amazon.

the ideas and some of the tech from the public clouds could and should be looked at by the internal infrastructure people, and not just the servers/storage, but also those that provide middleware services – messaging and monitoring, for instance.

clouds in your own data centers? some of the vendors seem to head this direction – give us the servers and the datacenters and we will create cloud infrastructure for you, taking care of management, monitoring, apis, provisioning, etc.

i wonder how the chargebacks in organizations will change between departments – since instead of a capital investment upfront with depreciation we can convert it to operating expense with pay for use billing – would that allow us to win anything? of course, for small companies it is great – no upfront costs, no hardware, etc.

ms azure is somewhat an outlier here – first of all, it lacks tinkerability (yes, it is a term) – the barrier for entry is too high, but at the same time it is not a vertical app, but a full-blown platform with all the development tools in place. the problem is that it is too opaque, and i would imagine it will be a while before the whole kitchen sink has proper api, billing, provisioning, etc working properly – as usual the idea is fantastic, but the implementation, knowing the track record, is something to be wary of. since windows is notoriously hard to automate, perhaps azure will finally sidestep this problem and offer that as a service, but i cannot see how it can completely do it, unless it provides something as easy and transparent as ssh+shell scripts and declarative versioned management for that.

coming up

development in several directions – granularity of service, privacy of the cloud; standards – one must talk about a certain place in the continuum – standards for infrastructure level monitoring/provisioning are perhaps possible, but once we run into more vertical services, standards make little sense, since you are buying into the provider.

there will be consolidation, since, as failures of ibm and sun showed and success of amazon shows, it is a low-margin business, and amazon knows it first-hand.

gotham ruby confrence is the nyc’s own ruby conference organized by nyc ruby users group. it was my second time attending it. this post is an attempt to organize my own notes, as well as an attempt at feedback that i think i owe to the speakers and the organizers.

i really like the smaller focused gatherings like this with around a hundred-plus attendees – there is definitely a community spirit there, since a lot of the folks are local and already know each other.

there is also more focused discussion, since the background of the people in the room is similar. a lot of the culture is shared, a lot of the values are implied, and the conversation zips along nicely. i have stopped attending local java groups precisely for the lack of this common background – the topics are too broad, the backgrounds are too different, and it takes a lot of effort to communicate ideas.

i am a bit uneasy about my own relationship with the ruby community – i am not really contributing, nor am i doing paid ruby work. i also am a bit weary of the monoculture (look at them macboys and macgirls!) that tends to re-invent the wheel way too often. but i am there for the excitement, for the bright-eyed kids that tinker and create things – this energy is infectious and i feed off it. it is inspirational and energizing.

so why am i going to the conferences like these? in addition to the energy boost i mentioned above, there is also the trivia of learning about tools, projects, approaches; getting the feel for the zeitgeist, where things are heading, what folks are thinking. deep down inside i am always looking for the “blow your mind” experience, something that can turn a familiar topic on its head, something that can make me discover things i have never suspected existed.

GoRuCo had a nice balance – things technical and detailed, and also approaches/techniques/principles. the reject conf at the end – a series of quick lightning talks – was an icing on the cake, stuffing you full of references, pointers, tips that you could take home and work through at your own pace.

Gregory Brown: Where is Ruby really heading?

more of a book report, talking about different versions of ruby out there. for anyone following the community, none of it was a surprise. no hard data either, just his personal anecdotal experience. this part of the talk was more suited to a short user group presentation.

there were a few nice tips and personal war stories related to moving between 1.8.6, 1.8.7 and 1.9.1, unicode, side-by-side installs and very basic crude techniques to code for different versions.

these are typical growing pains – everything from the compatibility issues between versions to the curse of the system-wide install that makes one go an extra mile in order to run different apps under different ruby versions on the same box (i always preferred the semi-structured self-contained java jdk installs and jars controlled by the classpath).

there were some props to jruby (it is a real distro, not a hack to reach out for when all else fails!), mentions of ffi in jruby that allowed gregory’s project to run on windows.

Eleanor McHugh: The Ruby Guide to *nix Plumbing

this could have been a great talk, but it seems like eleanor was really hungover, so instead it was a very disconnected series of ramblings on the general subject of unix and coding. very poor delivery, and at times plain embarrassing.

only towards the end she managed to find the message for the talk, which was “you can code against kernel internals using ruby, since it makes it much easier. do not automatically assume that you need to write in C for performance – try ruby first.”

she also highlighted ruby community’s respect for bare-metal – the ability to tinker, the taste for small simple tools that do the job well, and the affinity for unix. this is exactly what attracted me to the language in the first place.

there were a lot of references that at times seemed like name-dropping: ngnix, beej’s guide to network programming, beej’s guide to unix interprocess communication, ruby/dl, duby, event machine, c10k problem, advanced unix programming book

Dan Yoder: Resource-Oriented Architecture With Waves

dan briefly talked about his waves framework that tries to present a simple DSL around HTTP and resource representation in a REST fashion.

the meat of the talk that was interesting to me dealt with REST in general, its differences from MVC, resource-oriented architecture, self-describing data, returning links to other data inside of data to aid discoverability. he mentioned how a resource identifier (e.g. URL) should not specify representation (i.e. do not add .xml to specify that you need an XML document back), but rely on client’s Accept* headers to negotiate representation (caveat being that CDNs like Akamai do not currently care about these headers, so you will always get the same content).

there was also a mention of the fact that sometimes one can view HTTP protocol as something dealing with a distributed hash table (DHT) using get/put/delete operations (with post reserved for everything else).

RDF and freebase were also mentioned.

i have a strange relationship with REST: i get it on the technical level, i sort of get it on the architecture level, but it did not fully “click” yet, perhaps due to the fact that i do not have enough practical implementations under my belt. i should go through the restful webservice book again and play around.

good talk overall, but lacking concrete examples that could have brought things more into focus.

Jake Howerton: Into the Heart of Darkness: Rails Anti-Patterns

a bit of a disappointment, since the title promised so much. i usually really liked the anti-pattern talks, since you learn nothing when things work as expected – the real learning comes when things break, and you are forced to dig in and figure out why. plus these talks also give you an idea of applicability of certain techniques, which is really a required counterpart for all patterns to begin with (beware of dartboard-driven design).

instead we were treated to a very few amusing short code snippets, but no larger patterns in the sense of fowler’s refactoring book or even rails-specific patterns.

he did mention cucumber, reek and metric_fu and some general well-known testing techniques. i liked his term “flight check” for the smoke tests that run before deployment to prod, and a notion of sandbox test environment where mocks are replaced with real classes that do destructive things without impacting the real world (like sending emails).

i also like his term “irb-driven design” for something that was copy-pasted from the exploratory irb session into the production code.

there was also some treatise of legacy code, but i think dhh’s talk on the subject was much better.

overall it was a fun, light talk that was well-received. i blame the deceiving title and the lack of focus for initial feeling of disappointment; it felt like jake was simply talking about things he does and prefers to do during development, without specific overarching theme in mind.

Sandi Metz: SOLID Object Oriented Design

sandi stole the show; she set the level that none of the speakers matched and showed what it really means to have a solid, gripping presentation that is lucid, focused, well-prepared, and superbly delivered.

she talked about SOLID design principles: Single Responsibility, Open Closed, Liskov Substitution, Interface Segregation, Dependency Inversion which all boil down to managing dependencies.

she went through the fowler’s value of design argument, and then iteratively went through a refactoring example, invoking the SOLID principles along the way.

i think the most impressive for me was how well the whole talk came together, how the arguments were presented to support the refactorings, and how well it got into my head (almost uncanny, similar to the effect the best books in head first series have).

some of the principles she kept bringing up – using the rate of change as the indicator for splitting the functionality; refactor in small steps to let the design emerge – not because you know, but because you want to find the design; red-green-refactor; mock at the seam; only mock classes i own; those that change often should depend on those that change less often.

this once again brings up the importance of developing a language to discuss design (e.g. patterns) – i think this is a crucial step for every practitioner, when something intuitive and personal (e.g. a vague code smell) becomes something that you can articulate and communicate to others.

she also mentioned uncle bob, micronaut, steve freeman and nat pryce mock objects site

Benjamin Stein: Building Cross Platform Mobile Apps with Ruby & PhoneGap

the main promise of PhoneGap is quite compelling – build cross-platform apps on the phones using javascript while taking advantage of the native features (vibrate, storage, accelerometer, sound, gps, etc).

it was a fine presentation, with a story to tell and with a perspective that put all the low-level details in context. makes me itchy to get my hands on some phone development; i was also quite impressed with their adoption of latest standards (e.g. HTML5).

Yehuda Katz: From Rails to Rack: Making Rails 3 a Better Ruby Citizen

rails3 perspective from the horse’s mouth – quite detailed look at how rails is trying to be less opinionated in its choice of frameworks to work with, and how it exposes its internals for others to integrate with (orm, rack, js frameworks). some very interesting examples of design decisions and the overall future path of the framework.

this was a bit tedious, but quite informative talk.

Lightning Talks

i really like these, since most people do not have enough material for a long talk; in fact some of the main talks earlier should probably have been half their size.

the highlights include sunlight foundation and data.gov plug, a great pair of fast talks by aman gupta on google-perftools that he tweaked to work with ruby and used on some real-world code and joe damato on tweaking the thread performance in ruby 1.8 (in retrospect, these two talks were something i wish eleanor would have done).

i really enjoyed this talk by michael “map reduce is crap” stonebraker. it is a terse and mind-blowing romp through the current database landscape, focusing on inability of one-size-fits-all “dinosaurs” to deal with streaming data (like tick market data) as well as data warehousing. he has some benchmark numbers and proposes some alternatives.

i have been looking into streaming databases and column data stores for a while, but I am yet to have a production hands-on experience with them.

i always had somewhat of an inferiority complex when it came to databases, since these guys have done so much work decades ago that tackled so many problems, and it seems to have had a hard time escaping the database world and affecting CS population at large.

so this new twist in the evolutionary spiral that comes due to the availability of cheap machines, lots of RAM, and the need to process lots of data should yield interesting results. i do hope that this will lead to a whole bunch of lean and mean specialized solutions instead of (loosely quoting stonebraker), “here’s oracle, what’s your problem?” approach.

perhaps it is my destiny to stumble into things that i should have known (and probably have known at some point, but they only became relevant now). in any case, i am happy to add drucker to my pantheon of striking bald men (already staffed with the likes of foucault, seth godin, and crowley).

i really enjoyed this updated/revised “best of” collection of drucker essays. he is not instantly quotable – i do not think he goes after the effect of catchy parables, so it is hard to pick out the pearls to quote here.

but of course, i cannot resist:

[...] the three stonecutters [...] were asked what they were doing. The first replied, “I am making a living.” The second kept on hammering while he said, “I am doing the best job of stonecutting in the entire county.” The third one looked up with a visionary gleam in his eyes and said, “I am building a cathedral.”

perhaps proverbs and fables are a retreat for those that cannot come up with an original wording themselves, but dammit, this is the kind of stuff that you hit people over the head with to get your idea across.

anyway, i really liked his writing on building innovation into the organization. his essays on the place of the organization in the society were also quite eye-opening, especially with respect to the non-profits. he does not go into heavy theorizing – everything is quite high-level and appears simplified, but there are a lot of good observations, especially once you read them in context. yes, some of it is a bit dated, but it is easy to make a quick mental adjustment to see the core of what he is getting at.

i’ll have to pick up a few more of his books; at this rate i will be reading f. w. taylor in no time. oh boy.

there is something i noticed about myself – oftentimes, when faced with a problem, i immediately jump to the technical means of solving it, which usually implies building some sort of automation. after that jump, the initial problem becomes less and less important, while i am speeding away, thinking through the technical details.

it turns out that sometimes an excel spreadsheet and a manual process is far superior solution, although it might hurt your ego and make you feel vaguely dirty.

another variation of the above is fascination with “cool” tech (for your definition of “cool”) – i noticed that sometimes i would use every opportunity to startup wireshark or Process Monitor and gloriously wallow in data, gleefully probing tiniest details. it turns out that almost always there are better tools out there that can get the job done with less noise.

so as a personal first-line defense, i opt for trying to ask the right questions early on – perhaps a manual solution would do, or a problem really is a non-issue.

it seems that too much knowledge may be a curse, but the real challenge is to keep your personal toolbox organized.

building software for a business, i established an interesting test that shows me whether i am thinking in the right direction:

off the top of your head name five most important things that technology can improve for the business as it is right now

i’ll wait – can you do it? right here, on the spot?

the way i see it – if, after having worked in a new team for a few months, you cannot do it – you have failed. you are not thinking about the business, you do not know the business, you do not care.

as technologists we are all too often drawn to the technical problems, forgetting that it is really the business that we should have in mind. the catch is that the domain of the software problems is infinite, so one can submerge forever without delivering anything of value to the business.

disclaimers apply, of course – it particularly matters for those developers working closely with the business. but then your definition of “business” might vary; the point is that sometimes a correct technical solution is the wrong one.

so while you are at it, type up this list and stick it somewhere visible, so that you can have it in front of you every day (as i’ve talked about it before).

Alex Miller aka Pure Danger Tech has a great entry on most common concurrency bugs.

the timing is perfect – this very thing bit me today. long story short, an old app, previously perceived to be multi-threaded, was recently converted to actually be multi-threaded, and then, once traffic ramped up a bit, exhibited peculiar behavior when perfectly good dates could not be parsed. thank god it blew up, as opposed to quietly corrupting the data.

so something as innocent-looking as private static final SimpleDateFormat declaration was the culprit: java.text.DateFormat is not thread-safe.

luckily, it is easy enough to spot and reproduce (threadPoolSize and invocationCount in TestNG simplify it even further).

a pessimist would heave a mighty sigh, once again swear on the copy of JCIP to find and root out every frivolous static out there with the help of FindBugs or a simple regex.

meanwhile, there is joda-time and a promise of jsr310

but of course this whole experience still leaves you feeling cheated and dirty – why, god, why?! something so function-like and stateless in nature insists on stowing things away.

a damn nice quote from Eric Brechner’s I. M. Wright "Hard Code" blog regarding team dynamics, specifically about retaining people:

[...] the best way to deal with turnover is to expect it and embrace it. How? Think flow, flow, floooooooow.

Think of your team as a river instead of a lake. A lake stagnates. There’s no energy or impetus to change. The same is true of groups that stagnate. They cultivate mediocrity and complacency; they abhor risk. A river is always running and changing with lots of great energy. You want a river.

a nice metaphor and it strikes a chord – building an ideal team is one thing, but perhaps it is more about building a self-sustaining culture when individuals might be coming and going. the latter seems to be a more important goal in the long run.

• decide to write a blog post
  • play with formatting
    • realize that there is a new version of a blogging tool
      • start upgrading blogging tool in test in vmware instance
        • realize that new vmware server is out
          • upgrade vmware
            • upgrade vmware tools (never easy!)
            • play with new vmware server options
        • realize that the new version of os is out
          • upgrade the os
        • notice that logrotate is not working correctly
          • fix log rotation
        • realize that rsync backups are b0rked
          • attempt to fix them, but fail
        • upgrade db while at it
        • upgrade web server while at it
      • screw around with formatting on the upgraded blog platform
        • fail to get pagination to work (like i fail every single time)
        • fix the theme since upgrade broke it
      • start upgrading prod and realize it does not have enough physical resources
        • consider switching hosting providers
          • really consider switching hosting providers
            • ...
              • ...
                • ...
                  • ...
                    • no, no, no! there lies madness!
                      • stack overflow!
• two days later write this blog post

• part 1

message-passing

if shared memory makes concurrent programming difficult, what else is there that an app developer can use?

one way of representing coarse-grained parallelism is through message-passing concurrency.

the idea is pretty simple – the only way to share state between isolated components is through message passing. what happens inside the components, is their own business.

there is no global state of the whole system, unlike in shared memory program that behaves like a giant state machine. sending and receiving of messages happens concurrently together with the computations by the components themselves. this approach is much easier for the developer to reason about and it maps easily to multiple CPUs and multiple machines.

a lot of the enterprise workflow-like processing falls into this model. it is basically a pipeline of worker pools, configured independently and possibly running on separate machines. a unit of work travels through the pipeline, as it is being handed off from one set of workers to another.

actors

one of the common implementations of message-passing concurrency is actor model. i’ll take a liberty to interpret this model to fit my own developer needs, even though i am butchering decades of academic research in the process.

actors model represents well multiple computers talking to each other using network packets, components exchanging messages using JMS, independent processes or threads talking to each other using messages – basically anything where isolation is possible and interactions are loosely coupled.

usually each actor has a mailbox associated with it (often represented with a queue), where messages are stored until an actor processes them. messages map well to physical artifacts in the real world – they are immutable, and only one actor can handle a given message at a time.

actors are connected with channels; individual actors are isolated from each other – a failure of an actor does not affect another actor. no other actor can change the inner state of a given actor – the only way to communicate is through message-passing.

messaging is usually asynchronous, but synchronous messaging could also be useful.

depending on implementation, beware of deadlocks if you are using synchronous messaging. another issue to keep in mind is order of messages – depending on implementation it might not be preserved.

while some advocate “everything is an actor” approach, and I get dizzy imagining the possibilities, the pragmatic app developer in me is living in the real world among existing apps. in this case actors work best as a library for the existing language.

erlang

although i shied away from “actors everywhere” approach above, erlang is the most successful implementation that actually does just that. it is not just the language, but a whole platform that transparently runs actors within a single process as well as across multiple machines.

as this topic is heating up, one should at least read the book and play with the language. after all, a language that doesn’t affect the way you think about programming is not worth knowing, and erlang is enough of a paradigm shift to kickstart your concurrency thinking.

Tibco BusinessWorks

as i’ve described before, BusinessWorks (BW) is an example of an integration DSL that happens to use actors.

given an integration process (e.g. receive a message on JMS queue A, enrich it from a database, transform it, and send it to a JMS topic B), you describe it using BW language constructs. then it becomes an actor definition that you can deploy on an engine (really a managed JVM instance). there could be multiple engines running on multiple machines, and each engine can have many process instances (aka actors in our terminology) running inside of it. a process instance gets created from a process definition whenever a new message arrives on a queue (mailbox in actors’ terminology).

a scheduler inside the individual engine takes care of creating process instances (there could be thousands) and scheduling them on the worker threads.

all of this mapping happens at deploy time, as a developer you do not worry about it.

actors talk to each other using message-passing, thus your actor implementation does not even have to worry about threads or concurrency – you just express your integration logic. you could use shared memory, but it would not scale well, since you are limited to one JVM; nor would it be natural, since you have to use explicit language constructs; this language support for immutability is very convenient, as i have mentioned earlier

if you use a JMS server to pass messages around, it becomes a sort of a mailbox, holding messages for you in the queue. each incoming message would eventually spawn an instance of the actor, feeding it the message as an argument. multiple instances of the same actor can read from the same queue, thus achieving load-balancing.

once you recall that jms supports -filters- selectors you have the actors implementation that curiously matches something like erlang

note that this is not fine-grained parallelism; your units of work are more coarse-grained and very loosely coupled, but fundamentally, the model is the same, and it scales like crazy achieving massive throughput.

even if you do not end up using BW, you can implement this model by hand relatively easy.

so what if i wanted more fine-grained and more efficient support for actors in my language of choice (provided i am not using erlang)?

ruby

revactor networking library includes actors implementation (also see this great intro to actors by Tony Arciery), but i have not seen a more generic approach yet.

note that ruby is really hampered by lack of proper threading support; this is why jruby guys are in a much better shape if they were to roll their own actors implementation.

scala

this is probably the most mature implementation i’ve seen (see this paper). they take advantage of scala language features to simplify the syntax and unify synchronous and asynchronous message-passing. individual actors are represented as threads or more light-weight primitives that get scheduled to run on threads in the thread pool. it is type-safe, but it relies on convention to make sure you do not mutate your messages.

although i could see where representing actors as threads could be too heavyweight for some tasks, in the case of java and scala, your mileage may vary (see this presentation from Paul Tyme).

groovy

given language features like closures and general simpler syntax, together with the fact that it sits on top of JDK that includes java.util.concurrent, one would imagine that groovy would be a perfect candidate for actors implementation. however, the only thing i found so far was groovy actors, and it seems to have been dormant for a while.

python

i do not know enough about python’s memory model and its implementation, but i suspect is suffers from the same “feature” as ruby – i.e. global interpreter lock, which means that it won’t be able to scale to multiple CPUs (and, similar to ruby, jython that builds on JVM comes to the rescue).

the only thing i’ve looked at so far is stackless python, which is a modified version of python that makes concurrency easier (see this tutorial by Grant Olson that also includes actors). it introduces tasklets aka fibers, channels, and a scheduler among other things.

java

this is where i am a bit surprised – i do not see a good drop-in-a-jar-and-go actors library blessed and used by all. there seems to be some research projects out there, but i want something that works for me now and supports in-memory zero-copy message passing, sync/async messaging, and type safety. i am OK with abiding by conventions instead of compiler checking things for me.

i suspect that the reason for this is the fact that some rudimentary form of actors can be implemented relatively easy using existing concurrency libraries, and this approach is intuitive without putting labels on it.

nevertheless, this is what i found:

• jetlang is a port of a .NET library and looks at Scala actors for inspiration. it is still quite beta, but it looks promising
• kilim (from one of the principle engineers of weblogic server) still seems to be a bit too much of a research project for my taste, but the theory behind it is sound

and there is a number of research projects out there:

• SALSA
• functional java contains actors implementation; see this blog entry as well as this one for examples
• Communicating Sequential Processes for Java

bottom line

actors is a great abstraction, and “good enough” version of it is easy to implement – think about it, consider it, use it!

it helps if your language/platform supports concurrency primitives to build upon. this includes true threading support that scales to many CPUs, although we could also benefit from a standard fibers implementation, since they are more lightweight than typical threads and would allow creation of a large number of actors that later could be mapped onto threads for execution.

each language could benefit from a well thought-out actors library, since it would push developers in the right direction.

it is not right for everything though – it might not be fine-grained enough, it might not map well to problems that rely on ordering of messages or presence of any other state across multiple actors or multiple messages.

to be continued

what is on the horizon that is worth noting? what are some of the interesting research topics? what have we forgotten over the years? what other heuristics/patterns and libraries could be immediately useful?

• part 1

true to the purpose of this blog, below is an attempt to organize my (admittedly very superficial) experience with concurrency.

• part 2

my 10GHz CPU

you probably noticed that moore’s law does not really apply anymore when it comes to CPU speed. if it were holding up, we would have had 10GHz CPUs by now, but for half a decade we haven’t really moved past 3GHz.

that is to be expected for the current generation of hardware – the gains have to happen elsewhere. for a little while we’ll get performance boost due to increase in size and speed of the caches that would improve locality, but in the long run it seems that multiple CPUs is where the improvements are to be mined from (this also includes specialized CPUs like Cell and GPUs in general).

this means that more and more people will have to think about their applications in terms of parallel processing. this also means that optimizations will become more and more important for those workloads that cannot be parallelized and therefore will be stuck on a single CPU (for a good introduction see The Free Lunch Is Over at Dr. Dobb’s Journal).

the bottom line is that as an app developer you cannot ignore the problem any longer; to make matter worse, there is no automagical solution in the nearest future that would make your application take advantage of multiple processors.

my concurrency story

in past decade most of the stuff i’ve worked with had some sort of coarse-grained parallelism; the rest was taken care of by the underlying framework.

i started with a unix philosophy of small programs connected via pipes, each performing a simple task. a little later came in fork and signals. things were simple, and OS took care of everything.

then came the web – it was mostly stateless with the database doing all the heavy lifting when it came to shared state. we just added boxes if we needed to grow. in ETL multi-box, multi-cpu setup was also natural, and the tools were designed to conceal concurrency; same goes for integration, where concurrency was at the level of data flows, which made things rather simple.

it is only in the past year or so when i had to really dive in deeper into relatively low-level concurrent development with java.

my dog-eared copy of Java Concurrency in Practice has proved to be quite an indispensable reference. the book is a bit uneven, and the editor should have spent more time on it, but you get used to it. it is a great practical resource, especially in the presence of so much confusing and incomplete information online.

jsr-166 introduced in java5 (and the primary subject of the book) is such a productivity boost; being a part of JDK, it is a big step forward towards letting mere mortals like me really embrace concurrent programming.

i find myself using Executors convenience methods all the time: it is so easy to create a pool, and then just feed it Callable instances, getting a Future instance as a handle in return. if more flexibility is needed, i use ThreadPoolExecutor. Queues are great as a communication channel for any sort of producer/consumer scenario, anything that requires message-passing or any sort of other work hand-off. Atomics are also great – i do not have to think twice when implementing counters or any other simple data structures.

most of the time i do not even have to work with threads or low-level synchronization primitives directly – they are buried deep within the libraries. i have less nightmares, since i do not have to touch volatile as often.

at some point i’ve read both editions of doug lea’s book, but i was always hesitant to recommend it; i’d rather rely on libraries that abstracted all of this away. now that java.util.concurrent has been out for 4 years, and Java Concurrency in Practice has become a bestseller, there are no more excuses.

one thing i’ve learned though – when you think you got this stuff, you discover a whole new class of problems that make you realize how complicated all of this really is, and how truly difficult it is to write larger concurrent programs.

you really, really have to think hard about how you share your objects, how you compose them and operate on them. you need to really understand how the language and the runtime work (i find myself checking JLS quite often). this is where good OO practices like encapsulation become even more important, since you are not just risking maintenance overhead, but you are risking the correctness of your program.

now, i always told myself that programming is not an exercise in manliness. i am just an app developer; i want to ship a working code that solves customer’s problems, not spend countless hours trying to reason through non-blocking algorithms just because i decided to do something non-trivial with ConcurrentHashMap. at the same time i do not want to waste my precious CPUs, so what am i to do? shouldn’t this stuff be easier? is there something I am missing?

threads considered harmful

actually, there is no problem with threads per se; the problem is with shared state.

in a normal sequential program you only worry about the logic as it is unfolding before you – one statement after another, in order. in a concurrent program that uses threads and shared state in addition to all your usual problems you also have problem of the non-deterministic state: since at any point in time any thread can come in and mess with your data, even between the operations you considered atomic before (like counter++), the number of states that your program can be in suffers a combinatorial explosion. this makes it really hard to reason about its correctness.

your code becomes brittle, sacrificing failure isolation – one misbehaving thread can potentially harm the whole runtime (a good analogy is BSOD caused by a device driver).

in addition, things don’t compose – a transfer operation performed by a thread-safe customer between two thread-safe accounts is not going to be automatically thread-safe.

to make matter worse, some of the errors remain hidden when run on commodity 1-2 CPU IA32 hardware, but as the number of CPUs grow, or their architecture becomes less restrictive to help with concurrency, things start to break down.

for more thorough discussion see The Problem With Threads by Edward A. Lee and Cliff Click’s We Don’t Know How To Program…

now what?

a natural reaction is to forget about fine-grained parallelism and offload the hard stuff onto someone else. after all, i am an app programmer, i care about business problems, what’s all of this yak shaving about?!

in some cases we can get away with firing up individual processes to take advantage of multiple CPUs. most of the time though it means that the problem has been pushed further down the stack, which often turns out to be the database. this is the route that rails folks went, and it certainly was pragmatic approach at the time (now that they are forced to deal with efficiency, threading is back in the picture. for discussion of issues see Q/A: What Thread-safe Rails Means).

if you can get away with using individual processes, go for it (see google chrome) – you get failure isolation, you get immutability in respect to other processes (it won’t be as easy for another process to mess with your data), and as an additional benefit, you get to use all the standard tools that the OS has when it comes to managing and troubleshooting processes (as opposed to using often incomplete and idiosyncratic tools for thread management that your runtime platform of choice offers – if any).

still, as we need more and more fine-grained concurrency and as the level of concurrency increases (it is not just a handful of CPUs now, but dozens, and even hundreds), one process per task becomes too expensive (context switching, high costs of creating a new process, memory overhead, etc). so we are back to some sort of lightweight thread-like primitives running within the same process, sharing some common resources.

most of the popular languages/platforms these days provide some sort of threading and shared memory support. but as outlined above, they suffer from some fundamental problems. there are some practical things at various levels of abstractions that that can help: low-level constructs within the language/platform itself, tooling, and higher-level libraries/mini-languages

language

• make immutability easier – take note of functional languages, but also make it practical. in java case, for instance, it could mean extending immutability to some core data structures (see scala collections) or making it easier to tag an instance as immutable (see ruby’s freeze; this reeks of boilerplate though) – this way errors will be caught at compile time
• consider sharing data only through explicit, ideally checked at compile-time, means. thus by default nothing is shared, and in order to make something shared you have to explicitly tag it as such. ideally, this would also come with some sort of namespace support, thus limiting mutations to a sandbox (see clojure for reference)
• make language safer to use when it comes to exposing shareable state (this is when something like static becomes a problem – see Shared Data Considered Harmful for an example that applies to concurrency)

tooling

• static analysis tools might help, but we need to give them a bit more than just an infinite number of states. findbugs for instance, supports concurrency annotations and something like chord could also be promising. this stuff is complex though and there are limits to static analysis (and i do not even want to bring up formal proofs using process calculi)
• i want more support from the platform to help me troubleshoot lock contention, deadlocks, cpu-intensive threads, and other concurrency-related infrastructure. sun’s hotspot has some rudimentary stuff in place, but i want more things out of the box (azul claims that they have always-on built-in tools in their product, but i have not played with them)
• speaking of azul, i need to study them more. although perceived as a boutique solution, they are addressing issues that everyone will be facing in just a few years. seems like they ported sun’s hotspot to their hardware which allowed them to achieve scaling by automatically replacing synchronization with optimistic concurrency which scales much better. incidentally, this truism about optimistic concurrency has been obvious to database folks for decades

libraries/mini-languages

one of the approaches is to focus on your problem domain and come up with a library/language that solves your particular problem and abstracts away concurrency. web frameworks (J2EE, rails), or ETL tools, or even databases are all examples of such approaches.

this is where my interest lies as an app developer – how can i make concurrent programming easier for me, the layman.

the bottom line is that if we insist on using low-level synchronization primitives, it would be really hard to paper over the underlying complexities. right now there is no generic universal approach that will simplify concurrent programming. so at this point a pragmatic programmer is left with patterns, supporting libraries, and heuristics.

to be continued

there are some patterns (for the lack of a better word) that i found to be helpful in dealing with concurrency; there is also some stuff on the horizon that promises all sorts of benefits – is there really a silver bullet? but also there is plenty of stuff that has been with us for decades, and i would be the first one to bow my head in shame, acknowledging my ignorance.

• part 2

this is a bit of a wandering post, but i know i think best through writing things out, and while personal journals are a great thing, i cannot force myself to write them if there is no (even hypothetical) audience.

i think as i get older, i realize that there is so much stuff out there that i am interested in, that the importance of focusing is becoming increasingly apparent.

the distractions are getting more targeted and more fine-grained – it is getting simpler to fill up all the cracks of time between activities – reddit, twitter, google reader – scan the headlines while you are waiting for the build, and boom – you’ve blown your stack and lost the context. this is when discipline often loses to muscle memory; i suppose it is the usual story of combating the addiction. all this technology i surround myself with to be more effective just makes it simpler to waste time without achieving anything.

in this sense (all other things being equal), i find pair programming to be very helpful – it is so much harder to get distracted with someone else sitting next to you.

in certain cases i’ve observed that the proverbial order of magnitude difference in productivity between individual developers can simply be attributed to the ability to focus.

this became even more apparent to me in the past year at work as i have been faced with thousands of stored procedures for a given system – some of these storprocs being 2K+ lines long and recursive(!). talking about a new level of immersion!

one behavior i am guilty of is perfectionism in the face of overarching bigger goals – trying to make something flawless all the while the solution has to be out now and has to be good enough. i think i became immune to rolling out frameworks when a quick point solution is needed, so now the battle has moved to a finer level – testing as risk management, for instance – in the absence of time i test the stuff that is most likely to contain bugs, and i test coarse-grained user-level functionality first instead of polishing up individual unit tests.

but really, all this “micro focusing” business above is not that bad and is quite manageable on day to day basis. what i am trying to figure out though is how it applies to my career in general.

my problem right now is that after a decade in IT i realize that i have some hands-on skills that would always be in demand, and that there always will be plenty of filler work that will keep me busy and provide some sort of satisfaction (the kind similar to mindless exhaustion you feel after a workout).

i know i tend to become absorbed in day to day stuff, and lose the sight of everything else. NYC is dangerous in this respect – the pace and the energy make you work harder, and one can get fixated on something with a lot more intensity. this is when i find myself performing all sorts of yak shaving feats; is it really what i should be spending my time on?

i think paul graham got it right in his essay – NYC has the drive and the ambition, and i feed on it, but i also need to be fully cognizant of its true target. doing something pointless twice as hard isn’t something i am after.

so what is it that i should be doing? so let’s see if i can talk through this.

i always took pride in knowing a great deal about a lot of technical things all the while aspiring to grasp “the bigger picture.” i enjoy being able to build things, drive them to completion, and deliver. i also know that i like dealing with people, solve organizational problems, motivate and build teams.

i really enjoy doing hands-on stuff, and i am pretty good at it; at the same time i’ve seen all too often that most of the technology problems stem from people problems. it pains me to be on the receiving end of business blunders that render my technical efforts useless.

one of the trends in large companies is increasing specialization on many levels of IT. it is inevitable to a degree, but this is precisely where my skills would be least utilized. the image that haunts me is the nine to five drone with atrophied knowledge of everything but the immediate responsibilities of everyday job, unable to survive outside of the large corporation. there are plenty of slots like that – just waiting to be filled.

i want to work in the environment where people around me do have a vision that we all share, and this is what drives our everyday work.

i also like small teams (or small companies) – you get to do a lot of different stuff, you learn much faster, and you feel the impact you make.

so ideally it comes down to focusing on my strengths and consciously building upon them, which means spending less time on everything else.

it sounds quite absolute and final (an also quite fluffy), so perhaps a better approach is to set a goal, timebox it, and focus. if it works – great, and if not – review and start again.

• current work: spend less time on it; spend less time on filler work; focus on business – actually read the pile of books i have here instead of reading up on tech stuff.

• spend more time on hobby projects, get back into languages/platforms i abandoned, pick a few projects and actually contribute; build the brand – push for talks at user groups, use real name in forums, blog more on technical topics.

as far as career choices go:

• i’ve done the enterprise architecture thing for a couple of years before, and despite the raging pundits that the title has accumulated, i think there are great things to be done. you do not have to abandon the technical skills, but you have to realize that your success does not depend on tech anymore. you do end up building and running systems in this role, but they also include people, culture, and organization – not just technology. there is a lot to be said about systems thinking, and there is plenty more reading i need to do. i am also yet to find a good no-bullshit community that thinks along these lines (so far i’ve been really enjoying Stu Charlton’s blog for instance) and people i can work with and learn from…

• ...which leads me to consulting. what appeals to me here is focus on solving a particular problem (compare that to a less focused role of a regular employee), as well as a great variety of problems, and generally a higher level of expertise expected. you grow and learn so much faster, and ideally this might lead to a product idea, if you see the same problem repeated over and over again. one skill that i know i have is what jerry weinberg calls “jiggling” (see Secrets of Consulting and Are Your Lights On?) – helping people get “unstuck” by asking questions, suggesting tools, practices. often i realize that in many teams although i could do all the work myself, the team would get so much more out of me if i create an environment where people can learn and grow on their own.

• another option is a product/program owner in a larger company. the biggest selling point is the diverse set of skills required and the sense of ownership. you have a bigger picture, you are responsible for it, and at the same time you need tech, people, and organization to be a success.

• the refinement of the above is a startup – you build and deliver a product. your impact is tangible, you actually “own” something, and you contribute on many levels, which is a great use for a broad set of skills.

• and the last option is to get a foot in the door in a larger company as a regular developer (since this is an easy lowest common denominator that is versatile enough and faceless enough) and then try and grow into some sort of a role similar to one of the above. it is terribly inefficient though – most of the energy will be spent on the wrong things.

it does help to write all of this out – having a stake in the ground is a start. let’s see how it goes from here.

my work-related reading started with programming books and reference materials. then, as i was becoming more aware of things around me, i started reading up on development process, requirements gathering, and all this other “softer” stuff.

when it comes to applying the knowledge, the latter was more tricky – perhaps because it was not just up to me when and where to implement it – other people had to be involved.

with non-technical books it is all too easy to kick back and follow along as if reading an adventure novel – this stuff is engaging, and one might dream of making bold decisions, only if one were in the position of power – all of it from the safety of one’s couch.

in my case i realized that i had to consciously make a mental effort to move from mere recognition and acknowledgment to tangible steps – what can i do tomorrow, in my particular environment? otherwise it will just sit in the back of my mind, and i will mumble behind peoples’ backs recognizing yet another opportunity missed.

this seth godin’s blog entry really helped to illuminate this problem for me – the ideas in a book are just a short bullet list, the rest is persuading the reader to act. so i will follow his advice and pick a book from my recent reading to see if i can get something tangible out of it:

five dysfunctions of a team – is a light read, just a few simple points presented first as a novel, then discussed one by one. but then after reading it, as it was fresh in my mind, i gradually noticed how nicely they applied to the environment around me.

true to its title (admittedly catchy), the book goes over five qualities that are necessary for creating an effective team; they build upon each other, so i’ll pick the first three, since they are the foundation for the rest:

Trust

in the team-building context trust is knowing that all the team members have team priorities first in their agenda. it is not being afraid of showing ones’ vulnerabilities, knowing that others won’t take advantage of that. it is knowing that your peers’ intentions are good, and there is no reason to be protective or careful around the group.

trust makes delegation possible – which allows the team to scale and be more effective, as work gets distributed around with everyone aware of individual strengths and weaknesses within the team. the opposite of that is everyone working in their little cubbyhole and reporting up.

finally, trust makes risk taking possible.

i think i consciously made an effort a few years ago to make sure i say, “i do not know” more – this simple acknowledgment goes a long way towards removing barriers within the team. it makes offering help and asking for help possible (which is good telltale sign in itself).

another conscious effort was to apologize more – it works towards defusing the tension and focusing on the problem. obviously, one has to be somewhat self-aware for that to work, and luckily i had a misfortune of witnessing some monstrous examples of personal communication deficiencies, so i have learned from others’ mistakes as well as my own.

team identity

in school we are encouraged to focus on individual goals, so it takes a conscious effort to change that habit for the good of the team, and team identity is one of the powerful aids. one has to consciously build it up – starting from the name of the team, clear statement of short-term and long-term goals, the vision that provides perspective when you are saddled with everyday tasks, acknowledging successes and being proud of them, sharing the problems and treating them as team’s responsibility. there are plenty of smaller things – everything from eating lunches together to casual chit-chat.

secrets and politics within the team kill trust, so my personal resolution is to be more open – have no reservations about telling others what i am working on and why. i noticed that i do hold back sometimes, not sure of others’ intent, and this is definitely a problem.

i also need to give more benefit of the doubt before arriving to conclusions regarding others’ motivations, i think sometimes i am too eager to assume ill intent, and this obviously inhibits trust.

feedback

this is the next step, it might come naturally as a result of working together, but it helps to acknowledge and strive for it consciously.

the good thing is that large companies invest heavily in HR review processes. the sad thing is that for the bad manager these twice a year reviews make the situation even worse, since they come across as stilted corny HR drone-induced formalities, so anything that resembles them (like actual good feedback) is squashed even further. as for the good manager, they do not need these, since they do it on regular basis anyway.

so my personal resolution is to give more feedback to my peers (which i think i am getting better at – noticing individual achievements, taking the time to stop by and comment on something casually, etc), but also giving feedback to my management, which i am not that good at.

i think i also need to get better at offering help – taking into account the personalities of others and trying to tailor my offer to that.

Conflict

productive conflict is essential. if it is avoided, important issues would not get resolved, which will be even worse in the long run. so one must recognize its importance and embrace it, at the same time remembering that it should not be personal.

needless to say, without trust productive conflict is impossible.

i think this is where i could use a lot of work. at some point i became all too aware of others’, so i tend to come across as non-confrontational more and more. i often tend to avoid conflict in the misguided attempt of maintaining everyone’s comfort.

another problem is “picking your fights” – there are always plenty of problems, and one cannot tend to them all. so you either spend your time on all of them and accomplish nothing, or give up and let it all slide.

here’s a common scenario i am guilty of – something is wrong, something is not working, i try a few times to fix it, but it is not my responsibility, although it hurts me as much as the others. so i grumble half-heartedly around the people responsible, but never challenge them openly.

a natural reaction is to ignore it, but that leads to the “broken windows” syndrome (see pragmatic programmer), which, once it gains momentum within the team, is hard to reverse.

it also reinforces the silos – this is my own perch, i tend to it, and others’ should not stick their heads in. ideally, a problem will be out in the open, the whole team would regard it as their problem.

so instead of creating a permanent state of team guilt, perhaps these issues could be put out in the open and prioritized, so that at least these mistakes won’t be made again.

i also primarily worked in the teams where i knew that we could always solve problems ourselves, there was no need to reach out to “higher authority.” it turns out that in large organizations it is sometimes necessary to ask for arbitrage from above to get stuff done. although it pains me to do this, since it fundamentally goes against what i think a leader should be doing, i have to acknowledge the reality and focus on the problem that needs to be solved.

Accountability

ideally i want and environment where my teammates hold each other responsible for the high standard of work they produce as a team. if someone goofs up, others should be able to step in and challenge them.

here’s an example. given a multitude of assorted projects with various infrastructure problems, one is tempted to create an architectural police of sorts that goes through the checklist – proper logs are written and rolled, monitoring in place, DR in place, etc. while this could be a valid first step in some situations, in a perfect world the team itself will know what these standards are (perhaps since they are wikified) and their personal pride and identity as a team makes them follow these standards and make sure other teammates follow them as well.

it comes natural in a healthy teams, but in dysfunctional teams its absence is painfully obvious: everyone does just the bare minimum to get by, and only certain people in complete isolation try to achieve their own high standards.

peer pressure is a powerful thing – poor performers feel the pressure to improve; it promotes respect, as opposed to resentment between team members with different standards.

of course it builds on the previous principles – it is impossible without trust to give feedback, it is impossible to challenge people with fear of conflict.

so my personal takeaway, going back to fear of conflict, is to challenge people and speak up; i should collect and promote best practices; i should be doing more presentations and knowledge sharing.

it is a gradual process, and with the absence of slack this grass-roots movement is very hard, so at some point there needs to be some management support.

it also helps to hold project retrospectives, because if we lack self-awareness and succumb to “broken windows”, we will make the same mistakes over and over again and take it as a norm. i am guilty of not pushing for retrospectives as much as i could. perhaps as a start i can informally round up a few teammates and go through a few exercises suggested by the “agile retrospectives” book.

why am i so adamant on this being a peer effort? leaders should not be the ones creating accountability, in fact it can be counterproductive – the team will delegate the responsibility to the leader and as a result either become even more passive, or worse, turn against it, seeing it as unnecessarily imposed upon them.

there is a lot more practical stuff in the book, and skimming through it again makes me realize that i’ve missed quite a few tips, but let’s see if i can get some of these practical things done first.

i noticed an interesting phenomenon – when talking to a few recent grads or those that are about to graduate, i realized that a lot of the smarter ones are disillusioned about programming.

i find it interesting, because i went through the same transition. having done some pretty cool practical stuff at school, i got quickly bored at my first job, thinking that there was not much new and interesting stuff for me to do.

at one point i recall flipping through the Pragmatic Programmer book right after it came out and dismissing it as something that was too obvious to pay attention to.

i view it as a key telling sign now. this was my own attitude at the time, and i see it again and again in new hires fresh out of college.

being a smart kid, you (perhaps subconsciously) try to assert your own superiority, and perhaps being somewhat defensive, you are all too quick to nod in agreement and move on the moment you think you grasped something.

it turns out that when i go back and re-read some of the books i thought were full of obvious stuff, i discover something that i have learned much later through my own mistakes.

maybe this is the cursed fate of these books – you either read them when it is too late and you wish you read them earlier, or you read them too early and you wonder what all the fuss is about.

what i personally found helpful in this case was the “beginner’s mind” stance – embrace the fact that you know nothing; be humble and view everything that comes your way as a learning opportunity.

which brings me to an opposite learning dysfunction. my first year of university had cryptography, compiler theory, computer languages, two semesters of discrete math, data structures and algorithms, theory of computation, etc, etc (and this was just the CS stuff). it was simply too much for my little brain, so i employed FIFO / LRU eviction approach – without practical application these concepts were promptly forgotten after i passed the exams.

only years later i painfully learned that all of that stuff was actually relevant and important and used in practice. i went back to grad school to patch up the fundamentals and re-learn all of it again at my own pace.

this time around, having done practical stuff, and having established a much broader picture of the CS field, i could finally see how it all connected, and how this knowledge was used in practice.

maybe all of the above is irrelevant to everyone else and could be attributed to my own learning disabilities. however, having worked in corporate IT for past 10+ years, i think there are lessons to be learned for everyone, since those that come out of school have to spend their first year or two learning on the job.

most schools teach computer science which is not the same thing as software engineering. the problem is that the industry needs more software engineers – those that can build everyday quality software and make sure it runs and actually does what the business wants it to do.

sometimes the stuff that you learn in school is directly counterproductive – you work alone and get your own grade; most of the time your work is over once a particular problem or concept is implemented; no one evaluates the quality of design, no one cares whether your stuff is maintainable, or uses version control, or has repeatable builds, etc.

it is up to organizations to make sure that new hires get trained – a mentor/buddy, a stack of books, a pile of wiki entries to get started. it is an investment that requires time and management support.

as for the new hire – be a professional in your field, strive to join the 5% and have fun doing it. otherwise you will be one of those 9 to 5 folks cracking “it is case of mondays” jokes and doing the same thing year after year (well, maybe not, but this is my own personal hell scenario). the field is changing so fast, that you have to keep learning to stay relevant. think of it as climbing the escalator that is going down – the moment you stop climbing, you are falling behind.

there is so much cool stuff out there that is IT-related, and each year it is only getting more interesting. what we need right now is more good engineers with business aptitude to bring in a much needed rigor and professionalism to the field of business software. there is a lot of stuff to do, and hopefully we’ll be able to communicate it to the newcomers.

i have been wrestling with some legacy code recently. beating it over the head with a copy of Working Effectively with Legacy Code did not do much, so i started writing tests to see how it worked.

after a few minutes of waiting for eclipse + junit4.3.1¹ combo to load, i furiously coded a bunch of tests, and then realized that i could not make them fail. essentially it boiled down to the following:

assertEquals(1.0, 1.1);

...which quietly and happily passes. wtf?!! these are two doubles, just compare them and let’s move on with our lives!

then after a bit of thinking i recalled that most of my tests that involved doubles were written on projects that were still on jdk < 1.5, when method above simply won’t even compile, alerting me to the fact that junit expects assertEquals(double, double, delta), where delta is the precision you need.

why they couldn’t simplify my life by creating a couple of Double instances and calling equals() on them is beyond me. this is what i would want most of the time anyway.

since i was running tests under jdk1.5, autoboxing kicked in and now we have two Doubles on our hands. fine, this should not be a big deal, simply call double1.equals(double2) and be done with it – what’s a big deal?

but nooooooooo, check out this little bundle of joy:

private static boolean isEquals(Object expected, Object actual) {
    if (expected instanceof Number && actual instanceof Number)
        return ((Number) expected).longValue() == ((Number) actual).longValue();
    return expected.equals(actual);
}

what the hell?!! you correctly detect that this instance of Double is an instance of Numeric, then take its long value and throw fractional part out. quietly. so all my tests never even squeak. why?!!!

an obvious approach is to suck it up and appease the api:

assertEquals(1.0, 1.1, 0);

this will work. but damn! my eyes!!

TestNG

we all know that junit is legacy, and version 4 was an afterthought, so let’s see what testng does. the api is the same, and under jdk1.5 my primitives get autoboxed into Doubles and assertEquals(Object, Object) gets called:

public static void assertEquals(Object actual, Object expected, String message) {
    if (expected == null && actual == null)
        return;
    if (expected != null && expected.equals(actual)) {
        return;
    } else {
        failNotEquals(actual, expected, message);
        return;
    }
}

voila! this is exactly what i expected. and guess what – it actually works and correctly fails the original test.

Lessons

• man, this alone would scare me away from junit in favor of testng
• make sure your test fails before it ever works!
• what really frightens me is how many more of these quiet autoboxing errors are lurking out there. what previously would be caught by the compiler now silently works in unpredictable ways
• yet another thing to be aware of when upgrading your app from jdk1.[34] to jdk1.5+

Note

this junit bug has been fixed in junit4.4 and up

Ruby

interestingly enough, on a few skunk works projects this year i cobbled together a bunch of existing libraries with ruby glue and used (j)ruby’s Test::Unit (which conveniently comes with ruby distro) and rspec for testing the result.

it definitely makes sense for a project that is written in (j)ruby that uses many existing java libraries; it might be a bit of a mindset shift for a java project that is looking for simplified testing. for now i am keeping an eye on projects like JtestR

¹ default version of junit that ships with eclipse 3.3