qcon2009sf: other talks

Posted by anton
on Monday, February 01, 2010

for completeness sake, in addition to outstanding talks, i should mention a few others that were memorable.

i caught the second half of eric “DDD” evans talk – it was moving at a glacial pace (a trait of a professional consultant that is paid by the hour?) and was terribly overcrowded, but i liked what i heard. he took the roundabout path to get to the importance of evolution of the existing systems, but he made his point well with some great insights along the way. i have been setting aside his book for years, and i know it is long overdue, but i should read it.

i was surprised that none of the talks on the “cool stuff with java” track appeared all that cool to me. perhaps i have already looked at the tech behind them too closely, and the talks did not offer much on top of that.

nevertheless, project voldemort talk was a good refresher on brewer’s CAP theorem, consistent hashing (or DHTs), amazon’s "dynamo". i wish it were more technical, since the devil is in the details, and i wish there was more comparisons to other systems in this landscape (e.g. cassandra which seems to be further ahead).

hadoop is pretty much a household name these days, so a generic intro to the tech and its ecosystem did not do much for me. still, having built and run ETL environments in the past, i think that anyone in this position must consider hadoop these days – after all, if your ETL environment is trying to scale vertically and is using one of the big boys’ mammoth software, then it will make sense to compare 1MM for the license + hardware costs with 100K for commodity hardware + free software. this is a subject for a whole different post though.

by the end of the conference i have acquired a nervous twitch each time i heard another attempt at cloud computing definition. i did like stu charlton’s talk though – he had a good business perspective, and although i do not believe in his product, i would be keeping an eye on it, since they are trying to solve a real problem.

finally, i was disappointed by my own reception of “architectures you’ve always wondered about” track. facebook, linkedin, amazon – i think i’ve already knew enough about them, so signal to noise ratio in these talks was too low for me, and i could not justify staying there for long.

qcon2009sf: individual talks

Posted by anton
on Monday, February 01, 2010

as promised, these are a few talks that i have attended and found worth mentioning. i know i missed a lot due to scheduling conflicts, but that’s the nature of the game.

clojure talk by stu halloway

a great talk that had to be witnessed; it was a fast-paced flight through the language, illuminating its features and defining its place. there was a story, there was excitement, and there was a pragmatic take on it all. since languages track was aimed at actual usage in the field, half of the talk was spent on war stories – things that worked and things that did not. it was a tight, erudite talk, with just the right amount of details.

i am still ambivalent about the language – i do not have enough experience with it to see it used in production on the systems i am working on right now; it still exhibits growing pains, and feels a little rough around the edges. perhaps it is my lack of functional language exposure that shows. at the same time i am absolutely fascinated and excited about the things that it gets right – the immutable data structures and the whole concurrency story. the language feels nimble, finely honed, and it is great to witness its evolution, as it happens in front of my eyes. i love the way it makes my brain feel, the way it challenges my perspective on language design and features.

if tech is your competitive advantage, and you have small sharp teams, then by all means, give it a try. even if the language does not survive in its current state, the ideas and their implementation will live on – i think they are that important.

groovy on the trading desk by jonathan felch

there is always a bit of a stigma associated in my mind with conference “thought leaders” – unless they have proven their credibility by repeatedly building and shipping, i always take their words with a grain of salt. after all, those that have the time to float from conference to conference, from client to client, might (d)evolve into pundits. there is a definite value in that, and i certainly would still attend their talks and buy their books, but i would always remind myself of their perspective.

i really liked jonathan’s talk because it was ruthlessly pragmatic, coming from someone driven to ship, working with traders to solve real problems quick. in a sense, this environment is somewhat of an aberration, since outside of the trading desk the software development world is quite different.

his team was bending technology, doing very creative things right on the bleeding edge; by any means necessary they had to deliver software for the business under the tightest deadlines. in some respects, this is the technologist’s ultimate dream – when politics and money and all the other pressures of architecture and enterprise world are pushed aside and all that matters is whether you can deliver. this freedom is scary and exciting at the same time.

the talk was about a distributed computational in-memory engine they have built to support a trading desk, standardizing on groovy and building on top of terracota in 2006. this allowed them to deliver new features in a matter of hours for their traders.

essentially it was a graph where cascading properties were recomputed in response to events; logic could be injected via closures at runtime, state and behavior of individual nodes could be extended at runtime; all of it written in functional style, avoiding shared state.

he talked about the areas where groovy shined, and walked through some of the problems they have run into. predictably, the strength was writing DSLs, terseness, convenience, speed of writing code, dynamic nature, integration with other systems; pains were performance, math, and language gotchas.

jonathan also managed to give a perspective on quantitative finance in general – what is the business about, who are the people involved, what tech is used, and what problems they have to deal with. this is what really made the talk “sink in.”

i dismissed groovy early on, when the race to add new language features trumped the need for quality and thoroughness. the whole affair seemed to be too sloppy and haphazard, so i only watched its evolution from the distance. i suspect that the situation has changed, even before spring source acquisition, so i should give groovy another try.

architecture for the cloud by michael nygard

i am yet to read "release it", but i have come to appreciate his perspective based on the articles, blog entries, and book excerpts. i think michael has a great gift for organizing and presenting the patterns of operations architecture, a field that for the longest time has been the dirty secret of running systems; the proverbial “last mile” of software development, the achilles’ heel.

it takes someone straddling the fence between operations and developers to recognize the issues. having been in this role myself (and having ranted about it on this very blog), i am really grateful to him for illuminating and organizing the patterns in a manner that (hopefully) should help us as a community to avoid repeating the same mistakes.

why do patterns matter? it is a shared language that allows those responsible for operations and development to communicate with each other and recognize the problems and their solutions.

why does operations matter even more now? the proverbial admin/developer fence is disappearing, cloud means fast provisioning of many machines done by developers/users; the developers become more and more aware of infrastructure, which, in turn, becomes more of a commodity, while the job of the sysadmin is changing. i really liked his practical insights into the way cloud computing will be changing the roles within the enterprise.

overall, i mostly nodded along with the slides, but it helped me organize my own thoughts on the subject (which, in the end, might be far more important).

doug crockford on javascript

an immensely entertaining and sprawling talk that went into the history of the language, its evolution, and its future. it was exactly the kind of the talk i dismissed earlier, but at the end of the long day, and from someone as entertaining as doug turned out to be, i was ready to kick back and enjoy the ride. there were plenty of anecdotes and killer one-liners, so if you get a chance, watch him speak.

erik meijer on rx

as opposed to the doug’s talk mentioned above, this was a great example of a conversation, or riffing, where ola bini, don box, dan north, amanda laucher, sadek drobi, and a few others completely derailed the whole rx presentation into a joyous bantering about languages.

erik meijer is a joy to listen to as it is, but if you add a small responsive audience and a few beers, the whole experience is unforgettable. he loves to be paradoxical, and he revels in controversy. he was dropping tweetable gems at an astonishing rate – the sparks were flying, and my brain could hardly handle it.

at last we were pretty much forced out of the room by the staff – it was a perfect closing for the conference.

qcon2009sf

Posted by anton
on Thursday, December 03, 2009

it was my first time at qcon, and i found myself quite conflicted before the conference – i kept asking, what was the main reason for me to attend?

i already know roughly 75% of the speakers – i’ve read their books, i have been following their blogs and twitter feeds, i even saw their presentations online. i know for the most part what they would be telling the masses, thus i could not expect a blow-your-mind experience, the ultimate conference “catch” i am sure everyone is after. perhaps this is the curse and the blessing of the ultimate transparency.

if i am interested in this stuff, i have already taken all the easier steps to get up to speed; attending a conference comes next for me – not before, not instead.

as it turns out, once you get a dozen or so people that roughly have the same level of expertise and the same interests, and that do not expect to be spoonfed as they doze off, something else emerges – a conversation. this is the main attraction for me – conversations around the speakers – whether they are happening during their sessions (best case scenario) or in-between.

i do not arrive there to be re-told the ideas i have already heard, i would like to use them as a basis to explore further.

these days conferences where i know most of the topics and the speakers are definitely about conversations. this probably does not apply to a lot more focused/specialized conferences or those where i am not as familiar with the speakers. but here’s the paradox – i need to know the speakers well enough before i decide to go, at which point it seems like there is no reason to go anymore.

i am after passionate, articulate experts in their respective fields leading and participating in a conversation where most of the people are already familiar with each others’ backgrounds, arguments, works, and build on top of that.

which leads to a few conclusions, stemming from the need to create and nurture conversations. the whole idea of a keynote for a large audience is useless to me, since anything over a 15-minute intro is something i have probably seen/read already; any audience that is more than 20 people is too large; any audience that is not “up to speed” on the topics discussed will drag the conversation down.

the requirements above are probably a bit idealistic, but good user groups, and self-selecting gatherings (like hacker b&b at Stu Halloway’s i was fortunate enough to have been invited to this summer) prove that it is possible. another example is speakerconf, although its selection bias might turn it into an echo chamber.

so given all this, how did qcon fare? each day had something good, and i will try to write these highlights down in the next post.

would i attend again? it all depends on the speakers, and, even more importantly, whether i would be able to get to know more people beforehand and thus have good conversations outside of conference sessions.

cloudcamp nyc

Posted by anton
on Sunday, June 07, 2009

i am organizing my notes from cloudcamp nyc 2009. the whole thing was done the unconference style, and i bounced around different groups for a few hours.

taxonomy

it seems that folks were struggling to come up with the taxonomy of the cloud. i see it is a continuum from bare OS instances (amazon’s EC2) to increasingly specialized services (google app engine, S3, salesforce, and even facebook).

bare OS instances are easy – nothing there is new, and what is changing is the elasticity of the capacity – you can spin up and wind down instances much faster (and recently announced amazon’s auto scaling makes it even easier). the only catch there is that you have to design for scalability yourself. but you have all the power and the flexibility, and there is no lock-in.

the latter is when you give up some of the flexibility for scalability out of the box – the design and implementation are taken care of for you – google app engine and facebook just scale. there is a lock-in there, but you might not care. somewhat unrelated, but pmarca had some things to say about it a while ago.

automation

when dealing with OS instances, these developments make it obvious that if you have not embraced automation yet, it is time to do so. the lone admin doing things by hand does not cut it anymore; even barebones shell scripts and basic packaging might not be enough.

the sad thing is that i do not see an easy, thorough, cross-platform management tool that does everything out of the box. cfengine was the closest thing back in the day, but i want something that embraces source control and declarative configuration; something that is transparent and easy to use; something that is cross-platform. chef and puppet look promising.

so there are a lot of new vendors coming out and touting their products for managing the cloud.

monitoring

same thing with monitoring – sscope was a killer app almost a decade ago, but i have not seen a compelling polished replacement (until, possibly, hyperic) – something that could scale down and up, something that can be either agentless for small easy deployments and agent-based to be able to scale beyond that. amazon’s cloud watch is a step in the right direction, but it is just the beginning.

i hope that cloud computing developments will give this industry a push it needs to clean up its act and finally come up with comprehensive convenient set of tools that is sorely overdue.

bottom line is that with clouds there will be more stuff to manage, and the demands for the management tools will become stricter.

i want commodization of these tools – monitoring, deployment, etc. ideally they will be independent of cloud provider and i want them to be simple and transparent and composable and hackable.

usage scenarios

it is almost a no-brainer these days for startups writing green-field apps to rely on amazon for their capacity. it is great, since they are not as concerned with SLAs and lock-in is not that big of a deal.

amazon truly has kick-started the industry that lurked in the shadows for so many years with all the major players trying and failing to launch a successful pay-as-you-go cloud computing service. i guess their secret was extracting a service from their own successful projects, instead of trying to come up with something new for others.

as for enterprises though, i do not see them embracing the cloud platform as it is now, however there are some scenarios possible right now, and i am sure many more will emerge:

  • a single batch that needs to be processed (amazon now has hadoop services ; also consider load testing from the cloud, some sort of one-off processing, or anything else that is a spike in demand could be good)
  • overflow capacity for something like flash traffic during holidays for retailers – extend your private infrastructure, or certain pieces of it into the cloud and wind down when you are done
  • POC, development – it is hard to test on the same hardware as prod, so roll your stuff out on the cloud and then for prod/uat deployment put it back inside
  • one-off isolated apps that can deal with current state of SLAs and lack of private clouds, or even pieces of existing apps

there is a chance for internal infrastructure folks to learn something from the cloud experience. in most large companies servers and storage have been increasingly provided as a service and billed back to the projects, but provisioning is still not elastic, and utilization is often the usual low numbers. granted, more and more are using vmware to virtualize (at least dev/uat) to increase utilization, but it is a far cry from the experience of the likes of amazon.

the ideas and some of the tech from the public clouds could and should be looked at by the internal infrastructure people, and not just the servers/storage, but also those that provide middleware services – messaging and monitoring, for instance.

clouds in your own data centers? some of the vendors seem to head this direction – give us the servers and the datacenters and we will create cloud infrastructure for you, taking care of management, monitoring, apis, provisioning, etc.

i wonder how the chargebacks in organizations will change between departments – since instead of a capital investment upfront with depreciation we can convert it to operating expense with pay for use billing – would that allow us to win anything? of course, for small companies it is great – no upfront costs, no hardware, etc.

ms azure is somewhat an outlier here – first of all, it lacks tinkerability (yes, it is a term) – the barrier for entry is too high, but at the same time it is not a vertical app, but a full-blown platform with all the development tools in place. the problem is that it is too opaque, and i would imagine it will be a while before the whole kitchen sink has proper api, billing, provisioning, etc working properly – as usual the idea is fantastic, but the implementation, knowing the track record, is something to be wary of. since windows is notoriously hard to automate, perhaps azure will finally sidestep this problem and offer that as a service, but i cannot see how it can completely do it, unless it provides something as easy and transparent as ssh+shell scripts and declarative versioned management for that.

coming up

development in several directions – granularity of service, privacy of the cloud; standards – one must talk about a certain place in the continuum – standards for infrastructure level monitoring/provisioning are perhaps possible, but once we run into more vertical services, standards make little sense, since you are buying into the provider.

there will be consolidation, since, as failures of ibm and sun showed and success of amazon shows, it is a low-margin business, and amazon knows it first-hand.

GoRuCo 2009

Posted by anton
on Saturday, June 06, 2009

gotham ruby confrence is the nyc’s own ruby conference organized by nyc ruby users group. it was my second time attending it. this post is an attempt to organize my own notes, as well as an attempt at feedback that i think i owe to the speakers and the organizers.

i really like the smaller focused gatherings like this with around a hundred-plus attendees – there is definitely a community spirit there, since a lot of the folks are local and already know each other.

there is also more focused discussion, since the background of the people in the room is similar. a lot of the culture is shared, a lot of the values are implied, and the conversation zips along nicely. i have stopped attending local java groups precisely for the lack of this common background – the topics are too broad, the backgrounds are too different, and it takes a lot of effort to communicate ideas.

i am a bit uneasy about my own relationship with the ruby community – i am not really contributing, nor am i doing paid ruby work. i also am a bit weary of the monoculture (look at them macboys and macgirls!) that tends to re-invent the wheel way too often. but i am there for the excitement, for the bright-eyed kids that tinker and create things – this energy is infectious and i feed off it. it is inspirational and energizing.

so why am i going to the conferences like these? in addition to the energy boost i mentioned above, there is also the trivia of learning about tools, projects, approaches; getting the feel for the zeitgeist, where things are heading, what folks are thinking. deep down inside i am always looking for the “blow your mind” experience, something that can turn a familiar topic on its head, something that can make me discover things i have never suspected existed.

GoRuCo had a nice balance – things technical and detailed, and also approaches/techniques/principles. the reject conf at the end – a series of quick lightning talks – was an icing on the cake, stuffing you full of references, pointers, tips that you could take home and work through at your own pace.

Gregory Brown: Where is Ruby really heading?

more of a book report, talking about different versions of ruby out there. for anyone following the community, none of it was a surprise. no hard data either, just his personal anecdotal experience. this part of the talk was more suited to a short user group presentation.

there were a few nice tips and personal war stories related to moving between 1.8.6, 1.8.7 and 1.9.1, unicode, side-by-side installs and very basic crude techniques to code for different versions.

these are typical growing pains – everything from the compatibility issues between versions to the curse of the system-wide install that makes one go an extra mile in order to run different apps under different ruby versions on the same box (i always preferred the semi-structured self-contained java jdk installs and jars controlled by the classpath).

there were some props to jruby (it is a real distro, not a hack to reach out for when all else fails!), mentions of ffi in jruby that allowed gregory’s project to run on windows.

Eleanor McHugh: The Ruby Guide to *nix Plumbing

this could have been a great talk, but it seems like eleanor was really hungover, so instead it was a very disconnected series of ramblings on the general subject of unix and coding. very poor delivery, and at times plain embarrassing.

only towards the end she managed to find the message for the talk, which was “you can code against kernel internals using ruby, since it makes it much easier. do not automatically assume that you need to write in C for performance – try ruby first.”

she also highlighted ruby community’s respect for bare-metal – the ability to tinker, the taste for small simple tools that do the job well, and the affinity for unix. this is exactly what attracted me to the language in the first place.

there were a lot of references that at times seemed like name-dropping: ngnix, beej’s guide to network programming, beej’s guide to unix interprocess communication, ruby/dl, duby, event machine, c10k problem, advanced unix programming book

Dan Yoder: Resource-Oriented Architecture With Waves

dan briefly talked about his waves framework that tries to present a simple DSL around HTTP and resource representation in a REST fashion.

the meat of the talk that was interesting to me dealt with REST in general, its differences from MVC, resource-oriented architecture, self-describing data, returning links to other data inside of data to aid discoverability. he mentioned how a resource identifier (e.g. URL) should not specify representation (i.e. do not add .xml to specify that you need an XML document back), but rely on client’s Accept* headers to negotiate representation (caveat being that CDNs like Akamai do not currently care about these headers, so you will always get the same content).

there was also a mention of the fact that sometimes one can view HTTP protocol as something dealing with a distributed hash table (DHT) using get/put/delete operations (with post reserved for everything else).

RDF and freebase were also mentioned.

i have a strange relationship with REST: i get it on the technical level, i sort of get it on the architecture level, but it did not fully “click” yet, perhaps due to the fact that i do not have enough practical implementations under my belt. i should go through the restful webservice book again and play around.

good talk overall, but lacking concrete examples that could have brought things more into focus.

Jake Howerton: Into the Heart of Darkness: Rails Anti-Patterns

a bit of a disappointment, since the title promised so much. i usually really liked the anti-pattern talks, since you learn nothing when things work as expected – the real learning comes when things break, and you are forced to dig in and figure out why. plus these talks also give you an idea of applicability of certain techniques, which is really a required counterpart for all patterns to begin with (beware of dartboard-driven design).

instead we were treated to a very few amusing short code snippets, but no larger patterns in the sense of fowler’s refactoring book or even rails-specific patterns.

he did mention cucumber, reek and metric_fu and some general well-known testing techniques. i liked his term “flight check” for the smoke tests that run before deployment to prod, and a notion of sandbox test environment where mocks are replaced with real classes that do destructive things without impacting the real world (like sending emails).

i also like his term “irb-driven design” for something that was copy-pasted from the exploratory irb session into the production code.

there was also some treatise of legacy code, but i think dhh’s talk on the subject was much better.

overall it was a fun, light talk that was well-received. i blame the deceiving title and the lack of focus for initial feeling of disappointment; it felt like jake was simply talking about things he does and prefers to do during development, without specific overarching theme in mind.

Sandi Metz: SOLID Object Oriented Design

sandi stole the show; she set the level that none of the speakers matched and showed what it really means to have a solid, gripping presentation that is lucid, focused, well-prepared, and superbly delivered.

she talked about SOLID design principles: Single Responsibility, Open Closed, Liskov Substitution, Interface Segregation, Dependency Inversion which all boil down to managing dependencies.

she went through the fowler’s value of design argument, and then iteratively went through a refactoring example, invoking the SOLID principles along the way.

i think the most impressive for me was how well the whole talk came together, how the arguments were presented to support the refactorings, and how well it got into my head (almost uncanny, similar to the effect the best books in head first series have).

some of the principles she kept bringing up – using the rate of change as the indicator for splitting the functionality; refactor in small steps to let the design emerge – not because you know, but because you want to find the design; red-green-refactor; mock at the seam; only mock classes i own; those that change often should depend on those that change less often.

this once again brings up the importance of developing a language to discuss design (e.g. patterns) – i think this is a crucial step for every practitioner, when something intuitive and personal (e.g. a vague code smell) becomes something that you can articulate and communicate to others.

she also mentioned uncle bob, micronaut, steve freeman and nat pryce mock objects site

Benjamin Stein: Building Cross Platform Mobile Apps with Ruby & PhoneGap

the main promise of PhoneGap is quite compelling – build cross-platform apps on the phones using javascript while taking advantage of the native features (vibrate, storage, accelerometer, sound, gps, etc).

it was a fine presentation, with a story to tell and with a perspective that put all the low-level details in context. makes me itchy to get my hands on some phone development; i was also quite impressed with their adoption of latest standards (e.g. HTML5).

Yehuda Katz: From Rails to Rack: Making Rails 3 a Better Ruby Citizen

rails3 perspective from the horse’s mouth – quite detailed look at how rails is trying to be less opinionated in its choice of frameworks to work with, and how it exposes its internals for others to integrate with (orm, rack, js frameworks). some very interesting examples of design decisions and the overall future path of the framework.

this was a bit tedious, but quite informative talk.

Lightning Talks

i really like these, since most people do not have enough material for a long talk; in fact some of the main talks earlier should probably have been half their size.

the highlights include sunlight foundation and data.gov plug, a great pair of fast talks by aman gupta on google-perftools that he tweaked to work with ruby and used on some real-world code and joe damato on tweaking the thread performance in ruby 1.8 (in retrospect, these two talks were something i wish eleanor would have done).

"no fluff just stuff" day two

Posted by anton
on Saturday, May 27, 2006

day two started with neal ford's talk on practical agility. i feel that this is one of the areas i am lacking in - all of my daytime job experience has happened in the environment where very few agile practices took place. i am interested in these cookbook-like recipes that give one a tangible feel for the steps in the process.

no revelations here, but a nice talk reaffirming the facts. his subsequent two-part talk on sunday was far more interesting, actually showing the artifacts produced by the project managers, and telling real-life war stories. it turned out to be more of an introductory talk, but some quotes did stick (or perhaps resurfaced again):

if you are afraid of a piece of code, refactor it
agile methods expose poor developers
maximize the amount of work not done
agile development is a form of risk management

in fact, risk management is probably the best selling point of agile methodology (which neal constantly emphasized).

next was the talk by bruce tate, titled where agile meets argyle: new processes in established companies. sadly, almost nothing there was new to me, or particularly exciting. bruce is not an engaging speaker, compared to justin or venkat, but he is nevertheless a very nice guy to talk to, as long as he is not in the "speaker" mode. i really do enjoy his books and value his opinion, as well as his drive and energy. perhaps i just heard the same arguments over and over again throughout the whole weekend (not to mention reading various books on the subject), so the overall impression was a bit muted. i suppose i really never did stick up to the higher-ups, selling agile, so some of the points seemed somewhat irrelevant to me.

a couple of times i snuck out and dropped by glenn's talk on jvm performance. in retrospect i wish i spent more time there - although most of the subjects on paper looked like something i alerady knew (mostly through personal performance tweaking experience on developement and admin sides, as well as the classic java platform and java performance books), but there were a few points that were news to me (cost of uncontested serialization for instance). talking to him afterward i realized how thorough and how knowledgeable he was when it came down to the innards of java platform.

next talk was justin gehtland's ajax architecture. although i went there mostly for the entertainment value, i still got out a lot from it - just clarifying my own understanding, and, most importantly, learning to talk about this topic, and seeing how well he can engage the audience. him and venkat are complimentary - venkat is a better all-around speaker, but justin comes across a bit livelier, filled with a sort of boyish energy and eagerly stuffing a multitude of other topics into his talk.

talking to neal ford the night before, i was curious to see his "productive programmer" talk (especially since there is a book in the works, scheduled for this fall). i stopped by a couple of times during this talk, and saw all the right things in place - cygwin + unix + command-line tricks, scripting languages for automation, etc. it was a good basic talk, and there was nothing really that i learned (except reminding me the pain i felt when i first switched from norton commander to windows95 gui).

however the topic itself warrants a longer rant, and i am really glad neal brought it up. first of all, i make a difference between a programmer and a developer. i've worked with both kinds (and even extreme examples of both kinds), and for the most part programmers are only useful in a big organization, coding to the tight, verbose spec, perhaps outsourced. a developer is someone that has a toolbox and knows when and how to use the tools at hand. i've seen too many consultants or narrow specialists that do wonders in their respective app development gui/ide, but freak out when it is minimized and they face an OS in front of them, or a task that is done best with something else (just ask me sometimes about the "laptop boy" or "tight guys"). i used to wonder whether the knowledge of the OS and productivity tools is important to a developer, and these days i see it almost as a requirement, better yet - a sign of common sense. a good argument that neal brought up - developers know more about the underlying OS, the way it works, and the way to tweak it to maximize their productivity.

however, there is a difference between GUI/OS productivity tricks (shortcuts, "getting around in a hurry", note-taking apps, etc) and development tools. both are needed, but the former is a bit tricky, since one could use many machines, and tweaking the environment at that point becomes painful (i myself have four machines, and i used to run heavily customized litestep and the likes, but now i just stick to minimal customizations: mostly firefox and cygwin + a small list of IDEs/productivity tools). when it comes down to development tools in one's toolbox, i want to see common sense and knowledge of a wide variety of tools. unix background generally puts people in the right mindselt - given the famous "grep question" from stevey's list of five questions, i want to see a sensible response; and this sort of response comes from a developer well-versed in basic unix tools.

after i switched groups at work about a year ago, i found myself in the environment where all of a sudden i could really help less-technical folks out by a few quick scripts - be it data imports, creating test data, loading some stuff from LDAP, merging a bunch of spreadsheets, etc. it is surprising how much energy is spent on these tasks, only because there is no versatile developer around that could quickly whip up a bunch of scripts. this is when the unix ideology really pays off - a bunch of small tools that read stdin, print to stdout, and could be chained together using pipes in cygwin/unix. this is when the value of a simple for i in `ls *blah`; do stuff; done could not be overestimated. incidentally, someone in my environment that showed a great ability to help the folks out (not just business users, but certified thought-leading architects) was a testing guy - a great example of someone that can build scaffolding around a product if needed, understands why and how stuff works - anything from OS to the end-user product.

my personal acid test for the developer would include a few questions on pragmatic automation - know the scripting languages, know the OS scripting, understand when a 1000+ lines of Java code could be replaced with a few lines of shell scripts. this is when the books like data crunching from pragmatic programmers and "pragmatic automation" come in handy (but a good developer knows all that already, right?).

to conclude this rant, this is why i enjoy the "back to the unix basics" mentality switch that was brought by the rails community. rake/capistrano + linux/mac os x; back to the bare metal and hacking mentality (as in building small, simple things and taking advantage of the basic concepts, but pushing them further).

the last talk of the day was "three technologies to watch" by bruce tate. a very anticipated topic, although he gave away some of it during his previous talks or personal conversations. he started with rails' active record and the way it takes advantage of metaprogramming in ruby. i've already seen most of the talk, since his blog entry on the topic was posted on artima just a couple of months ago. great stuff nevertheless, i wish i've seen it a year ago. next set of slides was on continuations and their applicability in the web servers world. once again, i have already read bruce's article on developerworks. this time it made me acknowledge (once again), how much i am missing because i am not familiar with functional languages (lisp in school and incidental readings on haskell do not count). the idea is great, and i am yet to grasp all the implications of it. a takeway - take a look at continuation-based web frameworks (rife, webwork2, seaside). finally, bruce talked about erlang, yet again exposing my lack of knowledge about parallel systems and the math behind it. instead of my poor interpretation of it, i will suggest reading yet another passionate rant by stevey yegge that talks about parallelism as it applies to CS.

the day was a bit disappointing, since a lot of the talks were based on the blog entries i have read earlier; i guess this is the drawback of being too close to this community and mindset. but if nothing else, these guys are very good speakers, and it helped me to clear up a lot of things in my head which hopefully would allow me to be a bit better at relating this stuff to others.

"no fluff just stuff": day one, the keynote

Posted by anton
on Tuesday, May 23, 2006

the keynote is what finally delivered that "blow your mind" experience that i was looking for. it started rather innocently with the usual run-of-the-mill DSL talk - traditional coffee-ordering DSL, etc. however, a little bit into it, and there was a nice distinction made between internal DSLs and external DSLs - something I have not articulated as clearly to myself.

internal ones are built on top of the language they are written in; they have full access to the underlying language, but they are also bound by its syntax and features. external DSLs is something you write a parser for, and then either interpret or compile them. external DSLs can do anything, at the expense of writing and maintaining a grammar (strangely enough, JavaCC is not used as an example, although yacc and lex are (no bison though!)), and lack of IDE support (auto-completion, color-coding, etc). a very simple way of distinguishing the two, very nice.

next is a statement that OO is not a panacea - not everything can be neatly modelled in hierarchies; not everything is an object. instead a lot of problems are best modelled in a language. here i digress and refer to another very pertinent rant by steve yegge: "execution in the kingdom of nouns." this is a very welcome approach, especially going back to functional languages and DSLs. context is everything; the same functional language can result in different representations, depending on the context (an example with ruby DSL and several different includes at the top that change its behavior).

next is something i have not thought about until now - the fact that none of the IDEs these days work with text - it is all abstract syntax tree (AST) that we work with. this is what allows IDEs to perform refactorings and all kinds of other tricks. this is why IDEs used to require one to compile the code before it could be refactored, but now they are getting better and better at handling broken code (i.e. code where some of the branches are missing). also, do not forget the predecessor of all small languages - Unix shell. it would be interesting to see where it failed, and where microsoft monad is failing.

finally, he whipped out JetBrains MPS - a tool for metaprogramming developed by the creators of IntelliJ. it brings the power of IDE to external DSLs - code completion, color-coding, syntax verification. in other words, one can design their own DSLs, and have an IDE that supports it. the tools looks very promising, although somewhat lacking "productionalization" side. seems like the team is waiting to see what the big boys (microsoft with software factories, and with intentional software) will do, and then offer an product to support it.

this leads nicely to the stuff i have been struggling with lately at work. i do not believe in MDA tools - any generated code is the code one has to maintain, and abstractions there leak very easily, especially when it comes to distributed enterprise code. so the approach to raising the abstraction level and getting closer to the problem domain is to use terse, expressive languages. these languages could be internal or external DSLs, as neal showed in his presentation. alternatively, the same domain (as long as the underlying core representation of the domain is the same AST) could be represented with graphical tools. and this is the stuff that i do at work on a current project - a well-defined space of enterprise integration is modeled using simple GUI tools that include all the building blocks - queues, web services, mappings, flow control, database connectivity, adapters to other apps. i was uneasy at first, feeling uncomfortably close to the MDA world, but now I can view this as yet another form of DSL.

and this leads to another topic i have been pimping left and right: given these simple integration tools and packaged products, a typical enterprise (the one that can be fit into the business model offered by these packaged products and can afford them) will require a very different type of "architect" and "developer". since it takes only a couple of hours to knock out an integration scenario, mapping a message from one system to another, complete with HA/failover/load-balancing/security/etc, then majority of the work would require less and less in-depth technical skills, but more and more of a business knowledge. these people will know a lot more about the business, and a lot less about computer science. granted, the breadth of knowledge is still essential to avoid blunders, and they would be technical people first of all, but a lot of their expertise will shift to the business domain, once the infrastructure and packages are in place. my favorite tabloid happens to reaffirm these sentiments in a recent article: "SOA predicates rise of the enterprise architect".

this kind of architecture still requires a lot of technical knowledge and developer's background, so the promise of "situated software" is still just a dream. it does, however, highlight the difference between the developer and the architect, as well as the different between application architect and integration architect (especially the one that depends on business domain knowledge). my problem is that i still enjoy developing things so much - i like building them, i like seeing them being used by people, i like tweaking them as they run. i like doing hands-on stuff, this is why i still do things on the side; but i also like the challenge of developing large systems that do require a significant domain knowledge. the question is whether my current occupation is indeed a domain that i am interested in knowing more about.

in retrospect, i think a lot of the talk came from martin fowler's article on language workbenches, which follows the same general layout as neal's talk (neal also quotes fowler quite a lot, but then who doesn't? the man deserves it; i just wish i had a better memory). in any case, the talk was excellent, and i am looking forward to digging more into this topic.

the whole keynote is available on neal's site: slides, samples

no fluff just stuff: day first talks

Posted by anton
on Monday, May 22, 2006

this year i was actually anticipating the conference, waiting impatiently for the release from work tedium. i tend to feed on the energy of others, and these intense three-day gatherings are a perfect boost that lasts for months afterward.

this time i chose to attend fewer purely technical talks. i think some of it is the result that i have to work with people a lot more in my new role, and i am trying to figure out how to be better at it - there seems to be a lot of non-tangible things that i am struggling with. another side of it - i am trying to decide what architecture is as opposed to development, where is it that i am trying to go, what is it that i am interested in doing, and what my focus should be, given that the time is limited.

of course some of it also comes from the gratifying masturbatory effect of bitching about work and those cavemen coworkers (the theme that was running strong last year, and something i am too eager to jump into myself).

on friday afternoon i caught the end of ramnivas laddad talk on AOP ("state of aop", to be exact). nothing groundbreaking there, just a nice rundown on load-time weaving (relying on JVMTI agents), bea's take on aspects in jrockit (do not miss reflective calls), xml and annotation syntax for aspects, eclipse AJDT, java5 features support. it was a little disappointing to see the obligatory logging example (i thought any AOP guy is fed up with these). most of this is already familiar to anyone that followed the AOP landscape, even from the distance; not a bad talk for anyone that is just starting out. one of the best AOP talks last year was on step-by-step adoption techniques with cookbook examples. i think this is the best way of introduction, since a lot of people are just paralyzed by the power available to them, not knowing where to start.

next i stopped by venkat's "refactoring your code" talk. i remember him being an excellent speaker, and i primarily wanted to see how he explains this already familiar topic - perhaps copy a trick or two to use myself if i were to speak in front of others. instead i was simply drawn in and followed every moment of the presentation. he was coding at breakneck pace, and yet amazing lucid, humorous, and illustrative, tying it altogether with the agile development principles. the examples were excellent; he was getting feedback from the audience, and yet driving the talk; setting up little traps and problems and solving them effectively, illustrating the point. his real-life anecdotes are so vivid often crossing the boundary into absurd; and they stick in your mind, together with his deadpan delivery. he is so natural, none of this stuff is forced or awkward. excellent talk; as a teacher he is unsurpased. a nice example of repetition as a stepping stone that gradually lets one refactor methodically and safely. some quotes from william zinsser on writing. and some famouse quotes (perhaps if i write them down, i will actually remember them):

perfection is achieved, not when there is nothing left to add, but when there is nothing left to remove. (Antoine de Saint-Exupery)
Programs must be written for people to read, and only incidentally for machines to execute. (Abelson and Sussman)
unit tests and code is like my left leg and my right leg - i do not walk three miles just on my right leg and then switch

this set the theme for venkat's talks for the rest of the weekend - i attended his talks on seemingly familiar topics, but he managed to either illuminate them from a different perspective, or really arrange them properly in my head, clarify and organize them. i seem to get a lot of stuff on very intuitive level, and then it takes me a while until i can clearly formulate it. teaching would probably be the best way to get it organized, but venkat's talks is a nice shortcut for that. some of this stuff is also available on his website.

still under the impression from the books i've read, i went to see bruce tate's talk on effective teams ("effective teams: the dirty little secret"). it was pretty decent, although a little tedious at times (when we had to remember what the letters were corresponding to the personalities and then go through the examples). up to this date the best books on team organization were "peopleware", "rapid development", two joel books ("best in software writing 1" and "joel on software") and many-many blogs on interviewing (stevey's drunken blog rants is one of the recent favorites). so a lot of the stuff was already familiar, although it was nice to see a distillation of personalities (rock vs. visionary, as well as myers-briggs personality test), as well as reaffirmation that good teams need both (here it is also pertinent to mention the difference between a developer and a programmer, and where each one fits). another good point - mentoring (get one; perhaps even outside of company) and keeping up.

looking through the slides for the first day, i realized that glenn vanderburg had a very nice talk on collections api - filled with practical details, very hands on, very useful. i think this goes for all of his other talks that weekend - very well researched, very thorough, very technical, in retrospect, i wish i have attended more of them.