qcon2009sf: other talks

Posted by anton
on Monday, February 01, 2010

for completeness sake, in addition to outstanding talks, i should mention a few others that were memorable.

i caught the second half of eric “DDD” evans talk – it was moving at a glacial pace (a trait of a professional consultant that is paid by the hour?) and was terribly overcrowded, but i liked what i heard. he took the roundabout path to get to the importance of evolution of the existing systems, but he made his point well with some great insights along the way. i have been setting aside his book for years, and i know it is long overdue, but i should read it.

i was surprised that none of the talks on the “cool stuff with java” track appeared all that cool to me. perhaps i have already looked at the tech behind them too closely, and the talks did not offer much on top of that.

nevertheless, project voldemort talk was a good refresher on brewer’s CAP theorem, consistent hashing (or DHTs), amazon’s "dynamo". i wish it were more technical, since the devil is in the details, and i wish there was more comparisons to other systems in this landscape (e.g. cassandra which seems to be further ahead).

hadoop is pretty much a household name these days, so a generic intro to the tech and its ecosystem did not do much for me. still, having built and run ETL environments in the past, i think that anyone in this position must consider hadoop these days – after all, if your ETL environment is trying to scale vertically and is using one of the big boys’ mammoth software, then it will make sense to compare 1MM for the license + hardware costs with 100K for commodity hardware + free software. this is a subject for a whole different post though.

by the end of the conference i have acquired a nervous twitch each time i heard another attempt at cloud computing definition. i did like stu charlton’s talk though – he had a good business perspective, and although i do not believe in his product, i would be keeping an eye on it, since they are trying to solve a real problem.

finally, i was disappointed by my own reception of “architectures you’ve always wondered about” track. facebook, linkedin, amazon – i think i’ve already knew enough about them, so signal to noise ratio in these talks was too low for me, and i could not justify staying there for long.

qcon2009sf: individual talks

Posted by anton
on Monday, February 01, 2010

as promised, these are a few talks that i have attended and found worth mentioning. i know i missed a lot due to scheduling conflicts, but that’s the nature of the game.

clojure talk by stu halloway

a great talk that had to be witnessed; it was a fast-paced flight through the language, illuminating its features and defining its place. there was a story, there was excitement, and there was a pragmatic take on it all. since languages track was aimed at actual usage in the field, half of the talk was spent on war stories – things that worked and things that did not. it was a tight, erudite talk, with just the right amount of details.

i am still ambivalent about the language – i do not have enough experience with it to see it used in production on the systems i am working on right now; it still exhibits growing pains, and feels a little rough around the edges. perhaps it is my lack of functional language exposure that shows. at the same time i am absolutely fascinated and excited about the things that it gets right – the immutable data structures and the whole concurrency story. the language feels nimble, finely honed, and it is great to witness its evolution, as it happens in front of my eyes. i love the way it makes my brain feel, the way it challenges my perspective on language design and features.

if tech is your competitive advantage, and you have small sharp teams, then by all means, give it a try. even if the language does not survive in its current state, the ideas and their implementation will live on – i think they are that important.

groovy on the trading desk by jonathan felch

there is always a bit of a stigma associated in my mind with conference “thought leaders” – unless they have proven their credibility by repeatedly building and shipping, i always take their words with a grain of salt. after all, those that have the time to float from conference to conference, from client to client, might (d)evolve into pundits. there is a definite value in that, and i certainly would still attend their talks and buy their books, but i would always remind myself of their perspective.

i really liked jonathan’s talk because it was ruthlessly pragmatic, coming from someone driven to ship, working with traders to solve real problems quick. in a sense, this environment is somewhat of an aberration, since outside of the trading desk the software development world is quite different.

his team was bending technology, doing very creative things right on the bleeding edge; by any means necessary they had to deliver software for the business under the tightest deadlines. in some respects, this is the technologist’s ultimate dream – when politics and money and all the other pressures of architecture and enterprise world are pushed aside and all that matters is whether you can deliver. this freedom is scary and exciting at the same time.

the talk was about a distributed computational in-memory engine they have built to support a trading desk, standardizing on groovy and building on top of terracota in 2006. this allowed them to deliver new features in a matter of hours for their traders.

essentially it was a graph where cascading properties were recomputed in response to events; logic could be injected via closures at runtime, state and behavior of individual nodes could be extended at runtime; all of it written in functional style, avoiding shared state.

he talked about the areas where groovy shined, and walked through some of the problems they have run into. predictably, the strength was writing DSLs, terseness, convenience, speed of writing code, dynamic nature, integration with other systems; pains were performance, math, and language gotchas.

jonathan also managed to give a perspective on quantitative finance in general – what is the business about, who are the people involved, what tech is used, and what problems they have to deal with. this is what really made the talk “sink in.”

i dismissed groovy early on, when the race to add new language features trumped the need for quality and thoroughness. the whole affair seemed to be too sloppy and haphazard, so i only watched its evolution from the distance. i suspect that the situation has changed, even before spring source acquisition, so i should give groovy another try.

architecture for the cloud by michael nygard

i am yet to read "release it", but i have come to appreciate his perspective based on the articles, blog entries, and book excerpts. i think michael has a great gift for organizing and presenting the patterns of operations architecture, a field that for the longest time has been the dirty secret of running systems; the proverbial “last mile” of software development, the achilles’ heel.

it takes someone straddling the fence between operations and developers to recognize the issues. having been in this role myself (and having ranted about it on this very blog), i am really grateful to him for illuminating and organizing the patterns in a manner that (hopefully) should help us as a community to avoid repeating the same mistakes.

why do patterns matter? it is a shared language that allows those responsible for operations and development to communicate with each other and recognize the problems and their solutions.

why does operations matter even more now? the proverbial admin/developer fence is disappearing, cloud means fast provisioning of many machines done by developers/users; the developers become more and more aware of infrastructure, which, in turn, becomes more of a commodity, while the job of the sysadmin is changing. i really liked his practical insights into the way cloud computing will be changing the roles within the enterprise.

overall, i mostly nodded along with the slides, but it helped me organize my own thoughts on the subject (which, in the end, might be far more important).

doug crockford on javascript

an immensely entertaining and sprawling talk that went into the history of the language, its evolution, and its future. it was exactly the kind of the talk i dismissed earlier, but at the end of the long day, and from someone as entertaining as doug turned out to be, i was ready to kick back and enjoy the ride. there were plenty of anecdotes and killer one-liners, so if you get a chance, watch him speak.

erik meijer on rx

as opposed to the doug’s talk mentioned above, this was a great example of a conversation, or riffing, where ola bini, don box, dan north, amanda laucher, sadek drobi, and a few others completely derailed the whole rx presentation into a joyous bantering about languages.

erik meijer is a joy to listen to as it is, but if you add a small responsive audience and a few beers, the whole experience is unforgettable. he loves to be paradoxical, and he revels in controversy. he was dropping tweetable gems at an astonishing rate – the sparks were flying, and my brain could hardly handle it.

at last we were pretty much forced out of the room by the staff – it was a perfect closing for the conference.

eclipse import static

Posted by anton
on Tuesday, January 26, 2010

i very much believe in a craftsman approach to software development, which, among other things, advocates the value of mastering the tools in your toolbox.

i consider using keyboard shortcuts in your IDE of choice a part of this craftsmanship approach. i do sympathize with unclebob’s plight for mouse-less editing and with stevey’s earlier posts on the subject.

back in 2004 java5 went GA and introduced static imports among several other syntactic niceties.

this feature is most useful for static helper methods – instead of writing Assert.assertEquals, i would rather use assertEquals, since i know i am writing a test, and in the domain of testing, assertEquals does not need to be qualified. same goes for many internal utility methods that i tend to use a lot (e.g. asList()) or static factory methods (e.g. newDateTime()). as a side note, using newXY() as a static factory method as opposed to create() makes it more suitable for static importing.

in my current IDE (Eclipse) i rely on auto-completion Ctrl+1 programming, so i would start typing Assert., followed by Ctrl+Space and then manually convert normal import to import static. it was very undignified.

it turns out that in Eclipse Ctrl+Shift+M that i already used to import dependencies under the cursor, also works for converting static method calls into static imports.

now all i have to do is type Assert.assertEquals once, then press Ctrl+Shift+M (obsessively followed by Ctrl+Shift+O to organize imports), and i can start using assertEquals all over the place without qualifying it with Assert.

as an additional convenience, i always set Number of static imports needed for .* to 1 under Java -> Code Style -> Organize Imports in Eclipse preferences. this way a single static import of a method from Assert triggers import static of Assert.*, which is what i want.


Posted by anton
on Thursday, December 03, 2009

it was my first time at qcon, and i found myself quite conflicted before the conference – i kept asking, what was the main reason for me to attend?

i already know roughly 75% of the speakers – i’ve read their books, i have been following their blogs and twitter feeds, i even saw their presentations online. i know for the most part what they would be telling the masses, thus i could not expect a blow-your-mind experience, the ultimate conference “catch” i am sure everyone is after. perhaps this is the curse and the blessing of the ultimate transparency.

if i am interested in this stuff, i have already taken all the easier steps to get up to speed; attending a conference comes next for me – not before, not instead.

as it turns out, once you get a dozen or so people that roughly have the same level of expertise and the same interests, and that do not expect to be spoonfed as they doze off, something else emerges – a conversation. this is the main attraction for me – conversations around the speakers – whether they are happening during their sessions (best case scenario) or in-between.

i do not arrive there to be re-told the ideas i have already heard, i would like to use them as a basis to explore further.

these days conferences where i know most of the topics and the speakers are definitely about conversations. this probably does not apply to a lot more focused/specialized conferences or those where i am not as familiar with the speakers. but here’s the paradox – i need to know the speakers well enough before i decide to go, at which point it seems like there is no reason to go anymore.

i am after passionate, articulate experts in their respective fields leading and participating in a conversation where most of the people are already familiar with each others’ backgrounds, arguments, works, and build on top of that.

which leads to a few conclusions, stemming from the need to create and nurture conversations. the whole idea of a keynote for a large audience is useless to me, since anything over a 15-minute intro is something i have probably seen/read already; any audience that is more than 20 people is too large; any audience that is not “up to speed” on the topics discussed will drag the conversation down.

the requirements above are probably a bit idealistic, but good user groups, and self-selecting gatherings (like hacker b&b at Stu Halloway’s i was fortunate enough to have been invited to this summer) prove that it is possible. another example is speakerconf, although its selection bias might turn it into an echo chamber.

so given all this, how did qcon fare? each day had something good, and i will try to write these highlights down in the next post.

would i attend again? it all depends on the speakers, and, even more importantly, whether i would be able to get to know more people beforehand and thus have good conversations outside of conference sessions.


Posted by anton
on Monday, July 27, 2009

scarred, but not defeated by vim, i decided to try a recently released beta of rubymine 1.5, and it’s been great so far.

massive disclaimer: i have not tried anything else for ruby/rails coding except for vim.

unlike a similar offering from netbeans, rubymine is a standalone install that seems to reuse a lot of the existing intellij idea codebase.

it was great experience out of the box – i pointed it at the local svn working copy and it verified all the installed gems (even though i am fortunate enough to run on cygwin, it recognized them all).

it has great rake support, and all rake tasks run without any modifications. however, for script/server in cygwin evironment i had to replace -e"STDOUT.sync=true;STDERR.sync=true;load($0=ARGV.shift);" with -e"STDOUT.sync=true;STDERR.sync=true;RAILS_ROOT='/cygdrive/c/project/root/dir';load($0=ARGV.shift);" in ruby arguments field. once the server runs, it also displays its log with proper color-coding.

i still run script/console from cygwin command line, since rubymine does not do readline support (i use ctrl+L, ctrl+R, tab completion, ctrl+e/ctrl+a and other goodness quite a lot in my irb).

things to love
  • color-coding, auto-indenting (including color-coding of matching brace/bracket/do-end block)
  • ctrl+shift+n/ctrl+n for finding files and getting around; with alt+f1 to show current file in different contexts/views
  • ctrl+f12 for current file structure
  • visible spaces (otherwise the ruby coding standards make the code look too squeezed to me)
  • ctrl+click when mousing over (jump to all kinds of things, including template names, css style names – all of it very nicely integrated)
  • ctrl+/ for toggling comments
  • rails project structure
  • parsing my stacktraces and linking them to the source code
  • autocompletion, although i do not find myself using it too much
  • all the usual things that idea has – svn and git support out of the box (and shelving works, just like in idea, in case you have to work with svn)
  • pretty sweet diff (ctrl+d) that rivals tortoisesvn visual diff
  • ctrl+shift+up/down to move the current line
  • ctrl+d to clone selection
  • shift+delete to delete the whole line
  • ctrl+shift+f12 to go full-screen
  • alt+number to toggle between tool windows
  • file structure tool window (alt+7)
  • alt+f7 to find usages
  • alt+shift+f10 to bring up run menu
  • shift+f10 to run current run
  • simple refactorings (introduce method, rename variable/method, etc).
  • it is pretty damn stable, and occasional errors do not kill the IDE.
things to improve
  • unlike netbeans, it is not a full-blown ide with ruby support, so some things that exist in idea are missing (notably, database support, some of the team communication and code sharing features, and other bells and whistles)
  • ctrl-q for docs is a bit wonky (frankly, i’d rather jump to matching place in the online api docs – it gives me context)
  • code folding fscks stuff up sometimes
  • still do not know how to jump to matching brace/do-end block
  • svn switch could not be found
  • ctrl+shift+f10 to run current test (and any other ad-hoc run tasks) does not work on cygwin, unless you do RAILS_ROOT trick above

i have not tried all the other stuff, like haml support, cucumber support, rspec, and rspec w/ drb.

overall feel is nice and polished – most things just work out of the box (unlike the frankenstein monster that eclipse can be sometimes – truly a Windows of IDEs).

i do believe in using “idiomatic” shortcuts with an IDE, thus i did not try any of the “compatibility” keyboard modes.

for now, i do not see myself coming back to vi for rails development – for a hundred bucks, rubymine is a great development tool.

reading: maister

Posted by anton
on Saturday, July 25, 2009

managing the professional services firm is my second book on consulting (first one being weinberg’s secrets of consulting).

i have been resisting the temptation to gobble up the whole book in one sitting. i am taking it slowly, giving myself time to think through each chapter, since every one is loaded with so much insight and relevant quotes, that at times i even get frustrated, not able to take it all in.

it has it all – from marketing to clients, finding work, to running a consulting business, motivating and growing people. i am only half-way through, but i am tempted to label it as a required reading for anyone starting in consulting.

here’s a quote from “motivational crisis” chapter:

Are professionals different from other types of workers? Do they need to be managed (and motivated) in special ways? While it is difficult to support the assertion that all professionals are different from all other workers, my work has led me to suspect that, when it comes to motivating forces, the average professional is different from the average worker in other environments: a difference based, I suspect, not on such things as educational levels, but on the psyche of those who choose professional careers.

The typical professional is apt to describe him or herself in the following way: “I am the type of person who gets bored easily. I hate doing repetitive sorts of work, and always like to seek out new challenges. Once I know I can do something, it tends not to satisfy me anymore.” This is, of course, a somewhat self-flattering description. In my experience, however, it is an accurate one. Professionals, certainly the best among them, are constantly driven to seek out the new, the unfamiliar, the challenging. The key word here is driven.

People who feel the (neurotic?) need to constantly and repeatedly test their skills against unfamiliar problems with an uncertain probability of success are frequently insecure, with a low sense of self-worth (never expressed in public), in constant need of external tests of their merits to prove (to themselves) that they still “got it.”

Many professionals, I would assert, are prime examples of what is now termed “The Impostor Syndrome”—successful people who live in constant dread that someone will, one day, tap them on the shoulder and say “We’ve found you out. You’ve been faking it all these years.”

Because of this, professionals tend to exhibit some clearly defined behavioral characteristics. They require continual challenge and personal growth to retain their interest, and are impatient when they do not receive it. They constantly ask themselves, and their superiors, “Am I still on track?” Because of their insecurity, and the ambiguity that surrounds the definition of “good work” in professional contexts, they need quick, repeated feedback on their performance to validate their efforts. They tend to be “scoreboard-oriented”: eager for visible, well-defined measures of success that can reassure them. They like to have unambiguous goals to shoot at. From their need to achieve self-respect by receiving the respect of others, it follows that professionals value both autonomy in their work and involvement in policy decisions, whether on engagements or firm-management matters. As much as these “rewards” are valued in their own right, they are valued more as signs that the organization trusts and respects them.

ode to vi

Posted by anton
on Friday, July 24, 2009

i recently had to do a small rails project. so i did what i usually do in these cases – fired up the easiest IDE that runs anywhere – vim.

now let me reminisce a bit – i’ve been using vi on and off ever since i got my hands on linux in 1997, and i have not learned much beyond the basics over the years. it works the same way on half a dozen unixes i’ve used it on, as well as on cygwin and macs; even dreaded beasts like mks toolkit provide it. it is an indispensable cross-platform tool.

it does not require much horsepower, and it fits well with the back-to-command-line ideology of rails.

if all you are using it for is editing occasional file or two, it does everything you need out of the box. throw in basic syntax highlighting, auto-indenting, split windows, buffers – and you have enough to survive.

oh, did i mention the macho factor? it takes some effort to tame the menu-less monster of an interface with a barrage of keystrokes that appear as magic incantation to others.

vi has a peculiar physical effect – i often surprise myself when i remember certain editing commands, but at the same time i am utterly unable to remember them when standing behind someone, advising them what to type – my fingers twitch, but my higher brain functions are not firing.

curiously, this reminds me too much of some of the mainframe folks i’ve seen, or even an occasional SAP jockey. consider it a compliment – there is a lot of power in short mnemonic commands compared to drill-down menus. yes, every powerful system must have a command line, but it must degrade gracefully. with vi the discoverability of interface is pretty much non-existent, and the learning curve is steep.

my current theory is that the muscle memory vi creates leads to a particular form of addiction which explains its appeal (and perhaps the religious zeal).

when i work in a context of a project, where i constantly need to bounce around different files, vim UI starts to break down: built-in buffers support is inadequate. the editor needs to have a concept of a project i am currently working on, and, ideally, the framework i use.

i know that the usual answer is customize, customize, customize – and in this respect it follows very much a linux tradition – if you are 15yrs old, and have tons of time, and only one machine, you can spend days crafting that perfect setup that is just right. having done that a number of times, i have learned that it is just not worth my time – I switch computers often, work on client sites, bounce between different teams, so i want stuff that is reasonably workable out of the box.

i can take it easy and install some basic plugins – fuzzy finder to give me files i want fast, nerd tree for filesystem navigation, rails for rails integration, tComment for toggling comments on blocks of text.

but now i need to manage them across several machines, and perhaps i do not want to spoil my vi muscle memory that can cripple my vi-fu on that hp-ux 11.11 when i come across it (yeah right).

so perhaps i will draw the line and use some other ide for project work, leaving vi for simpler stuff.

to add some substance to this post, a few significant lines from my .vimrc:

syntax enable
filetype on
filetype plugin on

set tabstop=4
set shiftwidth=4
set expandtab

autocmd FileType ruby set shiftwidth=2|set tabstop=2|set expandtab

set number
set ai
set si

and some of the commands i use often (besides the usual navigation/editing/searching ones that are in my muscles, but refuse to be articulated):

  • :e! to reload the file i am currently editing
  • :retab i really hate those tabs
  • :ls to look at open buffers (the listing is a pain to read, trying to remember what those little symbols mean and matching numbers to file names)
  • :e filename to open a file in a new buffer
  • :bd to close the current buffer
  • :e# to bounce between two last buffers (how can i cycle between all the buffers ala alt+tab?)
  • ctrl+w followed by s or p to start splitting windows, then bounce around them with double ctrl+w or ctrl+w and arrows
  • ctrl+w followed by <number>+ or <number>- to shrink or grow the split windows
  • ctrl+w followed by q to close current window
  • zz to center screen on current line, accompanied by zt op and zb ottom
  • % to jump to matching brace
  • o and O to insert the line and switch to editing mode
  • I to insert at the beginning of the line (that i always forget, unlike its companion A)
  • m + letter to place a named mark, ' + letter to jump to the beginning of the line, ` + letter to exact position of the mark
  • >> and << are much easier for indenting, as opposed to my muscle-memorized number + > + enter
  • shift+v or v to do visual selection
  • once you visually selected stuff, you fold it with zf

finally, i find the whole :tabnew business utterly useless and insulting.

what i do miss in addition to project structure navigation, is easily looking up/jumping between methods, code block folding that follows language semantics, and decent tabs.

jamis buck blog entries on the subject were really informative and inspiring (especially the comments).

cloudcamp nyc

Posted by anton
on Sunday, June 07, 2009

i am organizing my notes from cloudcamp nyc 2009. the whole thing was done the unconference style, and i bounced around different groups for a few hours.


it seems that folks were struggling to come up with the taxonomy of the cloud. i see it is a continuum from bare OS instances (amazon’s EC2) to increasingly specialized services (google app engine, S3, salesforce, and even facebook).

bare OS instances are easy – nothing there is new, and what is changing is the elasticity of the capacity – you can spin up and wind down instances much faster (and recently announced amazon’s auto scaling makes it even easier). the only catch there is that you have to design for scalability yourself. but you have all the power and the flexibility, and there is no lock-in.

the latter is when you give up some of the flexibility for scalability out of the box – the design and implementation are taken care of for you – google app engine and facebook just scale. there is a lock-in there, but you might not care. somewhat unrelated, but pmarca had some things to say about it a while ago.


when dealing with OS instances, these developments make it obvious that if you have not embraced automation yet, it is time to do so. the lone admin doing things by hand does not cut it anymore; even barebones shell scripts and basic packaging might not be enough.

the sad thing is that i do not see an easy, thorough, cross-platform management tool that does everything out of the box. cfengine was the closest thing back in the day, but i want something that embraces source control and declarative configuration; something that is transparent and easy to use; something that is cross-platform. chef and puppet look promising.

so there are a lot of new vendors coming out and touting their products for managing the cloud.


same thing with monitoring – sscope was a killer app almost a decade ago, but i have not seen a compelling polished replacement (until, possibly, hyperic) – something that could scale down and up, something that can be either agentless for small easy deployments and agent-based to be able to scale beyond that. amazon’s cloud watch is a step in the right direction, but it is just the beginning.

i hope that cloud computing developments will give this industry a push it needs to clean up its act and finally come up with comprehensive convenient set of tools that is sorely overdue.

bottom line is that with clouds there will be more stuff to manage, and the demands for the management tools will become stricter.

i want commodization of these tools – monitoring, deployment, etc. ideally they will be independent of cloud provider and i want them to be simple and transparent and composable and hackable.

usage scenarios

it is almost a no-brainer these days for startups writing green-field apps to rely on amazon for their capacity. it is great, since they are not as concerned with SLAs and lock-in is not that big of a deal.

amazon truly has kick-started the industry that lurked in the shadows for so many years with all the major players trying and failing to launch a successful pay-as-you-go cloud computing service. i guess their secret was extracting a service from their own successful projects, instead of trying to come up with something new for others.

as for enterprises though, i do not see them embracing the cloud platform as it is now, however there are some scenarios possible right now, and i am sure many more will emerge:

  • a single batch that needs to be processed (amazon now has hadoop services ; also consider load testing from the cloud, some sort of one-off processing, or anything else that is a spike in demand could be good)
  • overflow capacity for something like flash traffic during holidays for retailers – extend your private infrastructure, or certain pieces of it into the cloud and wind down when you are done
  • POC, development – it is hard to test on the same hardware as prod, so roll your stuff out on the cloud and then for prod/uat deployment put it back inside
  • one-off isolated apps that can deal with current state of SLAs and lack of private clouds, or even pieces of existing apps

there is a chance for internal infrastructure folks to learn something from the cloud experience. in most large companies servers and storage have been increasingly provided as a service and billed back to the projects, but provisioning is still not elastic, and utilization is often the usual low numbers. granted, more and more are using vmware to virtualize (at least dev/uat) to increase utilization, but it is a far cry from the experience of the likes of amazon.

the ideas and some of the tech from the public clouds could and should be looked at by the internal infrastructure people, and not just the servers/storage, but also those that provide middleware services – messaging and monitoring, for instance.

clouds in your own data centers? some of the vendors seem to head this direction – give us the servers and the datacenters and we will create cloud infrastructure for you, taking care of management, monitoring, apis, provisioning, etc.

i wonder how the chargebacks in organizations will change between departments – since instead of a capital investment upfront with depreciation we can convert it to operating expense with pay for use billing – would that allow us to win anything? of course, for small companies it is great – no upfront costs, no hardware, etc.

ms azure is somewhat an outlier here – first of all, it lacks tinkerability (yes, it is a term) – the barrier for entry is too high, but at the same time it is not a vertical app, but a full-blown platform with all the development tools in place. the problem is that it is too opaque, and i would imagine it will be a while before the whole kitchen sink has proper api, billing, provisioning, etc working properly – as usual the idea is fantastic, but the implementation, knowing the track record, is something to be wary of. since windows is notoriously hard to automate, perhaps azure will finally sidestep this problem and offer that as a service, but i cannot see how it can completely do it, unless it provides something as easy and transparent as ssh+shell scripts and declarative versioned management for that.

coming up

development in several directions – granularity of service, privacy of the cloud; standards – one must talk about a certain place in the continuum – standards for infrastructure level monitoring/provisioning are perhaps possible, but once we run into more vertical services, standards make little sense, since you are buying into the provider.

there will be consolidation, since, as failures of ibm and sun showed and success of amazon shows, it is a low-margin business, and amazon knows it first-hand.

GoRuCo 2009

Posted by anton
on Saturday, June 06, 2009

gotham ruby confrence is the nyc’s own ruby conference organized by nyc ruby users group. it was my second time attending it. this post is an attempt to organize my own notes, as well as an attempt at feedback that i think i owe to the speakers and the organizers.

i really like the smaller focused gatherings like this with around a hundred-plus attendees – there is definitely a community spirit there, since a lot of the folks are local and already know each other.

there is also more focused discussion, since the background of the people in the room is similar. a lot of the culture is shared, a lot of the values are implied, and the conversation zips along nicely. i have stopped attending local java groups precisely for the lack of this common background – the topics are too broad, the backgrounds are too different, and it takes a lot of effort to communicate ideas.

i am a bit uneasy about my own relationship with the ruby community – i am not really contributing, nor am i doing paid ruby work. i also am a bit weary of the monoculture (look at them macboys and macgirls!) that tends to re-invent the wheel way too often. but i am there for the excitement, for the bright-eyed kids that tinker and create things – this energy is infectious and i feed off it. it is inspirational and energizing.

so why am i going to the conferences like these? in addition to the energy boost i mentioned above, there is also the trivia of learning about tools, projects, approaches; getting the feel for the zeitgeist, where things are heading, what folks are thinking. deep down inside i am always looking for the “blow your mind” experience, something that can turn a familiar topic on its head, something that can make me discover things i have never suspected existed.

GoRuCo had a nice balance – things technical and detailed, and also approaches/techniques/principles. the reject conf at the end – a series of quick lightning talks – was an icing on the cake, stuffing you full of references, pointers, tips that you could take home and work through at your own pace.

Gregory Brown: Where is Ruby really heading?

more of a book report, talking about different versions of ruby out there. for anyone following the community, none of it was a surprise. no hard data either, just his personal anecdotal experience. this part of the talk was more suited to a short user group presentation.

there were a few nice tips and personal war stories related to moving between 1.8.6, 1.8.7 and 1.9.1, unicode, side-by-side installs and very basic crude techniques to code for different versions.

these are typical growing pains – everything from the compatibility issues between versions to the curse of the system-wide install that makes one go an extra mile in order to run different apps under different ruby versions on the same box (i always preferred the semi-structured self-contained java jdk installs and jars controlled by the classpath).

there were some props to jruby (it is a real distro, not a hack to reach out for when all else fails!), mentions of ffi in jruby that allowed gregory’s project to run on windows.

Eleanor McHugh: The Ruby Guide to *nix Plumbing

this could have been a great talk, but it seems like eleanor was really hungover, so instead it was a very disconnected series of ramblings on the general subject of unix and coding. very poor delivery, and at times plain embarrassing.

only towards the end she managed to find the message for the talk, which was “you can code against kernel internals using ruby, since it makes it much easier. do not automatically assume that you need to write in C for performance – try ruby first.”

she also highlighted ruby community’s respect for bare-metal – the ability to tinker, the taste for small simple tools that do the job well, and the affinity for unix. this is exactly what attracted me to the language in the first place.

there were a lot of references that at times seemed like name-dropping: ngnix, beej’s guide to network programming, beej’s guide to unix interprocess communication, ruby/dl, duby, event machine, c10k problem, advanced unix programming book

Dan Yoder: Resource-Oriented Architecture With Waves

dan briefly talked about his waves framework that tries to present a simple DSL around HTTP and resource representation in a REST fashion.

the meat of the talk that was interesting to me dealt with REST in general, its differences from MVC, resource-oriented architecture, self-describing data, returning links to other data inside of data to aid discoverability. he mentioned how a resource identifier (e.g. URL) should not specify representation (i.e. do not add .xml to specify that you need an XML document back), but rely on client’s Accept* headers to negotiate representation (caveat being that CDNs like Akamai do not currently care about these headers, so you will always get the same content).

there was also a mention of the fact that sometimes one can view HTTP protocol as something dealing with a distributed hash table (DHT) using get/put/delete operations (with post reserved for everything else).

RDF and freebase were also mentioned.

i have a strange relationship with REST: i get it on the technical level, i sort of get it on the architecture level, but it did not fully “click” yet, perhaps due to the fact that i do not have enough practical implementations under my belt. i should go through the restful webservice book again and play around.

good talk overall, but lacking concrete examples that could have brought things more into focus.

Jake Howerton: Into the Heart of Darkness: Rails Anti-Patterns

a bit of a disappointment, since the title promised so much. i usually really liked the anti-pattern talks, since you learn nothing when things work as expected – the real learning comes when things break, and you are forced to dig in and figure out why. plus these talks also give you an idea of applicability of certain techniques, which is really a required counterpart for all patterns to begin with (beware of dartboard-driven design).

instead we were treated to a very few amusing short code snippets, but no larger patterns in the sense of fowler’s refactoring book or even rails-specific patterns.

he did mention cucumber, reek and metric_fu and some general well-known testing techniques. i liked his term “flight check” for the smoke tests that run before deployment to prod, and a notion of sandbox test environment where mocks are replaced with real classes that do destructive things without impacting the real world (like sending emails).

i also like his term “irb-driven design” for something that was copy-pasted from the exploratory irb session into the production code.

there was also some treatise of legacy code, but i think dhh’s talk on the subject was much better.

overall it was a fun, light talk that was well-received. i blame the deceiving title and the lack of focus for initial feeling of disappointment; it felt like jake was simply talking about things he does and prefers to do during development, without specific overarching theme in mind.

Sandi Metz: SOLID Object Oriented Design

sandi stole the show; she set the level that none of the speakers matched and showed what it really means to have a solid, gripping presentation that is lucid, focused, well-prepared, and superbly delivered.

she talked about SOLID design principles: Single Responsibility, Open Closed, Liskov Substitution, Interface Segregation, Dependency Inversion which all boil down to managing dependencies.

she went through the fowler’s value of design argument, and then iteratively went through a refactoring example, invoking the SOLID principles along the way.

i think the most impressive for me was how well the whole talk came together, how the arguments were presented to support the refactorings, and how well it got into my head (almost uncanny, similar to the effect the best books in head first series have).

some of the principles she kept bringing up – using the rate of change as the indicator for splitting the functionality; refactor in small steps to let the design emerge – not because you know, but because you want to find the design; red-green-refactor; mock at the seam; only mock classes i own; those that change often should depend on those that change less often.

this once again brings up the importance of developing a language to discuss design (e.g. patterns) – i think this is a crucial step for every practitioner, when something intuitive and personal (e.g. a vague code smell) becomes something that you can articulate and communicate to others.

she also mentioned uncle bob, micronaut, steve freeman and nat pryce mock objects site

Benjamin Stein: Building Cross Platform Mobile Apps with Ruby & PhoneGap

the main promise of PhoneGap is quite compelling – build cross-platform apps on the phones using javascript while taking advantage of the native features (vibrate, storage, accelerometer, sound, gps, etc).

it was a fine presentation, with a story to tell and with a perspective that put all the low-level details in context. makes me itchy to get my hands on some phone development; i was also quite impressed with their adoption of latest standards (e.g. HTML5).

Yehuda Katz: From Rails to Rack: Making Rails 3 a Better Ruby Citizen

rails3 perspective from the horse’s mouth – quite detailed look at how rails is trying to be less opinionated in its choice of frameworks to work with, and how it exposes its internals for others to integrate with (orm, rack, js frameworks). some very interesting examples of design decisions and the overall future path of the framework.

this was a bit tedious, but quite informative talk.

Lightning Talks

i really like these, since most people do not have enough material for a long talk; in fact some of the main talks earlier should probably have been half their size.

the highlights include sunlight foundation and data.gov plug, a great pair of fast talks by aman gupta on google-perftools that he tweaked to work with ruby and used on some real-world code and joe damato on tweaking the thread performance in ruby 1.8 (in retrospect, these two talks were something i wish eleanor would have done).

michael stonebraker

Posted by anton
on Tuesday, May 26, 2009

i really enjoyed this talk by michael “map reduce is crapstonebraker. it is a terse and mind-blowing romp through the current database landscape, focusing on inability of one-size-fits-all “dinosaurs” to deal with streaming data (like tick market data) as well as data warehousing. he has some benchmark numbers and proposes some alternatives.

i have been looking into streaming databases and column data stores for a while, but I am yet to have a production hands-on experience with them.

i always had somewhat of an inferiority complex when it came to databases, since these guys have done so much work decades ago that tackled so many problems, and it seems to have had a hard time escaping the database world and affecting CS population at large.

so this new twist in the evolutionary spiral that comes due to the availability of cheap machines, lots of RAM, and the need to process lots of data should yield interesting results. i do hope that this will lead to a whole bunch of lean and mean specialized solutions instead of (loosely quoting stonebraker), “here’s oracle, what’s your problem?” approach.

the essential drucker

Posted by anton
on Monday, April 13, 2009

perhaps it is my destiny to stumble into things that i should have known (and probably have known at some point, but they only became relevant now). in any case, i am happy to add drucker to my pantheon of striking bald men (already staffed with the likes of foucault, seth godin, and crowley).

i really enjoyed this updated/revised “best of” collection of drucker essays. he is not instantly quotable – i do not think he goes after the effect of catchy parables, so it is hard to pick out the pearls to quote here.

but of course, i cannot resist:

[...] the three stonecutters [...] were asked what they were doing. The first replied, “I am making a living.” The second kept on hammering while he said, “I am doing the best job of stonecutting in the entire county.” The third one looked up with a visionary gleam in his eyes and said, “I am building a cathedral.”

perhaps proverbs and fables are a retreat for those that cannot come up with an original wording themselves, but dammit, this is the kind of stuff that you hit people over the head with to get your idea across.

anyway, i really liked his writing on building innovation into the organization. his essays on the place of the organization in the society were also quite eye-opening, especially with respect to the non-profits. he does not go into heavy theorizing – everything is quite high-level and appears simplified, but there are a lot of good observations, especially once you read them in context. yes, some of it is a bit dated, but it is easy to make a quick mental adjustment to see the core of what he is getting at.

i’ll have to pick up a few more of his books; at this rate i will be reading f. w. taylor in no time. oh boy.

manual override

Posted by anton
on Thursday, January 29, 2009

there is something i noticed about myself – oftentimes, when faced with a problem, i immediately jump to the technical means of solving it, which usually implies building some sort of automation. after that jump, the initial problem becomes less and less important, while i am speeding away, thinking through the technical details.

it turns out that sometimes an excel spreadsheet and a manual process is far superior solution, although it might hurt your ego and make you feel vaguely dirty.

another variation of the above is fascination with “cool” tech (for your definition of “cool”) – i noticed that sometimes i would use every opportunity to startup wireshark or Process Monitor and gloriously wallow in data, gleefully probing tiniest details. it turns out that almost always there are better tools out there that can get the job done with less noise.

so as a personal first-line defense, i opt for trying to ask the right questions early on – perhaps a manual solution would do, or a problem really is a non-issue.

it seems that too much knowledge may be a curse, but the real challenge is to keep your personal toolbox organized.

elevator test

Posted by anton
on Thursday, January 29, 2009

building software for a business, i established an interesting test that shows me whether i am thinking in the right direction:

off the top of your head name five most important things that technology can improve for the business as it is right now

i’ll wait – can you do it? right here, on the spot?

the way i see it – if, after having worked in a new team for a few months, you cannot do it – you have failed. you are not thinking about the business, you do not know the business, you do not care.

as technologists we are all too often drawn to the technical problems, forgetting that it is really the business that we should have in mind. the catch is that the domain of the software problems is infinite, so one can submerge forever without delivering anything of value to the business.

disclaimers apply, of course – it particularly matters for those developers working closely with the business. but then your definition of “business” might vary; the point is that sometimes a correct technical solution is the wrong one.

so while you are at it, type up this list and stick it somewhere visible, so that you can have it in front of you every day (as i’ve talked about it before).

Public Enemy No.1

Posted by anton
on Wednesday, January 28, 2009

Alex Miller aka Pure Danger Tech has a great entry on most common concurrency bugs.

the timing is perfect – this very thing bit me today. long story short, an old app, previously perceived to be multi-threaded, was recently converted to actually be multi-threaded, and then, once traffic ramped up a bit, exhibited peculiar behavior when perfectly good dates could not be parsed. thank god it blew up, as opposed to quietly corrupting the data.

so something as innocent-looking as private static final SimpleDateFormat declaration was the culprit: java.text.DateFormat is not thread-safe.

luckily, it is easy enough to spot and reproduce (threadPoolSize and invocationCount in TestNG simplify it even further).

a pessimist would heave a mighty sigh, once again swear on the copy of JCIP to find and root out every frivolous static out there with the help of FindBugs or a simple regex.

meanwhile, there is joda-time and a promise of jsr310

but of course this whole experience still leaves you feeling cheated and dirty – why, god, why?! something so function-like and stateless in nature insists on stowing things away.

go with the flow

Posted by anton
on Monday, January 05, 2009

a damn nice quote from Eric Brechner’s I. M. Wright "Hard Code" blog regarding team dynamics, specifically about retaining people:

[...] the best way to deal with turnover is to expect it and embrace it. How? Think flow, flow, floooooooow.

Think of your team as a river instead of a lake. A lake stagnates. There’s no energy or impetus to change. The same is true of groups that stagnate. They cultivate mediocrity and complacency; they abhor risk. A river is always running and changing with lots of great energy. You want a river.

a nice metaphor and it strikes a chord – building an ideal team is one thing, but perhaps it is more about building a self-sustaining culture when individuals might be coming and going. the latter seems to be a more important goal in the long run.