i noticed an interesting phenomenon – when talking to a few recent grads or those that are about to graduate, i realized that a lot of the smarter ones are disillusioned about programming.

i find it interesting, because i went through the same transition. having done some pretty cool practical stuff at school, i got quickly bored at my first job, thinking that there was not much new and interesting stuff for me to do.

at one point i recall flipping through the Pragmatic Programmer book right after it came out and dismissing it as something that was too obvious to pay attention to.

i view it as a key telling sign now. this was my own attitude at the time, and i see it again and again in new hires fresh out of college.

being a smart kid, you (perhaps subconsciously) try to assert your own superiority, and perhaps being somewhat defensive, you are all too quick to nod in agreement and move on the moment you think you grasped something.

it turns out that when i go back and re-read some of the books i thought were full of obvious stuff, i discover something that i have learned much later through my own mistakes.

maybe this is the cursed fate of these books – you either read them when it is too late and you wish you read them earlier, or you read them too early and you wonder what all the fuss is about.

what i personally found helpful in this case was the “beginner’s mind” stance – embrace the fact that you know nothing; be humble and view everything that comes your way as a learning opportunity.

which brings me to an opposite learning dysfunction. my first year of university had cryptography, compiler theory, computer languages, two semesters of discrete math, data structures and algorithms, theory of computation, etc, etc (and this was just the CS stuff). it was simply too much for my little brain, so i employed FIFO / LRU eviction approach – without practical application these concepts were promptly forgotten after i passed the exams.

only years later i painfully learned that all of that stuff was actually relevant and important and used in practice. i went back to grad school to patch up the fundamentals and re-learn all of it again at my own pace.

this time around, having done practical stuff, and having established a much broader picture of the CS field, i could finally see how it all connected, and how this knowledge was used in practice.

maybe all of the above is irrelevant to everyone else and could be attributed to my own learning disabilities. however, having worked in corporate IT for past 10+ years, i think there are lessons to be learned for everyone, since those that come out of school have to spend their first year or two learning on the job.

most schools teach computer science which is not the same thing as software engineering. the problem is that the industry needs more software engineers – those that can build everyday quality software and make sure it runs and actually does what the business wants it to do.

sometimes the stuff that you learn in school is directly counterproductive – you work alone and get your own grade; most of the time your work is over once a particular problem or concept is implemented; no one evaluates the quality of design, no one cares whether your stuff is maintainable, or uses version control, or has repeatable builds, etc.

it is up to organizations to make sure that new hires get trained – a mentor/buddy, a stack of books, a pile of wiki entries to get started. it is an investment that requires time and management support.

as for the new hire – be a professional in your field, strive to join the 5% and have fun doing it. otherwise you will be one of those 9 to 5 folks cracking “it is case of mondays” jokes and doing the same thing year after year (well, maybe not, but this is my own personal hell scenario). the field is changing so fast, that you have to keep learning to stay relevant. think of it as climbing the escalator that is going down – the moment you stop climbing, you are falling behind.

there is so much cool stuff out there that is IT-related, and each year it is only getting more interesting. what we need right now is more good engineers with business aptitude to bring in a much needed rigor and professionalism to the field of business software. there is a lot of stuff to do, and hopefully we’ll be able to communicate it to the newcomers.

i have been wrestling with some legacy code recently. beating it over the head with a copy of Working Effectively with Legacy Code did not do much, so i started writing tests to see how it worked.

after a few minutes of waiting for eclipse + junit4.3.1¹ combo to load, i furiously coded a bunch of tests, and then realized that i could not make them fail. essentially it boiled down to the following:

assertEquals(1.0, 1.1);

...which quietly and happily passes. wtf?!! these are two doubles, just compare them and let’s move on with our lives!

then after a bit of thinking i recalled that most of my tests that involved doubles were written on projects that were still on jdk < 1.5, when method above simply won’t even compile, alerting me to the fact that junit expects assertEquals(double, double, delta), where delta is the precision you need.

why they couldn’t simplify my life by creating a couple of Double instances and calling equals() on them is beyond me. this is what i would want most of the time anyway.

since i was running tests under jdk1.5, autoboxing kicked in and now we have two Doubles on our hands. fine, this should not be a big deal, simply call double1.equals(double2) and be done with it – what’s a big deal?

but nooooooooo, check out this little bundle of joy:

private static boolean isEquals(Object expected, Object actual) {
    if (expected instanceof Number && actual instanceof Number)
        return ((Number) expected).longValue() == ((Number) actual).longValue();
    return expected.equals(actual);
}

what the hell?!! you correctly detect that this instance of Double is an instance of Numeric, then take its long value and throw fractional part out. quietly. so all my tests never even squeak. why?!!!

an obvious approach is to suck it up and appease the api:

assertEquals(1.0, 1.1, 0);

this will work. but damn! my eyes!!

TestNG

we all know that junit is legacy, and version 4 was an afterthought, so let’s see what testng does. the api is the same, and under jdk1.5 my primitives get autoboxed into Doubles and assertEquals(Object, Object) gets called:

public static void assertEquals(Object actual, Object expected, String message) {
    if (expected == null && actual == null)
        return;
    if (expected != null && expected.equals(actual)) {
        return;
    } else {
        failNotEquals(actual, expected, message);
        return;
    }
}

voila! this is exactly what i expected. and guess what – it actually works and correctly fails the original test.

Lessons

• man, this alone would scare me away from junit in favor of testng
• make sure your test fails before it ever works!
• what really frightens me is how many more of these quiet autoboxing errors are lurking out there. what previously would be caught by the compiler now silently works in unpredictable ways
• yet another thing to be aware of when upgrading your app from jdk1.[34] to jdk1.5+

Note

this junit bug has been fixed in junit4.4 and up

Ruby

interestingly enough, on a few skunk works projects this year i cobbled together a bunch of existing libraries with ruby glue and used (j)ruby’s Test::Unit (which conveniently comes with ruby distro) and rspec for testing the result.

it definitely makes sense for a project that is written in (j)ruby that uses many existing java libraries; it might be a bit of a mindset shift for a java project that is looking for simplified testing. for now i am keeping an eye on projects like JtestR

¹ default version of junit that ships with eclipse 3.3

face the enemy

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil¹

continuing the topic of my previous post (do not deploy black boxes, allow for measurement, metrics, monitoring), i offer you my two rules of performance optimization (heavily optimized for enterprise environment²).

rule #1: don’t do it

i’ve seen more crimes committed against software in the name of performance optimization than for any other reason³.

don’t bother with performance optimization – deliver working software first.

instead of spending time salivating over sexy distributing caching algorithms and debating the merits of lock striping approaches, implement by feature and most likely you will find out that performance is good enough.

a fitting quote from Refactoring:

The secret to fast software, in all but hard real-time contexts, is to write tunable software first and then to tune it for sufficient speed.

i know it is hard to resist the glimmering image of a performance superhero squeezing out a dramatic 100000x speed increase. we are all guilty of dreaming about it. face it – you are not writing hard real-time systems. you’ll just have to buckle up and stick to implementing business functionality without getting to play with those exciting computational problems.

finally, know your requirements upfront – what latency can you actually tolerate, does it even matter? what throughput do you need? make sure you have the actual numbers; both average and worst case scenarios. guess what, in many cases it turns out that you do not need to optimize to begin with.

after all, performance optimization is always about trade offs – leave your options open.

rule #2: don’t do it blindly

measure, then optimize. most of the software spends most of its time in just a fraction of the code – the “hot spot” – find it and optimize it away.

having a well-factored program leads to hotspots that are easier to isolate and optimize.

way too often i see people diving in and tweaking things left and right, just because they think they know where the problems are.

at best you will waste your time, but most likely you will actually make things worse.

if you live in the java platform world, you are in luck – there are so many tools out there for modern JVMs – use them!

notes

rules are meant to be broken, but i’d rather overreact upfront to discourage frivolous optimization.

yes, yes – you have to have basic knowledge so that you do not do stupid things all over the place and end up bleeding to death from a thousand cuts. luckily, in enterprise software most of these things are very basic – a few language rules, and a few design rules – good software developer will follow them automatically.

why performance optimization is so alluring?

here’s my take on it – enterprise software is boring. you’ve done it a few times, and you do not feel like cranking out the same stuff over and over again.

so you start creating complexity to entertain yourself, to give your mind something to chew on: you fall in love with design patterns, you build beautiful multi-tier distributed designs with transactional semantics all over, and you fiddle with performance optimization on every step.

most of us have gone through it; it is like a childhood disease that you suffer from in order to become immune. most of us survived and gained valuable insight in the process. it does take a bit of self-reflection and experience to realize this though.

(those that did not survive ascended to the stratosphere and became raving zombies – and we all know what to do with zombies).

i think the key is to understand that although enterprise software is boring, the vast majority of all software projects still fail. this is where the true complexity is – figuring out how to deliver working software that customers actually use. not to mention doing it on time, on budget, and without burning your team.

this is much harder and at the first sight a lot less sexy, but there are still plenty of technical challenges to work through in order to create well-engineered systems. it is just your definition of “well-engineered” that has to change.

once you re-adjust your focus, the work is cut out for you.

it is tempting to pitch “enterprise” software against the opposite “swing” championed by the pragmatic programmers, rails, and the whole community around it. and of course, it is not the technology but the mindset.

¹ donald knuth paraphrasing hoare

² with apologies to m. a. jackson

³ with apologies to w. a. wulf

i am still on the lookout for quality podcasts. one of the series i’ve been really enjoying lately is silver bullet security podcast by gary mcgraw.

i am a bit of a fanboy – i’ve got all three of his software security books, and i have been following him for a while – listening to the interviews and reading the articles.

i go through a lot of podcasts on regular basis, and “silver bullet” is one of the most impressive ones. first of all, the sound quality itself is pretty damn good. having suffered through some badly-recorded, non-normalized audio in the past, i am happily listening to articulate, passionate folks that actually sound good. perhaps the sound quality it is due to gary mcgraw’s music career.

secondly, he manages to pull together an impressive array of people that have proven themselves over the years in the field of security. in most cases they’ve been at it for decades, and it shows.

since security is a relatively new field, most of his guests come from different backgrounds, which makes interviews all the more interesting. gary mcgraw himself did his undergrad in philosophy, while his phd was in CS and cognitive science under douglas hofstadter (in fact, gary mcgraw contributed a chapter to Fluid Concepts and Creative Analogies).

having read his books, there is not much new in the podcast, but it really helps to clarify and reinforce the basics as well as acquire the vocabulary to be able to discuss these topics. this is why i am on my second round re-listening to the podcast.

one of the main insights i got from listening to the podcast came from its title. “silver bullet” is the famous essay by brooks where he argues that there are inherent complexities in software development that will not magically disappear by introducing better tools or practices.

security is complex. there are accidental complexities, that could be (and are) addressed – get rid of languages with memory overflow problems, introduce address space layout randomization, use static analysis tools, etc.

at the same time there are inherent, essential complexities to security that will always be in place due to new applications that create new attack vectors simply by the virtue of their functionality.

thus it is logical to see security overlapping with QA (QA being quality assurance as a process, which deals with ensuring that quality happens, as opposed to testing which is simply one aspect of it). given this approach, a lot of things about security fall into place: “security means risk management”, “security is a process, not a product”, “there is not absolute security”, “security faults are quality faults”, “it is harder to build a secure system than to break one”, “one cannot sprinkle some magic security dust at the end of the project”, usefulness of “badness-ometers” as gary calls them, etc (these one-liners have such a nice ring to them and stimulate thinking in the right direction; they are butchering the contents of the books though, so read them for the facts!).

since podcast series are not that technical at all (although most of the participants are quite hardcore hackers), i would wholeheartedly recommend them to the broadest audience.

i really like this idea of starlets grouplets (as used by googlers) – informal gatherings of people interested in the same topic and willing to dedicate their 20% “extra-curriculum” time to it.

they do not necessarily come from the same team (in fact most of them do not), but they come to work on something together. for some of them it might even become their “main” project.

there are probably half a dozen (at least) small-ish projects at work that i would like to hack on – from an api to a commonly used tool to some improvements to existing tools. this is the stuff that has been created and is maintained by other groups elsewhere, but i have the energy, the ideas, and the expertise to contribute.

the unfortunate thing is that right now it is pretty much impossible (at least with the projects i am interested in). it is not necessarily the direct managerial support – although it would make it easier. i think for motivated people this support does not matter as much – most of the stuff like that i’ve done on my own time as a skunkworks project.

i think the biggest roadblock in my case is the internal culture – people just do not think this way. their first reaction is suspicion and thinly veiled irritation. i understand that this is probably the result of many incompetent people bombarding them with impatient requests, but wouldn’t it be nice, if i simply could check out some code, tinker with it, and then submit a patch? and have someone answer my question or two? it would immediately help some of my daily activities, and benefit others. sort of like the way open source projects are – scratch my own itch and help others in the process.

perhaps, one of the first steps is to make it easier – open up source control, do not require one to chew through the permissions/requests muck to get access to code/environments, start an area on an internal forum (you got one, right?) and an area in an internal wiki (you got one, right?).

most of it will benefit you, but also allow others to come in and contribute. in fact, most of the stuff that can be done is simply a copy of best practices by open source projects.

speaking of motivation, this article by esther derby that muness pointed out to me a while ago is quite appropriate. here’s a quote:

Most people show up for a new job with high motivation. They’re excited and they want to do a good job. But as the weeks pass, motivation dribbles away. It’s not because managers are failing to motivate these once-enthusiastic people. It’s because organizational systems, policies-and yes, management actions-actively demotivate people.

sadly, i’ve seen this happen over and over again.

this time with the newcomer is quite special – take advantage of it (or cultivate it, if possible). they have a perspective and the motivation that might disappear later. they can bring in the new ideas, point out something you’ve never thought of, or simply challenge the status quo. this sort of disruptive energy is very healthy and should happen on regular basis… just stop demotivating me! (sorry, i could not resist – the article title is way too catchy)

well, actually, excitement matters.

in particular, how do you motivate people to contribute; to write that dreaded documentation, for instance?

perhaps one thing worth trying is appealing to the geeky side and giving them(us) toys tools that tap into intrinsic motivations.

in my experience it turns out that if output looks pretty and is immediately available (and especially instantly viewable by others) – people are motivated to tinker and create. in fact, this is precisely the stuff that made internet happen.

to reiterate – instant turn-around and aesthetically pleasing results will take you a long way. probably somewhat related, but better worded – optimize for happiness.

this post is brought to you by the scars and bruises acquired by yours truly as i crawl away from a slow and beaten up twiki instance (that looks like something designed by that stoic mosaic folk back in the day; man, i can hear the sound of knuckles scraping the floor).

great quote from ward cunningham in response to steve mcconnell’s post on technical debt:

I would also sometimes explain refactoring to my management as follows: “We tried to add the feature to our application but found that there was no place for it. So we first made a place, and then added the feature there.”

currently i have a privilege to work with sybase 12.5. perhaps i am spoiled with the ease of mysqldump db_name [tables] or echo .dump [tables] | sqlite3, but i expect any modern database to have a scriptable way to dump schema for selected tables in create statements, as well as data in insert statements that simply could be piped back when needed.

while dumping data is easy enough using bcp, scriptable schema extracts are a bit trickier (especially if you want it to be cross-platform).

but we are lucky – it took sybase only 20+ years to introduce a command-line utility written in java called ddlgen in version 15 of its flagship enterprise product (in my case it worked against 12.5 as well).

ddlgen

• rip it out from freely downloadable developers’ distro of sybase

$ ls -lR c:/programs/sybase/ddlgen/
c:/programs/sybase/ddlgen/:
    ddlgen.sh
    lib/

c:/programs/sybase/ddlgen/lib:
    DDLGen.jar
    dsparser.jar
    jconn3.jar

• rig up the wrapper script:

$ cat c:/programs/sybase/ddlgen/ddlgen.sh 
JAVA_HOME=c:/programs/java/jdk/jdk1.6.0_03
LIB_DIR=`dirname $0`/lib
CLASSPATH=$LIB_DIR/jconn3.jar:$LIB_DIR/dsparser.jar:$LIB_DIR/DDLGen.jar

$JAVA_HOME/bin/java \
-mx500m \
-classpath `cygpath --mixed --path $CLASSPATH` \
com.sybase.ddlgen.DDLGenerator $*

backup scripts

• schema-out.sh

source env.sh

[ ! -d $OUT_DIR ] && mkdir -p $OUT_DIR

for table in $TABLES; do
    out_file=`cygpath --mixed --absolute $OUT_DIR/${table}-schema.txt`
    printf "dumping $table schema to $out_file... " 
    $DDLGEN -U $USERNAME -P $PASSWORD -S $SERVER:$PORT -D $DATABASE -TU -N$table -O $out_file
    printf "done\n" 
done

• bcp-out.sh

source env.sh

[ ! -d $OUT_DIR ] && mkdir -p $OUT_DIR

LOG=`dirname $0`/bcp-out.log
cat /dev/null > $LOG

for table in $TABLES; do
    out_file=`cygpath --mixed --absolute $OUT_DIR/${table}-bcp.txt`
    printf "dumping $table to $out_file... " 
    bcp $DATABASE.dbo.$table out $out_file -c -t, -S $SERVER -U $USERNAME -P $PASSWORD >> $LOG
    printf "done\n" 
done

• schema-in.sh

source env.sh

LOG=`dirname $0`/schema-in.log
cat /dev/null > $LOG

for table in $TABLES; do
    in_file=`cygpath --mixed --absolute $OUT_DIR/${table}-schema.txt`
    [ ! -f $in_file ] && echo "$in_file does not exist for $table, skipping" && continue
    printf "loading $table schema from $in_file... " 
    isql -S$SERVER -U$USERNAME -P$PASSWORD < $in_file >> $LOG
    printf "done\n" 
done

• bcp-in.sh

source env.sh

[ ! -d $OUT_DIR ] && mkdir -p $OUT_DIR

LOG=`dirname $0`/bcp-in.log
cat /dev/null > $LOG

for table in $TABLES; do
    in_file=`cygpath --mixed --absolute $OUT_DIR/${table}-bcp.txt`
    [ ! -f $in_file ] && echo "$in_file does not exist for $table, skipping" && continue
    printf "loading $table from $in_file... " 
    bcp $DATABASE.dbo.$table in $in_file -c -t, -S $SERVER -U $USERNAME -P $PASSWORD >> $LOG
    printf "done\n" 
done

do i feel silly? yes. do i feel petty? yes. does it make me feel better about myself, given that the sybase DBA told me to contact dbartisan support to see if i could script their tool to do this? oh yes.

background: 10K+ compiled class files and sources that got out of sync¹; need to figure out which sources are valid, and which ones are not.

decompiling things is the last resort, since sources produced are not easily diff‘able against the sources you’ve got. the likes of diffj is not much help either, and i did not even want to go down the rabbit hole of normalization through obfuscators.

so if you do not feel like wielding antlr or javacc to normalize two sources, the obvious approach is to simply recompile and compare with the existing class files (just beware of missing class files that might not have any sources at all).

however, keep in mind that javac by default includes line number table in the class file it produces². this means that even if you added or removed a line of comments or even a blank line before any sources, it would result in a classfile that is different from the original.

sometimes you have another class inside the .java file (not to be confused with inner classes). in this case it gets compiled into a separate class file. so if your main class’ source code has changed, it will affect the line number table of the other class as well. this means that even though another class’ source has not changed, its generated class file will be different.

in my case i also had to check which jdk compiler produced the class files. one can always opt for javap that prints minor and major versions, or if you are feeling manly enough, whip out your favorite hex editor and check bytes 6 and 7 (according to the vm spec). in general, javap is the easiest way to check the internal structure of the class file.

finally, to diff files and directories i simply used svn – check the originals into the local repo, then put your stuff in a working copy – it will do the rest. after all, this is what it’s good at.

oh and why CAFE BABE? look it up

¹ this is a whole different and interesting topic – how any sort of generated content creates a possibility of this disconnect. all these xdoclets, jaxb-generated sources, and even compiled classfiles create artifacts that now have a potential of getting out of sync with the sources. yes, proper engineering practices mitigate the risks, but all things being equal, i like the fact that with scripting languages this problem is largely non-existent. what you see is what you run.

² you could always run javac with -g:none to get rid of line numbers, but it was no help in my case.

Only focusing on 50X just gives you faster Elephants, not the revolutionary new breeds of animals that can serve us better.

(by werner vogels)

if you enjoyed everyone’s favorite upside-down-ternet way of making new friends, this whimsical bit is right up your alley.

it is based on cross-site request forgery (CSRF) attack.

briefly, these are the attacks that trick you into submitting a potentially damaging request to the application you are logged in to. so if you receive an email with a link to http://www.google.com/setprefs?hl=ga, which you press, it will set your google language preferences to irish.

thus you could try to impress those inquisitive souls looking for things on your site with the following apache config directive:

RedirectMatch \.(php|phtml|phps|php3)$ http://www.google.com/setprefs?hl=xx-klingon

therefore any request to a booby-trapped url on your site (in this case anything that ends in php) would set their google search language to klingon.

(stolen from here)

of course, it does not have to be an explicit server-side redirect – similar behavior can be triggered with javascript, iframes, etc.

how do you protect from it? the app has to use unique tokens in the form presented to the user (or one can start lugging around those encrypted URLs again – anyone remembers IBM’s Net.Commerce?)

since i am (somewhat reluctantly and half-asleep) reading gibson’s latest, and since these days i mostly appreciate him for sensing the zeitgeist and popularizing new art forms, i cannot shake off the feeling that there is an art piece lurking in here.

looking back i am surprised at how little importance i saw in the process of interviewing the interviewer.

it’s pretty simple – for me the main criterion is how good the people are that i will be working with. they have to be better than me, and i have to be able to learn from them. once this is in place, everything else does not matter as much.

it would be ideal to work on a great project, but even given a bad project, good people can make it worthwhile.

how do you recognize great people? well, i wrote about it before. everything else is icing – useful, but not a replacement for the good people.

ideally, you already know them – user groups, conferences, friends, books, mailing lists, or you simply worked with them in the past. this is how it should be – you are setting out to work with people you look up to on the projects that interest you.

disclaimers galore: mid size to big company, certain amount of evilness is expected and tolerated for a greater goal. this is mostly a memory crutch for myself (i actually had it written down and would whip it out during the interviews).

personal

• how long have you worked here
• why did you come to this company
• what is some of the cool stuff that you are proud of
• why do you like it here
• what do you dislike about working here (blunt, but you’d be surprised what people blurt out)
• your background (education, previous companies, interests)
• top challenge you are struggling with right now

i had a prospective manager interview me once, and he kept telling me how boring the job was, and how stupid the users were, and how underpaid i would be. while i appreciated the frankness, i could not believe that he expected me to take up a job after such introduction. the whole concept of marketing your company to a prospective employee apparently never even entered the picture.

working conditions

• make sure you see where you will sit (i know i was ashamed to show prospective employees around; at least we had aeron chairs!)
• developer setup – monitors, machine, software installs, certain tools
• what does it take to get software (purchased, installed)
• do you get admin rights on your machine. can you install wireshark/vmware/cygwin/firefox and plugins/etc

team

in general i am after gelled teams – those that know each others’ strength and weaknesses, those that figured out how they can work together and get stuff done. those that simply have fun

• i want to talk to people i will be working with (often people from other teams interview you, and then you end up working with another set of people you’ve never talked to)
• collaboration – does everyone hide their little piles of documents on shared drives and guard them, or is it wiki, company-wide forums, at least sharepoint for the team? how do people share knowledge, within the group and between the groups?
• team culture – everyone for themselves or are they out to learn, help, share
• does team eat lunch together (i found it a pretty good indicator of how well people gel, and i always tried to get people to have lunch together at least once a week)
• day in life

org

• org chart – make sure you know who you are reporting to, who is your boss reporting to, where you fit on the org chart. it might seem silly, but if you are out to get stuff done and you work for a bigger company it matters what your actual title is. after you are hired it might turn out that you have landed at the bottom step of the ladder. that shiny title you got was just that – a title, meanwhile it is your actual pay grade that drives the salary and to an extent how much clout you get. once you are hired, it is much, much harder chew your way through, because then you are subject to the usual organizational inertia, policies, etc, etc. negotiate upfront!
• schedule flexibility (can i work from home?)
• do you expect me to work weekends
• how many hours is the team currently working
• dress code
• what does it take to get stuff done (get a new app/tool up and running in prod, roll your own monitoring, push through an idea, etc)
• mobility within company (in most places HR has a policy that you have to work for a year or more in the position you have been hired for before you can move; is mobility tolerated? encouraged? do they care?)
• growing people – training (books, classes, conferences, tuition reimbursement)
• top org problem you’ve got right now (they might balk, but they might reveal something interesting)
• turnover rate (could be very telling) – how long each of the person you talked to have been with the company, how many people leave
• career path – where can i go, what can i do, how can i move within the company, what are the options, what is the organizational chart like, what is being done to help understand it and climb it. i do not quite trust overly formal you-do-this-you-get-that approach (see Steve McConnell, for instance), but it is helpful to know where you stand and what it takes to grow

in addition, i really like the idea of mentorship – assigning a mentor to a new hire. ideally it happens naturally (in a nurturing and caring team), but it would help to make it a recognized practice. it would be someone with significant experience in the industry and in the company; this person will help to “bootstrap” the new hire (in the ways HR never could) as well as help them navigate the company and grow. having a fellow employee in this role as opposed to an HR generalist will be much more effective.

software development cycle

• source control, automated builds, tests
• qa people
• what sort of testing is done
• how are the requirements gathered and tracked
• how are defects and enhancements gathered and tracked
• how is development organized – timeboxed iterations? waterfall? chaos?
• infrastructure – how much access do you have, who runs it
• monitoring
• automated deploys
• how are releases promoted to prod
• support (be careful here – it might end up in a rabbit hole into the never-ending infrastructure improvements and tweakage – late-night pager fire drills and other joys. i believe in eating your own dog food when it comes to running your own stuff, but this is different from looking after the apps that someone else develops. if you are after a developer’s position, clarify the amount of support the team is doing currently and expects you to do)
• are people proactively fixing things and constantly improving, or is that discouraged (heavily regulated complicated development, throw over the wall support (especially if it is offshored), people just do not care, etc)
• what happens at the end of the project (retrospective? what happens to the team? i want to see the team gel and remain together for the next project)
• code reviews (encouraged? non-existent?)
• who maintains the app once it is live
• how are standards created and maintained
• how are best practices disseminated (arch review board with astronauts? hands-on architects? workshops? training?)

current project

• make sure you understand what you are being hired for – that new spiffy project might never materialize
• where will this project be in a year (could it be one of those “pump and dump” gigs, where you are merely a body to be used and disposed of at the end?)

other

• what are you looking for in the ideal candidate

this last one is interesting. it took me a while to realize that (surprise, surprise!) i simply might not be someone they are looking for.

i noticed that being interviewed puts me in a mode where i am trying so hard to be liked, to please the interviewer. this is certainly natural, but quickly becomes evil, if not held in check (“yes, i would love doing 24/7 support for that cobol app written by mental asylum patients during their work therapy course twenty years ago”).

it is simply counterproductive for both of us. yes, this is a job they’ve got, and yes, it is important, and someone has to do it. indeed, i could do it, just like i could do a myriad other things, but i am after something else. this is perfectly OK, unless you are desperate, of course. and this is where the trick is – know what you are worth and always remember what is important to you and what you are after.

a quote from marc andreessen:

most people who do great things are doing them for the first time

this is from an excellent “The Pmarca Guide to Startups” series on his blog

this is a rant, inspired by working in both developer and admin roles over the years (i strongly believe in “eating your own dogfood” when it comes to building and running the apps, but this is a whole different topic).

my experience is that given a choice of manageability/logging/monitoring vs. extra performance i will always choose the former. the amount of time spent troubleshooting performance and stability issues on live application in production trumps any hardware (and sometimes even development) costs.

so instead of satisfying your inner ricer and deploying a highly-performing black box hotrod, spend the time to put the probes in, make it declaratively manageable; if your OS/hardware provides any sort of isolation and partitioning – consider it; take advantage of existing platforms and tools.

take Tibco BusinessWorks for instance. besides having their own suite of monitoring/management tools they allow (perhaps serendipitously) individual “worker engines” to be deployed in separate JVMs (which could be on different machines) so you can analyze and manage them using not only an existing ecosystem of Java tools, but also fall back on your regular OS tools – per-user, per-process, per-box.

the benefit of this simplicity becomes obvious once you have worked with apps that insist on packing everything into one JVM – worker engines, daemon-like processes, queuing, etc, etc. management and tuning becomes a nightmare; on a flipside it is guaranteed job security and high salaries.

so what can a developer do? besides the obvious, consider an api to talk to your application and tweak it as it is running (look at those ol’ smalltalk dudes), or better yet – a command-line scriptable console that exposes your app’s domain. here’s props to bea folks – their flagship server product for years had a python (jython to be exact)-based console that allowed one to connect to a running cluster and make changes to it on the fly. similar functionality is provided by rails stack, although technically you only get connectivity to the database, not the actual running application instance. still, it is a big step.

another tip of the hat in bea’s direction – their JVM for years had actually usable manageability tools; sun was late, and even when they started delivering them, the tools were really clunky (i am still waiting for something similar to jrcmd tool from sun that allows me to do simple things like collecting threaddumps from a jvm on all platforms, including windows and redirecting them to a given file, since jvm might be running with stdout sent to /dev/null). bea’s mission control has been around for a while in various forms – i want to be able to attach to my production JVM and look at GUI representation of memory distribution, object counts, stack traces, heap info; but on top of that it gives me an ability to explore and act upon exposed JMX beans both from the JVM and the app, set up triggers and alerts that start recordings, memory dumps, send emails, etc. this becomes indispensable, especially for hand-me-down apps or third-party software.

this is actually a big change in mentality – gradually people are realizing that they should be able to monitor stuff in production, live, as it is running. hence we have things like (under appreciated) dtrace, and more and more investment into the platforms that support that sort of runtime lightweight dynamic analysis. these days it is expected that apps should be on 24/7, and ability to dynamically redeploy things, reconfigure things, analyze things is crucial.

finally, i have seen way too many folks that consciously refuse to learn about how their code runs – the minimum about the OS, the network, the tools, the tuning. i am willing to consider and understand, as long as they have those that do know around. sadly, too often this responsibility gets shifted to the OS admins that could not care less.

all sorts of disclaimers apply – in many cases the apps are so small that one can pile them together and forget about them. the apps that will benefit most from the manageability stuff mentioned above are the ones that churn through a lot of data and have pretty strict uptime/latency requirements. in addition, it is assumed that there are a lot of people working on them, so tools and approaches should be somewhat uniform.