Quality Assurance

After the New Year my current project entered its testing phase. I wrote in my last post that my QA policy was an Excel spreadsheet with check marks. That’s exaggerating slightly, but I’ve learned very quickly that our first couple attempts didn’t work too well.

There’s been two issues that quickly spring to mind:

Reappearing bugs

We’ve had some success managing change requests with our issue tracking system. The flow goes like this: a tester has a problem with the interface or something breaks, and enters the issue into the issue tracker. Our business analyst vets the issue against our current specification. Change requests are logged as meeting agenda items; spec violations are assigned to me. I either fix the bug myself or assign it to the responsible developer. The bug is fixed, and the business analyst verifies it has been fixed on our test server. So far so good.

What the consultants and best-sellers don’t tell you is that bugs have a nasty habit of re-breaking. A bug tracker can tell you that a bug was fixed at one point in time, but it has no information as to whether a bug that was fixed two months ago is still fixed. Software components behave in ways that are difficult to predict when integrated together. Poorly designed components also tend to crumble as features are added to them as well; about 75% of our reopened bugs were the handiwork one bad developer. To make matters worse, most of these bugs are user interface issues, which are both highly visible to the customer and difficult to unit test.

I’ve tried two approaches to this problem. The first was more of a punitive measure against my special egg: I had him print out his bugs from the past four weeks and verify that all were still fixed. This wasted a lot of my time and his, but it certainly slowed the re-emergence of bugs. The better solution I believe is to have a robust set of automated tests. Selenium is a open-source javascript test script suite designed specifically for rapidly generating and updating user interface tests. I use the Selenium IDE to record a user session, export the test as a junit test, and add it to my regression test suite. My Cruise Control build server can open a web browser and execute all tests within on every check-in, providing a much higher level of confidence that bugs are staying fixed.

Version Control

And by this I mean product version, not source control. In the beginning there was my development machine (read: laptop). Testers needed a server to hit without disrupting my work, so we set up a demo server with the latest HEAD build from my box pushed out at regular intervals. My client’s staff needed instances against which to develop reports and train on configuring the final product, so an internal release server was set up for them to download a copy to their workstations. This week we began setting up an external test and training sandbox for our first group of users, a performance and load testing internal server, and the production environment will be available any day now.

That’s a lot of copies of my application floating around. It’s particularly problematic when addressing bugs, since the bug can only be reproduced on the version that user is using. Who’s using what instance on what machine? Which log files do I check? Asking users to record their version in the issue tracker is an imperfect solution; folks are human and very rarely put more into the issue tracker than a title and a summary. Is there a better solution?

Cruise Control is riding to the rescue here as well. Once a build passes our test suite, it’s published to each environment. The internal distribution server points at the new build when client staff needs to train, the testing, demonstration, and load test servers all get copies of the new build, and the build is tagged in version control as the latest build to pass our QA tests. As for production, my idea here is to have a staging area that new builds are pushed to automatically, and then a technician can hand copy them to the production application server at a suitable time. All environments now have the same build.

Next Post

Just got out of my first No Fluff Just Stuff conference. Let me tell you, I recommend this HIGHLY to a Java developer at any level.

Right now I’m a “project lead” on a system in the QA phase getting ready for user acceptance testing and pilot roll out. I put project lead in quotes because I sort of fell into the role after being hired; really you can’t call me more than a Java developer.

I tried to focus mainly on topics I’m already interested in, such as productivity enhancements , plus get a few insights and ideas to help me through the QA process. Here’s a short list of the things I learned this weekend:

I really need to go out and get a MacBook Pro.

I’m quite certain Mr. Jobs and Co. don’t need any more raves about their products, but damn. All but one of the presenters at this conference use MacBook Pros as their tools of choice; all presented using the same machines they use for day-to-day development. The lone Windows user wrote every one of his demonstration code samples live in front of the audience using Windows Notepad, and must have hit the Backspace key about 5 times during an 1.5 hour talk. Barring any other freaks of nature out there, the gains from Mac OS X are staggering.

The clincher for me was Neal Ford’s talk on productivity. He demoed a small application on his MacBook called QuickSilver. To use his words, its a graphical command line. Nothing like this exists for Windows. With a series of simple keyboard gestures you can choose 3 files from separate hard disks, zip them together, and email the target to a recipient. Astounding.

Productivity gem: Mac OS X stores the keyboard shortcuts for every open application in the same place, and displays a cheat sheet based on whichever is active.

As soon as the funding gets lined up, I’m hitting up the Apple Store.

I really, REALLY need to install IDEA

I can’t cry poverty in this case, since my boss has generously offered me a floating license for IDEA whenever its convenient for me.

I first heard of IDEA at my first job, circa 2004. Due in part to cost and part to customer policies, we were restricted from running IDEA as our development machines. My mentor and tech lead would occasionally wax poetic about his salad days using IDEA for all his development, constantly ranting about how “slow” Eclipse was, and constantly looking for a somewhat decent JSP plugin.

Fast forward to today, when I’m one of two developers at my small ISV running Eclipse full time. When I first started a year ago, I thought I would stick with what I know, since it would be one less thing to spin up on. I got used to the idea of waiting 6 minutes for my incremental builds. I didn’t think a good JSP editor/debugger existed that could seamlessly integrate with an IDE’s method implementation and invocation lookups. Oops.

One more little productivity gem: IDEA will actually train you to use its keyboard shortcuts. If you use the same command three times, it will recommend you to create a keyboard shortcut if none already exists.

One of those things were, once you see them side by side, you can’t believe why any sane person would ever pick the first over the second.

We should have started QA much sooner

We used continuous integration at my previous company in a bad way. Sure we had a Cruise Control server running, but if all it does is check if the build completes successfully, it’s basically a glorified compiler.

But you can incorporate:

• Junit Tests
• Selenium Functional Tests
• Cyclomatic Complexity Reporting with JavaNCSS
• FindBugz static code checking
• JDepend dependency analysis

….and produce a ton of useful metrics about the overall health of your code base. You can’t catch everything, but you can catch a lot.

Unfortunately the underlying assumption here is an Agile development process with unit tests written as code is developed. Yea, we’re not quite there. Our “regression test” is an Excel spreadsheet with check marks, and that only came about after testers began complaining that previously closed bugs were reopening. Not really something you can automate. However, thanks to Selenium, we can at least automate our regression tests, even in the absence of more robust unit testing. You really shouldn’t re-factor to include unit tests, but better a repentant sinner than a self-righteous preacher.

As the industry (well, the hype anyway) trends toward dynamic languages like Smalltalk, Ruby, and Groovy, its going to become nearly impossible to ship code without a full suite of unit and regression tests. It’s just too easy to get your code to run, and bad developers just don’t care if their code passes tests or not. Particularly temp developers (there’s no such thing as a good temp developer). Even good developers cheat here and there when it comes to pure test driven development, especially with programs that are notoriously difficult to unit test (we’re looking at you, Struts!) This is a very, very, very good thing. I do not want to ever end up in this scenario again, and preventing it is motivation enough to pick up one of the Groovy books on the way out the door.

Best quote: “This will be the year of spectacular Ruby on Rails failures.”

One language just ain’t gonna cut it anymore

Speaking of buzzwords, you couldn’t fart without some windbag going on and on about how totally cool Groovy is. “Look! Hello, World in just one line!” My humble opinion is that the text representation of code is far less important than the bytecode; your audience here is another developer looking for bugs, not a fashionista. Groovy fails on both counts; the compiled Java byte code for a 10 line Groovy script runs about 5k, and the code samples I saw looked like the bastard stepchild child of c++ and python.

But a recurring theme was definitely the increasing specialization of languages. A language for scripting (Groovy), presentation (JSP), user interface (Swing and AJAX), database access (SQL), prototyping (Jython), rails (JRuby), multi-threading (Scala and Jaskell); all running using Java bytecode over the Java Virtual Machine. Combine this with the open sourcing of Java, and you have the emergence of Java as an assembly language for the Java Virtual Machine.

The JVM has existed for 12 years now, and despite the ever-present noise from its religious objectionists, it is a fast, stable, and secure platform that has survived a decade of trial by fire. It has maintained dominance in the software industry longer than its predecessor, C++. Yet the newsgroups all continually complain about Java’s “limitations”: no (direct) support for function pointers, rigorous type checking, vebosity. The constant whining has led to scope creep in Java itself. Take generics, which require type-safety when declared, but do nothing to enforce safety in their contents. Silly me who thought that was the whole problem in Java Collections that needed solving. Many features like this have crept into Java and are hardly in use anywhere.

But plain old Java is a well understand language. If someone puts Java code in front of a Java developer, chances are that developer can deduce its function given enough time. But if you did the same thing with Haskell code, the developer could probably figure it out, but you can be darn sure the first case would be easier. So what if you could compile your Haskell into Java? And use a Java debugger/profiler? And deploy it immediately to the millions of JVMs currently running on user and enterprise machines?

You don’t need generics or closures or autoboxing in Java itself. If Python takes less time to write, write Python and compile it to Java bytecode using Jython. Java itself is going to become less and less used by developers, and instead become a compilation target. So Java itself is now becoming a metaphor for JVM behavior, which is a metaphor for machine behavior.