Android versus iPhone Development: A Comparison

A few months ago I ventured into the world of Mobile development and created an application (Hudson Helper) for both iPhone and Android. This article is about my experiences, comparing Android and iPhone development with a focus on tools, platform and the developer experience.

Before going much further I should note that my comparison is with considerable bias. I’ve spent the past 12+ years in Java development, having spent much of my career building developer tools. Since January of 2004 I’ve been building plug-ins for Eclipse, and before that plug-ins for NetBeans. This bias is somewhat tempered with several years of C and C++ development. With this background I find that I’m very critical of developer tools. Developer productivity is key — anything that takes away from the flow of a developer in the zone is a real problem.

Language, Programming Model and Platform

Language

The language of choice for iPhone development is Objective-C. Objective-C is a language based on C with extensions for object-oriented concepts, such as classes, inheritance, interfaces, messages, dynamic typing, etc. The Java language is used when developing for Android (though it doesn’t actually get compiled to bytecode).

Java is a no-brainer. I have to say that it’s nice not to have to learn a new language to target mobile. Existing skillsets don’t come easy — so reuse of expertise is worth a lot.

It took a little while to wrap my head around some of the language features available with Objective-C. I soon discovered that I really loved certain language features, such as message passing (instead of calling methods), categories and named arguments. I did find however that the syntax of Objective-C is cumbersome. I’m still not used to ‘+’ and ‘-’ for static and member methods, too many parentheses are required, and in general I just felt like I had to type way to much to express a simple concept. The IDE didn’t help much with this either (more on that later).

One thing that really became clear to me is that Objective-C, though it may have been visionary for its time, is really a language of the '80s. Certain issues such as split header and implementation files and violation of DRY are really time-wasters, and not small ones at that. I found myself constantly switching back and forth between files, which not only has a cost in navigation (which file to open?) but with every file opened your sense of context must be recreated (where’s the caret, what’s selected, where am I in the file, how is this file organized).

As far as DRY, must I really do 5 things to declare a property?? (declare in the class definition, again to declare getter/settter, initialize in the init method, @synthesize in the implementation, release in dealloc). Here’s what I mean:

Server.h

@interface Server : Updatable {
NSString *name;  <-- declare the property 
}

@property (nonatomic,retain) NSString *name; <--- declare the property again

Server.m

@synthesize name;  <-- implement getter/setter

-(void) dealloc {
[name release];  <-- release memory
}

If you ask me, everything in Server.m should go away. Another gotcha here is the positional relevance of @synthesize.

Java has a similar problem with properties, though not quite so bad — and the IDE helps you write your getter/setter.

Pointers in Objective-C, though powerful, are also another time-waster. This is where Java really shines with its garbage collection. I found that I was constantly considering whether allocated objects were freed appropriately. Code flow is poor since application logic is littered with memory management. I only have so many brain cycles available — why do I have to think about this other cruft that’s not really a core concern of the application domain? Of course this gets even worse when trying to figure out where things went wrong if you make a mistake. Zombies help, but still don’t make it obvious if you’ve accessed something that was deallocated. Other issues include deallocating something twice, autoreleasing something twice. I also found it non-intuitive when to retain return values from methods.

Another annoyance of Objective-C is the patterns that must be followed: implementing correct init and dealloc methods is non-trivial. @synthesized getters and setters for properties with retain should not be called in these methods. So many conventions and rules to remember!

Though I understand why there’s a separation of alloc and init, it’s still overly wordy to specify [[aloc Foo] initWithArg: arg]. Why not just [new Foo arg]? Or how about new Foo(arg) -- oh, wait, that’s just like Java!

Objective-C’s imports and forward-declarations (@class) are a pain. Though these issues exist with Java development, Eclipse’s JDT is so good that I’ve almost forgotten what it’s like to write an import. All you have to do is Ctrl+Space to auto-complete a class name or Ctrl+Shift+O to organize imports and voila!

Of course Java is not perfect either, however this fact is hidden from me due to the fact that I’ve been living in Java for a very long time. Sometimes I wish that Java were more Groovy-like, however I’m used to it and the tooling is so good.

Platform

On Android I found that I could readily use the Java runtime classes. Some, but not all, of the standard Java RT classes are available on Android. I didn’t find this a problem, since most of the standard Java IO, network and regex libraries are available. Android RT classes appear to be based on Harmony, which has been around long enough to be stable.

With iPhone on the other hand, finding the functionality that I needed was painful. Classes and methods are poorly organized. When to look for a static method versus a class with members was not clear to me. Also depending on the framework used, naming conventions and code organization would differ. I suppose this is the legacy of an older platform. Areas where functionality was lacking that I found painful were regular expressions, string handling and XML parsing. I ended up using the excellent Regex Kit Lite for regular expressions. For XML parsing I implemented a parser abstraction over libxml, only to discover later that I may have had an easier time with NSXMLParser which is a lot more like SAX.

On the iPhone when things didn’t work as expected I had to resort to Google and hope that others had encountered the same problem. This technique was hampered by Apple’s earlier NDA policy, which meant that iPhone content is pretty thin on the net. In some cases I would resort to guesswork and experimentation to find a solution.

Android has the benefit of being open source. Within minutes I had the full Android platform source code on my system, and had re-built the SDK from sources to ensure that the source I had matched the runtime classes in the emulator. So not only could I see how things were implemented in the Android platform and learn by example, I could step through the platform code in the emulator and discover why my code wasn’t producing the desired results.

In general I found the layout, organization, and naming conventions of Android platform classes was consistent and predictable. This made it much easier to learn.

Programming Model

The iPhone platform does a great job of encouraging an MVC design pattern. With this design pattern built in to the platform, building the UI was simple and I didn’t have to figure out how to organize the UI component design myself. It also means that when looking at sample code, it’s all organized in the same way.

Android also does a good job with design patterns, though their concepts varied significantly from the iPhone. With Android’s support for multiple processes and component reuse, the platform itself provides support for Intents and Activities (an Intent is just a variant of a command). The design results in a better user experience, however it does introduce some complexity for the developer: when starting one Activity from another, an Intent is used to communicate any parameters. These parameters cannot be passed by reference — only by value. Where on the iPhone it’s simple to have screens sharing the same data structures, on Android this requires some forethought. Apparently Android applications can manage the back button and have everything occur inside a single Activity, however this is not the norm.

Both Android and iPhone provide a way of declaring user preferences in XML. Both platforms provide a default UI for editing those preferences, which is great. Android’s XML format is extensible allowing custom UI components to be integrated, which makes user preferences a breeze. iPhone developers that wish to customize preferences will have to implement a UI from scratch, which is a lot more work.

Testing and Continuous Integration

I’m of the opinion that every development effort should include unit tests. Teams of size greater than one should also include Continuous Integration.

Android developers will be happy to know that they can write JUnit tests. I could even launch these from the Eclipse UI after some classpath fiddling. Though I didn’t try it, I assume that it’s trivial to run these from Ant and your favorite CI server such as Hudson.

I did see some iPhone unit test documentation with the iPhone SDK but didn’t take the time to explore it — so I can’t comment there.

Resources

Apple does an excellent job of providing lots of resources for developers. Important concepts are explained in videos, which makes grasping concepts easy — however I did find that videos progressed slowly and I was watching for what seemed like hours to find information that should have taken minutes. Luckily Apple also provides lots of sample applications and code to demonstrate API usage.

Android developers also have access to loads of resources. The guide and API reference are installed with the SDK, so everything is available when offline (which for me is important since I do a lot of my work in transit). I found the Android development resources better organized and spent less time looking and more time finding. In particular the ApiDemos sample app provides a great starting point. I also downloaded many open source Android projects for ideas on architecture and API usage. This is an area where Android has the advantage, with Apple’s previous NDA policy there isn’t much out there in terms of open source for iPhone.

Tooling

For me tooling was a real shocker. These are the categories of tooling that I’ll cover: IDE, UI builder, debugger, profiler. Almost everything else is related to provisioning, and in that area I didn’t notice much in the way of differences between Android and iPhone.

IDE

Android development leverages the excellent JDT tools, which are pretty much stock and standard with every Eclipse installation. I’ve used these tools now for many years and they’re excellent. Everything Java is indexed, the IDE has a rich model of the source code, and refactoring is so seamless that it has changed the way that I work.

Perhaps the best feature of JDT is its incremental compiler, which provides immediate feedback with errors and warnings as you type. This eliminates the code-compile-wait-for-feedback cycle that was so common in the '80s and '90s. Errors and warnings are updated in the Java editor as I type, giving me instant feedback. I didn’t realize just how valuable this feature is until I was coding Objective-C in XCode — when I became acutely aware at how waiting for compiler feedback can break the flow of programming.

Other key features that make Eclipse so amazing to work with are:

• content assist
• quick-fixes
• organize imports
• open type (CTRL+Shift+T)
• refactorings

Integrated javadoc and content assist is quite possibly the best way to learn an unfamiliar API. In Ecipse not only are all classes and methods immediately available in the context in which you’re writing code, their documentation is presented alongside.

Content Assist with Integrated Javadoc

XCode is so shockingly bad that I almost don’t know where to start. Here’s a minimum list of things that I think need fixing in order for XCode to become a viable IDE:

• Content assist that actually works. Content assist provided by XCode is often wrong, and almost always suggests a small subset of what’s actually available.
• A decent window/editor management system. XCode and it’s associated tools (debugger) like to open lots of windows. Want to open a file? How about a new window for you! Very quickly I found myself in open-window-hell. The operating system’s window management is designed for managing multiple applications, not multiple editors within an IDE. It’s simply not capable of providing management of editors in an environment as sophisticated as an IDE.
• A project tree view that sorts files alphabetically. Really!
• Integrated API documentation. I found that I was constantly switching out of the IDE and searching for API documentation using Appkido. This may seem trivial, but it really breaks the flow.

One area of Eclipse that simply can’t be matched is Mylyn. Integrated task management and a focused interface introduce huge efficiencies into any project, small or large. If you haven’t yet tried out Mylyn, it’s definitely worth your time to take a look. A good place to start is Mylyn’s Getting Started page.

UI Builder

iPhone app developers are given a pretty good UI builder. It does a great job of showing the UI as it will actually appear. It’s flexible and can model some pretty sophisticated UIs, so I was impressed. I found that using it was a little tricky — I had to read the documentation two or three times before I could really figure out how to use it properly.

The Android UI builder I found pretty useless: it can’t display UIs how they’ll actually appear, and it’s UI is way too inefficient. I found that I coded all of the UIs directly in the XML source view of the UI builder. There the content assist and validation were pretty good, making it the easiest way for me to build a UI.

Debugger

Having used to the Java debugger in Eclipse I was shocked at the state of the debugger in XCode. With Eclipse I can see and modify variable values. Not so in XCode. Maybe this is simply the state of affairs when debugging native code, but it sure affects the usefulness of the debugger. XCode often seemed confused as to the type of an object and presented me with a pointer value and no detail. This is a sharp contrast to Eclipse, where I can drill down through an object graph with ease.

I found the XCode debugger UI extremely difficult to use. Clicking on the stack to show code in an editor caused new windows to open, eventually resulting in dozens of windows open. In addition I found that watch expressions rarely worked for me.

Profiler and Heap Analysis

An area where Apple development tools excel is in profiling and heap analysis. These tools seemed mature and easy to use. With no prior experience with these specific tools I was able to gain a better understanding of my app within minutes, find and fix several memory leaks and improve performance.

XCode Memory Leak Detection

Android developers must use Android’s traceview application, which I found worked well but required significantly more effort to configure and operate. I was surprised to find that the source code must be changed in order to get the trace files required for analysis.

I’m not sure if Android can provide heap dumps in hprof format. If it can then the awesome MAT tool could be used to analyze heap usage. According to this article Android can produce hprof heap data, though I haven’t tried it.

App Store

It goes without saying that the iPhone app store is excellent in that you can sell into many countries worldwide with a single setup. I was able to provide my Canadian bank account number, sign a few legal agreements and I was up and running.

Getting an app into the store however is frustrating to say the least. Apple must approve every app before it is accepted into the store. Mine got rejected multiple times. Each time it was rejected I was given almost no information about why. When I emailed them to clarify the problem, I received what looked like a canned response indicating that I should refer to previous correspondence. If it weren’t so frustrating I would have found it funny. I highly recommend reading Brian Stormont’s Avoiding iPhone App Rejection from Apple and Dan Grigsby’s Part 2 follow-up.

Of course once I started selling Hudson Helper I realized that Apple won’t send me any money unless the payout is greater than $250. This is true not only of the first payout, but every payout. Google market on the other hand requires a minimum of $1 for each payout. Both the iPhone app store and Google market take about %30 of your app selling price. $0.99 applications have to have high volume, or they’re simply not worth your time.

The Google market by comparison to the Apple app store is terrible in that you can only sell into a handful of countries. You also can’t see or install apps that cost money on a developer phone. Actually you can, but not if the app has copy protection — which is almost every non-free app. On the other hand when you upload your app to the app store it’s available within minutes, so you don’t have to worry about an approval process.

To set up a merchant account with Google market, I had to provide a US address and bank account number, since Google doesn’t support Canada. For me this was a pain, but not too bad since I live within a few kilometers of the US border. I rode my bike down to the US and opened an account with Horizon bank. The bank required a passport and driver’s license, so no problem there. Why Google doesn’t support more countries I don’t know. At the very least Google market should accept alternate payment methods for countries that are not supported by Google checkout.

Summary

Android’s platform and developer tools are excellent. Leveraging Java and the Eclipse IDE are major winning factors for Android. Apple’s developer tools are shockingly bad by comparison. The Objective-C language and platform APIs are cumbersome and poorly organized. Overall when developing for the iPhone I felt like I was back in 1993. These factors combined in my estimation make application development about three times more expensive when developing for iPhone. The only area where Apple’s developer tools excelled was in profiling and heap analysis.

Apple’s app store from a user’s standpoint and from a worldwide coverage standpoint are excellent. In this area Google market for Android is weak.

Development for iPhone may improve as tools such as iphonical (MDD for iPhone) and objectiveclipse (Eclipse plug-in for Objective-C) emerge.

We may see a shake-up in the mobile market, with at least 18 new Android handsets being released this year. Until that happens, iPhone will remain a market leader and developers will have to put up with XCode and Objective-C.

For me, my love is with Android. Sure, the iPhone is great — but can you install a new Linux kernel?

Hudson Helper for Android

Recently I launched Hudson Helper for iPhone and iPod Touch, enabling Continuous Integration fans to stay in touch with their projects. Android users can get in on the game now too, with Hudson Helper for Android.

Hudson Helper for Android provides all of the same features as the iPhone version including support for multiple servers and authentication. New for this version are build controls: start and stop builds right from your phone. CI can be even more fun with shake-to-build and sound effects.

To get Hudson Helper for Android, search for 'Hudson Helper' in the Google Market on your Android device.

Tip: Twittering About Eclipse Bugs

A quick tip for including an Eclipse bug URL in your 140-character tweets: Bugzilla for Eclipse supports short links. Instead of using the long-form URL or impossible-to-decipher URL shorteners, take out the 'bugs/show_bug.cgi?id=' part of the Bugzilla URL as follows: https://bugs.eclipse.org/277974

Hudson Helper and Untrusted Certificates on the iPhone

If you're using a self-signed certificate or a certificate signed by your own certificate authority with the Hudson Continuous Integration Server then you may be wondering how to get your iPhone to trust your certificate. Well, you're in luck because Benjamin Boksa has written an excellent tutorial explaining how to do just that: iPhone OS 3.0 – Root Certificate Authority installation. After following the tutorial you should be able to use Hudson Helper with untrusted certs and your secure Hudson server.

WikiText in Eclipse Galileo

Ian Bull of EclipseSource has provided an excellent summary of Mylyn WikiText and what it means for Mylyn users and Galileo. WikiText is number 4 on Ian's Eclipse Galileo Feature Top 10 List. Mylyn WikiText is part of the highly anticipated Eclipse Galileo release due out this Wednesday. You can find out more about WikiText from the Mylyn homepage, the FAQ, and the downloads page.

Galileo Builds On Your iPhone

With the Eclipse Galileo release exactly a week away, I'm presenting a quick tutorial on how you can monitor the Galileo hudson build on your iPhone with Hudson Helper. I'm also giving Hudson Helper away for free to the first 20 Eclipse contributors^* to send me an email.

Here are the steps to take to get Hudson Helper set up with the Galileo build:

1. If you haven't done so already, install Hudson Helper. You can get it by searching for 'Hudson Helper' in the app store, or via iTunes here.


2. Next, go to the Settings application on your iPhone or iPod touch. Under Hudson Helper, Server 1 URL enter the following: https://build.eclipse.org/hudson/view/Galileo
   Notice that it's https, not http.
   
   
   
   You may have noticed from the screenshot that a new version of Hudson Helper is now available with support for HTTP authentication.
   


3. That's it! Close the Settings app, launch Hudson Helper and you should see the following:

Even though Hudson Helper has blue balls, it can still help to keep you up to date with your builds!

^*If you're an Eclipse committer you qualify. If you're not an Eclipse committer but have contributed at least 2 patches to Eclipse, you qualify. If you were a committer you qualify. If you aren't a committer, but are a Friend of Eclipse you qualify.

Hudson Helper: Hudson On Your iPhone

I'm a big fan of Continuous Integration and automated builds. So much so that today I released Hudson Helper, a native iPhone and iPod Touch application for monitoring your Hudson continuous integration engine.

Hudson Helper has a simple interface that gives you what you need to know first. Offline data caching helps to minimize network bandwidth and makes it work when the network is unavailable. Includes job status, health and build trends.

Build status display adapts to your project to report on build health including JUnit test results, Clover coverage, findbugs and any other Hudson build report plug-ins your project is running.

To get started all you need to do is point Hudson Helper at the URL of your build server. You can get Hudson Helper by searching for 'Hudson Helper' in the app store, or click here to get it with iTunes.

Eclipse 3.5 (Galileo) on Java 7

Some time ago I blogged about running Eclipse (Ganymede) on SoyLatte. Since then there has been little improvement in the state of Java on the Mac. Apple has released a Java 6 for Mac, but it's a 64-bit only VM which crashes regularly when running Eclipse. Having heard about recent improvements in performance for 64-bit VMs using pointer compression (PDF paper), I decided to have another look at launching Eclipse with an OpenJDK-based VM such as JDK7. After a few false starts I managed to get it working. Here's what I did:

First, to get JDK7 on my machine. I followed these instructions to build JDK 7 from source. The instructions didn't work perfectly for me -- but they were pretty good. Basically followed everything verbatim with the exception that I had to install MacPorts and use it to get wget.

With a 32-bit JDK7 on my machine, now to use it to start Eclipse. I downloaded the latest I-build of 32-bit Eclipse 3.5 from here (Look for '3.5 Stream Integration Builds').

Using a combination of jconsole and guesswork I came up with this shell script to start Eclipse:

JDKPATH=/Users/dgreen/Documents/packages/jdk7-32bit-2009-05-08
DEBUG_OPTS=
# DEBUG_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=y
MEM_OPTS=-Xms1024m\ -Xmx1536m\ -XX:MaxPermSize=256m
ECLIPSE_LOC=/Users/dgreen/Downloads/eclipse
WORKSPACE=/Users/dgreen/Documents/workspace

$JDKPATH/bin/java $DEBUG_OPTS \
-cp $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher_1.0.200.v20090429-1630.jar \
-XstartOnFirstThread $MEM_OPTS \
-Dorg.eclipse.swt.internal.carbon.smallFonts \
-Djava.library.path=$ECLIPSE_LOC/Eclipse.app/Contents/MacOS/lib \
-Dswt.library.path=$ECLIPSE_LOC/Eclipse.app/Contents/MacOS/lib \
org.eclipse.equinox.launcher.Main \
-os macosx -ws cocoa -arch x86 \
-showsplash -launcher $ECLIPSE_LOC/Eclipse.app/Contents/MacOS/eclipse \
-name Eclipse \
--launcher.library $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher.cocoa.macosx_1.0.0.v20090429-1630  \
-startup $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher_1.0.200.v20090429-1630.jar \
-data $WORKSPACE \
-keyring /Users/dgreen/.eclipse_keyring -consoleLog -showlocation \
-vm $JDKPATH

Note that all of the above should appear on one line: your browser may split it into multiple lines.

After starting Eclipse as above I saw the following stack trace on startup:

java.lang.UnsatisfiedLinkError: no swt-pi-cocoa-3547 or swt-pi-cocoa in swt.library.path, java.library.path or the jar file
 at org.eclipse.swt.internal.Library.loadLibrary(Library.java:248)
 at org.eclipse.swt.internal.Library.loadLibrary(Library.java:159)
       ... snip ...

To solve the problem I extracted all of the *.jnilib files from org.eclipse.equinox.launcher_*.jar and put them in a lib folder. I then renamed all of those *.jnilib files in the lib folder so that they had *.dylib file extensions. I ended up with a file structure as follows:

david-greens-macbook-pro:MacOS dgreen$ pwd
/Users/dgreen/Downloads/eclipse/Eclipse.app/Contents/MacOS    <--- where I put the files
david-greens-macbook-pro:MacOS dgreen$ ls -l *.sh lib
-rwxr-xr-x@ 1 dgreen  staff  1100  8 May 15:47 eclipse.sh    <--- the startup script

lib:        <---- the lib folder for shared libraries
total 2296
-rw-r--r--  1 dgreen  staff   37104  8 May 14:20 libswt-awt-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  287116  8 May 14:20 libswt-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  539240  8 May 14:20 libswt-pi-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  300428  8 May 14:20 libswt-xulrunner-cocoa-3547.dylib

Trying the shell script again, Eclipse started up and ran beautifully! Once again I'm running Eclipse with a newer-than-Apple-will-give-me VM, this time JDK 7!! My next step is to compile a 64-bit JDK7 and see if it works too... I have high hopes.

Updated 2009-05-15 see this issue (bug 276564) for the underlying cause of failure

How Babel Saved My Bacon

Recently I blogged about the pain of moving to ICU4J, lamenting the fact that using NLS instead of ResourceBundle would require me to change the keys in my resource bundles. After much consideration and the advice of knowledgeable people, I decided that moving to NLS was the way to go. After changing resource bundle keys across the board, I expected that translations to other languages would need their corresponding keys changed as well. Not so. Apparently Babel can magically handle such refactorings, with the help of a mystery user named 'Babel Syncup'. I wonder what else it can do.

The Eclipse Babel project is truly impressive. Kudos to the Babel team for thinking ahead.

Peter Friese on Advanced WikiText

Peter Friese has upped the stakes in his latest article on Advanced WikiText. Friese tackles tough topics, such as how to optimize team collaboration with multiple source files and integrating with a DocBook toolchain. Great article, a must-read for those who are authoring documentation.

Peter Friese on Writing Documentation with WikiText

Peter Friese has published a great article titled Getting Started With WikiText. He starts off with some background on various documentation formats, then provides a step-by-step guide to getting started with Mylyn WikiText. Peter also shows how images can be scaled using some simple wiki markup using a new WikiText feature that Peter Friese recently contributed.

Galileo ICU4J Requirement Is Painful

Projects must use ICU4J if they wish to participate in the Galileo release of Eclipse, due this summer. Though such a requirement may make good sense in theory, in practice adopting ICU4J is painful. Not only does this policy stifle innovation for questionable gain, such a burden makes participating in the Eclipse ecosystem less attractive.

ICU4J is designed to be a drop-in replacement for Java classes that perform poorly or behave incorrectly. The idea is that you change your import statements, recompile and everything works. In applying this technique to Mylyn WikiText, I’ve discovered that it’s not that easy.

Mylyn WikiText makes use of two classes that are to be replaced by ICU4J equivalents: MessageFormat and ResourceBundle.

ResourceBundle

ResourceBundle conversion should be easy, right? ICU4J provides UResourceBundle, which extends ResourceBundle.... wow, that’s great. All I should have to do is replace ResourceBundle.getBundle() with UResourceBundle.getBundleInstance()... easy peasy. Unfortunately, not so easy.

Previously I had used the JDT ‘Externalize Strings’ wizard to externalize strings in my plug-in. The JDT nicely created a utility class Messages.java alongside the resource bundle messages.properties. This worked really well and has been in use for some time. Unfortunately after converting to ICU4J UResourceBundle throws a NoClassDefFoundError because of the presence of the Messages class. It seems that UResourceBundle cannot handle a resource bundle alongside a class by a similar name. While I’ve logged bug 272166 to track this issue and have it fixed, it doesn’t help me. Even if the bug gets fixed in time for Galileo, Mylyn WikiText must work on Eclipse 3.3 and 3.4 as well.

It gets worse. Mylyn WikiText is already translated to German and Japanese. Resource bundle message keys must remain relatively stable in order to avoid losing those translations. JDT creates message keys containing dots ‘.’, and the Eclipse externalized string pattern cannot have dots in message keys... so I cannot convert over to the Eclipse externalized string pattern without losing these translations.

MessageFormat

MessageFormat evolved for Java 5: it’s format() method became a var-args method, meaning that you can pass a variable number of arguments to it. WikiText makes extensive use of this method, so you’ll see code such as this:

MessageFormat.format(messageTemplate,arg1,arg2)

ICU4J does not have a var-args method equivalent — it seems that ICU4J has not evolved for Java 5. To adopt ICU4J’s MessageFormat, all calls had to be converted as follows:

MessageFormat.format(messageTemplate, new Object[] { arg1, arg2 })

While the conversion can be achieved using regular expressions, the resulting code is less readable and more prone to bugs. Localization is hard enough as it is without having to jump through age-old languageisms.

Summary

While converting all plug-ins to ICU4J may make sense for some, making it a requirement for Galileo participation has caused me and presumably others a lot of work. From what I can see ICU4J brings little or no value to the users of my project. This is one case where good intentions with policy have a substantial negative impact. In my case, the time I’m spending converting to ICU4J could have been used to innovate or improve the quality of WikiText. Instead I’m jumping through hoops with no apparent benefit for me, my project or my users.

A Tough Lesson in AJAX, JSF and Client-Side State

Last week I learned a tough lesson in AJAX, JSF and client-side component state. This one had me fooled for the better part of a day, in part due to the complexity of debugging multiple related AJAX requests, intermittent symptoms, and the opaque nature of JSF component state. Here's what happened:

The page that was problematic has multiple tabs (rich:tab), some combo boxes (h:selectOneMenu), and at least one suggest control (rich:suggestionbox). The tabs are server-side AJAX tabs. The suggest control also uses AJAX.

To reproduce the problem, the user takes the following steps:

1. Use a suggest control to select a value on the first tab of a
   multi-tab form
2. Switch tabs to a different part of the form
3. Press a button on the form in the 2nd tab

At this point the application complains with validation errors on the first tab. The experience is very confusing to the user: after switching tabs successfully with no errors, pressing a button on the second tab causes validation errors on the first tab. This should never happen in the application: validation is designed to occur only for components that are submitted. When the button is pressed on the second tab, validation should not occur for components in the first tab. So what's going on?

Here's what I expected to happen:

1. The AJAX suggest control sends requests and renders itself
2. The tab sends requests and renders its contents
3. The button submits the form to make a request

Here's what really happened, depending on timing of the requests:

1. The AJAX suggest control sends a request
2. The tab sends a request
3. The browser receives the response from the tab click and re-renders the tab and its form controls
4. The browser receives the response from the suggest control

After receiving every response the DOM is updated to contain the new JSF component state (remember: we're using client-side component state, so all JSF component state is stored in a hidden form parameter in the browser).

Due to out-of-order responses, the component state was being set to that of the first tab by the suggest control -- thus the application thinks that the current tab is the first, not the second as we would expect. Pressing the button on the second tab results in the application interpreting the submitted form as submitting values for components on the first tab. This is problematic for HTML <select> controls, since HTTP parameters are not submitted for <select> controls where no value is selected.

So how do we fix this problem? Have all AJAX controls use the same request queue. This is achieved by setting the attribute eventsQueue="myApplicationEventQueue" on all of the richfaces AJAX controls. By doing so we end up with the following sequence:

Lesson learned: AJAX requests update JSF component state on the client, and thus must all complete in sequence. Otherwise component state on the client can get out of sync with what we expect it to be.

Mylyn WikiText produces PDF

Mylyn WikiText becomes increasingly flexible with its latest addition of XSL-FO to the list of output formats supported by its wiki markup conversion capability. Combined with the excellent Apache FOP project, it becomes easy to produce PDF from your wiki markup.

 <-- create XSL-FO -->
 <wikitext-to-xslfo markupLanguage="Textile" file="My File.mediawiki"/>

 <-- create PDF -->
 <exec command="${fop.home}/fop">
  <arg value="${basedir}/help/My File.fo"/>
  <arg value="${basedir}/help/My File.pdf"/>
 </exec>

From a single source you can now produce HTML, Eclipse Help, DocBook, DITA and now XSL-FO and PDF. Wiki markup never looked so good!

If you're interested in trying out this new functionality you'll have to wait until the next weekly release is posted on the Mylyn downloads page.

Mylyn WikiText Experiences from EclipseCon 2009

I presented my session titled "WikiText: Generate Eclipse Help from Eclipsepedia" (about Mylyn WikiText) at EclipseCon 2009. I had a lot of fun and lots of positive feedback. If you're interested in the slides from my presentation, you can find them here. A couple of interesting experiences from EclipseCon:

Trying Something New

Before the presentation I saw the Keynote Remote application for iPhone. Thinking it would be fun, I tried it for the first time while presenting my session. It turns out that to control slides you must swipe, which means that you need to know the orientation of your iPhone. Every time you want to change slides you have to look at your phone. Very distracting for both the presenter and the audience.

I learned the hard way something that I already knew: don't try anything new in a presentation!

Twitter

Twitter helped to create a buzz both before, during and after my session. The best thing about it for me was that I got immediate feedback on my presentation even before it was over.

iPhone Alarm

Beware that your iPhone alarm will sound even if the quiet switch has been activated. I discovered this design feature as Mike Milinkovich, the executive director of the Eclipse foundation was presenting. I think he was as surprised as me when the get-to-a-bomb-shelter-now siren sounded from my phone's alarm. Sorry Mike!

All in all I had a great time. I highly recommend EclipseCon for those who wish to meet interesting people with a common focus and lots of ideas.

Mylyn WikiText 1.0 Released

I'm pleased to announce that Mylyn WikiText 1.0 has been released as part of Mylyn 3.1. Mylyn WikiText provides lightweight markup (wiki) parsing and editing capabilities to the Eclipse platform, Mylyn, Ant and stand-alone applications. Mylyn WikiText is the result of more than a year of development and over 170 bug fixes and enhancements. This marks the first release of WikiText after its move from the Textile-J project. Many thanks to the Mylyn team and the Eclipse community for making this possible. Here are some of the highlights:

Mylyn Task Editor Integration

With WikiText the Mylyn task editor is now markup-aware, providing source formatting, integrated preview, content assist and markup help. The following shows how markup can be authored and previewed in the task editor:

WikiText selects the markup language that best suits your task repository. This is great for users of JIRA, Trac and other repositories that provide support for wiki markup in task descriptions and comments. Users can also benefit from using wiki markup with task repositories such as Bugzilla that don't provide support for wiki markup.

Eclipse Integration

WikiText provides an editor for files containing wiki markup:

WikiText provides support for several markup languages, including MediaWiki, Textile, Confluence, TracWiki and TWiki. WikiText also provides an extensible parser framework for adding support for other markup languages.

Ant Integration

Users of Apache Ant now have access to Ant tasks for transforming wiki markup to alternative formats such as HTML, Eclipse Help, DocBook and OASIS DITA.

RCP, SWT and Server Applications

WikiText provides public APIs for integrating wiki markup capabilities into RCP applications and stand-alone SWT applications. These APIs make it easy to display markup and inspect its structure.

WikiText can also be used server-side to transform markup in web applications.

WikiText In The Field

WikiText is being used in many interesting ways:

• The Mylyn project uses WikiText to convert MediaWiki content from Eclipsepedia to Eclipse Help content.
• Mylyn WikiText authors help content in Textile and converts it at build time to Ecilpse Help content.
• A recent addition to the Eclipse Examples project uses WikiText to create slide presentations based on files authored in MediaWiki markup.

Mylyn WikiText will also be presented at EclipseCon 2009 in WikiText: Generate Eclipse Help from Eclipsepedia.

More Information

Mylyn WikiText can be installed using an Eclipse update site or by downloading the standalone jars. See Mylyn downloads for details.

For more information about WikiText, see the Mylyn 3.1 New and Noteworthy, Mylyn FAQ or the Mylyn homepage.

Open Type Dialog - Small Things Make A Big Difference

I'm a big fan of attention to detail when creating tooling for developers. Our brains are tuned to notice the smallest perceptible visual clues. We can absorb a lot of information with the right visual cues, however presenting that information without adding clutter to an IDE is a real challenge.

The JDT team has mangaged to improve the Open Type dialog in Eclipse 3.5 without falling into the trap of a cluttered interface:



Notice how a bold font is being used to identify regions of text that match the filter. The JDT team is using a new platform feature where multiple fonts can be used in one cell. Here's what it used to look like:



The best thing about this new feature is that my brain doesn't have to work harder to know more -- it's immediately evident to me which matches are better and why without having to think about it.

The importance of eliminating and minimizing visual noise is evidenced by products such as Mylyn that are designed to do just that: filter out what you don't need.

Kudos to the JDT team for raising the bar. I can't wait for Eclipse 3.5 Galileo to be released this summer.

Eclipse Foundation Committer Member

Today I became an Eclipse Foundation Committer Member. This will enable me to vote in the 2009 Eclipse Foundation Elections.

Being committed (no pun intended) to the Mylyn WikiText project and being interested in projects such as SWTBot and SWT (for Cocoa) I believe that small and innovative projects should be well-represented on the Eclipse Foundation board.

Having known and worked with Mik Kersten in various ways over the past few years, I am constantly surprised by his breadth of knowledge, leadership, and grass-roots understanding of the needs of developers and committers. I've seen Mik uphold a high level of integrity and demonstrate consistent follow-through. I know that he has a solid understanding of what is needed by such projects and is capable of using his board experience and influence to make a difference.

Unsung Heroes Of Eclipse

I'm glad to see a Top Contributor category in the Eclipse Community Awards that takes into account more than just code contributions. It's all too easy to lavish praise and recognition on those who contribute code -- but what about the others? The Mylyn WikiText project has been lucky to garner the attention of individuals who spend their own time installing, testing, performing quality assurance, and collaborating on Bugzilla.

In particular I'd like to thank Jörg Thönnes of Aachen, Germany for the many hours that he has contributed in making Mylyn WikiText a better product.

TPTP is Not For Me

I've been using the NetBeans profiler to help me develop Eclipse plug-ins. A couple of years ago I tried TPTP (Eclipse Test and Performance Tools Platform), but found it extremely difficult to configure and it couldn't compete with Sun's dynamic bytecode instrumentation technology. Today I thought I'd give TPTP another go. The first link that I clicked on the TPTP homepage was the Getting Started link, and this is what I got:

Given that Safari and WebKit are largely acclaimed as being some of the most advanced web technology available, it seems that the TPTP team really doesn't want me to use their profiler. I'll check back later -- in the meantime I'll continue using NetBeans to profile Eclipse.

JPA 2.0: Why AccessType is Relevant

It has long been debated whether field access is superior to property access for JPA entities. Most arguments that I've seen hinge on performance versus encapsulation. While these arguments may be interesting from a theoretical perspective, control over field access is in fact very useful in solving real-world problems.

Field access allows code to distinguish between the JPA provider setting properties while loading an entity versus application code setting a property.

This was very useful for a project that I'm involved with that has de-normalized data or other fields that are computed and persisted. Here's how it works:

• In the setter for various properties that affect the computation of a de-normalized property, the value of the de-normalized property is set to null. @AccessType(FIELD) is used for these properties.
• In the getter for the de-normalized property, if the value is null the de-normalized value is recomputed.
• In a @PrePersist method, the getter is called to ensure that the de-normalized value is properly persisted.

Using this technique maintaining de-normalized data becomes simple and robust.

While this feature has been available within Hibernate since 3.0, I'm glad to see that JPA 2.0 (JSR 317) introduces a standard annotation to control this behaviour.

WikiText as a Slideshow Presentation Technology

I'm constantly surprised by the versatility of wiki markup. Wayne Beaton has recently blogged about using MediaWiki markup to author presentation slides. He does this using Mylyn WikiText to parse wiki markup. Neat stuff!

Eclipse: Hitting a Breakpoint After The Fact

Ever wonder how to hit a breakpoint after you've passed it? You can do this with the Eclipse Java debugger. Here's how I do it:

1. Stop at a breakpoint
2. Realize that you want to step through a method called a few lines back
3. Put a breakpoint in that method
4. Highlight an expression that uses the method in the java editor and from the context menu select Inspect

That should do it! Before you see the value in the inspection pop-up, your debugger should hit the breakpoint that you just created. Very cool.

Working With Patches and CVS

Many contributions to Eclipse projects start as a patch. Patches are great because they can encompass changes to multiple files and make it easy to collaborate with others. Anyone can make a patch, so contributors don't need commit privileges to participate. Patches can be attached to tickets in the Eclipse Bugzilla so that committers can be confident of their IP cleanliness. The problem with patches is that they're static. That is, they don't evolve as the source code in CVS changes. Thus patches become stale, often resulting in committers asking contributors to re-cut a patch. So how do we overcome this problem?

I've had the pleasure of working from both sides of the problem. Starting as a contributor with the Textile-J contribution to Mylyn WikiText I created and maintained multiple patches, some small and some large. Eventually after earning Mylyn commit privileges I saw patches coming from contributors. The problem of the stale patch comes up again and again. Often contributions must wait a few days, weeks or even months before they're evaluated. During this time the files affected by the patch often change in CVS, thus rendering the patch unusable until it is massaged to fit the new code. Unfortunately this process of massaging is usually manual since the line-based merge algorithms used by CVS cannot account for non-trivial changes to source files.

So how do we evaluate and apply these valuable contributions? It's tempting to apply a stale patch and store conflicting changes in a temporary .rej file, and then merge those conflicting changes by hand afterwards. Though in some cases this approach makes sense, there is an easier way for those other cases.

The following formula can ease the pain for committers and contributors alike as they try to use a patch that has become stale:

1. Start by attempting to apply the patch using the Eclipse 'Apply patch' functionality, either from the Team menu or from within the Mylyn task editor.
2. Take note of which files have problems in the apply patch wizard.
3. Exit the wizard without applying the patch.
4. Take note of the time-stamp on the patch.
5. For each file in the patch that was problematic, use the CVS 'Team->Replace With->History...' wizard and select the most recent revision that predates the patch. Take note of which revision was used.
6. Now apply the patch. It should merge without issues. If not, you've missed something so go back to step 1.
7. For each file that was moved back in time in step 5, use the CVS History to compare the revision that was used with the patch to the latest revision in CVS. Apply the changes that are shown in the compare editor to the file in the workspace.
   

The formula turns the problem around. Instead of trying to apply the patch to the latest revision, we apply the changes made since the patch to the patched files. Sometimes this can be much easier.

How do you choose which approach to start with? I recommend starting with an up-to-date workspace and try applying the patch. If proceeding down that path is starting to feel too hard, then revert your changes and try the formula described here. In some cases, the hard problem becomes easy! If not.... well, patches aren't perfect. Sometimes hard is just hard.

Mylyn WikiText at EclipseCon 2009

WikiText will be at EcilpseCon 2009 in Santa Clara, California. The presentation time was expanded to allow for more coverage of WikiText -- so if there's something specific that you'd like to hear about please let me know.

The WikiText presentation is included in a session titled "50 minutes towards a better you". Don't ask me about the title -- I don't know if WikiText will make you a better person or not. I'm very excited to be there presenting this new technology alongside Steffen Pingel, who will be providing a Mylyn connector crash course.

Mylyn Context-Driven Domain Diagram

Lately I've been working on some large legacy modernization projects with MAKE's MDD tooling. These projects have upwards of 400 entities in their domain (read: minimum 400 database tables). With such a large application domain it's common to have large, complex domain diagrams which are often at the wrong level of granularity for programming tasks.

I found myself creating small, purpose-specific diagrams showing only those things that were relevant to the task at hand. These diagrams would get tweaked and maintained as my understanding of the task and domain evolved. While maintaining diagrams is not really all that hard, it is time-consuming and distracting.

After some thinking I realized that the answer to developer productivity was sitting right in front of me: Mylyn task context-driven diagrams. Why can't the MDD tooling create and maintain a domain diagram based on the interesting domain elements in the active Mylyn context? There would be no need to create or maintain such a diagram manually: it would always exist, and it would always show only those items that are relevant to the currently active task. Suddenly, purpose-specific diagrams become implied, easy, expected.

I realized that a few simple rules could make such a diagram really work:

• domain elements in the Mylyn context always appear in the diagram
• associations between elements in the diagram are always shown
• removing an element from the Mylyn context causes it to be removed from the diagram
• adding an element to the diagram causes it to be added to the Mylyn context
• diagram elements are positioned according to a default layout algorithm, but remember their positioning if moved manually
• diagram state is saved and associated with the Mylyn task id, such that it is restored when the task is re-activated
• switching active tasks automatically resets the diagram to correspond to the new active task

As my Mylyn task context evolves with its decaying interest model the diagram becomes ever more pertinent. Mylyn landmarks prevent those really essential things from disappearing, other things come and go as needed.

Now that I've been working with automatic domain diagramming I wonder how I ever got by without. Context-driven diagrams are now an essential part of my toolkit that let me focus on the task at hand without distraction.

Eclipse DemoCamp Vancouver

I had a great time at Eclipse DemoCamp last night. I was really impressed by two presentations in particular, both of them about testing tools.

One tool, called Tripoli is a differential code coverage tool. Tripoli shows you the difference in coverage between two execution runs of a program. It looked like a great tool to have for finding code relevant to a specific behaviour or event. Kaitlin Sherwood both created Tripoli and presented at DemoCamp.

The other tool presentation was about an Omniscient Debugger. Going into the DemoCamp, I was thinking "Omniscient, yeah right!". In reality, the debugger can show you exactly what happened on a timeline. You can go back in time and see when variables were assigned specific values. Yes, after it has already happened!! This is the first real time machine that I've seen actually working. The tool is called TOD and was presented by its creator Guillaume Pothier, who is a PHD student at the University of Chile.

Mik Kersten did a presentation about Tasktop and Mylyn. Being an avid Mylyn user, I can see how Tasktop adds a lot of value beyond the basics of Mylyn. As an early beta user of Tasktop ages ago, I'm impressed by how far they've come. At the time I was disappointed by their lack of support for the Mac. Though the Mac is still not officially supported by Tasktop, I've heard that it is much improved. Maybe it's time for another test drive.

My presentation on WikiText went well. If you're interested in the slides, here they are.

All in all, a great night. We ended up at the Lenox pub where I got to talk to more great people. I came home wishing that I'd had the opportunity to talk to more people. It's great to see such a lively Eclipse community in Vancouver.

Mylyn Converts Wiki-Based User Guide To Eclipse Help

The Mylyn user guide is maintained online in the Eclipse wiki. While this has the many benefits of using a collaborative wiki for help content, this has meant that the Mylyn user guide is only available online on the wiki, not from within the Eclipse help system. Until now. Using WikiText Ant tasks Mylyn is able to convert the online wiki content into Eclipse help using an automated process. Users of Mylyn now have access to the user guide by using the standard Eclipse help system, including search, table of contents, etc. even when offline.

Going forward this means that the Mylyn project will be able to continue maintaining the user guide in the Eclipse wiki while also providing the same content from within the Eclipse. Furthermore, other Eclipse projects will be able to do the same by adopting the same approach.

Users wishing to see the user guide in Eclipse can do so by installing the Mylyn weekly build as described here. Eclipse committers that would like to have their wiki-based content available within Eclipse can do so by looking at the Mylyn project (org.eclipse.mylyn.help.ui/build-helper.xml) and by making use of the WikiText stand-alone package.

Mylyn WikiText Stand-Alone Package Available

I'm pleased to announce that the Mylyn WikiText stand-alone package is available for download. The download contains jar files and documentation for including WikiText Ant tasks in your build or for using WikiText APIs outside of an Eclipse runtime.

The WikiText project is still in the incubation phase, thus users should be aware that the download has not yet completed a release review. The availability of WikiText stand-alone download enables users to try out WikiText prior to its release, including the much-anticipated wiki-to-DITA functionality.

This is the first time that WikiText stand-alone jars have been made available since the move of Textile-J to the WikiText project. WikiText includes many improvements and fixes since the Textile-J 2.2 release.

Many thanks to Mik Kersten, Steffen Pingel and others at Tasktop who have facilitated the WikiText project.

Mylyn WikiText targets OASIS DITA

Wiki content has become increasingly popular within the DITA community, as evidenced by the recent announcement of the DITA2Wiki open source project. While most of these projects enable DITA-to-wiki conversions, community interest has been expressed for full-cycle wiki-DITA-wiki transformation capability. Mylyn WikiText hopes to close this loop with the addition of its support for DITA output. Mylyn WikiText can now create OASIS DITA as output from wiki markup.

Mylyn WikiText provides a flexible architecture supporting multiple wiki markup languages. This provides organizations with many options when considering a DITA toolchain, whether it be a one-time conversion of existing wiki assets to DITA, or as an integrated part of a publishing process. Currently supported are MediaWiki, Textile, Confluence, TWiki and TracWiki.

I am also very excited about the recent DITA Open Platform project, which aims to enable Eclipse as a full-fledged DITA authoring and publishing platform. With the capabilities provided by Mylyn WikiText and DITA-op, Eclipse as a platform becomes a compelling cross-platform DITA authoring toolset.

Mylyn WikiText originated from Textile-J, where parsing capabilities evolved to support more than 5 wiki markup languages, conversion of wikitext to DocBook, integrated support for the Ant build system and first-class integration with the Eclipse platform.

Mylyn WikiText is looking for community feedback on wikitext to DITA conversion. Please feel free to contact me directly via email or post a bug or enhancement request to the WikiText component of the Mylyn project.

Eclipse GUI Testing Is Viable With SWTBot

For several years I've struggled with the viability of automated Eclipse GUI test frameworks. Now for the first time I've found an approach that works reliably. This article discusses the approach and details some specific techniques that greatly increase productivity in creating useful tests.

In the past automated GUI test frameworks have failed to meet my needs because they have focused too much on recording and playback. The complexity of the Eclipse environment makes it near impossible to get this approach to work reliably. Simple things such as background jobs and processing become major issues. Testing of GEF-based editors is not possible since all record/playback frameworks rely on identifying controls and widgets -- which GEF does not use.

Recently I've come across SWTBot, which uses a novel approach. SWTBot tests are written in Java, and run inside the Eclipse process as an Eclipse test application. This gives SWTBot full access to the SWT and Eclipse APIs. SWTBot tests are written as JUnits, which makes integration with common technologies such as Ant, Continuous Integration and code coverage tools easy.

But all is not rosy when writing tests with SWTBot. There are some Eclipse-specific idiosyncrasies that come back to bite you such as context menus that are reconstructed when shown and the GEF framework that does not use controls or widgets. That said, SWTBot provides and excellent starting point. Here's how I built on SWTBot to create a powerful test environment that is easy to use.

Initially my tests contained code that looks like this:

public void resetPerspective() {
bot.menu("Window").menu("Reset Perspective...").click();
bot.shell("Reset Perspective").activate();
bot.button("OK").click();
}

Not bad! It's easy to see what's going on. In other cases the code looked more like this:

SWTBotTree tree = bot.tree();
String[] path = name.split("/");
SWTBotTreeItem[] items = tree.getAllItems();
SWTBotTreeItem selectedItem = null;
for (SWTBotTreeItem item: items) {
if (path[0].equals(item.getText())) {
item.expand();
sleep();
selectedItem = item;
break;
}
}
for (int x = 1;selectedItem != null && x

As you can see, it's hard to see the forest for the trees. Way too much code is required to do simple things.

Here's what I did to make things easier:

1. Create classes that directly model the UI parts that you're working with in your tests. For example, instead of directly using SWTBotView to manipulate the Package Explorer view, create a class called PackageExplorer that delegates to SWTBotView and provides richer functionality. For example, in my PackageExplorer class I have the following method:

/**
* select the first element that adapts to the given resource
*/
public void select(final IResource resource)

2. Create a method to find and click context menu items in one go. In SWTBot the finding and clicking occur in two seperate UI runnables. In Eclipse this can cause problems for some context menus as the menu item gets disposed before it is clicked due to a loss of focus.

3. Instead of using sleep() to wait for something to be done, wait for the real thing to be done. A heavily loaded machine can cause processing times to vary. Instead of having a fragile sleep(500L), use a reliable technique to determine when the job is really done.

For example, if you know that your processing is holding a resource lock, post a no-op empty workspace job and wait on it inside your test. It will only be invoked once all other resource locks are released, so when it's complete you're guaranteed that your other job is done:

// ensure that all queued workspace operations and locks are released
ResourcesPlugin.getWorkspace().run(new IWorkspaceRunnable() {
public void run(IProgressMonitor monitor) throws CoreException {
// nothing to do!
}
}, new NullProgressMonitor());

4. Leverage and extend the SWTBot framework with Eclipse-specific behaviour. For example, make use of SWTBot's conditional waiting APIs by creating Eclipse-specific conditions like this one that is used to wait until an editor is opened on a resource:

/**
* a condition that is used to wait for an editor to open on a specific file.
*
* @author dgreen
*/
public class EditorOpenCondition extends DefaultCondition {
private final IFile file;

public EditorOpenCondition(IFile file) {
this.file = file;
}

public String getFailureMessage() {
return String.format("Timed out waiting for editor on %s to open",file.getFullPath());
}

public boolean test() throws Exception {
if (!file.exists()) {
return false;
}
return UIThreadRunnable.syncExec(new UIThreadRunnable.BoolResult() {
public boolean run() {
IEditorReference[] editorReferences = PlatformUI.getWorkbench().getActiveWorkbenchWindow().getActivePage().getEditorReferences();
for (IEditorReference reference: editorReferences) {
 try {
  IEditorInput input = reference.getEditorInput();
  if (input instanceof IFileEditorInput) {
   IFileEditorInput editorInput = (IFileEditorInput) input;
   if (editorInput.getFile().equals(file)) {
    return true;
   }
  }
  if (input instanceof IStorageEditorInput) {
   IStorageEditorInput editorInput = (IStorageEditorInput) input;
   IPath fullPath = editorInput.getStorage().getFullPath();
   if (fullPath.equals(file.getFullPath())) {
    return true;
   }
  }
 } catch (PartInitException e) {
  e.printStackTrace();
 } catch (CoreException e) {
  e.printStackTrace();
 }
}
return false;
}
});
}
}

5. Create a Java-based DSL for often-repeated use of editors or views. For example, for a static class diagram editor you might end up with a DSL that could be used as follows:

DomainModelDsl domainModelDsl = new DomainModelDsl();
domainModelDsl.create(project, "domain.dm");

domainModelDsl.
createEntity("A").
createEntity("B");

domainModelDsl.entity("A").extension("B");

Inside the DSL implementation the dirty work of manipulating the editor occurs. This makes it really fast to create complex tests that cover lots of ground.

6. Create GEF EditPart wappers. For GEF-based editors SWTBot doesn't give a lot of lift. For these you'll need to create classes similar to those provided by SWTBot, but instead of being widget-focused, they'll need to be EditPart-focused.

All in all I'm very impressed with SWTBot, which has finally delivered a viable automated GUI test framework for Eclipse and Eclipse RCP applications. I'm pleased to see that SWTBot has made an Eclipse project proposal, which if approved will hopefully lead to continued improvements and community adoption.

Beyond Rich Text (Part 2): Displaying Images in an SWT StyledText

Recently I wrote about a few simple tricks that make it possible to display more than just text in a StyledText widget. In this article I take it a step further: displaying images in an SWT StyledText widget.

Normally a StyledText widget cannot display images. Why would you want to display images in a text control, you might ask? Well, there are cases where a rich text control is used to render text which may have embedded images. Take for example the Mylyn task editor. It displays bug descriptions and comments. In some repositories (such as Trac or JIRA) it's possible to use markup to display rich text and images. It's common to see screenshots or other visual data included in bug reports.

To display images in a StyledText we use a small variation on the tricks from my previous post. Previously I used an annotation to mark the location of the thing I wanted to draw, and a painter to paint it. This approach works well when we know the size of the thing we're about to draw (such as a character or horizontal rule). In the case of images, it gets more complicated. We need to know the size of the image so that we can create enough space for it in the text.

Since loading image data can be slow (over a network or from a busy disk) we must load the image asynchronously and render it when it becomes available. We can get away with it since users are used to this experience -- that's how web browsers work.

So here's a run-down of the technique I used:

1. Mark the location of the image with an annotation
2. Asynchronously download the image
3. When the image becomes available, determine its height (in pixels)
4. Determine the font in use at the annotated location
5. Using font metrics compute the number of blank lines required to create enough vertical space (in pixels) to display the image without obscuring text
6. Insert the required number of newlines into the text widget
7. Adjust text presentation offsets (some styled regions may need to be translated by the same number of characters as we inserted in the previous step)
8. Associate the image data with the annotation so that the painter will have access to it
9. Cause the text widget to repaint
10. Use the custom painter to paint the image in the empty space that we created

Here's a screenshot demonstrating this technique in action:

It works! The SWT StyledText can now display images!

Though the result is pleasing, there remain a few issues with this approach:

• Images don't flow with the text, that is we can't wrap text around the image on the left or right side, only above and below.
• Images aren't included in clipboard operations such as copy and paste.

If you're interested in the code involved in this technique you can find it here.

The fact that we can do this using standard JFace and SWT APIs is a testament to the quality and completeness of the platform. Thanks Eclipse!

Beyond Rich Text: Tricks Using SourceViewer, Annotations and AnnotationPainter

An SWT StyledText can display text attributes such as bold, italic and strikethrough, alter colors and fonts. What are we to do when we need to go beyond rich text formatting? In this article we present three simple tricks using standard SWT and JFace APIs to create a polished presentation where text attributes alone won't do the job.

Marking The Spot

In order to display something interesting, we need to mark the spot. Normally this is done with a TextPresentation, which specifies character offsets and style ranges. Fortunately the Eclipse APIs give us another mechanism to mark regions of text in an extensible manner: annotations.

The key APIs at our disposal are as follows:

org.eclipse.jface.text.source.Annotation
org.eclipse.jface.text.source.AnnotationModel

Using these APIs we can create annotations and specify their location. An easy place to do this is in your document partitioner. It will be called at the appropriate times to partition your document. By using a RuleBasedPartitionScanner you can modify your rules to create the appropriate annotations.

Now that our document is annotated, we know where we need to draw. To hook up the drawing strategy, we add the following code to the initialization of the SourceViewer:

IAnnotationAccess annotationAccess = new IAnnotationAccess() {
public Object getType(Annotation annotation) {
    return annotation.getType();
}
public boolean isMultiLine(Annotation annotation) {
    return true;
}
public boolean isTemporary(Annotation annotation) {
    return true;
}
};

AnnotationPainter painter = new AnnotationPainter(sourceViewer, annotationAccess);

Now all we have to do is add a drawing strategy to the painter for every kind of annotation that we're interested in drawing. Read on to find out how we do that.

Repainting Characters

Not all fonts can display all characters. This is problematic in an application that is internationalized or one where the user can change the font.

Take for example bullets. Unicode \u2022 can be used to display a solid round bullet with most fonts, but what about an empty one, or a square one? These characters cannot be reliably found in commonly used fonts. The trick we use is to always use the \u2022 character, but repaint it where we want to display something more interesting. By doing that we get the display just right and the text works nicely with copy/paste operations.

To make it work, we create annotations where our bullet characters are in the document. We then hook up a bullet drawing strategy to our painter as follows:

painter.addDrawingStrategy(BulletAnnotation.TYPE, new BulletDrawingStrategy());
painter.addAnnotationType(BulletAnnotation.TYPE, BulletAnnotation.TYPE);
painter.setAnnotationTypeColor(BulletAnnotation.TYPE, getTextWidget().getForeground());

The painter won't invoke our drawing strategy unless the type and type color are also added.

What does our bullet annotation look like? It needs to have enough information for the drawing strategy to know what to draw. In this case the shape of the bullet is dependent on the 'level' of indentation. Here's what I used:

public class BulletAnnotation extends Annotation {

public static final String TYPE = "org.eclipse.mylyn.internal.wikitext.ui.viewer.annotation.bullet";

private final int indentLevel;

public BulletAnnotation(int indentLevel) {
super(TYPE, false, Integer.toString(indentLevel));
this.indentLevel = indentLevel;
}

public int getIndentLevel() {
return indentLevel;
}

}

Now we need to implement our drawing strategy. The drawing strategy must 'erase' the existing bullet character and then draw the new bullet shape where the old bullet was.

We erase the previous character by drawing a rectangle the size of the character in the background color:

// erase whatever character was there
gc.fillRectangle(left.x, left.y, right.x - left.x, lineHeight);

then we draw the new shape:

// now paint the bullet
switch (bullet.getIndentLevel()) {
case 1: // round solid bullet
 gc.setBackground(color);
 gc.fillOval(hcenter - 3, vcenter - 2, 5, 5);
 break;
case 2: // round empty bullet
 gc.setForeground(color);
 gc.drawOval(hcenter - 3, vcenter - 3, 5, 5);
 break;
default: // square bullet
 gc.setBackground(color);
 gc.fillRectangle(hcenter - 3, vcenter - 2, 5, 5);
 break;
}

Here's a screenshot showing an example of this technique in use:

Drawing Non-Characters

Sometimes there's a need to display non-characters. For example, browsers display a horizontal line for <hr /> (horizontal rule). By marking the spot with annotations and registering a custom painter, we can do the same thing. Here's the result we're looking for.

To create this effect we put an empty line in the text we're displaying and annotate it with a HorizontalRuleAnnotation. Drawing the annotation is easy:

public void draw(Annotation annotation, GC gc, StyledText textWidget, int offset, int length, Color color) {
if (gc != null) {
final Color foreground = gc.getForeground();

Point left = textWidget.getLocationAtOffset(offset);
Point right = textWidget.getLocationAtOffset(offset + length);
if (left.x > right.x) {
 // hack: sometimes linewrapping text widget gives us the wrong x/y for the first character of a line that
 // has been wrapped.
 left.x = 0;
 left.y = right.y;
}
right.x = textWidget.getClientArea().width;

int baseline = textWidget.getBaseline(offset);

int vcenter = left.y + (baseline / 2) + (baseline / 4);

gc.setLineWidth(0); // NOTE: 0 means width is 1 but with optimized performance
gc.setLineStyle(SWT.LINE_SOLID);

left.x += 3;
right.x -= 5;
vcenter -= 2;

if (right.x > left.x) {
 // draw the shadow
 gc.setForeground(shadowForeground);
 gc.drawRectangle(left.x, vcenter, right.x - left.x, 2);

 // draw the horizontal rule
 gc.setForeground(color);
 gc.drawLine(left.x, vcenter, right.x, vcenter);
 gc.drawLine(left.x, vcenter, left.x, vcenter + 2);
}

gc.setForeground(foreground);
} else {
textWidget.redrawRange(offset, length, true);
}
}

As before, we hook the drawing strategy up to the painter:

  painter.addDrawingStrategy(HorizontalRuleAnnotation.TYPE, new HorizontalRuleDrawingStrategy());
  painter.addAnnotationType(HorizontalRuleAnnotation.TYPE, HorizontalRuleAnnotation.TYPE);
  painter.setAnnotationTypeColor(HorizontalRuleAnnotation.TYPE,  getTextWidget().getForeground());

Conclusion

Eclipse provides some powerful APIs for hooking into the painting of StyledText. Using some simple tricks we can create powerful polished visuals in an SWT user interface. All of these techniques are applied in the Mylyn WikiText project, where you can find source code that works.

Mik Kersten on Rich Editing for Tasks via Mylyn WikiText

Mik Kersten has writen a great article about WikiText and what it means for Mylyn users, titled Rich Editing for Tasks via Mylyn WikiText. He really does a great job of capturing the essence behind Eclipse's powerful source editing features and how it was applied to the markup editing features of WikiText.

Textile-J Is Moving to Mylyn WikiText!

The Textile-J project announced today that it is moving to Eclipse as the Mylyn WikiText component. Textile-J, now WikiText, is a library for parsing and displaying lightweight markup languages (wikitext). With the addition of WikiText, Mylyn and Tasktop will be able to display markup as it is intended and provide a markup-aware authoring environment.

The move was primarily motivated by my desire to see Mylyn provide first-class support for lightweight markup languages in its task editor. Even though most issue-tracking systems provide support for markup (for example Trac and JIRA), WikiText can also benefit those who use issue-tracking systems with no markup support. For example, using Bugzilla with WikiText will enable users to use markup in their descriptions and comments.

Overall I'm very excited about the move to Mylyn WikiText, as it will not only improve Mylyn and Tasktop but will also benefit existing Textile-J users by ensuring a higher quality library. In designing the move I built some protection for existing Textile-J users into the WikiText project charter so that existing users will be able to continue using WikiText independently of an Eclipse runtime. This is important for users that have integrated Textile-J into their non-Eclipse projects and for those who use the Textile-J Ant tasks.

This project move has been many months in the making, and with the backing and help of the Mylyn team has finally come to fruition. Many thanks go to the team, especially Seffen Pingel, Mik Kersten and Jingwen 'Owen' Ou for their help in making this a reality.

WikiText is available immediately for download from the Mylyn incubation weekly update site. To find out more about WikiText, see the original announcement and the WikiText project homepage.

Eclipse Ganymede on SoyLatte (Mac+Java6+Eclipse)

Mac has no Java 6. The Java 6 beta that Apple released is only 64-bit, which is no good for running Eclipse. Not only is this a major annoyance, it is a problem for developers that need to profile and take heap dumps of their applications.

Some time ago I found this post by Landon Fuller which supposedly launches Eclipse on SoyLatte. While I believe that Landon is a genius, it did not work for me. Well lately I've needed to take some heap dumps, so I tried again.

Here's what worked for me:


/Users/dgreen/Documents/java/soylatte16-i386-1.0.2/bin/java -server -Xms128m -Xmx1536m -XX:MaxPermSize=256m -Dosgi.requiredJavaVersion=1.5 -Dorg.eclipse.swt.internal.carbon.smallFonts -Djava.library.path=/Users/dgreen/bin/jnilib -cp /Applications/eclipse/Eclipse.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar org.eclipse.equinox.launcher.Main -os macosx -ws carbon -arch x86 -showsplash -launcher /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse -name Eclipse --launcher.library /Applications/eclipse/Eclipse.app/Contents/MacOS//../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.100.v20080509-1800/eclipse_1114.so -startup /Applications/eclipse/Eclipse.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar -data /Users/dgreen/Documents/workspace -launcher /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse -keyring /Users/dgreen/.eclipse_keyring -consoleLog -showlocation -vm /Users/dgreen/Documents/java/soylatte16-i386-1.0.2


I also had to extract all *.jnilib files from eclipse/plugins/org.eclipse.swt.carbon.macosx_3.4*.jar and copy them into /Users/dgreen/bin/jnilib

Finally, Eclipse running in a Java 6 VM on a Mac! Apple, time to get your Java act in gear.

Update 2008-08-21: SoyLatte 1.0.2 works for me, but 1.0.3 and OpenJDK 7 darwin packages posted here don't. If you find a solution, please let me know!

Update 2009-05-11: See this related article Eclipse 3.5 (Galileo) on Java 7

Optimizing Screen Real-Estate in Eclipse

I recently read an article stating that the Eclipse Package Explorer and Outline view are wasted space. Mylyn also recommends minimizing or closing the Outline view. Eclipse makes it really easy to make views go away with fast views and minimizing tab groups. An in-place Outline can be activated with Ctrl+O. All of these focus on providing maximum screen real-estate for editors. While these paradigms work really well for many editors, some editors are designed to make use of the Outline view and Properties view as integral parts of the editor. So how does a tool designer resolve this tension between editor styles?

Imagine that a developer is switching from a Java source editor -- where views should be minimized -- to an editor that makes use of the properties view and outline view. When switching back and forth the developer is going to feel friction. To try this out, try using the XSD editor in Eclipse with the properties view.

Enter the editor-controls-view paradigm.

For editors that require the outline and properties views, the editor 'restores' the views to their visible state when the editor gains focus. When such an editor loses focus, the views are minimized to their iconized state. Eclipse could support this editor-controls-view paradigm by allowing editors to declare their affinity with specific views.

This solution provides an optimal use of screen real-estate for both editing scenarios without requiring the developer to take additional action every time she switches editors. It also does not require the developer to switch perspectives, which is a heavier solution to the same problem.

Why Is Eclipse.org Still On CVS?

With the recent announcement of their 1.5 release, Subversion (SVN) has clearly surpassed their original goal "to build a compelling replacement for CVS in the open source community". So why are all Eclipse projects using CVS as their source repository? I explore some of the reasons that go beyond the commonly extolled superiority of SVN.

In a new collaboration with the Eclipse Mylyn project, I've started working with CVS again after about 4 years of active SVN usage. Although I had used CVS previously (for several years) in my career, I had forgotten how bad it really is. After having near-instant branching, fast history retrieval, atomic commits, efficient storage, directory versioning, commit version numbers and web browser access, it is really, really hard to go back. What the Eclipse foundation may not realize is that using CVS is costing them time and money. So why CVS?

1. Tooling

Eclipse committers use Eclipse. From the early days, Eclipse has had really good support for CVS. UI integration is tight, and CVS plug-in features such as workspace-relative patches make it easy to collaborate with others.

The Subclipse plug-in for Eclipse is excellent, however it's still not as good as the CVS plug-in. The Subversive plug-in for Eclipse (which is now an Eclipse.org incubation project) is unusable in its current state.

One Eclipse committer claims that there remain some major usability issues with SVN inside of Eclipse. I'm not sure if he was talking about Subclipse or Subversive -- and in part I agree. If more Eclipse committers were using SVN, then these plug-ins would improve to the same polished level of quality seen in the CVS plug-in.

2. Inertia

Projects that are already using CVS have inertia in the form of commit history, tagging and branching habits, build scripts, etc. It's difficult to overcome this inertia. I've done it on several large projects, and it can be painful.

3. Training

Eclipse committers who use CVS daily are used to CVS and the way it works. Minor differences in UI paradigms of the Eclipse plug-ins and some larger differences with how SVN works take effort to learn and overcome. There is a cost associated with this, and people dislike change.

4. Lack of Organizational Support

Many issues facing the Eclipse project teams would be greatly diminished if the Eclipse foundation provided tools, tutorials, documentation, recommendations and guidelines for making the switch. With the right support, Eclipse could make the switch one project at a time over weeks, months or even years.

5. Time Pressures

Eclipse committers are already busy supporting Eclipse and meeting their milestones. The last thing they want is to be spending time learning new tools and managing project infrastructure. The perceived time-savings of remaining on CVS is a falsehood. I can't tell you how much time I've wasted waiting for a CVS update to complete.

Conclusion

Eclipse and Eclipse users suffer because of continued use of an antiquated source versioning system. The cost in time and money is substantial. With the right vision and organizational support, Eclipse can overcome their roots in CVS and become a more agile organization.

Update July 4 2008: A few Eclipse projects are using SVN for revision control. See Eclipse SVN technology projects for a list of technology projects in SVN.

Focus Your Work Week (Even More): Beyond Mylyn 3.0

Mylyn and Tasktop provide a revolutionary approach to focusing the Eclipse UI. They work on the concept of having a task 'context', that defines a degree of interest (DOI) for resources related to a task. We can make this task context even richer and deliver efficiencies by providing first-class integration of our MDD tools with Mylyn.

When using Mylyn, developers build up their task context using their natural navigation though source code while performing the task. For example, to fix a bug a developer opens the relevant Java source files, and Mylyn will automatically add these to the task context. Based on the frequency and type of interactions with these files, Mylyn will increase or decrease the DOI.

The issue here is that developers start a task with an empty context. The context is empty because Mylyn doesn't know what the developer is doing (or what resources are related to a task), and as a result the Package Explorer and other views in Eclipse are initially empty.

Developers can get around this problem by disabling the Mylyn filter on those views, at the cost of having views filled with all kinds of information that the developer doesn't need to complete the task.

Enter MDD tooling.

MDD tools have the advantage of knowing which things are related. For example, an MDD model is typically used to generate source code. Thus the source code is related to the model. Models rarely exist in isolation: they usually have associations with other models. Thus MDD tools know which source files are related to which models, and which models are related to other models. Given a specific model, most files in a system can be related in some way with a 'degree of separation'.

How can this information be used to our advantage with Mylyn? With first-class Mylyn integration MDD tools can let Mylyn 'see' into the MDD models. Furthermore, these MDD tools can tell Mylyn which files to put into a task context based on how they are related to models in the context.

Developers can start their task by selecting the relevant MDD model, and tell the MDD tools to 'populate' the Mylyn task context. The developer starts their task with a context that already has interesting things in it.

The richer and more expressive the MDD models, the better the integration can be. For example, if the MDD models can express business requirements and methodology, development tasks and other project tasks can be created in a task repository such as Bugzilla or JIRA, prepopulated with a Mylyn context before the developer even starts.

Code coverage tools can be misleading

Software development best-practices encourage a thorough automated test strategy and continuous integration (for me that's Hudson, Ant and JUnit). Even better is to measure code coverage of your tests using a tool such as Cobertura. Code coverage tools will tell you which code was executed by your tests. This kind of analysis is ideal for finding untested code, however it's easy to be mislead by code coverage reports.

Take for example the following ficticious example:

public class Calculator {
BigDecimal currentValue = new BigDecimal(0);
Stack history = new Stack();

public void add(BigDecimal augend) {
 currentValue = currentValue.add(augend);
 commandPerformed(Type.ADD,augend);
}

private void commandPerformed(Type type, BigDecimal augend) {
 // TODO Auto-generated method stub
 System.out.println(type+": "+augend);
}

public void undo() {
 // TODO
}

public void redo() {
 // TODO
}
}

Initially the coverage report will show that all of this class is not covered.

Create a test for the add() function, and the code coverage reports will show that the commandPerformed() has been covered -- even though it is clearly not doing what it is suppoed to.

Be careful when reading code coverage reports -- don't assume that code that is covered is tested. The most a coverage report can tell you is if code is not covered at all by your tests.

Authoring Cross-Platform Wiki Markup

I'll never think of line delimiters the same way again. Mac uses \r, Unix uses \n, and Windows uses \r\n. Who cares? Well, when it comes to writing platform-independent Wiki markup, it matters.

Take for example a document that was written on a Unix platform. Every line is separated by a single \n character. For Wiki markup, paragraphs are separated by an empty line, which is represented in the document as two consecutive newlines \n\n. When a Wiki markup parser converts this document to HTML, it looks for the empty newline (the second \n) and uses that to close the previous paragraph and start a new one.

Now what happens if the document is opened and edited on a Mac? Mac uses \r as a newline. Suppose the Mac user adds a new empty line just before an existing one in this same document. For example:

Prior to editing:

some text\nmore text

After editing:

some text\r\nmore text

Prior to adding the newline, the document contains a single \n character, and afterwards the document contains \r\n. In the users editor (on a Mac) the \r\n will appear visually as two lines, however \r\n happens to be a Windows line delimiter and thus will be parsed by Windows-accomodating Wiki markup parsers as a single line delimiter. In many documents this may not matter, however in Wiki markup this can make a big difference.

Editors that support editing Wiki markup on multiple platforms must be coded carefully to avoid this issue. For Textile-J this means that the end-of-line markers are converted to the platform default when the editor first opens a file.

Who knew that line delimiters could be so important.

Roller and Textile-J

The Roller Support Project has announced plugins for parsing markup using Textile-J.  This brings support for Textile, MediaWiki, Confluence and TracWiki markup to the Apache Roller blog server.  Great to see projects like this picking up Textile-J, good work Anil!

JUnit Made It Easy!

Recently I posted about evolving the architecture of a markup parser.  During its evolution I ended up completely rewriting the parser to a new architecture.  A solid JUnit test suite made it possible to do this while maintaining confidence in the quality of the new code.  This article details the nature of JUnits that make such a switch easy.

Methodology:

While developing the Textile-J markup parser I incrementally improved the parser by adding support for markup syntax.  Paragraphs were added first, then bold text, then lists, you get the idea.  As support was added for each syntax feature, I created one or more JUnit tests to exercise the feature.  I inspected the results of the HTML output from the small snippet of Textile used, and when I was satisfied created one or more assertions in the JUnit.  This made development easy because:

• I could isolate the syntax that I wanted to support and test it independently of other syntax
• Since the tests were small, stepping through the code in the debugger only involved the relevant code
• It's easy to see when something breaks that worked before
• I could be confident in the quality of the parser

The Result

The result of using this methodology was a JUnit test suite of over 100 tests, and I could easily measure progress in supporting Textile markup features.

Markup Parser Rewrite

Rewriting the parser first involved stubbing out the new parser API, and then making the old parser API a facade to the new one.  In doing this my new parser API automatically gained over 100 JUnit tests -- which of course were now failing.  By incrementally improving the new parser until all JUnits passed, I could be certain that the new parser supports the same markup language features.

As I finished the last language feature and saw green for every JUnit test, I had a huge sense of relief.  I now knew that the new parser was working with a guaranteed level of quality.

The Textile-J project now has over 250 JUnit tests.  To get an idea what feedback JUnit can give, take a look at the test report here, and the code coverage report here.

Attributes of a Test Oriented Project

The following are some attributes of my project that make JUnit tests so powerful:

• A JUnit test for every feature
• Meaningful assertions
• An environment that makes running JUnits painless and easy (like Eclipse)
• A development methodology that encourages or requires tests for all new code
• An Ant build script that runs the JUnits whenever a new build is created
• Integrated code coverage (such as cobertura) so that you can see which code is tested

Once your project is set up to run tests, it's easy to add them... so jump that first hurdle and get testing with JUnit!

Evolving a Wiki Markup Parser

Initially my Textile-J open-source project started out with modest goals: to provide a Textile markup parser for Java. Since the project has started, interest in other features drove evolution of the code-base in some interesting directions. This article discusses architectural choices and the evolution of the Textile-J parser architecture to meet changing requirements.

The initial design of the Textile-J parser was a monolithic stateful parser class that used regular expressions and local variables to parse the markup. The results of parsing markup were passed to the XMLStreamWriter interface as HTML elements and attributes. This architecture worked fine initially, as it allowed me to evolve my understanding of Textile markup structure (blocks, phrases and tokens) and the XMLStreamWriter provided a solid means of outputting XHTML.

Feature: Multiple Output Formats

The first major feature request was to support multiple output formats: HTML and DocBook. This required a major rethink of the parser. While XMLStreamWriter was a great means of outputting XML, the parser had to know that HTML was the output format. After some thinking and reviewing the ever-relevant GOF Design Patterns book, I recognized that the Builder design pattern was an ideal fit. So I created a new interface called DocumentBuilder, with an HtmlDocumentBuilder and DocbookDocumentBuilder implementations. Now the parser need not know about the output format, meaning that a single parser could drive multiple output formats.

Feature: Markup Dialects

Next extensions to the Textile markup language were requested. For example, Confluence markup dialect is very similar to Textile but has some additional syntax features. With a better understanding of markup structure (blocks, phrases and tokens) I designed a new 'Dialect' concept that allowed for markup extensions to be added to the base Textile language. The 'Dialect' design was object-oriented, making extensions modular and relatively easy to add without disturbing the base markup parser code.

Feature: Markup Languages

While the approach of markup dialects worked well for adding extensions to Textile parsing, it did not solve the problem of fully supporting new markup languages. Dialects at this point were only capable of extending Textile.

Community demand for supporting new markup languages was increasing, including requests for MediaWiki and Markdown. Supporting these languages was not possible using the existing Textile parser, as the markup rules of Textile were embedded in a single monolithic class. After some prolonged hesitation (this would be a big job), I got down to the design of a complete rewrite of the parser architecture.

The first step was to read up on various markup languages. Most languages that I've looked at use a simple line-based parsing approach that consists of dividing the markup into blocks, phrases and tokens. Blocks are usually multi-line constructs that have certain attributes, such as paragraphs and lists. Phrases are modifiers that affect text on a single line, and tokens are match-and-replace elements in the text.

Requirements of the new design had to include:

• ease of adding new markup languages
• modular object-oriented design for better maintainability
• facilitate comprehensive JUnit tests
• easier learning curve for community contributions
• pluggable architecture
• output format agnostic

Using my experience with the previous Dialect design here's what I came up with:

The parser delegates all language-specific parsing to the Dialect. The Dialect defines a language with a collection of Blocks, phrase modifiers (PatternBasedElement) and tokens (PatternBasedElement). Blocks implement rules specific to paragraphs, lists, tables, etc. Concrete PatternBasedElementProcessor classes know how to emit portions of content affected by markup (phrase modifiers or tokens).

So far the following languages have been imlemented using this new architecture:

• Textile
• MediaWiki (a la WikiPedia fame)
• Confluence

I hope to see contributions from the community for supporting other languages, such as Markdown and Creole.

Hacking SWT

Talking to Mik at Tasktop about what it will take to bring his revolutionary Mylyn-based product to the mac platform, Mik pointed me to an issue with the way the Safari-based SWT browser widget accesses GMail. Though Safari is a supported browser for GMail, the embedded Safari browser in eclipse is not recognized by GMail. This post highlights the ups-and-downs of my investigation into this issue.

Others at Tasktop had discovered that the SWT browser works if gmail is accessed via the http://mail.google.com/mail/?nocheckbrowser URL, which was promising. This lead me to believe that the user-agent header of an embedded Safari was different than that produced by Safari launched from the desktop. A quick test browsing to http://whatsmyuseragent.com proved my suspicions were correct.

Here were the user-agent headers:

Safari Inside Eclipse: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko)

Stand-alone Safari: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko) Version/3.0.4 Safari/523.12.2

The next step then was to see if Safari provides an API for changing the user-agent header. Having already installed XCode on my system, it was easy to search the API documentation. I found a section called 'Spoofing', which is specifically about an API for modifying the 'user-agent' header:

• setCustomUserAgent
• setApplicationNameForUserAgent

By the looks of the difference in user-agent headers, the second one was exactly what I was looking for.

Knowing that SWT involves native calls to Cocoa, I was a little wary about the next step. Looking at Safari.java I could see in the create() method other calls to the Safari web view to configure various properties. All of them were performed using JNI native calls, one of the various Cocoa.objc_msgSend variants. Unfortunately, there wasn't one with the signature that I needed.  The one that I needed would look like this:

public static final native int objc_msgSend(int object, int selector, String string);

I added the method to Cocoa.java, and added a call to it in Safari.create() as follows:

// [webView setApplicationNameForUserAgent:];
Cocoa.objc_msgSend(webView, Cocoa.S_setApplicationNameForUserAgent,"Safari/unknown");

 The next step is to provide the native implementation of the method.

Using the eclipse search command, I could see that the other objc_msgSend implementations were in cocoa.c  Never having done JNI before, it was time to do a little looking around.  The SWT FAQ had most of what I needed, indicating how to build the native libraries, etc.  The tricky parts were as follows:

• how to determine the mangled function name for the native method?
• now to convert the jstring argument to an NSString?

For the first issue, a little guessing and a look at the JNI spec where it talks about mangling and I got it mostly right.  Stubbing out the method, building it (according to the instructions in the SWT FAQ) and I got a java.lang.UnsatisfiedLinkError. Running it again in the debugger, I was able to see the method name it was looking for and fixed up the name of the method in cocoa.c.  Rebuild, re-run, and so far so good!  My stubbed out method was being called whenever a new SWT Browser control is created. This is what I had so far:

#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;
return rc;
}
#endif

So now on to the method implementation: how to call the right method?  A quick google on jstring to NSString resulted in the following:

#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;

Cocoa_NATIVE_ENTER(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
const char *str = (*env)->GetStringUTFChars(env, arg3, 0);

/* use NSString as follows: [NSString stringWithCString: str] */

(*env)->ReleaseStringUTFChars(env, arg3, str);
Cocoa_NATIVE_EXIT(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
return rc;
}
#endif

Okay, so now all I had to do is actually send the Objective-C message to the object!  Copying from other examples in the same file, here's what it I ended up with:

#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;

Cocoa_NATIVE_ENTER(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
const char *str = (*env)->GetStringUTFChars(env, arg3, 0);

rc = (jint)objc_msgSend((id)arg0, (SEL)arg1, [NSString stringWithCString: str]);
(*env)->ReleaseStringUTFChars(env, arg3, str);
Cocoa_NATIVE_EXIT(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
return rc;
}
#endif

I rebuilt the native libraries, re-ran eclipse, and voila!! It worked!  The SWT Safari browser widget was now sending a user-agent string as follows:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko) Safari/unknown

I browsed back to GMail with this new patch, and GMail now recognized Safari embedded in eclipse as a supported browser.  For my first foray into hacking SWT, using JNI and Objective-C, it was a great success.... now if only the SWT team will accept my patch ;)

updated Feb 29: the fix was committed by the SWT team: see https://bugs.eclipse.org/bugs/show_bug.cgi?id=220836 for details

Integrating scripting languages into your DSL using JSR-223

I've been building tools for Model-Driven Engineering (sometimes known as Model-Driven Development) now for about 11 years, but this is the first time that I've integrated scripting capabilities into a DSL using Java.  The opportunity came up when creating a DSL for modernizing legacy data for my company MAKE Technologies Inc.  Following is a summary of how this was done.

The idea is that the functional capabilities of the DSL can be extended by scripting the behaviour in the DSL itself.  By integrating with JSR-223 APIs we can support as many scripting languages as are JSR-223 compliant.  By the looks of scripting.dev.java.net, there are many including Java itself, Groovy, JRuby, Python, BeanShell, ECMAScript (JavaScript) and PNuts.

The DSL defines the concept of a 'caster', which can cast an arbitrary string value to some other value.  For example, a trivial boolean caster might consist of a function that can cast a 'Y' or 'N' value to a boolean true or false.  So we define it as follows:

<caster language="groovy" name="YesNoToBooleanCaster">
String cast(text) {
   if (text == 'Yes') {
       return true
   } else if (text == 'No') {
       return false
   } else {
       throw new Exception("Unexpected value '$text'");
   }
}
</caster>

Elsewhere in the DSL we can refer to the new caster by it's name YesNoToBooleanCaster.

The code to make the new caster work is relatively simple using the javax.script APIs:

String languageName = "groovy"; // get the language from the DSL instance
String script = ""; // get the script from the DSL instance

ScriptEngine scriptEngine = new ScriptEngineManager().getEngineByName(languageName);
Invocable invocable = (Invocable) scriptEngine.eval(script);

Object value = invocable.invokeFunction("cast", "Yes"); // test it out
assert Boolean.TRUE.equals(value);

With a little more work, I was able to hook up auto-suggest for the language attribute in the DSL editor (ScriptEngineManager can tell you which languages are supported) and provide validation of the scripting (ScriptEngine.eval can be used to detect script syntax errors).

All of this and the script runs very quickly at runtime, thanks to bytecode compilation by most supported scripting languages!

The only real issues that I encountered were:

• extracting useful error messages from exceptions thrown by ScriptEngine.eval can be tricky.  For example, JRuby provides the information in a getException() method on the Exception object.  Some reflection and chained exception unraveling was used to overcome this problem in a language-independent manner.
• Older Java 6 VMs (developer preview versions) have old preview versions of the JSR-223 APIs... stick to a non-beta version of the Java 6 VM if you can, or if you're developing for Macs then you may have to catch NoSuchMethodError and use some reflection magic.

All in all I was very impressed with the ease of using JSR-223.  The end result when used with DSLs is that it is very easy to provide an extensible DSL API with a very low barrier to entry.