Green's Opinion

Testing REST Web Services with JPA and Spring

2010-07-29T09:00:00.000-07:00

REST Web Services can be particularly difficult to test, with the need for networking, a web container, multiple threads and transaction management creating extra complexity beyond your standard unit test. In this article I demonstrate patterns designed to address this complexity while enabling complete testing of your REST web service stack.

The ideal web service unit test will use the same principles discussed in my previous article:

clean database with mocked data so that we can reliably test specific scenarios
rollback to avoid side-effects after our test
in-memory database so that no environment setup is required

In addition, we’ll need to run a web container for our unit test. To maintain a zero-setup test environment, we’ll have our unit test start a web container for the purpose of running our web service. By having each test start and stop the web container, tests become very simple to run and we’re guaranteed that our tests will have a clean environment with no external dependencies.

Web Container

Normally starting a web container is difficult to do in a unit test, and container startup time can be a problem — however with the right choice of web container we can overcome these problems. Winstone is a small, fast web container that is designed to be embedded in Java programs. By using Winstone, we’ll be able to start and stop the web container as part of our unit test setup and tear-down with relative ease.

Starting Winstone is as simple as this:

 Map args = new HashMap();
 args.put("webroot", pathToWebRoot);
 args.put("httpPort", String.valueOf(port));
 Launcher.initLogger(args);

 // start winstone
 winstoneLauncher = new Launcher(args);

Winstone only takes a couple of seconds to start up. Shutting it down is as simple as this:

 if (winstoneLauncher.isRunning()) {
  winstoneLauncher.shutdown();
 }

To hide the details of winstone from our unit tests, we can create a class WebApplicationContainer, which is responsible for starting choosing a port, starting/stopping Winstone, and providing a base URL for our tests:

public class WebApplicationContainer {

 private static final Random random = new Random(System.currentTimeMillis());

 private Launcher winstoneLauncher;

 private File webRoot;

 private int port;
 
 /**
  * start the webserver, guarantees that the webserver is started upon return.
  * @see #stop()
  */
 @SuppressWarnings({ "unchecked", "rawtypes" })
 public void start() {
  if (winstoneLauncher != null) {
   throw new IllegalStateException("already started");
  }
  port = findAvailablePort();
  
  
  Logger log = Logger.getLogger(WebApplicationContainer.class.getName());
  log.fine("Starting web container on "+getBaseUrl());
  
  Map args = new HashMap();
  try {
   args.put("ajp13Port", "-1");
   args.put("useJasper", "false");
   args.put("webroot", webRoot.getAbsolutePath());
   args.put("httpPort", String.valueOf(port));
   Launcher.initLogger(args);

   // start winstone
   winstoneLauncher = new Launcher(args);

   // wait for Winstone to finish starting up
   // we do that by attempting to connect via socket
   final int maxRetries = 150;
   for (int x = 0; x < maxRetries; ++x) {
    if (testForSuccessfulStartup()) {
     break;
    }
    if (x == maxRetries - 1) {
     throw new IllegalStateException(
      String.format("Connection to localhost:%s failed."+
       "  Did the web container start up successfully?"));
    }
    // wait and then try again
    Thread.sleep(100L);
   }
   Logger.getLogger(WebApplicationContainer.class.getName()).
    info("Started web container at "+getBaseUrl());
  } catch (Exception e) {
   throw new IllegalStateException(e);
  }
 }

 private int findAvailablePort() {
  ...
 }
 
 private boolean testForSuccessfulStartup() {
  // test to see if the web container is listening on the address/port combo
  try {
   Socket socket = SocketFactory.getDefault().createSocket("localhost", port);
   socket.close();
   return true;
  } catch (UnknownHostException e) {
   throw new IllegalStateException(e);
  } catch (IOException e) {
   return false;
  }
 }

 /**
  * stop the web container
  * @see #start()
  */
 public void stop() {
  if (winstoneLauncher == null) {
   throw new IllegalStateException();
  }
  
  winstoneLauncher.shutdown();
  winstoneLauncher = null;
 }

 public String getBaseUrl() {
  return String.format("http://localhost:%s/", port);
 }
 ...
}

REST Service Unit Test

A complete test for our web service ends up looking like this:

@RunWith(SpringJUnit4ClassRunner.class)
@Transactional
public class BlogServiceClientTest extends BlogServiceTest {
 @Autowired
 private BlogServiceClient blogServiceClient;

 @Autowired
 private WebApplicationContainer webContainer;

 @Before
 public void before() {
  webContainer.setWebRoot(computeWebRoot());
  webContainer.start();

  blogServiceClient.setBaseUrl(webContainer.getBaseUrl()+"api");
  // use the client instead of the service directly
  service = blogServiceClient;
 }

 @After
 public void after() {
  if (webContainer.isStarted()) {
   webContainer.stop();
  }
 }
}

You might be thinking "where are the @Test methods?". The beauty of our web service test is that it extends our service test: all of the test methods we were running before using direct method calls are now also run over our web service protocol. By using inheritance, other than setup of our web service there’s nothing else to do. In other words, our service tests are run twice: once using Java method calls, and again over-the-wire using our web service. This ensures that we see consistent behaviour in both modes of execution.

BlogServiceClient is an implementation of BlogService (our service interface under test) based on Spring’s RestTemplate. It’s a REST client for our service, implementing the same Java interface as our server-side service. This enables us to substitute the REST client service implementation in our test.

Transaction Management and JPA

Now that we’re able to start our web service and run tests we’re done, right? Not quite: we still need to manage transactions and our test data. This is somewhat more complicated with our web service: recall that we want our unit test to have the following qualities:

clean database on startup
run in a transaction that rolls back after the test
mocks all of its own data

Employing the techniques we’ve already established in Patterns for Better Unit Testing with JPA helps, but is not quite enough.

Our service implementation is running in a web container. The web container services incoming HTTP requests with threads in a thread pool. Our service is pretty much guaranteed to be running on a distinct thread, not the same thread as that running our unit test method. As a result our service bean will be running in it’s own transaction context. There are two problems that arise from this:

our service won’t be able to see the data we’ve mocked up in our test
our service method will commit its transaction

Both of these things are undesirable. To overcome this problem we’ll apply a little trickery: we’ll provide an entity manager that is shared by both the test thread and the web container thread servicing our HTTP requests, and we’ll prevent any commit or rollback from occurring on that entity manager while it’s in use in a unit test. We do this by wrapping the entity manager factory in our test environment:

<bean class="greensopinion.restexample.test.jpa.TestEntityManagerFactory" 
 id="blogDomain">
 <property name="delegate">
  <bean
   class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
   <property name="dataSource" ref="dataSource" />
   <property name="persistenceXmlLocation" 
    value="classpath*:/persistence-test.xml"/>
  </bean>
 </property>
</bean>

Our TestEntityManagerFactory employs the standard delegate pattern, with the exception that it changes the default behaviour of createEntityManager():

 public EntityManager createEntityManager() {
  if (theEntityManager == null) {
   theEntityManager = delegate.createEntityManager();
  }
  return new TestEntityManager(theEntityManager);
 }

It creates and returns a singleton entity manager instance. This allows us to share a single entity manager between our unit test and the web container. We also wrap the entity manager so that we can control it’s lifecycle (we prevent close()) and that of it’s transaction (we prevent commit() and rollback()).

While we know that the EntityManager is not thread safe, it’s okay to share it between threads providing that no two threads access it at the same time. Since our REST client will block on any calls to the server, we’re not at risk of concurrent access.

Source Code

Complete, working source code demonstrating these patterns and others is available on GitHub:

http://github.com/dgreen99/greensopinion.restexample

The source contains two Eclipse projects: one for the web application, and one for unit tests. The unit tests all run and pass, and the web application can be started and deployed with a MySQL database.

Service Patterns

The service is split into the following:

Service interface – defines the service contract
Service bean – implements the service over JPA
Service controller – exposes the service using Spring REST
Service client – implements client end of the REST service, used in our tests

This gives us a good separation of concerns, and client code need not always know if it’s calling an in-process or out-of-process service.

JPA domain objects are exposed by the service interface. This helps in a few ways:

The service is usable in other parts of the application that deal with the domain
It helps to reduce the amount of code that must be written

However it has drawbacks:

It exposes the design of our domain, limiting our ability to change it in the future
Changes to the domain may inadvertently break the service interface for third-party clients
The service controller must be careful to expose shallow copies of our domain objects

This approach may be feasible for some projects. It’s common to see transfer objects as well, which allows a service to be explicit about it’s data model without exposing the domain implementation details.

Missing Functionality

Notably this sample code is missing two crucial aspects of a normal web application:

Exception handling should be built-in to our service controller so that errors can be propagated gracefully.
Data validation is not performed by the service bean.

These aspects of the sample application are left unimplemented.

Summary

Testing REST web services can be easy by following the following patterns:

implement a web service client to exercise your service from unit tests
run the web service in your tests with a lightweight embedded web container
share the entity manager and transaction context between the service and test

By applying these straight-forward techniques it’s possible to overcome the complexity inherent in testing a web service, enabling better coverage and higher quality applications.

References

Technologies used in this project:

REST services are implemented using Spring REST and Jackson
EclipseLink is used as a JPA provider
JUnit is used for unit tests
Hypersonic is used for an in-memory database
Winstone is used for an embedded web container

Patterns for Better Unit Testing with JPA

2010-07-15T08:07:00.000-07:00

Over the years I’ve run across many projects, which tend to fall into one of two categories: those projects that have great confidence, energy and momentum, running hundreds (or thousands) of unit tests dozens of times every day; and those projects where the effort of writing and running unit tests is high enough that tests aren’t maintained and as a result no one on the team really knows if their software works. So how do we get our project into the first category? Below I highlight a few patterns and practices that significantly lower the bar to creating a solid test suite, enabling your team to adopt a practice and culture of complete test coverage.

Setup and Speed

Configuration, setup and maintenance of environments creates a significant barrier to a well-exercised and maintained test suite. Ideally we want to enable running tests with zero setup. Data-intensive applications need a database, which are notoriously hard to configure and setup. We’ll use a database for our tests, but instead of connecting to an external one the database will start up and run as part of the unit test. This gives us the following:

zero setup: no installation, no shared machine, no maintenance, no JDBC urls, no passwords
in-memory and fast
can run anywhere, even when not on a network
clean, with no rogue data that could affect our tests
no deadlocks

My first reaction when I saw this approach in use was “how can tests be meaningful if they're not run on the same database software that is used in production”? The answer is two-sided. We should be running these tests on the same database software. In fact, I encourage you to do that — on your continuous integration build server. The tests are meaningful in that they can make assertions and exercise your code. Database platform-specific issues can be flushed out on the build server, which can run these same tests using a different configuration.

To set this up I recommend using Apache Derby or Hypersonic. Here’s how it’s done with Hypersonic and Eclipselink as our JPA provider:

add hsqldb.jar to the test project classpath (this is for Hypersonic)
configure your JDBC connection to use an in-memory database URL as follows: jdbc:hsqldb:mem:tests
configure your persistence.xml with the right SQL dialect. For Eclipselink this is done by setting eclipselink.target-database to HSQL as follows:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" 
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" 
 xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
  http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
    <persistence-unit name="blogDomain" transaction-type="RESOURCE_LOCAL">
        <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
        ... snip ...
        <properties>
            <property name="eclipselink.target-database" value="HSQL"/>
            <property name="eclipselink.ddl-generation" value="create-tables"/>
            <property name="eclipselink.ddl-generation.output-mode" 
                             value="database"/>
            <property name="eclipselink.weaving" value="false"/>
            <property name="eclipselink.logging.level" value="INFO"/>
        </properties>
    </persistence-unit>
</persistence>

With this configuration the test suite will create a new in-memory database every time it’s run, and Eclipselink will create the necessary tables to persist our data. We’ve managed to eliminate the need for any kind of setup or external database to run our tests. That’s great, but now we’ve got an empty database, where does our data come from?

Test Data

Data-intensive applications need data for their tests — that’s a given. For our tests to run reliably and check specific scenarios every time, we need the data used by each test case to be the same every time tests are run. The best way to do this is to have our tests mock up the data. Mocking data can be cumbersome, however that can be alleviated by using mock factories for our domain objects. Mock factories populate domain objects with values such that they’re ready to use as-is. Where needed, tests can alter the state of domain objects to create the desired scenario. Here’s an example:

 @Test
 public void testUpdateBlog() {
  Blog blog = MockFactory.on(Blog.class).create(entityManager);
  entityManager.flush();
  entityManager.clear();
  
  blog.setName(blog.getName()+"2");
  
  Blog updatedBlog = service.updateBlog(blog);
  assertNotNull(updatedBlog);
  assertNotSame(updatedBlog,blog);
  assertEquals(blog.getName(),updatedBlog.getName());
  assertEquals(blog.getId(),updatedBlog.getId());
 }

In this example the test verifies that the Blog entity is properly updated by the service. Notice that the test didn’t have to populate the blog object before persisting it: all of the required values were filled in by the MockFactory.

For this to work, we’ll need a MockFactory implementation. This is what ours looks like:

/**
 * A factory for domain objects that mocks their data.
 * Example usage:
 * <pre><code>
 * Blog blog = MockFactory.on(Blog.class).create(entityManager);
 * </code></pre>
 * @author David Green
 */
public abstract class MockFactory<T> {

 private static Map<Class<?>,MockFactory<?>> factories = 
      new HashMap<Class<?>, MockFactory<?>>();
 static {
  register(new MockBlogFactory());
  register(new MockArticleFactory());
 }
 private static void register(MockFactory<?> mockFactory) {
  factories.put(mockFactory.domainClass,mockFactory);
 }
 @SuppressWarnings("unchecked")
 public static <T> MockFactory<T> on(Class<T> domainClass) {
  MockFactory<?> factory = factories.get(domainClass);
  if (factory == null) {
   throw new IllegalStateException(
    "Did you forget to register a mock factory for "+
      domainClass.getClass().getName()+"?");
  }
  return (MockFactory<T>) factory;
 }
 
 private final Class<T> domainClass;

 private int seed;
 
 protected MockFactory(Class<T> domainClass) {
  if (domainClass.getAnnotation(Entity.class) == null) {
   throw new IllegalArgumentException();
  }
  this.domainClass = domainClass;
 }

 /**
  * Create several objects
  * @param entityManager the entity manager, or null if the mocked objects
  *            should not be persisted
  * @param count the number of objects to create
  * @return the created objects
  */
 public List<T> create(EntityManager entityManager,int count) {
  List<T> mocks = new ArrayList<T>(count);
  for (int x = 0;x<count;++x) {
   T t = create(entityManager);
   mocks.add(t);
  }
  return mocks;
 }

 /**
  * Create a single object
  * @param entityManager the entity manager, or null if the mocked object
  *        should not be persisted
  * @return the mocked object
  */
 public T create(EntityManager entityManager) {
  T mock;
  try {
   mock = domainClass.newInstance();
  } catch (Exception e) {
   // must have a default constructor
   throw new IllegalStateException();
  }
  populate(++seed,mock);
  if (entityManager != null) {
   entityManager.persist(mock);
  }
  return mock;
 }

 /**
  * Populate the given domain object with data
  * @param seed a seed that may be used to create data
  * @param mock the domain object to populate
  */
 protected abstract void populate(int seed, T mock);
 

 private static class MockBlogFactory extends MockFactory<Blog> {
  public MockBlogFactory() {
   super(Blog.class);
  }
  
  @Override
  protected void populate(int seed, Blog mock) {
   mock.setName("Blog "+seed);
  }
 }

 private static class MockArticleFactory extends MockFactory<Article> {
  
  public MockArticleFactory() {
   super(Article.class);
  }
  
  @Override
  protected void populate(int seed, Article mock) {
   mock.setAuthor("First Last");
   mock.setTitle("Article "+seed);
   mock.setContent("article "+seed+" content");
  }
 }
}

This implementation uses static methods and a static initializer. It’s not very Spring-like. If you prefer you could get rid of all the statics and instead have Spring inject (autowire) your mock factories into your test classes.

In our implementation we use an integer seed to help us produce values that vary. This implementation is fairly trivial, however if needed we could apply more advanced techniques such as using data dictionaries to source data. Such mock factories should populate enough values that the mocked domain object can be persisted: all not-null properties should have values. To ensure that this is true over the lifespan of our project, it’s important to have a test:

/**
 * test that {@link MockFactory mock factories} are working as expected
 * 
 * @author David Green
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration( { "/applicationContext-test.xml" })
@Transactional
public class MockFactoryTest {

 @PersistenceContext
 private EntityManager entityManager;

 @Test
 public void testCreateBlog() {
  Blog blog = MockFactory.on(Blog.class).create(entityManager);
  assertNotNull(blog);
  assertNotNull(blog.getName());
  entityAssertions(blog);
  entityManager.flush();
 }
 @Test
 public void testCreateArticle() {
  Blog blog = MockFactory.on(Blog.class).create(entityManager);
  Article article = MockFactory.on(Article.class).create(null);
  article.setBlog(blog);
  entityManager.persist(article);
  assertNotNull(article);
  assertNotNull(article.getTitle());
  assertNotNull(article.getContent());
  entityAssertions(article);
  entityManager.flush();
 }
 private void entityAssertions(AbstractEntity entity) {
  assertNotNull(entity.getId());
  assertNotNull(entity.getCreated());
  assertNotNull(entity.getModified());
 }
}

The key behind this approach is that each test mocks its own data. Where data models are signifcantly more complex, utility functions can use these mock factories to create common scenarios.

Now that we’ve got an easy way to produce data in our tests, how do we eliminate potential for side-effects between tests? That part is easy: we ensure that transactions are rolled back after every test. In our example we’re using Spring to run our tests via SpringJUnit4ClassRunner. Its default behaviour is to roll back after each test is run, however if you’re not using Spring you should have something like this in an abstract test case class:

 /**
  * Overriding methods should call super.
  */ 
 @After
 public void after() {
  if (entityManager.getTransaction().isActive()) {
   entityManager.getTransaction().rollback();
  }
 }

Summary

We’ve seen how we can make unit testing easy with a few simple patterns:

in-memory database
mocked data
roll back transactions after each test

By employing these simple techniques, tests are easy to write and run. Your team will get addicted to thorough test coverage as the number of tests being run increases and provides tangible feedback of their progress.

In my next article I’ll post complete working source code and show how these techniques can be taken a step further in testing Spring REST web services.

Joining the Tasktop Team

2010-06-15T09:40:00.000-07:00

My experience with Mylyn began shortly after it was released in 2006 as a way to access bug reports from my IDE. It didn’t take long for me to discover how radically my effectiveness and productivity were affected by Mylyn’s ability to track the context of each task and trigger my situational memory. Mylyn’s effect on my work was so profound that over the following years I was inspired to create several extensions for Mylyn, including automatic creation of task contexts for model-driven development (MDD) tools, context-driven domain diagrams, and WikiText. It was through these projects and others that I came to know a world-class team, the creators of Mylyn at Tasktop.

Mylyn’s frameworks and Tasktop’s products have become the hub of a new movement in software development, focused on integrating Agile methodologies and ALM tools to provide the groundbreaking productivity gains of the task-focused interface. It’s with great excitement that I join the team at Tasktop as Vice President of Engineering. I look forward to working with one of the most innovative software development companies that exists at the center of the Agile and ALM space and leading edge of the Eclipse community.

You can find out more by reading the press release.

Lava Lamps, Android and Continuous Integration

2010-05-12T08:45:00.001-07:00

A few years ago I came across this very cool setup by Ken Nichols where Lava Lamps are used to indicate build status. Real, tangible feedback on your builds somehow has that cool factor that just can't be ignored. With circa-2004 lava lamps as inspiration, the latest Hudson Helper for Android has been updated.

Android devices have a multi-color LED that can be used for notifications. On my Nexus One, the trackball doubles as the LED, resulting in a glowing orb. Hudson Helper will monitor your builds and make the glowing orb throb when your builds change status to red, yellow or blue.

Fix your build and the orb throbs blue.

That and Hudson Helper for Android now has support for host aliases (a la /etc/hosts), making integration with misconfigured Hudson servers and VPNs much easier.

To find Hudson Helper search for it in the market, or scan this QR code with your Android device:

Faster Builds: A Continuous Integration Build Strategy

2010-04-19T15:00:00.000-07:00

We all know that Continuous Integration (CI) can deliver real, tangible benefits to our projects. So why are the benefits so elusive? Many software projects claim to be exercising CI, but have builds that run for 30 minutes or more, or worse, just have a nightly build. Build times that exceed a few minutes are excessive; all too commonly we see build times reaching 20 minutes to an hour or two. Real projects tend to have build times that gradually increase as the project evolves, resulting in a failure to reach the full potential that CI promises to deliver. Luckily for many of these projects, there's a solution that can get your project back on track.

I've seen many projects with this problem, and they all have the same pattern: a monolithic build that builds components in a serial fashion. While many of these projects are modular, where modules can be built independently, dependencies between modules prevent them from being built in parallel.

So how do we get shorter build times? We cheat.

By building modules in independent CI jobs, we can get faster feedback for independent modules of our project without reducing the total build time. As we make discrete changes to the source code, only the affected modules will rebuild.

With this approach each module is built separately from the others. As a result we get very fast feedback as to the impact of our changes, possibly even before downstream modules have finished building. Furthermore, in many cases only part of the project will need building — thus resulting in a reduction in total build time. I know, I said that total build time is not reduced using this approach, but for many cases the total build time is in fact reduced since we don't always need to build the whole project!

So how do we employ this approach in practice? These are the steps that I took on a recent project:

Identify logical modules of related functionality in your project. For Eclipse projects, this could be related plug-ins. For my project which involves about 100 Eclipse plug-ins (OSGi bundles) this meant splitting the bundles into about 7 modules.
Next, change your build process so that each module builds independently. To do this I started with the module that would need to be built without any dependencies on other modules. After creating a CI job to built it, I moved on to the next module, which could now download its dependencies from the first CI job. Repeat this process until each module is built by an independent CI job that downloads its dependencies as binaries from other CI jobs.
After all modules were building on the CI server using binary dependencies, some minor tweaks were required to limit job concurrency and unnecessary thrashing on the build server. This is where the power of Hudson really shines.
1. Configure each module's job under Post-build Actions to trigger the next module in the series of dependant modules: check Build other projects and list them there.
2. Using the Locks and Latches plugin ensure that all module jobs are using the same lock, preventing them from all building at once.

Here's what I ended up with:

Before: 1 job, 1100 tests, 30 minute build time.

After: 7 jobs, 1075 tests, 20 minute 30 second build time, with a break down as follows in dependency order:

Job 1:	20 seconds
Job 2:	26 seconds
Job 3:	1 minute 31 seconds
Job 4:	5 minutes 54 seconds
Job 5:	5 minutes 45 seconds
Job 6:	5 minutes 12 seconds
Job 7:	1 minute 12 seconds

The net result is that most of the time, we get feedback within 6 minutes, whereas we were having to wait 30 minutes. Even if you can't parallelize your build, splitting your build into independent modules that build serially can produce measurable benefits such as faster feedback and shorter build times.

From iPhone to iPad: Creating a Universal Application

2010-04-08T13:29:00.000-07:00

I’ve recently had the pleasure of porting an iPhone app to the iPad. The process was remarkably easy. This brief article covers the technical issues as well as the relatively minor UI design issues that came up when doing the port.

Apple’s iPhone OS SDK is capable of producing ‘Universal Applications’ which run on multiple devices, including the iPad and iPhone. The recommended approach for existing iPhone app projects is to run an upgrade wizard, which worked beautifully for me. After running the wizard I was initially concerned that nothing happened, because the wizard simply exited with no indication of what occurred — but after comparing project files against the version in my VCS there were obviously some minor changes to the project files, and one new file, the iPad’s main view MainWindow-iPad.xib.

Immediately after running the upgrade wizard I was able to run the app in the iPad simulator. Very impressive. Having implemented the app with relative layouts, the whole UI scaled very nicely to the larger screen. Running through some basic workflows in the app, I expected some things to break — but no, everything worked perfectly!

There were two places in the code where visually things needed adjusting based on the size of the screen. Initially I tried Apple’s recommended approach of conditional coding based on (UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad), however I quickly discovered that it doesn’t work for iPhone 3.1.x apps because UIUserInterfaceIdiomPad is not defined. Clearly Apple missed something here. After thinking through some creative ways to make this work (see Jeff LaMarche’s blog for an approach that works) I realized that instead of trying to detect the platform, using the screen size was a better approach. An easy way to do that is as follows:

[[UIScreen mainScreen] bounds]

The iPad has some new UI paradigms which are well-described in the iPad HIG. Most apps will benefit from following these guidelines, however I was surprised at how good the app looked in its current form on the iPad.

The moment of truth of course came when I deployed the app to an iPad. Apart from having to download a new provisioning profile from Apple, the process was easy. Within a few minutes it was up and running flawlessly on the device. Kudos go to Apple for making it so easy to port apps from the iPhone to iPad. It’s hard to imagine it being any easier.

Crowdsourcing Documentation Made Easy

2010-03-12T07:32:00.000-08:00

Momentum is building around documentation best-practices at Eclipse. EGit, Eclipse's Git integration project, recently became the latest project to adopt a crowdsourcing approach to documentation, and is reaping the benefits: in one day they've already seen the contribution of Git for Eclipse Users from Alex Blewitt.

They are able to do this by taking advantage a new feature of the latest Mylyn WikiText, which can generate Eclipse help directly from a wiki. While it was possible to process individual pages before, now Mylyn WikiText makes it easy to process multiple wiki pages at once while automatically downloading images, maintaining page cross-references, and building a unified table of contents.

You can learn more about crowdsourcing documentation at EclipseCon 2010, where Chris Aniszczyk and I are co-presenting Documentation: Single-Sourcing, Crowd-Sourcing And Other Voodoo. If you can't wait two weeks, you can find out how they're doing it at EGit from their contributor guide.

Updated 4/21/2010: Thanks to Chris, a working example with documentation and source is available at DocumentationGuidelines/CrowdSourcingExample

Good Things Come In Threes

2010-02-05T21:31:00.000-08:00

Hundreds of developers at Eclipse.org are on the cusp of something great. After lamenting the pains of the dark ages and thinking it wouldn't end, Eclipse's focus on best practices has resulted in a decision to move to Git. With tooling integration being such a key factor, it's no surprise that everyone's eyes are on the EGit project. I figured it was time to give it a spin after a recent call for project trials. This is how my adventures with GitHub, EGit and Mylyn began.

To kick it off, reading the progit book helped with the Git basics, and Lars Vogel's excellent EGit tutorial got me up to speed with EGit. Knowing the only way to learn is to do it, I created an account on GitHub and kicked off a new project for the purpose.

Keeping focused on the issue at hand is key, so naturally I use Mylyn — but wait! There's even a GitHub connector for Mylyn! To my disappointment after installing it, I realized that the connector was missing some key features, such as being able to edit and close tasks from within the Eclipse IDE. Time to get hacking! This is where the power of Git really kicked in.

Within moments I cloned the GitHub Mylyn connector project and got to work. Luckily the code was in great shape before I started, and within a few hours I had implemented a functional Mylyn task editor for GitHub. It's trivial for the connector project committers to pull my contributions back into their project.

Though Git itself is not simple to use, it eliminates many barriers and eases community collaboration. EGit integrates first-class support for Git into Eclipse, overcoming much of the complexity of using Git.

Eclipse's ability to collaborate within its project teams and with its wide user community will step into high gear this year. A continued focus on best practices and adoption of enabling technologies such as Git and EGit will make it happen.

Eclipse 3.6 on OpenJDK on Mac OS X

2010-01-19T11:00:00.000-08:00

Eclipse 3.6 (aka Helios) will run on OpenJDK on Mac when it's released this year. This will free us from Apple's oft-criticized and historically slow release cycle of the Java VM.

With a little persistence and help from some committers at Eclipse, the most glaring problems were resolved:

bug 276564:Eclipse won't launch using JDK 7 because SWT native libraries don't load
bug 276763: [launcher] launcher fails to use OpenJDK-based VM when -vm is specified

If you're interested in trying it out, you'll need to get an integration build of Eclipse (see 3.6 Stream Integration Builds) I20100119-0800 or newer, or a nighly build N20100117-2000 or newer. Prior to these builds, running Ecilpse on OpenJDK on a mac was only possible with some trickery. You'll also need a working OpenJDK on your mac, which can be built from source using these instructions, or installed using mac ports.

Many thanks to the Eclipse team for helping, notably Andrew Niefer, Mike Wilson, Kevin Barnes, Grant Gayed, Felipe Heidrich, Thomas Watson, Steve Northover and Silenio Quarti.

Continuous Integration at Eclipse.org

2009-12-31T16:30:00.000-08:00

As we near the end of the year it's a good time to look back at how we're doing things and to look for opportunities for improvement. Last year Eclipse.org has taken some significant steps forward. One notable change for the better is the increasing adoption of running builds on a central build server (in this case, Hudson). While red balls look great on the Christmas tree, they can really detract from the benefits of continuous integration.

At my work we've developed a culture that focuses on maintaining a fast, clean build. It is light-hearted and fun: those that break the build are quick to respond and restore integrity. This behavior is encouraged by good-humored ribbing by other team members. This culture takes a small amount of effort to maintain, however the team recognizes that everyone benefits:

Integration problems are detected and fixed continuously
Early warning of broken/incompatible code
Early warning of conflicting changes
Immediate unit testing of all changes
Constant availability of a "current" build for testing, demo, or release purposes
Immediate feedback to developers on the quality, functionality, or system-wide impact of code they are writing^[1]

At Eclipse.org we can do the same thing: let's strive to eliminate broken builds entirely. Builds should (almost) always be blue; when they're broken, those that caused the breakage should either roll back their changes or fix them.

Here's to a great 2010 at Eclipse.org, where continuous integration meets with continuous improvement!

Hudson Helper for Android Update

2009-12-12T23:20:00.000-08:00

Hudson Helper for Android has been updated. New in this version:

Support for untrusted SSL Certificates (Self-signed, or signed by an untrusted certificate authority)
Layout improvements, bug fixes

Thanks to Jim Hague who assisted with providing an environment for testing untrusted SSL certificates.

To get Hudson Helper for Android, search for 'Hudson Helper' in the Google Market on your Android device, or install via SlideME if the Google Market doesn't support paid applications in your region.

Ant Build Analysis Gets A Facelift

2009-12-09T22:31:00.000-08:00

Wondering why your Ant build is slow? Are your continuous integration builds taking too long? Analysis of Ant builds has had a major facelift, with a new version of Ant Utility now fully integrated into the Eclipse UI. With Ant Utility, you can run your build and get immediate feedback:

As your build runs, Ant Utility measures how much time is spent in Ant tasks and targets. When the build is complete, build metrics are presented in an Eclipse editor, showing exactly which parts of your build file are taking the most time. Double-click on an entry to open the build file in the Ant editor, with problematic lines presented in varying degrees of red depending on the level to which the task or target contributes to the overall build time.

Ant Utility is not designed to replace a full-fledged profiler, however it can give you fast insight into build file problems in the language of your build script. Ant Utility has almost no overhead and is simple to configure.

Ant Utility can be installed into Eclipse via the following update site:
https://antutility.dev.java.net/site/

To enable Ant Utility on your build, add
-listener net.java.dev.antutility.core.MetricsBuildListener to the Arguments section of your Ant launch configuration:

The Ant build must also be configured to run in the same VM as the Eclipse IDE:

At this point you can run your build. When your build completes successfully, build metrics will appear in Eclipse. Hopefully you're off to the races, returning to snappy, healthy builds in no time flat!

How To Be A Successful Developer

2009-12-07T12:49:00.000-08:00

I was recently asked for advice from a young student on how to become a successful software developer. This is a complicated question. I put some thought into it, and realized that every individual will become successful in different ways. Here are some things that helped me:

Always strive to improve yourself and learn more.
Share information freely with others — be generous.
Focus on developing good working relationships with your coworkers, both technical staff and others.
Effective communication, both written and spoken is crucial.
Get involved in open source.
Be precise.
Deliver on commitments, or if you need to renegotiate your commitments.
In everything that you do, do it with integrity.

Almost none of these have anything to do with knowledge of technology. I believe that social aspects have far more impact on success than anything else. Of course being knowledgeable helps too, however what's more important than knowing a specific technology is being able to pick up the knowledge that you need, when you need it.

A few things that I missed in my response because I take them for granted:

Have passion for what you do.
Strive for excellence.
Avoid being self-righteous.

I'm sure that there are many things that contribute to being successful. I'd love to hear from others: what do you think are key contributing factors to becoming a successful software developer?

iPhone Bluetooth Tethering Not Working?

2009-12-02T09:01:00.000-08:00

I love my iPhone and I love it's Bluetooth tethering capability. I use it every day for at least two hours. Much to my dismay, one day it stopped working without explanation. On my mac the Bluetooth 'Connect to Network' menu item was disabled, and for the life of me I couldn't figure out what was wrong — no error messages, no log entries, no clues to speak of. Yesterday I stumbled upon the problem by accident. What follows is some basic instructions that helped me to get it working again. These instructions are only applicable to software developers provisioning apps to their iPhone via XCode.

The underlying problem turned out to be some iPhone provisioning profiles that had expired. If you've had expiring provisioning profiles on your device before, then you know that the iPhone will warn you many times as the expiry date approaches. In the Settings app of your iPhone, tap through General -> Profiles, select any expired profiles and press the Remove button. Once the rogue profiles are removed from your phone, start up XCode on your mac and from the menu select Window -> Organizer. From the Organizer, select Provisioning Profiles under IPHONE DEVELOPMENT.

Delete any expired provisioning profiles using the context menu. After completing this step, exit XCode and you're done! Your Connect to Network menu item should now be enabled, and tethering should now be restored to its normal greatness.

Encoding And Other Character Woes

2009-12-01T08:55:00.000-08:00

Ever have a troublesome file containing unprintable characters or other character encoding problems? Ever wonder if your line delimiter (EOL) is correct for your platform? Over the years I've seen many incarnations of such problems, hidden deeply within XML and other text files. Due to the nature of such characters, being unprintable and sometimes occupying 0 space, these problems can be hard to find and diagnose -- until now. I've created a tiny utility for Eclipse that provides a way to see the invisible: Text Inspection is a new project specifically designed to aid in looking at text in Eclipse.

Text Inspection does this by providing a view that displays the current workbench text selection, exposing unprintable characters as a Java String literal using unicode escape sequences where appropriate. All you have to do to see the text of interest is open your file in a text editor (any text editor will do) and select the area of interest.

The project also augments the Java source editor with a paste command alternative for pasting text as a Java string literal, escaping characters as needed. This can come in really handy when writing unit tests. My unit tests often compare output to an expected value, which is often copy/pasted from the console.

I invite you to try it out and provide feedback.

Eclipse DemoCamp Vancouver 2009

2009-11-26T23:51:00.000-08:00

I had a great time at Eclipse DemoCamp Vancouver 2009. There were some really interesting talks, and the pace was good with 9 minutes per presentation. In particular I was really impressed by a presentation by Emerson Murphy-Hill titled Identifying Code Smells in Eclipse. Emerson demo'd a tool which augments the Java source editor with ambient code smell decorations. Neat stuff.

Anyone that is interested can find my slides on Crowdsourcing Documentation here. In a nutshell I presented on how Mylyn WikiText can enable the use of a wiki (such as EclipsePedia) for creating and maintaining documentation while still publishing to other formats, such as the Eclipse help system. This idea is evolving within discussions by the Eclipse Architecture Council on bug 283734: Documentation Best Practices.

NSLog Revisited

2009-11-13T09:10:00.000-08:00

Logging is an essential tool when developing for iPhone. Keeping things simple and code readable is a major challenge when scattering logging code throughout your application. A few recent posts on the topic have provided some great solutions, but if you're like me sometimes a dead-simple solution is exactly what you need.

This is what I use:

// in Implementation.m
#define DLog NSLog

...

// I need to log something
DLog(@"something interesting happended: %@",what)

When I want to disable logging in that file, I simply change the #define as follows:

#define DLog // NSLog

That's all!

TreeViewer: Two Clicks To Edit

2009-11-05T09:06:00.000-08:00

TreeViewer is the foundation behind most trees and tree tables found in Eclipse. It provides excellent extensibility and options for customization. One feature, in-place editing of tree nodes, is very useful -- however the default implementation activates editing on the first click. This can be frustrating and unintuitive to users who expect the first click to select a node, and the second click to edit. This tip shows a simple way to cause TreeViewer to require two clicks to edit.

Since Eclipse 3.3, ColumnViewerEditorActivationStrategy has been available as API to configure how a TreeViewer starts editing. Our strategy will require the tree's current selection to be the node being clicked in order for a click to be considered an editing event. We do this by extending ColumnViewerEditorActivationStrategy as follows:


/**
 * an activation strategy that only allows editor activation if the item under the
 * event was already selected. This has the effect of requiring a click to first select
 * the item, then a second click to edit. 
 */
class SecondClickColumnViewerEditorActivationStrategy extends
  ColumnViewerEditorActivationStrategy implements ISelectionChangedListener {

 private Object selectedElement;

 public SecondClickColumnViewerEditorActivationStrategy(ColumnViewer viewer) {
  super(viewer);
  viewer.addSelectionChangedListener(this);
 }
 
 @Override
 protected boolean isEditorActivationEvent(
   ColumnViewerEditorActivationEvent event) {
  IStructuredSelection selection = (IStructuredSelection)getViewer().getSelection();
  
  return selection.size() == 1 && super.isEditorActivationEvent(event) && 
    selectedElement == selection.getFirstElement();
 }

 public void selectionChanged(SelectionChangedEvent event) {
  IStructuredSelection ss = (IStructuredSelection) event.getSelection();
  
  if (ss.size() == 1) {
   selectedElement = ss.getFirstElement();
   return;
  }
 
  selectedElement = null;
 }

}

This code relies on the order of events when comparing the tree's current selection against it's previous current selection.

Now all we have to do is install the strategy on our TreeViewer:


viewer = new org.eclipse.jface.viewers.TreeViewer(parent, 
   SWT.MULTI | SWT.H_SCROLL | SWT.V_SCROLL | SWT.FULL_SELECTION);
TreeViewerEditor.create(viewer,null,new SecondClickColumnViewerEditorActivationStrategy(viewer),
   ColumnViewerEditor.DEFAULT);

Voila! We're done: our TreeViewer now edits on the second click instead of the first.

Movember!

2009-11-02T09:05:00.001-08:00

Normally my wife would pay me to shave it off... now I've found the best excuse ever to grow a Mo! I'm growing a Mo to raise money and awareness for Prostate Cancer Canada. Now to decide.... what'll it be?

Please donate, or better yet sign up and start raising too! Want to find out more? Check out the official Movember site.

Key Bindings in Eclipse Editors

2009-10-27T12:23:00.000-07:00

For tool writers, the Eclipse platform provides great facilities for binding keys to commands so that specific actions can be invoked with the right key combination. In most cases hooking into these facilities is a simple matter of activating the right context via IContextService and using a few extension points in your plugin.xml. For more complex editors, such as those with multiple Text or StyledText controls, it's not so easy. This article examines the mechanisms that the Eclipse platform uses to determine how commands are handled, and outlines an approach for implementing editors that wish to alter command handling depending on the focus control.

Suppose we are building an editor that has more than one StyledText control and uses IOperationHistory to support undo/redo. We want CTRL+Z and CTRL+Y (Command-Z and Command-Y on a Mac) to cause undo/redo in the editor. In this editor what we want is for these keys to manipulate the IOperationHistory, however when a StyledText has focus in our editor we want to invoke undo/redo on the StyledText. In other words, the user will expect undo/redo to behave differently depending on what they're doing. This is consistent with how undo/redo work throughout the rest of the platform.

Eclipse uses the following abstractions to represent the concept of keys invoking commands:

KeyBinding - binds a sequence of keys to a logical command
Command - an abstract representation for some semantic behaviour
IHandler - an implementation of the behaviour for a specific command

Let's take (for example) CTRL+Z. When key presses are made, they are matched with a KeyBinding. Eclipse knows which Command to invoke based on the KeyBinding (in this case the 'undo' command). Eclipse then determines which IHandler to use by looking up the handler using the logical command in IHandlerService. (There are many layers of indirection in the actual implementation -- this example is simplified for ease of understanding).

In our editor we want the operation history to be associated with undo/redo. We hook it up as follows:


OperationHistoryActionHandler undoAction= new UndoActionHandler(site, moduleEditor.getOperationContext());
PlatformUI.getWorkbench().getHelpSystem().setHelp(undoAction, IAbstractTextEditorHelpContextIds.UNDO_ACTION);
undoAction.setActionDefinitionId(IWorkbenchActionDefinitionIds.UNDO);
undoAction.setId(ITextEditorActionConstants.UNDO);

IHandlerService handlerService = (IHandlerService) site.getService(IHandlerService.class);
handlerService.activateHandler(undoAction.getActionDefinitionId(), new ActionHandler(undoAction));

When we test our editor this works as expected: we can undo changes in the editor by pressing CTRL+Z. After focusing on a StyledText in our editor however, we expect CTRL+Z to undo text that we type. In our editor these text changes aren't in our operation history until after the text control loses focus. So how do we set that up? Here's how:


TextViewer textViewer = // create the text viewer
TextViewerSupport support = new TextViewerSupport(textViewer);

Most of the work is done by this helper class:



 protected class TextViewerSupport implements FocusListener, DisposeListener {

  private final TextViewer textViewer;
  private List handlerActivations = new ArrayList();
  

  public TextViewerSupport(TextViewer textViewer) {
   this.textViewer = textViewer;
   StyledText textWidget = textViewer.getTextWidget();
   textWidget.addFocusListener(this);
   textWidget.addDisposeListener(this);

   if (textViewer.getTextWidget().isFocusControl()) {
    activateContext();
   }
  }
  public void focusLost(FocusEvent e) {
   deactivateContext();
  }
  public void focusGained(FocusEvent e) {
   activateContext();
  }
  public void widgetDisposed(DisposeEvent e) {
   deactivateContext();
  }

  protected void activateContext() {
   if (handlerActivations.isEmpty()) {
    activateHandler(ISourceViewer.QUICK_ASSIST,ITextEditorActionDefinitionIds.QUICK_ASSIST);
    activateHandler(ISourceViewer.CONTENTASSIST_PROPOSALS,ITextEditorActionDefinitionIds.CONTENT_ASSIST_PROPOSALS);
    activateHandler(ITextOperationTarget.CUT, ITextEditorActionDefinitionIds.CUT);
    activateHandler(ITextOperationTarget.COPY, ITextEditorActionDefinitionIds.COPY);
    activateHandler(ITextOperationTarget.PASTE, ITextEditorActionDefinitionIds.PASTE);
    activateHandler(ITextOperationTarget.DELETE, ITextEditorActionDefinitionIds.DELETE);
    activateHandler(ITextOperationTarget.UNDO, ITextEditorActionDefinitionIds.UNDO);
    activateHandler(ITextOperationTarget.REDO, ITextEditorActionDefinitionIds.REDO);
   }
  }
  

  protected void activateHandler(int operation, String actionDefinitionId) {
   StyledText textWidget = textViewer.getTextWidget();
   IHandler actionHandler = createActionHandler(operation, actionDefinitionId);
   IHandlerActivation handlerActivation = handlerService.activateHandler(actionDefinitionId, actionHandler,new ActiveFocusControlExpression(textWidget));
   
   handlerActivations.add(handlerActivation);
  }

  private IHandler createActionHandler(final int operation, String actionDefinitionId) {
   Action action = new Action() {
    @Override
    public void run() {
     if (textViewer.canDoOperation(operation)) {
      textViewer.doOperation(operation);
     }
    }
   };
   action.setActionDefinitionId(actionDefinitionId);
   return new ActionHandler(action);
  }
  
  protected void deactivateContext() {
   if (!handlerActivations.isEmpty()) {
    for (IHandlerActivation activation: handlerActivations) {
     handlerService.deactivateHandler(activation);
     activation.getHandler().dispose();
    }
    handlerActivations.clear();
   }
  }  
 }

Though TextViewerSupport looks complicated, it's really just activating different command handlers whenever the textViewer gains focus.

When the text control has focus the editor now has two handlers for the undo command. So how does Eclipse know which one to use? Digging into the internals of Eclipse reveals that it uses the handler with highest priority. But we don't set a priority no these handlers! Eclipse calculates the priority of handlers automatically based on the enablement expression of the handler. The priority of the enablement expression is based in part on the variables that the expression references. This leads us to the last part of our solution, ActiveFocusControlExpression:


/**
 * An expression that evaluates to true if and only if the current focus control is the one provided.
 * Has a very high priority in order to ensure proper conflict resolution.
 */
public class ActiveFocusControlExpression extends Expression {

 private Control focusControl;

 public ActiveFocusControlExpression(Control control) {
  focusControl = control;
 }

 @Override
 public void collectExpressionInfo(ExpressionInfo info) {
  info.markDefaultVariableAccessed(); // give it a very high priority
  info.addVariableNameAccess(ISources.ACTIVE_SHELL_NAME);
  info.addVariableNameAccess(ISources.ACTIVE_WORKBENCH_WINDOW_NAME);
 }

 @Override
 public EvaluationResult evaluate(IEvaluationContext context)
   throws CoreException {
  if (Display.getCurrent() != null && focusControl.isFocusControl()) {
   return EvaluationResult.TRUE;
  }
  return EvaluationResult.FALSE;
 }
}

In collectExpressionInfo we ensure that the expression indicates that it uses the default variable. This gives the expression a very high priority. Since the expression is only enabled when our control has focus, we've ensured that undo is directed to our text control at the right time.

Credit: The approach described in this article is inspired by a class CommonTextSupport (authored by Steffen Pingel) from the Mylyn project.

Hudson Helper for iPhone Update

2009-09-03T11:49:00.000-07:00

Hudson Helper for iPhone and iPod Touch has been updated.

New in this version:

Build controls (start/stop builds)
Build details with blame!
Build progress indicator
More compact layout
Trend graphs now display when using Hudson CI servers version 1.321 and greater

You can get Hudson Helper by searching for 'Hudson Helper' in the app store, or click here to get it with iTunes.

If you're an Eclipse contributor my original offer still stands: you can get Hudson Helper for free.

How I Lost My Annotations: JPA, Hibernate and FetchType.LAZY

2009-09-03T08:00:00.000-07:00

Many JPA applications use Java annotations on their domain model classes in order to specify validation rules. Runtime frameworks use reflection to discover these validation rules and enforce them at runtime. This all works nicely until you realize that you need FetchType.LAZY on some of your @ManyToOne associations. After specifying lazy fetching you may find that some of your annotations disappear – only not all the time. So what's going on here? This article discusses the implications of FetchType.LAZY and how to overcome this intermittent problem.

In order to implement lazy fetching under the covers Hibernate enhances your classes. What does this mean? Basically, Hibernate extends your classes and overrides its methods at runtime. It does this by using one of two bytecode manipulation libraries: Javassist or CGLib. If you ask one of these enhanced classes for its name, you'll end up with something like this:


// JPA entity class name
greensopinion.Person

// CGLib enhanced class name
greensopinion.Person$$EnhancerByCGLIB$$4ad0592d

// Javassist enhanced class name
greensopinion.Person_$$_javassist_0

Hibernate only does this when necessary. So when you get an instance of your entity it might be enhanced, and it might not – depending on how it was loaded. This can be confusing, since instances of a Person may not always be the same class. For example, code like this can be problematic:


Person person1 = entityManager.find(Person.class,id1);
Person person2 = entityManager.find(Person.class,id2);

// bad: contrary to what we'd expect, this comparison may
// be true sometimes, but not always
if (person1.getClass() == person2.getClass()) {
}
// bad: this may not work as expected either
if (person1.getClass().isAssignableFrom(person2.getClass())) {
}
// bad: again, sometimes true sometimes false
if (person1.getClass().isAssignableFrom(Person.class)) {
}
// this is ok: comparison will always be true (assuming person1 is not null)
if (person1 instanceof Person) {
}

In the above example comparisons that look straight-forward may behave differently depending on how and if these entities were already loaded by the same entity manager. This can be tricky when implementing hashCode and equals.

Furthermore, we're in for more trouble if we declare Person as follows:


@Entity
public class Person {

  ...

  @Required
  public String getFirstName() { ... }

}

Here we're using an annotation to indicate to our validation framework that the firstName property is a required field (see JSR-303, Hibernate Validation, Spring Modules, OVal). Our validation framework may do something like this:


Class clazz = entity.getClass();

// iterate on class accessors
...

if (method.getAnnotation(Required.class) != null) {
  ...
}

This may work well most of the time, and may work in all of our unit tests – however in the running application it may appear as if our class has lost its annotations! Why is this happening? Don't worry, your JVM isn't losing its head, or your annotations. This code may not work as expected if our entity is an enhanced version since Hibernate's ProxyFactory enhancers don't consider annotations when overriding methods on your entity.

So why use FetchType.LAZY at all? Most non-trivial JPA applications will have to make use of FetchType.LAZY in order to avoid @ManyToOne associations from causing cascading loads. Using FetchType.LAZY can also reduce the number of joins when fetching collections, which in some cases greatly improves performance.

So how to we solve this problem? Our reflection code must be aware of HibernateProxy. Here's an example of how this can be fixed:



Class clazz = PersistenceUtil.getEntityClass(entity);
... now do reflection ...

// in PersistenceUtil.java
public static Class getEntityClass(Object entity) {
   if (entity instanceof HibernateProxy) {
       return ((HibernateProxy)entity).getHibernateLazyInitializer().getPersistentClass();
   }
   return entity.getClass();
}

The intermittent and in some cases occasional nature of the symptoms involved make this problem hard to reproduce. Lazy fetching may seem like a panacea, however it's another case where leaky abstractions make our lives more challenging. As with most things in software development, it's important to understand the implementation details in order to work effectively with JPA.

Hudson Helper for Android Update

2009-09-02T16:37:00.000-07:00

Hudson Helper for Android has been updated. New in this version:

background status monitoring
trend graphs now display when using Hudson CI servers version 1.321 and greater
bug fixes

To get Hudson Helper for Android, search for 'Hudson Helper' in the Google Market on your Android device.

Regular Expression Gotchas

2009-08-14T14:03:00.000-07:00

Mylyn WikiText uses regular expressions to parse wiki markup. For Mylyn users, this means that your Bugzilla, JIRA and Trac bugs look good in the Eclipse UI. With the Galileo release of Eclipse, Mylyn WikiText added automatic stack trace detection.
Little did I know that the regular expression that I wrote would hang the Eclipse UI.

The intention was to format stack traces using a monospace font to improve layout:

Stack traces are detected by a regular expression. The regex that I created for this purpose worked well in testing, however for some stack traces would pin the CPU at near 100%, hanging the Eclipse UI. Here's what the regular expression looked like:

([A-Za-z][a-z0-9_$]*)+

Actually, the full regular expression is a lot bigger:

\s*((((Caused by:\s+)|(at\s+))?([a-z][a-z0-9]*)(\.([a-z][a-z0-9]*))*\.([A-Za-z][a-z0-9_$]*)+((:\s+\w.*)|(\.((\)|([a-zA-Z0-9_$]+))\(.*?\)))?)|(\.\.\.\s\d+\smore))

The regex segment attempts to match a camel-case Java class name. What I didn't notice the first time around is that it contains nested greedy quantifiers. For some input can cause the CPU spiral of death... not good news.

To fix the offending regular expression is simple: just eliminate any nested quantifiers. My improved regular expression segment now looks like this:

[A-Za-z][a-zA-Z0-9_$]*

Two things that I learned from this experience:

Never write a regular expression that nests greedy quantifiers.
Always ensure that testing involves sufficient diversity of input data.

Android versus iPhone Development: A Comparison

2009-07-02T23:01:00.001-07:00

A few months ago I ventured into the world of Mobile development and created an application (Hudson Helper) for both iPhone and Android. This article is about my experiences, comparing Android and iPhone development with a focus on tools, platform and the developer experience.

Before going much further I should note that my comparison is with considerable bias. I’ve spent the past 12+ years in Java development, having spent much of my career building developer tools. Since January of 2004 I’ve been building plug-ins for Eclipse, and before that plug-ins for NetBeans. This bias is somewhat tempered with several years of C and C++ development. With this background I find that I’m very critical of developer tools. Developer productivity is key — anything that takes away from the flow of a developer in the zone is a real problem.

Language, Programming Model and Platform

Language

The language of choice for iPhone development is Objective-C. Objective-C is a language based on C with extensions for object-oriented concepts, such as classes, inheritance, interfaces, messages, dynamic typing, etc. The Java language is used when developing for Android (though it doesn’t actually get compiled to bytecode).

Java is a no-brainer. I have to say that it’s nice not to have to learn a new language to target mobile. Existing skillsets don’t come easy — so reuse of expertise is worth a lot.

It took a little while to wrap my head around some of the language features available with Objective-C. I soon discovered that I really loved certain language features, such as message passing (instead of calling methods), categories and named arguments. I did find however that the syntax of Objective-C is cumbersome. I’m still not used to ‘+’ and ‘-’ for static and member methods, too many parentheses are required, and in general I just felt like I had to type way to much to express a simple concept. The IDE didn’t help much with this either (more on that later).

One thing that really became clear to me is that Objective-C, though it may have been visionary for its time, is really a language of the '80s. Certain issues such as split header and implementation files and violation of DRY are really time-wasters, and not small ones at that. I found myself constantly switching back and forth between files, which not only has a cost in navigation (which file to open?) but with every file opened your sense of context must be recreated (where’s the caret, what’s selected, where am I in the file, how is this file organized).

As far as DRY, must I really do 5 things to declare a property?? (declare in the class definition, again to declare getter/settter, initialize in the init method, @synthesize in the implementation, release in dealloc). Here’s what I mean:

Server.h


@interface Server : Updatable {
NSString *name;  <-- declare the property 
}

@property (nonatomic,retain) NSString *name; <--- declare the property again

Server.m


@synthesize name;  <-- implement getter/setter

-(void) dealloc {
[name release];  <-- release memory
}

If you ask me, everything in Server.m should go away. Another gotcha here is the positional relevance of @synthesize.

Java has a similar problem with properties, though not quite so bad — and the IDE helps you write your getter/setter.

Pointers in Objective-C, though powerful, are also another time-waster. This is where Java really shines with its garbage collection. I found that I was constantly considering whether allocated objects were freed appropriately. Code flow is poor since application logic is littered with memory management. I only have so many brain cycles available — why do I have to think about this other cruft that’s not really a core concern of the application domain? Of course this gets even worse when trying to figure out where things went wrong if you make a mistake. Zombies help, but still don’t make it obvious if you’ve accessed something that was deallocated. Other issues include deallocating something twice, autoreleasing something twice. I also found it non-intuitive when to retain return values from methods.

Another annoyance of Objective-C is the patterns that must be followed: implementing correct init and dealloc methods is non-trivial. @synthesized getters and setters for properties with retain should not be called in these methods. So many conventions and rules to remember!

Though I understand why there’s a separation of alloc and init, it’s still overly wordy to specify [[aloc Foo] initWithArg: arg]. Why not just [new Foo arg]? Or how about new Foo(arg) -- oh, wait, that’s just like Java!

Objective-C’s imports and forward-declarations (@class) are a pain. Though these issues exist with Java development, Eclipse’s JDT is so good that I’ve almost forgotten what it’s like to write an import. All you have to do is Ctrl+Space to auto-complete a class name or Ctrl+Shift+O to organize imports and voila!

Of course Java is not perfect either, however this fact is hidden from me due to the fact that I’ve been living in Java for a very long time. Sometimes I wish that Java were more Groovy-like, however I’m used to it and the tooling is so good.

Platform

On Android I found that I could readily use the Java runtime classes. Some, but not all, of the standard Java RT classes are available on Android. I didn’t find this a problem, since most of the standard Java IO, network and regex libraries are available. Android RT classes appear to be based on Harmony, which has been around long enough to be stable.

With iPhone on the other hand, finding the functionality that I needed was painful. Classes and methods are poorly organized. When to look for a static method versus a class with members was not clear to me. Also depending on the framework used, naming conventions and code organization would differ. I suppose this is the legacy of an older platform. Areas where functionality was lacking that I found painful were regular expressions, string handling and XML parsing. I ended up using the excellent Regex Kit Lite for regular expressions. For XML parsing I implemented a parser abstraction over libxml, only to discover later that I may have had an easier time with NSXMLParser which is a lot more like SAX.

On the iPhone when things didn’t work as expected I had to resort to Google and hope that others had encountered the same problem. This technique was hampered by Apple’s earlier NDA policy, which meant that iPhone content is pretty thin on the net. In some cases I would resort to guesswork and experimentation to find a solution.

Android has the benefit of being open source. Within minutes I had the full Android platform source code on my system, and had re-built the SDK from sources to ensure that the source I had matched the runtime classes in the emulator. So not only could I see how things were implemented in the Android platform and learn by example, I could step through the platform code in the emulator and discover why my code wasn’t producing the desired results.

In general I found the layout, organization, and naming conventions of Android platform classes was consistent and predictable. This made it much easier to learn.

Programming Model

The iPhone platform does a great job of encouraging an MVC design pattern. With this design pattern built in to the platform, building the UI was simple and I didn’t have to figure out how to organize the UI component design myself. It also means that when looking at sample code, it’s all organized in the same way.

Android also does a good job with design patterns, though their concepts varied significantly from the iPhone. With Android’s support for multiple processes and component reuse, the platform itself provides support for Intents and Activities (an Intent is just a variant of a command). The design results in a better user experience, however it does introduce some complexity for the developer: when starting one Activity from another, an Intent is used to communicate any parameters. These parameters cannot be passed by reference — only by value. Where on the iPhone it’s simple to have screens sharing the same data structures, on Android this requires some forethought. Apparently Android applications can manage the back button and have everything occur inside a single Activity, however this is not the norm.

Both Android and iPhone provide a way of declaring user preferences in XML. Both platforms provide a default UI for editing those preferences, which is great. Android’s XML format is extensible allowing custom UI components to be integrated, which makes user preferences a breeze. iPhone developers that wish to customize preferences will have to implement a UI from scratch, which is a lot more work.

Testing and Continuous Integration

I’m of the opinion that every development effort should include unit tests. Teams of size greater than one should also include Continuous Integration.

Android developers will be happy to know that they can write JUnit tests. I could even launch these from the Eclipse UI after some classpath fiddling. Though I didn’t try it, I assume that it’s trivial to run these from Ant and your favorite CI server such as Hudson.

I did see some iPhone unit test documentation with the iPhone SDK but didn’t take the time to explore it — so I can’t comment there.

Resources

Apple does an excellent job of providing lots of resources for developers. Important concepts are explained in videos, which makes grasping concepts easy — however I did find that videos progressed slowly and I was watching for what seemed like hours to find information that should have taken minutes. Luckily Apple also provides lots of sample applications and code to demonstrate API usage.

Android developers also have access to loads of resources. The guide and API reference are installed with the SDK, so everything is available when offline (which for me is important since I do a lot of my work in transit). I found the Android development resources better organized and spent less time looking and more time finding. In particular the ApiDemos sample app provides a great starting point. I also downloaded many open source Android projects for ideas on architecture and API usage. This is an area where Android has the advantage, with Apple’s previous NDA policy there isn’t much out there in terms of open source for iPhone.

Tooling

For me tooling was a real shocker. These are the categories of tooling that I’ll cover: IDE, UI builder, debugger, profiler. Almost everything else is related to provisioning, and in that area I didn’t notice much in the way of differences between Android and iPhone.

IDE

Android development leverages the excellent JDT tools, which are pretty much stock and standard with every Eclipse installation. I’ve used these tools now for many years and they’re excellent. Everything Java is indexed, the IDE has a rich model of the source code, and refactoring is so seamless that it has changed the way that I work.

Perhaps the best feature of JDT is its incremental compiler, which provides immediate feedback with errors and warnings as you type. This eliminates the code-compile-wait-for-feedback cycle that was so common in the '80s and '90s. Errors and warnings are updated in the Java editor as I type, giving me instant feedback. I didn’t realize just how valuable this feature is until I was coding Objective-C in XCode — when I became acutely aware at how waiting for compiler feedback can break the flow of programming.

Other key features that make Eclipse so amazing to work with are:

content assist
quick-fixes
organize imports
open type (CTRL+Shift+T)
refactorings

Integrated javadoc and content assist is quite possibly the best way to learn an unfamiliar API. In Ecipse not only are all classes and methods immediately available in the context in which you’re writing code, their documentation is presented alongside.

Content Assist with Integrated Javadoc

XCode is so shockingly bad that I almost don’t know where to start. Here’s a minimum list of things that I think need fixing in order for XCode to become a viable IDE:

Content assist that actually works. Content assist provided by XCode is often wrong, and almost always suggests a small subset of what’s actually available.
A decent window/editor management system. XCode and it’s associated tools (debugger) like to open lots of windows. Want to open a file? How about a new window for you! Very quickly I found myself in open-window-hell. The operating system’s window management is designed for managing multiple applications, not multiple editors within an IDE. It’s simply not capable of providing management of editors in an environment as sophisticated as an IDE.
A project tree view that sorts files alphabetically. Really!
Integrated API documentation. I found that I was constantly switching out of the IDE and searching for API documentation using Appkido. This may seem trivial, but it really breaks the flow.

One area of Eclipse that simply can’t be matched is Mylyn. Integrated task management and a focused interface introduce huge efficiencies into any project, small or large. If you haven’t yet tried out Mylyn, it’s definitely worth your time to take a look. A good place to start is Mylyn’s Getting Started page.

UI Builder

iPhone app developers are given a pretty good UI builder. It does a great job of showing the UI as it will actually appear. It’s flexible and can model some pretty sophisticated UIs, so I was impressed. I found that using it was a little tricky — I had to read the documentation two or three times before I could really figure out how to use it properly.

The Android UI builder I found pretty useless: it can’t display UIs how they’ll actually appear, and it’s UI is way too inefficient. I found that I coded all of the UIs directly in the XML source view of the UI builder. There the content assist and validation were pretty good, making it the easiest way for me to build a UI.

Debugger

Having used to the Java debugger in Eclipse I was shocked at the state of the debugger in XCode. With Eclipse I can see and modify variable values. Not so in XCode. Maybe this is simply the state of affairs when debugging native code, but it sure affects the usefulness of the debugger. XCode often seemed confused as to the type of an object and presented me with a pointer value and no detail. This is a sharp contrast to Eclipse, where I can drill down through an object graph with ease.

I found the XCode debugger UI extremely difficult to use. Clicking on the stack to show code in an editor caused new windows to open, eventually resulting in dozens of windows open. In addition I found that watch expressions rarely worked for me.

Profiler and Heap Analysis

An area where Apple development tools excel is in profiling and heap analysis. These tools seemed mature and easy to use. With no prior experience with these specific tools I was able to gain a better understanding of my app within minutes, find and fix several memory leaks and improve performance.

XCode Memory Leak Detection

Android developers must use Android’s traceview application, which I found worked well but required significantly more effort to configure and operate. I was surprised to find that the source code must be changed in order to get the trace files required for analysis.

I’m not sure if Android can provide heap dumps in hprof format. If it can then the awesome MAT tool could be used to analyze heap usage. According to this article Android can produce hprof heap data, though I haven’t tried it.

App Store

It goes without saying that the iPhone app store is excellent in that you can sell into many countries worldwide with a single setup. I was able to provide my Canadian bank account number, sign a few legal agreements and I was up and running.

Getting an app into the store however is frustrating to say the least. Apple must approve every app before it is accepted into the store. Mine got rejected multiple times. Each time it was rejected I was given almost no information about why. When I emailed them to clarify the problem, I received what looked like a canned response indicating that I should refer to previous correspondence. If it weren’t so frustrating I would have found it funny. I highly recommend reading Brian Stormont’s Avoiding iPhone App Rejection from Apple and Dan Grigsby’s Part 2 follow-up.

Of course once I started selling Hudson Helper I realized that Apple won’t send me any money unless the payout is greater than $250. This is true not only of the first payout, but every payout. Google market on the other hand requires a minimum of $1 for each payout. Both the iPhone app store and Google market take about %30 of your app selling price. $0.99 applications have to have high volume, or they’re simply not worth your time.

The Google market by comparison to the Apple app store is terrible in that you can only sell into a handful of countries. You also can’t see or install apps that cost money on a developer phone. Actually you can, but not if the app has copy protection — which is almost every non-free app. On the other hand when you upload your app to the app store it’s available within minutes, so you don’t have to worry about an approval process.

To set up a merchant account with Google market, I had to provide a US address and bank account number, since Google doesn’t support Canada. For me this was a pain, but not too bad since I live within a few kilometers of the US border. I rode my bike down to the US and opened an account with Horizon bank. The bank required a passport and driver’s license, so no problem there. Why Google doesn’t support more countries I don’t know. At the very least Google market should accept alternate payment methods for countries that are not supported by Google checkout.

Summary

Android’s platform and developer tools are excellent. Leveraging Java and the Eclipse IDE are major winning factors for Android. Apple’s developer tools are shockingly bad by comparison. The Objective-C language and platform APIs are cumbersome and poorly organized. Overall when developing for the iPhone I felt like I was back in 1993. These factors combined in my estimation make application development about three times more expensive when developing for iPhone. The only area where Apple’s developer tools excelled was in profiling and heap analysis.

Apple’s app store from a user’s standpoint and from a worldwide coverage standpoint are excellent. In this area Google market for Android is weak.

Development for iPhone may improve as tools such as iphonical (MDD for iPhone) and objectiveclipse (Eclipse plug-in for Objective-C) emerge.

We may see a shake-up in the mobile market, with at least 18 new Android handsets being released this year. Until that happens, iPhone will remain a market leader and developers will have to put up with XCode and Objective-C.

For me, my love is with Android. Sure, the iPhone is great — but can you install a new Linux kernel?

Hudson Helper for Android

2009-06-29T08:00:00.000-07:00

Recently I launched Hudson Helper for iPhone and iPod Touch, enabling Continuous Integration fans to stay in touch with their projects. Android users can get in on the game now too, with Hudson Helper for Android.

Hudson Helper for Android provides all of the same features as the iPhone version including support for multiple servers and authentication. New for this version are build controls: start and stop builds right from your phone. CI can be even more fun with shake-to-build and sound effects.

To get Hudson Helper for Android, search for 'Hudson Helper' in the Google Market on your Android device.

Tip: Twittering About Eclipse Bugs

2009-06-22T09:23:00.000-07:00

A quick tip for including an Eclipse bug URL in your 140-character tweets: Bugzilla for Eclipse supports short links. Instead of using the long-form URL or impossible-to-decipher URL shorteners, take out the 'bugs/show_bug.cgi?id=' part of the Bugzilla URL as follows: https://bugs.eclipse.org/277974

Hudson Helper and Untrusted Certificates on the iPhone

2009-06-21T14:47:00.000-07:00

If you're using a self-signed certificate or a certificate signed by your own certificate authority with the Hudson Continuous Integration Server then you may be wondering how to get your iPhone to trust your certificate. Well, you're in luck because Benjamin Boksa has written an excellent tutorial explaining how to do just that: iPhone OS 3.0 – Root Certificate Authority installation. After following the tutorial you should be able to use Hudson Helper with untrusted certs and your secure Hudson server.

WikiText in Eclipse Galileo

2009-06-21T14:34:00.000-07:00

Ian Bull of EclipseSource has provided an excellent summary of Mylyn WikiText and what it means for Mylyn users and Galileo. WikiText is number 4 on Ian's Eclipse Galileo Feature Top 10 List. Mylyn WikiText is part of the highly anticipated Eclipse Galileo release due out this Wednesday. You can find out more about WikiText from the Mylyn homepage, the FAQ, and the downloads page.

Galileo Builds On Your iPhone

2009-06-17T08:28:00.001-07:00

With the Eclipse Galileo release exactly a week away, I'm presenting a quick tutorial on how you can monitor the Galileo hudson build on your iPhone with Hudson Helper. I'm also giving Hudson Helper away for free to the first 20 Eclipse contributors^* to send me an email.

Here are the steps to take to get Hudson Helper set up with the Galileo build:

If you haven't done so already, install Hudson Helper. You can get it by searching for 'Hudson Helper' in the app store, or via iTunes here.

Next, go to the Settings application on your iPhone or iPod touch. Under Hudson Helper, Server 1 URL enter the following: https://build.eclipse.org/hudson/view/Galileo
Notice that it's https, not http.

You may have noticed from the screenshot that a new version of Hudson Helper is now available with support for HTTP authentication.

That's it! Close the Settings app, launch Hudson Helper and you should see the following:

Even though Hudson Helper has blue balls, it can still help to keep you up to date with your builds!

^*If you're an Eclipse committer you qualify. If you're not an Eclipse committer but have contributed at least 2 patches to Eclipse, you qualify. If you were a committer you qualify. If you aren't a committer, but are a Friend of Eclipse you qualify.

Hudson Helper: Hudson On Your iPhone

2009-05-28T13:00:00.000-07:00

I'm a big fan of Continuous Integration and automated builds. So much so that today I released Hudson Helper, a native iPhone and iPod Touch application for monitoring your Hudson continuous integration engine.

Hudson Helper has a simple interface that gives you what you need to know first. Offline data caching helps to minimize network bandwidth and makes it work when the network is unavailable. Includes job status, health and build trends.

Build status display adapts to your project to report on build health including JUnit test results, Clover coverage, findbugs and any other Hudson build report plug-ins your project is running.

To get started all you need to do is point Hudson Helper at the URL of your build server. You can get Hudson Helper by searching for 'Hudson Helper' in the app store, or click here to get it with iTunes.

Eclipse 3.5 (Galileo) on Java 7

2009-05-08T15:49:00.000-07:00

Some time ago I blogged about running Eclipse (Ganymede) on SoyLatte. Since then there has been little improvement in the state of Java on the Mac. Apple has released a Java 6 for Mac, but it's a 64-bit only VM which crashes regularly when running Eclipse. Having heard about recent improvements in performance for 64-bit VMs using pointer compression (PDF paper), I decided to have another look at launching Eclipse with an OpenJDK-based VM such as JDK7. After a few false starts I managed to get it working. Here's what I did:

First, to get JDK7 on my machine. I followed these instructions to build JDK 7 from source. The instructions didn't work perfectly for me -- but they were pretty good. Basically followed everything verbatim with the exception that I had to install MacPorts and use it to get wget.

With a 32-bit JDK7 on my machine, now to use it to start Eclipse. I downloaded the latest I-build of 32-bit Eclipse 3.5 from here (Look for '3.5 Stream Integration Builds').

Using a combination of jconsole and guesswork I came up with this shell script to start Eclipse:


JDKPATH=/Users/dgreen/Documents/packages/jdk7-32bit-2009-05-08
DEBUG_OPTS=
# DEBUG_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=y
MEM_OPTS=-Xms1024m\ -Xmx1536m\ -XX:MaxPermSize=256m
ECLIPSE_LOC=/Users/dgreen/Downloads/eclipse
WORKSPACE=/Users/dgreen/Documents/workspace

$JDKPATH/bin/java $DEBUG_OPTS \
-cp $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher_1.0.200.v20090429-1630.jar \
-XstartOnFirstThread $MEM_OPTS \
-Dorg.eclipse.swt.internal.carbon.smallFonts \
-Djava.library.path=$ECLIPSE_LOC/Eclipse.app/Contents/MacOS/lib \
-Dswt.library.path=$ECLIPSE_LOC/Eclipse.app/Contents/MacOS/lib \
org.eclipse.equinox.launcher.Main \
-os macosx -ws cocoa -arch x86 \
-showsplash -launcher $ECLIPSE_LOC/Eclipse.app/Contents/MacOS/eclipse \
-name Eclipse \
--launcher.library $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher.cocoa.macosx_1.0.0.v20090429-1630  \
-startup $ECLIPSE_LOC/plugins/org.eclipse.equinox.launcher_1.0.200.v20090429-1630.jar \
-data $WORKSPACE \
-keyring /Users/dgreen/.eclipse_keyring -consoleLog -showlocation \
-vm $JDKPATH

Note that all of the above should appear on one line: your browser may split it into multiple lines.

After starting Eclipse as above I saw the following stack trace on startup:


java.lang.UnsatisfiedLinkError: no swt-pi-cocoa-3547 or swt-pi-cocoa in swt.library.path, java.library.path or the jar file
 at org.eclipse.swt.internal.Library.loadLibrary(Library.java:248)
 at org.eclipse.swt.internal.Library.loadLibrary(Library.java:159)
       ... snip ...

To solve the problem I extracted all of the *.jnilib files from org.eclipse.equinox.launcher_*.jar and put them in a lib folder. I then renamed all of those *.jnilib files in the lib folder so that they had *.dylib file extensions. I ended up with a file structure as follows:


david-greens-macbook-pro:MacOS dgreen$ pwd
/Users/dgreen/Downloads/eclipse/Eclipse.app/Contents/MacOS    <--- where I put the files
david-greens-macbook-pro:MacOS dgreen$ ls -l *.sh lib
-rwxr-xr-x@ 1 dgreen  staff  1100  8 May 15:47 eclipse.sh    <--- the startup script

lib:        <---- the lib folder for shared libraries
total 2296
-rw-r--r--  1 dgreen  staff   37104  8 May 14:20 libswt-awt-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  287116  8 May 14:20 libswt-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  539240  8 May 14:20 libswt-pi-cocoa-3547.dylib
-rw-r--r--  1 dgreen  staff  300428  8 May 14:20 libswt-xulrunner-cocoa-3547.dylib

Trying the shell script again, Eclipse started up and ran beautifully! Once again I'm running Eclipse with a newer-than-Apple-will-give-me VM, this time JDK 7!! My next step is to compile a 64-bit JDK7 and see if it works too... I have high hopes.

Updated 2009-05-15 see this issue (bug 276564) for the underlying cause of failure
Updated 2010-01-19 see Eclipse 3.6 on OpenJDK on Mac OS X for an update

How Babel Saved My Bacon

2009-05-05T22:30:00.000-07:00

Recently I blogged about the pain of moving to ICU4J, lamenting the fact that using NLS instead of ResourceBundle would require me to change the keys in my resource bundles. After much consideration and the advice of knowledgeable people, I decided that moving to NLS was the way to go. After changing resource bundle keys across the board, I expected that translations to other languages would need their corresponding keys changed as well. Not so. Apparently Babel can magically handle such refactorings, with the help of a mystery user named 'Babel Syncup'. I wonder what else it can do.

The Eclipse Babel project is truly impressive. Kudos to the Babel team for thinking ahead.

Peter Friese on Advanced WikiText

2009-05-05T22:12:00.000-07:00

Peter Friese has upped the stakes in his latest article on Advanced WikiText. Friese tackles tough topics, such as how to optimize team collaboration with multiple source files and integrating with a DocBook toolchain. Great article, a must-read for those who are authoring documentation.

Peter Friese on Writing Documentation with WikiText

2009-04-23T20:22:00.000-07:00

Peter Friese has published a great article titled Getting Started With WikiText. He starts off with some background on various documentation formats, then provides a step-by-step guide to getting started with Mylyn WikiText. Peter also shows how images can be scaled using some simple wiki markup using a new WikiText feature that Peter Friese recently contributed.

Galileo ICU4J Requirement Is Painful

2009-04-14T08:33:00.000-07:00

Projects must use ICU4J if they wish to participate in the Galileo release of Eclipse, due this summer. Though such a requirement may make good sense in theory, in practice adopting ICU4J is painful. Not only does this policy stifle innovation for questionable gain, such a burden makes participating in the Eclipse ecosystem less attractive.

ICU4J is designed to be a drop-in replacement for Java classes that perform poorly or behave incorrectly. The idea is that you change your import statements, recompile and everything works. In applying this technique to Mylyn WikiText, I’ve discovered that it’s not that easy.

Mylyn WikiText makes use of two classes that are to be replaced by ICU4J equivalents: MessageFormat and ResourceBundle.

ResourceBundle

ResourceBundle conversion should be easy, right? ICU4J provides UResourceBundle, which extends ResourceBundle.... wow, that’s great. All I should have to do is replace ResourceBundle.getBundle() with UResourceBundle.getBundleInstance()... easy peasy. Unfortunately, not so easy.

Previously I had used the JDT ‘Externalize Strings’ wizard to externalize strings in my plug-in. The JDT nicely created a utility class Messages.java alongside the resource bundle messages.properties. This worked really well and has been in use for some time. Unfortunately after converting to ICU4J UResourceBundle throws a NoClassDefFoundError because of the presence of the Messages class. It seems that UResourceBundle cannot handle a resource bundle alongside a class by a similar name. While I’ve logged bug 272166 to track this issue and have it fixed, it doesn’t help me. Even if the bug gets fixed in time for Galileo, Mylyn WikiText must work on Eclipse 3.3 and 3.4 as well.

It gets worse. Mylyn WikiText is already translated to German and Japanese. Resource bundle message keys must remain relatively stable in order to avoid losing those translations. JDT creates message keys containing dots ‘.’, and the Eclipse externalized string pattern cannot have dots in message keys... so I cannot convert over to the Eclipse externalized string pattern without losing these translations.

MessageFormat

MessageFormat evolved for Java 5: it’s format() method became a var-args method, meaning that you can pass a variable number of arguments to it. WikiText makes extensive use of this method, so you’ll see code such as this:


MessageFormat.format(messageTemplate,arg1,arg2)

ICU4J does not have a var-args method equivalent — it seems that ICU4J has not evolved for Java 5. To adopt ICU4J’s MessageFormat, all calls had to be converted as follows:


MessageFormat.format(messageTemplate, new Object[] { arg1, arg2 })

While the conversion can be achieved using regular expressions, the resulting code is less readable and more prone to bugs. Localization is hard enough as it is without having to jump through age-old languageisms.

Summary

While converting all plug-ins to ICU4J may make sense for some, making it a requirement for Galileo participation has caused me and presumably others a lot of work. From what I can see ICU4J brings little or no value to the users of my project. This is one case where good intentions with policy have a substantial negative impact. In my case, the time I’m spending converting to ICU4J could have been used to innovate or improve the quality of WikiText. Instead I’m jumping through hoops with no apparent benefit for me, my project or my users.

A Tough Lesson in AJAX, JSF and Client-Side State

2009-04-13T14:30:00.000-07:00

Last week I learned a tough lesson in AJAX, JSF and client-side component state. This one had me fooled for the better part of a day, in part due to the complexity of debugging multiple related AJAX requests, intermittent symptoms, and the opaque nature of JSF component state. Here's what happened:

The page that was problematic has multiple tabs (rich:tab), some combo boxes (h:selectOneMenu), and at least one suggest control (rich:suggestionbox). The tabs are server-side AJAX tabs. The suggest control also uses AJAX.

To reproduce the problem, the user takes the following steps:

Use a suggest control to select a value on the first tab of a
multi-tab form
Switch tabs to a different part of the form
Press a button on the form in the 2nd tab

At this point the application complains with validation errors on the first tab. The experience is very confusing to the user: after switching tabs successfully with no errors, pressing a button on the second tab causes validation errors on the first tab. This should never happen in the application: validation is designed to occur only for components that are submitted. When the button is pressed on the second tab, validation should not occur for components in the first tab. So what's going on?

Here's what I expected to happen:

The AJAX suggest control sends requests and renders itself
The tab sends requests and renders its contents
The button submits the form to make a request

Here's what really happened, depending on timing of the requests:

The AJAX suggest control sends a request
The tab sends a request
The browser receives the response from the tab click and re-renders the tab and its form controls
The browser receives the response from the suggest control

After receiving every response the DOM is updated to contain the new JSF component state (remember: we're using client-side component state, so all JSF component state is stored in a hidden form parameter in the browser).

Due to out-of-order responses, the component state was being set to that of the first tab by the suggest control -- thus the application thinks that the current tab is the first, not the second as we would expect. Pressing the button on the second tab results in the application interpreting the submitted form as submitting values for components on the first tab. This is problematic for HTML <select> controls, since HTTP parameters are not submitted for <select> controls where no value is selected.

So how do we fix this problem? Have all AJAX controls use the same request queue. This is achieved by setting the attribute eventsQueue="myApplicationEventQueue" on all of the richfaces AJAX controls. By doing so we end up with the following sequence:

Lesson learned: AJAX requests update JSF component state on the client, and thus must all complete in sequence. Otherwise component state on the client can get out of sync with what we expect it to be.

Mylyn WikiText produces PDF

2009-04-03T22:42:00.000-07:00

Mylyn WikiText becomes increasingly flexible with its latest addition of XSL-FO to the list of output formats supported by its wiki markup conversion capability. Combined with the excellent Apache FOP project, it becomes easy to produce PDF from your wiki markup.


 <-- create XSL-FO -->
 <wikitext-to-xslfo markupLanguage="Textile" file="My File.mediawiki"/>

 <-- create PDF -->
 <exec command="${fop.home}/fop">
  <arg value="${basedir}/help/My File.fo"/>
  <arg value="${basedir}/help/My File.pdf"/>
 </exec>

From a single source you can now produce HTML, Eclipse Help, DocBook, DITA and now XSL-FO and PDF. Wiki markup never looked so good!

If you're interested in trying out this new functionality you'll have to wait until the next weekly release is posted on the Mylyn downloads page.

Mylyn WikiText Experiences from EclipseCon 2009

2009-03-30T11:33:00.000-07:00

I presented my session titled "WikiText: Generate Eclipse Help from Eclipsepedia" (about Mylyn WikiText) at EclipseCon 2009. I had a lot of fun and lots of positive feedback. If you're interested in the slides from my presentation, you can find them here. A couple of interesting experiences from EclipseCon:

Trying Something New

Before the presentation I saw the Keynote Remote application for iPhone. Thinking it would be fun, I tried it for the first time while presenting my session. It turns out that to control slides you must swipe, which means that you need to know the orientation of your iPhone. Every time you want to change slides you have to look at your phone. Very distracting for both the presenter and the audience.

I learned the hard way something that I already knew: don't try anything new in a presentation!

Twitter

Twitter helped to create a buzz both before, during and after my session. The best thing about it for me was that I got immediate feedback on my presentation even before it was over.

iPhone Alarm

Beware that your iPhone alarm will sound even if the quiet switch has been activated. I discovered this design feature as Mike Milinkovich, the executive director of the Eclipse foundation was presenting. I think he was as surprised as me when the get-to-a-bomb-shelter-now siren sounded from my phone's alarm. Sorry Mike!

All in all I had a great time. I highly recommend EclipseCon for those who wish to meet interesting people with a common focus and lots of ideas.

Mylyn WikiText 1.0 Released

2009-03-17T06:41:00.000-07:00

I'm pleased to announce that Mylyn WikiText 1.0 has been released as part of Mylyn 3.1. Mylyn WikiText provides lightweight markup (wiki) parsing and editing capabilities to the Eclipse platform, Mylyn, Ant and stand-alone applications. Mylyn WikiText is the result of more than a year of development and over 170 bug fixes and enhancements. This marks the first release of WikiText after its move from the Textile-J project. Many thanks to the Mylyn team and the Eclipse community for making this possible. Here are some of the highlights:

Mylyn Task Editor Integration

With WikiText the Mylyn task editor is now markup-aware, providing source formatting, integrated preview, content assist and markup help. The following shows how markup can be authored and previewed in the task editor:

WikiText selects the markup language that best suits your task repository. This is great for users of JIRA, Trac and other repositories that provide support for wiki markup in task descriptions and comments. Users can also benefit from using wiki markup with task repositories such as Bugzilla that don't provide support for wiki markup.

Eclipse Integration

WikiText provides an editor for files containing wiki markup:

WikiText provides support for several markup languages, including MediaWiki, Textile, Confluence, TracWiki and TWiki. WikiText also provides an extensible parser framework for adding support for other markup languages.

Ant Integration

Users of Apache Ant now have access to Ant tasks for transforming wiki markup to alternative formats such as HTML, Eclipse Help, DocBook and OASIS DITA.

RCP, SWT and Server Applications

WikiText provides public APIs for integrating wiki markup capabilities into RCP applications and stand-alone SWT applications. These APIs make it easy to display markup and inspect its structure.

WikiText can also be used server-side to transform markup in web applications.

WikiText In The Field

WikiText is being used in many interesting ways:

The Mylyn project uses WikiText to convert MediaWiki content from Eclipsepedia to Eclipse Help content.
Mylyn WikiText authors help content in Textile and converts it at build time to Ecilpse Help content.
A recent addition to the Eclipse Examples project uses WikiText to create slide presentations based on files authored in MediaWiki markup.

Mylyn WikiText will also be presented at EclipseCon 2009 in WikiText: Generate Eclipse Help from Eclipsepedia.

More Information

Mylyn WikiText can be installed using an Eclipse update site or by downloading the standalone jars. See Mylyn downloads for details.

For more information about WikiText, see the Mylyn 3.1 New and Noteworthy, Mylyn FAQ or the Mylyn homepage.

Open Type Dialog - Small Things Make A Big Difference

2009-03-09T14:07:00.000-07:00

I'm a big fan of attention to detail when creating tooling for developers. Our brains are tuned to notice the smallest perceptible visual clues. We can absorb a lot of information with the right visual cues, however presenting that information without adding clutter to an IDE is a real challenge.

The JDT team has mangaged to improve the Open Type dialog in Eclipse 3.5 without falling into the trap of a cluttered interface:

Notice how a bold font is being used to identify regions of text that match the filter. The JDT team is using a new platform feature where multiple fonts can be used in one cell. Here's what it used to look like:

The best thing about this new feature is that my brain doesn't have to work harder to know more -- it's immediately evident to me which matches are better and why without having to think about it.

The importance of eliminating and minimizing visual noise is evidenced by products such as Mylyn that are designed to do just that: filter out what you don't need.

Kudos to the JDT team for raising the bar. I can't wait for Eclipse 3.5 Galileo to be released this summer.

Eclipse Foundation Committer Member

2009-02-24T09:10:00.000-08:00

Today I became an Eclipse Foundation Committer Member. This will enable me to vote in the 2009 Eclipse Foundation Elections.

Being committed (no pun intended) to the Mylyn WikiText project and being interested in projects such as SWTBot and SWT (for Cocoa) I believe that small and innovative projects should be well-represented on the Eclipse Foundation board.

Having known and worked with Mik Kersten in various ways over the past few years, I am constantly surprised by his breadth of knowledge, leadership, and grass-roots understanding of the needs of developers and committers. I've seen Mik uphold a high level of integrity and demonstrate consistent follow-through. I know that he has a solid understanding of what is needed by such projects and is capable of using his board experience and influence to make a difference.

Unsung Heroes Of Eclipse

2009-02-20T10:53:00.000-08:00

I'm glad to see a Top Contributor category in the Eclipse Community Awards that takes into account more than just code contributions. It's all too easy to lavish praise and recognition on those who contribute code -- but what about the others? The Mylyn WikiText project has been lucky to garner the attention of individuals who spend their own time installing, testing, performing quality assurance, and collaborating on Bugzilla.

In particular I'd like to thank Jörg Thönnes of Aachen, Germany for the many hours that he has contributed in making Mylyn WikiText a better product.

TPTP is Not For Me

2009-02-13T09:21:00.000-08:00

I've been using the NetBeans profiler to help me develop Eclipse plug-ins. A couple of years ago I tried TPTP (Eclipse Test and Performance Tools Platform), but found it extremely difficult to configure and it couldn't compete with Sun's dynamic bytecode instrumentation technology. Today I thought I'd give TPTP another go. The first link that I clicked on the TPTP homepage was the Getting Started link, and this is what I got:

Given that Safari and WebKit are largely acclaimed as being some of the most advanced web technology available, it seems that the TPTP team really doesn't want me to use their profiler. I'll check back later -- in the meantime I'll continue using NetBeans to profile Eclipse.

JPA 2.0: Why AccessType is Relevant

2009-02-10T12:07:00.000-08:00

It has long been debated whether field access is superior to property access for JPA entities. Most arguments that I've seen hinge on performance versus encapsulation. While these arguments may be interesting from a theoretical perspective, control over field access is in fact very useful in solving real-world problems.

Field access allows code to distinguish between the JPA provider setting properties while loading an entity versus application code setting a property.

This was very useful for a project that I'm involved with that has de-normalized data or other fields that are computed and persisted. Here's how it works:

In the setter for various properties that affect the computation of a de-normalized property, the value of the de-normalized property is set to null. @AccessType(FIELD) is used for these properties.
In the getter for the de-normalized property, if the value is null the de-normalized value is recomputed.
In a @PrePersist method, the getter is called to ensure that the de-normalized value is properly persisted.

Using this technique maintaining de-normalized data becomes simple and robust.

While this feature has been available within Hibernate since 3.0, I'm glad to see that JPA 2.0 (JSR 317) introduces a standard annotation to control this behaviour.

WikiText as a Slideshow Presentation Technology

2009-02-02T09:31:00.000-08:00

I'm constantly surprised by the versatility of wiki markup. Wayne Beaton has recently blogged about using MediaWiki markup to author presentation slides. He does this using Mylyn WikiText to parse wiki markup. Neat stuff!

Eclipse: Hitting a Breakpoint After The Fact

2009-01-30T15:02:00.000-08:00

Ever wonder how to hit a breakpoint after you've passed it? You can do this with the Eclipse Java debugger. Here's how I do it:

Stop at a breakpoint
Realize that you want to step through a method called a few lines back
Put a breakpoint in that method
Highlight an expression that uses the method in the java editor and from the context menu select Inspect

That should do it! Before you see the value in the inspection pop-up, your debugger should hit the breakpoint that you just created. Very cool.

Working With Patches and CVS

2009-01-28T09:20:00.001-08:00

Many contributions to Eclipse projects start as a patch. Patches are great because they can encompass changes to multiple files and make it easy to collaborate with others. Anyone can make a patch, so contributors don't need commit privileges to participate. Patches can be attached to tickets in the Eclipse Bugzilla so that committers can be confident of their IP cleanliness. The problem with patches is that they're static. That is, they don't evolve as the source code in CVS changes. Thus patches become stale, often resulting in committers asking contributors to re-cut a patch. So how do we overcome this problem?

I've had the pleasure of working from both sides of the problem. Starting as a contributor with the Textile-J contribution to Mylyn WikiText I created and maintained multiple patches, some small and some large. Eventually after earning Mylyn commit privileges I saw patches coming from contributors. The problem of the stale patch comes up again and again. Often contributions must wait a few days, weeks or even months before they're evaluated. During this time the files affected by the patch often change in CVS, thus rendering the patch unusable until it is massaged to fit the new code. Unfortunately this process of massaging is usually manual since the line-based merge algorithms used by CVS cannot account for non-trivial changes to source files.

So how do we evaluate and apply these valuable contributions? It's tempting to apply a stale patch and store conflicting changes in a temporary .rej file, and then merge those conflicting changes by hand afterwards. Though in some cases this approach makes sense, there is an easier way for those other cases.

The following formula can ease the pain for committers and contributors alike as they try to use a patch that has become stale:

Start by attempting to apply the patch using the Eclipse 'Apply patch' functionality, either from the Team menu or from within the Mylyn task editor.
Take note of which files have problems in the apply patch wizard.
Exit the wizard without applying the patch.
Take note of the time-stamp on the patch.
For each file in the patch that was problematic, use the CVS 'Team->Replace With->History...' wizard and select the most recent revision that predates the patch. Take note of which revision was used.
Now apply the patch. It should merge without issues. If not, you've missed something so go back to step 1.
For each file that was moved back in time in step 5, use the CVS History to compare the revision that was used with the patch to the latest revision in CVS. Apply the changes that are shown in the compare editor to the file in the workspace.

The formula turns the problem around. Instead of trying to apply the patch to the latest revision, we apply the changes made since the patch to the patched files. Sometimes this can be much easier.

How do you choose which approach to start with? I recommend starting with an up-to-date workspace and try applying the patch. If proceeding down that path is starting to feel too hard, then revert your changes and try the formula described here. In some cases, the hard problem becomes easy! If not.... well, patches aren't perfect. Sometimes hard is just hard.

Mylyn WikiText at EclipseCon 2009

2009-01-02T08:55:00.000-08:00

WikiText will be at EcilpseCon 2009 in Santa Clara, California. The presentation time was expanded to allow for more coverage of WikiText -- so if there's something specific that you'd like to hear about please let me know.

The WikiText presentation is included in a session titled "50 minutes towards a better you". Don't ask me about the title -- I don't know if WikiText will make you a better person or not. I'm very excited to be there presenting this new technology alongside Steffen Pingel, who will be providing a Mylyn connector crash course.

Mylyn Context-Driven Domain Diagram

2008-12-09T12:51:00.000-08:00

Lately I've been working on some large legacy modernization projects with MAKE's MDD tooling. These projects have upwards of 400 entities in their domain (read: minimum 400 database tables). With such a large application domain it's common to have large, complex domain diagrams which are often at the wrong level of granularity for programming tasks.

I found myself creating small, purpose-specific diagrams showing only those things that were relevant to the task at hand. These diagrams would get tweaked and maintained as my understanding of the task and domain evolved. While maintaining diagrams is not really all that hard, it is time-consuming and distracting.

After some thinking I realized that the answer to developer productivity was sitting right in front of me: Mylyn task context-driven diagrams. Why can't the MDD tooling create and maintain a domain diagram based on the interesting domain elements in the active Mylyn context? There would be no need to create or maintain such a diagram manually: it would always exist, and it would always show only those items that are relevant to the currently active task. Suddenly, purpose-specific diagrams become implied, easy, expected.

I realized that a few simple rules could make such a diagram really work:

domain elements in the Mylyn context always appear in the diagram
associations between elements in the diagram are always shown
removing an element from the Mylyn context causes it to be removed from the diagram
adding an element to the diagram causes it to be added to the Mylyn context
diagram elements are positioned according to a default layout algorithm, but remember their positioning if moved manually
diagram state is saved and associated with the Mylyn task id, such that it is restored when the task is re-activated
switching active tasks automatically resets the diagram to correspond to the new active task

As my Mylyn task context evolves with its decaying interest model the diagram becomes ever more pertinent. Mylyn landmarks prevent those really essential things from disappearing, other things come and go as needed.

Now that I've been working with automatic domain diagramming I wonder how I ever got by without. Context-driven diagrams are now an essential part of my toolkit that let me focus on the task at hand without distraction.

Eclipse DemoCamp Vancouver

2008-11-26T08:58:00.000-08:00

I had a great time at Eclipse DemoCamp last night. I was really impressed by two presentations in particular, both of them about testing tools.

One tool, called Tripoli is a differential code coverage tool. Tripoli shows you the difference in coverage between two execution runs of a program. It looked like a great tool to have for finding code relevant to a specific behaviour or event. Kaitlin Sherwood both created Tripoli and presented at DemoCamp.

The other tool presentation was about an Omniscient Debugger. Going into the DemoCamp, I was thinking "Omniscient, yeah right!". In reality, the debugger can show you exactly what happened on a timeline. You can go back in time and see when variables were assigned specific values. Yes, after it has already happened!! This is the first real time machine that I've seen actually working. The tool is called TOD and was presented by its creator Guillaume Pothier, who is a PHD student at the University of Chile.

Mik Kersten did a presentation about Tasktop and Mylyn. Being an avid Mylyn user, I can see how Tasktop adds a lot of value beyond the basics of Mylyn. As an early beta user of Tasktop ages ago, I'm impressed by how far they've come. At the time I was disappointed by their lack of support for the Mac. Though the Mac is still not officially supported by Tasktop, I've heard that it is much improved. Maybe it's time for another test drive.

My presentation on WikiText went well. If you're interested in the slides, here they are.

All in all, a great night. We ended up at the Lenox pub where I got to talk to more great people. I came home wishing that I'd had the opportunity to talk to more people. It's great to see such a lively Eclipse community in Vancouver.

Mylyn Converts Wiki-Based User Guide To Eclipse Help

2008-11-18T12:39:00.000-08:00

The Mylyn user guide is maintained online in the Eclipse wiki. While this has the many benefits of using a collaborative wiki for help content, this has meant that the Mylyn user guide is only available online on the wiki, not from within the Eclipse help system. Until now. Using WikiText Ant tasks Mylyn is able to convert the online wiki content into Eclipse help using an automated process. Users of Mylyn now have access to the user guide by using the standard Eclipse help system, including search, table of contents, etc. even when offline.

Going forward this means that the Mylyn project will be able to continue maintaining the user guide in the Eclipse wiki while also providing the same content from within the Eclipse. Furthermore, other Eclipse projects will be able to do the same by adopting the same approach.

Users wishing to see the user guide in Eclipse can do so by installing the Mylyn weekly build as described here. Eclipse committers that would like to have their wiki-based content available within Eclipse can do so by looking at the Mylyn project (org.eclipse.mylyn.help.ui/build-helper.xml) and by making use of the WikiText stand-alone package.

Mylyn WikiText Stand-Alone Package Available

2008-11-10T16:08:00.000-08:00

I'm pleased to announce that the Mylyn WikiText stand-alone package is available for download. The download contains jar files and documentation for including WikiText Ant tasks in your build or for using WikiText APIs outside of an Eclipse runtime.

The WikiText project is still in the incubation phase, thus users should be aware that the download has not yet completed a release review. The availability of WikiText stand-alone download enables users to try out WikiText prior to its release, including the much-anticipated wiki-to-DITA functionality.

This is the first time that WikiText stand-alone jars have been made available since the move of Textile-J to the WikiText project. WikiText includes many improvements and fixes since the Textile-J 2.2 release.

Many thanks to Mik Kersten, Steffen Pingel and others at Tasktop who have facilitated the WikiText project.

Mylyn WikiText targets OASIS DITA

2008-11-03T09:13:00.000-08:00

Wiki content has become increasingly popular within the DITA community, as evidenced by the recent announcement of the DITA2Wiki open source project. While most of these projects enable DITA-to-wiki conversions, community interest has been expressed for full-cycle wiki-DITA-wiki transformation capability. Mylyn WikiText hopes to close this loop with the addition of its support for DITA output. Mylyn WikiText can now create OASIS DITA as output from wiki markup.

Mylyn WikiText provides a flexible architecture supporting multiple wiki markup languages. This provides organizations with many options when considering a DITA toolchain, whether it be a one-time conversion of existing wiki assets to DITA, or as an integrated part of a publishing process. Currently supported are MediaWiki, Textile, Confluence, TWiki and TracWiki.

I am also very excited about the recent DITA Open Platform project, which aims to enable Eclipse as a full-fledged DITA authoring and publishing platform. With the capabilities provided by Mylyn WikiText and DITA-op, Eclipse as a platform becomes a compelling cross-platform DITA authoring toolset.

Mylyn WikiText originated from Textile-J, where parsing capabilities evolved to support more than 5 wiki markup languages, conversion of wikitext to DocBook, integrated support for the Ant build system and first-class integration with the Eclipse platform.

Mylyn WikiText is looking for community feedback on wikitext to DITA conversion. Please feel free to contact me directly via email or post a bug or enhancement request to the WikiText component of the Mylyn project.

Eclipse GUI Testing Is Viable With SWTBot

2008-09-23T20:49:00.000-07:00

For several years I've struggled with the viability of automated Eclipse GUI test frameworks. Now for the first time I've found an approach that works reliably. This article discusses the approach and details some specific techniques that greatly increase productivity in creating useful tests.

In the past automated GUI test frameworks have failed to meet my needs because they have focused too much on recording and playback. The complexity of the Eclipse environment makes it near impossible to get this approach to work reliably. Simple things such as background jobs and processing become major issues. Testing of GEF-based editors is not possible since all record/playback frameworks rely on identifying controls and widgets -- which GEF does not use.

Recently I've come across SWTBot, which uses a novel approach. SWTBot tests are written in Java, and run inside the Eclipse process as an Eclipse test application. This gives SWTBot full access to the SWT and Eclipse APIs. SWTBot tests are written as JUnits, which makes integration with common technologies such as Ant, Continuous Integration and code coverage tools easy.

But all is not rosy when writing tests with SWTBot. There are some Eclipse-specific idiosyncrasies that come back to bite you such as context menus that are reconstructed when shown and the GEF framework that does not use controls or widgets. That said, SWTBot provides and excellent starting point. Here's how I built on SWTBot to create a powerful test environment that is easy to use.

Initially my tests contained code that looks like this:


public void resetPerspective() {
bot.menu("Window").menu("Reset Perspective...").click();
bot.shell("Reset Perspective").activate();
bot.button("OK").click();
}

Not bad! It's easy to see what's going on. In other cases the code looked more like this:


SWTBotTree tree = bot.tree();
String[] path = name.split("/");
SWTBotTreeItem[] items = tree.getAllItems();
SWTBotTreeItem selectedItem = null;
for (SWTBotTreeItem item: items) {
if (path[0].equals(item.getText())) {
item.expand();
sleep();
selectedItem = item;
break;
}
}
for (int x = 1;selectedItem != null && x

As you can see, it's hard to see the forest for the trees. Way too much code is required to do simple things.

Here's what I did to make things easier:

1. Create classes that directly model the UI parts that you're working with in your tests. For example, instead of directly using SWTBotView to manipulate the Package Explorer view, create a class called PackageExplorer that delegates to SWTBotView and provides richer functionality. For example, in my PackageExplorer class I have the following method:


/**
* select the first element that adapts to the given resource
*/
public void select(final IResource resource)

2. Create a method to find and click context menu items in one go. In SWTBot the finding and clicking occur in two seperate UI runnables. In Eclipse this can cause problems for some context menus as the menu item gets disposed before it is clicked due to a loss of focus.

3. Instead of using sleep() to wait for something to be done, wait for the real thing to be done. A heavily loaded machine can cause processing times to vary. Instead of having a fragile sleep(500L), use a reliable technique to determine when the job is really done.

For example, if you know that your processing is holding a resource lock, post a no-op empty workspace job and wait on it inside your test. It will only be invoked once all other resource locks are released, so when it's complete you're guaranteed that your other job is done:


// ensure that all queued workspace operations and locks are released
ResourcesPlugin.getWorkspace().run(new IWorkspaceRunnable() {
public void run(IProgressMonitor monitor) throws CoreException {
// nothing to do!
}
}, new NullProgressMonitor());

4. Leverage and extend the SWTBot framework with Eclipse-specific behaviour. For example, make use of SWTBot's conditional waiting APIs by creating Eclipse-specific conditions like this one that is used to wait until an editor is opened on a resource:


/**
* a condition that is used to wait for an editor to open on a specific file.
*
* @author dgreen
*/
public class EditorOpenCondition extends DefaultCondition {
private final IFile file;

public EditorOpenCondition(IFile file) {
this.file = file;
}

public String getFailureMessage() {
return String.format("Timed out waiting for editor on %s to open",file.getFullPath());
}

public boolean test() throws Exception {
if (!file.exists()) {
return false;
}
return UIThreadRunnable.syncExec(new UIThreadRunnable.BoolResult() {
public boolean run() {
IEditorReference[] editorReferences = PlatformUI.getWorkbench().getActiveWorkbenchWindow().getActivePage().getEditorReferences();
for (IEditorReference reference: editorReferences) {
 try {
  IEditorInput input = reference.getEditorInput();
  if (input instanceof IFileEditorInput) {
   IFileEditorInput editorInput = (IFileEditorInput) input;
   if (editorInput.getFile().equals(file)) {
    return true;
   }
  }
  if (input instanceof IStorageEditorInput) {
   IStorageEditorInput editorInput = (IStorageEditorInput) input;
   IPath fullPath = editorInput.getStorage().getFullPath();
   if (fullPath.equals(file.getFullPath())) {
    return true;
   }
  }
 } catch (PartInitException e) {
  e.printStackTrace();
 } catch (CoreException e) {
  e.printStackTrace();
 }
}
return false;
}
});
}
}

5. Create a Java-based DSL for often-repeated use of editors or views. For example, for a static class diagram editor you might end up with a DSL that could be used as follows:


DomainModelDsl domainModelDsl = new DomainModelDsl();
domainModelDsl.create(project, "domain.dm");

domainModelDsl.
createEntity("A").
createEntity("B");

domainModelDsl.entity("A").extension("B");

Inside the DSL implementation the dirty work of manipulating the editor occurs. This makes it really fast to create complex tests that cover lots of ground.

6. Create GEF EditPart wappers. For GEF-based editors SWTBot doesn't give a lot of lift. For these you'll need to create classes similar to those provided by SWTBot, but instead of being widget-focused, they'll need to be EditPart-focused.

All in all I'm very impressed with SWTBot, which has finally delivered a viable automated GUI test framework for Eclipse and Eclipse RCP applications. I'm pleased to see that SWTBot has made an Eclipse project proposal, which if approved will hopefully lead to continued improvements and community adoption.

Beyond Rich Text (Part 2): Displaying Images in an SWT StyledText

2008-09-08T19:56:00.000-07:00

Recently I wrote about a few simple tricks that make it possible to display more than just text in a StyledText widget. In this article I take it a step further: displaying images in an SWT StyledText widget.

Normally a StyledText widget cannot display images. Why would you want to display images in a text control, you might ask? Well, there are cases where a rich text control is used to render text which may have embedded images. Take for example the Mylyn task editor. It displays bug descriptions and comments. In some repositories (such as Trac or JIRA) it's possible to use markup to display rich text and images. It's common to see screenshots or other visual data included in bug reports.

To display images in a StyledText we use a small variation on the tricks from my previous post. Previously I used an annotation to mark the location of the thing I wanted to draw, and a painter to paint it. This approach works well when we know the size of the thing we're about to draw (such as a character or horizontal rule). In the case of images, it gets more complicated. We need to know the size of the image so that we can create enough space for it in the text.

Since loading image data can be slow (over a network or from a busy disk) we must load the image asynchronously and render it when it becomes available. We can get away with it since users are used to this experience -- that's how web browsers work.

So here's a run-down of the technique I used:

Mark the location of the image with an annotation
Asynchronously download the image
When the image becomes available, determine its height (in pixels)
Determine the font in use at the annotated location
Using font metrics compute the number of blank lines required to create enough vertical space (in pixels) to display the image without obscuring text
Insert the required number of newlines into the text widget
Adjust text presentation offsets (some styled regions may need to be translated by the same number of characters as we inserted in the previous step)
Associate the image data with the annotation so that the painter will have access to it
Cause the text widget to repaint
Use the custom painter to paint the image in the empty space that we created

Here's a screenshot demonstrating this technique in action:

It works! The SWT StyledText can now display images!

Though the result is pleasing, there remain a few issues with this approach:

Images don't flow with the text, that is we can't wrap text around the image on the left or right side, only above and below.
Images aren't included in clipboard operations such as copy and paste.

If you're interested in the code involved in this technique you can find it here.

The fact that we can do this using standard JFace and SWT APIs is a testament to the quality and completeness of the platform. Thanks Eclipse!

Beyond Rich Text: Tricks Using SourceViewer, Annotations and AnnotationPainter

2008-09-02T08:36:00.000-07:00

An SWT StyledText can display text attributes such as bold, italic and strikethrough, alter colors and fonts. What are we to do when we need to go beyond rich text formatting? In this article we present three simple tricks using standard SWT and JFace APIs to create a polished presentation where text attributes alone won't do the job.

Marking The Spot

In order to display something interesting, we need to mark the spot. Normally this is done with a TextPresentation, which specifies character offsets and style ranges. Fortunately the Eclipse APIs give us another mechanism to mark regions of text in an extensible manner: annotations.

The key APIs at our disposal are as follows:

org.eclipse.jface.text.source.Annotation
org.eclipse.jface.text.source.AnnotationModel

Using these APIs we can create annotations and specify their location. An easy place to do this is in your document partitioner. It will be called at the appropriate times to partition your document. By using a RuleBasedPartitionScanner you can modify your rules to create the appropriate annotations.

Now that our document is annotated, we know where we need to draw. To hook up the drawing strategy, we add the following code to the initialization of the SourceViewer:


IAnnotationAccess annotationAccess = new IAnnotationAccess() {
public Object getType(Annotation annotation) {
    return annotation.getType();
}
public boolean isMultiLine(Annotation annotation) {
    return true;
}
public boolean isTemporary(Annotation annotation) {
    return true;
}
};

AnnotationPainter painter = new AnnotationPainter(sourceViewer, annotationAccess);

Now all we have to do is add a drawing strategy to the painter for every kind of annotation that we're interested in drawing. Read on to find out how we do that.

Repainting Characters

Not all fonts can display all characters. This is problematic in an application that is internationalized or one where the user can change the font.

Take for example bullets. Unicode \u2022 can be used to display a solid round bullet with most fonts, but what about an empty one, or a square one? These characters cannot be reliably found in commonly used fonts. The trick we use is to always use the \u2022 character, but repaint it where we want to display something more interesting. By doing that we get the display just right and the text works nicely with copy/paste operations.

To make it work, we create annotations where our bullet characters are in the document. We then hook up a bullet drawing strategy to our painter as follows:


painter.addDrawingStrategy(BulletAnnotation.TYPE, new BulletDrawingStrategy());
painter.addAnnotationType(BulletAnnotation.TYPE, BulletAnnotation.TYPE);
painter.setAnnotationTypeColor(BulletAnnotation.TYPE, getTextWidget().getForeground());

The painter won't invoke our drawing strategy unless the type and type color are also added.

What does our bullet annotation look like? It needs to have enough information for the drawing strategy to know what to draw. In this case the shape of the bullet is dependent on the 'level' of indentation. Here's what I used:



public class BulletAnnotation extends Annotation {

public static final String TYPE = "org.eclipse.mylyn.internal.wikitext.ui.viewer.annotation.bullet";

private final int indentLevel;

public BulletAnnotation(int indentLevel) {
super(TYPE, false, Integer.toString(indentLevel));
this.indentLevel = indentLevel;
}

public int getIndentLevel() {
return indentLevel;
}

}

Now we need to implement our drawing strategy. The drawing strategy must 'erase' the existing bullet character and then draw the new bullet shape where the old bullet was.

We erase the previous character by drawing a rectangle the size of the character in the background color:


// erase whatever character was there
gc.fillRectangle(left.x, left.y, right.x - left.x, lineHeight);

then we draw the new shape:


// now paint the bullet
switch (bullet.getIndentLevel()) {
case 1: // round solid bullet
 gc.setBackground(color);
 gc.fillOval(hcenter - 3, vcenter - 2, 5, 5);
 break;
case 2: // round empty bullet
 gc.setForeground(color);
 gc.drawOval(hcenter - 3, vcenter - 3, 5, 5);
 break;
default: // square bullet
 gc.setBackground(color);
 gc.fillRectangle(hcenter - 3, vcenter - 2, 5, 5);
 break;
}

Here's a screenshot showing an example of this technique in use:

Drawing Non-Characters

Sometimes there's a need to display non-characters. For example, browsers display a horizontal line for <hr /> (horizontal rule). By marking the spot with annotations and registering a custom painter, we can do the same thing. Here's the result we're looking for.

To create this effect we put an empty line in the text we're displaying and annotate it with a HorizontalRuleAnnotation. Drawing the annotation is easy:



public void draw(Annotation annotation, GC gc, StyledText textWidget, int offset, int length, Color color) {
if (gc != null) {
final Color foreground = gc.getForeground();

Point left = textWidget.getLocationAtOffset(offset);
Point right = textWidget.getLocationAtOffset(offset + length);
if (left.x > right.x) {
 // hack: sometimes linewrapping text widget gives us the wrong x/y for the first character of a line that
 // has been wrapped.
 left.x = 0;
 left.y = right.y;
}
right.x = textWidget.getClientArea().width;

int baseline = textWidget.getBaseline(offset);

int vcenter = left.y + (baseline / 2) + (baseline / 4);

gc.setLineWidth(0); // NOTE: 0 means width is 1 but with optimized performance
gc.setLineStyle(SWT.LINE_SOLID);

left.x += 3;
right.x -= 5;
vcenter -= 2;

if (right.x > left.x) {
 // draw the shadow
 gc.setForeground(shadowForeground);
 gc.drawRectangle(left.x, vcenter, right.x - left.x, 2);

 // draw the horizontal rule
 gc.setForeground(color);
 gc.drawLine(left.x, vcenter, right.x, vcenter);
 gc.drawLine(left.x, vcenter, left.x, vcenter + 2);
}

gc.setForeground(foreground);
} else {
textWidget.redrawRange(offset, length, true);
}
}

As before, we hook the drawing strategy up to the painter:


  painter.addDrawingStrategy(HorizontalRuleAnnotation.TYPE, new HorizontalRuleDrawingStrategy());
  painter.addAnnotationType(HorizontalRuleAnnotation.TYPE, HorizontalRuleAnnotation.TYPE);
  painter.setAnnotationTypeColor(HorizontalRuleAnnotation.TYPE,  getTextWidget().getForeground());

Conclusion

Eclipse provides some powerful APIs for hooking into the painting of StyledText. Using some simple tricks we can create powerful polished visuals in an SWT user interface. All of these techniques are applied in the Mylyn WikiText project, where you can find source code that works.

Mik Kersten on Rich Editing for Tasks via Mylyn WikiText

2008-08-17T09:51:00.000-07:00

Mik Kersten has writen a great article about WikiText and what it means for Mylyn users, titled Rich Editing for Tasks via Mylyn WikiText. He really does a great job of capturing the essence behind Eclipse's powerful source editing features and how it was applied to the markup editing features of WikiText.

Textile-J Is Moving to Mylyn WikiText!

2008-08-12T17:04:00.000-07:00

The Textile-J project announced today that it is moving to Eclipse as the Mylyn WikiText component. Textile-J, now WikiText, is a library for parsing and displaying lightweight markup languages (wikitext). With the addition of WikiText, Mylyn and Tasktop will be able to display markup as it is intended and provide a markup-aware authoring environment.

The move was primarily motivated by my desire to see Mylyn provide first-class support for lightweight markup languages in its task editor. Even though most issue-tracking systems provide support for markup (for example Trac and JIRA), WikiText can also benefit those who use issue-tracking systems with no markup support. For example, using Bugzilla with WikiText will enable users to use markup in their descriptions and comments.

Overall I'm very excited about the move to Mylyn WikiText, as it will not only improve Mylyn and Tasktop but will also benefit existing Textile-J users by ensuring a higher quality library. In designing the move I built some protection for existing Textile-J users into the WikiText project charter so that existing users will be able to continue using WikiText independently of an Eclipse runtime. This is important for users that have integrated Textile-J into their non-Eclipse projects and for those who use the Textile-J Ant tasks.

This project move has been many months in the making, and with the backing and help of the Mylyn team has finally come to fruition. Many thanks go to the team, especially Seffen Pingel, Mik Kersten and Jingwen 'Owen' Ou for their help in making this a reality.

WikiText is available immediately for download from the Mylyn incubation weekly update site. To find out more about WikiText, see the original announcement and the WikiText project homepage.

Eclipse Ganymede on SoyLatte (Mac+Java6+Eclipse)

2008-08-06T17:07:00.000-07:00

Mac has no Java 6. The Java 6 beta that Apple released is only 64-bit, which is no good for running Eclipse. Not only is this a major annoyance, it is a problem for developers that need to profile and take heap dumps of their applications.

Some time ago I found this post by Landon Fuller which supposedly launches Eclipse on SoyLatte. While I believe that Landon is a genius, it did not work for me. Well lately I've needed to take some heap dumps, so I tried again.

Here's what worked for me:


/Users/dgreen/Documents/java/soylatte16-i386-1.0.2/bin/java  -server -Xms128m -Xmx1536m -XX:MaxPermSize=256m -Dosgi.requiredJavaVersion=1.5 -Dorg.eclipse.swt.internal.carbon.smallFonts  -Djava.library.path=/Users/dgreen/bin/jnilib  -cp /Applications/eclipse/Eclipse.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar org.eclipse.equinox.launcher.Main -os macosx -ws carbon -arch x86 -showsplash -launcher /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse -name Eclipse --launcher.library /Applications/eclipse/Eclipse.app/Contents/MacOS//../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.100.v20080509-1800/eclipse_1114.so -startup /Applications/eclipse/Eclipse.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.100.v20080509-1800.jar -data /Users/dgreen/Documents/workspace -launcher /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse -keyring /Users/dgreen/.eclipse_keyring -consoleLog -showlocation -vm /Users/dgreen/Documents/java/soylatte16-i386-1.0.2

I also had to extract all *.jnilib files from eclipse/plugins/org.eclipse.swt.carbon.macosx_3.4*.jar and copy them into /Users/dgreen/bin/jnilib

Finally, Eclipse running in a Java 6 VM on a Mac! Apple, time to get your Java act in gear.

Update 2008-08-21: SoyLatte 1.0.2 works for me, but 1.0.3 and OpenJDK 7 darwin packages posted here don't. If you find a solution, please let me know!

Update 2009-05-11: See this related article Eclipse 3.5 (Galileo) on Java 7

Optimizing Screen Real-Estate in Eclipse

2008-07-08T20:26:00.000-07:00

I recently read an article stating that the Eclipse Package Explorer and Outline view are wasted space. Mylyn also recommends minimizing or closing the Outline view. Eclipse makes it really easy to make views go away with fast views and minimizing tab groups. An in-place Outline can be activated with Ctrl+O. All of these focus on providing maximum screen real-estate for editors. While these paradigms work really well for many editors, some editors are designed to make use of the Outline view and Properties view as integral parts of the editor. So how does a tool designer resolve this tension between editor styles?

Imagine that a developer is switching from a Java source editor -- where views should be minimized -- to an editor that makes use of the properties view and outline view. When switching back and forth the developer is going to feel friction. To try this out, try using the XSD editor in Eclipse with the properties view.

Enter the editor-controls-view paradigm.

For editors that require the outline and properties views, the editor 'restores' the views to their visible state when the editor gains focus. When such an editor loses focus, the views are minimized to their iconized state. Eclipse could support this editor-controls-view paradigm by allowing editors to declare their affinity with specific views.

This solution provides an optimal use of screen real-estate for both editing scenarios without requiring the developer to take additional action every time she switches editors. It also does not require the developer to switch perspectives, which is a heavier solution to the same problem.

Why Is Eclipse.org Still On CVS?

2008-06-18T20:17:00.000-07:00

With the recent announcement of their 1.5 release, Subversion (SVN) has clearly surpassed their original goal "to build a compelling replacement for CVS in the open source community". So why are all Eclipse projects using CVS as their source repository? I explore some of the reasons that go beyond the commonly extolled superiority of SVN.

In a new collaboration with the Eclipse Mylyn project, I've started working with CVS again after about 4 years of active SVN usage. Although I had used CVS previously (for several years) in my career, I had forgotten how bad it really is. After having near-instant branching, fast history retrieval, atomic commits, efficient storage, directory versioning, commit version numbers and web browser access, it is really, really hard to go back. What the Eclipse foundation may not realize is that using CVS is costing them time and money. So why CVS?

1. Tooling

Eclipse committers use Eclipse. From the early days, Eclipse has had really good support for CVS. UI integration is tight, and CVS plug-in features such as workspace-relative patches make it easy to collaborate with others.

The Subclipse plug-in for Eclipse is excellent, however it's still not as good as the CVS plug-in. The Subversive plug-in for Eclipse (which is now an Eclipse.org incubation project) is unusable in its current state.

One Eclipse committer claims that there remain some major usability issues with SVN inside of Eclipse. I'm not sure if he was talking about Subclipse or Subversive -- and in part I agree. If more Eclipse committers were using SVN, then these plug-ins would improve to the same polished level of quality seen in the CVS plug-in.

2. Inertia

Projects that are already using CVS have inertia in the form of commit history, tagging and branching habits, build scripts, etc. It's difficult to overcome this inertia. I've done it on several large projects, and it can be painful.

3. Training

Eclipse committers who use CVS daily are used to CVS and the way it works. Minor differences in UI paradigms of the Eclipse plug-ins and some larger differences with how SVN works take effort to learn and overcome. There is a cost associated with this, and people dislike change.

4. Lack of Organizational Support

Many issues facing the Eclipse project teams would be greatly diminished if the Eclipse foundation provided tools, tutorials, documentation, recommendations and guidelines for making the switch. With the right support, Eclipse could make the switch one project at a time over weeks, months or even years.

5. Time Pressures

Eclipse committers are already busy supporting Eclipse and meeting their milestones. The last thing they want is to be spending time learning new tools and managing project infrastructure. The perceived time-savings of remaining on CVS is a falsehood. I can't tell you how much time I've wasted waiting for a CVS update to complete.

Conclusion

Eclipse and Eclipse users suffer because of continued use of an antiquated source versioning system. The cost in time and money is substantial. With the right vision and organizational support, Eclipse can overcome their roots in CVS and become a more agile organization.

Update July 4 2008: A few Eclipse projects are using SVN for revision control. See Eclipse SVN technology projects for a list of technology projects in SVN.

Focus Your Work Week (Even More): Beyond Mylyn 3.0

2008-06-16T11:36:00.000-07:00

Mylyn and Tasktop provide a revolutionary approach to focusing the Eclipse UI. They work on the concept of having a task 'context', that defines a degree of interest (DOI) for resources related to a task. We can make this task context even richer and deliver efficiencies by providing first-class integration of our MDD tools with Mylyn.

When using Mylyn, developers build up their task context using their natural navigation though source code while performing the task. For example, to fix a bug a developer opens the relevant Java source files, and Mylyn will automatically add these to the task context. Based on the frequency and type of interactions with these files, Mylyn will increase or decrease the DOI.

The issue here is that developers start a task with an empty context. The context is empty because Mylyn doesn't know what the developer is doing (or what resources are related to a task), and as a result the Package Explorer and other views in Eclipse are initially empty.

Developers can get around this problem by disabling the Mylyn filter on those views, at the cost of having views filled with all kinds of information that the developer doesn't need to complete the task.

Enter MDD tooling.

MDD tools have the advantage of knowing which things are related. For example, an MDD model is typically used to generate source code. Thus the source code is related to the model. Models rarely exist in isolation: they usually have associations with other models. Thus MDD tools know which source files are related to which models, and which models are related to other models. Given a specific model, most files in a system can be related in some way with a 'degree of separation'.

How can this information be used to our advantage with Mylyn? With first-class Mylyn integration MDD tools can let Mylyn 'see' into the MDD models. Furthermore, these MDD tools can tell Mylyn which files to put into a task context based on how they are related to models in the context.

Developers can start their task by selecting the relevant MDD model, and tell the MDD tools to 'populate' the Mylyn task context. The developer starts their task with a context that already has interesting things in it.

The richer and more expressive the MDD models, the better the integration can be. For example, if the MDD models can express business requirements and methodology, development tasks and other project tasks can be created in a task repository such as Bugzilla or JIRA, prepopulated with a Mylyn context before the developer even starts.

Code coverage tools can be misleading

2008-05-15T10:54:00.000-07:00

Software development best-practices encourage a thorough automated test strategy and continuous integration (for me that's Hudson, Ant and JUnit). Even better is to measure code coverage of your tests using a tool such as Cobertura. Code coverage tools will tell you which code was executed by your tests. This kind of analysis is ideal for finding untested code, however it's easy to be mislead by code coverage reports.

Take for example the following ficticious example:


public class Calculator {
BigDecimal currentValue = new BigDecimal(0);
Stack history = new Stack();

public void add(BigDecimal augend) {
 currentValue = currentValue.add(augend);
 commandPerformed(Type.ADD,augend);
}

private void commandPerformed(Type type, BigDecimal augend) {
 // TODO Auto-generated method stub
 System.out.println(type+": "+augend);
}

public void undo() {
 // TODO
}

public void redo() {
 // TODO
}
}

Initially the coverage report will show that all of this class is not covered.

Create a test for the add() function, and the code coverage reports will show that the commandPerformed() has been covered -- even though it is clearly not doing what it is suppoed to.

Be careful when reading code coverage reports -- don't assume that code that is covered is tested. The most a coverage report can tell you is if code is not covered at all by your tests.

Authoring Cross-Platform Wiki Markup

2008-04-29T07:50:00.000-07:00

I'll never think of line delimiters the same way again. Mac uses \r, Unix uses \n, and Windows uses \r\n. Who cares? Well, when it comes to writing platform-independent Wiki markup, it matters.

Take for example a document that was written on a Unix platform. Every line is separated by a single \n character. For Wiki markup, paragraphs are separated by an empty line, which is represented in the document as two consecutive newlines \n\n. When a Wiki markup parser converts this document to HTML, it looks for the empty newline (the second \n) and uses that to close the previous paragraph and start a new one.

Now what happens if the document is opened and edited on a Mac? Mac uses \r as a newline. Suppose the Mac user adds a new empty line just before an existing one in this same document. For example:

Prior to editing:

some text\nmore text

After editing:

some text\r\nmore text

Prior to adding the newline, the document contains a single \n character, and afterwards the document contains \r\n. In the users editor (on a Mac) the \r\n will appear visually as two lines, however \r\n happens to be a Windows line delimiter and thus will be parsed by Windows-accomodating Wiki markup parsers as a single line delimiter. In many documents this may not matter, however in Wiki markup this can make a big difference.

Editors that support editing Wiki markup on multiple platforms must be coded carefully to avoid this issue. For Textile-J this means that the end-of-line markers are converted to the platform default when the editor first opens a file.

Who knew that line delimiters could be so important.

Roller and Textile-J

2008-04-24T20:39:00.000-07:00

The Roller Support Project has announced plugins for parsing markup using Textile-J. This brings support for Textile, MediaWiki, Confluence and TracWiki markup to the Apache Roller blog server. Great to see projects like this picking up Textile-J, good work Anil!

JUnit Made It Easy!

2008-04-15T06:53:00.000-07:00

Recently I posted about evolving the architecture of a markup parser. During its evolution I ended up completely rewriting the parser to a new architecture. A solid JUnit test suite made it possible to do this while maintaining confidence in the quality of the new code. This article details the nature of JUnits that make such a switch easy.

Methodology:

While developing the Textile-J markup parser I incrementally improved the parser by adding support for markup syntax. Paragraphs were added first, then bold text, then lists, you get the idea. As support was added for each syntax feature, I created one or more JUnit tests to exercise the feature. I inspected the results of the HTML output from the small snippet of Textile used, and when I was satisfied created one or more assertions in the JUnit. This made development easy because:

I could isolate the syntax that I wanted to support and test it independently of other syntax
Since the tests were small, stepping through the code in the debugger only involved the relevant code
It's easy to see when something breaks that worked before
I could be confident in the quality of the parser

The Result

The result of using this methodology was a JUnit test suite of over 100 tests, and I could easily measure progress in supporting Textile markup features.

Markup Parser Rewrite

Rewriting the parser first involved stubbing out the new parser API, and then making the old parser API a facade to the new one. In doing this my new parser API automatically gained over 100 JUnit tests -- which of course were now failing. By incrementally improving the new parser until all JUnits passed, I could be certain that the new parser supports the same markup language features.

As I finished the last language feature and saw green for every JUnit test, I had a huge sense of relief. I now knew that the new parser was working with a guaranteed level of quality.

The Textile-J project now has over 250 JUnit tests. To get an idea what feedback JUnit can give, take a look at the test report here, and the code coverage report here.

Attributes of a Test Oriented Project

The following are some attributes of my project that make JUnit tests so powerful:

A JUnit test for every feature
Meaningful assertions
An environment that makes running JUnits painless and easy (like Eclipse)
A development methodology that encourages or requires tests for all new code
An Ant build script that runs the JUnits whenever a new build is created
Integrated code coverage (such as cobertura) so that you can see which code is tested

Once your project is set up to run tests, it's easy to add them... so jump that first hurdle and get testing with JUnit!

Evolving a Wiki Markup Parser

2008-04-02T12:43:00.001-07:00

Initially my Textile-J open-source project started out with modest goals: to provide a Textile markup parser for Java. Since the project has started, interest in other features drove evolution of the code-base in some interesting directions. This article discusses architectural choices and the evolution of the Textile-J parser architecture to meet changing requirements.

The initial design of the Textile-J parser was a monolithic stateful parser class that used regular expressions and local variables to parse the markup. The results of parsing markup were passed to the XMLStreamWriter interface as HTML elements and attributes. This architecture worked fine initially, as it allowed me to evolve my understanding of Textile markup structure (blocks, phrases and tokens) and the XMLStreamWriter provided a solid means of outputting XHTML.

Feature: Multiple Output Formats

The first major feature request was to support multiple output formats: HTML and DocBook. This required a major rethink of the parser. While XMLStreamWriter was a great means of outputting XML, the parser had to know that HTML was the output format. After some thinking and reviewing the ever-relevant GOF Design Patterns book, I recognized that the Builder design pattern was an ideal fit. So I created a new interface called DocumentBuilder, with an HtmlDocumentBuilder and DocbookDocumentBuilder implementations. Now the parser need not know about the output format, meaning that a single parser could drive multiple output formats.

Feature: Markup Dialects

Next extensions to the Textile markup language were requested. For example, Confluence markup dialect is very similar to Textile but has some additional syntax features. With a better understanding of markup structure (blocks, phrases and tokens) I designed a new 'Dialect' concept that allowed for markup extensions to be added to the base Textile language. The 'Dialect' design was object-oriented, making extensions modular and relatively easy to add without disturbing the base markup parser code.

Feature: Markup Languages

While the approach of markup dialects worked well for adding extensions to Textile parsing, it did not solve the problem of fully supporting new markup languages. Dialects at this point were only capable of extending Textile.

Community demand for supporting new markup languages was increasing, including requests for MediaWiki and Markdown. Supporting these languages was not possible using the existing Textile parser, as the markup rules of Textile were embedded in a single monolithic class. After some prolonged hesitation (this would be a big job), I got down to the design of a complete rewrite of the parser architecture.

The first step was to read up on various markup languages. Most languages that I've looked at use a simple line-based parsing approach that consists of dividing the markup into blocks, phrases and tokens. Blocks are usually multi-line constructs that have certain attributes, such as paragraphs and lists. Phrases are modifiers that affect text on a single line, and tokens are match-and-replace elements in the text.

Requirements of the new design had to include:

ease of adding new markup languages
modular object-oriented design for better maintainability
facilitate comprehensive JUnit tests
easier learning curve for community contributions
pluggable architecture
output format agnostic

Using my experience with the previous Dialect design here's what I came up with:

The parser delegates all language-specific parsing to the Dialect. The Dialect defines a language with a collection of Blocks, phrase modifiers (PatternBasedElement) and tokens (PatternBasedElement). Blocks implement rules specific to paragraphs, lists, tables, etc. Concrete PatternBasedElementProcessor classes know how to emit portions of content affected by markup (phrase modifiers or tokens).

So far the following languages have been imlemented using this new architecture:

Textile
MediaWiki (a la WikiPedia fame)
Confluence

I hope to see contributions from the community for supporting other languages, such as Markdown and Creole.

Hacking SWT

2008-02-28T14:04:00.000-08:00

Talking to Mik at Tasktop about what it will take to bring his revolutionary Mylyn-based product to the mac platform, Mik pointed me to an issue with the way the Safari-based SWT browser widget accesses GMail. Though Safari is a supported browser for GMail, the embedded Safari browser in eclipse is not recognized by GMail. This post highlights the ups-and-downs of my investigation into this issue.

Others at Tasktop had discovered that the SWT browser works if gmail is accessed via the http://mail.google.com/mail/?nocheckbrowser URL, which was promising. This lead me to believe that the user-agent header of an embedded Safari was different than that produced by Safari launched from the desktop. A quick test browsing to http://whatsmyuseragent.com proved my suspicions were correct.

Here were the user-agent headers:

Safari Inside Eclipse: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko)

Stand-alone Safari: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko) Version/3.0.4 Safari/523.12.2

The next step then was to see if Safari provides an API for changing the user-agent header. Having already installed XCode on my system, it was easy to search the API documentation. I found a section called 'Spoofing', which is specifically about an API for modifying the 'user-agent' header:

setCustomUserAgent
setApplicationNameForUserAgent

By the looks of the difference in user-agent headers, the second one was exactly what I was looking for.

Knowing that SWT involves native calls to Cocoa, I was a little wary about the next step. Looking at Safari.java I could see in the create() method other calls to the Safari web view to configure various properties. All of them were performed using JNI native calls, one of the various Cocoa.objc_msgSend variants. Unfortunately, there wasn't one with the signature that I needed. The one that I needed would look like this:


public static final native int objc_msgSend(int object, int selector, String string);

I added the method to Cocoa.java, and added a call to it in Safari.create() as follows:


// [webView setApplicationNameForUserAgent:];
Cocoa.objc_msgSend(webView, Cocoa.S_setApplicationNameForUserAgent,"Safari/unknown");

The next step is to provide the native implementation of the method.

Using the eclipse search command, I could see that the other objc_msgSend implementations were in cocoa.c Never having done JNI before, it was time to do a little looking around. The SWT FAQ had most of what I needed, indicating how to build the native libraries, etc. The tricky parts were as follows:

how to determine the mangled function name for the native method?
now to convert the jstring argument to an NSString?

For the first issue, a little guessing and a look at the JNI spec where it talks about mangling and I got it mostly right. Stubbing out the method, building it (according to the instructions in the SWT FAQ) and I got a java.lang.UnsatisfiedLinkError. Running it again in the debugger, I was able to see the method name it was looking for and fixed up the name of the method in cocoa.c. Rebuild, re-run, and so far so good! My stubbed out method was being called whenever a new SWT Browser control is created. This is what I had so far:


#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;
return rc;
}
#endif

So now on to the method implementation: how to call the right method? A quick google on jstring to NSString resulted in the following:


#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;

Cocoa_NATIVE_ENTER(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
const char *str = (*env)->GetStringUTFChars(env, arg3, 0);

/* use NSString as follows: [NSString stringWithCString: str] */

(*env)->ReleaseStringUTFChars(env, arg3, str);
Cocoa_NATIVE_EXIT(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
return rc;
}
#endif

Okay, so now all I had to do is actually send the Objective-C message to the object! Copying from other examples in the same file, here's what it I ended up with:


#ifndef NO_objc_1msgSend__IILjava_lang_String_2
JNIEXPORT jint JNICALL Cocoa_NATIVE(objc_1msgSend__IILjava_lang_String_2)
(JNIEnv *env, jclass that, jint arg0, jint arg1, jstring arg3)
{
jint rc = 0;

Cocoa_NATIVE_ENTER(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
const char *str = (*env)->GetStringUTFChars(env, arg3, 0);

rc = (jint)objc_msgSend((id)arg0, (SEL)arg1, [NSString stringWithCString: str]);
(*env)->ReleaseStringUTFChars(env, arg3, str);
Cocoa_NATIVE_EXIT(env, that, objc_1msgSend__IILjava_lang_String_2I_FUNC);
return rc;
}
#endif

I rebuilt the native libraries, re-ran eclipse, and voila!! It worked! The SWT Safari browser widget was now sending a user-agent string as follows:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/523.12.2 (KHTML, like Gecko) Safari/unknown

I browsed back to GMail with this new patch, and GMail now recognized Safari embedded in eclipse as a supported browser. For my first foray into hacking SWT, using JNI and Objective-C, it was a great success.... now if only the SWT team will accept my patch ;)

updated Feb 29: the fix was committed by the SWT team: see https://bugs.eclipse.org/bugs/show_bug.cgi?id=220836 for details

Integrating scripting languages into your DSL using JSR-223

2008-02-25T10:52:00.000-08:00

I've been building tools for Model-Driven Engineering (sometimes known as Model-Driven Development) now for about 11 years, but this is the first time that I've integrated scripting capabilities into a DSL using Java. The opportunity came up when creating a DSL for modernizing legacy data for my company MAKE Technologies Inc. Following is a summary of how this was done.

The idea is that the functional capabilities of the DSL can be extended by scripting the behaviour in the DSL itself. By integrating with JSR-223 APIs we can support as many scripting languages as are JSR-223 compliant. By the looks of scripting.dev.java.net, there are many including Java itself, Groovy, JRuby, Python, BeanShell, ECMAScript (JavaScript) and PNuts.

The DSL defines the concept of a 'caster', which can cast an arbitrary string value to some other value. For example, a trivial boolean caster might consist of a function that can cast a 'Y' or 'N' value to a boolean true or false. So we define it as follows:


<caster language="groovy" name="YesNoToBooleanCaster">
String cast(text) {
   if (text == 'Yes') {
       return true
   } else if (text == 'No') {
       return false
   } else {
       throw new Exception("Unexpected value '$text'");
   }
}
</caster>

Elsewhere in the DSL we can refer to the new caster by it's name YesNoToBooleanCaster.

The code to make the new caster work is relatively simple using the javax.script APIs:


String languageName = "groovy"; // get the language from the DSL instance
String script = ""; // get the script from the DSL instance

ScriptEngine scriptEngine = new ScriptEngineManager().getEngineByName(languageName);
Invocable invocable = (Invocable) scriptEngine.eval(script);

Object value = invocable.invokeFunction("cast", "Yes"); // test it out
assert Boolean.TRUE.equals(value);

With a little more work, I was able to hook up auto-suggest for the language attribute in the DSL editor (ScriptEngineManager can tell you which languages are supported) and provide validation of the scripting (ScriptEngine.eval can be used to detect script syntax errors).

All of this and the script runs very quickly at runtime, thanks to bytecode compilation by most supported scripting languages!

The only real issues that I encountered were:

extracting useful error messages from exceptions thrown by ScriptEngine.eval can be tricky. For example, JRuby provides the information in a getException() method on the Exception object. Some reflection and chained exception unraveling was used to overcome this problem in a language-independent manner.
Older Java 6 VMs (developer preview versions) have old preview versions of the JSR-223 APIs... stick to a non-beta version of the Java 6 VM if you can, or if you're developing for Macs then you may have to catch NoSuchMethodError and use some reflection magic.

All in all I was very impressed with the ease of using JSR-223. The end result when used with DSLs is that it is very easy to provide an extensible DSL API with a very low barrier to entry.