Using ANTLR in your RCP

ANTLR is a parser generator widely used to create DSLs and all kinds of parsers for special purposes. In this tutorial I will show how to create a simple parser and how to implement it into an RCP.

Source code for this tutorial is available on googlecode as a single zip archive, as a Team Project Set or you can checkout the SVN projects directly.

Prerequisites

We need some stuff from the ANTLR download page. Get Complete ANTLR 4.0 Java binaries jar and ANTLR 4.0 Java runtime binaries jar from there. The full package includes the build tool for grammars, the runtime binaries are needed to execute a parser in our RCP later on.

Step 1: Create a simple grammar

As this tutorial is not about ANTLR itself, but its integration into an RCP, we will use a very simple grammar to parse CSV files (actually just a single line of separated integers).

Currently there seems to be no eclipse support for ANTLRv4. To write grammars you could use ANTLRWorks 2 or write it directly in an editor of your choice.

Create a new Plug-in Project named com.example.antlr.parser. Afterwards create a grammar file CSV.g4 in the project root and add following content:

grammar CSV;

csv: NUMBER (DELIMITER NUMBER)*;

DELIMITER: ',';
NUMBER: DIGIT+;

fragment
DIGIT: [0-9];

Step 2: Compile grammar

To compile our grammar file we need the Complete ANTLR package we downloaded earlier and a simple script. As there is no native eclipse builder available, we will use an external build script.

Create a new folder build and copy antlr-4.0-complete.jar to it. Afterwards create a new file build.bat with following content:

"C:\Program Files\Java\jdk1.7.0_04\bin\java.exe" -cp antlr-4.0-complete.jar org.antlr.v4.Tool -visitor -package com.example.antlr.gen -o ..\src\com\example\antlr\gen\foo ..\CSV.g4

Extend this to your needs, port it to ANT or integrate it in a maven build (there seems to exist a maven target for ANTLRv4). It seems that the -o parameter needs some dummy folder name at the end. At least for me the very last path entry was skipped by the build tool.

You may launch this batch file from the External Tools located in the Run menu. The launcher is available in the source repository in the build folder.

Step 3: Add ANTLR runtime

The created files show lots of error markers because we have no runtime available yet. Create a new Plug-in from Existing JAR Archives and add antlr-runtime-4.0.jar to it. Name the project org.antlr.v4.runtime and add it as a dependency to com.example.antlr.parser.  All the error markers should be gone by now.

ANTLR comes with a BSD like license so you have to check if this fits to your application needs. According to this wiki entry it seems that it fits to EPL.

Step 4: Implement the parser

To run the parser we need 2 more classes. Lets start with com.example.antlr.CSVVisitor

package com.example.antlr;

import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;

import org.antlr.v4.runtime.tree.RuleNode;
import org.antlr.v4.runtime.tree.TerminalNode;

import com.example.antlr.gen.CSVBaseVisitor;
import com.example.antlr.gen.CSVLexer;

public class CSVVisitor extends CSVBaseVisitor<List<Integer>> {

 @Override
 public List<Integer> visitChildren(RuleNode ctx) {
  List<Integer> numbers = new LinkedList<Integer>();
  for (int index = 0; index < ctx.getChildCount(); index++)
   numbers.addAll(visit(ctx.getChild(index)));

  return numbers;
 }

 @Override
 public List<Integer> visitTerminal(TerminalNode node) {

  if (node.getSymbol().getType() == CSVLexer.NUMBER) {
   int number = Integer.parseInt(node.getText());
   return Arrays.asList(number);
  }

  return Collections.emptyList();
 }
}

Now we can finally call the parser from com.example.antlr.CSVProcessor

package com.example.antlr;

import java.util.List;

import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;

import com.example.antlr.gen.CSVLexer;
import com.example.antlr.gen.CSVParser;

public class CSVProcessor {

 public static List<Integer> parseCSV(final String data) {
  CSVLexer lexer = new CSVLexer(new ANTLRInputStream(data));
  CSVParser parser = new CSVParser(new CommonTokenStream(lexer));

  return parser.csv().accept(new CSVVisitor());
    }
 
 public static void main(String[] args) {
  List<Integer> numbers = parseCSV("2,45,66,123,24,6,7,7,8");
  System.out.println(numbers);
 }
}

If you need support on ANTLRv4 check out the webpage or - even better - support the project and buy the book.