Wednesday, July 27, 2016

Oomph 02: A setup in action

During our first tutorial we started an installation using the Oomph installer. Now we will have a closer look on the applied tasks, how to monitor and relaunch them and where these settings get persisted.

Oomph Tutorials

For a list of all Oomph related tutorials see my Oomph Tutorials Overview.

Workspace Setup

Right after the installation Oomph prepares your workspace. While busy you can see a spinning icon in the status bar at the bottom of your eclipse installation. A double click reveals a progress dialog where you can investigate all actions Oomph performs.
Oomph provides a toolbar, which is hidden by default. Enable it in Preferences / Oomph / Setup Tasks by checking Show tool bar contributions. Now we can repeat setup tasks or add additional project setups to our installation using the Import Projects... wizard from the toolbar setup entry.

Preferences Recorder

One of the most interesting features of Oomph is the preferences recorder. It can be enabled in the preferences window by selecting the record item in the bottom left corner. Once enabled it records all preference changes and stores them for you. When switching to another workspace these settings are applied directly. In practice this means: change a setting once and as long as you stick to Oomph you never have to think about it anymore.

Generally setup tasks (like setting preferences) may be stored to one of three different locations:
  1. User
    This is a global storage on your local machine shared for all installations and workspaces. Most of your changes will go here.
  2. Installation
    Settings get stored in the configuration folder of your current eclipse installation. These settings apply as long as you stick to the current eclipse binary.
  3. Workspace
    These settings get stored in the .metadata folder of your current workspace. So they are workspace specific, no matter which eclipse binary you use to access this workspace.
Personally I did not find a use case for options 2 or 3 yet.

Investigate Oomph Setups

Now that we know of the different storage locations, we can have a look at their content. The second toolbar item allows to open each one of them in the Setup Editor (setups are also available from the Navigate Open Setup menu).

The editor displays a tree structure of all Oomph tasks. As it is based on EMF we have to open the Properties view to display details of each tree element. If an element has a [restricted] annotation next to its name this means that the definition of this item is referenced by the current setup file. Typically this refers to a setup stored on the web. Such entries are readable, but cannot be changed without opening the original setup file.

Now that you are familiar with the basic ingredients we are ready to start building our own project setups.

Thursday, July 21, 2016

Oomph 01: A look at the eclipse installer

This will be the start of a new series of posts on Oomph. It is the basis for the eclipse installer but with the right configuration it can do much more:
  • serve your own RCP applications
  • provide fully configured development environments
  • store and distribute your settings over multiple installations
to name a few. This first part will have a look at the installer itself. Further posts of this series will focus on custom setups and how we can leverage the power of Oomph.

Oomph Tutorials

For a list of all Oomph related tutorials see my Oomph Tutorials Overview.

Step 1: The Eclipse Installer

All starts with the Eclipse Installer, a tool we will need throughout this series. Download and unzip it. The installer is an eclipse application by itself, so it provides the familiar folder structure with plugins, features, ini file and one executable. As we will need the installer continuously find a safe home for it.

After startup you will be in Simple Mode, something we will not cover here. Use the configuration icon in the top right corner to switch to Advanced Mode. The first thing we are presented with is a catalog of products to install.
The top right menu bar allows to add our own catalogs and to select which catalogs are displayed. For now we will ignore these settings, they will be treated in a separate tutorial. After selecting a product the bottom section allows us to select from different product versions, 32/64 bit, the Java runtime to use and if we want to use bundle pools.


Bundle Pools

A bundle pool is a place that stores - among some other things - plugins and features. Basically everything that a typical eclipse application would host in its plugins/features folders. Further it may host the content of target platforms.

Using a shared bundle pool saves you from a lot of redundant downloads from eclipse servers and can provide offline abilities. For everything available in the bundle pool you do not require an internet connection anymore. A nice feature if you are sitting behind a company firewall. While it is not required to use them, bundle pools save you a lot of time and are safe and convenient to use. At first I was quite hesitant of splitting my installations and move stuff to bundle pools, but after giving it a try I do not want to step back anmore.

To have some control over the used bundle pools, click on the icon next to the location and setup a New Agent... on a dedicated location. Further eclipse installations will use this pool, so do not alter the directory content manually. The Bundle Pool Management view will allow you to analyze, cleanup and even repair the directory content.
Step 2: Project Selection

The 2nd page of the installer presents eclipse projects we want to add to our installation. Selecting a project typically triggers actions after the plain eclipse installation:
  • automatically checkout projects
  • import into the workspace
  • set the target platform
  • apply preference settings
  • setup Mylyn
  • install additional components
The target is that you get everything pre-configured to start coding on the selected projects.

Step 3: Installer Variables

Installations do need some user input for the install location, repository checkout preferences, credentials and more. All these accumulated variables will be presented on the next page of the installer. By default the installer creates three folders below the Root install folder:
  • eclipse
    to host the eclipse binary and configuration. If you use bundle pools plugins and features go there. Otherwise they will be located here.
  • ws
    the default workspace for this installation
  • git
    the default folder for git checkouts
You may go with these defaults or change them to your needs. While developing a setup (which we will start in an upcoming tutorial) I would recommend to use these settings. For a final installation I prefer to host my workspace elsewhere.

Oomph stores all your settings in a global profile. So the next time you install something it will use your previously entered values here. You may always revisit your choices by enabling Show all variables in the bottom left corner.

The last page finally allows you to enable offline mode and to enable/disable download mirrors. On the next tutorial we will have a closer look at setup tasks and where these settings get persisted.

Optional: Preferences

The icons on the bottom allow to set two kinds of important preferences: proxy and ssh settings. If you are behind a proxy activate those settings and they will automatically be copied to any installation done be Oomph.

Ssh might be needed for git checkouts depending on your repository choices. If you do not use the default ssh settings you might need to wait for Neon.1 to have these settings applied (see bug 497057).

Thursday, May 12, 2016

EASE release, Community award & EclipseCon France

EASE v0.3.0 release

Today we shipped the brand new v0.3.0 release of EASE, your beloved scripting framework. Most important we improved the installation process for Jython as you do not need to know about external update sites anymore.

Code completion got introduced to help you browse your script objects and to improve coding speed. Completion proposals are provided for module functions, Java classes and even for some parameters (like file paths). In case you run eclipse from a JDK we also provide help popups for java methods.



Then we added a whole bunch of new commands to our modules to simplify your scripts. The Scripting module now provides means to exchange data between multiple script instances, the UI module got new methods to manipulate views and the brand new P2 module allows to install or update packages.

Of course we also had lots of bugfixes, if you are interested you may browse the complete list of changes.

Community Award

While the release might be the most important news for you, the development team is really proud of the Eclipse Community Award for the Most Innovative Project 2016.


It is a real honor that you, the community, rate this little project so high. While there seems to be quite some userbase using EASE, I have little knowledge of the adopters out there. I would love to read who you are and what you are using EASE for. If you got a minute left, please leave a comment to this blog entry.

EclipseCon France

In case you have no idea why this project might be innovative, meet us at EclipseCon France, where there will be several talks on EASE. I will be involved in 2 of them:

On Tuesday afternoon there will be a tutorial session EASE-ily Make the Most of Eclipse with Python, where we will do a lot of scripting (of course in Python). Besides scripts we will also look on how to integrate such scripts into the IDE and how to distribute them successfully.

On Thursday morning I will be giving a talk on the EASE framework itself, showing how to Elevate your IDE with scripts. We will see what scripting is capable of and I will give a sneak preview of the upcoming features for the 0.4 release. I will just say so much: your scripts never integrated so well and easily with your IDE.

While those talks are already worth going to EclipseCon, there is also tons of other interesting topics to investigate, so be there!

Monday, April 25, 2016

A new interpreter for EASE (5): Support for script keywords

EASE scripts registered in the preferences support  a very cool feature: keyword support in script headers. While this does not sound extremely awesome, it allows to bind scripts to the UI and will allow for more fancy stuff in the near future. Today we will add support for keyword detection in registered BeanShell scripts.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: Provide a code parser

Code parser is a big word. Currently all we need to detect in given script code are comments. As there already exists a corresponding base class, all we need to do is to provide a derived class indicating comment tokens:
package org.eclipse.ease.lang.beanshell;

import org.eclipse.ease.AbstractCodeParser;

public class BeanShellCodeParser extends AbstractCodeParser {

 @Override
 protected boolean hasBlockComment() {
  return true;
 }

 @Override
 protected String getBlockCommentEndToken() {
  return "*/";
 }

 @Override
 protected String getBlockCommentStartToken() {
  return "/*";
 }

 @Override
 protected String getLineCommentToken() {
  return "//";
 }
}

Step 2: Register the code parser

Similar to registering the code factory, we also need to register the code parser. Open the plugin.xml and select the scriptType extension for BeanShell. There register the code parser from above. Now EASE is able to parse script headers for keywords and interprets them accordingly.

Wednesday, December 16, 2015

A new interpreter for EASE (4): Provide modules support

EASE is shipped with modules: libraries written in Java, made available for script interpreters. The intention of such modules is to write them once and use them in all available script languages.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 


Introduction

Typically modules are plain java classes. We could easily create an instance in BeanShell and then call instance.method(). While this is perfectly ok for the average programmer, some of our user might expect a more simple, functional interface. What module loading does is:
  • create an instance of the module
  • inject the instance into the interpreter
  • create dynamic script code to wrap method calls
  • execute the created code
For BeanShell we would create code like
method(param1, param2) { 

result = __MOD_module_instance.method(param1, param2);

return result;

}
After loading this fragment we could access method() without the knowledge of modules, instances and object orientation at all.

Step 1: The code factory

To dynamically create code EASE needs an ICodeFactory implementation. To start you may look at existing implementations, basically we just need to build strings containing script code.
package org.eclipse.ease.lang.beanshell;

import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.List;

import org.eclipse.ease.Logger;
import org.eclipse.ease.modules.AbstractCodeFactory;
import org.eclipse.ease.modules.IEnvironment;
import org.eclipse.ease.modules.IScriptFunctionModifier;
import org.eclipse.ease.modules.ModuleHelper;

public class BeanShellCodeFactory extends AbstractCodeFactory {

 public static final List<String> RESERVED_KEYWORDS = Arrays.asList("abstract", "continue", "for", "new", "switch", "assert", "default", "goto", "package",
   "synchronized", "boolean", "do", "if", "private", "this", "break", "double", "implements", "protected", "throw", "byte", "else", "import", "public",
   "throws", "case", "enum", "instanceof", "return", "transient", "catch", "extends", "int", "short", "try", "char", "final", "interface", "static",
   "void", "class", "finally", "long", "strictfp", "volatile", "const", "float", "native", "super", "while");

 private static boolean isValidMethodName(final String methodName) {
  return BeanShellHelper.isSaveName(methodName) && !RESERVED_KEYWORDS.contains(methodName);
 }

 @Override
 public String getSaveVariableName(final String variableName) {
  return BeanShellHelper.getSaveName(variableName);
 }

 @Override
 public String createFunctionWrapper(final IEnvironment environment, final String moduleVariable, final Method method) {

  final StringBuilder scriptCode = new StringBuilder();

  // parse parameters
  final List<Parameter> parameters = ModuleHelper.getParameters(method);

  // build parameter string
  final StringBuilder parameterList = new StringBuilder();
  for (final Parameter parameter : parameters)
   parameterList.append(", ").append(parameter.getName());

  if (parameterList.length() > 2)
   parameterList.delete(0, 2);

  final StringBuilder body = new StringBuilder();

  // insert hooked pre execution code
  body.append(getPreExecutionCode(environment, method));

  // insert deprecation warnings
  if (ModuleHelper.isDeprecated(method))
   body.append("\tprintError('" + method.getName() + "() is deprecated. Consider updating your code.');\n");

  // insert method call
  body.append("\t ");
  if (!method.getReturnType().equals(Void.TYPE))
   body.append(IScriptFunctionModifier.RESULT_NAME).append(" = ");

  body.append(moduleVariable).append('.').append(method.getName()).append('(');
  body.append(parameterList);
  body.append(");\n");

  // insert hooked post execution code
  body.append(getPostExecutionCode(environment, method));

  // insert return statement
  if (!method.getReturnType().equals(Void.TYPE))
   body.append("\treturn ").append(IScriptFunctionModifier.RESULT_NAME).append(";\n");

  // build function declarations
  for (final String name : getMethodNames(method)) {
   if (!isValidMethodName(name)) {
    Logger.error(IPluginConstants.PLUGIN_ID,
      "The method name \"" + name + "\" from the module \"" + moduleVariable + "\" can not be wrapped because it's name is reserved");

   } else if (!name.isEmpty()) {
    scriptCode.append(name).append("(").append(parameterList).append(") {\n");
    scriptCode.append(body);
    scriptCode.append("}\n");
   }
  }

  return scriptCode.toString();
 }

 @Override
 public String classInstantiation(final Class<?> clazz, final String[] parameters) {
  final StringBuilder code = new StringBuilder();
  code.append("new ").append(clazz.getName()).append("(");

  if (parameters != null) {
   for (final String parameter : parameters)
    code.append(parameter).append(", ");

   if (parameters.length > 0)
    code.delete(code.length() - 2, code.length());
  }

  code.append(")");

  return code.toString();
 }

 @Override
 public String createFinalFieldWrapper(final IEnvironment environment, final String moduleVariable, final Field field) {
  return getSaveVariableName(field.getName()) + " = " + moduleVariable + "." + field.getName() + ";\n";
 }

 @Override
 protected String getNullString() {
  return "null";
 }
}
createFunctionWrapper() is called for each exposed method, createFinalFieldWrapper() is called for each exposed final field. The function wrapper injects preExecutionCode and postExecutionCode. This is interesting for modules implementing IScriptFunctionModifier. If such a module is loaded, it may influence code generation for all other modules. Use cases are creation of log files or unit testing.

Step 2: Register code factory

Now we need to register our code factory, so EASE is able to find it. Open the Extensions tab of org.eclipse.ease.lang.beanshell/plugin.xml. Navigate to the scriptType extension and add the code factory class to the BeanShell type.
Step 3: Bootstrapping

There exists a dedicated Environment module that provides basic commands like loadModule() to load further modules. Lets try to load it in a BeanShell:

run EASE and open a bean shell in the Script Shell View. We will create an instance of EnvironmentModule and advise it to load itself:
new org.eclipse.ease.modules.EnvironmentModule().loadModule("/System/Environment");
If your code factory works as expected, commands like print() and loadModule() should be available.

To automate this task we will add a launch extension to our BeanShell language definition.
Switch to the plugin.xml and add a launchExtension to the org.eclipse.ease.language node.We bind it to the engineID of BeanShell and need to provide an implementation:
public class BootStrapper implements IScriptEngineLaunchExtension {

 @Override
 public void createEngine(final IScriptEngine engine) {
  ICodeFactory codeFactory = ScriptService.getCodeFactory(engine);
  if (codeFactory != null) {
   StringBuilder stringBuilder = new StringBuilder();
   stringBuilder.append(codeFactory.classInstantiation(EnvironmentModule.class, new String[0]));
   stringBuilder.append(".loadModule(\"");
   stringBuilder.append(EnvironmentModule.MODULE_NAME);
   stringBuilder.append("\");\n");

   engine.executeAsync(new Script("Bootloader", stringBuilder.toString()));
  }
 }
}
Of course we could directly use the string we tried manually before, but using the code factory is somewhat nicer. The implementation given above is already available from the BootStrapper class from EASE. If you do want to add more stuff to the bootloader you need to provide your own implementation.

With that in place you should be able to load all available modules into BeanShell and use its functions.

Optional: Loading external jars

We added support to register URLs to the classloader in our previous tutorial. Now as we have modules working we may try this out by calling
loadJar("http://central.maven.org/maven2/com/github/lalyos/jfiglet/0.0.7/jfiglet-0.0.7.jar")
 true
com.github.lalyos.jfiglet.FigletFont.convertOneLine("I      love      scripting");
  ___        _                                             _         _    _               
 |_ _|      | |  ___  __   __  ___        ___   ___  _ __ (_) _ __  | |_ (_) _ __    __ _ 
  | |       | | / _ \ \ \ / / / _ \      / __| / __|| '__|| || '_ \ | __|| || '_ \  / _` |
  | |       | || (_) | \ V / |  __/      \__ \| (__ | |   | || |_) || |_ | || | | || (_| |
 |___|      |_| \___/   \_/   \___|      |___/ \___||_|   |_|| .__/  \__||_||_| |_| \__, |
                                                             |_|                    |___/ 


Tuesday, December 15, 2015

A new interpreter for EASE (3): Loading native classes

In this tutorial we will teach our interpreter to load all kinds of classes from Eclipse and dedicated URLs. So we will have a closer look on the classloader, but don't worry - it will not get too complicated.

Our BeanShell interpreter is already capable of calling java code, so we only need to make sure that its classloader will have access to all desired resources.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: Access java classes from eclipse

Our BeanShell interpreter already accepts calls to JRE classes like
java.lang.System.out.println("Hello world");
However if we try to access eclipse classes like
org.eclipse.jface.resource.JFaceColors
    Sourced file: inline evaluation of: ``org.eclipse.jface.resource.JFaceColors;'' :
    Class or variable not found: org.eclipse.jface.resource.JFaceColors
We get an exception indicating that the class cannot be found. This is a classloader issue as the interpreter uses the bundle classloader from our org.eclipse.ease.lang.beanshell bundle. We could add dependencies to jface to solve the problem, but how would we introduce dependencies to all plug-ins that might be used by a customer?

Fortunately eclipse has a solution for this. We may allow plug-ins to access classes from all exported packages out there without  adding a dedicated dependency by defining
Eclipse-BuddyPolicy: global
in the MANIFEST.MF file. This comes with a penalty on class loading performance but allows to access all available classes.

Add this setting to the manifest of org.eclipse.ease.lang.beanshell. Afterwards switch to the BeanShellEngine and add following line to setupEngine():
fInterpreter.setClassLoader(getClass().getClassLoader());
This line changes the classloader of the BeanShell with the one from the org.eclipse.ease.lang.beanshell bundle.

Now go and try to load some eclipse classes:
org.eclipse.jface.resource.JFaceColors
    Class Identifier: org.eclipse.jface.resource.JFaceColors

Step 2: Load classes from external URLs

EASE allows to register external jar files and make them available to scripting. This mechanism  is provided by a module - a concept we will introduce in the next tutorial. For now we will make all necessary preparations. EASE provides a generic classloader with all necessary functionality. Go to your BeanShellEngine and modify the code as follows:
public class BeanShellEngine extends AbstractScriptEngine {

 private DelegatingJarClassLoader fClassLoader;

 @Override
 protected boolean setupEngine() {
  fInterpreter = new Interpreter();

  fClassLoader = new DelegatingJarClassLoader(getClass().getClassLoader());
  fInterpreter.setClassLoader(fClassLoader);

  [...]
 }

 @Override
 public void registerJar(final URL url) {
  if (fClassLoader != null)
   fClassLoader.registerURL(url);
 }
}
The DelegatingClassLoader will take care of registered URLs. For everything else its parent classloader will be used. We cannot test this currently, so you need to trust me on this until the next tutorial.

Friday, December 11, 2015

A new interpreter for EASE (2): Shell and script integration

Today we will focus on integrating BeanShell into the script shell and to allow to execute .bsh files.

To follow this tutorial you will either need to install EASE into your development IDE or even better create a target platform containing EASE core plugins. I expect that you are familiar with that process.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: A very basic EASE interpreter

Create a new Plug-in Project called org.eclipse.ease.lang.beanshell. Add dependencies to org.eclipse.ease and to our org.beanshell plug-in.

Switch to the Extensions tab and add a new extension point for org.eclipse.ease.language. Create an engine, provide an ID, a nice name and a link to the implementing class.

Instead of implementing the IScriptEngine interface directly you may inherit from AbstractScriptEngine:
package org.eclipse.ease.lang.beanshell;

import org.eclipse.ease.AbstractScriptEngine;
import org.eclipse.ease.Script;

import bsh.Interpreter;

public class BeanShellEngine extends AbstractScriptEngine {

 private Interpreter fInterpreter = null;

 public BeanShellEngine() {
  super("BeanShell");
 }

 @Override
 protected boolean setupEngine() {
  fInterpreter = new Interpreter();

  fInterpreter.setOut(getOutputStream());
  fInterpreter.setErr(getErrorStream());

  return true;
 }

 @Override
 protected boolean teardownEngine() {
  fInterpreter = null;

  return true;
 }

 @Override
 protected Object execute(final Script script, final Object reference, final String fileName, final boolean uiThread) throws Throwable {
  return fInterpreter.eval(script.getCode());
 }
}
Use the QuickFix to Add unimplemented methods.

Launch a new RCP application adding org.eclipse.ease.* and org.beanshell plugins to your run configuration.

Now open the Script Shell view and use the engine selection to switch to BeanShell. Your console name should change to BeanShellScript Shell and you should be able to enter and execute some beanshell commands.

Step 2: Adding launch target support

To launch files with an engine we need to bind our engine to dedicated content types. That means that we first need a content type for BeanShell files. Open the Plug-in Manifest Editor, switch to the Extensions tab and add a new extension for org.eclipse.contenttype.contentTypes. Add a new content-type based on org.eclipse.core.runtime.text and set file-extensions to bsh.

Now create a new extension for org.eclipse.ease.scriptType. Set name to BeanShell and the defaultExtension to bsh.  Afterwards add a binding to our previously created content type.

Finally switch to your engine definition and create a binding to the scriptType.

When launching you may now run .bsh files from your workspace using the EASE launch target (eg by using Run As/EASE Script from the context menu).

Step 3: Add variables support

Getting and setting variables improves the Script Shell experience but is also needed for modules support, which we investigate in a following tutorial.

Open your BeanShellEngine and replace your default methods for variables with these:
 @Override
 protected Object internalGetVariable(final String name) {
  try {
   return fInterpreter.get(name);
  } catch (EvalError e) {
   Logger.error("org.eclipse.ease.lang.beanshell", "Cannot retrieve variable \"" + name + "\"", e);
  }

  return null;
 }

 @Override
 protected Map<String, Object> internalGetVariables() {
  Map<String, Object> variables = new HashMap<String, Object>();

  for (Variable variable : fInterpreter.getNameSpace().getDeclaredVariables())
   variables.put(variable.getName(), internalGetVariable(variable.getName()));

  return variables;
 }

 @Override
 protected boolean internalHasVariable(final String name) {
  for (Variable variable : fInterpreter.getNameSpace().getDeclaredVariables()) {
   if (variable.getName().equals(name))
    return true;
  }

  return false;
 }

 @Override
 protected void internalSetVariable(final String name, final Object content) {
  try {
   fInterpreter.set(name, content);
  } catch (EvalError e) {
   Logger.error("org.eclipse.ease.lang.beanshell", "Cannot set variable \"" + name + "\"", e);
  }
 }

 @Override
 protected Object internalRemoveVariable(final String name) {
  Object content = internalGetVariable(name);
  fInterpreter.getNameSpace().unsetVariable(name);

  return content;
 }

 @Override
 public String getSaveVariableName(final String name) {
  return BeanShellHelper.getSaveName(name);
 }
We purely dig into the beanshell interpreter and investigate its namespace. The BeanShellHelper class is copied from the JavaScriptHelper and provides basic checks on variable names.

Once launched, the Script Shell will show a list of all available variables in your interpreter attached in the container right to the shell.