So far, I have worked with traditional web frameworks such as Selenium, Cucumber, and Cypress and there are abundant resources that helped me get started.

In our case, we are building an IntelliJ plugin, and testing it end-to-end was challenging. I couldn’t find an easy-to-get-started guide that explained automated UI tests for IDE plugins. Hope that this post serves the purpose.

Unlogged has two components: the Unlogged SDK (this adds probes to the user code in compile time and logs the code execution) and an IntelliJ plugin. TLDR: The SDK writes the logs, and the IntelliJ plugin scans those logs.

Here is how the plugin works from a developer's point of view:

1. Developer identifies a method
2. Developer clicks on the gutter icon
3. Developer executes the method using Direct Invoke
4. Developer then goes to the next method
5. Developer then goes to the next Java file.

Although, I created a complete automation mimicking all the steps above, I want to focus only on automating the UI Interaction part of the IntelliJ UI interface.

Getting Started

I came across the Remote Robot tool from IntelliJ. This tool seems well-suited for IntelliJ plugins. It allows you to simulate user actions, such as key presses and mouse movements.

It goes a step further, enabling us to interact with IntelliJ's APIs, offering a way to mimic a real world use case. This coupled with reporting would automate manual testing for me.

To set up an IntelliJ plugin project for Remote Robot, include the following dependencies in your <span  class="pink" >build.gradle</span>:

 
dependencies {
    testImplementation 'com.intellij.remoterobot:remote-robot:' + remoteRobotVersion
    testImplementation 'com.intellij.remoterobot:remote-fixtures:' + remoteRobotVersion
}

Once your project has these dependencies, define test environment properties, such as the port:

 
runIdeForUiTests {
    systemProperty "robot-server.port", "8082"
}

Now, create an instance of Remote Robot, specifying the URL and port number:

 
RemoteRobot remoteRobot = new RemoteRobot("<http://127.0.0.1:8082>");

IntelliJ’s remote robot builds a robot client based on an okHttpClient, JavaScriptAPI and a LambdaAPI. The robot client connects to the robot server (http://127.0.0.1:8082) only when we try to interact with or find a component.

Now, I can interact with any visible component on IntelliJ's screen, with actions like mouse clicks based on component coordinates.

Note that you don’t need to work with coordinates, the remote robot does it for you.

To obtain a list of visible components in IntelliJ, open your browser and navigate to localhost:8082 after running the Gradle task <span  class="pink" >runIdeForUiTests</span> from your IntelliJ plugin project. This list displays all visible components rendered in the IntelliJ window.

Each component can be accessed through Fixture classes, which serve as access points for interacting with these components. Components are identified using XPath. It’s useful to have a unique XPath for major components you wish to interact with, like prominent buttons on your plugin's UI.

Here’s a snapshot of the components visible on an instance of IDEA.

This is what the idea window looked like when the above screenshot was taken.

The list of components are massive and this screenshot doesn’t cover all the components visible on the IntelliJ window.

You can fetch a Fixture class for a specific component using the <span  class="teal" >find</span> method. Some fixtures, such as the main idea frame, should be organized into their own classes, extending <span  class="pink" >CommonContainerFixture</span> to simplify interactions.

There are two major types of Fixture classes:

• <span  class="pink" >ComponentFixture</span>: for individual components

• <span  class="pink" >ContainerFixture</span>: for containers that encompass multiple components

Here's an example of an <span  class="pink" >IdeaFrame</span> class pointing to the main IntelliJ idea panel:

 
import com.intellij.remoterobot.RemoteRobot;
import com.intellij.remoterobot.data.RemoteComponent;
import com.intellij.remoterobot.fixtures.*;
import org.jetbrains.annotations.NotNull;

@FixtureName(name = "Idea frame")
@DefaultXpath(by = "IdeFrameImpl type", xpath = "//div[@class='IdeFrameImpl']")
public class IdeaFrame extends CommonContainerFixture {

    private RemoteRobot remoteRobot;

    public IdeaFrame(@NotNull RemoteRobot remoteRobot, @NotNull RemoteComponent remoteComponent) {
        super(remoteRobot, remoteComponent);
        this.remoteRobot = remoteRobot;
    }
}

Fixture classes offer two methods that use the underlying JavaScriptAPI:

• <span  class="pink" >runJs</span>: for executing JavaScript code

• <span  class="pink" >callJs</span>: for executing JavaScript code and returning a value

These methods enable interactions with IntelliJ's underlying APIs, such as the <span  class="teal" >Project</span> API. For example, you can use them to determine if the IDE is in "dumb" mode, a critical aspect for some testing scenarios.

Dumb mode is a state where IDE doesn’t have all its features available, this happens when the project is still indexing. Once indexed, you’ll have all of the IDE’s features at your disposal.

Here's an example of how to use <span  class="pink"> callJs</span> to check if the IDE is in "dumb" mode:

 
 public boolean isDumbMode() {
    return callJs(
        """
        const frameHelper = com.intellij.openapi.wm.impl.ProjectFrameHelper.getFrameHelper(component)
        if (frameHelper) {
            const project = frameHelper.getProject()
            project ? com.intellij.openapi.project.DumbService.isDumb(project) : true
        } else { true }
        """, true);
}
 
 

With a class representing the main IDE window, such as <span  class="pink">IdeaFrame</span>, you can create tests to open specific files and perform actions. These tests can be executed using JUnit 5.

For instance, here's a simple test that opens a file and clicks the debug button:

 
public class RemoteRobotTests {
    private RemoteRobot remoteRobot = new RemoteRobot("");
    private final Keyboard keyboard = new Keyboard(remoteRobot);

    @Test
    public void debugTest() {
        final IdeaFrame idea = remoteRobot.find(IdeaFrame.class, ofSeconds(10));
        waitFor(ofMinutes(2), () -> !idea.isDumbMode());
        String fileName = "UserController";

        keyboard.hotKey(VK_SHIFT, VK_META, VK_O);
        keyboard.enterText(fileName);
        keyboard.hotKey(VK_ENTER);
        idea.getDebugButton().click();
    }
}

This test waits for the IdeaFrame, ensures that the IDE is not in "dumb" mode, opens a specific file (UserController in this case), and clicks the debug button.

To interact with the debug button, a <span  class="pink">ComponentFixture</span> is required. You can obtain it using a method like this:

 
public ComponentFixture getDebugButton() {
    return remoteRobot.find(ComponentFixture.class, byXpath("//div[@myicon='startDebugger.svg']"));
}

To obtain the Xpath to identify the debug button, you can fetch it from the browser end-point:

Now, you're ready to run this test. Once executed, you will see the Remote Robot opening a specific file and clicking the debug button in your IntelliJ instance.

Steps and Pause

For better organization and to introduce pauses in your tests, Remote Robot provides two handy methods: <span  class="pink">step</span> and <span  class="pink">pause</span>.

Use <span  class="pink">step</span> to organize sequences of code. For instance, you can split the code into two steps: opening a file and clicking the debug button.

 
step("Open file", () -> {
    keyboard.hotKey(VK_SHIFT, VK_META, VK_O);
    keyboard.enterText(fileName);
    keyboard.hotKey(VK_ENTER);
});
step("Click Debug button", () -> {
    idea.getDebugButton().click();
});

To introduce pauses at specific points in your automation, the <span  class="pink">pause</span> method is valuable. For example, to pause for one second:

 
pause(ofSeconds(1).toMillis());

With IntelliJ's Remote Robot, you're not limited to button clicks; you can script interactions with Java files and methods within your project.

What does our Test Automation look like?

This is a demonstration of our automation, in this case every java file gets visited and every testable method marked with a gutter icon gets directInvoked with autogenerated inputs.

Here’s what the test looks like at a high level :

<span  class="pink">projectTreeView</span> gives us the list of all files seen on the project tree view.

 
List elements = idea.getProjectViewTree().getData().getAll();

This lists only what you can see on the IntelliJ screen.

Double clicking on an instance of <span  class="pink">RemoteText</span> will open that file.

 
element.doubleClick();

The gutter icons you see on the method line numbers - are the starting point of Unlogged’s user journey. I wanted a program to click on this icon and open the panel first.

Here is how to get a list of gutter icons.

 
TextEditorFixture editor = idea.textEditor(Duration.ofSeconds(2));
List icons = editor.getGutter().getIcons();

In our case, I created a filter to click only on the “Process Running” icon.

Icons can be filtered based on many factors, here I’ve used the name of the image.

 
icon.toString().contains("process_running.svg")

The line number where the gutter icon is rendered can be obtained using <span  class="pink">GutterIcon</span> class.

With <span  class="pink">TextEditorFixture</span>scrolling to any point in the editor but this is caret-based.

To scroll down to the correct point in the editor, the caret offset pointing to the start of the line that the gutter icon is rendered on is needed. This is possible with the help of a convenient method in the document model of the editor.

 
private void scrollDownToIcon(TextEditorFixture editorFixture, GutterIcon icon) {
        int startingOffset = editorFixture.getEditor().callJs("local.get('editor').getDocument().getLineStartOffset(" + icon.getLineNumber() + ")", true);
        editorFixture.getEditor().scrollToOffset(startingOffset);
    }

Once the editor has scrolled so that the icon is in view next is to

1. Click on it.
2. Open Unlogged’s toolbar.
3. Invoke the selected method.

The Fixture for the “Execute Method” button is obtained the same way as seen with the debug button from the above example.

Once a method is executed, it may take sometime to respond depending on what it is doing. I added an artificial delay of 5 seconds so that the method completes its execution and the input/return values + CPU usage can be added to the report file.

Report Generation

Our Automation testing service is paired with a reporting service that records the inputs and return values of methods every time the methods are executed along with information like CPU and memory usage to monitor performance and visual/functional bugs.

Voilà! We found bugs with our tests

I want to highlight 2 issues found using this automation.

1. Incorrect Rendering of Gutter Icons.‍

The correct gutter icon state was not rendered when multiple IntelliJ idea instances were open at once.

2. Performance Drops on longer sessions.

Longer running applications means longer sessions both in storage and duration.

The time taken to update a gutter icon post-execution and the average time taken for the same methods to display responses increased by as much as 40% when the sessions were longer.

More info and examples Here

‍

‍