navigate_next

Java

How does the lombok magic work underneath?

Parth

October 4, 2023

Much has been already written about what lombok does and how it reduces boilerplate for java developers. This post is not about how to use lombok.

‍

At Unlogged, we extended the lombok implementation and were amazed to learn how flawlessly it works across different build systems, java versions and IDEs.

AnnotationProcessor introduced in J2SE 1.5 cannot make changes to existing files. It could only create new files or bytecode. This makes lombok’s implementation intriguing, since they use AnnotationProcessor to modify the existing java class files, during the compilation phase.

I want to explain how lombok achieves this feat, in detail.

‍

This briefly explains the several steps creating the lombok magic.

‍

Using ShadowClassLoader

Lombok uses this ant step called “mappedresources”and renames all .class files to .SCL.lombok - in short it hides the class files in a given jar by renaming it.

Why is it important?

Namespace Isolation

By renaming the .class files with the .SCL.lombok ending, Lombok ensures that it doesn't interfere with or "contaminate" the namespace of any project that uses a JAR file containing Lombok. This means that when you use Lombok in your project, it won't clutter your code suggestions or auto-complete features in your IDE with anything other than Lombok's actual public API.

Dependency Management

Similar to a tool called "jarjar," SCL allows Lombok to include its own dependencies (like ASM, a popular Java bytecode manipulation library) without imposing these dependencies on the projects that use the Lombok JAR. Lombok can use other libraries it needs to function correctly without forcing those libraries onto you when you use Lombok in your project.

Enhanced Debugging

SCL also plays a role in debugging. It allows an agent that must be packaged in a JAR file to load everything except the SCL infrastructure from class files generated by your IDE. This is beneficial because it makes debugging easier. You can rely on your IDE's built-in auto-recompile features instead of having to run a full build every time you make changes to your code. Additionally, it can help with features like hot code replacement, which allows you to modify code during debugging without restarting your application.

Assembling the jar with SCL

“mappedresources” is an ant step and I use maven for unlogged-sdk. So, I created a maven plugin that renames files based on regex just before the final jar is assembled.

Here is how it’s used.

 
 <plugin>
	<groupId>video.bug</groupId>
	<artifactId>rename-file-maven-plugin</artifactId>
	<version>1.0-SNAPSHOT</version>
	<executions>
		<execution>
			<phase>compile</phase>
			<goals>
				<goal>rename-file</goal>
			</goals>
		</execution>
	</executions>
	<configuration>
		<source>(.+).class</source>
		<target>$1.scl.unlogged</target>
		<workingDirectory>${build.directory}/classes/io/unlogged/processor</workingDirectory>
	</configuration>
</plugin>

You can read more about how class loaders work here.

Discovery of Annotation processors

When you compile your Java code, the process happens in several steps, or "rounds." In each round, the compiler might find new files that haven't been processed yet. It asks each annotation processor if it should do something with these files. The decision is based on another annotation called @SupportedAnnotationTypes. This annotation tells the processor which types of annotations it's interested in. It can even use a * to say it wants to handle all annotations, including those on classes that aren't annotated.

AnnotationProcessors primarily implement two methods, init() and process() . The first method, init() is called when the AnnotationProcessor is initialized and it receives an instance of ProcessingEnvironment as an argument. The second method, process() , is called for every round the java compiler finds a new set of files eligible to be processed (based on the supported annotation types earlier)

Lombok defines two annotation processors in its jar

 
lombok.launch.AnnotationProcessorHider$AnnotationProcessor
lombok.launch.AnnotationProcessorHider$ClaimingProcessor

I wasn’t very clear on the necessity of 2 processors especially when ClaimingProcessordidn’t seem to do anything except returning true, signaling the compiler that the set of annotations has been processed. But then the below comment inside the codebase makes it clear.

 
// Normally we rely on the claiming processor to claim away all lombok annotations
// One of the many Java9 oversights is that this 'process' API has not been
// fixed to address the point that 'files I want to look at' and 'annotations’
// I want to claim must be one and the same and yet in Java 9 you can no longer have
// 2 providers for the same service. Thus if you go by module path,
// lombok no longer loads the ClaimingProcessor.
// This doesn't do as good a job, but it'll have to do.
// The only way to go from here, I think, is either 2 modules, or use reflection
// hackery to add ClaimingProcessor during our init.

Debugging the compilation phase

If you want a hook inside the compilation process, place a lombok jar on your class path and here is how you can run the build process with remote debugging enabled.

 
MAVEN_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
mvn clean package

 
GRADLE_OPTS='-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005
./gradlew clean build

With the above parameters, the build process will pause (suspend=y) and start the remote debug server (server=y) at the port 5005. Now the debugger can connect to it.

In IntelliJ you can do this by creating a new Debug Profile of type “Remote JVM Debug”. Make sure the port number matches in your intellij debug config and the one you have used to start the process.

Once you have created that, click the debug button and you will be inside the javac compilation process in debug mode. Remember to put a breakpoint in the entry point of the annotation processor which is the init() method

Patching Java Compilation

Lombok needs to work with different compilers (javac/eclipse), and different build systems (maven/gradle) - so we will come across a lot of patching.

What’s the Java 9 Silly warning?

‍

Java 9 added warnings to prevent the code accessing other private parts of the code. Creators of lombok think that these warnings are unnecessary. So, disableJava9SillyWarning just suppresses the warning message that Java developers may otherwise see. The comment inside the code explains what the developer was thinking.

 
// JVM9 complains about using reflection to access packages from a module that 
// aren't exported. This makes no sense; the whole point of reflection
// is to get past such issues. The only comment from the jigsaw team lead on this
// was some unspecified mumbling about security which makes no sense,
// as the SecurityManager is invoked to check such things. Therefore this warning
// is a bug, so we shall patch java to fix it.

You may have come across the --add-opens and --add-exports arguments when running with certain libraries/java-agents which allows module A to access module B via reflection. This access was allowed to begin with, until java 1.9 when warnings/limitations were added for such access. You can read more about it in this very succinct answer on stackoverflow by Nicolai Parlog.

What does the process descriptor do?

Processor Descriptor is like a set of rules. It follows a design pattern called the "delegator pattern”. This pattern helps organize code by having different parts do different jobs.

Want() Method and Conditions

The Processor Descriptor has a method called want(). This method is like a decision maker and is called when the annotation processor starts.
If want() says "yes" (returns true), then the Processor Descriptor will be involved in the compilation process. If it says "no" (returns false), it won't be used.

There are two implementations of this method: EcjDescriptor and JavacDescriptor.

How Lombok Tries to Find It

Lombok tries a few different ways to find the real JavacProcessingEnvironment:

First, it looks for a field named "delegate" (for Gradle).
If that doesn't work, it looks for a field named "processingEnv" (for Kotlin).
If still no luck, it looks for a field named "val$delegateTo" (for IntelliJ IDEA version 2020.3 or later).
If any of these fields contain a non-null value, Lombok uses the same logic on that value until it finds the real JavacProcessingEnvironment.

If none of these attempts succeed, Lombok disables its processor because it can't find the necessary environment to work.

Finding and Patching the ClassLoader

This process is about finding and preparing a special component in Java called a "ClassLoader." The main goal is to load a specific AnnotationProcessor called "lombok.javac.apt.LombokProcessor."

Three classes play a role in this process:

AnnotationProcessorHider: This is loaded by the built-in ClassLoader and does some setup work.
AnnotationProcessor: This is loaded by a special ShadowClassLoader (SCL). It handles certain tasks and delegates work to other parts.
LombokProcessor: This is the actual AnnotationProcessor we want to use. It's loaded later after we get the ShadowClassLoader.

There's a special case when using the Plexus Apache Compiler, especially in the Eclipse IDE. In this case, Lombok uses a technique called "reflection" to add its own code to the ClassLoader. This allows Lombok to create an instance of LombokProcessor and set it up.

First, Lombok finds the ClassLoader it needs to use.
With this ClassLoader, it creates an instance of LombokProcessor.
Then, it calls the init() method of this processor to prepare it for its tasks.

LombokProcessor's Initialization

The real work of modifying Javac's fields happens inside the method.

Before diving in, we see a disable switch

 
if (System.getProperty("lombok.disable") != null) { lombokDisabled = true;
return;
}

The ConfigurationKeys java source itself carries a scary notice with this

"Disables lombok transformers. It does not flag any lombok mentions (so, @Cleanup silently does nothing), and does not disable patched operations in eclipse either. Don't use this unless you know what you're doing. (default: false).

I did come across this thread where the developer moved the lombok dependency from 500 sub-modules to a global parent module, ending up with a

 
java.lang.ClassNotFoundException:
com.sun.tools.javac.processing.JavacProcessingEnvironment.

And they resolved it by disabling lombok during tests.

 
<!-- disable lombok during tests -->
<lombok.disable>true</lombok.disable>

I couldn't make much sense of this. Disabling lombok should have resulted in non-compilable code (since the @Getters/@Setters are not effective resulting in missing methods). But we move on.

Getting a different JavacProcessingEnvironment

This part of the code is about obtaining another instance of the Javac ProcessingEnvironment, but with a slight difference in purpose.

Checking for Java Version

First, it checks if the Java version being used is 1.9 or above. It does this by trying to load a class named "java.lang.Module". If that class doesn't exist, it means it's not Java 1.9 or above, and it stops here.

Accessing the sun.misc.Unsafe Class

It then gets access to a class called "sun.misc.Unsafe". This class holds a special singleton instance named "theUnsafe". Note that there's a security check to ensure only trusted code can access this instance.

Purpose of Unsafe Operations

The code then does something seemingly strange: it gets the offset of a field named "first" in a class called "Parent" using the "theUnsafe" instance. This offset is then used to set a value on a "Method" instance. The purpose becomes clear when we look at the Method.invoke() method. The code sets the "override" field of the "Method" instance to "true". This field is used for access checks.

Why Setting "override" to True

In the Method.invoke() method, there's a check that fails if "override" is not true. This is part of access control. So, by setting "override" to true, the code ensures that it can access the necessary methods without access checks.

Invoking a Method for Modules:

Finally, the code invokes a method for 10 modules in a loop. This part is relatively straightforward.

This paper deep dives into the usage of unsafe class.

Getting Javac Filer

JavacFiler is responsible for creating files where the Java compiler stores its results. The JavacProcessingEnvironment can provide the JavacFiler, but sometimes it's wrapped in another proxy class. To ensure Lombok gets access to it, there's a getJavacFiler() method that uses similar tactics to what we saw earlier in getJavacProcessingEnvironment().

Post-Compile and No Force Round Dummies Hook: (unusually long name for a method!)

It creates an anonymous ClassLoader to prevent the original ClassLoader from being closed when JavacProcessingEnvironment finishes its work. The reason for this isn't entirely clear.
Force Multiple RoundsIn NetBeans Editor: This is specific to NetBeans and sets a value to indicate background compilation.
Disable Partial Reparse In NetBeans Editor: Again, NetBeans-specific, it seems to disable partial reparse of Java code.
Patching the JavaFileManager Context: Lombok wraps the JavaFileManager to intercept calls to getJavaFileForOutput. This is crucial because Lombok needs to apply bytecode transformations before the class file is written to disk.

Initializing Trees and JavacTransformer

Lombok proceeds to create an instance of com.sun.source.util.Trees and a JavacTransformer.
Trees class bridges different parts of Java processing, and JavacTransformer is a delegator pattern that passes information to actual handlers.

You can read more about Tree API, and pluggable annotation APIs, The Java Compiler API allows a Java program to select and invoke a Java Language Compiler programmatically. The interfaces abstract the way a compiler interacts with its environment.

Completing Initialization

The init() process for both LombokProcessor and the original AnnotationProcessor is now complete.
The Java compiler can start sending the Abstract Syntax Tree (AST) of source code to Lombok's annotation processor through the process() method.

Modifying the Java AST in-place

The in-place modification of the java code during compilation happens in three parts as we see in the call flow earlier. One is the AST based handlers like HandleFieldDefaults and most notably the HandleVal . The second one is annotation based and most handlers are of this second type. @Getter / @Setter / @Builder are all annotation based handler implementations.

Order of Transformations

Each transformation handler can specify a priority level.
The priority determines the order in which these handlers are executed during compilation.
For example, HandleVal specifies a priority slightly higher than HandleDelegate because it needs to work after certain code generation tasks.
LombokProcessor organizes these priorities and handlers, ensuring they are executed in the correct order during compilation rounds.

Here is how to set the priority when there is dependency between 2 handlers.

In HandleVal

 
@HandlerPriority(HANDLE_DELEGATE_PRIORITY + 100)

/*run slightly after HandleDelegate; resolution needs to work, so if the RHS expression is a call to a generated getter,  we have to run after that getter has been generated*/

public class HandleVal extends JavacASTAdapter {

And in HandleSynchronized we have

@HandlerPriority(value = 1024)

/*2^10; @NonNull must have run first, so that we wrap around the statements*/

public class HandleSynchronized extends JavacAnnotationHandler<Synchronized>

Execution of Handlers

During each compilation round, LombokProcessor identifies the next priority value and the associated handlers.
It then calls a JavacTransformer, which in turn invokes the handlers for that priority.
Handlers at the same priority level are executed in the order they were originally loaded.

AST Visitors and Annotation Handlers

It's a way to visit different parts of your code.
Lombok uses this pattern with the JavacASTVisitor interface.
When you visit a part of your code, you get an instance of JavacNode, which is a wrapper around the actual code (represented by JCTree).
JCTree is the core part of your code that Lombok wants to change or modify.

Modifying Code with Annotations

Let's say you have an annotation like @Setter. Lombok has a handler for it called HandleSetter. It adds a new method to your code's structure (AST).

If it's a constructor, it might also do some extra checks to remove an existing default constructor.

It takes care of updating the overall code structure.

Here is a code snippet from their code base.

 
public static void injectMethod(JavacNode typeNode, JCMethodDecl method) {

  JCClassDecl type = (JCClassDecl) typeNode.get();

  if (method.getName().contentEquals("<init>")) {

    //Scan for default constructor, and remove it.

    int idx = 0;

    for (JCTree def: type.defs) {

      if (def instanceof JCMethodDecl) {

        if ((((JCMethodDecl) def).mods.flags & Flags.GENERATEDCONSTR) != 0) {

          JavacNode tossMe = typeNode.getNodeFor(def);

          if (tossMe != null) tossMe.up().removeChild(tossMe);

          type.defs = addAllButOne(type.defs, idx);

          ClassSymbolMembersField.remove(type.sym, ((JCMethodDecl) def).sym);

          break;

        }

      }

      idx++;

    }

  }

  addSuppressWarningsAll(method.mods, typeNode, typeNode.getNodeFor(getGeneratedBy(method)), typeNode.getContext());

  addGenerated(method.mods, typeNode, typeNode.getNodeFor(getGeneratedBy(method)), typeNode.getContext());

  type.defs = type.defs.append(method);

  EnterReflect.memberEnter(method, typeNode);

  typeNode.add(method, Kind.METHOD);

}

PostCompiler Transformers

Bytecode-Level Transformations

Lombok does not just modify your Java source code; it also performs transformations at the bytecode level, but not too many. It uses objectweb.asm library for bytecode transformations.
One example of bytecode-level transformation is the "PreventNullAnalysisRemover". This transformer removes calls to the Lombok.preventNullAnalysis method from the generated bytecode.

Intercepting the Bytecode Writing Process

We saw that earlier, Lombok replaced the standard JavacFiler with its own "InterceptingFiler".

When it's time for the Java compiler to save the final bytecode to class files on disk (like when you compile your code), the InterceptingFiler steps in. It receives the bytecode, applies any necessary transformations (like removing calls to Lombok.preventNullAnalysis), and then writes the modified bytecode to the disk as expected.

Usage of Modified Class Files

Once the modified class files are on your disk, they are ready for use by the Java Virtual Machine (JVM) when you run your application in the future. These modified class files can also be packaged into a JAR file if needed.

‍

Parth

October 4, 2023