Add support for gradle resolver #1357

smocherla-brex · 2025-04-27T17:22:13Z

This attempts to add support for a gradle resolver to support usecases like #864 and #909 (downloading the relevant JVM artifact without having to specify the JVM specific one, aka rely on Gradle metadata to do its magic).

I borrowed some of the work from @shs96c's PR in #1360. It works as follows:

Sets up a fake gradle project with the build file templated using handlebars
Uses the gradle-tooling-api and also injects a plugin dynamically into the gradle classpath
The plugin runs the dependency resolution for runtimeClasspath configuration, collects the dependency graph, collects all the resolved artifacts (and requests additional things like javadoc, sources, poms). In this process, conflicts are also collected, and failures are reported.
The gradle tooling api with ToolingModelBuilder requires serialization with interfaces, so there's a bit of boilerplate I had to add.
This supports BOMs, private repos and those tests pass.

Forgive me if my code is not idiomatic Java (it's been a while since I've worked with Java).

Checklist:

Added regression tests
Isolated gradle daemon and gradle cache per execution
Handle BOMs and pom files
Conflict resolution
Basic support for KMP artifacts (and support for gradle metadata) as illustrated by regression tests. I have not tested this with an android project though.
All tests pass

Follow-up work:

Use hermetic JDK - this unfortunately seems tricky. rules_java 8.x (presumably aligning with Bazel 8) has support for it https://github.com/bazelbuild/rules_java/pull/272/files#diff-485891a23eb8c524bc4bb0eb6670902e8ff2defaf7ce575924e7a11bb1213778R156 but we need to support Bazel 6/7 here still, so need to look into an alternative approach. It might be working already though implicitly

Process command line: /private/var/tmp/_bazel_smocherla/536b5c8af4e86fad57fc24f39380ac8a/external/rules_java~~toolchains~remotejdk11_macos_aarch64/zulu-11.jdk/Contents/Home/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.xml/javax.xml.namespace=ALL-UNNAMED -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005 -Dfile.encoding=UTF-8 -Duser.country=CA -Duser.language=en -Duser.variant -cp /private/var/tmp/_bazel_smocherla/536b5c8af4e86fad57fc24f39380ac8a/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tests/com/github/bazelbuild/rules_jvm_external/resolver/gradle/GradleResolverTest.runfiles/_main~_repo_rules~gradle/gradle-bin/lib/gradle-daemon-main-8.13.jar -javaagent:/private/var/tmp/_bazel_smocherla/536b5c8af4e86fad57fc24f39380ac8a/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/tests/com/github/bazelbuild/rules_jvm_external/resolver/gradle/GradleResolverTest.runfiles/_main~_repo_rules~gradle/gradle-bin/lib/agents/gradle-instrumentation-agent-8.13.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 8.13

Make tests faster (they're a bit slow now)
Create alias targets for KMP artifact that resolve to the actual target based on the selected platform (android, JVM...)

shs96c · 2025-05-08T17:31:14Z

This is great to see! Thank you!

@smocherla-brex, I've had a stalled branch to do this exact same thing for months. Would you like to work together on this? I've put up my WIP as #1360, but I'm more than happy to drop it in favour of your work. Please feel free to lift anything you think is useful from my WIP PR!

smocherla-brex · 2025-05-08T18:19:13Z

@shs96c Thanks for sharing your in-progress changes. I'm happy to help get this over the line. I'll take a look at your PR and see if I can leverage some of it in here later this week when I have cycles and hopefully get it in a place where it can be open for review.

smocherla-brex · 2025-05-20T00:25:38Z

Update:

I have a lot of the tests passing now. Some are still failing and need to fix - related to handling BOMs and Kotlin Multiplatform/gradle metadata artifacts.
The tests are slow because Gradle incurs 10-15 seconds in setup for every test case, so I had to bump the timeout (but need to make them faster).
Need to cleanup the code and add comments especially in the tooling API.

I'm hoping to get the above unblocked and the tests passing in the next few days and will open it up for review.

shs96c · 2025-05-21T10:13:57Z

@smocherla-brex, I really appreciate all the work you're doing on this. Thank you! When you're ready for review, just let us know. If you need a hand with anything, just ask :)

smocherla-brex · 2025-06-01T13:39:15Z

MODULE.bazel

+    name = "regression_testing_gradle",
+    artifacts = [
+        # https://github.com/bazel-contrib/rules_jvm_external/issues/909
+        "androidx.compose.foundation:foundation-layout:1.5.0-beta01",


Added a couple of regression tests for the issues linked above/below.

That's great. Thank you

smocherla-brex · 2025-06-01T13:41:09Z

tests/com/github/bazelbuild/rules_jvm_external/resolver/gradle/BUILD.bazel

+
+java_test(
+    name = "GradleResolverTest",
+    timeout = "eternal",


Increased the timeout here since the target takes about 5-6 minutes on my machine (seems to be longer about 8-9 minutes in CI) to complete as using an isolated instance of gradle per test and running things serially adds up quite a bit. We could look into test sharding if this is unacceptably slow.

Thanks for the explanation. I think it's fine to leave as-is

smocherla-brex · 2025-06-01T13:58:16Z

This should be ready for review now @shs96c. It's possible my gradle implementation may not be optimal (as I'm not a gradle expert) but all tests are passing and I also added some regression tests and added comments where relevant for context. There's one job failure in rules-jvm-external-examples but that seems potentially unrelated (kt_jvm_export) to this PR.

shs96c · 2025-06-01T16:01:04Z

This is fantastic. Thank you! I'll make sure to review this next week.

shs96c

This is great stuff. Thank you so much for working on it. The only real blocker is the comment about the exclusions being included in the handlebars context as a Stream rather than a collection.

shs96c · 2025-06-05T14:49:23Z

MODULE.bazel

+    name = "regression_testing_gradle",
+    artifacts = [
+        # https://github.com/bazel-contrib/rules_jvm_external/issues/909
+        "androidx.compose.foundation:foundation-layout:1.5.0-beta01",


That's great. Thank you

shs96c · 2025-06-05T14:49:45Z

MODULE.bazel

+    fail_if_repin_required = True,
+    generate_compat_repositories = True,
+    lock_file = "//tests/custom_maven_install:regression_testing_gradle_install.json",
+    repin_instructions = "Please run `REPIN=1 bazel run @regression_testing_gradle//:pin` to refresh the lock file.",


We shouldn't need this line.

shs96c · 2025-06-05T14:50:28Z

MODULE.bazel

+    "regression_testing_gradle",
    "regression_testing_maven",
    "unpinned_regression_testing_coursier",
+    "unpinned_regression_testing_gradle",


Nit: we shouldn't need to expose the unpinned_ version either. We used to need it for the :pin target, but that's no longer the case.

shs96c · 2025-06-05T14:51:23Z

private/extensions/maven.bzl


        # How do we do artifact resolution?
-        "resolver": attr.string(doc = "The resolver to use. Only honoured for the root module.", values = ["coursier", "maven"], default = _DEFAULT_RESOLVER),
+        "resolver": attr.string(doc = "The resolver to use. Only honoured for the root module.", values = ["coursier", "maven", "gradle"], default = _DEFAULT_RESOLVER),


shs96c · 2025-06-05T14:53:13Z

...ava/com/github/bazelbuild/rules_jvm_external/resolver/gradle/GradleBuildScriptGenerator.java

+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;


If you bazel run scripts:format this wild card import should be expanded properly.

shs96c · 2025-06-05T16:00:12Z

...ithub/bazelbuild/rules_jvm_external/resolver/gradle/plugin/GradleDependencyModelBuilder.java

+      List<GradleResolvedDependency> resolvedRoots,
+      Map<Coordinates, GradleResolvedDependency> coordinatesGradleResolvedDependencyMap,
+      List<GradleDependency> declaredDeps) {
+    ArtifactView sourcesView =


I had no idea about this mechanism for getting artifacts. Great stuff

shs96c · 2025-06-05T16:03:33Z

...ithub/bazelbuild/rules_jvm_external/resolver/gradle/plugin/GradleDependencyModelBuilder.java

+  }
+
+  private boolean isVerbose() {
+    return System.getenv("RJE_VERBOSE") != null && System.getenv("RJE_VERBOSE").equals("true");


Quite often people set RJE_VERBOSE to 1. It's enough that the env var isn't null

shs96c · 2025-06-05T16:05:01Z

private/tools/java/com/github/bazelbuild/rules_jvm_external/resolver/gradle/plugin/PomUtil.java

+public class PomUtil {
+  public static String extractPackagingFromPom(File pomFile) {
+    try {
+      DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();


I think this is fine for now, but experience suggests that we'll need to use the Maven classes for parsing poms. We can definitely leave that change for later.

shs96c · 2025-06-05T16:06:20Z

tests/com/github/bazelbuild/rules_jvm_external/resolver/gradle/BUILD.bazel

+
+java_test(
+    name = "GradleResolverTest",
+    timeout = "eternal",


Thanks for the explanation. I think it's fine to leave as-is

shs96c · 2025-06-05T16:06:58Z

tests/com/github/bazelbuild/rules_jvm_external/resolver/gradle/GradleResolverTest.java

+import com.github.bazelbuild.rules_jvm_external.resolver.netrc.Netrc;
+
+
+public class GradleResolverTest extends ResolverTestBase {


I am genuinely impressed that you managed to get all the tests passing. Very cool.

smocherla-brex · 2025-06-06T03:10:33Z

This is great stuff. Thank you so much for working on it. The only real blocker is the comment about the exclusions being included in the handlebars context as a Stream rather than a collection.

Thanks, updated the PR and addressed the feedback you shared. Also added a small update to the README about the new resolver.

smocherla-brex · 2025-06-15T09:29:32Z

I've been looking at improving performance of the resolver as it's extremely slow right now (on our monorepo, it takes 15 mins+ which isn't great). My hunch for one of the reasons (not everything) based on testing so far is that it's really slow right now due to the serial nature of artifact resolution in the plugin for various artifact views (javadocs, source jars) and additionally serially downloading POMs for every artifact rather than batching. I'm going to try make some of those resolutions async and see if it helps the performance.

…ache support

smocherla-brex · 2025-06-15T17:45:10Z

After looking further into the slowness, I've made a few optimizations to improve the resolver:

I removed the artifact views for javadocs/source jars which I added earlier. This actually isn't required (I assumed it was needed because a javadoc related test was failing at one point) as we fetch javadoc and sources explicitly in the Downloader after resolution. This gives a massive improvement as this also ballooned the dependency graph into thousands of nodes and resulted in unnecessary downloads. After removing this, the tests still pass. Building the dependency graph was taking 2 minutes+ as a result, and removing them brought it down to seconds on an example.
The Downloader has support for populating ~/.m2/repository currently with RJE_UNSAFE_CACHE. I also added support for it to populate the gradle cache. (While gradle supports m2local, it only appears to be for reading).

To illustrate the improvement, I tested the gradle resolver on the rules_jvm_external_deps_install.json in this repo before and after the improvements:

Before:

Currently: building lock file
RJE_VERBOSE=true REPIN=1 bazel run @rules_jvm_external_deps//:pin  168.25s user 5.99s system 30% cpu 9:39.62 total

It took nearly 9 minutes to build the lockfile even for the small repo here.

After (cold cache/no unsafe cache with removal of Javadoc/source jar resolution)

RJE_VERBOSE=true REPIN=1 bazel run @rules_jvm_external_deps//:pin  13.82s user 2.65s system 10% cpu 2:31.35 total

It takes around 2.5 minutes, which is pretty significant.

With warm/unsafe/gradle cache support, it is even faster

Currently: building lock file
RJE_UNSAFE_CACHE=1 RJE_VERBOSE=true REPIN=1 bazel run   7.76s user 0.82s system 29% cpu 29.441 total

It takes about 30 seconds in total.

Overall, I think that's a pretty healthy improvement and could be considered something close to usable for an experimental implementation. There's likely room for more improvement though, especially for larger repos..

shs96c · 2025-06-19T16:09:48Z

@smocherla-brex, I plan on merging this on Monday since it LGTM, but I'd like the other committers to also have a final chance to review. Thank you for working on this!

…into smocherla/gradle-resolver

shs96c

It's great to see this working. We've had enough time to review, and I think the right thing to do now is to land this, and then iterate on top of it as we find things that we need to tinker with. The other committers have had time to review this, so I think it's reasonable to land this now.

Thank you for putting this together, and thank you for focusing on improving the speed!

* master: Update maven-metadata.xml when publishing locally (bazel-contrib#1369) Add support for gradle resolver (bazel-contrib#1357) Prepare for 6.8 release (bazel-contrib#1380)

* master: (39 commits) Fix resolution of Android/AAR artifacts with Gradle resolver (bazel-contrib#1395) fail_if_repin_required is now True by default and minor improvement to failure message (bazel-contrib#1397) Housekeeping before we release 6.8 (bazel-contrib#1384) Add dll, dylib and so types to maven package mappings (bazel-contrib#1392) [bzlmod] Allow suppressing warning about multiple contributing modules. (bazel-contrib#1393) Use the artifact default values when adding them to a struct and add tests for coursier artifacts that have empty versions provided by BOMs and don't inlcude them in outdated (bazel-contrib#1390) Allow package exclusions and inclusions in javadocs (bazel-contrib#1293) Document well-known issues with `bzlmod` (bazel-contrib#1388) Begin documenting the gradle resolver (bazel-contrib#1389) Run gradle regression tests in CI (bazel-contrib#1385) Modify maven_export to allow exporting zip archives (bazel-contrib#1368) Allow root module's override tags to take precedence over the overridees from transitive deps. (bazel-contrib#1381) Ensure root module artifacts and boms take precedence with warnings (bazel-contrib#1373) Update maven-metadata.xml when publishing locally (bazel-contrib#1369) Add support for gradle resolver (bazel-contrib#1357) Prepare for 6.8 release (bazel-contrib#1380) Remove the Windows `kt_jvm_export` example (bazel-contrib#1383) Avoid spurious warnings about poorly formatted artifact coordinates (bazel-contrib#1374) Flip `fail_if_repin_required` to `True` by default (bazel-contrib#1371) Allow the same coordinate to be overridden in different repos (bazel-contrib#1378) ...

smocherla-brex added 7 commits April 26, 2025 15:58

Initial changes for gradle resolver

c02906d

Add a few more tests

f3156fe

Some more progress and improvements

2749929

Actually parse dependencies and add license notices

259b03f

Add an example and make some more progress

fca81a9

More progress + fixes

5a7b588

Remove bazelrc changes

5db4844

smocherla-brex added 10 commits May 13, 2025 00:32

Fix some tests and incorporate some changes from another draft PR

82343f2

Attempt to fix Bazel 6/WORKSPACE build

8ddbc65

Repin lock file

6b79bf9

Fix lockfile for both WORKSPACE and bzlmod

02eca16

Remove unused repositories

e15442a

Use a plugin with gradle tooling API

3fcd9b4

Use plugin fully, some more progress

2490007

Adapt all models to interfaces to work with tooling API

d6c7007

Get some tests working

8a0a4fd

Switch to ArtifactView API, fetch javadocs/sources explicitly

d2a9f9f

smocherla-brex changed the title ~~[WIP] Add support for gradle resolver~~ Add support for gradle resolver May 18, 2025

smocherla-brex added 7 commits May 18, 2025 16:30

Resolve classpath conflict due to jansi from gradle distribution

8bfefad

Fix comment

cbdbb11

Switch to groovy scripts to improve perf

1238432

Handle insecure maven repos for tests

cccd6e5

Fix some tests

0565d12

Fix some tests and reduce timeout

71263e3

Some cleanup and handle cycles

8d2edcf

Fix some more tests around POMs and conflicts

598dd62

smocherla-brex commented Jun 1, 2025

View reviewed changes

Attempt to fix tests on windows

c38de71

Do some minor code cleanup

57d37a8

shs96c requested changes Jun 5, 2025

View reviewed changes

smocherla-brex added 5 commits June 6, 2025 07:33

Formatting and address some of the feedback

8c0c006

Address some more feedback

63a7feb

Update README

79f8ca1

Address remaining feedback and get rid of wildcard imports

40e7fd7

Retrigger windows build

e2891dc

This was referenced Jun 10, 2025

Add a resolver backed by gradle #1360

Closed

Resolving Kotlin Multiplatform via Gradle Metadata (AndroidX) #864

Open

smocherla-brex mentioned this pull request Jun 11, 2025

Kotlin Bazel Cross-Platform Roadmap bazelbuild/rules_kotlin#567

Open

liutikas mentioned this pull request Jun 11, 2025

AndroidX considering dropping support for .pom files #1376

Open

smocherla-brex added 3 commits June 15, 2025 18:16

Parallelize artifact view resolution to improve performance

f521059

Batch retrieval of poms with detached configuration

18dabbe

perf: Remove artifact views for javadocs/source jars and add gradle c…

5d3740d

…ache support

smocherla-brex added 2 commits June 20, 2025 20:12

Merge branch 'master' of github.com:bazel-contrib/rules_jvm_external …

1a5b910

…into smocherla/gradle-resolver

Resolve merge conflict

c4d31b7

shs96c approved these changes Jun 23, 2025

View reviewed changes

shs96c merged commit e8c84f8 into bazel-contrib:master Jun 23, 2025
7 checks passed

smocherla-brex deleted the smocherla/gradle-resolver branch June 23, 2025 14:10

		import com.github.bazelbuild.rules_jvm_external.resolver.netrc.Netrc;


		public class GradleResolverTest extends ResolverTestBase {

Uh oh!

Add support for gradle resolver #1357

Add support for gradle resolver #1357

Uh oh!

Conversation

smocherla-brex commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shs96c commented May 8, 2025

Uh oh!

smocherla-brex commented May 8, 2025

Uh oh!

smocherla-brex commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shs96c commented May 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smocherla-brex Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smocherla-brex commented Jun 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shs96c commented Jun 1, 2025

Uh oh!

shs96c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smocherla-brex commented Jun 6, 2025

Uh oh!

smocherla-brex commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smocherla-brex commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shs96c commented Jun 19, 2025

Uh oh!

shs96c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

smocherla-brex commented Apr 27, 2025 •

edited

Loading

smocherla-brex commented May 20, 2025 •

edited

Loading

smocherla-brex Jun 1, 2025 •

edited

Loading

smocherla-brex commented Jun 1, 2025 •

edited

Loading

smocherla-brex commented Jun 15, 2025 •

edited

Loading

smocherla-brex commented Jun 15, 2025 •

edited

Loading