vortex: Add CI, rust driver #3017

brson · 2025-06-09T00:05:44Z

Here are a bunch of changes to get vortex moving again.

Background: vortex runs as a group of processes, one of which is a language-specific driver. There is a built-in driver for zig, a java driver, and this PR adds a rust driver. Vortex really wants to be run in its own linux namespace so that it controls the entire network, and theoretically I think so that all processes can be torn down automatically when the supervisor exits. As such it only really works correctly under linux.

This PR:

adds a rust driver
adds java/rust vortex smoke tests to the existing client ci infra
adds a run command that sets up the linux namespace, previously not handled directly by vortex

It also makes various small tweaks to accomplish the above.

The reduction of the JDK minimum version is because my development machine has that particular JDK. Seems to work fine.

This is useful for smoke tests and development.

brson · 2025-06-09T00:17:34Z

The ubuntu runnner failure is because ubuntu needs a system-level configuration change to support the unshare command (other linuxes don't). I'll figure out the best place to do that later.

src/clients/rust/src/conversions.rs

src/testing/vortex/faulty_network.zig

src/testing/vortex/java_driver/ci.zig

sentientwaffle · 2025-06-10T18:04:56Z

src/testing/vortex/run.zig

+
+const log = std.log.scoped(.run);
+
+pub fn main(allocator: std.mem.Allocator) !void {


Instead of shelling out to unshare, could the supervisor just run the appropriate std.os.linux.unshare() syscalls directly to set up the new namespaces? Then we don't need a separate binary/process to handle it.

(Now that I think of it, the same approach would probably be useful for the cfo supervisor. That is out of scope here though.)

Done.

I also implemented ip link set up dev lo as netlink commands.

src/testing/vortex/rust_driver/src/main.rs

brson · 2025-08-10T21:40:41Z

Some reviews done, a few left.

brson · 2025-08-10T22:40:28Z

@sentientwaffle all reviews addressed.

brson · 2025-08-10T23:25:30Z

CI failing because the unshare implementation is not working, can't open /proc/self/setgroups

+- run scripts failure
rust ci: 5m41.413s
2025-08-10 22:44:30.858Z error: process failed with ExecNonZeroExitStatus: ../../../../zig-out/bin/vortex run --driver-command=../../../../src/testing/vortex/rust_driver/target/debug/vortex-driver-rust --tigerbeetle-executable=../../../../zig-out/bin/tigerbeetle --output-directory=/home/runner/work/tigerbeetle/tigerbeetle/.zig-cache/tmp/8530484706316026764 --disable-faults --test-duration-seconds=10
2025-08-10 22:44:30.858Z error: stderr:
++++
2025-08-10 22:44:30.669Z error(run): Failed to open /proc/self/setgroups: error.AccessDenied

brson · 2025-08-13T20:10:45Z

Neither the hand-coded unshare implementation nor just calling unshare work on the CI for slightly different access-denied reasons. It seems like there's some capability that the processes in the CI container don't have.

brson · 2025-08-13T22:33:37Z

I think unshare is working with CI now. Problem now is that the java vortex driver doesn't build with JDK 11. I'll just rewrite it so it does.

brson · 2025-08-14T04:43:22Z

I converted the vortex driver to Java 11 to be compatible with the CI JDK. The resulting code is sadly much more verbose by not having record syntax.

I think this is all ready to go.

sentientwaffle

Mostly LGTM, modulo a few nits and questions!

src/testing/vortex/faulty_network.zig

sentientwaffle · 2025-08-18T16:01:38Z

src/testing/vortex/java_driver/pom.xml

-    <maven.compiler.source>21</maven.compiler.source>
-    <maven.compiler.target>21</maven.compiler.target>


Why do we need to use such an old jvm?

I can not find that information other than CI tests JDK 11.

sentientwaffle · 2025-08-18T16:20:39Z

src/testing/vortex/run.zig

+    // Run the supervisor
+    var child = std.process.Child.init(argv.items, allocator);
+    child.stdin_behavior = .Inherit;
+    child.stdout_behavior = .Inherit;
+    child.stderr_behavior = .Inherit;
+
+    try child.spawn();
+    const result = try child.wait();
+    std.process.exit(@intFromEnum(result));


Out of scope, but I'm curious -- do unshare/ip_link_loopback affect this current process, or only the child process? e.g. is it strictly required that the supervisor be a child process here?

Some things unshare does affect the current process, some only take effect on the child process.

The child process is required here to make the process reaping behavior work. With the new pid namespace, the child supervisor process will become the special pid 1 and we'll get the benefit of the behavior where every other process dies when the supervisor dies.

This does lead to the question of what happen when the parent "run" process (not pid 1) is killed - is the supervisor not killed? I have not investigated closely nor noticed any annoying behavior.

src/testing/vortex/run.zig

sentientwaffle · 2025-08-18T16:54:24Z

src/testing/vortex/run.zig

+    // Create user namespace first
+    const unshare_user_result = std.os.linux.unshare(linux.CLONE.NEWUSER);
+    const unshare_user_errno = std.posix.errno(unshare_user_result);
+    if (unshare_user_errno != .SUCCESS) {
+        log.err("Failed to create user namespace: {}", .{unshare_user_errno});
+        return error.UnshareFailure;
+    }
+
+    // Create PID namespace
+    const unshare_pid_result = std.os.linux.unshare(linux.CLONE.NEWPID);
+    const unshare_pid_errno = std.posix.errno(unshare_pid_result);
+    if (unshare_pid_errno != .SUCCESS) {
+        log.err("Failed to create pid namespace: {}", .{unshare_pid_errno});
+        return error.UnshareFailure;
+    }
+
+    // Create network namespace
+    const unshare_net_result = std.os.linux.unshare(linux.CLONE.NEWNET);
+    const unshare_net_errno = std.posix.errno(unshare_net_result);
+    if (unshare_net_errno != .SUCCESS) {
+        log.err("Failed to create net namespace: {}", .{unshare_net_errno});
+        return error.UnshareFailure;
+    }


Nice -- I guess the unshare calls could be combined, but the more detailed error messages make splitting them up worth it!

Exactly yes. And I think there's some technical reason the user namespace has to be created separately anyway.

src/testing/vortex/rust_driver/ci.zig

sentientwaffle · 2025-08-18T17:24:08Z

src/testing/vortex/java_driver/README.md

Could the example test command be swapped with a simpler vortex run command?

I've updated the readme.

I will write better vortex docs once it comes together a bit more. Right now it has the big problem that the ability to build and configure individual drivers is spread across several places. In a future patch I will make it easier to use vortex manually: putting the driver build steps into the tigerbeetle build system, and probably also putting some kind of run-vortex:java commands into the build system.

sentientwaffle · 2025-08-18T20:45:23Z

src/testing/vortex/rust_driver/src/main.rs

+                }
+                Ok(Some(Request::LookupTransfers(events)))
+            }
+            _ => todo!("{op}"),


Does this actually print the op variable, or just the string "{op}"?

Yeah it is a formatting string interpolation that captures op.

Co-authored-by: djg <sentientwaffle@gmail.com>

brson · 2025-08-20T18:41:00Z

@sentientwaffle I think I've addressed all new reviews.

brson added 5 commits June 8, 2025 18:18

vortex: Define ports at comptime

b1a4e67

vortex: Make replica-count a CLI argument

b80767c

vortex java driver: Target JDK 17

3314788

vortex java driver: Rename to 'vortex-driver-java'

d271cea

vortex: Change --test-duration-minutes to --test-duration-seconds

39b5974

This is useful for smoke tests and development.

brson assigned sentientwaffle Jun 9, 2025

brson added 2 commits June 8, 2025 20:16

vortex: Add run command

7c19fc4

ci: Add java vortex driver

17d8a57

brson force-pushed the vortex-1 branch from 28bc792 to 8b34881 Compare June 9, 2025 00:16

vortex: Add Rust driver

36912fd

brson force-pushed the vortex-1 branch from 8b34881 to 36912fd Compare June 9, 2025 02:25

sentientwaffle reviewed Jun 11, 2025

View reviewed changes

brson added 8 commits July 14, 2025 13:54

rust client: Fix new clippy lint

2206b40

vortex rust driver: Fix new lint

96b9a2a

rust client: Fix new clippy lint

3481581

Merge remote-tracking branch 'origin/main' into vortex-1

4e969f1

vortex: Remove replica_count argument

1c84631

vortex: Use create_tmp_dir

2f6d1b1

vortex: warn if not testing vortex

d7a7cd5

vortex: Implement unshare and ip link directly

5839ba5

brson added 3 commits August 10, 2025 16:06

vortex rust driver: Use Request and Reply names

b9fdea0

vortex rust driver: results_len -> results_length

3c248b4

Merge remote-tracking branch 'origin/main' into vortex-1-next

b50a115

brson force-pushed the vortex-1 branch 2 times, most recently from 74693f7 to d947815 Compare August 13, 2025 20:43

brson force-pushed the vortex-1 branch 4 times, most recently from d59e8b3 to 12be0a3 Compare August 13, 2025 21:52

brson added 2 commits August 13, 2025 16:12

vortex: Minimize namespace setup

ece210c

ci: Set up 'clients' config to work with vortex

c2f9e6e

brson force-pushed the vortex-1 branch from 12be0a3 to 32886f8 Compare August 13, 2025 22:12

vortex java driver: Minimum compiler version 11

0bcbcfa

brson force-pushed the vortex-1 branch from 32886f8 to 0bcbcfa Compare August 13, 2025 22:24

brson added 2 commits August 13, 2025 22:07

vortex java driver: Convert to Java 11

e3ed38b

Merge branch 'main' into vortex-1

200b84b

sentientwaffle reviewed Aug 18, 2025

View reviewed changes

brson and others added 4 commits August 20, 2025 12:18

Update src/testing/vortex/run.zig

5046bf4

Co-authored-by: djg <sentientwaffle@gmail.com>

Update src/testing/vortex/rust_driver/ci.zig

6996bcb

Co-authored-by: djg <sentientwaffle@gmail.com>

vortex: Rework

020f728

vortex: Update java readme

6541fef

sentientwaffle approved these changes Aug 20, 2025

View reviewed changes

brson added this pull request to the merge queue Aug 20, 2025

Merged via the queue into tigerbeetle:main with commit 0321207 Aug 20, 2025
38 checks passed

brson deleted the vortex-1 branch August 20, 2025 19:24


		const log = std.log.scoped(.run);

		pub fn main(allocator: std.mem.Allocator) !void {

		<maven.compiler.source>21</maven.compiler.source>
		<maven.compiler.target>21</maven.compiler.target>

vortex: Add CI, rust driver #3017

vortex: Add CI, rust driver #3017

Uh oh!

Conversation

brson commented Jun 9, 2025

Uh oh!

brson commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brson commented Aug 10, 2025

Uh oh!

brson commented Aug 10, 2025

Uh oh!

brson commented Aug 10, 2025

Uh oh!

brson commented Aug 13, 2025

Uh oh!

brson commented Aug 13, 2025

Uh oh!

brson commented Aug 14, 2025

Uh oh!

sentientwaffle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brson Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brson commented Aug 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brson commented Jun 9, 2025 •

edited

Loading

brson Aug 20, 2025 •

edited

Loading