RFC: Accelerate xdiff and begin its rustification #1980

ezekielnewren · 2025-05-30T00:09:34Z

This series accelerates xdiff by 5-19%.

It also introduces Rust as a hard dependency.

…and it doesn’t yet pass a couple of the github workflows; hints from Windows experts, and opinions on ambiguous primitives would be appreciated (see below).

This is just the beginning of many patches that I have to convert portions of, maybe eventually all of, xdiff to Rust. While working on that conversion, I found several ways to clarify the code, along with some optimizations.

So...

This obviously raises the question of whether we are ready to accept a hard dependency on Rust. Previous discussions on the mailing list and at Git Merge 2024 have not answered that question. If not now, will we be willing to accept such a hard dependency later? And what route do we want to take to get there?

About the optimizations in this series:

1. xdiff currently uses DJB2a for hashing (even though it is not explicitly named as such). This is an older hashing algorithm, and modern alternatives are superior. I chose xxhash because it’s faster, more collision resistant, and designed to be a standard. Other hash algorithms like aHash, MurMurHash, SipHash, and Fnv1a were considered, but my local testing made me feel like xxhash was the best choice for usage in xdiff.

2. In support of switching to xxhash, parsing and hashing were split into separate steps. And it turns out that memchr() is faster for parsing than character-by-character iteration.

About the workflow builds/tests that aren’t working with this series:

1. Windows fails to build. I don’t know which rust toolchain is even correct for this or if multiple are needed.  Example failed build: https://github.com/git/git/actions/runs/16353209191

2. I386/ubuntu:focal will build, but fails the tests. The kernel reports the bitness as 64 despite the container being 32. I believe the issue is that C uses ambiguous primitives (which differ in size between platforms). The new code should use unambiguous primitives from Rust (u32, u64, etc.) rather than perpetuating ambiguous primitive types.  Since the current xdiff API hardcodes the ambiguous types, though, those places will need to be migrated to unambiguous primitives. Much of the C code needs a slight refactor to be compatible with the Rust FFI and usually requires converting ambiguous to unambiguous types. What does this community think of this approach?

My brother (Elijah, cc’ed) has been guiding and reviewing my work here.

cc: Elijah Newren newren@gmail.com
cc: "brian m. carlson" sandals@crustytoothpaste.net
cc: Taylor Blau me@ttaylorr.com
cc: Christian Brabandt cb@256bit.org
cc: Phillip Wood phillip.wood123@gmail.com
cc: Eli Schwartz eschwartz@gentoo.org
cc: "Haelwenn (lanodan) Monnier" contact@hacktivis.me
cc: Johannes Schindelin Johannes.Schindelin@gmx.de
cc: Matthias Aßhauer mha1993@live.de
cc: Patrick Steinhardt ps@pks.im
cc: Sam James sam@gentoo.org
cc: Collin Funk collin.funk1@gmail.com
cc: Mike Hommey mh@glandium.org
cc: Pierre-Emmanuel Patry pierre-emmanuel.patry@embecosm.com
cc: Ben Knoble ben.knoble@gmail.com

gitgitgadget-git · 2025-05-30T00:11:02Z

Welcome to GitGitGadget

Hi @ezekielnewren, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that either:

Your Pull Request has a good description, if it consists of multiple commits, as it will be used as cover letter.
Your Pull Request description is empty, if it consists of a single commit, as the commit message should be descriptive enough by itself.

You can CC potential reviewers by adding a footer to the PR description with the following syntax:

CC: Revi Ewer <revi.ewer@example.com>, Ill Takalook <ill.takalook@example.net>

NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description,
because it will result in a malformed CC list on the mailing list. See
example.

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

the lines should not exceed 76 columns,
the first line should be like a header and typically start with a prefix like "tests:" or "revisions:" to state which subsystem the change is about, and
the commit messages' body should be describing the "why?" of the change.
Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

Search: is:pr is:open "/allow"

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the Libera Chat IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this. Note that any reviewers CC'd via the list in the PR description will not actually be sent emails.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Libera Chat. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

gitgitgadget-git · 2025-05-30T00:11:03Z

There are issues in commit 2178ac1:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T20:57:42Z

There are issues in commit 2178ac1:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T20:57:43Z

There are issues in commit 67f6c92:
Random stab in the dark with Rust building in ci
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-05-30T21:14:16Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T22:08:28Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T23:40:20Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-31T01:23:04Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

dscho · 2025-06-01T15:50:47Z

/allow

gitgitgadget-git · 2025-06-01T15:51:45Z

User ezekielnewren is now allowed to use GitGitGadget.

WARNING: ezekielnewren has no public email address set on GitHub; GitGitGadget needs an email address to Cc: you on your contribution, so that you receive any feedback on the Git mailing list. Go to https://github.com/settings/profile to make your preferred email public to let GitGitGadget know which email address to use.

gitgitgadget-git · 2025-06-03T21:06:54Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T21:06:55Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T21:29:27Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T21:29:28Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T21:29:28Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:35Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T22:03:35Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:36Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:36Z

There are issues in commit c28361b:
.github/workflows/main.yml: use github actions to setup rust
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:35Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-05T18:19:36Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:36Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:37Z

There are issues in commit c28361b:
.github/workflows/main.yml: use github actions to setup rust
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:37Z

There are issues in commit 564a178:
Revert "meson.build: link libdl"
Commit not signed off

gitgitgadget-git · 2025-07-19T00:57:10Z

On the Git mailing list, "Haelwenn (lanodan) Monnier" wrote (reply to this):

[2025-07-18 17:25:01-0400] Eli Schwartz:
>On 7/18/25 9:34 AM, Phillip Wood wrote:
>> Hi Ezekiel
>>
>> Thanks for working on this
>>
>> On 17/07/2025 21:32, Ezekiel Newren via GitGitGadget wrote:
>>
>>> So...
>>>
>>> This obviously raises the question of whether we are ready to accept a
>>> hard
>>> dependency on Rust. Previous discussions on the mailing list and at Git
>>> Merge 2024 have not answered that question. If not now, will we be
>>> willing
>>> to accept such a hard dependency later? And what route do we want to
>>> take to
>>> get there?
>>
>> As far as git goes I think introducing a hard dependency on rust is
>> fine. It is widely supported, the only issue I'm aware of is the lack of
>> support on NonStop and I don't think it is reasonable for such a
>> minority platform to hold the rest of the project to ransom. There is a
>> question about the other users of the xdiff code though. libgit2 carries
>> a copy as do other projects like neovim. I've cc'd the libgit2
>> maintainer and posted a link to this thread in neovim github [1]
>
>
>A hard dependency on rust for Gentoo amd64 would potentially require
>building https://github.com/thepowersgang/mrustc followed by building 13
>and counting versions of rustc in order to get to the latest version.
>What is the minimum supported version in this series, by the way?
>
>bin packages for rust do exist but not everyone wants to use non-distro
>provided binaries, sometimes for auditability reasons.
>
>
>For Gentoo HPPA, Alpha, m68k it will simply mean the removal (or end of
>life and staying forever on 2.50, perhaps) of Git. There is no rust
>compiler there.
>
>Even s390 support for rust is limited to a precompiled version not
>everyone is willing to use.

Also in other distro concerns, if it trickles down to libgit2,
extra care should be taken to avoid creating circular dependencies
due to cargo depending on libgit2 (via git2 crate).

For example with making sure it can reasonably be built via meson's
Rust support rather than through cargo.

>
>GCC-rs will probably fix this general issue.
>
>-- >Eli Schwartz

gitgitgadget-git · 2025-07-19T00:57:12Z

User "Haelwenn (lanodan) Monnier" <contact@hacktivis.me> has been added to the cc: list.

gitgitgadget-git · 2025-07-19T21:57:57Z

On the Git mailing list, Johannes Schindelin wrote (reply to this):

Hi Ezekiel,                                                                                                                                                                                                           
pleasure to make your acquaintance!

On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote:

> 1. Windows fails to build. I don’t know which rust toolchain is even
>    correct for this or if multiple are needed.  Example failed build:
>    https://github.com/git/git/actions/runs/16353209191

There are a couple of problems, not just one. Here are the patches that I
would like to ask you to take custody of (for your convenience, I have
pushed them to https://github.com/dscho/git as the `xdiff_rust_speedup`
branch). Please find them below. They _just_ fix the build, but the tests
with win+Meson still fail (and as "win+Meson test" jobs keep the logs of
the failed tests a well-guarded secret, due to time constraints I have to
stop looking into this for now).

Thank you for working on this,
Johannes

-- snipsnap --
From 72c50ee3f9df5ccfe48bf6f44b2c6bba05a680bf Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Sat, 19 Jul 2025 21:24:07 +0200
Subject: [PATCH 1/3] Do support Windows again after requiring Rust

By default, Rust wants to build MS Visual C-compatible libraries on
Windows, because that is _the_ native C compiler.

Git is historically lacking in its MSVC support, and the official Git
for Windows versions are built using GCC instead. As a consequence, a
(subset of a) GCC toolchain is installed as part of the `windows-build`
job of every CI build.

Naturally, this requires adjustments in how Rust is called, most
importantly it requires installing support for a GCC-compatible build
target.

Let's make the necessary adjustment both in the CI-specific code that
installs Rust as well as in the Windows-specific configuration in
`config.mak.uname`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 ci/install-rust.sh | 3 +++
 config.mak.uname   | 9 +++++++++
 2 files changed, 12 insertions(+)

diff --git a/ci/install-rust.sh b/ci/install-rust.sh
index 141ceddb17cfe..c22baa629ceb7 100644
--- a/ci/install-rust.sh
+++ b/ci/install-rust.sh
@@ -28,6 +28,9 @@ if [ "$BITNESS" = "32" ]; then
   $CARGO_HOME/bin/rustup default --force-non-host $RUST_VERSION || exit $?
 else
   $CARGO_HOME/bin/rustup default $RUST_VERSION || exit $?
+  if [ "$CI_OS_NAME" = "windows" ]; then
+    $CARGO_HOME/bin/rustup target add x86_64-pc-windows-gnu || exit $?
+  fi
 fi
 
 . $CARGO_HOME/env
diff --git a/config.mak.uname b/config.mak.uname
index 3e26bb074a4b5..fbe7cebf40edd 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -727,19 +727,28 @@ ifeq ($(uname_S),MINGW)
 		prefix = /mingw32
 		HOST_CPU = i686
 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,_mainCRTStartup
+		CARGO_BUILD_TARGET = i686-pc-windows-gnu
         endif
         ifeq (MINGW64,$(MSYSTEM))
 		prefix = /mingw64
 		HOST_CPU = x86_64
 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,mainCRTStartup
+		CARGO_BUILD_TARGET = x86_64-pc-windows-gnu
         else ifeq (CLANGARM64,$(MSYSTEM))
 		prefix = /clangarm64
 		HOST_CPU = aarch64
 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,mainCRTStartup
+		CARGO_BUILD_TARGET = aarch64-pc-windows-gnu
         else
 		COMPAT_CFLAGS += -D_USE_32BIT_TIME_T
 		BASIC_LDFLAGS += -Wl,--large-address-aware
         endif
+
+	export CARGO_BUILD_TARGET
+	RUST_TARGET_DIR = rust/target/$(CARGO_BUILD_TARGET)/$(RUST_BUILD_MODE)
+	# Unfortunately now needed because of Rust
+	EXTLIBS += -luserenv
+
 	CC = gcc
 	COMPAT_CFLAGS += -D__USE_MINGW_ANSI_STDIO=0 -DDETECT_MSYS_TTY \
 		-fstack-protector-strong
-- 
2.50.1.windows.1


From ef6e4394ae26d8f28cb0d9e456810ce0818e623b Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Sat, 19 Jul 2025 23:08:11 +0200
Subject: [PATCH 2/3] win+Meson: allow for xdiff to be compiled with MSVC

The `build_rust.sh` script is quite opinionated about the naming scheme
of the C compiler: It assumes that the xdiff library file will be named
`libxdiff.a`.

However, MS Visual C generates `xdiff.lib` files instead; This naming
scheme has been in use in a very, very long time.

Let's allow for that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 build_rust.sh |  7 ++++++-
 meson.build   | 12 +++++++++---
 2 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/build_rust.sh b/build_rust.sh
index 4c12135cd2050..694d48d857a58 100755
--- a/build_rust.sh
+++ b/build_rust.sh
@@ -44,7 +44,12 @@ fi
 
 cd $dir_rust && cargo clean && pwd && cargo build -p $crate $rust_args; cd ..
 
-libfile="lib${crate}.a"
+if grep x86_64-pc-windows-msvc rust/target/.rustc_info.json
+then
+  libfile="${crate}.lib"
+else
+  libfile="lib${crate}.a"
+fi
 dst=$dir_build/$libfile
 
 if [ "$dir_git_root" != "$dir_build" ]; then
diff --git a/meson.build b/meson.build
index 047d7e5b66306..5e89a5dd0e00f 100644
--- a/meson.build
+++ b/meson.build
@@ -277,8 +277,16 @@ else
   rustflags = '-Aunused_imports -Adead_code -C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes'
 endif
 
+compiler = meson.get_compiler('c')
+
+if compiler.get_id() == 'msvc'
+  xdiff_lib_filename = 'xdiff.lib'
+else
+  xdiff_lib_filename = 'libxdiff.a'
+endif
+
 rust_build_xdiff = custom_target('rust_build_xdiff',
-  output: 'libxdiff.a',
+  output: xdiff_lib_filename,
   build_by_default: true,
   build_always_stale: true,
   command: [
@@ -288,8 +296,6 @@ rust_build_xdiff = custom_target('rust_build_xdiff',
   install: false,
 )
 
-compiler = meson.get_compiler('c')
-
 libgit_sources = [
   'abspath.c',
   'add-interactive.c',
-- 
2.50.1.windows.1


From 9c3b017cfa069211027fbb1f6d3b97c8e7edda81 Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Sat, 19 Jul 2025 23:22:57 +0200
Subject: [PATCH 3/3] win+Meson: do allow linking with the Rust-built xdiff

When linking against the Rust-built `xdiff`, there is now a new required
dependency: Without _also_ linking to the system library `userenv`, the
compile would fail with this error message:

  xdiff.lib(std-c85e9beb7923f636.std.df32d1bc89881d89-cgu.0.rcgu.o) :
  error LNK2019: unresolved external symbol __imp_GetUserProfileDirectoryW
  referenced in function _ZN3std3env8home_dir17hfd1c3b6676cd78f6E

Therefore, just like we do in case of Makefile-based builds on Windows,
we now also link to that library when building with Meson.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 meson.build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meson.build b/meson.build
index 5e89a5dd0e00f..af015f04763fd 100644
--- a/meson.build
+++ b/meson.build
@@ -1260,6 +1260,7 @@ elif host_machine.system() == 'windows'
   ]
 
   libgit_dependencies += compiler.find_library('ntdll')
+  libgit_dependencies += compiler.find_library('userenv')
   libgit_include_directories += 'compat/win32'
   if compiler.get_id() == 'msvc'
     libgit_include_directories += 'compat/vcbuild/include'
-- 
2.50.1.windows.1

gitgitgadget-git · 2025-07-19T21:57:59Z

User Johannes Schindelin <Johannes.Schindelin@gmx.de> has been added to the cc: list.

gitgitgadget-git · 2025-07-19T21:58:01Z

rust/Cargo.lock

+
+[[package]]
+name = "xdiff"
+version = "0.1.0"


On the Git mailing list, Johannes Schindelin wrote (reply to this):

Hi Ezekiel, On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote: > diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs > index e69de29bb2d1..96975975a1ba 100644 > --- a/rust/xdiff/src/lib.rs > +++ b/rust/xdiff/src/lib.rs > @@ -0,0 +1,7 @@ > + > + > +#[no_mangle] > +unsafe extern "C" fn xxh3_64(ptr: *const u8, size: usize) -> u64 { > + let slice = std::slice::from_raw_parts(ptr, size); > + xxhash_rust::xxh3::xxh3_64(slice) > +} I know that this is a pretty small file, but I do notice that it does not have a license header. This reminds me of the unfortunate oversight to be careful about making (and keeping) libgit.a's source files compatible with libgit2's license to nurture a fruitful exchange between those two projects. With Rust, we still have a really good chance to learn from history and avoid that mistake: Gitoxide is a very exciting project with clear overlap in its mission to implement Git functionality in Rust. Gitoxide is dual-licensed under the Apache License v2 and the MIT license (see https://github.com/GitoxideLabs/gitoxide?tab=readme-ov-file#license). Would you mind adding a license header to that file that explicitly allows the contents of the file to be used in Gitoxide, to get the Rust effort started on a good foot? Thank you, Johannes

On the Git mailing list, Phillip Wood wrote (reply to this):

Hi Johannes On 19/07/2025 22:53, Johannes Schindelin wrote: > Hi Ezekiel, > > On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote: > >> diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs >> index e69de29bb2d1..96975975a1ba 100644 >> --- a/rust/xdiff/src/lib.rs >> +++ b/rust/xdiff/src/lib.rs >> @@ -0,0 +1,7 @@ >> + >> + >> +#[no_mangle] >> +unsafe extern "C" fn xxh3_64(ptr: *const u8, size: usize) -> u64 { >> + let slice = std::slice::from_raw_parts(ptr, size); >> + xxhash_rust::xxh3::xxh3_64(slice) >> +} > > I know that this is a pretty small file, but I do notice that it does not > have a license header. > > This reminds me of the unfortunate oversight to be careful about making > (and keeping) libgit.a's source files compatible with libgit2's license to > nurture a fruitful exchange between those two projects. I'm not sure I follow your reasoning here. libgit2 was started after git and chose to use an incompatible license. I wasn't around at the time but isn't there a list of git contributors who are happy to re-license their contributions with the linking exception used by libgit2? > With Rust, we still have a really good chance to learn from history and > avoid that mistake: Gitoxide is a very exciting project with clear overlap > in its mission to implement Git functionality in Rust. Gitoxide is > dual-licensed under the Apache License v2 and the MIT license (see > https://github.com/GitoxideLabs/gitoxide?tab=readme-ov-file#license). > > Would you mind adding a license header to that file that explicitly allows > the contents of the file to be used in Gitoxide, to get the Rust effort > started on a good foot? I wary of that for two reasons. Firstly over time it is de-facto re-licensing git as the amount of rust code grows and the amount of C code shrinks which deserves a wider discussion. Secondly it makes it harder to convert our C code which is licensed under GPL2 (or in the case of xdiff LGPL) to rust if the rust code uses a different license. If someone wants to start a discussion about re-licensing git (and is prepared to do all of the associated admin in the event that it happens) then by all means do so but I don't think it we want to slip such a change into this series. Thanks Phillip

gitgitgadget-git · 2025-07-19T21:58:03Z

.github/workflows/main.yml

@@ -4,6 +4,7 @@ on: [push, pull_request]



On the Git mailing list, Johannes Schindelin wrote (reply to this):

Hi Ezekiel, On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote: > +if [ "$dir_git_root" != "$dir_build" ]; then > + src=$dir_rust/target/$rust_target/$libfile > + if [ ! -f $src ]; then > + echo >&2 "::error:: cannot find path of static library" > + exit 5 > + fi As I found out the hard way, this error message could be more helpful if it specified a couple of those variables that play into the failure (or all of them). Would you mind changing the error message accordingly? Thank you, Johannes

gitgitgadget-git · 2025-07-20T01:42:43Z

xdiff/xdiffi.c

@@ -418,7 +418,7 @@ static int get_indent(xrecord_t *rec)
 	long i;


On the Git mailing list, Johannes Schindelin wrote (reply to this):

Hi Ezekiel, On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote: > diff --git a/xdiff/xtypes.h b/xdiff/xtypes.h > index 8b8467360ecf..6e5f67ebf380 100644 > --- a/xdiff/xtypes.h > +++ b/xdiff/xtypes.h > @@ -39,9 +39,9 @@ typedef struct s_chastore { > } chastore_t; > > typedef struct s_xrecord { > - char const *ptr; > - long size; > - unsigned long ha; > + u8 const* ptr; > + usize size; > + u64 ha; > } xrecord_t; > > typedef struct s_xdfile { You cannot do this on its own, you'll also have to do the following (which incidentally fixes the linux32 failures as well as the win test and win+Meson test failures, see https://github.com/dscho/git/actions/runs/16394351471): -- snipsnap -- From 8693c83858a7c9308e54fb470cd7e82bcf67c758 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin <johannes.schindelin@gmx.de> Date: Sun, 20 Jul 2025 02:34:35 +0200 Subject: [PATCH] fixup! xdiff: make fields of xrecord_t Rust friendly To make `xdl_classify_record()` work, the `ha` attributes of `xrecord_t` and of `s_xdlclass` _must_ have the same range. Otherwise the function won't be able to recognize previously-classified records correctly when the `ha` recorded in the `xrecord_t` is so wide that it won't fit into the `s_xdlclass`' attribute and therefore they won't match when they need to match. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> --- xdiff/xprepare.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c index 5a2e52f102cf7..c0463bacd94b0 100644 --- a/xdiff/xprepare.c +++ b/xdiff/xprepare.c @@ -32,7 +32,7 @@ typedef struct s_xdlclass { struct s_xdlclass *next; - unsigned long ha; + u64 ha; char const *line; long size; long idx; -- 2.50.1.windows.1

gitgitgadget-git · 2025-07-20T08:49:10Z

On the Git mailing list, Matthias Aßhauer wrote (reply to this):

On Sat, 19 Jul 2025, Johannes Schindelin wrote:

> Hi Ezekiel,
> pleasure to make your acquaintance!
>
> On Thu, 17 Jul 2025, Ezekiel Newren via GitGitGadget wrote:
>
>> 1. Windows fails to build. I don’t know which rust toolchain is even
>>    correct for this or if multiple are needed.  Example failed build:
>>    https://github.com/git/git/actions/runs/16353209191
>
> There are a couple of problems, not just one. Here are the patches that I
> would like to ask you to take custody of (for your convenience, I have
> pushed them to https://github.com/dscho/git as the `xdiff_rust_speedup`
> branch). Please find them below. They _just_ fix the build, but the tests
> with win+Meson still fail (and as "win+Meson test" jobs keep the logs of
> the failed tests a well-guarded secret, due to time constraints I have to
> stop looking into this for now).
>
> Thank you for working on this,
> Johannes
>
> -- snipsnap --
> From 72c50ee3f9df5ccfe48bf6f44b2c6bba05a680bf Mon Sep 17 00:00:00 2001
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> Date: Sat, 19 Jul 2025 21:24:07 +0200
> Subject: [PATCH 1/3] Do support Windows again after requiring Rust
>
> By default, Rust wants to build MS Visual C-compatible libraries on
> Windows, because that is _the_ native C compiler.
>
> Git is historically lacking in its MSVC support, and the official Git
> for Windows versions are built using GCC instead. As a consequence, a
> (subset of a) GCC toolchain is installed as part of the `windows-build`
> job of every CI build.
>
> Naturally, this requires adjustments in how Rust is called, most
> importantly it requires installing support for a GCC-compatible build
> target.
>
> Let's make the necessary adjustment both in the CI-specific code that
> installs Rust as well as in the Windows-specific configuration in
> `config.mak.uname`.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> ci/install-rust.sh | 3 +++
> config.mak.uname   | 9 +++++++++
> 2 files changed, 12 insertions(+)
>
> diff --git a/ci/install-rust.sh b/ci/install-rust.sh
> index 141ceddb17cfe..c22baa629ceb7 100644
> --- a/ci/install-rust.sh
> +++ b/ci/install-rust.sh
> @@ -28,6 +28,9 @@ if [ "$BITNESS" = "32" ]; then
>   $CARGO_HOME/bin/rustup default --force-non-host $RUST_VERSION || exit $?
> else
>   $CARGO_HOME/bin/rustup default $RUST_VERSION || exit $?
> +  if [ "$CI_OS_NAME" = "windows" ]; then
> +    $CARGO_HOME/bin/rustup target add x86_64-pc-windows-gnu || exit $?
> +  fi
> fi
>
> . $CARGO_HOME/env
> diff --git a/config.mak.uname b/config.mak.uname
> index 3e26bb074a4b5..fbe7cebf40edd 100644
> --- a/config.mak.uname
> +++ b/config.mak.uname
> @@ -727,19 +727,28 @@ ifeq ($(uname_S),MINGW)
> 		prefix = /mingw32
> 		HOST_CPU = i686
> 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,_mainCRTStartup
> +		CARGO_BUILD_TARGET = i686-pc-windows-gnu

While i686-pc-windows-gnu is fine for CI, it would mean we'd have to bump our supported Windows version up to Windows 10. If we want to keep supporting Windows 8.1, we'll need i686-win7-windows-gnu, at least on rust 1.78 and newer.[1][2] We'd probably build Windows versions on rust 1.88 currently.[3]

[1] https://blog.rust-lang.org/2024/02/26/Windows-7/
[2] https://doc.rust-lang.org/rustc/platform-support/win7-windows-gnu.html
[3] https://packages.msys2.org/base/mingw-w64-rust

>         endif
>         ifeq (MINGW64,$(MSYSTEM))
> 		prefix = /mingw64
> 		HOST_CPU = x86_64
> 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,mainCRTStartup
> +		CARGO_BUILD_TARGET = x86_64-pc-windows-gnu

For x86_64 we'llprobably  also want x86_64-win7-windows-gnu if we want to keep Windows 8.1 support.

>         else ifeq (CLANGARM64,$(MSYSTEM))
> 		prefix = /clangarm64
> 		HOST_CPU = aarch64
> 		BASIC_LDFLAGS += -Wl,--pic-executable,-e,mainCRTStartup
> +		CARGO_BUILD_TARGET = aarch64-pc-windows-gnu

Are we sure this target currently exists? It's at least undocumented.[4] I think we might want aarch64-pc-windows-gnullvm for CLANGARM64, either way.[5]

[4] https://doc.rust-lang.org/rustc/platform-support/windows-gnu.html
[5] https://doc.rust-lang.org/rustc/platform-support/windows-gnullvm.html

Best regards

Matthias

>         else
> 		COMPAT_CFLAGS += -D_USE_32BIT_TIME_T
> 		BASIC_LDFLAGS += -Wl,--large-address-aware
>         endif
> +
> +	export CARGO_BUILD_TARGET
> +	RUST_TARGET_DIR = rust/target/$(CARGO_BUILD_TARGET)/$(RUST_BUILD_MODE)
> +	# Unfortunately now needed because of Rust
> +	EXTLIBS += -luserenv
> +
> 	CC = gcc
> 	COMPAT_CFLAGS += -D__USE_MINGW_ANSI_STDIO=0 -DDETECT_MSYS_TTY \
> 		-fstack-protector-strong
> -- > 2.50.1.windows.1
>
>
> From ef6e4394ae26d8f28cb0d9e456810ce0818e623b Mon Sep 17 00:00:00 2001
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> Date: Sat, 19 Jul 2025 23:08:11 +0200
> Subject: [PATCH 2/3] win+Meson: allow for xdiff to be compiled with MSVC
>
> The `build_rust.sh` script is quite opinionated about the naming scheme
> of the C compiler: It assumes that the xdiff library file will be named
> `libxdiff.a`.
>
> However, MS Visual C generates `xdiff.lib` files instead; This naming
> scheme has been in use in a very, very long time.
>
> Let's allow for that.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> build_rust.sh |  7 ++++++-
> meson.build   | 12 +++++++++---
> 2 files changed, 15 insertions(+), 4 deletions(-)
>
> diff --git a/build_rust.sh b/build_rust.sh
> index 4c12135cd2050..694d48d857a58 100755
> --- a/build_rust.sh
> +++ b/build_rust.sh
> @@ -44,7 +44,12 @@ fi
>
> cd $dir_rust && cargo clean && pwd && cargo build -p $crate $rust_args; cd ..
>
> -libfile="lib${crate}.a"
> +if grep x86_64-pc-windows-msvc rust/target/.rustc_info.json
> +then
> +  libfile="${crate}.lib"
> +else
> +  libfile="lib${crate}.a"
> +fi
> dst=$dir_build/$libfile
>
> if [ "$dir_git_root" != "$dir_build" ]; then
> diff --git a/meson.build b/meson.build
> index 047d7e5b66306..5e89a5dd0e00f 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -277,8 +277,16 @@ else
>   rustflags = '-Aunused_imports -Adead_code -C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes'
> endif
>
> +compiler = meson.get_compiler('c')
> +
> +if compiler.get_id() == 'msvc'
> +  xdiff_lib_filename = 'xdiff.lib'
> +else
> +  xdiff_lib_filename = 'libxdiff.a'
> +endif
> +
> rust_build_xdiff = custom_target('rust_build_xdiff',
> -  output: 'libxdiff.a',
> +  output: xdiff_lib_filename,
>   build_by_default: true,
>   build_always_stale: true,
>   command: [
> @@ -288,8 +296,6 @@ rust_build_xdiff = custom_target('rust_build_xdiff',
>   install: false,
> )
>
> -compiler = meson.get_compiler('c')
> -
> libgit_sources = [
>   'abspath.c',
>   'add-interactive.c',
> -- > 2.50.1.windows.1
>
>
> From 9c3b017cfa069211027fbb1f6d3b97c8e7edda81 Mon Sep 17 00:00:00 2001
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
> Date: Sat, 19 Jul 2025 23:22:57 +0200
> Subject: [PATCH 3/3] win+Meson: do allow linking with the Rust-built xdiff
>
> When linking against the Rust-built `xdiff`, there is now a new required
> dependency: Without _also_ linking to the system library `userenv`, the
> compile would fail with this error message:
>
>  xdiff.lib(std-c85e9beb7923f636.std.df32d1bc89881d89-cgu.0.rcgu.o) :
>  error LNK2019: unresolved external symbol __imp_GetUserProfileDirectoryW
>  referenced in function _ZN3std3env8home_dir17hfd1c3b6676cd78f6E
>
> Therefore, just like we do in case of Makefile-based builds on Windows,
> we now also link to that library when building with Meson.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> meson.build | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/meson.build b/meson.build
> index 5e89a5dd0e00f..af015f04763fd 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1260,6 +1260,7 @@ elif host_machine.system() == 'windows'
>   ]
>
>   libgit_dependencies += compiler.find_library('ntdll')
> +  libgit_dependencies += compiler.find_library('userenv')
>   libgit_include_directories += 'compat/win32'
>   if compiler.get_id() == 'msvc'
>     libgit_include_directories += 'compat/vcbuild/include'
> -- > 2.50.1.windows.1
>

gitgitgadget-git · 2025-07-20T08:49:12Z

User Matthias Aßhauer <mha1993@live.de> has been added to the cc: list.

gitgitgadget-git · 2025-07-21T10:23:34Z

On the Git mailing list, Phillip Wood wrote (reply to this):

On 18/07/2025 15:38, Junio C Hamano wrote:
> "Ezekiel Newren via GitGitGadget" <gitgitgadget@gmail.com> writes:
> >> This series accelerates xdiff by 5-19%.
> > ;-)
> > Do we know how much of that can be attributed to the hash algorithm
> difference, and how much for languages?

That's an interesting question. The two patches below [1] switch
xdiff to use xxhash from libxxhash. On my computer the rust and
C implementations both speed up "git log --oneline --shortstat"
by 15%. Just over half of that seems to come from hoisting the
check for whitespace flags in xdl_hash_record() out of the loop
in xdl_prepare_ctx() and the rest comes from the change in hash
function. As I understand it the hash is implemented using SIMD
compiler intrinsics and the rust implementation is basically a
copy of the C code in libxxhash. I wonder how well xxhash performs
compared to our existing hash on platforms without an optimized
implementation.

Thanks

Phillip

[1] These patches are available in the xdiff-hashing-experiments
    branch at https://github.com/phillipwood/git

---- 8< ----
From 06e7abdcfb9fc3f143ef84644966d6fce128d8ae Mon Sep 17 00:00:00 2001
From: Phillip Wood <phillip.wood@dunelm.org.uk>
Date: Sat, 19 Jul 2025 10:58:48 +0100
Subject: [PATCH 1/2] xdiff: refactor xdl_hash_record()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Inline the check for whitespace flags so that the compiler can hoist
it out of the loop in xdl_prepare_ctx(). This improves the performance
by 8%.

$ hyperfine --warmup=1 -L rev HEAD,HEAD^  --setup='git checkout {rev} -- :/ && make git' ': {rev}; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'
Benchmark 1: : HEAD; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.670 s ┬▒  0.044 s    [User: 1.473 s, System: 0.196 s]
  Range (min ÔÇª max):    1.619 s ÔÇª  1.754 s    10 runs

Benchmark 2: : HEAD^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.801 s ┬▒  0.021 s    [User: 1.605 s, System: 0.192 s]
  Range (min ÔÇª max):    1.766 s ÔÇª  1.831 s    10 runs

Summary
  ': HEAD^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0' ran
    1.08 ┬▒ 0.03 times faster than ': HEAD^^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'

Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---
 xdiff/xutils.c |  7 ++-----
 xdiff/xutils.h | 10 +++++++++-
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index 444a108f87..e070ed649f 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -249,7 +249,7 @@ int xdl_recmatch(const char *l1, long s1, const char *l2, long s2, long flags)
 	return 1;
 }
 -static unsigned long xdl_hash_record_with_whitespace(char const **data,
+unsigned long xdl_hash_record_with_whitespace(char const **data,
 		char const *top, long flags) {
 	unsigned long ha = 5381;
 	char const *ptr = *data;
@@ -294,13 +294,10 @@ static unsigned long xdl_hash_record_with_whitespace(char const **data,
 	return ha;
 }
 -unsigned long xdl_hash_record(char const **data, char const *top, long flags) {
+unsigned long xdl_hash_record_verbatim(char const **data, char const *top) {
 	unsigned long ha = 5381;
 	char const *ptr = *data;
 -	if (flags & XDF_WHITESPACE_FLAGS)
-		return xdl_hash_record_with_whitespace(data, top, flags);
-
 	for (; ptr < top && *ptr != '\n'; ptr++) {
 		ha += (ha << 5);
 		ha ^= (unsigned long) *ptr;
diff --git a/xdiff/xutils.h b/xdiff/xutils.h
index fd0bba94e8..13f6831047 100644
--- a/xdiff/xutils.h
+++ b/xdiff/xutils.h
@@ -34,7 +34,15 @@ void *xdl_cha_alloc(chastore_t *cha);
 long xdl_guess_lines(mmfile_t *mf, long sample);
 int xdl_blankline(const char *line, long size, long flags);
 int xdl_recmatch(const char *l1, long s1, const char *l2, long s2, long flags);
-unsigned long xdl_hash_record(char const **data, char const *top, long flags);
+unsigned long xdl_hash_record_verbatim(char const **data, char const *top);
+unsigned long xdl_hash_record_with_whitespace(char const **data, char const *top, long flags);
+static inline unsigned long xdl_hash_record(char const **data, char const *top, long flags)
+{
+	if (flags & XDF_WHITESPACE_FLAGS)
+		return xdl_hash_record_with_whitespace(data, top, flags);
+	else
+		return xdl_hash_record_verbatim(data, top);
+}
 unsigned int xdl_hashbits(unsigned int size);
 int xdl_num_out(char *out, long val);
 int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,
-- 
2.49.0.897.gfad3eb7d21


From 16f3b26624dc17002f3e507cd1e260deadfe1de8 Mon Sep 17 00:00:00 2001
From: Phillip Wood <phillip.wood@dunelm.org.uk>
Date: Sat, 19 Jul 2025 14:52:48 +0100
Subject: [PATCH 2/2] xdiff: use xxhash
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Using XXH3_64bits() from libxxhash to hash the input lines improves
the performance by about 6% and equals the performance of using
xxhash-rust.

$ hyperfine --warmup=1 -L rev en/xdiff-rust/v1,HEAD,HEAD^,HEAD^^  --setup='git checkout {rev} -- :/ && make git' ': {rev}; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'
Benchmark 1: : en/xdiff-rust/v1; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.575 s ┬▒  0.032 s    [User: 1.406 s, System: 0.168 s]
  Range (min ÔÇª max):    1.541 s ÔÇª  1.651 s    10 runs

Benchmark 2: : HEAD; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.569 s ┬▒  0.018 s    [User: 1.382 s, System: 0.185 s]
  Range (min ÔÇª max):    1.546 s ÔÇª  1.596 s    10 runs

Benchmark 3: : HEAD^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.661 s ┬▒  0.026 s    [User: 1.475 s, System: 0.186 s]
  Range (min ÔÇª max):    1.630 s ÔÇª  1.696 s    10 runs

Benchmark 4: : HEAD^^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0
  Time (mean ┬▒ ¤â):      1.800 s ┬▒  0.023 s    [User: 1.611 s, System: 0.187 s]
  Range (min ÔÇª max):    1.772 s ÔÇª  1.837 s    10 runs

Summary
  ': HEAD; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0' ran
    1.00 ┬▒ 0.02 times faster than ': en/xdiff-rust/v1; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'
    1.06 ┬▒ 0.02 times faster than ': HEAD^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'
    1.15 ┬▒ 0.02 times faster than ': HEAD^^; GIT_CONFIG_GLOBAL=/dev/null ./git log --oneline --shortstat v2.0.0..v2.5.0'

Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---
 Makefile       |  1 +
 xdiff/xutils.c | 14 ++++++--------
 2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index 5f7dd79dfa..6de7ccdf3b 100644
--- a/Makefile
+++ b/Makefile
@@ -1390,6 +1390,7 @@ UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/lib-reftable.o
 # xdiff and reftable libs may in turn depend on what is in libgit.a
 GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE)
 EXTLIBS =
+EXTLIBS += -lxxhash
   GIT_USER_AGENT = git/$(GIT_VERSION)
 diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index e070ed649f..43fce4b5b1 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -21,7 +21,7 @@
  */
   #include "xinclude.h"
-
+#include <xxhash.h>
   long xdl_bogosqrt(long n) {
 	long i;
@@ -295,14 +295,12 @@ unsigned long xdl_hash_record_with_whitespace(char const **data,
 }
   unsigned long xdl_hash_record_verbatim(char const **data, char const *top) {
-	unsigned long ha = 5381;
-	char const *ptr = *data;
+	long ha;
+	char const *eol = memchr(*data, '\n', top - *data);
+	size_t len = (eol ? eol : top) - *data;
 -	for (; ptr < top && *ptr != '\n'; ptr++) {
-		ha += (ha << 5);
-		ha ^= (unsigned long) *ptr;
-	}
-	*data = ptr < top ? ptr + 1: ptr;
+	ha = XXH3_64bits(*data, len);
+	*data += len + !!eol;
   	return ha;
 }
-- 
2.49.0.897.gfad3eb7d21

gitgitgadget-git · 2025-07-21T18:37:02Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

Phillip Wood <phillip.wood123@gmail.com> writes:

> ... by 15%. Just over half of that seems to come from hoisting the
> check for whitespace flags in xdl_hash_record() out of the loop
> in xdl_prepare_ctx() and the rest comes from the change in hash
> function.

The first half of that alone is interesting enough ;-).

> As I understand it the hash is implemented using SIMD
> compiler intrinsics and the rust implementation is basically a
> copy of the C code in libxxhash. I wonder how well xxhash performs
> compared to our existing hash on platforms without an optimized
> implementation.

Yeah, that indeed is worth investigating.

Thanks.

gitgitgadget-git · 2025-07-22T12:23:24Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Fri, Jul 18, 2025 at 12:29:16AM +0000, brian m. carlson wrote:
> On 2025-07-17 at 22:25:23, Taylor Blau wrote:
> > I agree. I don't think that there is ever going to be a "perfect" time
> > to introduce a hard dependency on Rust, and I don't think that should
> > hold the project back from adopting it.
> > 
> > I am far from a Rust expert, but I think that a more modern, memory-safe
> > language will attract newer contributors who may have a fresher
> > perspective on the project, and I think that's a good thing.
> 
> Yes, I think that's true.  Rust is by far the most admired programming
> language to work with, according to the 2024 Stack Overflow Developer
> Survey.  We will likely attract new contributors who find C intimidating
> or a bit of a hassle[0] but are excited about working on Rust,
> especially in a project as compelling as Git[1].

I am also aligned with allowing Rust into Git. I think the ecosystem has
kind of settled on Rust as the next system-level programming language,
and it does have good interop with C.

I think with the ongoing efforts to reduce our reliance on global state
we should eventually be able to encapsulate more and more of our
subsystems. And once they are neatly encapsulated we would be able to
swap out their respective implementation and plug in a Rust replacement.

Good candidates are for example the reftable library, as I've already
proposed in the past.

> > The alternative, of course, is to continue to use C and not take any
> > dependency on Rust. I think there is a middle-ground in there somewhere
> > to be able to build with (e.g.) "make" or "make RUST=1", but I would
> > really like to see the project take a firmer stance here.
> > 
> > I worry that having build support for both "with Rust" and "C only" will
> > create a headache not just at the build system level, but also in the
> > code itself. Having a patchwork of features, optimizations, or bug fixes
> > that either are or aren't supported depending on whether Rust support
> > was specified at build-time seems like a worst-of-all-worlds outcome.
> 
> I definitely agree.  I already find it terribly inconvenient when I end
> up when `git grep` doesn't support `-P` and I imagine that having lots
> of features that weren't available would be bothersome.
> 
> I also think that using a combination of C and Rust will end up with us
> still writing a lot of unsafe Rust code to interoperate with C.  If we
> want to reap the benefits in terms of memory and thread safety[2], we'll
> be better off sticking with just Rust.
> 
> I will also say that while it may be more challenging to compile Git at
> first on Windows, as we move more towards an all-Rust codebase, Git may
> end up being easier to maintain there as we depend more on the standard
> library.

Fully agreed. I've said so at the last contributors summit, but I think
it would become awfully unmaintainable if we retain two implementations
of every subsystem that we convert to Rust. If we decide to use Rust I
would strongly advocate for going all-in.

Patrick

gitgitgadget-git · 2025-07-22T12:23:26Z

User Patrick Steinhardt <ps@pks.im> has been added to the cc: list.

gitgitgadget-git · 2025-07-22T12:23:27Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Sat, Jul 19, 2025 at 02:48:39AM +0200, Haelwenn (lanodan) Monnier wrote:
> [2025-07-18 17:25:01-0400] Eli Schwartz:
> > On 7/18/25 9:34 AM, Phillip Wood wrote:
> > > Hi Ezekiel
> > > 
> > > Thanks for working on this
> > > 
> > > On 17/07/2025 21:32, Ezekiel Newren via GitGitGadget wrote:
> > > 
> > > > So...
> > > > 
> > > > This obviously raises the question of whether we are ready to accept a
> > > > hard
> > > > dependency on Rust. Previous discussions on the mailing list and at Git
> > > > Merge 2024 have not answered that question. If not now, will we be
> > > > willing
> > > > to accept such a hard dependency later? And what route do we want to
> > > > take to
> > > > get there?
> > > 
> > > As far as git goes I think introducing a hard dependency on rust is
> > > fine. It is widely supported, the only issue I'm aware of is the lack of
> > > support on NonStop and I don't think it is reasonable for such a
> > > minority platform to hold the rest of the project to ransom. There is a
> > > question about the other users of the xdiff code though. libgit2 carries
> > > a copy as do other projects like neovim. I've cc'd the libgit2
> > > maintainer and posted a link to this thread in neovim github [1]
> > 
> > 
> > A hard dependency on rust for Gentoo amd64 would potentially require
> > building https://github.com/thepowersgang/mrustc followed by building 13
> > and counting versions of rustc in order to get to the latest version.
> > What is the minimum supported version in this series, by the way?
> > 
> > bin packages for rust do exist but not everyone wants to use non-distro
> > provided binaries, sometimes for auditability reasons.
> > 
> > 
> > For Gentoo HPPA, Alpha, m68k it will simply mean the removal (or end of
> > life and staying forever on 2.50, perhaps) of Git. There is no rust
> > compiler there.
> > 
> > Even s390 support for rust is limited to a precompiled version not
> > everyone is willing to use.
> 
> Also in other distro concerns, if it trickles down to libgit2,
> extra care should be taken to avoid creating circular dependencies
> due to cargo depending on libgit2 (via git2 crate).
> 
> For example with making sure it can reasonably be built via meson's
> Rust support rather than through cargo.

I think it's unlikely that this eventually trickles down into libgit2.
The bundled versions of xdiff have already diverged for a long time, and
unfortunately libgit2 is mostly in maintenance mode nowadays. So I guess
that this change here just means that things will diverge even further
in the future, which is probably okay-ish. After all, the whole xdiff
library didn't really evolve in a fast pace over the last years.

That being said, there is an xdiff fork located at [1] that libgit2
maintains nowadays. So if the Rust dependency ever became a problem for
any of the downstream users I think we could simply redirect them to
that fork and make it the canonical upstream for C-only xdiff.

Patrick

[1]: https://github.com/libgit2/xdiff

gitgitgadget-git · 2025-07-22T14:26:13Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Fri, Jul 18, 2025 at 05:25:01PM -0400, Eli Schwartz wrote:
> On 7/18/25 9:34 AM, Phillip Wood wrote:
> > Hi Ezekiel
> > 
> > Thanks for working on this
> > 
> > On 17/07/2025 21:32, Ezekiel Newren via GitGitGadget wrote:
> >
> >> So...
> >>
> >> This obviously raises the question of whether we are ready to accept a
> >> hard
> >> dependency on Rust. Previous discussions on the mailing list and at Git
> >> Merge 2024 have not answered that question. If not now, will we be
> >> willing
> >> to accept such a hard dependency later? And what route do we want to
> >> take to
> >> get there?
> > 
> > As far as git goes I think introducing a hard dependency on rust is
> > fine. It is widely supported, the only issue I'm aware of is the lack of
> > support on NonStop and I don't think it is reasonable for such a
> > minority platform to hold the rest of the project to ransom. There is a
> > question about the other users of the xdiff code though. libgit2 carries
> > a copy as do other projects like neovim. I've cc'd the libgit2
> > maintainer and posted a link to this thread in neovim github [1]
> 
> 
> A hard dependency on rust for Gentoo amd64 would potentially require
> building https://github.com/thepowersgang/mrustc followed by building 13
> and counting versions of rustc in order to get to the latest version.
> What is the minimum supported version in this series, by the way?
> 
> bin packages for rust do exist but not everyone wants to use non-distro
> provided binaries, sometimes for auditability reasons.
> 
> 
> For Gentoo HPPA, Alpha, m68k it will simply mean the removal (or end of
> life and staying forever on 2.50, perhaps) of Git. There is no rust
> compiler there.
> 
> Even s390 support for rust is limited to a precompiled version not
> everyone is willing to use.
> 
> GCC-rs will probably fix this general issue.

Hm. It would be nice to assemble a list of common or semi-common
distributions that do not have proper support for Rust for all or at
least some platforms. Should we maybe consider reaching out to other
distros (e.g. Debian, Fedora, BSDs) before we commit to any change that
has an outsized impact on the larger ecosystem?

I would really love to start adopting Rust, and if it's only going to be
architectures that are extremely niche I'm probably fine with that. But
if there are many small systems that are impacted by such a change we
might have to reconsider.

Meh :/

Patrick

gitgitgadget-git · 2025-07-22T15:18:03Z

On the Git mailing list, Eli Schwartz wrote (reply to this):

On 7/22/25 10:24 AM, Patrick Steinhardt wrote:
> On Fri, Jul 18, 2025 at 05:25:01PM -0400, Eli Schwartz wrote:

>> For Gentoo HPPA, Alpha, m68k it will simply mean the removal (or end of
>> life and staying forever on 2.50, perhaps) of Git. There is no rust
>> compiler there.
>>
>> Even s390 support for rust is limited to a precompiled version not
>> everyone is willing to use.
>>
>> GCC-rs will probably fix this general issue.
> 
> Hm. It would be nice to assemble a list of common or semi-common
> distributions that do not have proper support for Rust for all or at
> least some platforms. Should we maybe consider reaching out to other
> distros (e.g. Debian, Fedora, BSDs) before we commit to any change that
> has an outsized impact on the larger ecosystem?
> 
> I would really love to start adopting Rust, and if it's only going to be
> architectures that are extremely niche I'm probably fine with that. But
> if there are many small systems that are impacted by such a change we
> might have to reconsider.
> 
> Meh :/

To elaborate a bit w.r.t. Gentooo.

Gentoo Prefix-on-macOS and Prefix-on-Solaris don't support rust either.
I think at least macOS is reasonably popular. Obviously Rust supports
macOS, and the Prefix maintainer would like it to work but hasn't been
able to -- no idea why. Arguably you can tell these users "install a
better OS so you can use git".

musl has lots of issues with rust, and is disabled for Gentoo musl
editions on arm (not arm64), ppc, i686, m68k, mips. Arguably you can
tell these users "musl sucks, why are you using it, use glibc like a
sensible person".

i486 is entirely disabled due to mandatory sse2. Hopefully those users
are rare even compared to i686 users. ;)

s390 only works on s390x

sparc 64ul works, but 32ul does not.

riscv rv64gc works, rv32imac does not.

A general trend here is 32-bit issues.

For alpha/hppa, no references at all -- not even tier 3 support -- on
https://doc.rust-lang.org/beta/rustc/platform-support.html, and Gentoo
doesn't support LLVM there either. ;) In general, porting rustc to a new
arch means *first* porting LLVM, and then after that, *also* porting
rustc, so who's going to try the latter before the former? ;)

Hence the interest in GCC-rs, which already has a backend supporting all
this for C/C++/Fortran plus interest by users of these arches in a
portable rust compiler.

If rust is added and doesn't have a fallback C impl, all this becomes a
relevant topic for consideration. (I don't have strong opinions on
optional rust.)

...

See

$ git clone https://github.com/gentoo/gentoo && cd gentoo
$ git grep --name-only features/wd40 profiles/| grep -v 17.0

(wd40 is the inheritance tree for disabling all features in any package
that rely on a rust compiler. See README at
https://github.com/gentoo/gentoo/tree/master/profiles/features/wd40 for
details)

-- 
Eli Schwartz

gitgitgadget-git · 2025-07-22T15:59:48Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

Patrick Steinhardt <ps@pks.im> writes:

> Fully agreed. I've said so at the last contributors summit, but I think
> it would become awfully unmaintainable if we retain two implementations
> of every subsystem that we convert to Rust. If we decide to use Rust I
> would strongly advocate for going all-in.

True.  

We do not have subsystems with clear boundaries yet, and introducing
Rust in such a state would not allow us to pick some parts (e.g.
merge backends, etc.) and do them optionally in Rust, while keeping
and/or adding others in C.

gitgitgadget-git · 2025-07-22T15:59:49Z

On the Git mailing list, Sam James wrote (reply to this):

There's a few issues from our perspective:

* Old platforms which don't have LLVM can't yet have Rust either, as
  rustc is based on LLVM.

  These need gccrs to be unblocked. I can understand not caring too much
  about these, though it is unfortunate, because I think if git hadn't
  supported many platforms to begin with, I doubt it'd have the adoption
  it does today.

  (There is another effort which seeks to take rustc and bolt on
  libgccjit as a replacement backend, but that isn't feasible for use
  yet either.)

* New platforms where rustc or various Rust crates don't support it
  and we have to go around patching them.

  The crate model makes this much harder. Not having git available when
  doing such porting if doing it natively is going to suck. It also
  means even more software needs Rust ported first.
  
* Platforms which aren't ancient, just not "the default", which tend not
  to work well with Rust.

  For example, rustc assumes that all musl configurations will be
  statically linked, which isn't the case. Working around this is a
  hassle.

* rustc doesn't have LTS releases or the like.

  The only supported release is the latest one. Upgrading to the latest
  release often means we have to deal with new portability problems
  but we can't not upgrade because:
  a) some software will start to require bleeding-edge Rust immediately,
  and
  b) it means we're missing out on bug fixes (miscompilations are
  serious)

* Crate creep

  Rust projects tend to end up having a huge list of crates that they
  pull-in which makes us worried about something nasty creeping in, but
  there's also popular crates with serious portability problems like the
  'ring' crate for TLS.

git is a fundamental piece of system software and making it harder to
build it or use it is a real worry for us.

gitgitgadget-git · 2025-07-22T15:59:51Z

User Sam James <sam@gentoo.org> has been added to the cc: list.

gitgitgadget-git · 2025-07-22T16:05:55Z

On the Git mailing list, Sam James wrote (reply to this):

> I am far from a Rust expert, but I think that a more modern, memory-safe
> language will attract newer contributors who may have a fresher
> perspective on the project, and I think that's a good thing.

Aren't they likely to contribute to gitoxide? There, they get a clean
slate without having to deal with the least-fun part (bidings).

> It is also not the Git project's responsibility to ensure that every
> platform is Rust-friendly.

That's true, of course. And nobody is entitled to indefinie updates, but
on the other hand, there's still some implicit contract with users. I
really don't think git would have the adoption it does today if it had
adopted a Rust-like language in the same state Rust is now from the
start.

(In exactly the same way, git doesn't gratuitously break compatibility
every release either. Can it? Yes, and git can change the platforms it
runs on, but it's something to be taken seriously.)

> Hopefully the platforms that we currently support but won't after this
> patch series have niche enough workloads that they do not need the
> absolute latest-and-greatest Git release at all times.

I mention this in my other email, but it's not just about ancient
platforms. It's also about new ones, or ones where Rust supports them
poorly despite them being relevant.

> Yeah, I think that this is the most interesting part of the discussion
> here. I am not knowledgeable enough about Rust's release cadence and
> platform compatibility to have an opinion here. But I trust brian's
> judgement ;-).

It gets a new release every 6 weeks and no other releases are supported.

gitgitgadget-git · 2025-07-22T21:39:30Z

On the Git mailing list, Elijah Newren wrote (reply to this):

Hi,

On Tue, Jul 22, 2025 at 9:03 AM Sam James <sam@gentoo.org> wrote:

First of all, thanks to all the Gentoo folks for chiming in and
providing specifics about platforms and their state.

> > I am far from a Rust expert, but I think that a more modern, memory-safe
> > language will attract newer contributors who may have a fresher
> > perspective on the project, and I think that's a good thing.
>
> Aren't they likely to contribute to gitoxide? There, they get a clean
> slate without having to deal with the least-fun part (bidings).

I'm sure some are.  But clearly there are others where the draw is
improving git itself because of its installed base; in fact, we need
look no further than this exact series we are commenting on to find
proof of that -- one such new contributor submitted patches to use
Rust in git, and found a significant speedup while doing so.

Further, there's considerable interest from existing git developers to
use Rust in git as well; last year at the Git contributor summit,
usage of Rust in git was not only one of the topics of discussion, it
was the top voted topic (meaning, the topic that the greatest number
of git contributors wanted to discuss).

> > It is also not the Git project's responsibility to ensure that every
> > platform is Rust-friendly.
>
> That's true, of course. And nobody is entitled to indefinie updates, but
> on the other hand, there's still some implicit contract with users. I
> really don't think git would have the adoption it does today if it had
> adopted a Rust-like language in the same state Rust is now from the
> start.
>
> (In exactly the same way, git doesn't gratuitously break compatibility
> every release either. Can it? Yes, and git can change the platforms it
> runs on, but it's something to be taken seriously.)

This feels kind of close to a false dichotomy between breaking
compatibility every release and indefinite update entitlements.  There
is certainly some middle ground: discussing reducing the breadth of
platform support in order to gain other benefits, then gathering
feedback, making a plan, and announcing the upcoming change, etc.

And we're already pretty deep into it.  Concerns about losing out on
some platforms have repeatedly slowed us down from adopting Rust years
ago.  Yet, the desire for Rust adoption keeps coming up anyway; see
the threads starting at

  * https://lore.kernel.org/git/ZZ77NQkSuiRxRDwt@nand.local/
  * https://lore.kernel.org/git/Zu2D%2Fb1ZJbTlC1ml@nand.local/
  * https://lore.kernel.org/git/20241128-pks-meson-v10-22-79a3fb0cb3a6@pks.im/
(search for "Rust")
  * https://lore.kernel.org/git/cover.1723242556.git.steadmon@google.com/

The discussion has also been picked up and reported outside the Git
mailing list, e.g. https://lwn.net/Articles/998115/.

And so, in addition to the optional contrib/libgit-rs and
contrib/libgit-sys Rust components that have already been merged into
git, and a new build system added in part to make it easier to adopt
Rust, we now have the first patch series that proposes a hard
dependency on Rust.

Further, I'd like to comment a bit on the support of our users from
another angle.  We're also responsible for security for our users, and
feel Rust would help (see e.g.
https://litchipi.github.io/infosec/2023/01/24/git-code-audit-viewed-as-rust-programmer.html
and https://github.com/bk2204/git/commit/fbeb1180c7473635a964daed2da642c53487782d).
We're responsible for performance of Git for our users, and feel Rust
would help (see the email that started this thread,
https://lore.kernel.org/git/CABPp-BFOmwV-xBtjvtenb6RFz9wx2VWVpTeho0k=D8wsCCVwqQ@mail.gmail.com/,
and brian's notes about [CPU multi-]threading elsewhere in this email
thread we are in).  And there are other benefits from using Rust that
we believe would benefit our users.  Thus, it's not just a question of
responsibility to our users, because such a responsibility pulls us in
different directions regarding usage of Rust.  So we need to figure
out how to weigh the needs of our different users.  For many of us,
and forgive the geeky comparison, we'll probably weigh those needs
with something more akin to an L2 norm (most good for the most users)
rather than an L-infinity norm (maximal difference in usability for a
single user), which probably isn't to your liking.

Anyway, there's been lots of discussion already.  We can certainly
still discuss more about exactly how to announce, when to adopt Rust,
whether we'll support an existing C-only version of git for a longer
period of time than normal, and even whether to continue to delay
adopting Rust for a little longer.  But my personal guess is that
attempting to stop adoption of Rust is unlikely to win at this point.

> > Hopefully the platforms that we currently support but won't after this
> > patch series have niche enough workloads that they do not need the
> > absolute latest-and-greatest Git release at all times.
>
> I mention this in my other email, but it's not just about ancient
> platforms. It's also about new ones, or ones where Rust supports them
> poorly despite them being relevant.

This feels like you're trying to push the decision for a given
platform to be a dichotomy between latest-and-greatest-Git or
no-version-of-Git-at-all, despite the fact that Taylor suggested an
alternative and you even quoted him.  Can you comment on that
alternative?  Why would using the last C-only version of Git[1] until
gccrs bridges the gap be a problem for these platforms?

Thanks,
Elijah

[1] Well, C-only other than optional Rust components like
contrib/libgit-rs and contrib/libgit-sys that have already been
released.

gitgitgadget-git · 2025-07-22T21:57:21Z

On the Git mailing list, Sam James wrote (reply to this):

Elijah Newren <newren@gmail.com> writes:

> Hi,
>
> On Tue, Jul 22, 2025 at 9:03 AM Sam James <sam@gentoo.org> wrote:
>
> First of all, thanks to all the Gentoo folks for chiming in and
> providing specifics about platforms and their state.

Thanks. I've been trying to be very specific about what the issues
are. I don't deny Rust is the future of many projects, but it still has
rough parts, and I'd like to ensure they're discussed. It is not my
intention to just scream whenever someone considers adopting Rust.

>
>> > I am far from a Rust expert, but I think that a more modern, memory-safe
>> > language will attract newer contributors who may have a fresher
>> > perspective on the project, and I think that's a good thing.
>>
>> Aren't they likely to contribute to gitoxide? There, they get a clean
>> slate without having to deal with the least-fun part (bidings).
>
> I'm sure some are.  But clearly there are others where the draw is
> improving git itself because of its installed base; in fact, we need
> look no further than this exact series we are commenting on to find
> proof of that -- one such new contributor submitted patches to use
> Rust in git, and found a significant speedup while doing so.

Part of my opinion there is coloured by how generally working on a
polylang codebase often has pain when dealing with bindings and the
edges, so I figure that anyone most-keen on Rust would surely want to
avoid that ;)

>
> Further, there's considerable interest from existing git developers to
> use Rust in git as well; last year at the Git contributor summit,
> usage of Rust in git was not only one of the topics of discussion, it
> was the top voted topic (meaning, the topic that the greatest number
> of git contributors wanted to discuss).
>
>> > It is also not the Git project's responsibility to ensure that every
>> > platform is Rust-friendly.
>>
>> That's true, of course. And nobody is entitled to indefinie updates, but
>> on the other hand, there's still some implicit contract with users. I
>> really don't think git would have the adoption it does today if it had
>> adopted a Rust-like language in the same state Rust is now from the
>> start.
>>
>> (In exactly the same way, git doesn't gratuitously break compatibility
>> every release either. Can it? Yes, and git can change the platforms it
>> runs on, but it's something to be taken seriously.)
>
> This feels kind of close to a false dichotomy between breaking
> compatibility every release and indefinite update entitlements.  There
> is certainly some middle ground: discussing reducing the breadth of
> platform support in order to gain other benefits, then gathering
> feedback, making a plan, and announcing the upcoming change, etc.
>

Of course. I'm just making the point that it is indeed a compatibility
change, and perhaps that perspective is useful.

> And we're already pretty deep into it.  Concerns about losing out on
> some platforms have repeatedly slowed us down from adopting Rust years
> ago.  Yet, the desire for Rust adoption keeps coming up anyway; see
> the threads starting at
>
>   * https://lore.kernel.org/git/ZZ77NQkSuiRxRDwt@nand.local/
>   * https://lore.kernel.org/git/Zu2D%2Fb1ZJbTlC1ml@nand.local/
>   * https://lore.kernel.org/git/20241128-pks-meson-v10-22-79a3fb0cb3a6@pks.im/
> (search for "Rust")
>   * https://lore.kernel.org/git/cover.1723242556.git.steadmon@google.com/
>
> The discussion has also been picked up and reported outside the Git
> mailing list, e.g. https://lwn.net/Articles/998115/.
>
> And so, in addition to the optional contrib/libgit-rs and
> contrib/libgit-sys Rust components that have already been merged into
> git, and a new build system added in part to make it easier to adopt
> Rust, we now have the first patch series that proposes a hard
> dependency on Rust.

Yes, that's why it's of concern. I have no issue with the optional parts.

>
> Further, I'd like to comment a bit on the support of our users from
> another angle.  We're also responsible for security for our users

Supply-chain issues become more of a problem with Rust if we end up
making heavy use of crates. A policy moderating their use is something
we should talk about.

> and
> feel Rust would help (see e.g.
> https://litchipi.github.io/infosec/2023/01/24/git-code-audit-viewed-as-rust-programmer.html
> and https://github.com/bk2204/git/commit/fbeb1180c7473635a964daed2da642c53487782d).
> We're responsible for performance of Git for our users, and feel Rust
> would help (see the email that started this thread,
> https://lore.kernel.org/git/CABPp-BFOmwV-xBtjvtenb6RFz9wx2VWVpTeho0k=D8wsCCVwqQ@mail.gmail.com/,
> and brian's notes about [CPU multi-]threading elsewhere in this email
> thread we are in).  And there are other benefits from using Rust that
> we believe would benefit our users.  Thus, it's not just a question of
> responsibility to our users, because such a responsibility pulls us in
> different directions regarding usage of Rust.  So we need to figure
> out how to weigh the needs of our different users.  For many of us,
> and forgive the geeky comparison, we'll probably weigh those needs
> with something more akin to an L2 norm (most good for the most users)
> rather than an L-infinity norm (maximal difference in usability for a
> single user), which probably isn't to your liking.
>

:)

> Anyway, there's been lots of discussion already.  We can certainly
> still discuss more about exactly how to announce, when to adopt Rust,
> whether we'll support an existing C-only version of git for a longer
> period of time than normal, and even whether to continue to delay
> adopting Rust for a little longer.  But my personal guess is that
> attempting to stop adoption of Rust is unlikely to win at this point.

I wouldn't characterise my position as attempting to flat-out stop
adoption of Rust (see beginning of this email).

>
>> > Hopefully the platforms that we currently support but won't after this
>> > patch series have niche enough workloads that they do not need the
>> > absolute latest-and-greatest Git release at all times.
>>
>> I mention this in my other email, but it's not just about ancient
>> platforms. It's also about new ones, or ones where Rust supports them
>> poorly despite them being relevant.
>
> This feels like you're trying to push the decision for a given
> platform to be a dichotomy between latest-and-greatest-Git or
> no-version-of-Git-at-all, despite the fact that Taylor suggested an
> alternative and you even quoted him.  Can you comment on that
> alternative?  Why would using the last C-only version of Git[1] until
> gccrs bridges the gap be a problem for these platforms?

What I was saying there was: it matters for platforms where they may not
have a git at all (because they're new, and we have a bit of a
bootstrapping problem), not just old ones where they're stuck on an old git.

Part of what I had in mind here is that sticking on old versions even
temporarily isn't necessarily a great option, see the recent issues w/
backports done (https://lore.kernel.org/git/xmqqldov4rpt.fsf@gitster.g/,
and https://lore.kernel.org/git/20250708210529.1214574-1-tmz@pobox.com/).

---

As a final note: I am genuinely not trying to be a member of the peanut
gallery wishing to prevent git's progress if the project wants to adopt
Rust, just there's some real practical obstacles for us right now.

I hope it didn't come across that way, but "dichotomy" appearing a few
times in your reply made me fear it did.

>
> Thanks,
> Elijah

thanks,
sam

>
> [1] Well, C-only other than optional Rust components like
> contrib/libgit-rs and contrib/libgit-sys that have already been
> released.

gitgitgadget-git · 2025-07-22T22:09:53Z

On the Git mailing list, Collin Funk wrote (reply to this):

Sam James <sam@gentoo.org> writes:

>> Further, I'd like to comment a bit on the support of our users from
>> another angle.  We're also responsible for security for our users
>
> Supply-chain issues become more of a problem with Rust if we end up
> making heavy use of crates. A policy moderating their use is something
> we should talk about.

+1. I find it a bit worrying when I see 500+ dependencies (mostly
transitive) being downloaded when running 'cargo build'.

Not saying we should go to the extreme of Not Invented Here syndrome
[1], since easy use of packages via 'cargo' is a major reason why people
enjoy Rust. But we should consider whether they provide enough value to
be included.

Collin

[1] https://en.wikipedia.org/wiki/Not_invented_here

gitgitgadget-git · 2025-07-22T22:09:55Z

User Collin Funk <collin.funk1@gmail.com> has been added to the cc: list.

gitgitgadget-git · 2025-07-22T22:26:41Z

User Mike Hommey <mh@glandium.org> has been added to the cc: list.

gitgitgadget-git · 2025-07-23T04:33:37Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Tue, Jul 22, 2025 at 04:56:12PM +0100, Sam James wrote:
> There's a few issues from our perspective:
> 
> * Old platforms which don't have LLVM can't yet have Rust either, as
>   rustc is based on LLVM.
> 
>   These need gccrs to be unblocked. I can understand not caring too much
>   about these, though it is unfortunate, because I think if git hadn't
>   supported many platforms to begin with, I doubt it'd have the adoption
>   it does today.
> 
>   (There is another effort which seeks to take rustc and bolt on
>   libgccjit as a replacement backend, but that isn't feasible for use
>   yet either.)

It would be great to know about the general timelines of these
alternative implementations. If e.g. gccrs were to achieve compatibility
with one of the editions of Rust next year it would be a good enough
reason to defer the rustification from my point of view so that we don't
break the ecosystem and have wider platform support. If the answer is
"They'll land in 10 years" then I don't know...

I sifted through their project sites and found various status reports,
and they do seem to be making steady progress. But as far as I see
critical language features are still missing as of now.

[snip]
> * rustc doesn't have LTS releases or the like.
> 
>   The only supported release is the latest one. Upgrading to the latest
>   release often means we have to deal with new portability problems
>   but we can't not upgrade because:
>   a) some software will start to require bleeding-edge Rust immediately,
>   and
>   b) it means we're missing out on bug fixes (miscompilations are
>   serious)

I'm not a big fan of this in the Rust ecosystem indeed. It feels like
every second project requires nightly features or at least a version of
the compiler that was released in the last couple months. This may work
for a language like Go, which is more targeted towards deploying server
applications. But for a system-level language like Rust I think it's
rather a sign of it being immature.

In any case, the burden would fall on us to ensure that we carefully
consider which version of Rust to target. And as it was said elsewhere
in the thread, we would need to make sure that things build on old
versions of Debian. Which may be easier said than done if we also rely
on lots of crates which may update to newer Rust versions at any point
in time.

> * Crate creep
> 
>   Rust projects tend to end up having a huge list of crates that they
>   pull-in which makes us worried about something nasty creeping in, but
>   there's also popular crates with serious portability problems like the
>   'ring' crate for TLS.

True. I think if we were to adopt Rust we ought to be as conservative as
we are now with picking up new dependencies. I don't want to have a big
open door for supply chain attacks. And neither do I want to be forced
into the situation where we cannot update a crate because they decided
to drop support for older Rust versions.

Patrick

gitgitgadget-git · 2025-07-24T09:05:46Z

On the Git mailing list, Pierre-Emmanuel Patry wrote (reply to this):

On Tue, Jul 23, 2025 at 06:32:06 +0200, Patrick Steinhardt wrote:
> It would be great to know about the general timelines of these
> alternative implementations.

We still think we'll be able to compile libcore before the end of the summer, we've made great progress and few items are left. But keep in mind we're targeting an older version of rust (1.49) and libcore is smaller than the standard library. We still have a lot of testing to do and we expect many bugs.

The next targeted version will probably be rust 1.78 as we want to keep up with rust for linux. This shouldn't be too long as most of the features are coming from either standard library modifications or nightly features we already had to support for 1.49.

We expect to be able to compile some 1.49 code correctly next year at best. I would like to bring to your attention rustc_codegen_gcc which adds a gcc backend to the rustc frontend, although not a full gcc compiler it could help supporting some architectures that are currently not supported by llvm.

Pierre-Emmanuel

gitgitgadget-git · 2025-07-24T09:05:49Z

User Pierre-Emmanuel Patry <pierre-emmanuel.patry@embecosm.com> has been added to the cc: list.

gitgitgadget-git · 2025-07-24T10:03:09Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Thu, Jul 24, 2025 at 11:01:22AM +0200, Pierre-Emmanuel Patry wrote:
> 
> On Tue, Jul 23, 2025 at 06:32:06 +0200, Patrick Steinhardt wrote:
> > It would be great to know about the general timelines of these
> > alternative implementations.
> 
> We still think we'll be able to compile libcore before the end of the
> summer, we've made great progress and few items are left. But keep in mind
> we're targeting an older version of rust (1.49) and libcore is smaller than
> the standard library. We still have a lot of testing to do and we expect
> many bugs.

Understood. Given that we don't plan to roll with the latest version of
Rust anyway I think it could be a viable tradeoff for us to also
consider gccrs when we determine the minimum required Rust version.

> The next targeted version will probably be rust 1.78 as we want to keep up
> with rust for linux. This shouldn't be too long as most of the features are
> coming from either standard library modifications or nightly features we
> already had to support for 1.49.
> 
> We expect to be able to compile some 1.49 code correctly next year at best.

And I expect that 1.78 will be another significant effort that won't
land before the year after?

> I would like to bring to your attention rustc_codegen_gcc which adds a gcc
> backend to the rustc frontend, although not a full gcc compiler it could
> help supporting some architectures that are currently not supported by llvm.

For my own understanding: is this something that the Git project would
have to support or something that the distributor needs to set up?

Patrick

gitgitgadget-git · 2025-07-25T23:58:42Z

User Ben Knoble <ben.knoble@gmail.com> has been added to the cc: list.

gitgitgadget-git bot added the new user label May 30, 2025

newren force-pushed the xdiff_rust_speedup branch 2 times, most recently from 8b3cc99 to 39211e9 Compare May 30, 2025 21:12

newren force-pushed the xdiff_rust_speedup branch from 39211e9 to 94ccfab Compare May 30, 2025 22:07

newren force-pushed the xdiff_rust_speedup branch from 94ccfab to d1acb81 Compare May 30, 2025 23:39

ezekielnewren force-pushed the xdiff_rust_speedup branch from d1acb81 to 9af7ba0 Compare May 31, 2025 01:22

gitgitgadget-git bot reviewed Jul 19, 2025

View reviewed changes

gitgitgadget-git bot reviewed Jul 20, 2025

View reviewed changes

		@@ -418,7 +418,7 @@ static int get_indent(xrecord_t *rec)
		long i;

RFC: Accelerate xdiff and begin its rustification #1980

Are you sure you want to change the base?

RFC: Accelerate xdiff and begin its rustification #1980

Conversation

ezekielnewren commented May 30, 2025 • edited by gitgitgadget-git bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Welcome to GitGitGadget

Contributing the patches

Need help?

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 31, 2025

Uh oh!

dscho commented Jun 1, 2025

Uh oh!

gitgitgadget-git bot commented Jun 1, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jul 19, 2025

Uh oh!

gitgitgadget-git bot commented Jul 19, 2025

Uh oh!

gitgitgadget-git bot commented Jul 19, 2025

Uh oh!

gitgitgadget-git bot commented Jul 19, 2025

Uh oh!

gitgitgadget-git bot Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot commented Jul 20, 2025

Uh oh!

ezekielnewren commented May 30, 2025 •

edited by gitgitgadget-git bot

Loading