Pass down value of `BUNDLE_JOBS` to RubyGems before compiling & introduce a new `gem install -j` flag #9171

Edouard-chin · 2025-12-08T21:19:44Z

What was the end-user or developer problem that led to this PR?

Since #9131, it's possible that Etc.nprocessors returns a value that isn't representative of the number of processors, especially in a docker environment. This may end up crashing/slowing the compilation process.

What is your fix for the problem, implemented in this PR?

Passing down the value of BUNDLE_JOBS to make would fix this issue, since they already run bundle install --jobs 4 in their docker container to prevent Bundler ParallelInstaller to fallback to Etc.processors.

Make sure the following tasks are checked

Describe the problem / feature
Write tests for features and bug fixes
Write code to solve the problem
Make sure you follow the current code style and write meaningful commit messages without tags

Edouard-chin · 2025-12-08T21:29:30Z

lib/rubygems/installer.rb

  end

+  def build_jobs
+    @build_jobs ||= Etc.nprocessors + 1


Etc.nprocessors + 1 is the number of jobs for two scenarios:

When running bundle install and no --jobs or BUNDLE_JOBS have been configured

When running gem install <gem>.

For 2., it's a bit tricky because RubyGems doesn't have the concept of parallelization with jobs. If this is a concern where gem install <gem> in a docker container takes too much resources, users don't really have a way to opt-out, unless they manually set the MAKEFLAGS env (which they have to know about).
Should we introduce a new command line flag to gem install & gem update ?

The alternative is to only add the MAKEFLAGS in the context of Bundler.

Should we introduce a new command line flag to gem install & gem update ?

+1

kou · 2025-12-09T02:24:23Z

lib/rubygems/installer.rb

  end

+  def build_jobs
+    @build_jobs ||= Etc.nprocessors + 1


Should we introduce a new command line flag to gem install & gem update ?

+1

kou · 2025-12-09T02:27:09Z

lib/rubygems/ext/ext_conf_builder.rb

+
+  def initialize(build_jobs)
+    return unless build_jobs
+
+    unless RUBY_PLATFORM.include?("mswin") && RbConfig::CONFIG["configure_args"]&.include?("nmake")
+      ENV["MAKEFLAGS"] ||= "-j#{build_jobs}"
+    end
+  end
+
+  def build(*args)
+    self.class.build(*args)
+  end


This may introduce a needless complexity...

How about adding a keyword argument to Gem::Ext::ExtConfBuilder.build instead?

And it may be better that we don't set MAKEFLAGS environment variable because it may affect other processes.

For example:

diff --git a/bundler/lib/bundler/rubygems_gem_installer.rb b/bundler/lib/bundler/rubygems_gem_installer.rb index 0da5ed236b..64ce6193d3 100644 --- a/bundler/lib/bundler/rubygems_gem_installer.rb +++ b/bundler/lib/bundler/rubygems_gem_installer.rb @@ -103,6 +103,10 @@ def generate_bin_script(filename, bindir) end end + def build_jobs + Bundler.settings[:jobs] || super + end + def build_extensions extension_cache_path = options[:bundler_extension_cache_path] extension_dir = spec.extension_dir diff --git a/lib/rubygems/ext/builder.rb b/lib/rubygems/ext/builder.rb index 600a6a5ff6..bfc8273cda 100644 --- a/lib/rubygems/ext/builder.rb +++ b/lib/rubygems/ext/builder.rb @@ -22,7 +22,8 @@ def self.class_name end def self.make(dest_path, results, make_dir = Dir.pwd, sitedir = nil, targets = ["clean", "", "install"], - target_rbconfig: Gem.target_rbconfig) + target_rbconfig: Gem.target_rbconfig, + n_jobs: nil) unless File.exist? File.join(make_dir, "Makefile") # No makefile exists, nothing to do. raise NoMakefileError, "No Makefile found in #{make_dir}" @@ -34,8 +35,17 @@ def self.make(dest_path, results, make_dir = Dir.pwd, sitedir = nil, targets = [ make_program_name ||= RUBY_PLATFORM.include?("mswin") ? "nmake" : "make" make_program = shellsplit(make_program_name) + is_nmake = /\bnmake/i.match?(make_program_name) # The installation of the bundled gems is failed when DESTDIR is empty in mswin platform. - destdir = /\bnmake/i !~ make_program_name || ENV["DESTDIR"] && ENV["DESTDIR"] != "" ? format("DESTDIR=%s", ENV["DESTDIR"]) : "" + destdir = !is_nmake || ENV["DESTDIR"] && ENV["DESTDIR"] != "" ? format("DESTDIR=%s", ENV["DESTDIR"]) : "" + + # nmake doesn't support parallel build + unless is_nmake + have_make_arguments = make_program.size > 1 + if !have_make_arguments and !ENV["MAKEFLAGS"] and n_jobs + make_program << "-j#{n_jbos}" + end + end env = [destdir] @@ -147,11 +157,12 @@ def self.shelljoin(command) # have build arguments, saved, set +build_args+ which is an ARGV-style # array. - def initialize(spec, build_args = spec.build_args, target_rbconfig = Gem.target_rbconfig) + def initialize(spec, build_args = spec.build_args, target_rbconfig = Gem.target_rbconfig, n_jobs: nil) @spec = spec @build_args = build_args @gem_dir = spec.full_gem_path @target_rbconfig = target_rbconfig + @n_jobs = n_jobs @ran_rake = false end @@ -208,7 +219,7 @@ def build_extension(extension, dest_path) # :nodoc: FileUtils.mkdir_p dest_path results = builder.build(extension, dest_path, - results, @build_args, lib_dir, extension_dir, @target_rbconfig) + results, @build_args, lib_dir, extension_dir, @target_rbconfig, n_jobs: @n_jobs) verbose { results.join("\n") } diff --git a/lib/rubygems/ext/cargo_builder.rb b/lib/rubygems/ext/cargo_builder.rb index 6bf3b405ad..42dca3b102 100644 --- a/lib/rubygems/ext/cargo_builder.rb +++ b/lib/rubygems/ext/cargo_builder.rb @@ -15,7 +15,7 @@ def initialize end def build(extension, dest_path, results, args = [], lib_dir = nil, cargo_dir = Dir.pwd, - target_rbconfig = Gem.target_rbconfig) + target_rbconfig = Gem.target_rbconfig, n_jobs: nil) require "tempfile" require "fileutils" diff --git a/lib/rubygems/ext/cmake_builder.rb b/lib/rubygems/ext/cmake_builder.rb index 2915568b39..e660ed558b 100644 --- a/lib/rubygems/ext/cmake_builder.rb +++ b/lib/rubygems/ext/cmake_builder.rb @@ -37,7 +37,7 @@ def initialize end def build(extension, dest_path, results, args = [], lib_dir = nil, cmake_dir = Dir.pwd, - target_rbconfig = Gem.target_rbconfig) + target_rbconfig = Gem.target_rbconfig, n_jobs: nil) if target_rbconfig.path warn "--target-rbconfig is not yet supported for CMake extensions. Ignoring" end diff --git a/lib/rubygems/ext/configure_builder.rb b/lib/rubygems/ext/configure_builder.rb index 76c1cd8b19..230b214b3c 100644 --- a/lib/rubygems/ext/configure_builder.rb +++ b/lib/rubygems/ext/configure_builder.rb @@ -8,7 +8,7 @@ class Gem::Ext::ConfigureBuilder < Gem::Ext::Builder def self.build(extension, dest_path, results, args = [], lib_dir = nil, configure_dir = Dir.pwd, - target_rbconfig = Gem.target_rbconfig) + target_rbconfig = Gem.target_rbconfig, n_jobs: nil) if target_rbconfig.path warn "--target-rbconfig is not yet supported for configure-based extensions. Ignoring" end @@ -19,7 +19,7 @@ def self.build(extension, dest_path, results, args = [], lib_dir = nil, configur run cmd, results, class_name, configure_dir end - make dest_path, results, configure_dir, target_rbconfig: target_rbconfig + make dest_path, results, configure_dir, target_rbconfig: target_rbconfig, n_jobs: n_jobs results end diff --git a/lib/rubygems/ext/ext_conf_builder.rb b/lib/rubygems/ext/ext_conf_builder.rb index 81491eac79..033f925839 100644 --- a/lib/rubygems/ext/ext_conf_builder.rb +++ b/lib/rubygems/ext/ext_conf_builder.rb @@ -8,7 +8,7 @@ class Gem::Ext::ExtConfBuilder < Gem::Ext::Builder def self.build(extension, dest_path, results, args = [], lib_dir = nil, extension_dir = Dir.pwd, - target_rbconfig = Gem.target_rbconfig) + target_rbconfig = Gem.target_rbconfig, n_jobs: nil) require "fileutils" require "tempfile" @@ -40,11 +40,8 @@ def self.build(extension, dest_path, results, args = [], lib_dir = nil, extensio end ENV["DESTDIR"] = nil - unless RUBY_PLATFORM.include?("mswin") && RbConfig::CONFIG["configure_args"]&.include?("nmake") - ENV["MAKEFLAGS"] ||= "-j#{Etc.nprocessors + 1}" - end - make dest_path, results, extension_dir, tmp_dest_relative, target_rbconfig: target_rbconfig + make dest_path, results, extension_dir, tmp_dest_relative, target_rbconfig: target_rbconfig, n_jobs: n_jobs full_tmp_dest = File.join(extension_dir, tmp_dest_relative) @@ -63,7 +60,7 @@ def self.build(extension, dest_path, results, args = [], lib_dir = nil, extensio destent.exist? || FileUtils.mv(ent.path, destent.path) end - make dest_path, results, extension_dir, tmp_dest_relative, ["clean"], target_rbconfig: target_rbconfig + make dest_path, results, extension_dir, tmp_dest_relative, ["clean"], target_rbconfig: target_rbconfig, n_jobs: n_jobs ensure ENV["DESTDIR"] = destdir end diff --git a/lib/rubygems/ext/rake_builder.rb b/lib/rubygems/ext/rake_builder.rb index 0eac5a180c..d702d7f339 100644 --- a/lib/rubygems/ext/rake_builder.rb +++ b/lib/rubygems/ext/rake_builder.rb @@ -8,7 +8,7 @@ class Gem::Ext::RakeBuilder < Gem::Ext::Builder def self.build(extension, dest_path, results, args = [], lib_dir = nil, extension_dir = Dir.pwd, - target_rbconfig = Gem.target_rbconfig) + target_rbconfig = Gem.target_rbconfig, n_jobs: nil) if target_rbconfig.path warn "--target-rbconfig is not yet supported for Rake extensions. Ignoring" end diff --git a/lib/rubygems/installer.rb b/lib/rubygems/installer.rb index 4c3038770d..ebbb211bf3 100644 --- a/lib/rubygems/installer.rb +++ b/lib/rubygems/installer.rb @@ -635,6 +635,7 @@ def process_options # :nodoc: @build_root = options[:build_root] @build_args = options[:build_args] + @build_jobs = options[:build_jobs] @gem_home = @install_dir || user_install_dir || Gem.dir @@ -803,7 +804,7 @@ def windows_stub_script(bindir, bin_file_name) # configure scripts and rakefiles or mkrf_conf files. def build_extensions - builder = Gem::Ext::Builder.new spec, build_args, Gem.target_rbconfig + builder = Gem::Ext::Builder.new spec, build_args, Gem.target_rbconfig, n_jobs: build_jobs builder.build_extensions end @@ -941,6 +942,10 @@ def build_args end end + def build_jobs + @build_jobs ||= Etc.nprocessors + 1 + end + def rb_config Gem.target_rbconfig end

Thank you, that make sense to me. I originally went with this approach as I wanted to avoid having a large diff and modify the signature of all builders. But I agree it adds complexity.

Regarding passing the -j to make command line, do you think it's ok if it takes precedence over MAKEFLAGS? If for instance a user explicitly sets MAKEFLAGS=-j6 it would be overriden by the -j value.

Regarding passing the -j to make command line, do you think it's ok if it takes precedence over MAKEFLAGS? If for instance a user explicitly sets MAKEFLAGS=-j6 it would be overriden by the -j value.

No. I think MAKEFLAGS provided by users should be used.

In my example code, this part handles this case:

+ if !have_make_arguments and !ENV["MAKEFLAGS"] and n_jobs + make_program << "-j#{n_jbos}" + end

If MAKEFLAGS is provided, -j isn't added automatically.

Ah right, sorry I missed that. Ok sounds great !

kou · 2025-12-09T02:28:19Z

lib/rubygems/installer.rb


  def build_extensions
-    builder = Gem::Ext::Builder.new spec, build_args, Gem.target_rbconfig
+    builder = Gem::Ext::Builder.new spec, build_args, Gem.target_rbconfig, build_jobs


It's not a related to this PR but it seems that we should use rb_config instead of Gem.target_rbconfig here.

Edouard-chin · 2025-12-10T22:38:26Z

I made the required change to be able to pass the BUNDLE_JOBS down to make as well as adding a new gem instal -j option flag.

This PR is already pretty big, and if that's ok I'd like to implement the jobserver in a separate patch.

kou

+1

lib/rubygems/ext/configure_builder.rb

colby-swandale · 2025-12-12T01:32:02Z

lib/rubygems/install_update_options.rb

+               "Defaults to the number of processors.",
+               "This option is only useful when installing gems",
+               "with native extensions.") do |value, options|
+      options[:build_jobs] = value


It seems this option is basically ignored if the user is on a windows computer correct? We should probably print a warning to let the user know about this.

it seems this option is basically ignored if the user is on a windows computer correct?

Its only ignored for users on windows using Ruby built for the mswin platform (compiled using Microsoft's Visual C++).

For ruby built on more modern windows toolchain like msys2 (where the platform is different and called mingw-*), Bundler will continue using make instead of nmake and therefore we can use the parallel feature. For instance on RG/Bundler windows CI, this works fine

lib/rubygems/install_update_options.rb

- ### Problem Since ruby#9131, we are now compiling make rules simultaneously. The number of jobs is equal to the number of processors. This may be problematic for some users as they want to control this value. ### Solution The number of jobs passed to `make` will now be equal to the `BUNDLE_JOBS` value. ### Side note It's also worth to note that since Bundler installs gems in parallel, we may end up running multiple `make -j<JOB>` in parallel which would cause exhaust the number of processors we have. This problem can be fixed by implementing a GNU jobserver, which I plan to do. But I felt that this would be too much change in one PR.

- Added a new `-j` option to `gem install` and `gem update`. This option allows to specify the number of jobs we pass to `make` when compiling gem with native extensions. By default its the number of processors, but users may want a way to control this. You can use it like so: `gem install json -j8`

Edouard-chin commented Dec 8, 2025

View reviewed changes

Edouard-chin force-pushed the ec-make-jobs-configurable branch 2 times, most recently from b3479fb to a0d5022 Compare December 8, 2025 23:31

hsbt added the rubygems: bug fix label Dec 8, 2025

kou reviewed Dec 9, 2025

View reviewed changes

Edouard-chin mentioned this pull request Dec 9, 2025

Running make with -j causes issues in docker environments #9170

Open

Edouard-chin force-pushed the ec-make-jobs-configurable branch 3 times, most recently from 2d04106 to e1fe5b8 Compare December 10, 2025 22:04

kou approved these changes Dec 11, 2025

View reviewed changes

Edouard-chin changed the title ~~Pass down value of BUNDLE_JOBS to RubyGems before compiling:~~ Pass down value of BUNDLE_JOBS to RubyGems before compiling & introduce a new gem install -j flag Dec 11, 2025

colby-swandale reviewed Dec 12, 2025

View reviewed changes

Edouard-chin force-pushed the ec-make-jobs-configurable branch from e1fe5b8 to 58cecb0 Compare December 12, 2025 16:22

Edouard-chin added 2 commits December 12, 2025 17:50

Edouard-chin force-pushed the ec-make-jobs-configurable branch from 58cecb0 to b3fb981 Compare December 12, 2025 16:50

Uh oh!

Pass down value of BUNDLE_JOBS to RubyGems before compiling & introduce a new gem install -j flag #9171

Are you sure you want to change the base?

Pass down value of BUNDLE_JOBS to RubyGems before compiling & introduce a new gem install -j flag #9171

Conversation

Edouard-chin commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was the end-user or developer problem that led to this PR?

What is your fix for the problem, implemented in this PR?

Make sure the following tasks are checked

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Edouard-chin Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Edouard-chin commented Dec 10, 2025

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Edouard-chin Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Pass down value of `BUNDLE_JOBS` to RubyGems before compiling & introduce a new `gem install -j` flag #9171

Pass down value of `BUNDLE_JOBS` to RubyGems before compiling & introduce a new `gem install -j` flag #9171

Edouard-chin commented Dec 8, 2025 •

edited

Loading

Edouard-chin Dec 9, 2025 •

edited

Loading

Edouard-chin Dec 12, 2025 •

edited

Loading