Counting Processor Cores in Emacs

One of the great advantages of dependency analysis is parallelization. Modern processors reorder instructions whose results don’t affect each other. Compilers reorder expressions and statements to improve throughput. Build systems know which outputs are inputs for other targets and can choose any arbitrary build order within that constraint. This article involves the last case.

The build system I use most often is GNU Make, either directly or indirectly (Autoconf, CMake). It’s far from perfect, but it does what I need. I almost always invoke it from within Emacs rather than in a terminal. In fact, I do it so often that I’ve wrapped Emacs’ compile command for rapid invocation.

I recently helped a co-worker set this set up for himself, so it had me thinking about the problem again. The situation in my config is much more complicated than it needs to be, so I’ll share a simplified version instead.

First bring in the usual goodies (we’re going to be making closures):

;;; -*- lexical-binding: t; -*-
(require 'cl-lib)

We need a couple of configuration variables.

(defvar quick-compile-command "make -k ")
(defvar quick-compile-build-file "Makefile")

Then a couple of interactive functions to set these on the fly. It’s not strictly necessary, but I like giving each a key binding. I also like having a history available via read-string, so I can switch between a couple of different options with ease.

(defun quick-compile-set-command (command)
  (interactive
   (list (read-string "Command: " quick-compile-command)))
  (setf quick-compile-command command))

(defun quick-compile-set-build-file (build-file)
  (interactive
   (list (read-string "Build file: " quick-compile-build-file)))
  (setf quick-compile-build-file build-file))

Now finally to the good part. Below, quick-compile is a non-interactive function that returns an interactive closure ready to be bound to any key I desire. It takes an optional target. This means I don’t use the above quick-compile-set-command to choose a target, only for setting other options. That will make more sense in a moment.

(cl-defun quick-compile (&optional (target ""))
  "Return an interaction function that runs `compile' for TARGET."
  (lambda ()
    (interactive)
    (save-buffer)  ; so I don't get asked
    (let ((default-directory
            (locate-dominating-file
             default-directory quick-compile-build-file)))
      (if default-directory
          (compile (concat quick-compile-command " " target))
        (error "Cannot find %s" quick-compile-build-file)))))

It traverses up (down?) the directory hierarchy towards root looking for a Makefile — or whatever is set for quick-compile-build-file — then invokes the build system there. I don’t believe in recursive make.

So how do I put this to use? I clobber some key bindings I don’t otherwise care about. A better choice might be the F-keys, but my muscle memory is already committed elsewhere.

(global-set-key (kbd "C-x c") (quick-compile)) ; default target
(global-set-key (kbd "C-x C") (quick-compile "clean"))
(global-set-key (kbd "C-x t") (quick-compile "test"))
(global-set-key (kbd "C-x r") (quick-compile "run"))

Each of those invokes a different target without second guessing me. Let me tell you, having “clean” at the tip of my fingers is wonderful.

Parallel Builds

An extension common to many different make programs is -j, which asks make to build targets in parallel where possible. These days where multi-core machines are the norm, you nearly always want to use this option, ideally set to the number of logical processor cores on your system. It’s a huge time-saver.

My recent revelation was that my default build command could be better: make -k is minimal. It should at least include -j, but choosing an argument (number of processor cores) is a problem. Today I use different machines with 2, 4, or 8 cores, so most of the time any given number will be wrong. I could use a per-system configuration, but I’d rather not. Unfortunately GNU Make will not automatically detect the number of cores. That leaves the matter up to Emacs Lisp.

Emacs doesn’t currently have a built-in function that returns the number of processor cores. I’ll need to reach into the operating system to figure it out. My usual development environments are Linux, Windows, and OpenBSD, so my solution should work on each. I’ve ranked them by order of importance.

Number of cores on Linux

Linux has the /proc virtual filesystem in the fashion of Plan 9, allowing different aspects of the system to be explored through the standard filesystem API. The relevant file here is /proc/cpuinfo, listing useful information about each of the system’s processors. To get the number of processors, count the number of processor entries in this file. I’ve wrapped it in if-file-exists so that it returns nil on other operating systems instead of throwing an error.

(when (file-exists-p "/proc/cpuinfo")
  (with-temp-buffer
    (insert-file-contents "/proc/cpuinfo")
    (how-many "^processor[[:space:]]+:")))

Number of cores on Windows

When I was first researching how to do this on Windows, I thought I would need to invoke the wmic command line program and hope the output could be parsed the same way on different versions of the operating system and tool. However, it turns out the solution for Windows is trivial. The environment variable NUMBER_OF_PROCESSORS gives every process the answer for free. Being an environment variable, it will need to be parsed.

(let ((number-of-processors (getenv "NUMBER_OF_PROCESSORS")))
  (when number-of-processors
    (string-to-number number-of-processors)))

Number of cores on BSD

This seems to work the same across all the BSDs, including OS X, though I haven’t yet tested it exhaustively. Invoke sysctl, which returns an undecorated number to be parsed.

(with-temp-buffer
  (ignore-errors
    (when (zerop (call-process "sysctl" nil t nil "-n" "hw.ncpu"))
      (string-to-number (buffer-string)))))

Also not complicated, but it’s the heaviest solution of the three.

Putting it all together

Join all these together with or, call it numcores, and ta-da.

(setf quick-compile-command (format "make -kj%d" (numcores)))

Now make is invoked correctly on any system by default.

Have a comment on this article? Start a discussion in my public inbox by sending an email to ~skeeto/public-inbox@lists.sr.ht [mailing list etiquette] , or see existing discussions.

null program

Chris Wellons

wellons@nullprogram.com (PGP)
~skeeto/public-inbox@lists.sr.ht (view)