Handling brute force attacks in the kernel

A number of different attacks against Linux systems rely on brute-force techniques using the fork() system call, so a new Linux security module (LSM), called "Brute", has been created to detect and thwart such attacks. Repeated fork() calls can be used for various types of attacks, such as exploiting the Stack Clash vulnerability or Heartbleed-style flaws. Version 6 of the Brute patch set was recently posted and looks like it might be heading toward the mainline.

This patch set has been in the works since it was first posted as an RFC by John Wood in September 2020 (the resend from Kees Cook a few days later may make it easier to see the whole set). It was originally called "fork brute force attack mitigation" or "fbfam", but that name was deemed too cryptic by Jann Horn and Cook. In addition, it was suggested that turning it into an LSM would be desirable. Both of those suggestions were adopted in version 2, which was posted in October.

But the idea goes back a lot further than that. The grsecurity kernel patches have long had the GRKERNSEC_BRUTE feature to mitigate brute-force exploits of server programs that use fork() as well as exploits of setuid/setgid binaries. A patch from Richard Weinberger in 2014 used a similar technique to delay fork() calls if forked children die due to a fatal error (which may imply it is part of an attack). That effort was not pushed further, so Cook added an issue to the kernel self-protection project (KSPP) GitHub repository, which is where Wood picked up the idea.

In the documentation patch, Wood described the kinds of behaviors that are being targeted by the Brute LSM. The basic idea is that there are several types of attacks that can use fork() multiple times in order to receive a desired memory layout; each child forked can be probed in various ways, if those probes fail and cause the child to crash, another child can simply be forked to try again. Because the child created with fork() shares the same memory layout as the parent, successful probes can give information that can be used to defeat address-space layout randomization (ASLR), determine the value of stack canaries, or for other nefarious purposes.

Brute takes a different approach than either grsecurity or Weinberger's patch did, in that it does not simply delay subsequent fork() calls when a problem is detected. Instead, Brute kills all of the processes associated with the attack. In addition, Brute detects more types of fork()-using attacks, including those that probe the parent, rather than the child process. It also focuses on crashes in processes that have crossed a privilege boundary to try to reduce the number of false positives.

It does its detection by focusing on the rate of crashes, rather than just their occurrence. Brute collects information on the number of "faults" that occur in a group of processes that have been forked, but where nothing new has been executed with execve(). A brute_stats structure is shared between all of those processes; executing a new program results in a new structure to track faults in the new (potential) fork() hierarchy.

The period of time between a process being started and it or any of its children that share its memory layout (i.e. no execve()) have crashed, or between consecutive crashes, is ultimately what is being used to determine if an attack is taking place. But in order to not be too sensitive, the exponential moving average (EMA) of the period is calculated once five crashes have occurred. The EMA is used to determine if a "fast brute force" attack variant is occurring; if the EMA of the period between crashes drops below a threshold of 30 seconds, attack mitigation is triggered. For "slow brute force" variants, the absolute number of crashes in the hierarchy is compared against a threshold of 200. Some way to configure these values would seem like a desirable addition to Brute.

The crashes are detected using the task_fatal_signal() LSM hook that was added as the first patch in the set. It will be called whenever the kernel is delivering a fatal signal to a process. Brute also uses the existing task_alloc() hook to detect fork() calls, the bprm_committing_creds() hook to detect execve() calls, and the task_free() hook to clean everything up.

The security boundary checks are implemented by tracking changes to the various user and group IDs (real, effective, saved, and filesystem) that occur when executing new programs. There is no mention of Linux capabilities in the patches, but capability changes would also indicate that a privilege boundary is being crossed; perhaps that is something that will be added down the road. Beyond the ID changes, the use of networking is detected using the socket_sock_rcv_skb() LSM hook. The idea is to restrict the crash checking to those processes that are crossing privilege boundaries by either doing things like executing setuid/setgid programs or receiving data over the network. That is intended to reduce the number of false positives.

As can be seen in the changelog in the top-level patch, the last few versions (which are helpfully linked) have drawn minimal comments needing attention; this latest round has not drawn any at all at the time of this writing. It seems like a useful feature for some users without imparting any real burden on the rest of the kernel when it is not configured in; the new security hook that gets called in the case of a fatal signal being delivered is the only change in that case. LSMs are often looked upon as a place to put code that some folks want, but others don't want to pay a price for in their kernels—Brute seems to fit that model well.