Merge branch 'bpf-jit-overridable-alloc'

Ard Biesheuvel says:

====================
On arm64, modules are allocated from a 128 MB window which is close to
the core kernel, so that relative direct branches are guaranteed to be
in range (except in some KASLR configurations). Also, module_alloc()
is in charge of allocating KASAN shadow memory when running with KASAN
enabled.

This means that the way BPF reuses module_alloc()/module_memfree() is
undesirable on arm64 (and potentially other architectures as well),
and so this series refactors BPF's use of those functions to permit
architectures to change this behavior.

Patch #1 breaks out the module_alloc() and module_memfree() calls into
__weak functions so they can be overridden.

Patch #2 implements the new alloc/free overrides for arm64

Changes since v3:
- drop 'const' modifier for free() hook void* argument
- move the dedicated BPF region to before the module region, putting it
  within 4GB of the module and kernel regions on non-KASLR kernels

Changes since v2:
- properly build time and runtime tested this time (log after the diffstat)
- create a dedicated 128 MB region at the top of the vmalloc space for BPF
  programs, ensuring that the programs will be in branching range of each
  other (which we currently rely upon) but at an arbitrary distance from
  the kernel and modules (which we don't care about)

Changes since v1:
- Drop misguided attempt to 'fix' and refactor the free path. Instead,
  just add another __weak wrapper for the invocation of module_memfree()
====================

Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Rick Edgecombe <rick.p.edgecombe@intel.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Jann Horn <jannh@google.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Jessica Yu <jeyu@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: netdev@vger.kernel.org
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index b964429..ee20fc6 100644 (file)
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -62,8 +62,11 @@
 #define PAGE_OFFSET            (UL(0xffffffffffffffff) - \
        (UL(1) << (VA_BITS - 1)) + 1)
 #define KIMAGE_VADDR           (MODULES_END)
+#define BPF_JIT_REGION_START   (VA_START + KASAN_SHADOW_SIZE)
+#define BPF_JIT_REGION_SIZE    (SZ_128M)
+#define BPF_JIT_REGION_END     (BPF_JIT_REGION_START + BPF_JIT_REGION_SIZE)
 #define MODULES_END            (MODULES_VADDR + MODULES_VSIZE)
-#define MODULES_VADDR          (VA_START + KASAN_SHADOW_SIZE)
+#define MODULES_VADDR          (BPF_JIT_REGION_END)
 #define MODULES_VSIZE          (SZ_128M)
 #define VMEMMAP_START          (PAGE_OFFSET - VMEMMAP_SIZE)
 #define PCI_IO_END             (VMEMMAP_START - SZ_2M)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index b872859..0a7371a 100644 (file)
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -943,3 +943,16 @@ out:
                                           tmp : orig_prog);
        return prog;
 }
+
+void *bpf_jit_alloc_exec(unsigned long size)
+{
+       return __vmalloc_node_range(size, PAGE_SIZE, BPF_JIT_REGION_START,
+                                   BPF_JIT_REGION_END, GFP_KERNEL,
+                                   PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE,
+                                   __builtin_return_address(0));
+}
+
+void bpf_jit_free_exec(void *addr)
+{
+       return vfree(addr);
+}

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index f93ed66..86817ab 100644 (file)
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -623,6 +623,16 @@ static void bpf_jit_uncharge_modmem(u32 pages)
        atomic_long_sub(pages, &bpf_jit_current);
 }
 
+void *__weak bpf_jit_alloc_exec(unsigned long size)
+{
+       return module_alloc(size);
+}
+
+void __weak bpf_jit_free_exec(void *addr)
+{
+       module_memfree(addr);
+}
+
 struct bpf_binary_header *
 bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
                     unsigned int alignment,
@@ -640,7 +650,7 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
 
        if (bpf_jit_charge_modmem(pages))
                return NULL;
-       hdr = module_alloc(size);
+       hdr = bpf_jit_alloc_exec(size);
        if (!hdr) {
                bpf_jit_uncharge_modmem(pages);
                return NULL;
@@ -664,7 +674,7 @@ void bpf_jit_binary_free(struct bpf_binary_header *hdr)
 {
        u32 pages = hdr->pages;
 
-       module_memfree(hdr);
+       bpf_jit_free_exec(hdr);
        bpf_jit_uncharge_modmem(pages);
 }