From b69e2ef24b7b4867f80f47e2781e95d0bacd15cb Mon Sep 17 00:00:00 2001 From: Keith Busch Date: Fri, 8 May 2020 13:04:06 -0700 Subject: [PATCH] nvme-pci: dma read memory barrier for completions Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads. Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion. Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic. Reported-by: John Garry Suggested-by: Will Deacon Signed-off-by: Keith Busch Tested-by: John Garry Acked-by: Will Deacon Reviewed-by: Sagi Grimberg Signed-off-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index e13c370de830..3726dc780d15 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq) while (nvme_cqe_pending(nvmeq)) { found++; + /* + * load-load control dependency between phase and the rest of + * the cqe requires a full read memory barrier + */ + dma_rmb(); nvme_handle_cqe(nvmeq, nvmeq->cq_head); nvme_update_cq_head(nvmeq); } -- 2.11.0