nvmet-rdma: allocate RW ctxs according to mdts

Current nvmet-rdma code allocates MR pool budget based on queue size,
assuming both host and target use the same "max_pages_per_mr" count.
After limiting the mdts value for RDMA controllers, we know the factor
of maximum MR's per IO operation. Thus, make sure MR pool will be
sufficient for the required IO depth and IO size.

That is, say host's SQ size is 100, then the MR pool budget allocated
currently at target will also be 100 MRs. But 100 IO WRITE Requests
with 256 sg_count(IO size above 1MB) require 200 MRs when target's
"max_pages_per_mr" is 128.

Reported-by: Krishnamraju Eraparaju <krishna2@chelsio.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Max Gurtovoy <maxg@mellanox.com>

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index f47a79b..9e1b8c6 100644 (file)
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -978,7 +978,7 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
 {
        struct ib_qp_init_attr qp_attr;
        struct nvmet_rdma_device *ndev = queue->dev;
-       int comp_vector, nr_cqe, ret, i;
+       int comp_vector, nr_cqe, ret, i, factor;
 
        /*
         * Spread the io queues across completion vectors,
@@ -1011,7 +1011,9 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
        qp_attr.qp_type = IB_QPT_RC;
        /* +1 for drain */
        qp_attr.cap.max_send_wr = queue->send_queue_size + 1;
-       qp_attr.cap.max_rdma_ctxs = queue->send_queue_size;
+       factor = rdma_rw_mr_factor(ndev->device, queue->cm_id->port_num,
+                                  1 << NVMET_RDMA_MAX_MDTS);
+       qp_attr.cap.max_rdma_ctxs = queue->send_queue_size * factor;
        qp_attr.cap.max_send_sge = max(ndev->device->attrs.max_sge_rd,
                                        ndev->device->attrs.max_send_sge);