io_uring: don't delay iopoll'ed req completion

->iopoll() may have completed current request, but instead of reaping
it, io_do_iopoll() just continues with the next request in the list.
As a result it can leave just polled and completed request in the list
up until next syscall. Even outer loop in io_iopoll_getevents() doesn't
help the situation.

E.g. poll_list: req0 -> req1
If req0->iopoll() completed both requests, and @min<=1,
then @req0 will be left behind.

Check whether a req was completed after ->iopoll().

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 9464f94..60f1a81 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2015,6 +2015,10 @@ static int io_do_iopoll(struct io_ring_ctx *ctx, unsigned int *nr_events,
                if (ret < 0)
                        break;
 
+               /* iopoll may have completed current req */
+               if (READ_ONCE(req->iopoll_completed))
+                       list_move_tail(&req->list, &done);
+
                if (ret && spin)
                        spin = false;
                ret = 0;