a performance issue for epoll idle

[lihuiba]

I came across a performance issue in epoll mode, when there were thousands concurrent connections. Profiling shows that _st_epoll_dispatch() consumed a lot of CPU.

After reviewing the function, I think I've found the reason: there's a loop that enumerates ALL threads in the I/O queue.

for (q = _ST_IOQ.next; q != &_ST_IOQ; q = q->next) {

As I'm using one thread per connection model, I believe this loop make epoll mode degraded effectively to select mode.

[winlinvip]

I find the code in

state-threads/event.c

Line 1281 in d04dcb5

for (q = _ST_IOQ.next; q != &_ST_IOQ; q = q->next) {

When there are some epoll events(the nfd >0), it will iterate the _ST_IOQ, a loop link list. The _ST_IOQ is the waiting queue, fd will push in queue in st_poll in

state-threads/sched.c

Line 81 in d04dcb5

_ST_ADD_IOQ(pq);

I don't know why need to iterate all blocked fds? How to fix this?

[winlinvip]

It seems to find the thread which is waiting on fd, and switch to the thread.

[lihuiba]

Yes it does, but I don't think it's necessary.

[lihuiba]

I believe we can store thread info in _st_epoll_data[], and resume the threads without iterating the IOQ.

[xiaosuo]

I do think there aren't many users now, so nobody cares.

[lihuiba]

I'm a heavy user of ST. Please make sure there is a problem, and I'll arrange a patch.

[xiaosuo]

Yes. It is a issue. But I am afraid that your way doesn't work. There maybe more threads waiting on a fd, and there isn't any reference to the monitoring fds. I think you'd better extend _epoll_fd_data, and record _st_pollq in _epoll_fd_data when inserting it into IOQ.

BTW: Maybe EPOLLONESHOT and EPOLLET can be used to get more better performance.

[winlinvip]

For performance issue, please write a test example and do some benchmarks, we SHOULD NEVER guess about it.

[xiaosuo]

FYI: I just cooked a patch for this issue, and will submit a PR after testing.

@winlinvip Is there any unit testing case?

[xiaosuo]

See #5 @lihuiba @winlinvip

[winlinvip]

@xiaosuo Thanks for your work and PR, that's great. And can you write a example code, which illustrate the problem? For example, we can write a program, it use 90% CPU or there is a performance report indicates the bottleneck. Then we patch ST with your PR, it use less CPU or the performance report finger it out.

Please write a benchmark to prove the PR really fixed the problem.

[xiaosuo]

@lihuiba reported this issue. I think he will test the PR with his workload.

I tested the server in the example directory, and ab reported slight improvements on both QPS and latency.

[lihuiba]

@winlinvip @xiaosuo I found the issue in my production server, where there were lots of idle connections. The server consumed several times more CPU resource than expected. I believe the situation could be easily reproduced, and I'll try the patch in a test environment, before launching it to my production server.

[winlinvip]

@lihuiba Please test it, thanks~ If you can write a simple test example, it will be better, because it's necessary for any performance issue.

[xiaosuo]

@lihuiba what is the result？