All libinput devices share the same file descriptor so make sure we don't remove from the server's select loop until the last device is disabled. Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>