In some cases (e.g. using x2x) the previous model broke, with a window ending
not counting down to 0 anymore. Special treatment for NotifyUngrab seems to
help here.
Current solution: If a NotifyGrab is to be sent ignore it. If a NotifyUngrab
enter is sent, only adjust the semaphore if it is on 0. Likewise, do the same
for a NotifyUngrab leave if the semaphore is on 1. This seems to work alright
so far.