randr: do full transform when checking SetScreenSize size

When validating the size passed to SetScreenSize, the CRTC mode size
needs to be applied the full CRTC transform, otherwise the check may
bogusly fail.

Do a full transform on the CRTC mode size when checking the
SetScreenSize request size instead of the current code that only
manually handles rotation.

Signed-off-by: Icenowy Zheng <uwu@icenowy.me>
Part-of: <https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/1971>
This commit is contained in:
Icenowy Zheng
2025-05-13 21:34:04 +08:00
committed by Enrico Weigelt, metux IT consult
parent a5741d057f
commit 7e77b5cdef

View File

@@ -266,17 +266,14 @@ ProcRRSetScreenSize(ClientPtr client)
RRModePtr mode = crtc->mode;
if (!RRCrtcIsLeased(crtc) && mode) {
int source_width = mode->mode.width;
int source_height = mode->mode.height;
Rotation rotation = crtc->rotation;
struct pixman_box16 display_box = {
crtc->x, crtc->y,
crtc->x + mode->mode.width,
crtc->y + mode->mode.height
};
pixman_f_transform_bounds(&crtc->f_transform, &display_box);
if (rotation & (RR_Rotate_90 | RR_Rotate_270)) {
source_width = mode->mode.height;
source_height = mode->mode.width;
}
if (crtc->x + source_width > stuff->width ||
crtc->y + source_height > stuff->height)
if (display_box.x2 > stuff->width || display_box.y2 > stuff->height)
return BadMatch;
}
}