randr: do full transform when checking SetScreenSize size

When validating the size passed to SetScreenSize, the CRTC mode size
needs to be applied the full CRTC transform, otherwise the check may
bogusly fail.

Do a full transform on the CRTC mode size when checking the
SetScreenSize request size instead of the current code that only
manually handles rotation.

Signed-off-by: Icenowy Zheng <uwu@icenowy.me>
Part-of: <https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/1971>
This commit is contained in:
Icenowy Zheng
2025-05-13 21:34:04 +08:00
committed by Enrico Weigelt, metux IT consult
parent 7f9a56390d
commit 692c3b4557

View File

@@ -266,17 +266,14 @@ ProcRRSetScreenSize(ClientPtr client)
RRModePtr mode = crtc->mode;
if (!RRCrtcIsLeased(crtc) && mode) {
int source_width = mode->mode.width;
int source_height = mode->mode.height;
Rotation rotation = crtc->rotation;
struct pixman_box16 display_box = {
crtc->x, crtc->y,
crtc->x + mode->mode.width,
crtc->y + mode->mode.height
};
pixman_f_transform_bounds(&crtc->f_transform, &display_box);
if (rotation & (RR_Rotate_90 | RR_Rotate_270)) {
source_width = mode->mode.height;
source_height = mode->mode.width;
}
if (crtc->x + source_width > stuff->width ||
crtc->y + source_height > stuff->height)
if (display_box.x2 > stuff->width || display_box.y2 > stuff->height)
return BadMatch;
}
}