GTK supports the environment variable GTK_MODULES which specifies arbitrary dynamic modules to be loaded and executed when GTK is initialized. It is somewhat similar to the LD_PRELOAD environment variable. However, this (and similar functionality such as specifying theme engines) is not disabled when running setuid or setgid. Is this a security hole? No. Writing setuid and setgid programs using GTK is bad idea and will never be supported by the GTK team.
You should not write setuid GTK programs because:
GTK is too big. GTK+-2.0 and its dependent libraries (ignoring Xlib) total over 600,000 lines of code. For GTK+-3.0 (ignoring backend specific and image loading libraries), this figure is over 800000 lines of code.
GTK is too complex. GTK takes input from dozens of sources, from drag-and-drop, to root-window properties, to keyboard input, to configuration files. This is a much broader scope for compromises than a typical server and makes auditing GTK especially tricky.
Securing GTK requires the security of the underlying windowing system backend. The GTK team is not prepared to make that guarantee. Security bugs have been found in the recent past in such areas of Xlib as the input method code.
You should not make your GUI setuid at all. Why run the risk of security bugs in code that does not need to be running with elevated privileges?
In the opinion of the GTK team, the only correct way to write a setuid program with a graphical user interface is to have a setuid backend that communicates with the non-setuid graphical user interface via a mechanism such as a pipe and that considers the input it receives to be untrusted.
For this reason, no effort is made in GTK to disable the obvious ways that you could compromise a setuid GTK program - GTK_MODULES and the ability for the user to specify extensions to the toolkit, because we consider this to be only papering over the fundamental problems of writing setuid programs with any GUI toolkit. GTK may be modified in the future to simply refuse to run with elevated privileges, though it does not do this currently.
Does this mean that there are no security considerations for GTK? No. In particular image loaders have been and will continue to be an area of special care, since users may load images from untrusted sources. And in addition to the possibility of this variety of exploit, most potential security holes are essentially bugs and even as mere bugs, must be squashed. To help accomplish this goal, GTK extensively uses high-level data structure abstractions which minimize the risk of most traditional buffer overflows.
However, the secure setuid program is a 500 line program that does only what it needs to, rather than a 800,000 line library whose essential task is user interfaces.