X provides the basic framework, or primitives, for building such GUI environments: drawing and moving windows on the display and interacting with a mouse, keyboard or touchscreen.

The X server is typically the provider of graphics resources and keyboard/mouse events to X clients, meaning that the X server is usually running on the computer in front of a human user, while the X client applications run anywhere on the network and communicate with the user's computer to request the rendering of graphics content and receive events from input devices including keyboards and mice.

This approach allows both 2D and (through extensions like GLX) 3D operations by an X client application which might be running on a different computer to still be fully accelerated on the X server's display.

For example, in classic OpenGL (before version 3.0), display lists containing large numbers of objects could be constructed and stored entirely in the X server by a remote X client program, and each then rendered by sending a single glCallList(which) across the network.

[4] The server accepts requests for graphical output (windows) and sends back user input (from keyboard, mouse, or touchscreen).

Practical examples of remote clients include: X primarily defines protocol and graphics primitives – it deliberately contains no specification for application user-interface design, such as button, menu, or window title-bar styles.

[6] Instead, application software – such as window managers, GUI widget toolkits and desktop environments, or application-specific graphical user interfaces – define and provide such details.

Commercial Unix vendors have tended to take the reference implementation and adapt it for their hardware, usually customizing it and adding proprietary extensions.

Microsoft Windows is not shipped with support for X, but many third-party implementations exist, as free and open source software such as Cygwin/X, and proprietary products such as Exceed, MKS X/Server, Reflection X, X-Win32 and Xming.

X terminals explore the network (the local broadcast domain) using the X Display Manager Control Protocol to generate a list of available hosts that are allowed as clients.

Dedicated (hardware) X terminals have fallen out of use; a PC or modern thin client with an X server typically provides the same functionality at the same, or lower, cost.

The Inter-Client Communication Conventions Manual (ICCCM), a specification for client interoperability, has a reputation for being difficult to implement correctly.

Like all thin clients, when using X across a network, bandwidth limitations can impede the use of bitmap-intensive applications that require rapidly updating large portions of the screen with low latency, such as 3D animation or photo editing.

Even a relatively small uncompressed 640×480×24 bit 30 fps video stream (~211 Mbit/s) can easily outstrip the bandwidth of a 100 Mbit/s network for a single client.

Most of the overhead comes from network round-trip delay time between client and server (latency) rather than from the protocol itself: the best solutions to performance issues depend on efficient application design.

Historical alternatives include Sun's NeWS and NeXT's Display PostScript, both PostScript-based systems supporting user-definable display-side procedures, which X lacked.

Carnegie Mellon University produced a remote-access application called Alto Terminal, that displayed overlapping windows on the Xerox Alto, and made remote hosts (typically DEC VAX systems running Unix) responsible for handling window-exposure events and refreshing window contents as necessary.

Demonstrations of the first commercial application for X (a mechanical computer-aided engineering system from Cognition Inc. that ran on VAXes and remotely displayed on PCs running an X server ported by Jim Fulton and Jan Hardenbergh) took place at the Autofact trade show at that time.

Attempts were made to enable X servers as real-time collaboration devices, much as Virtual Network Computing (VNC) would later allow a desktop to be shared.

[22] The X11 protocol design, led by Scheifler, was extensively discussed on open mailing lists on the nascent Internet that were bridged to USENET newsgroups.

Gettys moved to California to help lead the X11 development work at WSL from DEC's Systems Research Center, where Phil Karlton and Susan Angebrandt led the X11 sample server design and implementation.

Gettys joined the design team for the VAXstation 2000 to ensure that X—which DEC called DECwindows—would run on it, and the company assigned 1,200 employees to port X to both Ultrix and VMS.

In January 1988, the MIT X Consortium formed as a non-profit vendor group, with Scheifler as director, to direct the future development of X in a neutral atmosphere inclusive of commercial and educational interests.

Donna Converse, Chris D. Peterson, and Stephen Gildea joined later that year, focusing on toolkits and widget sets, working closely with Ralph Swick of MIT Project Athena.

Controversially, X11R6.4 departed from the traditional liberal licensing terms, as the Open Group sought to assure funding for the development of X, and specifically cited XFree86 as not significantly contributing to X.

[33] In 1999, the XFree86 team joined X.Org as an honorary (non-paying) member,[34] encouraged by various hardware companies[35][failed verification] interested in using XFree86 with Linux and in its status as the most popular version of X.

[39] In March 2003, the XFree86 organization expelled Keith Packard, who had joined XFree86 after the end of the original MIT X Consortium, with considerable ill feeling.

[48] Other groups saw it as against the spirit of the original X. Theo de Raadt of OpenBSD, for instance, threatened to fork XFree86 citing license concerns.

The Foundation takes an oversight role over X development: technical decisions are made on their merits by achieving rough consensus among community members.

In the early part of the 2000s, however, it was overhauled to resolve a number of problems that had surfaced over the years, including a "flawed" font architecture, a 2D graphics system "which had always been intended to be augmented and/or replaced", and latency issues.