XDMCP Session Failed for Display

When starting an Xdmcp session in X-Win32, if you get a message in X-Win32’s Messages window that says Cannot open display, then you may be having a problem with your reverse DNS lookups. Note: Xdmcp sessions may be used to connect to remote desktops, such as the Common Desktop Environment (CDE), Gnome, or KDE running on Red Hat Linux, SuSE, HP-UX, IBM’s AIX (RS/6000), BSD, or Solaris.

NOTE: You should confirm that you do not have a simple firewall blocking the display connection attempt on your Windows machine; please search the knowledge base for articles on firewall configuration.

To understand this problem, a little background on how an Xdmcp Query session works will help.

Background on Xdmcp connections and reverse DNS

  1. Your Windows machine sends a UDP packet to your remote UNIX, Linux, or BSD host
  2. The remote machine replies to the UDP packet, sending the reply to the address that it received the UDP packet from
  3. Some hand-shaking takes place:
    • If the remote host is not willing to manage the display, then it sends back a UDP packet saying that it is unwilling to manage the display; this error will be shown in X-Win32’s Messages window
    • If the remote host is willing to manage the display, then it tries to connect back to the address that it was sent by X-Win32, taking the steps below
  4. The remote host performs a reverse DNS lookup to convert the IP address that X-Win32 sent it into a host name; this converted address is now known as the display address (this is done so that your display address will be more meaningful than a simple IP address)
  5. The remote host then passes the display address (with a possibly incorrect host name) to the xlogin process
  1. The xlogin process then performs a DNS lookup on the display address and connects to the IP address that is returnedIt is immediately obvious on a network when DNS lookups are not functioning, as you simply cannot connect to hosts by their DNS names. However, it is much less obvious when reverse DNS lookups are misconfigured, as you very rarely need to convert an IP address into a host name, but Xdmcp connections are just one of the cases where failing reverse DNS lookups will have an effect.

    It is simple to confirm that this is the problem by adding an entry to the /etc/hosts file on your remote host. If the Xdmcp connection then succeeds, it means that reverse DNS lookups are not working correctly; the real solution is to fix the reverse DNS lookups, not to add entires to the /etc/hosts file on all of your Xdmcp hosts.

    Confirming that reverse DNS is at fault

  2. Using nslookup
    1. Get the IP address of your Windows machine via ipconfig in a Windows command prompt
    2. Login to the remote host (e.g. on the console or with ssh, etc.)
    3. Run nslookup
    4. Enter the IP address of your Windows machine and press Enter
    5. Note the host name returned
    6. Enter the host name returned and press Enter
    7. Note the IP address returned
    1. If the IP address of your Windows machine does not match the IP address returned in the previous step, then reverse DNS lookups are misconfigured on your network
    2. Using a temporary /etc/hosts entry
    3. Get the IP address of your Windows machine via ipconfig in a Windows command prompt
    4. Become root on the remote host
    5. Edit /etc/hosts
    6. Add a line to the file like the line below, but substituting your Windows machine’s IP address 
      192.168.0.1 any_host_name_will_do
    7. Save the file
    8. Attempt your Xdmcp connection again
  3. If your Xdmcp connection now works then it means that reverse DNS lookups are misconfigured on your network. If the connection still does not work, then reconfirm that you do not have a firewall blocking the connection to your X-Win32 machine.

    NOTE: Misconfigured reverse DNS entries will cause Xdmcp connections to fail; however, unconfigured reverse DNS will still work because the IP address will be passed to xlogin instead of the host name. This is different than rexec because rexec on some platforms will fail completely if reverse DNS is unconfigured or misconfigured.

Reverse DNS solution did not work

Check your Windows firewall.  Windows may block the XDMCP connection.  This is usually cause when Windows prompts to “Always Allow” when first starting a session with X-Win32 and the user declines the prompt.