If ssh -X fails with X11 forwarding request failed on channel 0 when connecting to an IPv4 only host, check your /etc/ssh/sshd_config for AddressFamily inet:

Specifies which address family should be used by sshd(8). Valid arguments are “any”, “inet” (use IPv4 only), or “inet6” (use IPv6 only). The default is “any”.