The issue: Sometimes, when you connect to a 11gR1 database remotely (or locally using TCP), it takes a very long time until the connection is established. Once connected, everything works fine.
The cause: The DNS servers listed in the database server configuration are taking too long to respond or are unavailable from the database server perspective.
The catch: There is one significant change in behavior from 10g to 11g. According to Metalink notes 561429.1 and 856820.1, in 11gR1 the listener code that checks DNS for every incoming connection was changed and it will wait for the DNS answer anywhere from 10 seconds to 2 minutes. But why would the database need to wait for this anyway?
The solution: Apply patch 9593134 or upgrade to 11gR2 (184.108.40.206 at least), where the code was again changed, so it no longer waits too long for DNS. And if possible, fix your DNS infrastructure also.