Java doesn't resolve remote dns properly and how to deal with it
I don't think that this story will be short, but it's definitely interesting. So, i got a new project where i need to cover SOAP API with automated tests written on Java. As a start point i have URL to WSDL file to work with. The "tricky" part is the environment configuration. Our test environment is hidden behind SOCKS proxy server. So, if you want to work with test environment, you need to make your requests through that proxy, otherwise you'll "talk" to another environment that we don't need right now. Not usual configuration, but it had it's own reasons in the past for doing so. My first problem was to decide how to work with WSDL and in what direction my future framework would go. After some googling and trials, i've stopped on JAX-WS lib. It's handy because with wsimport maven task it can easily generate all needed classes and objects where your stubs will be held. Here's how it looks like in maven :
<plugin>
<group>org.jvnet.jax-ws-commons</groupid>
<artifact>jaxws-maven-plugin</artifactid>
<version>${jaxws.plugin.version}</version>
<executions>
<execution>
<id>{put_some_id}</id>
<phase>validate</phase>
<goals>
<goal>wsimport</goal>
</goals>
<configuration>
<!--<wsdldirectory>${project.basedir}srcwsdl</wsdlDirectory>-->
<wsdllocation>{your_wsdl_location}</wsdllocation>
<destdir>${project.basedir}srcmainjavaStubsoop</destdir>
<wsdlfiles>
<wsdlfile>${project.basedir}
srcmainresourceswsdlmyWSDL.wsdl
</wsdlfile>
</wsdlfiles>
<packagename>Stubs.oop</packagename>
<!-- Without this, multiple WSDLs won't be processed :( -->
<stalefile>${project.build.directory}/jaxws/stale/wsdl.done
</stalefile>
</configuration>
</execution>
<execution>
<id>{put_some_id}</id>
<phase>validate</phase>
<goals>
<goal>wsimport</goal>
</goals>
<configuration>
<!--<wsdldirectory>${project.basedir}srcwsdl</wsdlDirectory>-->
<wsdllocation>{your_wsdl_location}</wsdllocation>
<destdir>${project.basedir}srcmainjavaStubswsdl2</destdir>
<wsdlfiles>
<wsdlfile>${project.basedir}
srcmainresourceswsdlyour_wsld2.wsdl</wsdlfile>
</wsdlfiles>
<packagename>your_package_name</packagename>
<!-- Without this, multiple WSDLs won't be processed :( -->
<stalefile>${project.build.directory}/jaxws/stale/wsdl.done</stalefile>
</configuration>
</execution>
</executions>
</plugin>
It was a bit hard for me to get WSDL from URL, so i decided to save it locally and read it from there then. Since changes are not being made frequently in API, i can easily accept this. This configuration allows to add as many WSDL sources as i want and to store stubs for each of them in separate packages in the project. First problem was solved. Now, i had to figure out how to put my requests through Proxy . My first trial was basic and most common in Java world:
System.getProperties().put("proxySet", "true");
System.getProperties().put("proxyHost", getProxyHost());
System.getProperties().put("proxyPort", getProxyPort());
But i kept getting errors and no successful connection. My next guess was that problem could be in SSL , since we are using https. So, i went into the browser, imported security certificate ( they were self-signed since it's a test environment), added them to java cacerts keystore and added next to my code :
System.setProperty("javax.net.ssl.keyStore", "keystore.jks");
System.setProperty("javax.net.ssl.trustStrore", "cacerts.jks");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
I tried to debug and to make sure that JVM is using my proxy settings and it was true - everything was there. What would be the next step in debugging for me ? I decided to install local proxy and try to put my requests through it, so i could get more details. I've installed Charles- a fantastic proxy server that helped me a lot in the past with REST API automation. It has a life changing ( for me in this case) "External Proxy" option . I've set it to forward all requests to our SOCKS proxy and then told to my tests to make calls through Charles. And it worked! You can't even believe how happy i was because of it:) But now i got another problem - Charles cost 50$ for one license:) So, there were 2 questions that i needed to solve :
- Why my Java calls works fine with Charles and without it they fail?
- Where to find free and easy to setup http proxy with forwarding possibility?:)
With help from one of our developers we could find out why Java kept getting authentification errors and didn't want to connect to test environment properly. The problem was that java was connecting to SOCKS proxy properly, but it was resolving remote dns to another environment to which i didn't have proper certificates and credentials for authorization! I found only one article on stackoverflow regarding this issue and from that point i new that i need to have local http proxy that is :
- free
- can forward my requests to SOCKS proxy
I've spent whole day trying to find some easy to use and adequate proxy server for Windows ( it's my Workstation that i need to use) , looked over 15 different apps and still couldn't find anything that i could use. In the end of the day i've installed VirtualBox with Ubuntu and installed there squid3 proxy server. It's the most popular and common proxy server for Unix, from what i saw on Google. But, this crazy proxy has configuration file with more then 3000 lines!! It's not that easy to make it run as i want, it's not even that easy to restart it. So, after couple hours i gave up on it and started looking for alternatives. Luckily, i found Polipo - tiny, easy to install and setup proxy server. It has few main options that i had to setup and make everything work as a charm :
- proxyAddress - set it to ip address of your local virtual box, or your local machine, if you use linux or mac
- allowedClients - list of ip addresses from which polipo will allow to access it and forward requests towards another proxy or web directly
- socksParentProxy = "host name and port of your proxy to which i needed to forward my requests"
- socksProxyType = socks5
Save changes and restart - that's it! After that i pointed my Java framework to local proxy and got green tests! To set proxy for my tests i used custom MyProxySelector class :
package Base;
import java.net.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.io.IOException;
public class MyProxySelector extends ProxySelector {
// Keep a reference on the previous default
public ProxySelector defsel = null;
/**
* Inner class representing a Proxy and a few extra data
*/
class InnerProxy {
Proxy proxy;
SocketAddress addr;
// How many times did we fail to reach this proxy?
int failedCount = 0;
InnerProxy(InetSocketAddress a) {
addr = a;
proxy = new Proxy(Proxy.Type.HTTP, a);
}
SocketAddress address() {
return addr;
}
Proxy toProxy() {
return proxy;
}
int failed() {
return ++failedCount;
}
}
/**
* A list of proxies, indexed by their address.
*/
HashMap<socketAddress, InnerProxy> proxies = new HashMap<socketAddress,
InnerProxy>();
public MyProxySelector(ProxySelector def, String host, int port) {
// Save the previous default
defsel = def;
// Populate the HashMap (List of proxies)
InnerProxy i = new InnerProxy(new InetSocketAddress(host, port));
proxies.put(i.address(), i);
}
/**
* This is the method that the handlers will call.
* Returns a List of proxy.
*/
public java.util.List<proxy> select(URI uri) {
// Let's stick to the specs.
if (uri == null) {
throw new IllegalArgumentException("URI can't be null.");
}
/**
* If it's a http (or https) URL, then we use our own
* list.
*/
String protocol = uri.getScheme();
if ("http".equalsIgnoreCase(protocol) ||
"https".equalsIgnoreCase(protocol)) {
ArrayList<proxy> l = new ArrayList<proxy>();
for (InnerProxy p : proxies.values()) {
l.add(p.toProxy());
}
return l;
}
/**
* Not HTTP or HTTPS (could be SOCKS or FTP)
* defer to the default selector.
*/
if (defsel != null) {
return defsel.select(uri);
} else {
ArrayList<proxy> l = new ArrayList<proxy>();
l.add(Proxy.NO_PROXY);
return l;
}
}
/**
* Method called by the handlers when it failed to connect
* to one of the proxies returned by select().
*/
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
// Let's stick to the specs again.
if (uri == null || sa == null || ioe == null) {
throw new IllegalArgumentException("Arguments can't be null.");
}
/**
* Let's lookup for the proxy
*/
InnerProxy p = proxies.get(sa);
if (p != null) {
/**
* It's one of ours, if it failed more than 3 times
* let's remove it from the list.
*/
if (p.failed() >= 3)
proxies.remove(sa);
} else {
/**
* Not one of ours, let's delegate to the default.
*/
if (defsel != null)
defsel.connectFailed(uri, sa, ioe);
}
}
}
And to turn proxy on and off i wrote next switch methods :
private ProxySelector defaultProxy = ProxySelector.getDefault();
public void setLocalProxy(){
MyProxySelector ps = new MyProxySelector(ProxySelector.getDefault(),
localProxyName,localProxyPort);
ProxySelector.setDefault(ps);
}
public void disableProxy(){
ProxySelector.setDefault(defaultProxy);
}
That's it. Now i can run my tests easily with control of when to use proxy and when not. In the future i'll move my tests to CI ( Jenkins most probably) and will setup on that environment Polipo proxy in 2 minutes. It's always nice to solve such non-ordinary problems. Most probably my solution is not very elegant and "right" , but it works right now and for this moment this what matters for me, since i can start writing automated tests rather then fighting with this configuration issues.
Update . One of my colleagues proposed me to set next settings for Java :
System.setProperty("sun.net.spi.nameservice.nameservers", "ip_here");
System.setProperty("sun.net.spi.nameservice.provider.1", "dns,sun");
But it didn't work for me. So will stick with my current solution for now.