The API is a little misleading, the correct way to return an exception in the ResultSet is to use a DataException.
For instance you can do this in the getProperties() of your PropertyProviderAdapter:
import com.vmware.vise.data.query.DataException;
....
public ResultSet getProperties(PropertyRequestSpec propertyRequest) {
...
result.error = DataException.newInstance(new Exception("your message"),
propertyRequest.objects, propertyRequest.properties[0].propertyNames);
There is a small issue in 5.1 (to be fixed in 5.1 U1) is that we didn't publish com/vmware/vise/core/model/MutableMessageException in the SDK library and DataException is a subclass of MutableMessageException, so the compiler will complain.
The work-around is to add the library \server\repository\usr\java-model-core-1.0.0.jar to the project build to compile your code.