I tried to change more configuration in vmware to improve the latency.
1. Change from Broadcom iSCSI adapter to software iSCSI adapter and enable jumbo frames, however it seems that jumbo frames can be enabled on only one ethernet port of the SAN, which is 253.66.
2. Disable virtual interrupt coalescing on both hosts.
3. Use vmxnet3 instead of E1000 on all virtual machines.
4. Set power performance to High Performance on Application Server and Database Server.
5. Enable RSS (Receive Side Scaling) on Application Server and Database Server.
After these kind of changes, the warnings of latency increase still come up, but this happens less than before. However, I'm afraid that if these warnings happen many times, the system will crash again. I tried to replace the cables, but it seems the cables are not the cause. So, is there anything that I can do to improve the latency, and I just wonder why the latency is still so high on Database server. As I can see from the performance chart of the Database server, the write latency from the SAN can reach 600ms sometimes. Thanks a lot!