Use the Keyboard Layout of the operator instead of the client. Even TightVNC can do this out of the box.