If you have a typical propane water heater, look for a valve in the plumbing directly in front of the water heater, probably in the back of a cabinet. Wherever your water heater is placed, the by-pass is probably not far away.
As far as shutting off your water pump, it should be off and stay off if any one of the switches is turned off. Not sure I understand the question.
As far as my system goes, the controller seems to have been the problem; the new one has been in service for over two weeks and the system has stayed on, so I’d say the operation was a success.