There are those people in the gym who fly from one exercise to another with no longer in between the sets than it takes them to get from one machine to the other. Then there are those who talk for 5 minutes before they move on. Both are probably correct for the individual.
There is no "one time fits all" any more than there is a "one program fits all". From the hyperactive to the lethargic each of us has a built in rhythm that determines the pace of our life. The workout should flow easily with that rhythm. Whatever the pace of your day before you started the workout should be the pace of the workout for most people. Your body should feel ready to be able to perform the next set well. Certain workouts will by their very nature require differing amounts of rest.
When I was Olympic and power lifting we would perform a maximum poundage lift for one rep and then sit and rest for several minutes before we recuperated enough to try again. In my bodybuilding workouts I prefer to train with two partners so that my turn doesn't come around quite so fast as with only one partner.
Find a pace you are comfortable with and that allows you to finish the workout. An endurance workout should be performed with shorter rest periods than a strength or weight gain workout. Strength and weight gain require using as much weight as you can handle in good form for a predetermined number of reps, usually around 8 to 10, and constantly increasing the poundage at every opportunity. A weight loss workout will get better results from shorter rest periods which will force you to use lighter weights in order to recuperate enough between sets to keep the faster pace.
But no matter what the goal of the particular workout, you should be comfortable with it.