A good tip off that the tempo is wrong is if one or more of the players is having trouble keeping time. If they are drifting around the time, there is a good chance they are hearing the song slightly slower or faster. It can be a big help to have a metronome for this so you can be sure exactly how much you are adjusting the tempo, even if you are not using a click on the final recording.
It can also be helpful to test existing recordings of a song with a program like the Amazing Slowdowner which allow you to keep the pitch the same but speed up or slow down a recording.
Background image by Pauline Moss. at http://paulinemoss.deviantart.com/art/Stars-307119758