Yes, I agree to what you said. Speaking is just one sub-part of the whole test. I've never been living in an English speaking country, but just had experience working with some of those who did stay there for a number of years for their degree, and found that their spoken English was horrible...