When you look at how fiercely the Tamil Tigers or Japanese fought you can realise that athletic build is not enough. The motivation for their cause and organisation were more important.
If athletic build was the most important factor then Sikhs, Pathans and blacks would be having great power. Such groups can not succeed in building modern empires on the basis of their physiques. You need to have brains, modern technology, etc. like the Israelies.