I agree! Yoga during first few weeks would actually help you as your baby grows inside and your back has to carry the weight of the child.
It also helps you open up your pelvis in preparation for birth.
I've read a great article about pregnancy yoga and the author is from Malaysia too!