I would definitely say the media is a huge factor in the misconception. A lot of celebrities (especially models & actors) that society deems beautiful/handsome have extremely low weights. This teaches us to associate being skinny with being beautiful.
