Asking why certain dinosaurs had feathers is no different, in principle, from asking why fish have scales or why dogs have fur. Why should the bare epidermis of any animal possess any type of covering (or, in the case of human beings, practically no covering at all)? In order to answer this question, we have to address a deeper conundrum: what evolutionary advantage did feathers confer on dinosaurs that couldn't be accomplished with fur, or bristles or simple, reptilian scales?
The Majority of Feathered Dinosaurs Were Theropods
Before we start, though, it's important to recognize that not all dinosaurs had feathers. The vast majority of feathered dinosaurs were theropods, a broad category that includes raptors, tyrannosaurs, ornithomimids and "dino-birds," as well as the earliest dinosaurs like Eoraptor and Herrerasaurus. Furthermore, not all theropods were feathered: it's a pretty sure bet that the late Jurassic Allosaurus had scaly skin, as did other large theropods like Spinosaurus and Tyrannosaurus Rex (though an increasing number of paleontologists believe that the hatchlings and juveniles of these dinosaurs may have been adorably tufted).
Theropods weren't the only members of the order of saurischian ("lizard-hipped") dinosaurs: oddly enough, their closest relatives were the giant, lumbering, elephant-legged sauropods, which were about as different in appearance and behavior from theropods as you can possibly get! To date, there's absolutely no evidence for any feathered relatives of Brachiosaurus or Apatosaurus, and such a discovery seems extremely unlikely. The reason has to do with the differing metabolisms of theropod and sauropod dinosaurs, of which more below.
What Is the Evolutionary Advantage of Feathers?
Extrapolating from the example of modern birds, you might think that the primary purpose of feathers is to sustain flight; feathers trap small pockets of air and provide the crucial "lift" that enables a bird to soar into the air. By all indications, though, the employment of feathers in flight is strictly secondary, one of those contingent developments for which evolution is so famous. First and foremost, the function of feathers is to provide insulation, just like the aluminum siding of a house or the polyurethane foam packed in its rafters.
And why should an animal need insulation, you ask? Well, in the case of theropod dinosaurs (and modern birds), it's because it possesses an endothermic (warm-blooded) metabolism. When a creature has to generate its own heat, it needs a way to retain that heat as efficiently as possible, and a coat of feathers (or fur) is one solution that has been repeatedly favored by evolution. While some mammals (like human beings and elephants) lack fur, all birds have feathers--and the insulating prowess of feathers is no better demonstrated than in flightless, aquatic birds inhabiting cold climates, i.e., penguins.
Of course, this raises the question of why Allosaurus and other large theropod dinosaurs lacked feathers (or why those feathers were only present in juveniles or hatchlings). This may have something to do with the climatic conditions in the regions where these dinosaurs lived, or with a quirk in the metabolism of large theropods; we don't yet know the answer. (As for the reason sauropods lacked feathers, that's because they were almost certainly cold-blooded, and needed to efficiently absorb and radiate heat to regulate their internal body temperature. If they had been covered with feathers, they would have baked themselves from the inside out, like microwaved potatoes.)
Dinosaur Feathers Were Favored by Sexual Selection
When it comes to otherwise mysterious features in the animal kingdom―the long necks of sauropods, the triangular plates of stegosaurs, and, possibly, the bright feathers of theropod dinosaurs―one should never discount the power of sexual selection. Evolution is notorious for picking out seemingly random anatomical features and putting them into sexual overdrive: witness the enormous noses of male proboscis monkeys, a direct result of the fact that females of the species prefer to mate with the biggest-nosed males.
Once insulating feathers had evolved in theropod dinosaurs, there was nothing to prevent sexual selection from taking over and driving the process even further. As yet, we know very little about the color of dinosaur feathers, but it's a sure bet that some species sported bright greens, reds, and oranges, probably in a sexually dimorphic fashion (i.e., the males were more brightly colored than the females or vice versa). Some otherwise bald theropods may have sported tufts of feathers in odd locations, such as their forearms or hips, another means of signaling sexual availability, and some early, famous dino-birds like Archaeopteryx were equipped with dark, glossy feathers.
What About Flight?
Finally, we come to the behavior that most people associate with feathers: flight. There's still a lot we don't know about the evolution of theropod dinosaurs into birds; this process may have happened multiple times during the Mesozoic Era, with only the last evolutionary wave resulting in the birds we know today. It's an almost open-and-shut case that modern birds evolved from the small, skittery, feathered "dino-birds" of the late Cretaceous period. But how?
There are two main theories. It could be that these dinosaurs' feathers provided an extra bit of lift when they were chasing prey or running away from larger predators; natural selection favored increasing amounts of lift, and finally, one lucky dinosaur achieved takeoff. In contrast to this "ground-up" theory, there's the less popular "arboreal" theory, which posits that small, tree-living dinosaurs evolved aerodynamic feathers while leaping from branch to branch. Whatever the case, the important lesson is that flight was the unintended byproduct, not the foreordained purpose, of dinosaur feathers!
One new development in the feathered dinosaurs debate is the discovery of small, feathered, plant-eating ornithopods like Tianyulong and Kulindadromeus. Might this imply that ornithopods, as well as theropods, possessed warm-blooded metabolisms? Is it at least possible that birds evolved from plant-eating ornithopods, rather than meat-eating raptors? We don't know yet but count on this being an active area of research for at least the next decade.