Look, off the field (or track or gym or pool), some athletes like to have a good time. Whether or not that fun is legal/good PR is another matter. Just ask Michael Phelps. But as far as the WADA is concerned, certain drugs are totally fine any other time except for performances.
Most of these are popular street drugs, including stimulants (cocaine, meth), narcotics (heroin, methadone, oxycodone), and cannabis (marijuana, weed, reefer, Mary Jane). Not that smoking weed would be particularly helpful for someone looking to run a marathon or swim 400 meters, but taking a stimulant delivers a quick burst of energy that would be beneficial to someone like a sprinter.
Glucocorticoids, on the other hand, are powerful anti-inflammatories, and may be prescribed for something like a slipped disc or to treat an autoimmune disease. They're also a form of steroids that produce energy when taken, and are therefore banned during competition. But they carry severe health side effects -- glucocorticoids weaken the immune system, making people prone to serious infection, and they raise blood sugar, which is why they are only prescribed for a few days at a time. Other potential problems include psychological effects such as mood changes and memory problems, and the onset of Cushing syndrome.