Look, off the field (or track or gym or pool), some athletes like to have a good time. Whether or not that fun is legal/good PR is another matter. Just ask Michael Phelps. But as far as the WADA is concerned, certain drugs are totally fine any other time except for performances.
Most of these are popular street drugs, including stimulants (cocaine, meth), narcotics (heroin, methadone, oxycodone), and cannabis (marijuana, weed, reefer, Mary Jane). Not that smoking weed would be particularly helpful for someone looking to run a marathon or swim 400 meters, but taking a stimulant delivers a quick burst of energy that would be beneficial to someone like a sprinter.