The American Academy of Pediatrics agrees with this, saying fevers can help children fight off infections.
OK, but 104˚ F still kinda sucks...
Admittedly, yes -- as do headaches, sweating, shivering, fatigue, weakness, aches, loss of appetite, and other fever symptoms you might get in addition to a high temperature. If you really want to fight off a fever and get back to work, your best courses of action are the following:
How to get rid of a fever
Drink lots and lots of fluids. And alcoholic ones don't count. You should try to drink at least a tall glass of water every couple hours when you have a fever, if not more.
Don't take medication unless you need it. That's if you're feeling really uncomfortable or in pain, or typically if your fever is higher than 103˚ F. Generally you're a better judge of that than anyone writing on the Internet will be. A few recommended fever-reducing meds include: aspirin, acetaminophen (found in Tylenol, NyQuil, and generic brands), and ibuprofen (found in Advil, Motrin, and generic brands). These are all mild pain relievers.