There are a lot of anachronistic events at the Olympics, like the modern pentathlon, that you assume are just grandfathered into the Olympics at this point. But how does a sport like golf wind up debuting at the 2016 Rio Olympics? It's actually an involved process to add a sport to the Olympic Games.
To start, the sport has to have a non-governmental international federation. Traditionally, that body can then petition the International Olympic Committee (IOC) for Olympic inclusion. However, in 2014 the IOC formalized another avenue that created a more flexible Olympic program to make it easier for events like skateboarding and surfing to be added, as they will for the 2020 Tokyo Olympics. That will be the first Olympics to have sports who have taken advantage of that new procedure. Golf had to take the more traditionally used route.
Britannica notes that a sport can be admitted "as a sport, a discipline, which is a branch of a sport, or an event, which is a competition within a discipline. For instance, triathlon was admitted as a sport, debuting at the 2000 Games in Sydney. Women's wrestling was a new discipline in the sport of wrestling at the Athens Games, and women's pole vaulting was the most recently added track and field event."