Scepter -- As of Avengers: Age of Ultron, the Mind Stone has been sitting in heroic robot Vision's forehead. Unfortunately, the events of Captain America: Civil War dealt a severe emotional and mental blow after he accidentally blasted War Machine out of the sky. At the end of the film, Vision is still an Avenger, but processing friendly-fire emotions isn't something the robot has any experience dealing with. The Mind Stone doesn't help matters.
Tesseract -- With Loki defeated at the end of the first Avengers, Thor took him and the Tesseract back to Asgard for safekeeping. There, in theory, it could be kept safe. Later in the chronology, however, Thor: The Dark World ends with Loki in disguise as Odin, and Thor being none the wiser. This should be bad news.