They said you couldn't change your future by going to the past, but you create alternate timelines when take the stones.
They did. But they also said that as long as you don't change the past you won't create new timelines either.
So Loki took the Tesseract and disappeared in 2012. What happened to that?
According to the rules, new timeline. However, I don't remember it stated how Loki ended up imprisoned in Asgard. So since it's a time loop, he may have escaped then got recaptured and nothing changed.
How can Cap return the Soul Stone at the exact moment it was taken?
Didn't need to. He only needed to return it. But he could have returned it right after Clint left.
2014 Thanos and his army (including Nebula) went to 2023 and got killed. So what happens to that timeline?
According to the rules, new timeline.
Did 2014 Gamora get dusted? Or did she disappear?
2014 Gamora is still alive. But now she's from another timeline. So it doesn't affect the OG 2014 Gamora.