After taking out the hashbang from my routes by using
$locationProvider.html5Mode(true);
Now, when I go to a page like "domain.com/download", it works. But if I refresh this same page, I get a 404 Error. URLs like "domain.com/download" can only be accessed by entering "domain.com/#!/download". After it loads, it automatically redirects me to the regular "domain.com/download" page.
I'm using v1.3.2 for routing and v1.6.3 for the rest (don't ask me why I didn't build this site myself lol).