16 GB RAM is a must at some point, but with 8 GB you aren't exactly in a desparate position to upgrade anyway. I suggest you look for deals and just buy a matching stick when you find a good bargain and have no higher priority expenses.
You definitely want to reconsider your position on buying used stuff. I hear you about warranty etc., but you can still but used stuff that's still on warranty.
Right now the most sensible step for you is to sell the 1050ti while you still can, and get the best used but reliable GPU you can afford. Some sellers of used hardware are either hardware stores or companies that use strong GPUs for work, so either way they aren't going to disappear on you, they can write a proper formal invoice for you and even give you seller's warranty on top of the manufacturer's warranty. Private folks, well, it boils down to their credibility. But a lot of them have kept the invoice and warranty and have only used the card for several months.
That said, 1050ti => 1060 would be too much of a sidegrade (let alone RX 570…) to even bother with unless you had a lot of time on your hands and could sell the 1050ti for nearly as much as you paid for the 1060. Otherwise it's Vega or 1070 & up for you. If you don't have the budget for a 1070, either wait or, well, concentrate on making some money on the side. Ask your boss for some paid overtime, ask at local businesses if you could lend a hand after your normal work or school for some extra buck, whatever works. That's usually the easier way out of a budget problem.
And get that new CPU cooler (preferably one that enables easy fan swap, and you want the fans to be at least 12 cm large if not 14 cm, depending on what fits in your case) and replace your case fans with the best you can afford (if possible, you want to go and test them in the store before buying, rather than relying on manufacturer specs, especially if noise bothers you). Even the best fans aren't normally horribly expensive, and they can make a big difference, as you can get away with more aggressive OC.
But the most immediate fps boost will always come from getting a stronger GPU, so don't get distracted. Upgrading the GPU should be your top priority.
EDIT: If you're on a tight budget, lowering the settings intelligently might give you more peace of mind. There are always some settings that impose a comparably significant load on the GPU without giving you a proportionate gain in image quality. Differences between ultra vs high textures or more shadows vs less, or anti-aliasing levels (just as long as there is some) aren't always noticeable unless you know what to look for and focus hard on it. In other words, you can free up some resources for more fps without degrading your visual experience. It's more about the psychological feeling of 'I know it's high and not ultra' or 'I know it's medium and not high'. If you can overcome that, you don't need to upgrade as aggressively as most people.
