I find it stupid to have to do this, but there is a good solution to avoid skilling things you don’t want: There is a Shopkeeper in Novigrad that sells the Potion of Clearance (for 1000 Gold) that resets all your ability Points. So you can save drink it, reskill and load again – and maybe use the Potion later when needed

There are 3 location where the Potion is sold: 1x at Keira Metz, 1x Shopkeeper in Novigrad and the 5x Alchemist in Gedyneith (Skellige)