Yes, I know you can't set your Charmer. But it's a more live card compared to your x3 Scroll, which is overkill especially you already have Genex Changer.
The use of Hidden Armory is not meant to be activated in whatever turn you want, or to be spammed, if you do this, its drawback is gonna hurt you just like you said. With Possession Release, if you already have any monster on the field then that's fine to not be able to summon/set monster, since Charmers cannot be destroyed by battle, and you get your opponent's monster with Snatch Steal. Also, if they MST/Castel your Snatch Steal, then ok, they waste those removal for it instead of your better backrow: Possession Release.
Remember your number of Attribute changer in this deck is too much, not only you already have Genex Changer, but you also use x3 Scroll, and also Possession Release do similar job by searching your Charmer with any attribute you want. You can throw in x1 or 2 Hidden Armory in exchange of your Scroll for more options and versatility. With Snatch Steal around, if you use Hidden Armory, you'll have less dead Scroll in your hand.
It's better to draw Hidden Armory than having 1 or several of your Scrolls stuck in your hand without available Charmer on the field.
Test it and complain later before taking conclusion by judging its a bad card.