How to keep idle setting stable?

My bike is an '86 YX600 Radian with CV carbs. When starting the bike it idles with choke off nice and smooth at 1300rpm. I always let the bike warm-up a couple of minutes before riding.....after 5 miles the idle is up to 2500-3000 rpm. I have a short piece of 3/8 inch dowel that I use to turn the thumbscrew under the carb bank (I hate to burn my knuckles and I cant get my hands under the carbs with gloves on) to lower the idle speed. Then it's alright, idle stays there at 1300-1400 rpm so I leave the thumbscre alone.
The next day when I start my bike, it idles smoothly at 1300 rpm but after riding it 5 miles I once again have to pull the dowel from under the seat and lower the idle speed.
Carbs were recently rebuilt and I also tried a spare bank of carbs off a FZ600 and dont have the idle problem but I dont have the correct main jets and needle jets the Radian requires.
What should I do, JBWeld the thumbscrew or convert the FZ600 carbs for use on my YX600?

Sounds like an air leak caused by heat expansion to me.

I replaced the manifolds last month and got the carbs sync'd last week. Didnt think a vacuum leak would raise the idle at all. To check, should I use spray starting fluid or propane? Got another carb bank on the way from EBay, I'll permatex the manifolds while I have the old carbs out.

Air leaks can raise the idle, but remember, this is basically guess work as I'm not there.

Make sure you didn't lose any idle or mixture screws as well.