My sweet spot regarding soundcard buffer seems to be 256.In Buzz i always need to set it to 2+1 async (as opposed to ableton) otherwise i also get some crackles.Then i'm able to have the CPU peaking up to 40% (which i avoid if possible) without crackles but only if the cores are (more or less) balanced.Sometimes it happens that one core is processing too much.Clicking on Buzz's "Disable Audio Devices" (F12) one or two times often helps to re-balance the cores.Also a simple "save" seems to work well.
When the cores are balanced Windows 10 does a great job in keeping them balanced while working in a session..
This is my experience with an RME AIO (on different computers,also on a i7-3930K) and a Audient ID4 which behave similar,minus some ms better latency for the RME card.
Oh,and closing the task manager helps too.
Interestingly.For a while i worked on a Asus J1900 Board (to be forced to work with limited resources

) with my Yamaha Ag03 and i could push the CPU by far more than i ever was able to do it on a much stronger i7.
Wifi USB can be a bad thing,there are many reports about audio/asio related problems,some Asus USB-Wifi devices can work though but i avoid Wifi.
My routine when working with Windows and low latency audio is always to start Windows and let it do its start up routine for about 20 minutes (checking the task manager) ,after a while when the CPU is idle i check the maintenance window too,to see if Windows got its job done.So don't touch a mouse or keyboard otherwise Windows will stop maintenance.
I'm currently working on a i7-7700 and i'm using the inbuilt GPU,There are sometimes glitches,at least a heavy CPU strain with certain VST GUI's but this isn't bothering me.
I haven't read much comments on about how much CPU people are able to use with low latency audio but there is a certain sweet spot which i'm able to work.
Btw,before Windows 1803 i needed to manually stop the Superfetch service because i got a huge glitch every 10 minutes or so .Superfetch comes back after a restart but i'm avoiding any tweaking that's why i only stop the service.
With 1803 it seems to be much better but i only want to mention it.
Regarding USB 3 interfaces.You should thoroughly read about them if there are any problems.Usually USB 2 for a small interface is the way to go (at the moment).
I'm also using a gaming mouse, a Logitech G502 without a problem.
One more thing to mention.In Windows 1803 there is a new "ultimate performance mode" power plan which i find helps stabilizing.
http://www.thewindowsclub.com/ultimate- ... power-plan
All in all,working with low latency isn't simple at all.