@jk about that, I have no idea about audio stuff, but...
https://www.crowdsupply.com/benti/democracy-dev
I've heard some good things about mixing an FPGA with a Pi to process audio. I'd guess it is kinda like your suggestion of connecting the pins directly to the CPU?