Verifying the sources is a separate (yet connected) issue indeed, but it's useless if you don't know whether the sources you're verifying are the ones that have been packaged.
As a developer you should never blindly trust any dependencies. If you depend on some code, you will inherit all its flaws and issues, as well. That's your responsibility.
Luckily you're not alone and it's a collective process. The same needs be achieved for verifying build integrity.