You're not supposed to do that yourself and alone. The point is that we need to set up systems that let us verify integrity collectively. Reproducible builds are just one step towards that. Only once we achieve that, we can actually rest assured, that the source code we're verifying is actually the code we're running. And again, that's a collective process with open source.