(1) If policy is defined as “everything that isn’t consensus”, yet those policy checks are included in the consensus files… what exactly are they doing there? Are they deprecated?
No, not deprecated. There isn’t a clean dividing line in the Bitcoin Core file system where say one file only checks policy rules and one file only checks consensus rules. (See this StackExchange post from Suhas Daftuar discussing consensus and policy subdirectories). This is for various reasons. One is the inheritance of the codebase from Satoshi who created a large bundle of code that although refactored and cleaned up over a decade by Core contributors still bears a resemblance to what they inherited from Satoshi. Perhaps more importantly it is really difficult to outline what all the consensus rules actually are because there are edge cases where you think the code you’re touching isn’t consensus related but then it turns out you introduced a consensus bug that results in a potential chain split. Pieter Wuille discussed this on the Chaincode Labs podcast in 2020:
One of the things I think learned from that is specifying what your consensus rules is really hard. That doesn’t mean you can’t try but who would’ve thought that a configuration setting in the database layer you are using actually leaked semantically into Bitcoin’s implicitly defined consensus rules. You can attribute that to human failure of course. We should’ve read the documentation and been aware of that.
There could be a bug in your C library. There can be a bug in your kernel. There can even be a bug in your CPU.
We can talk about the boundary in trying to abstract the part of the codebase that intentionally contributes to consensus but it is very hard to say clearly this code has no impact on consensus code because bugs can leak. I think one of the things to learn there is you really want software that is intended for use in a consensus system where not only you have the requirement that if everyone behaves correctly everybody accepts the right answer but also that everybody will disagree about what is an invalid piece of data in lockstep.
(2) Are ALL of the consensus rules included somewhere in these consensus files? I’m assuming not. So, I ask…
No. There have been attempts in the past to create a consensus library (libbitcoinconsensus) and more recently libbitcoinkernel that Carl Dong has discussed and contributed to but it is really hard. Consensus is a really slippery concept in Bitcoin.
(3) Could we compile a list, perhaps, of everything that a valid block would need in order to be accepted as valid that is NOT already included somewhere in these consensus files??
If you mean like an exhaustive list in the English language then you’re going to struggle. You can try and you can probably include most of the obvious things but it is the edge cases and things that seem they are entirely non consensus related that turn out not to be which will be missing.
