Physicalism entails the notion that every fact about existence is a physical fact. It does not stipulate that one can have all the facts about any aspect of existence. Physicalism does not require, or even suggest, P2.
We have no way of contextualizing the notion of “all of the facts” about something such as color vision. Without any such limitations to guide our thinking, "all of the facts" is not a well-defined concept.
I present and critique a version of the Knowledge Argument that does not rely on P2 here: Omniscience, Testability and The Knowledge Argument.