In a 2011 Stone column, Timothy Williamson writes:
"One challenge to naturalism is to find a place for mathematics. Natural sciences rely on it, but should we count it a science in its own right? If we do, then the description of scientific method just given is wrong, for it does not fit the science of mathematics, which proves its results by pure reasoning, rather than the hypothetico-deductive method. Although a few naturalists, such as W.V. Quine, argued that the real evidence in favor of mathematics comes from its applications in the natural sciences, so indirectly from observation and experiment, that view does not fit the way the subject actually develops. When mathematicians assess a proposed new axiom, they look at its consequences within mathematics, not outside. On the other hand, if we do not count pure mathematics a science, we thereby exclude mathematical proof by itself from the scientific method, and so discredit naturalism. For naturalism privileges the scientific method over all others, and mathematics is one of the most spectacular success stories in the history of human knowledge."
Quine's naturalistic position is that we judge mathematical validity by relation to empirical observation. In contrast, Williamson says that mathematical validity is determined by looking only at mathematics itself, and not the world. Williamson's reasoning seems obviously flawed. If Quine is correct, then whenever we look at internal consistency or coherence in mathematics, we are looking at consistency/coherence within an empirically-grounded framework. So, when mathematicians determine whether or not a proof is mathematically valid, they are determining whether or not it fits with empirical givens. They might not always be doing so directly, but then, physicists and chemists don't always deal directly with observable givens, either.
Edited to add the following clarifying remarks (which, unfortunately, are a bit on the rambling side): I have strongly naturalistic tendencies. I might not say that science is the only way to knowledge, though. I would rather say that science is the most reliable way to shared knowledge of the world. (Actually, I would define "science" as the pursuit of shared methods for discovering new knowledge about the world.) And I would say that all facts about the world can, in theory, be discovered scientifically--even if nobody will ever be in a position to discover them. That makes me a naturalist, I think, but it leaves open two possibilities: one is that we can have private knowledge of the world; the other is that we can have knowledge which is not worldly.
Philosophical knowledge--knowledge of logical relationships (and we might include mathematics here)--is not necessarily worldly. I think philosophical knowledge must have a worldly foundation, but it might not be reducible to facts about the world. It might better be thought of as facts about all possible worlds, even if our knowledge of all possible worlds must, in some way, be limited by the facts about the world we live in.
For example, consider mathematics: Our mathematical knowledge might be empirically grounded, as Quine claims, but, at the same time, it is knowledge of all possible worlds. Our knowledge of logical relationships--like the analytic truth of "all bachelors are unmarried"--is similarly grounded in empirical knowledge. (We know through experience what "bachelor" and "unmarried" are, as well as the verb "to be" and the qualifier "all.") Yet, the extension of that knowledge is beyond the actual world. All bachelors are unmarried in all possible worlds.
The issue here is between claims about particulars (facts about the actual world, or individual possible worlds) and universals (facts about all possible worlds). Science and philosophy deal in both, but in different ways. Science comes down to developing methods for testing claims about the actual world against claims about possible worlds. A scientific experiment uses facts about individual possible worlds (empirical hypotheses) and all possible worlds (mathematical and logical relationships) to test claims about the actual world. In contrast, philosophy explores the consistency and coherence of facts about possible worlds. These are two different ways of pursuing knowledge, and they are not mutually exclusive. So it seems to be that science (as the pursuit of methods for discovering shared knowledge of the world) is not the only path to gaining knowledge. We also have philosophy, which can work with science for their mutual benefit. But philosophical knowledge as such is not knowledge of the actual world (even if it has an empirical foundation). (This is leaving aside the possibility of an intrinsically private knowledge.)
We have to wonder, though. If mathematics is true of all possible worlds, then how can it be arrived at by empirical means? How can we determine universal truths if we are limited by the particulars of the actual world? It's tempting to dismiss this as a language game. Mathematical truths are true of all possible worlds only because we define them that way. Mathematics is a universalized construction based on certain empirical features of our world--specifically, features which allow for the analysis of patterns. To say that mathematics (or logical relationships) are true in all possible worlds is only to say that the features which make pattern analysis possible are universalizable. But that universalizability is a feature of the actual world. It is a feature of all possible worlds. To put it another way, you cannot have a world with features that make pattern analysis possible without those features being universalizable. This is part of the identity of those features. At this point, I have to pause and reflect on what it means for such features to be universalizable, and also on how it can be that we can know that they are universalizable.
Update: See follow up: Possibility, Actuality and Necessity