Orchestrating Mathematical Reasoning Agents with Fact-Graph Memory