Look at the QB's they've faced.
David Gerrard, Alex Smith, Kerry Collins, Chad Henne, Seneca Wallace, Marc Bulger. .
The only 2 respectable QB's they played were Warner and Schaub, and they both put up 330 yards on them. Brady is already over 300.
Their secondary isn't the reason why they've been able to keep things in check. . it's a favorable schedule and the pressure that they put on a QB.