The IRS had good reason to question the tax-exempt status of some right-wing groups.