So then does that mean the speaker basket doesn't offer any acoustical advantage? I assumed it did that's why I didn't want to remove them. I bought some speakers a while back that I thought would fit with the basket installed but are a little to deep. They're 2 5/16" from the magnet to the bottom of the metal flange.

 

The speaker basket offers no acoustical advantage; however it does offer some weather protection.
Many people substitute foam speaker baffles for that purpose.