I think you need to worry about RSS feeds if there is lot of content added in your site everyday, and many people have subscribed to your rss feeds. Then you will get traffic from the people who have subscribed to your rss feeds.
Normally, I think we should block the Google Bot and other SEs from indexing the RSS feeds. As RSS feeds can create duplicate content problem within sites.