Matt Cutts the Google Engineer has stated in his blog , “We want the web to be faster, we want sites to load quickly”, means Google consider “user experience" tor showing search results.
I have seen Matt Cutts talking about this matter. He said that a site loading time do not have a direct impact on SERPs/SEO. The only thing that we should worry about if a site taking too much time to load: load time that will result in a browser error or something like that.
Just try to make your site load as fast as possible but do not worry much about loading time if your site do not take lot of time to load.
it might not just be your site it might be your server. If you work hours on your site but your server is still over loaded due to other websites (running scripts, traffic, oversold server etc.) no matter what you improve your site will still load slowly.