what kind of sites are you talking about? if you only mean domain portfolio sites with 1,000 - 100,000 domains in a database and no heavy computing on the srver side you definitely do not need dedicated server - cheap $10/mo vps running debian/ubuntu with 512mb ram is more than enough with almost any 1 core proc.
if your sites are slow the answer is a function of how many requests per second you got (caching + connection), geography of requests (serving Singapore users from the US), software optimization (do you use tons of wp plugins, unneeded social widgets, numerous js libs only to add outdated eye damaging button animations and all that funny bells and whistles stuff)
get rid of Apache, use lighttpd or nginx anyway
get rid of huge images, and no you dont have to serve videos on a portfolio site
dont use more than 2 google fonts on the site
move all js tags to the bottom of the body tag, compress js/css files
get rid of all social buttons/widgets - buyers are not going to follow you on twitter, who cares about your sales/acquisitions
get rid of jquery and rewrite all js in native javascript - unless you are developing complex web app with advanced browser interface everything including sliders should/can be made with plain js
use good light fast php/nodejs framework (yii, falcon, fatfree etc)
do not load more than 1 js/css file or better use inline js on simple sites
strip off unused css when using bootstrap or other generic css framework
use icon font (fontawesome+fontello) instead of multiple image icons
get good fast not oversold vps (digitalocean etc) with easy upgrade options + diverse locations
simple portfolio site on debian + lighttpd + mysql on $10 vps on 1gb/s link can easily serve tens of thousands of visitors a day even with defaul settings if your app is coded properly. even on wordpress (providing you dont have 1001 plugins running for every simple task that can be accomplished by a couple of php/js code strings instead)
remember that your sites need to be responsive nowadays whatever they are, and that does not put any additional load on your server if done properly
imo