It depends on what you have. Most sites, you won't have to do a major redesign. I've been throwing html sites into Bootstrap. Certain parts of a site may need to be rethought for mobile. For example, I mentioned in another thread that I have pages with multi-column comparison tables - how do I handle that for 320 wide Portrait mode? (The Mobile Ready Test checks mobile compatibility down to that screen size - horizontal scrolling of the window is not allowed.)
Some CMS driven sites will be just a matter of switching to a responsive template, if you have a lot of customization it could be trickier.
If responsive doesn't work for you, make a mobile version of your site and serve it by default on screen sizes too small to render your current design.