Did you know that Google searches work best with mobile friendly web sites. If you want to make sure that your website is higher up the rankings then you need to have a mobile friendly web site.
Not only will Google not like your site but what is your customers view of the site? Do they find themselves looking at a page where the text is too small, the links are tiny, and have to scroll sideways to see all the content? This usually happens when the website has not been optimised to be viewed on a mobile phone.