I think you are missing that SPA is not server side rendering. At least for the majority. So, when you access /anything your web server won't redirect it to index.html. However, if you click on any vuejs router link, it will work due to the fact that the javascript got in action, but not the server side.
To solve this, use .htaccess and redirect all requests to index.html like so
<ifModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index.html$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.html [L]
</ifModule>
Hope it helps someone!