CODE EXAMPLE FOR JAVA

spring serve robots.txt

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests()
            .antMatchers("/robots.txt").permitAll()
    }