I'm running .Net Core middleware and an AngularJS front-end. On my main page, I have google analytics script tags, and other script tags necessary for verifying with third-party providers. Prerender.io removes these by default, however, there's a plugin "removeScriptTags". Does anyone have experience turning this off with the .Net Core Middleware?
A better solution may be to blacklist the crawlers you don't want seeing cached content, though I'm not sure this is configurable. In my case, it looks like all the user-agents below are accessing Prerender.io cached content.
Here is my "crawlerUserAgentPattern" which are the crawlers that should be allowed to access the cached content. I don't see the ones above on this list so I'm confused as to why they're allowed to access.
"(SeobilityBot)|(Seobility)|(seobility)|(bingbot)|(googlebot)|(google)|(bing)|(Slurp)|(DuckDuckBot)|(YandexBot)|(baiduspider)|(Sogou)|(Exabot)|(ia_archiver)|(facebot)|(facebook)|(twitterbot)|(rogerbot)|(linkedinbot)|(embedly)|(quora)|(pinterest)|(slackbot)|(redditbot)|(Applebot)|(WhatsApp)|(flipboard)|(tumblr)|(bitlybot)|(Discordbot)"
Copyright Notice：Content Author:「Brandon McAlees」，Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article：https://stackoverflow.com/questions/59464236/allow-script-tags-in-net-core-prerender-io-middlewear