Why does Googlebot need to retrieve HTML from JSON-exclusive URLs?

Link to a page like this:

The following code is found:

$.getJSON("/newsfeeds/61?order=activity&type=discussion", function(response) {
  $(".discussion-post-stream").replaceWith($(response.newsfeed_html));
  $(".stream-posts").before($("<div class=\'newsfeed-sorting-panel generic-12\' data-id=\'61\'>\n<div class=\'newsfeed-type-menu generic-12\'>\n<ul class=\'newsfeed-sorting-buttons\'>\n<li>\n<span>\nShow\n<\/span>\n<\/li>\n<li>\n<select id=\"type\" name=\"type\"><option selected=\"selected\" value=\"discussion\">Show All (15)<\/option>\n<option value=\"discussion_answered\">Answered Questions (15)<\/option>\n<option value=\"discussion_unanswered\">Unanswered Questions (0)<\/option><\/select>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n"));
  Newsfeed.prepare_for_newsfeed_sort($(".newsfeed-sorting-panel"));
});

Googlebot attempts to crawl the URL 

/newsfeeds/61?order=activity&amp;type=discussion
in search of interesting HTML. However, an error is reported by my app stating "ActionView::MissingTemplate: Missing template newsfeeds/show..."

  1. What prompts Googlebot to crawl this URL? Is it exploring all possibilities or is there something amiss in the code?
  2. How can this be effectively addressed in Rails? Disregarding all MissingTemplate errors could mask genuine issues later on. Any alternatives to consider instead of ignoring errors caused by bots?
javascriptruby-on-railsbotsgoogle-searchgooglebot

Answer №1

It seems like the URL was extracted from the source code of the page, indicating that it is attempting to index your website.

To better control what Google indexes on your site, it's recommended to utilize a sitemap.xml file and a robots.txt file.

If you want to prevent Googlebot from crawling pages with specific GET parameters, include the following rule in your robots.txt file:

Disallow: /*?

Answer №2

It's perfectly fine for bots to explore and discover new links on your webpage as that is their purpose.

If you want to instruct search engine bots to ignore certain content, you can make use of specific metatags. Check out this link for more information: Is there a way to make robots ignore certain text?

These metatags send a signal to Googlebot to refrain from indexing the specified content.

<!--googleoff: all-->

$.getJSON("/newsfeeds/61?order=activity&amp;type=discussion", function(response) {
$(".discussion-post-stream").replaceWith($(response.newsfeed_html));
$(".stream-posts").before($("<div class=\'newsfeed-sorting-panel generic-12\' data-id=\'61\'>\n<div class=\'newsfeed-type-menu generic-12\'>\n<ul class=\'newsfeed-sorting-buttons\'>\n<li>\n<span>\nShow\n<\/span>\n<\/li>\n<li>\n<select id=\"type\" name=\"type\"><option selected=\"selected\" value=\"discussion\">Show All (15)<\/option>\n<option value=\"discussion_answered\">Answered Questions (15)<\/option>\n<option value=\"discussion_unanswered\">Unanswered Questions (0)<\/option><\/select>\n<\/li>\n<\/ul>\n<\/div>\n<\/div>\n"));
Newsfeed.prepare_for_newsfeed_sort($(".newsfeed-sorting-panel"));
});

<!--googleon: all>

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Ajax is known for inundating servers with requests at a rapid rate, causing a significant

I am running an application that sends multiple requests per second to fetch data from API and return responses to clients. The process flow is as follows: Ajax sends a request View sends a request API returns response for view View returns response for ...

javascriptajaxdjangowebwebsocket

Filter a div based on multiple conditions using JavaScript/jQuery

In my filtering system, I have two sections: place and attraction. When I select a place, the corresponding div is displayed according to the selected place. Similarly, when I select an attraction, only the attractions are filtered accordingly. <ul clas ...

javascripthtmljquery

Errors encountered while running `npm install discord.js`

I am currently facing an issue while trying to install discord.js. Unfortunately, I keep encountering the following errors: npm ERR! cb() never called! npm ERR! This is an error with npm itself. Please report this error at: npm ERR! <https://npm.co ...

javascriptnode.jsnpm

The selected plugin appears to be incompatible with mobile browsers

My project involves implementing the Chosen plugin for a select field, allowing users to type-search through lengthy lists. It functions perfectly on desktop but is disabled on both Apple and Android phones, reverting to the default user interface for sele ...

javascriptandroidjqueryjquery-chosen

The issue with Framer motion's whileInView is that it does not animate the element when it is within the viewport in Next.js

In my current Next.js 14 project with the App Router, I decided to play around with animations using Framer Motion. One specific feature I wanted to implement was animating text elements into view as they enter the viewport. The idea is for the text to gra ...

javascripthtmlreactjsnext.jsframer-motion

Express functions properly when handling the root route, but encounters issues with sub routes as it sends empty response bodies

Inside the routes.ts file: const router:Router = express.Router() // Route to get all blogs router.get('/',async (req:Request,res:Response)=>{ res.status(200).send("message sent") }) router.get('/admin',async (req:Requ ...

javascriptnode.jstypescriptexpressnext.js

Leveraging the power of map in an Angular typescript file

I've been attempting to populate a Map in Angular by setting values dynamically. When certain buttons are clicked, the onClick function is invoked. typeArray: Map<number,string>; Rent(movieId: number){ this.typeArray.set(movieId,"Rental ...

javascripthtmlangulartypescript

The state initialization process is beginning with an [object Object] being passed into the setState function

Within the async function provided below, I am invoking stationData to verify that an array of objects is being passed into bartData (which currently consists of an empty array). Upon exploration, I have observed a response containing the anticipated array ...

javascriptreactjs

Generating unique ObjectIDs for each object is crucial before saving documents in Mongoose

I need to generate a unique ObjectID for every object within my array. The challenge is that I am fetching products from another server using a .forEach statement and adding them to my array without a Schema that automatically creates an ObjectID.... Prod ...

javascriptnode.jsmongoosemongoose-schemaobjectid

Can $event be transferred from cshtml to Vue.component?

I am facing an issue when trying to pass an event into my Vue component. No matter what method I try, I keep getting a "TypeError: Cannot read property 'preventDefault' of undefined" error. Here is the code snippet for my Vue component: Vue.com ...

javascriptvue.jsrazoremit

Console is displaying an error message stating that the $http.post function is returning

Just diving into angular and I've set up a controller to fetch data from a factory that's loaded with an $http.get method connecting to a RESTful API: videoModule.factory('myFactory', function($http){ var factory = {}; facto ...

javascriptnode.jsangularjspost

Creating distinctive, unchanging colors for individual ellipses within a collection in p5.js

It's clear that each ellipse is part of a wave object, with the fill color applied on every frame causing a blinking effect. I'm trying to assign a random color to each ellipse when it's drawn, so it maintains that fill color instead of chan ...

javascriptp5.js

Enhance your cloud functions by updating data

Recently, I encountered an issue with a function I wrote that interacts with the Real Time Database. Every time I attempted to write data to a specific node, it would add the complete dataset and then promptly delete everything except one entry. https://i ...

javascriptnode.jsfirebase-realtime-databasegoogle-cloud-functions

Trigger the React custom hook when the form is submitted

I have developed a custom useFetch hook to handle API requests in various sections of the application. It works smoothly when a component is rendered, but I am facing an issue when trying to make a request upon form submission. An error message pops up say ...

javascriptreactjsreact-hooks

Step-by-Step Guide: Unveiling a Particular Modal Post-Submission of Form with

My website has a form inside a modal, and when the form is submitted, I don't want the modal to close. However, I have encountered an issue because my SQL UPDATE statement redirects to the same page after updating the database. This disrupts the funct ...

javascriptphpjquerytwitter-bootstrapbootstrap-modal

Tips on choosing and showcasing information from jQuery date and time picker

I am struggling to show the selected data from a jQuery date and time picker and save it to a database using PHP with MySQL. I am not sure how to retrieve the information. Here is the jQuery code for the date and time picker along with a suggested jQuery f ...

javascriptphpjquerymysql

I'm working on separating the functionality to edit and delete entries on my CRM model, but I'm having trouble finding a way to connect these buttons with my data fields

I am encountering some difficulties while trying to implement separate functionality for editing and deleting items on my CRM model. I have already created the necessary API in Angular, but I am struggling to bind these buttons with my field. Any assistanc ...

javascriptnode.jsangulartwitter-bootstrap-3

The SyntaxError is triggered by the React-Native FlatList component

As a newcomer to React Native, I am facing an issue with refreshing the component to load city names stored in async storage. The error occurs when I utilize the FlatList component for the first time. The specific error message I encountered is: SyntaxE ...

javascriptandroidreactjsreact-native

Tips for sending parameters using arrow functions within ReactJS hooks

I've recently started working with ReactJS and I'm a bit confused about how to pass parameters using arrow functions. I attempted to use .bind() in order to bind the value so it can be used in the function, but unfortunately that didn't work ...

javascriptreactjsreact-hooksarrow-functions

Transitioning from a traditional CURL method to utilizing AJAX and XMLHttp

I'm currently facing a challenge converting the curl code from an API named TextRazor to AJAX XMLHttp due to limitations on the platform I am working with. Despite trying various solutions shared by the community, I have been unsuccessful in retrievin ...

javascriptajaxcurlxmlhttprequest