The Facebook bots are unable to crawl our AngularJS application because the JavaScript is not being properly

Question

The Facebook bots are unable to crawl our AngularJS application because the JavaScript is not being properly

I have a unique setup where my website is built with AngularJS and Wordpress as a single page application. Depending on the routing of the page, I dynamically define meta tags in the controller. Here's a snippet of my HTML header:

<meta property="og:url" content="{{ngMeta['og:url']}}" />
<meta property="og:title" content="{{ngMeta['og:title']}}" />
<meta property="og:description" content="{{ngMeta['og:description']}}" />
<meta property="og:image" content="{{ngMeta['og:image']}}" />

While this setup works correctly, I encountered an issue when trying to share a page on Facebook. The crawler dispatched by Facebook doesn't seem to recognize the meta tags, resulting in incorrect display:

https://i.sstatic.net/VH8Mk.png

I suspect this is due to the fact that the crawler doesn't load the Javascript, hence failing to fetch the meta tags.

To address this, I'm considering implementing a server-side solution using Apache mod rewrite. This involves setting up different redirects for human visitors and social media crawlers (Facebook, Twitter, etc.).

In an article I came across, they suggest using mod rewrite with Apache to handle requests from social media crawlers and redirect them to server-side static pages with metadata processed by the server.

Currently, my website uses Wordpress with permalinks structured as:

https://<dns>/%category%/%postname%/

. Here's a snippet of my .htaccess file:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

To implement the suggested solution, I would need to add specific Rewrite rules for each category like so:

RewriteCond %{HTTP_USER_AGENT} (facebookexternalhit/[0-9]|Twitterbot|Pinterest|Google.*snippet)
RewriteRule tracks/(\d*)$ https://<dns>/server/static-page.php?id=$1 [P]
RewriteRule articles/(\d*)$ https://<dns>/server/static-page.php?id=$1 [P]

These rules are intended to redirect crawlers to static pages. However, I'm uncertain how to handle the requests on these server-side static pages. Any suggestions or recommendations?

javascript angularjs facebook facebook-graph-api

Answer 1

Answer №1

I devised a solution by redirecting all Facebook crawlers to a static server page in order to enhance the open graph meta tags.

WordPress automatically redirects to index.php, so it's necessary to block this redirection for Facebook (and any other social bots) in the .htaccess file as well.

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTP_USER_AGENT} !facebookexternalhit/1.1|Twitterbot|Pinterest|Google.*snippet
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} facebookexternalhit/1.1|Twitterbot|Pinterest|Google.*snippet
RewriteRule tracks/(.*) http://<fqdn>o/static.php?id=$1 [R]
</IfModule>

You can adjust the RewriteRule to fit your permalinks (tracks represents a category in my WordPress) using the regex of your choice. My custom WordPress permalinks are structured like this: /%category%/%postname%/.

After configuring the rewrite in the Apache configuration, you need to create your static page. In my case, I'm utilizing WP API V2, and here is an example of my page:

<?php 

$SITE_ROOT = "http://<fqdn>";

$jsonData = getData($SITE_ROOT);
makePage($jsonData, $SITE_ROOT);

function getData($siteRoot) {
    $id = $_GET['id'];
    $rawData = file_get_contents($siteRoot.'/wp-json/wp/v2/posts?slug='.$id);
    return json_decode($rawData);
}

function makePage($data, $siteRoot) {
    ?>
    <!DOCTYPE html>
    <html>
    <head>
        <meta property="og:type" content="articles" />
        <meta property="og:url" content="<?php echo $data[0]->link; ?>" />
        <meta property="og:title" content="<?php echo $data[0]->title->rendered; ?> | Going Solo" />
        <meta property="og:description" content="<?php echo $data[0]->excerpt->rendered; ?>" />
        <meta property="og:image" content="<?php echo $data[0]->better_featured_image->media_details->sizes->medium->source_url; ?>" />
        <meta property="og:image:width" content="500" />
        <meta property="og:image:height" content="500" />
    </head>
    <body>
        <img src="<?php echo $data[0]->better_featured_image->media_details->sizes->medium->source_url; ?>">    
        <h1><?php echo $data[0]->title->rendered; ?></h1>
        <p><?php echo $data[0]->excerpt->rendered; ?></p>

    </body>
    </html>
<?php
}
?>

The variable $id corresponds to the post's slug. By accessing your static page with the specified slug as a parameter, you will observe the server-side management of all open graph meta tags:

http://<fqdn>/static.php?id=<slug>

Answer 2

I devised a solution by redirecting all Facebook crawlers to a static server page in order to enhance the open graph meta tags.

WordPress automatically redirects to index.php, so it's necessary to block this redirection for Facebook (and any other social bots) in the .htaccess file as well.

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{HTTP_USER_AGENT} !facebookexternalhit/1.1|Twitterbot|Pinterest|Google.*snippet
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} facebookexternalhit/1.1|Twitterbot|Pinterest|Google.*snippet
RewriteRule tracks/(.*) http://<fqdn>o/static.php?id=$1 [R]
</IfModule>

You can adjust the RewriteRule to fit your permalinks (tracks represents a category in my WordPress) using the regex of your choice. My custom WordPress permalinks are structured like this: /%category%/%postname%/.

After configuring the rewrite in the Apache configuration, you need to create your static page. In my case, I'm utilizing WP API V2, and here is an example of my page:

<?php 

$SITE_ROOT = "http://<fqdn>";

$jsonData = getData($SITE_ROOT);
makePage($jsonData, $SITE_ROOT);

function getData($siteRoot) {
    $id = $_GET['id'];
    $rawData = file_get_contents($siteRoot.'/wp-json/wp/v2/posts?slug='.$id);
    return json_decode($rawData);
}

function makePage($data, $siteRoot) {
    ?>
    <!DOCTYPE html>
    <html>
    <head>
        <meta property="og:type" content="articles" />
        <meta property="og:url" content="<?php echo $data[0]->link; ?>" />
        <meta property="og:title" content="<?php echo $data[0]->title->rendered; ?> | Going Solo" />
        <meta property="og:description" content="<?php echo $data[0]->excerpt->rendered; ?>" />
        <meta property="og:image" content="<?php echo $data[0]->better_featured_image->media_details->sizes->medium->source_url; ?>" />
        <meta property="og:image:width" content="500" />
        <meta property="og:image:height" content="500" />
    </head>
    <body>
        <img src="<?php echo $data[0]->better_featured_image->media_details->sizes->medium->source_url; ?>">    
        <h1><?php echo $data[0]->title->rendered; ?></h1>
        <p><?php echo $data[0]->excerpt->rendered; ?></p>

    </body>
    </html>
<?php
}
?>

The variable $id corresponds to the post's slug. By accessing your static page with the specified slug as a parameter, you will observe the server-side management of all open graph meta tags:

http://<fqdn>/static.php?id=<slug>

The Facebook bots are unable to crawl our AngularJS application because the JavaScript is not being properly

Answer №1

Similar questions

Fresh ajax requests are not clearing the current data displayed in the JSP table

What is the procedure for turning off hover color on an href element?

Combine two arrays of objects and merge properties using the Ramda library

Is there a way to individually apply loading to only the button that has been clicked in vuejs?

Beginner's guide to integrating the @jsplumb/browser-ui into your Vuejs/Nuxtjs project

change visibility:hidden to visible in a css class using JavaScript

The program keeps encountering the issue of returning undefined while attempting to calculate the sum of elements in an array

I will receive a 404 error if I try to access a URL without including the hashtag symbol

Adding options to a dropdown list dynamically within a ASP.NET Datagrid using Jquery

Clicking a button in JavaScript can pass an ID by triggering an onClick

How can I display input only when a checkbox is selected? React with Next.js

Using ReactJS to insert an array of objects into an object nested within another array object

Automatically filling in related information in multiple input fields after choosing a value in a specific input field using JavaScript

Angular HTTP request URLs are dynamically appended to the current URL in Internet Explorer within a custom MVC route leading to

Angular is failing to retrieve data from FS.readFile using promises

Display the div only after all images have finished loading in response to the AJAX call

Adjust the tally of search results and modify the selection depending on the frequency of the user's searches within an array of objects

Tips for receiving feedback when uploading multiple images

Encountered an issue retrieving audio during recording with Recorder.js

Parent state in AngularJS, identifying state transitions