Filtering out inappropriate language using special characters

Question

Filtering out inappropriate language using special characters

Utilizing https://www.npmjs.com/package/bad-words, I have created a regex to filter out special characters.

const Filter = require('bad-words');
const badWordsFilter = new Filter({replaceRegex:  /[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/g});
badWordsFilter.addWords(['badword', 'şğ'])

The filtering works when the word does not contain Turkish characters. However, when Turkish characters like ş or ğ are included, it does not filter them out.

Is there an issue with my regex?

I came across this code in the documentation:

var filter = new Filter({ regex: /\*|\.|$/gi });
var filter = new Filter({ replaceRegex:  /[A-Za-z0-9가-힣_]/g }); 
//multilingual support for word filtering

javascript regex npm

Answer 1

Answer №1

It appears that there might be an encoding issue with your regex, as it seems to work fine outside of your application. You can check it out here: https://regex101.com/r/VpItfH/3/.

To potentially solve this problem, try encoding the characters in your regex within your application:

You can view the result of the encoded regex here: https://regex101.com/r/VpItfH/4/

Further Details

If you test the encoded regex in a PCRE regex engine, it should work (https://regex101.com/r/VpItfH/5):

/[A-Za-z0-9\x{f6}\x{d6}\x{c7}\x{e7}\x{15e}\x{15f}\x{11e}\x{11f}\x{130}\x{131}\x{dc}\x{fc}_]/g

However, if you are using a JavaScript regex engine, the { and } may cause issues with the Unicode characters. In such cases, simply remove them and replace \x with \u0 if needed. For example, \x{15e} can be written as \u015e

By following these steps, you should be able to achieve the same results as when using

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/g

.

Note: To obtain the unicode representation of a character, you can use "Ğ".charCodeAt(0).toString(16); and then add either \x or \u0 as prefix.

I hope this information proves helpful, and demonstrates that you can encode characters within a regex while still achieving the desired matches. :)

Answer 2

It appears that there might be an encoding issue with your regex, as it seems to work fine outside of your application. You can check it out here: https://regex101.com/r/VpItfH/3/.

To potentially solve this problem, try encoding the characters in your regex within your application:

You can view the result of the encoded regex here: https://regex101.com/r/VpItfH/4/

Further Details

If you test the encoded regex in a PCRE regex engine, it should work (https://regex101.com/r/VpItfH/5):

/[A-Za-z0-9\x{f6}\x{d6}\x{c7}\x{e7}\x{15e}\x{15f}\x{11e}\x{11f}\x{130}\x{131}\x{dc}\x{fc}_]/g

However, if you are using a JavaScript regex engine, the { and } may cause issues with the Unicode characters. In such cases, simply remove them and replace \x with \u0 if needed. For example, \x{15e} can be written as \u015e

By following these steps, you should be able to achieve the same results as when using

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/g

.

Note: To obtain the unicode representation of a character, you can use "Ğ".charCodeAt(0).toString(16); and then add either \x or \u0 as prefix.

I hope this information proves helpful, and demonstrates that you can encode characters within a regex while still achieving the desired matches. :)

Answer 3

Answer №2

Could you please attempt the following code:

let filter = new Filter({ replaceRegex:  /(\w+)/gi });

Make sure to utilize the replaceRegex option.

This pattern will match everything regardless of case sensitivity.

Here is a breakdown of /(\w+)/gi (with appreciation to regex101):

1st Capturing Group (\w+).
1. \w+ matches any word character (equivalent to [a-zA-Z0-9_])
2. + Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
Global pattern flags
1. i modifier: insensitive. Case insensitive match (ignores case of [a-zA-Z])
2. g modifier: global. All matches (don't return after first match)

Answer 4

Could you please attempt the following code:

let filter = new Filter({ replaceRegex:  /(\w+)/gi });

Make sure to utilize the replaceRegex option.

This pattern will match everything regardless of case sensitivity.

Here is a breakdown of /(\w+)/gi (with appreciation to regex101):

1st Capturing Group (\w+).
1. \w+ matches any word character (equivalent to [a-zA-Z0-9_])
2. + Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
Global pattern flags
1. i modifier: insensitive. Case insensitive match (ignores case of [a-zA-Z])
2. g modifier: global. All matches (don't return after first match)

Answer 5

Answer №3

To ensure that your regular expression is Unicode-aware, you should include the u flag. Specifically, modify

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/g

to

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/gu

(added a u at the end). This will function properly in modern browsers (excluding Internet Explorer), as outlined in this resource. Additionally, there are alternative solutions available for older browser compatibility.

Answer 6

To ensure that your regular expression is Unicode-aware, you should include the u flag. Specifically, modify

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/g

to

/[A-Za-z0-9öÖÇçŞşĞğİıÜü_]/gu

(added a u at the end). This will function properly in modern browsers (excluding Internet Explorer), as outlined in this resource. Additionally, there are alternative solutions available for older browser compatibility.

Answer 7

Answer №4

To ensure optimal encoding, convert your JavaScript file to UTF-8 and remember to adjust your meta tag accordingly:

<meta http-equiv="content-type" content="text/html;charset=utf-8" />

Hopefully this adjustment will be beneficial for you.

Answer 8

To ensure optimal encoding, convert your JavaScript file to UTF-8 and remember to adjust your meta tag accordingly:

<meta http-equiv="content-type" content="text/html;charset=utf-8" />

Hopefully this adjustment will be beneficial for you.

Filtering out inappropriate language using special characters

Answer №1

Answer №2

Answer №3

Answer №4

Similar questions

What is the best way to find a partial string match within an array of objects when using Jest?

Steps to fill a select dropdown with numbers ranging from 1 to 10, ensuring that each number has a distinct index or value

Ways to create different sections for ng-repeat data in AngularJS

OpenLayers' circular frames surrounding the icons

Having trouble launching a React Native project on an Android simulator?

What is the best method for having tooltips hover over textboxes?

What is the best way to showcase a collapsible tree using AngularJS and Bootstrap?

Angular TypeScript Directive Link function not being executed

Express is unable to locate the specified property

Facing an issue with sending data in AJAX Post request to Django View

"Encountered a 400 error while trying to add a user in npm

How can one address the issue of undefined data within table cells?

Can the browser console be used to make a $.get request?

Error: The package [email protected] requires @angular/core@^6.0.3 as a peer dependency, but it is nowhere to be found. This error message is being misleading

The passport JWT authorization failed, not a single message is being printed to the console

Encountering a Mongoose error during the upsert operation with findOneAndUpdate

Does CSS offer a way to specify overflow for selections?

Showing or hiding a div element based on a changing boolean value in Angular

<select> dropdown menu for selecting specific files opened by JavaScript

Displaying only a section of a webpage using AJAX