Getting data from a latin1 (iso-8859-1) database using javascript/nodejs: Tips and Tricks

My ancient mysql database (mysql 5.0.2) is in latin1 format and I'm encountering an issue with getting data from it. Non-ascii characters such as Â, À, and Á are appearing as 'ef bf bd' in hex, which means different characters are being displayed the same way.

I need assistance in retrieving these characters correctly so I can match each one to its corresponding utf-8 character.

I've attempted changing the charset, but unfortunately, it has not resolved the problem for me!

Could someone please assist me in extracting meaningful data?

var mysql = require('mysql')

var con = mysql.createConnection({
    host: "localhost",
    user: "root",
    //charset: "utf8mb4",
    //charset: "utf8",
    charset: "latin1",
    database : 'my_db'
})

con.connect()

var query = con.query("SELECT data from my_table where id='07'", function 
(error, results, fields) {
    var b = Buffer.from (results[0].data)
    console.log ('Retrieved data in hex -> ', b)
})

con.end()

Whenever I update the data in the database to contain only ascii characters, I can successfully retrieve the information in JavaScript without any issues. However, when I substitute this data with characters like 'á' or 'à', I consistently receive 'ef bf bd' in hex (-17 -65 -67 in decimal).

Answer №1

When dealing with Latin1 hex code EF BF BD, it translates to �, but if you are talking about a BOM (Byte-Order-Mark), then the code is EF BB BF which appears as . You can read more about this here.

Issues like Â, À, and Á often arise when incorrectly mixing latin1 and utf8 encoding.

If you need further assistance, provide a hexadecimal snippet of text along with the characters or visit this link for troubleshooting common problems: Trouble with UTF-8 characters; what I see is not what I stored.

In older versions like MySQL 4.0, character sets were not well-defined. Operations such as LENGTH() provided byte counts instead of character counts, leading to inaccurate results with non-ascii characters in string comparisons (col_a < col_b).

To play it safe, consider doing minimal processing in PHP with your characters. Directly store and retrieve them in VARCHAR/TEXT (or BLOB) columns without manipulation.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Browser lacks proper credentials for passport authentication

I'm currently learning how to incorporate user authentication using passport.js. I have successfully set up a basic passport "local" strategy on the server side, along with a single POST route for logging in users. When testing with insomnia, everythi ...

Utilize promise-style for Sequelize associations instead, please

I am in the process of merging information from 3 tables - Products, Suppliers, and Categories. My goal is to retrieve the row with SupplierID = 13. I recently came across a helpful explanation on How to implement many to many association in sequelize, whi ...

The technique to retrieve the row from a table with the highest value

Below is the table I am working with: id date custid billno month amount balance 64 07-Jun-2015 1 102 5 9192 0 134 05-Jul-2015 1 172 6 9744 0 235 01-Aug-2015 1 277 7 4032 0 435 ...

I am currently struggling with the mapquest GeoJson integration within my Node/Express application. I keep encountering an error message that reads "body used already."

I attempted to utilize the mapquest API by following the provided documentation, however I encountered an error message: HttpError: body used already for: Below is my geoCoder.js configuration: const NodeGeocoder = require('node-geocoder'); con ...

Problem encountered in NextJS/ReactJS when attempting to dynamically load a new component by clicking a button within the current component

In my NextJS project, I am working with 3 components named "Sidebar", "Woven", and "ToolsPage". Below are the respective codes for each: ToolsPage Component: "use client" import Woven from './components/weaved'; import Sidebar from &ap ...

Converting PHP arrays into JavaScript arrays with the help of json_encode()

Is there a way to pass an array from PHP to the JavaScript function console.log()? I'm currently simulating a database and need help with this specific task. Despite not having an array declared in my code, I attempted to use the .getJSON() function w ...

Effectively handle multiple connections from nodejs to postgres using the pg library

I need to run a script that performs multiple queries using the pg library for managing connections. However, I am facing an issue where my program stops working when the connection pool is full and does not queue future queries. I have tried setting the p ...

Automatically Switch Font Colors to Red and Green

I am looking to automate the process of changing multiple textbox colors based on the class property. <input type="text" Class="ChangeColor" /> <input type="text" Class="ChangeColor" /> <input type=& ...

Ways to efficiently incorporate data into App.vue from the constructor

My app initialization uses main.js in the following way, import App from './App.vue'; const store = { items: [{ todo: 'Clean Apartment.', },{ todo: 'Mow the lawn!', },{ todo: 'Pick up ...

submission of form data and automatic redirection to another page

Seeking advice on managing form submissions and page redirection. I have a landing page with a basic form, where users select criteria and are then redirected to page2 displaying tabular data and query information. When submitting the form on Page1, data ...

Once the ajax request is finished, load only the <script> tags that have specific ids

I'm currently implementing infinite-scroll to dynamically load more content which includes <script> tags that need to be executed. To achieve this, I have created the following code as an ajax-callback: JS on load ajax callback: function a ...

How can I embed certain returned values into JavaScript for display on a webpage using PHP's PDO?

I recently started learning about MVC and I'm working on retrieving an array of image paths from a MySQL database to create a grid using JavaScript. The controller needs a model that can fetch the paths from the database. Before, I had an ajax call ...

How can you prevent an element from overflowing when employing window.pageYOffset and using a switch case to alter the background color at specified pixel thresholds?

I want the <body> element's background-color to change every 500px as I scroll down. I've scoured the web for answers, but still can't figure out what's going wrong. Please review the code. However, Firefox Browser indicates that ...

Best practice for updating Form.Control text using a custom onChange method

Today, I observed a unique behavior while utilizing a custom onChange in a Form.Control. The text in the field no longer updates when a file is selected. I thoroughly checked the Git documentation HERE, but unfortunately, it does not provide information on ...

PHP JSON encoding is not valid

I'm currently working on a PHP project (version 5.3.1) where I need to send a JSON string to a Python webservice. However, the JSON result generated by json_encode does not validate as a valid JSON according to JSLint. It's worth mentioning that ...

Creating a Structured Database Schema for Multiple Extensive Lists

Imagine planning a travel itinerary. With 20 potential stops to choose from, the possibilities are endless. A typical tour would involve visiting stops 1 through 20 in sequence. However, users have the freedom to create their own unique tours with a minimu ...

My goal is to monitor every action (button) that users take while using the PDF viewer

Each browser displays the view differently, and I am interested in monitoring specific user actions such as button presses on a PDF viewer. I am currently setting up an iframe and configuring its attributes. Is there a way to achieve this? ...

What is the method for displaying x-axis dates below a highchart?

I'm encountering an issue with Highcharts where dates are not showing under the chart when passing series data as an array of objects. See result image The documentation mentions using an object instead of an array ([1649153340000, 45]docs I need t ...

When utilizing useEffect in Next.js, an error may occur stating that the window is

It seems like there is a challenge with executing script code server side in next.js 13 when it needs to be executed client side. Currently, I am trying to implement the bulma calendar found at When importing the required library: import PasswordStrengthB ...

Leveraging AJAX for database variable checks and PHP and JavaScript to redirect to specific pages based on the results

I am currently exploring the use of AJAX to validate variables in a database. The goal is to redirect to a specific page if the validation is successful. However, during this testing phase, I am not actually checking any variables. My focus is on verifying ...