Deciphering Korean for crawling using got

Question

Deciphering Korean for crawling using got

I am currently attempting to perform web scraping on a website using the got library.

Below is the simple code that I have written:

import got from 'got';

async function test(){
  const data = await got('https://dhlottery.co.kr/store.do?method=topStore&pageGubun=L645', { encoding: 'utf8'});
  console.log(data.body);

}
test();

The code works fine, but it does not display Korean characters correctly.

An excerpt of the output is shown below:

<div class="foot_txt2">
  <p>Copyright (c) 2018 ȸ&amp;ູ. All rights reserved</p>
  <p> Ȩ Խõ ̸ ּҰ ڵ Ǵ  źϸ, ̸ ݽ Ÿ  ó Ͽ ֽñ ٶϴ.</p>
  <p class="f_blue2">ûҳ  ϰų ÷   ϴ.</p>
</div>

All the corrupted words appear to be in Korean.

I am curious as to why this issue is occurring and how it can be resolved.

javascript encoding web-crawler domcrawler

Answer 1

Answer №1

I am not familiar with this particular package and have not tested the solution provided below, but it could potentially resolve your issue.

In the example given, you are specifying the utf8 encoding, however, the website in question actually uses EUC-KR encoding...

https://i.sstatic.net/EzrC2.png

To address this issue, try updating the encoding property in your request as shown below:

import got from 'got';

async function test(){
  const url = 'https://dhlottery.co.kr/store.do?method=topStore&pageGubun=L645';

  const data = await got(url, {
    encoding: 'EUC-KR'
  });

  console.log(data.body);

}
test();

Answer 2

I am not familiar with this particular package and have not tested the solution provided below, but it could potentially resolve your issue.

In the example given, you are specifying the utf8 encoding, however, the website in question actually uses EUC-KR encoding...

https://i.sstatic.net/EzrC2.png

To address this issue, try updating the encoding property in your request as shown below:

import got from 'got';

async function test(){
  const url = 'https://dhlottery.co.kr/store.do?method=topStore&pageGubun=L645';

  const data = await got(url, {
    encoding: 'EUC-KR'
  });

  console.log(data.body);

}
test();

Deciphering Korean for crawling using got

Answer №1

Similar questions

Angular allows for the creation of dynamic content, much like the append function in jQuery

I love the idea of the music playing on as I navigate between pages or tabs. Such a cool feature with Next

Choose a single image by clicking on the <img> tag

The sequence of error middleware in Express4

JavaScript - splitting numbers into multiple parts

What is the technique for filtering multiple values using the OR operation in ng-model functions?

Using ValidationGroup to trigger JavaScript calls from controls

Emberjs 1.0: Data Changes don't Refresh Computed Property and Template

Encountering a 404 error when attempting to use the jQuery .load() function with parameters that include periods ('.')

Implementing a feature in ReactJS that allows users to upload multiple images in base64 format

Problems with Navbar rendering on multiple occasions

Is it possible to rotate an image with a random angle when hovering in Angular?

Having trouble loading an image with texture loader in Three.js? The issue may be related to the size and scale of the plane geometry

Testing in NodeJS - revealing the application

Modify the DOM at a later time once the images have loaded. This particular action is being executed within the context of an

Creating dynamic grid data based on various filter criteria using JavaScript (AngularJS)

I encountered Hydration errors while attempting to use dangerouslySetInnerHTML to render a script on my webpage

Exploring location-based services using React-Redux

When positioned at high or low angles, the camera starts acting strangely

Incorporating PWA functionality into Next.js for seamless notifications and push notifications