Is it possible to retrieve an array from the console.log using selenium webdriver?

Question

Is it possible to retrieve an array from the console.log using selenium webdriver?

Every time I attempt to retrieve the console.log using

driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.log(values);')
time.sleep(4)
for entry in driver.get_log('browser'):
print(entry)

I only see the console output but not the actual array contents which should resemble this screenshot of array

Instead, what I end up with is:

{'level': 'INFO', 'message': 'console-api 2:121 Array(261)', 'source': 'console-api', 'timestamp': 1647448899327}

I've tried multiple solutions to address this issue, but none have been successful for me. Can anyone provide a resolution? Thank you!

EDIT: After discovering this script () that saves the console output to a file when executed manually in the browser, I attempted to integrate it into Selenium as follows:

driver.execute_script('(function(console){console.save=function(data,filename){if(!data){console.error("Console.save: No data")return}if(!filename)filename="console.json"if(typeof data==="object"){data=JSON.stringify(data,undefined,4)}var blob=new Blob([data],{type:"text/json"}),e=document.createEvent("MouseEvents"),a=document.createElement("a")a.download=filename a.href=window.URL.createObjectURL(blob)a.dataset.downloadurl=["text/json",a.download,a.href].join(":")e.initMouseEvent("click",true,false,window,0,0,0,0,0,false,false,false,false,0,null)a.dispatchEvent(e)}})(console)')

driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.save(values, add);')

The response received is as follows:

[19980:18944:0316/183240.115:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 13 errno 112: database or disk is full sql: CREATE TABLE IF NOT EXISTS previews_v1 (host_name VARCHAR NOT NULL, time INTEGER NOT NULL, opt_out INTEGER NOT 
NULL, type INTEGER NOT NULL, PRIMARY KEY(host_name, time DESC, opt_out, type))
[19980:18944:0316/183240.121:ERROR:database.cc(1777)] OptOutBlacklist SQLite error: code 1 errno 112: no such table: enabled_previews_v1 sql: SELECT type, version FROM enabled_previews_v1
...

javascript selenium web-scraping selenium-chromedriver

Answer 1

Answer №1

I believe the issue at hand is related to what's known as the XYProblem.

You're inquiring about extracting output from console.log(), but it appears that the real challenge lies in obtaining data from Highcharts and integrating it into Python - this doesn't necessarily require the use of console.log().

In JavaScript, you can utilize return values to fetch the data and then work with it in Python.

This process does not even necessitate the utilization of JSON.stringify() in JavaScript or json.loads() in Python.

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(data)

UPDATE:

By using .map() instead of forEarch(), you can condense the JavaScript code into one line:

return Highcharts.charts[0].series[0].data.map(d => d.y);

Here's a simple example utilizing https://www.highcharts.com/demo/line-basic

from selenium import webdriver
#from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
import time

url = 'https://www.highcharts.com/demo/line-basic'

#driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

driver.get(url)

time.sleep(5)

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(type(data), data)

# or shorter

data = driver.execute_script('return Highcharts.charts[0].series[0].data.map(d => d.y);')

print(type(data), data)

Outcome:

<class 'list'> [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]

UPDATE:

For instance, you could use this method to convert timestamps (in milliseconds) represented by x to dates like MM/DD/YY or YYYY-MM-DD

import datetime

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

new_data = []

for x in data:
    dt = datetime.datetime.fromtimestamp(x//1000)
    new_data.append(d.strftime('%D'))
    print(d.strftime('%D'), '|', d.strftime('%Y-%m-%d'))
        
print('new_data:', new_data)

Outcome:

03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
new_data: ['03/17/22', '03/17/22', '03/17/22', '03/17/22']

If you store the data in a pandas.DataFrame, you can leverage pd.to_datetime() and .dt.strftime('%D') for ease of use, but for computations, keeping it as a datetime object might be more beneficial

import pandas as pd

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

df = pd.DataFrame({'date': data})
print(df)

df['date'] = pd.to_datetime(df['date'], unit='ms')
df['date'] = df['date'].dt.strftime('%D')
print(df)

Outcome:

            date
0  1647212400000
1  1647298800000
2  1647385200000
3  1647471600000
       date
0  03/13/22
1  03/14/22
2  03/15/22
3  03/16/22

Answer 2

I believe the issue at hand is related to what's known as the XYProblem.

You're inquiring about extracting output from console.log(), but it appears that the real challenge lies in obtaining data from Highcharts and integrating it into Python - this doesn't necessarily require the use of console.log().

In JavaScript, you can utilize return values to fetch the data and then work with it in Python.

This process does not even necessitate the utilization of JSON.stringify() in JavaScript or json.loads() in Python.

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(data)

UPDATE:

By using .map() instead of forEarch(), you can condense the JavaScript code into one line:

return Highcharts.charts[0].series[0].data.map(d => d.y);

Here's a simple example utilizing https://www.highcharts.com/demo/line-basic

from selenium import webdriver
#from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
import time

url = 'https://www.highcharts.com/demo/line-basic'

#driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

driver.get(url)

time.sleep(5)

data = driver.execute_script('''
values = [];
Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
return values;
''')

print(type(data), data)

# or shorter

data = driver.execute_script('return Highcharts.charts[0].series[0].data.map(d => d.y);')

print(type(data), data)

Outcome:

<class 'list'> [43934, 52503, 57177, 69658, 97031, 119931, 137133, 154175]

UPDATE:

For instance, you could use this method to convert timestamps (in milliseconds) represented by x to dates like MM/DD/YY or YYYY-MM-DD

import datetime

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

new_data = []

for x in data:
    dt = datetime.datetime.fromtimestamp(x//1000)
    new_data.append(d.strftime('%D'))
    print(d.strftime('%D'), '|', d.strftime('%Y-%m-%d'))
        
print('new_data:', new_data)

Outcome:

03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
03/17/22 | 2022-03-17
new_data: ['03/17/22', '03/17/22', '03/17/22', '03/17/22']

If you store the data in a pandas.DataFrame, you can leverage pd.to_datetime() and .dt.strftime('%D') for ease of use, but for computations, keeping it as a datetime object might be more beneficial

import pandas as pd

data = [1647212400000, 1647298800000, 1647385200000, 1647471600000]

df = pd.DataFrame({'date': data})
print(df)

df['date'] = pd.to_datetime(df['date'], unit='ms')
df['date'] = df['date'].dt.strftime('%D')
print(df)

Outcome:

            date
0  1647212400000
1  1647298800000
2  1647385200000
3  1647471600000
       date
0  03/13/22
1  03/14/22
2  03/15/22
3  03/16/22

Answer 3

Answer №2

I managed to resolve the issue by including an additional JavaScript code.

driver.execute_script("""(function(console){

    console.save = function(data, filename){

        if(!data) {
            console.error('Console.save: No data')
            return;
        }

        if(!filename) filename = 'console.json'

        if(typeof data === "object"){
            data = JSON.stringify(data, undefined, 4)
        }

        var blob = new Blob([data], {type: 'text/json'}),
            e    = document.createEvent('MouseEvents'),
            a    = document.createElement('a')

        a.download = filename
        a.href = window.URL.createObjectURL(blob)
        a.dataset.downloadurl =  ['text/json', a.download, a.href].join(':')
        e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
        a.dispatchEvent(e)
    }
})(console)""")
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });')
driver.execute_script('console.save(values, add);')

This action will store the array as a JSON file.

Answer 4

I managed to resolve the issue by including an additional JavaScript code.

driver.execute_script("""(function(console){

    console.save = function(data, filename){

        if(!data) {
            console.error('Console.save: No data')
            return;
        }

        if(!filename) filename = 'console.json'

        if(typeof data === "object"){
            data = JSON.stringify(data, undefined, 4)
        }

        var blob = new Blob([data], {type: 'text/json'}),
            e    = document.createEvent('MouseEvents'),
            a    = document.createElement('a')

        a.download = filename
        a.href = window.URL.createObjectURL(blob)
        a.dataset.downloadurl =  ['text/json', a.download, a.href].join(':')
        e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
        a.dispatchEvent(e)
    }
})(console)""")
driver.execute_script('values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });')
driver.execute_script('console.save(values, add);')

This action will store the array as a JSON file.

Is it possible to retrieve an array from the console.log using selenium webdriver?

Answer №1

Answer №2

Similar questions

What is the process for retrieving user data from Postgresql using axios.get when the data is already stored within the database?

Installing Vue JavaScript using the npm command

Error: The ng-click directive is encountering a parsing syntax error. The token 'Object' is unexpected and is causing the error, it is expected to be enclosed in

Adaptive video player for complete YouTube experience

Using varying data types in a byte array with Node.js

Incorporating Framer Motion into traditional React class components (non-functional approach)

How can I streamline the process of opening and handling links in Selenium using a loop for maximum efficiency?

Steps for inserting an item into a div container

A comprehensive guide on displaying data in Angular using an API

My three.js program isn't displaying anything on the screen, even though I'm confident that my

Provide users with the option to select a specific destination for saving their file

What is the process for transferring a JavaScript variable to a Java servlet?

Use jQuery to easily add and remove rows from a table dynamically

Getting a specific nested child component from the parent component's slot in the render function: A step-by-step guide

The retrieval of JSON data in a JavaScript function is malfunctioning

Congratulations! Your product has been successfully added to Magento using Ajax

ERROR UnhandledTypeError: Unable to access attributes of null (attempting to retrieve 'pipe')

Pick the item when the checkbox is selected

Configuring the Accept-Language header in PhantomJSDriver within a Play Framework Specification

The occurrence of events for a basic model in Backbone is inexplicably non