I am trying to retrieve the HTML webpage from . However, a portion of the HTML file is loaded through JavaScript. When using HTTP.jl to fetch the webpage with HTTP.request(), I only receive the part of the HTML file that was loaded before the execution of JavaScript. As a result, the webpage obtained differs from what Chrome displays. How can I obtain the webpage identical to what Chrome renders? Is it necessary to utilize WebDriver.jl, which serves as a wrapper for Selenium WebDriver's Python bindings?
The snippet of my source code:
function get_page(w::word)::Bool
response = nothing
try
response = HTTP.request("GET", "https://www.collinsdictionary.com/dictionary/$(dictionary)/$(w.org_word)",
connect_timeout=connect_timeout, readtimeout=readtimeout, retries=retries, redirect=true,proxy=proxy)
catch e
push!(w.err_log, [get_page_http_err, string(e)])
return falses
end
open("./assets/org_page.html", "w") do f
write(f, String(response.body))
end
return true
end
dictionary and w.org_word are both of type String, and the function resides within a module.