457106_en-US

liliconcom

Cookie Consent issue

Hi All,

when attempting to open the html code of a webpage i only get the Cookie Consent text rather than the page content.

For example, being in the UK and attempting to open the jmp homepage this is what i get:

 

//:*/
Page = open("https://www.jmp.com/");

/*:

String( "<HEA... 2588="" total="" characters="" ...="">

" ) assigned.

//:*/
print (Page);
/*:

"<SCRIPT type="\!"text/javascript\!"">
function callAjax(url, fallbackUrl, callback){
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.onreadystatechange = function(){
        if (xmlhttp.readyState === 4 &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp; xmlhttp.status === 200){
            callback(xmlhttp.responseText);
        }
    }
    xmlhttp.open(\!"GET\!", url, true);
    xmlhttp.timeout = 3000; // time in milliseconds
    xmlhttp.ontimeout = function() {
        console.log(\!"countrycode xhr request timed out\!");
        window.location.replace(fallbackUrl);
    }
    xmlhttp.send();
}

function findInLocales(locales, cookielocale) {
    var i = 0;
    var found = \!"\!";
    while (i &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lt; locales.length &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp; !found) {
        if (locales[i].substring(2) === '_' + cookielocale.toLowerCase()) {
            found = locales[i];
        }
        i++;
    }
    return found;
}

function geoResetURL(locales) {
    var windowReplacement = window.location.href;
    var regex = /\/[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]\//;
    var regextest =  /[a-zA-Z][a-zA-Z]_[a-zA-Z][a-zA-Z]/;
    windowReplacement = windowReplacement.replace('.geo.', '.');
    var match = document.cookie.match(new RegExp('(^| )usr_locale=([^;]+)'));
    var cookielocale = '';
    if (match != null &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp; match.length &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;gt; 2) {cookielocale = match[2];}
    if (!cookielocale.match(regextest)) {
        var result;
        var code = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
        callAjax( '/services/countrycode?'+code, windowReplacement,
                function (result) {
                    var jsonresult = result.replace(/[()]/g, '');
                    cookielocale = findInLocales(locales, (JSON.parse(jsonresult)).address.country_code);
                    if (locales.indexOf(cookielocale) != -1) {
                        windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/');
                    }
                  window.location.replace(windowReplacement);
                });
    } else {
        if (locales.indexOf(cookielocale) != -1) {
            windowReplacement = windowReplacement.replace(regex, '/' + cookielocale + '/'); 
        }
        window.location.replace(windowReplacement);
    }
}

(function () { 
var locales = ['en_us','en_be','de_at','zh_cn','en_dk','fr_fr','de_de','it_it','ja_jp','ko_kr','en_nl','en_ch','en_gb','en_ca','fr_ca','zh_tw','pt_br','es_mx','es_es','en_au','en_hk','en_my','en_ph','en_in','en_sg','es_ar','es_cl','es_co','es_pe']; 
geoResetURL(locales); 
  }) ();</SCRIPT>
<P>Redirecting ...</P>

"
</HEA...>

 

Many thanks for any suggestions in the right direction.

ron

ScriptingRe: Cookie Consent issue

Open Developer Tools in your browser (F12), select Network tab and go to jmp.com. From there take a look at the domains / files and try to find "correct one":

Now try opening that one.

 

This approach might work or at least get you closer, depending on the website and what you want to get from there.

Re: Cookie Consent issue

Thank you @jthi,

I had a look in the developer tools but didn't manage to crack it yet as a command.

i managed to get the html of http://jmp.com/ by performing the following steps:

1) file>> internet open >> webpage.

2) open jmp.com as a webpage

3) accept / not accept the cookies interactively

4) run the following statement:

page = open ("https://www.jmp.com/en_gb/home.html")

notes: cookie approval only lasts within the same jmp session. After that preferences are not remembered.

This method is useful in my case since only fetch information from one domain in each session. therefore, approving manually is not too difficult. if the scrip crosses domains while fetching data it would be nice to somehow automate this.

 

this method also worked for the webpage i was actually fetching data from so it is reliable to that extent.