nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Pulling data from a webpage with multiple table pages

jearls11

Hey Folks,

I am pulling data from a webpage but running into a problem, I cant provide the web address due to IP.

when i pull the page it opens a table but not all the rows - it opens the first 1000 rows:

open("URL", HTML Table(2));

The problem is it has 3 pages if that makes sense see screen shot

my question is how do I pull the next two pages?
Thank you

Accepted answers

All comments

Jed_Campbell

When you select the next page on the website in a browser, does the URL change in a way that you can decipher? If so, you could run your open script multiple times, one for each page/URL, then use the Concatenate() command to bring all the tables together.

jearls11

I tried this the URL remains the same, i tried page=2. Unfortunately as its an internal system I cannot share

Craige_Hales

There are several approaches, in addition to the @Jed_Campbell idea (which is the easiest if the browser's address bar shows an address change in a predictable way.)

0. If the site provides an API, possibly for rest services, that will be the best choice. Maybe you can ask the owner of the system for an API.

or, 1. Use F12 in your browser (I'm using FireFox, but Chrome and Edge are similar). Pick the network tab which shows you what is sent over the network. Click the next page button on the web site and start studying the requests that are sent. It's hard to say for sure what you are looking for, but it will often be JSON data. You'll see if it was a POST or GET and what sort of headers were used. If there is a choice, JSON will likely be better than HTML.

or, 2. This will be significantly harder, but will often work when other choices fail. Browser Scripting with Python Selenium

Scraping data from web pages is hard in the best case. In the worst case, some sites are actively trying to prevent automated bots from working, generally by detecting not a real browser or not a real human. Sites that don't update the address bar are using some sort of ajax-like protocol with JavaScript; that will show up in the F12 window, perhaps with an easy-to-decode URL and headers. Sometimes with a cookie or encoded parameters or password.

JMP has special handling for HTML tables, but that might not work for you unless there is a simple URL that returns each page as HTML with a table. JMP also has JSON, XML, and CSV import wizards that might help.

lala

If there is JavaScript in the web page to improve the request of the timestamp algorithm conversion, how to use JSL to implement this JavaScript algorithm, so that JSL can download to the web data smoothly?

Thank Craige!