Gran parte de la web utiliza API relajantes para mover datos hacia y desde los servidores. Descansar es un concepto sencillo que no tiene nada que ver con dormir; descansar medio Transferencia de estado representacional , pero este artículo no trata sobre descansar . Este artículo trata sobre un mecanismo complicado para solucionar la falta de una API de descanso cuando realmente necesitas recuperar algunos datos de un sitio.
Es posible que los sitios web no quieran que usted haga esto por varias razones: el ancho de banda para datos cuesta dinero, las licencias de datos cuestan dinero y no ver los anuncios también puede costar dinero. Esta herramienta, selenio , es nominalmente para probar un sitio web, no para acelerar sobre un obstáculo. La mayoría de los sitios tienen condiciones de uso; Puede encontrar los términos de JMP al final de esta página.
Se adjunta un archivo JSL completo. Está escrito para Firefox, Windows y el sitio web de JMP tal como se veía el 8 de mayo de 2022. La parte de Firefox se puede cambiar, probablemente, a muchos otros navegadores. Puede que también funcione en Mac, no se realizaron pruebas. El sitio web de JMP cambiará con el tiempo y será necesario modificar el JSL. Esa es la desventaja de no utilizar una API oficial.
Antes de comenzar, descargue un controlador e instale selenium como se muestra en los comentarios a continuación. También necesitarás Firefox o investigar un poco sobre el controlador de tu navegador preferido.
// JMP + Python + Selenium + Firefox (probably works with Chrome or Edge with a few tweaks)
// https://firefox-source-docs.mozilla.org/testing/geckodriver/index.html
// https://selenium-python.readthedocs.io/
// downloaded https://github.com/mozilla/geckodriver/releases -- geckodriver-...-win64.zip (pick latest)
// expand on desktop geckodriver.exe
// installed selenium python -m pip install selenium
// the following code is welded into the JMP.COM's HTML of 8May2022. It could change at anytime.
// the functions are sprinkled through the JSL below, near where I first needed them, reused later.
// you'll need similar functions, probably, and want to consult the Python Selenium bindings to write them.
// I only wrote enough to make this simple example work. There is no actual need to be logged in to JMP.
//
// Using try...except on snips of python code makes debugging much easier. JMP loses the exception message without it.
//
// the JSL functions are thin wrappers around the python calls to selenium which (I believe) is
// a thin wrapper to generate an API call to the gecko (etc) driver. I *think* it might be possible to
// remove the python layer and load/call the driver directly from JSL. But Selenium is documented.
Python Init(); // one-time startup...
// 0: startup
xrc = Python Execute( {}, {By_ID, By_XPATH, rc},
"\[
try:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
#
options = webdriver.FirefoxOptions()
#options.add_argument("--private") # example. you most likely don't want private, some things don't work.
#
service=Service(r'C:\Users\v1\Desktop\geckodriver.exe')
# sometimes people use "browser" rather than "driver". It will be used below.
driver = webdriver.Firefox(service=service,options=options)
# return two magic values. you may need some others, just add them in the same way...
By_ID = By.ID
By_XPATH = By.XPATH
rc = "ok"
except Exception as e:
rc = repr(e)
]\"
);
If( xrc != 0 | rc != "ok", Throw( "start up Selenium failed" || Try( ": " || Char( rc ), "" ) ) );
PythonInit() solo debe realizarse una vez; conecta JMP a Python y tarda unos segundos la primera vez. Puedes volver a llamarlo sin penalización. PythonExecute(...) no envía variables pero recupera tres del código que ejecuta. Se necesita un poco para cargar todo e iniciar el navegador.
You are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPYou are looking at an empty browser controlled by JMPEstás viendo un navegador vacío controlado por JMP.
Abra la página JMP.COM a continuación. Es posible que veas una redirección que normalmente pasa desapercibida.
// 1: navigate to jmp.com
nav = Function( {url}, {rc},
Python Execute( {url}, {rc},
"\[
try:
driver.get(url)
rc = "ok"
except Exception as e:
rc = repr(e)
]\" );
return(rc);
);
rc = nav( "https://www.jmp.com/" );
if( rc != "ok", throw("nav: "||char(rc)));
La función de navegación devuelve "ok" o un mensaje de error. La página web de JMP se carga en el navegador. Ignora a las personas en la captura de pantalla.
The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.The icon means the browser is remote controlled.El icono significa que el navegador está controlado de forma remota.
Cree un script para iniciar sesión en el sitio JMP. Haga clic derecho en el botón Iniciar sesión para conocer el valor de identificación HTML del botón. Recuerda cómo hacer esto; Me saltaré esta explicación al final...
F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 might bring you to the next screen, but this way the control will already be selected.F12 puede llevarlo a la siguiente pantalla, pero de esta manera el control ya estará seleccionado.
Y la consola del desarrollador se abre mostrando la identificación del control. Más abajo habrá controles que tendrán una clase pero no una identificación. XPATH de Selenium puede manejarlo. El truco es similar a la navegación por el cuadro de visualización: encontrar una ruta que no sea demasiado frágil y que sea lo suficientemente específica.
Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Use the button id in the JSL that follows.Utilice la identificación del botón en el JSL que sigue.
Las identificaciones suelen ser la mejor opción cuando están disponibles porque son únicas en la página. WaitID espera hasta 10 segundos hasta que aparezca el botón de inicio de sesión. Quizás no sea necesario esperar. No lleva tiempo si ya está ahí.
// 2: login with userid/password credentials
// wait for an id to be available
waitID = Function( {id, timeout = 10, BYformat=By_ID}, {rc},
Python Execute( {id, timeout, BYformat}, {rc},
"\[
try:
myElem = WebDriverWait(driver, timeout).until(EC.presence_of_element_located((BYformat, id)))
rc = "ok"
except TimeoutException:
rc = "timeout"
except Exception as e:
rc = repr(e)
]\"
);
Return( rc );
);
rc = waitID( "loginButton", 5 );
If( rc != "ok", Throw( "no login button: " || char(rc) ) );
// click a button ID
clickID = Function( {id, BYformat=By_ID}, {rc},
Python Execute( {id, BYformat}, {rc},
"\[
try:
driver.find_element(BYformat, id).click()
rc = "ok"
except Exception as e:
rc = repr(e)
]\"
);
return(rc);
);
rc = clickID( "loginButton" );
if( rc != "ok", throw("login button: "||char(rc)));
¡Fresco! Aparece el letrero en la pantalla. Busque el campo de nombre de usuario a continuación.
Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Now get the id for the user name field by right-click...Ahora obtenga la identificación para el campo de nombre de usuario haciendo clic derecho...
Nuevamente, espere el campo esperado. Ahora se necesita una función de pulsación de tecla...
rc = waitID( "idp-discovery-username", 10 );
If( rc != "ok", Throw( "no username field: " || char(rc) ) );
// type a value into a field
keysToID = Function( {id, txt}, {rc},
Python Execute( {id, txt}, {rc},
"\[
try:
driver.find_element(By.ID, id).send_keys(txt)
rc = "ok"
except Exception as e:
rc = repr(e)
]\"
);
return(rc);
);
rc = keysToID( "idp-discovery-username", Include( "$documents/UserID.jsl" ) ); // file contains "AliBaba@1000&OneNights.com", in quotation marks, possibly encrypted
if( rc != "ok", throw("keysToID username: "||char(rc)));
Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Fake user name for fake password.Nombre de usuario falso para contraseña falsa.
Mi ID de usuario se desplaza fuera de la pantalla y el botón siguiente está visible... encuentra su nombre...
Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Click the Next button to get the password prompt.Haga clic en el botón Siguiente para que se le solicite la contraseña.
haz clic en él, luego espera el campo de contraseña...
// click Next button
rc = clickID( "idp-discovery-submit" );
if( rc != "ok", throw("click submit user name: "||char(rc)));
// wait for password field
rc = waitID( "okta-signin-password", 5 );
If( rc != "ok", Throw( "no password field: " || char(rc) ) );
luego ingrese la contraseña
// enter password
rc = keysToID( "okta-signin-password", Include( "$documents/password.jsl" ) ); // file contains "OpenSesame", in quotation marks, possibly encrypted
if( rc != "ok", throw("keysToID password: "||char(rc)));
y repite el proceso...busca el botón de iniciar sesión...
After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.After entering the password, click the sign in button.Después de ingresar la contraseña, haga clic en el botón de iniciar sesión.
Pinchalo
// click Sign In
rc = clickID( "okta-signin-submit" );
if( rc != "ok", throw("click signin submit: "||char(rc)));
Estamos registrados.
Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Must be signed in, there is an edit profile choice.Debe iniciar sesión, hay una opción de edición de perfil.
Hay un campo de búsqueda en la imagen. Escriba JSL y haga clic en la lupa. "searchField" es la identificación. Se puede hacer clic en la lupa, pero Selenium tiene un mecanismo de envío de formulario que funcionará con el campo de búsqueda, que es un campo de entrada en el formulario.
The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.The search field needs a bigger window to be visible, here it is.El campo de búsqueda necesita una ventana más grande para ser visible, aquí está.
Hay algo de JavaScript asincrónico que carga algunas partes de la página. Es posible que no sea necesario esperar por algún campo en particular si el campo se carga como parte de la página.
// 3: query for articles about JSL
rc = waitID( "searchField" );
If( rc != "ok", Throw( "no search field: " || char(rc) ) );
// enter a search string. "jsl" currently retruns 3 pages
rc = keysToID( "searchField", "jsl" );
if( rc != "ok", throw("keysToID search field: "||char(rc)));
submitForm = Function( {id}, {rc},
Python Execute( {id}, {rc},
"\[
try:
driver.find_element(By.ID, id).submit()
rc = "ok"
except Exception as e:
rc = repr(e)
]\"
);
return(rc);
);
rc = submitForm( "searchField" ); // submit form works OK off of this input field.
if( rc != "ok", throw("submitForm searchField: "||char(rc)));
Ahora prepárese para revisar los resultados. Los elementos del listado de varias páginas se ven así
The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.The pink outer element holds three inner parts: title, description, link.El elemento exterior rosa contiene tres partes interiores: título, descripción y enlace.
hay una lista de elementos de datos con un círculo rosa que abarca varias páginas.
// 4: page through the results to capture them
getElements = Function( {id, BYformat=By_ID}, {rc},
Python Execute( {id, BYformat}, {rc},
"\[
try:
list = driver.find_elements(BYformat,id)
rc = "ok"
except Exception as e:
rc = repr(e)
]\"
);
Return( rc );
);
getNElements = function({},{n},
Python Execute( {}, {n},
"\[
try:
n = len(list)
except Exception as e:
print(repr(e))
n = -1
]\"
);
return(n);
);
getElementItext = function({i,id, BYformat=By_ID},{txt},
Python Execute( {i, id, BYformat}, {txt},
"\[
try:
txt = list[int(i)].find_element(BYformat, id).text
except Exception as e:
txt = "Error: getElementItext: " + repr(e)
]\"
);
return(txt);
);
getElementIattribute = function({i,id, BYformat=By_ID, attr},{txt},
Python Execute( {i, id, BYformat, attr}, {txt},
"\[
try:
txt = list[int(i)].find_element(BYformat, id).get_attribute(attr)
except Exception as e:
txt = "Error: getElementIattribute: " + repr(e)
]\"
);
return(txt);
);
Arriba: algunas funciones para usar en el bucle siguiente. Hay botones en la parte inferior de la página para ir a la página siguiente; ejecutan algo de JavaScript que destruye y recrea la lista de elementos. Las funciones se llaman nuevamente para recuperar la nueva lista. JSL y Python son lo suficientemente buenos para este ejemplo. Se descompondrán si hay más de una lista para realizar un seguimiento al mismo tiempo; consulte Python lista variable. Estoy bastante seguro de que es necesario esperar a que se carguen los datos después de cada página siguiente...
dt = New Table( "articles",
New Column( "link", character,
Set Property("Event Handler",
Event Handler(
Click(JSL Quote(Function( {thisTable, thisColumn, iRow}, Web( Char( thisTable:thisColumn[ iRow ] ) ); );)),
Tip(JSL Quote(Function( {thisTable, thisColumn, iRow}, "Open " || Char( thisTable:thisColumn[ iRow ] ) || " in your browser."; );)),
Color(JSL Quote(Function( {thisTable, thisColumn, iRow}, RGBColor("link"); );))
)
)
),
New Column( "title", character ),
New Column( "description", character )
);
// this while loop will grab screens of answers and break() when the NextScreen button goes dim
while(1,
// not sure, yet, what to wait for. maybe the card(s)...
// this html is updated in the background by a json ajax mechanism that
// is hard to see.
rc = waitID("//div[@id='searchresults']//div[@class='result-card']",10,By_XPATH);
If( rc != "ok", Throw( "no search results" ) );
// get the current set
rc = getElements("//div[@id='searchresults']//div[@class='result-card']",By_XPATH);
if( rc != "ok", throw("getElements: "||char(rc)));
n = getNElements();
for(i=0,i<N></N>
En este punto el navegador está abierto y esta tabla está en la pantalla.
Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Today there were 57 entries spanning three pages.Hoy hubo 57 entradas que abarcaron tres páginas.
Es hora de cerrar el navegador.
// 5: quit
Python Submit(
"\[
driver.quit() # close the browser
]\" );
Python Term();
Hacia el final hay un XPATH
rc = clickID("//ul[@id='pager']//a[@class='pager-next']",By_XPATH);
eso significa
// - somewhere below the root of the document find a
ul - a tag (some sort of HTML list)
[@id='pager'] - the list has this id
// - more nested tags, followed by...
a = a tag (link)
[@class='pager-next']
Puede que no sea necesario que sea tan complicado. Es un ejemplo que utiliza una identificación única para encontrar un elemento que podría no ser único si solo se considerara la clase.
