ARTICLE AD BOX
I'm trying to get into this website and retrieve fund status data https://fundfinder.panfoundation.org
I need to log into the website (this is an open website, anybody can create username and pwd to sign in), scroll down, click on 'prostate cancer' and then retrieve all the fund status (locked or unlocked). I'm using the selenium to do webscraping.
This is my code:
from selenium import webdriver from selenium.webdriver.chrome.service import Service from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys import time # Set up Chrome options for headless mode chrome_options = Options() chrome_options.add_argument("--headless") chrome_options.add_argument("--no-sandbox") chrome_options.add_argument("--disable-dev-shm-usage") # Set up the WebDriver service = Service('/usr/local/bin/chromedriver') # Update with the path to your ChromeDriver driver = webdriver.Chrome(service=service, options=chrome_options) # Open the website HOME_PAGE = 'https://fundfinder.panfoundation.org' driver.get(HOME_PAGE) # Wait for the page to load time.sleep(2) # Find the username and password fields and log in username_field = driver.find_element(By.NAME, 'email') password_field = driver.find_element(By.NAME, 'phrase') username_field.send_keys('Your_UserName') # Replace with your username password_field.send_keys('Your_Password') # Replace with your password password_field.send_keys(Keys.RETURN) # Press Enter to log inWhat I'm struggling with is next steps. After log in, I need to scroll down to click on the link 'Prostate Cancer'. I tried many different ways, such as using xpath below. But unfortunately nothing works.
data_element = driver.find_element(By.XPATH, '//div[@class="data"]')I'm new to selenium. Does anyone know how I can achieve the task below:
click on link 'Prostate Cancer' after log in
retrieve the 5 fund status (currently all locked - means unavailable)
Thank you so much in advance!
