ARTICLE AD BOX
I am new to python and programming in general and I got to Chapter 13 of Automate The Boring Stuff With Python which teaches Web Scraping, I succeeded in some projects but I'm having trouble getting the HTML from pypi.org, the task is to get the response from a search page in there and open the first few links, I can do it if I manually download the page as a .html file but i can't do it through the URL.
Here is the code that's supposed to work:
import requests, sys, webbrowser, bs4 print('Searching...') # Display text while downloading the search results page. res = requests.get('https://pypi.org/search/?q=' + ' '.join(sys.argv[1:])) res.raise_for_status() # Retrieve top search result links. soup = bs4.BeautifulSoup(res.text, 'html.parser') # Error happens here # Open a browser tab for each result. link_elems = soup.select('.package-snippet') # Returns an empty list num_open = min(5, len(link_elems)) for i in range(num_open): url_to_open = 'https://pypi.org' + link_elems[i].get('href') print('Opening', url_to_open) webbrowser.open(url_to_open)Here is what the beautiful soup object returns: https://pastebin.com/nkfPuh9h
I turned off uBlock so judging by `Failed to load script: ${src}, Please contact the service administrator.` and:
A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings.Maybe it has something to do with Javascript? That code was fully provided by Automate the Boring Stuff so pypi.org could have changed something since then. I would just like to know if it is a mistake on my end or not so I can get this right because at the end I will have to write a script to download images from Imgur or Flickr, but I won't be able to do that if I can't do simple webscraping.
