Best Library To Parse HTML With Python 3 And Example?


Answer :

Web-scraping in Python 3 is currently very poorly supported; all the decent libraries work only with Python 2. If you must web scrape in Python, use Python 2.

Although Beautiful Soup is oft recommended (every question regarding web scraping with Python in Stack Overflow suggests it), it's not as good for Python 3 as it is for Python 2; I couldn't even install it as the installation code was still Python 2.

As for adequate and simple-to-install solutions for Python 3, you can try the library's HTML parser, although quite barebones, it comes with Python 3.


If your HTML is well formed, you have many options, such as sax and dom. If it is not well formed you need a fault tolerant parser such as Beautiful soup, element tidy, or lxml's HTML parser. No parser is perfect, when presented with a variety of broken HTML, sometimes I have to try more then one. Lxml and Elementree use a mostly compatible api that is more of a standard than Beautiful soup.

In my opinion, lxml is the best module for working with xml documents, but the ElementTree included with python is still pretty good. In the past I have used Beautiful soup to convert HTML to xml and construct ElementTree for processing the data.


BeautifulSoup, with its version 3.1.0.1 (January 2009) also work with Python 3.x.

I do not have have direct experience with BeautifulSoup under Py3k (although this soon should change...).   I just read, however, that Version 3.1.0 of Beautiful Soup does significantly worse on real-world HTML than its previous versions, so I may try and wait if possible (i.e. stay with Python 2.6 a bit longer).


Comments

Post a Comment

Popular posts from this blog

Are Regular VACUUM ANALYZE Still Recommended Under 9.1?

Answer : VACUUM is only needed on updated or deleted rows in non-temporary tables. Obviously you're doing lots of INSERTs but it's not obvious from the description that you're also doing lots of UPDATEs or DELETEs. These operations can be tracked with the pg_stat_all_tables view, specifically the n_tup_upd and n_tup_del columns. Also, even more to the point, there is a n_dead_tup column that tells, per table, how much rows need to be vacuumed. (see Monitoring statistics in the doc for functions and views related to statistics gathering). A possible strategy in your case would be to suppress the scheduled VACUUM, keeping an eye on this view and checking on which tables the n_dead_tup is going up significantly. Then apply the aggressive VACUUM to these tables only. This will be a win if there are large tables whose rows never get deleted nor updated and the aggressive VACUUM is really necessary only on smaller tables. But keep running the ANALYZE for the optimiz
Read more

Can Feynman Diagrams Be Used To Represent Any Perturbation Theory?

Answer : Diagram machinery works also for perturbation theory in classical statistical mechanics and classical field theories. Generally, various kinds of diagrams constitute a pictorial way of talking about tensor products and their contractions while hiding the multi-linear algebra from the layman. In the simplest case, vertices (or blobs) represent vectors, matrices, tensors; vertices have ingoing and/or outgoing ports; ingoing ports denote (contravariant) ro indices; outgoing ports denote (covariant) co indices; directed arcs between blobs give a pair of equal indices summed over; different base spaces ⇔ \Leftrightarrow ⇔ different arc types; symmetric tensors ⇔ \Leftrightarrow ⇔ undirected diagrams; no labels are needed for internal lines. In many cases, the directed arcs are decorated (as thin, thick, broken,wavy, curly lines), each decoration indicating the presence of a so-called propagator, a function of its label to use as a weight in the sum, which may beco
Read more