Resources for article extraction from HTML pages

Boilerpipe library

Boilerpipe is one of the best open source packages for full article text extraction using machine learning techniques. Both text and structural features of the document are used to train a classifier to predict whether the observed part of the document belongs to the article text or not.

Boilerplate adopts the following features of HTML documents:

• text frequency in the whole corpus: to obtain useless phrases that are not the main text of the article.
• particular tags that enclose a block of text: <h#> headline, <p> paragraph, <a> anchor and <div> division
• shallow text features: average word length, average sentence length, absolute number of words in the segment
• local context of text: absolute and relative position of the text block
• heuristic features: number of words that start with an uppercase letter, number of words written in all-caps, number of date and time tokens, link density and certain ratios of those previously listed
• density of text blocks: number of words in a wrapped fixed column width text block divided by number of lines of the same block

The original html document is transfered into atomic text blocks that are annotated with features listed above and labeled manually with content or boilerpate class.

Extracting Article Text from the Web with Maximum Subsequence Segmentation

This algorithm (MSS) transforms the problem of detecting article text in HTML documents to maximum subsequence optimization. Local token-level classifier outputs a sequence of scores for applying maximum subsequence optimization. Indexes of the subsequence are used to generate the set of tokens to represent the extracted text.

Given a sequence of numbers, the task is to find a continuous subsequence where the sum of it’s elements is maximal. This is easy if the elements of the sequence are all non-negative. In other words:

In MSS, a document is tokenized using the following steps:

1. discard everything between <script> and <style> tags
2. break up HTML into a list of tags, words and numbers
3. apply porter stemming to all words
4. generalize numeric tokens

To apply maximum subsequence optimization, local token-level classifiers are used to find a score for each token of the document. A negative score indicates that the observed token is not likely to be considered as content and vica-versa for tokens with positive scores. For the Experiments, Naive Bayes model is trained with 2 types of features for every labeled token in the document:

• trigram of token: the token itself and its 2 successors
• parent tag of token in the DOM tree (this can easily be implemented by maintaining a stack of tags when passing through the token array)

Each score produced by the NB classifier is then transformed with f(p) = p - 0.5 to obtain a sequence of scores ranging from [-0.5, 0.5].

Text Extraction from the Web via Text-to-Tag Ratio

If MSS transformed the problem of article text extraction to maximum subsequence optimization; this approach transforms it to histogram clustering.

A Cluster algorithm is adopted to the text-to-tag ratio array (TTRArray) that is generated using the following steps:

1. delete all empty lines and the contents between <script> tags from the original document
2. initialize the TRRArray
3. for each line in the html document
   1. let x be the number of non-tag ASCII characters
   2. let y be the number of tags in that line
   3. if there are no tags in the current line than TTRArray[ current line ] = x
   4. otherwise TTRArray[ current line ] = x/y

The resulting TTRArray holds a text-to-tag ratio for every line in the filtered HTML document.

Clustering can be applied to the TTRArray to obtain lines of text that represent the article by considering the following heuristic:

TTRArray (image courtesy of Weninger & Hsu: Text Extraction from the Web via Text-to-Tag Ratio)

For each k in TTRArray, the higher the TTR is for an element k relative to the mean TTR of the entire array the more likely that k represents a line of content-text within the HTML-page.

Prior to clustering, TTRArray is passed by a smoothing function to prevent the loss of short paragraph lines that might still be part of the content at the edges of the article.

Weninger & Hsu propose the following clustering techniques to obtain article text:

• K-means
• Expectation Maximization
• Farthest First
• Threshold clustering; using standard deviation as a cut-of threshold

VIPS: a Vision based Page Segmentation Algorithm

Semantic tree structure (source http://www.zjucadcg.cn/dengcai/VIPS/VIPS.html)

This algorithm is different from others in that it is making full use of the visual page layout features.

It first extracts blocks from the HTML DOM tree and assigns them a value that indicates the coherency of the block based on visual perception. A hierarchical semantic structure is then employed to represent the structure of the website. The algorithm applies vertical and horizontal separators to the block layout to construct such a semantic structure.

The constructed structure is then used to find parts of the website that represents article text and other common building blocks.

Readabillity

Readability is the most popular article text extraction tool. Readability relies solely on common HTML coding practices to extract article text, title, corresponding images and even the next button if the article seems to be segmented into several pages.

It provides both python and ruby and implementations.

Reference:

https://www.quora.com/Whats-the-best-method-to-extract-article-text-from-HTML-documents

http://cleaneval.sigwac.org.uk/