1. Introduction

Rapid development of e-commerce has brought vast opportunities to the customers as well as product manufacturers or retailers. The main objective of each product manufacturer is to highlight the special features of the product. Product differentiation is a marketing technique which depicts the differences between the products. Differentiation makes the product more interesting by comparing its unique qualities with other competing products in the market. Successful product differentiation helps product manufacturers to gain competitive advantage in the market. It also helps the customers to choose the right product of their choice. The major factors on which product differentiation can be performed are price, features, quality and availability (time and location) of the product. In this paper, we focus on extracting automatically entities, price and features of the product for the specified product class.

Currently, customers visit different manufacturer or retailers site for purchasing the product. For a particular class of a product, only a limited amount of information is available on a single website regarding the product features. Customers will also find different values of the features as well as prices for the product on various websites. Therefore, the customer has to visit numerous websites to choose the right product. If any retailer or manufacturer has introduced new product feature, the feature information is manually updated on his website. Some websites display a comparison of similar class products. But, this comparison process lacks new feature introduced by another manufacturer.

For carrying out fruitful product differentiation, a web-spreadsheet view is a solution which will handle all above discussed scenarios. Hence, in the proposed research, we focus on novel search problem: Automatic Extraction of Attributes, their corresponding values and Entity Information (AEAVEI). The main objective of AEAVEI is to extract relevant attribute-value information related to entities along with the entity (product) name of the specified product class in the form of a spreadsheet. Web pages are crawled for extraction of attributes and their respective value-measure pairs. Extracted attributes will form the columns of the spreadsheet and values (value-measure pair) along with entity name will form rows of a spreadsheet. This approach provides all product information for particular product attributes under one roof.

AEAVEI is inspired by Relational aggregated Search (RAS). On the web, large amount of relations exists between the information. The goal of the RAS is to find out these relations between the information, extract and aggregate this information in the form of small nuggets. RAS consists of three main parts, namely Query Dispatching (QD), Nuggets Retrieval (NR) and Result Aggregation (RA) [1]. In the proposed work, we mainly focus on relation retrieval between information placed on the web and aggregating the information in spreadsheet format. In relation retrieval, we focused on instance-attribute relation to automatically extract attributes and their respective values and instance-class relation to automatically extract entity (product) name of a particular class. For example, if a user submits a query for the product class Mobile Phone, then the entities belonging to the class Mobile-Phone and attributes and their values of those entities are extracted.

By applying this approach, the effort of user by crawling through each hyperlink of product pages for finding relevant information is reduced. This approach also aggregates updated product information (attribute and value-measure pairs) dynamically in a web-spreadsheet.

Aggregating information in a consolidated spreadsheet is having following challenges:

• •

  On the web page, attributes and their values are not directly mentioned, so correct pairs of attribute and their values has to be extracted.

• •

  On the web page, descriptions of entities are generally stored in an unstructured manner. The product (entity) name may appear on a web page in an ambiguous manner. For example, when user intention is electronic tablet, it may search webpage of medicinal tablet.

2. Related work

Extracting information from unstructured text that is natural language documents have been proposed [2,3]. A template independent wrapper has been developed which exercises labeled news pages from a single site [4]. This wrapper has been used to extract plain texts of news embedded in news article body. The relation between news article and news body is also explored by developing novel features for them. Various text summarization techniques has been discussed to summarize unstructured documents like news articles, nursing narratives and country economic reports [5]. There are some approaches that deal with semistructured documents [6]. In these documents, information is stored in different HTML structures like tables, list and forms. Using ORASS (object-relationship-attribute model), semantics has been efficiently extracted from semistructured data of web documents [7]. The task of designing semistructured databases will become easy using ORASS model.

Extracting entities from web pages is a challenging problem, because entities are dispersed over web pages in an ambiguous and unstructured manner. Nowadays, by identifying instance-class relation to extract entity (instance) from web pages is also demanding. Using predicate and patterns, named entities are extracted from web corpus [8]. In this work, using Bootstrapping a set of rules are created which utilizes domain independent extraction patterns. Accuracy is calculated using mutual information between extracted instances of a class and set of discriminator associated with that class. Various techniques based on ontologies have been applied for enhancing the performance of information management systems. Ontology is one of the best techniques to represent background knowledge in information management systems. Background knowledge or domain knowledge such as likings of the user is very necessary for understanding situations and problems in Information management systems. To understand mutual monitoring for enhancing recommendation quality in autonomous multiagent systems, [9] focused on ontologies and ANNs to serve as behavior and interests of users.

In [10], the framework named as Artequakt has been developed. Artequakt utilizes ontology relation declaration and lexical data to identify a relation between entities. These entities are stored in unstructured web documents. In this work initially, an ontology for the artists and paintings has been created. Using various information extraction (IE) tools and techniques, ontology is automatically populated with the information extracted from web documents by utilizing ontology representation and WordNet lexicons. The information is then stored in the knowledge base (KB). Afterwards, a query is fired to KB to search and extract the relevant text or fact. This text or fact is used to generate biography dynamically. A technique for finding semantic associations between web entities by considering the complex hierarchical structure of web entities has been proposed in [11]. In this work, additional information such as a logical collection of web entities into groups has been considered for extracting semantic association between web entities. Experimentation has been performed by considering MUADDIB [12,13] recommender systems.

Work described in [14] addresses the technique of extracting and grouping the words or entities from the text into pre-defined domain-specific concepts. This technique utilizes parts of speech (POS) tagging. For the extraction purpose, Engineering Ontology from the mechanical domain has been taken into consideration. For automatic extraction of entities, data from various online sources which depict mechanical components have been taken into account. In this work, four domain, i.e. concepts, device, material, and a process has been automatically extracted.

Automated feature induction method has been proposed [15]. In this method, Named entity recognition is done using undirected graphical models called as Conditional Random Fields (CRFs). Initially, the user creates atomic features. In this work, as compared to traditional approaches accuracy is improved. This method also reduces feature count. CONLL-2003 named entity shared task dataset has been used for experimentation.

Focusing on context patterns for extracting named entity from unlabeled data has been suggested [16]. This work utilizes language independent technique. The initial seed list is created and it is gradually increased by searching more instances for matching context patterns from unlabeled data.

Work addressed in [17], Support Vector Machine (SVM) based parser has been presented. N-best words with the features based on their character position and type has been calculated using statistical techniques for an input sequence. Finally, the characters are aggregated into words using SVM and named entity is recognized. In this work, CRL NE data have been used for evaluation. A System developed for extracting named entities from unstructured biomedical documents using integrated SVM and Decision Tree classifier has been proposed in [18].

In the past few years, extracting attributes and their values from web pages had gained the attention to researchers. Extracting implicit and explicit attributes from the textual product description has been proposed [19]. Implicit attributes are not directly mentioned in the product description. They are semantic attributes. While, explicit attributes are physical attributes. Using Naïve Bayes as a supervised learning and Co-training Expectation Maximization (Co-EM) as a Multiview semi-supervised learning algorithm, explicit attributes are extracted. In this work, sporting goods category website has been used for extracting attribute values.

Work described in [20], extracted textual attribute value pairs under three diversities such as Tweets, Web-Links and Tweets+ Web-Links for 15 events. By using set of delimiters, sentences are split and attribute-value pairs are extracted. Some special cases are handled using common noun or proper noun. Remaining sentences which doesnt fall under special cases are handled using dependencies (subject and object concept).

Work on extracting attribute-value pairs using Knowledge Base (KB) for a particular category have been demonstrated [21]. In this work, unstructured web pages are used for creation of KB. Table and other structures available in unstructured pages are used for automatic creation of KB. Using this KB, web pages are annotated automatically. These annotated pages are utilized for building training model for extraction. Some attribute value pairs are missing in this model because, annotated corpus is not created manually. Therefore, from unstructured test data, the model fails to extract some attribute-value pairs.

Recently lot of work has been done for extracting attributes and their values from online reviews of the product. In the work done by [22], features are extracted from the pros and cons sentences of the customer views of the product. For extracting product features and opinions from reviews, [23] calculate point of mutual information between the noun-phrases and related context patterns. A Technique based on association rule mining and natural language processing for mining frequent features has been proposed [24].

3. Proposed work

Following subsections describe proposed work. These subsections present the meaning of definitions and terms used in the Algorithm 1, important steps of the algorithm and explanation of each step of the algorithm. The attribute-value extraction which is given in Algorithm 1 is implemented as a two step process. In the first step entity name is extracted [25] and after entity name extraction, attribute-value pairs are extracted in a second step.

Input:Search Query specifying class name CLSNAME, Price as a single attribute and maximum number of URLs crawled (n) to limit the search space.

Output: Spreadsheet (WSP) indicating product names of the class and attribute values represented as rows of a spreadsheet.

Prepare Synonym list(SYL) for product class name CLSNAME.

Extract URLs from the search engine by specifying classname CLSNAME and Price as an attribute and store them in URLSTORE.

Rank the documents pointed by URL based on presence of classname and some of the attribute names belonging to class CLSNAME. Sort URLSTORE based on rank.

for every document Wi present in URLSTORE do

  Remove Stop words, unwanted symbols and perform stemming using Word Net Stemmer.

  For each word in a sentence construct Feature Vector specified in sections ”Extraction of entity name” and ” Attribute-Value extraction”

  Classify word using Decision Tree Classifier for determining entity name (Ei)

  Classify word using Decision tree Classifier to determine Attribute-Value pair (A1 - V2, A2 - V2 ,..., An - Vn).

  Prepare Columns of spreadsheet using (A1, A2, …, An) attributes which is obtained in step 10.

  Prepare row of spreadsheet using results from step 9 and values (V1, V2, …, Vn) obtained in step 10.

end for

Output: spreadsheet(WSP)

3.4. Preparation of synonym list

We also maintain Synonym List (SYL) for product class name for which the spreadsheet has to be generated. All the synonym words for product class name are preserved in this list. Synonym list for class CLSNAME is created using Word Net. Synonym list plays a crucial role in extracting instance-class relation. The main aim of creating a synonym list is to target only those sentences of a web page, which contains at least one of the synonym words from list SYL.

Every Retailer describes their products in a different fashion. They also use different words corresponding to class name CLSNAME to describe the complete name of the product. For e.g., Figure 1 shows the initial part of product description page for Lenovo Tablet. In this, complete name (Lenovo Tab 2 A8) in the heading part comprises word “Tab” while in Figure 2, the complete name (UBI Slate 7CZ Tablet) consists of word “Tablet”.

Fig. 1.

Initial Part of Product Description Page for Lenovo Tablet.

Fig. 2.

Initial Part of Product Description Page for UBI Slate Tablet.

While describing the product, retailers use different substring words. These substring words which are basically a part of a complete entity name. For e.g., Figure 3 shows the part of product description page for the Samsung Galaxy Tablet. The Complete Name of the product is “Samsung Galaxy Tab 4”. The product is described using n number of sentences. Consider the two sample sentences described as follows:

• Sentence 1: ”Packed with a host of features, Samsung Galaxy Tab 4 T231 Tablet is a user friendly budget device.”

• Sentence 2: ”Android operating system and 1.5 GB RAM boosts the performance of this Galaxy Tablet.” In sentence 1, complete entity name, i.e. “Samsung Galaxy Tab 4 T231” is used while in sentence 2, substring word “Galaxy” is used. In sentence 1, “Tablet” and “Tab” are used as a synonym words while in sentence 2 “Tablet” is used as a synonym word.

Fig. 3.

Product Description Page for Samsung Galaxy Tablet.

4. Experimental setup

Experiments were performed on Windows Server 2012(R2) and softwares used are as follows:

• •

  NetBeans IDE 8.0.2

• •

  Java JDK 1.7.0.80

• •

  Jsoup1.8.2 API is used, which is used to connect to the web document url and extracts all the sentences.

• •

  Stanford POS (Part Of Speech) tagger v. 2.0 for tagging the terms of web pages.

• •

  Word Net Stemmer for obtaining word stems.

5. Results and Discussions

Both the Classifiers for entity and attribute-value extraction are trained using class names as Tablet, Washing Machine and Tennis Racket. Effectiveness of algorithm which is described in Section 3 is evaluated by calculating classification accuracy (Cross validation) using N-Fold cross validation and 80-20 split i.e. 80 percent data are used for training and 20 percent data are used for testing.

6. Conclusions and Future work

A huge amount of product related data is spread over various web pages. Every manufacturer or retailer displays attribute (feature) information related to a particular class of product, but this information is insufficient. The user has to crawl different websites for choosing the right product because different values for same feature is available on various websites. If any manufacturer introduces new feature for a particular product, it is updated manually on their website. In this work, for a particular class of products, exceptional product differentiation based on features has been carried out by introducing novel concept of web-spreadsheet which is a solution to the above discussed problem.

The web-spreadsheet is generated dynamically and it stores consolidated information of product names and their respective attribute-value pairs, for a particular class of product. We have crawled nearly every websites of retailers for generating spreadsheet. The user will create search query which consists of a particular class of product for which the spreadsheet has to be generated. The search query is fired to search engine which returns related URLs of nearly every product manufacturers or retailers. Web documents corresponding to these URLs are crawled one by one and product names and attribute-value pairs are automatically extracted from unstructured natural language documents.

Using separate J48 decision tree based classifier, entities (Product names) and attribute-value pairs are extracted successfully. Decision tree classifier is trained by using product classes of different categories. As training data are very expensive to generate, we experimentally showed that, the trained classifier can be used to test (classify) product classes without requiring training data of each class. In this work we have dealt with Alphanumeric and Numeric-Alpha type value-measure pair word. We have extensively performed experiments on various product classes and we obtained encouraging results in each product category. The proposed method accuracy is exceptionally good as compared with many entity and attribute-value extraction techniques as shown in Table 24 and Table 25. Immediate Future work is the classification accuracy can be improved by utilizing more detailed information from ontologies and using word phrase information at statement and document level. Similar approach as we have discussed in this paper can also be adopted for different kinds of problems such as comparing services. Various classifiers can be designed and tested for the same.

Acknowledgment

Authors would like to thank Computer Science and Engineering Department, Visvesvaraya National Institute of Technology, Nagpur for providing necessary facilities for carrying out this research work.