Optimal Diversity of Recommendation List for Recommender Systems based on the Users’Desire Diversity

1. INTRODUCTION

Nowadays, the necessity of recommender systems to help users in finding their intended items among a mass of information is completely evident. Recommender systems recommend an appropriate list of items according to the interests and past behaviors of the users. An online movie market website which introduces and sells movies is an example of an e-commerce website in which a recommender system can be utilized. There is a huge number of movie items on that market about which most users have no information. It is obvious that if users could be aware of the existence of their intended items, the profitability of the market would increase.

The main focus of primitive recommender systems was just the accuracy of the recommendation list (Bennett & Lanning, 2007; Ricci, Rokach, & Shapira, 2011). However, focusing only on accuracy may prevent recommender systems from taking the risk of recommending unknown items to users. McNee, Riedl, and Konstan (2006) showed that being accurate is not enough, and illustrated how accuracy metrics have hurt recommender systems. Also, considering only the accuracy criterion would not satisfy users without having sufficient novelty and diversity (Di Noia, Ostuni, Rosati, Tomeo, & Di Sciascio, 2014; McNee et al., 2006). Therefore, other criteria such as diversity and novelty had been suggested to be considered in the recommendation lists to increase the efficiency of recommender systems (Castells, Hurley, & Vargas, 2015). But finding the optimal amount of novelty and diversity of the recommendation list is a great challenge. An online experiment with over 1,800 users for six months on a live recommendation system was conducted by Nguyen, Harper, Terveen, and Konstan (2018). The authors concluded that users with different personalities have different preferences for diversity, popularity, and serendipity properties. Consequently, the paper suggested that user satisfaction can be improved when users’ personality traits are integrated into the process of generating recommendations.

Certainly, some people show less variety in their behaviors; on the other hand, others enjoy more variety. Hence, the necessity of a recommender system which considers users’ desire diversity in their recommendation list is completely evident. To achieve this goal, our approach finds the similarity of users based on their desire diversity and combines it as a new parameter to the usual similarity calculation of the state-of-the-art collaborative filtering algorithm. By taking into account the similarity of users’ past behaviors in how diverse they have rated or followed items, our recommender system method will recommend lists such that they are more diverse for those users who seek lots of diversity, and vice versa. Hence the main contribution of this study is to tune the degree of diversity of the recommendation list based on the users’ variety-seeking feature by defining a new users’ similarity calculation function.

It is worth noting that studies have demonstrated that there is a tradeoff between diversity and accuracy (Zhou et al., 2010), so that by some increase in one, the other one decreases. Therefore, while adding the diversity criterion to a recommender system it is also emergent to keep accuracy at an acceptable level too. The proposed recommender algorithm adjusts the amount of diversity for the intended user accompanied by keeping the accuracy criterion unaffected. The experiments’ results show that the proposed approach not only increases the users’ desire diversity but also increases intra-list diversity without negative impact on the accuracy criterion, which leads to greater satisfaction for users.

The remainder of this paper is organized as follows: In the next section, related work is discussed. Section 3 introduces the collaborative filtering algorithm in detail. In Section 4, first the personal diversity criterion and then the proposed approach will be explained. Section 5 is dedicated to experimental results. Finally, Section 6 concludes the paper with some future directions for researchers.

2. LITERATURE REVIEW

Some studies have paid attention to the idea of considering user interests in the diversification approach in order to personalize recommendation diversity. Di Noia et al. (2014) proposed the first attribute-based diversification approach to adjust the degree of diversity of the recommendation list by taking into account the inclination to diversity of the user across different item attributes. Martinez-Cruz, Porcel, Bernabe-Moreno, & Herrera-Viedma (2015) proposed a user modeling technique to characterize deeply the users-items interactions and to move beyond the network of users just based on the rating history, where a trustworthy network made of users which a user can rely on has been built.

Vargas and Castells (2013) identified the diversity within user profiles and generate partial recommendations based on homogeneous subsets of user preferences (sub-profiles), which they combined later to produce a final recommendation. Their experimental results showed that the approach indeed improves the quality of state-of-the-art recommenders, and is competitive against diversification methods that explicitly use item categories as the units for diversification. Wu, Chen, and He (2013) proposed a strategy that explicitly embeds personality to adjust the diversity degree within multiple recommendations. They performed a user evaluation on the developed system. Their experimental results demonstrated an effective solution to generate personality-based diversity in recommender systems.

A pre-filtering approach to personalizing diversity was proposed by Eskandanian, Mobasher, and Burke (2017). They assumed that each user’s diversity preference can differ from others’. Their method first automatically segmented users according to their diversity preference level, which was determined as the distribution across categories (e.g., movie genres) of the user’s rated items. Then, a standard collaborative recommendation algorithm on each segment was run separately. They proved that their approach can achieve satisfactory performance in terms of both accuracy and diversity on two datasets, MovieLens and Yelp. However, by clustering users in different groups, their method may face the local optimum issue. In addition to the mentioned issue they ignore the similarity of users between different clusters, which affects the performance of their recommender system negatively.

Comparing to literature, our proposed method considers the effect of all users’ similarity regarding how diverse they have behaved in their ratings in the past. This novel similarity definition would be combined to user-based collaborative filtering similarity aspect to achieve optimal diversity.

3. COLLABORATIVE FILTERING

Collaborative filtering is one of the most useful algorithms that is used in many recommender systems (Medina-Moreira et al., 2017; Nilashi, Ibrahim, & Bagherifard, 2018). The collaborative filtering algorithm first predicts the vote of the target user for the items which have not been visited by that user yet. Then the unvisited items are sorted based on their predicted votes, and L of the items from the head of the list—items with greatest vote values—will be suggested to the user. The collaborative filtering algorithm can also be divided into two categories: user-based collaborative filtering and item-based collaborative filtering algorithms (Bennett & Lanning, 2007; Ricci et al., 2011).

User-based approaches include two phases of learning and prediction. In the learning phase, the similarity of the users is calculated and a similarity matrix is constructed. Various methods have been developed to extract similarities between users (Ahn, 2008; Tkalcic, Kunaver, Tasic, & Kosir, 2009). Among the valid methods for obtaining similarity of users, Pearson similarity and the Cosine similarity of the vector of the users’ votes are the most common ones (Felfernig, Jeran, Ninaus, Reinfrank, & Reiterer, 2013). Pearson similarity, which is used in this research, utilizes Eq. 1 to obtain the similarity of user i and j:

$S_{p e a r s o n} (i, j) = \frac{\sum_{k = 1}^{n} (r_{i k} - \bar{r_{i}}) (r_{i k} - \bar{r_{i}})}{\sqrt{\sum_{k = 1}^{n} {(r_{i k} - \bar{r_{i}})}^{2}} \sqrt{\sum_{k = 1}^{n} {(r_{i k} - \bar{r_{i}})}^{2}}} (1)$

where $\bar{r_{i}}$ is the average votes of the user i and r_ik is the vote of user i for item k. n denotes the number of common items in the votes of user i and user j.

Then in the prediction phase, using similarities between the users which are obtained from the previous phase, the predicted value of the vote of user i for item o is calculated based on the sum of the weighted votes of the other users to item o. The weight of the votes is relevant to the similarity of the voter and the target user; Eq. 2:

$P (u, o) = \bar{R_{u}} + \frac{\sum_{h = 1}^{m} (R_{h, o} - \bar{R_{h}}) s i m (u, h)}{\sum_{h = 1}^{m} s i m (u, h)} (2)$

in which sim(u, h) demonstrates the similarity of the vote of user u and user h. m is the number of the voters of item o.

4. PROPOSED METHOD

This section explains the suggested personal diversity criterion and then the proposed recommender system method.

4.1. Personal Diversity Criterion

Users’ desire diversity in recommendation lists is an important criterion to which insufficient attention has hitherto been paid. Some people are variety-seeking and embrace diversity in the suggested list, whereas others might not do so.

To calculate the degree of personal diversity of each user, the content diversity of the items which are voted by that user can be used. It is obvious that if the contents of the voted items by the user are identical or too similar, then it can be concluded that the user has a low interest in diversity and vice versa. Hence, this idea is considered in presenting the suggested list to the user. Therefore, the amount of interest in diversity for each user, called users’ desire diversity, can be calculated by utilizing the list of those items which are voted on by that user and their contents.

First, the algorithm needs to calculate the similarity amount of two items based on their contents. Eq. 3 is proposed to do this:

$s i m_{i j} = \frac{\sum_{k = 1}^{m} 1 (i_{k} = = j_{k})}{m} (3)$

in which sim_ij indicates the similarity amount of item i and j, m states the number of attributes of contents, and i_k shows the k’th attribute of item i, which might be zero or one. More details in addition to an example will be expressed in Section 5.1. 1(x) equals 1 if x is true and equals 0 if x is false.

Then, the amount of diversity of the user’s past votes should be calculated. After that the intra-list diversity of the suggested lists is calculated. The difference between these two parameters indicates the users’ desire diversity criterion. While this difference amount tends toward zero, the system is more efficient regarding the users’ desire diversity criterion because the diversity amount in suggestions to the user is relevant to the variety amount that the user has shown in past behaviors. The following formula, Eq. 4, indicates users’ desire diversity criterion mathematically.

$P e r s o n a l D i v e r s i t y = \frac{\sum_{u = 1}^{U} | P a s t I n t r a D i v_{u} - L i s t I n t r a D i v_{u} |}{U} (4)$

where U is the number of users and PastIntraDiv indicates the past personal diversity of the users which is calculated through the following formula:

$P a s t I n t r a D i v_{u} = 1 - \frac{\sum_{i = 1}^{m} \sum_{j = 1}^{m} s i m_{i j}}{n^{2}} (5)$

wherein the above formula, n is the number of the items which are voted by user u in the past. Intra-list diversity amount is also calculated by Eq. 6:

$L i s t I n t r a D i v_{u} = 1 - \frac{\sum_{i = 1}^{L} \sum_{j = 1}^{L} s i m_{i j}}{L^{2}} (6)$

where L indicates the number of items in the suggested list.

4.2. Personal Desire Diversity Collaborative Filtering

The proposed method tries to decrease the difference amount between diversity in the suggested list and the users’ desire diversity. The method includes two phases which will be explained as follow.

4.2.1. Phase 1: Calculation of Users’ Similarity Based on Diversity

In the first phase, the similarity of the users should be calculated based on the diversity of their past votes. To calculate the user variety-seeking of each user, the amount of diversity of the items which have been voted on by that user should be determined. First, the diversity of rated items is calculated based on Eq. 6, then the similarity amount of the users is calculated by Eq. 7:

$S i m_{p, q} = 1 - | P a s t I n t r a D i v_{q} - L i s t I n t r a D i v_{p} | (7)$

After finding the degree of similarity between the users, these values are combined in the user-based collaborative filtering algorithm. As previously mentioned, this algorithm compares the similarity of the users’ votes to the same items to find out the similarity of the users. The proposed hybrid collaborative filtering algorithm, Eq. 8, constructs two matrixes of size n×n where n is the number of users. One of them contains the similarity of the users based on the diversity of their votes, and the next one is based on the conventional similarity of the votes (regular user-based collaborative filtering) of the users for the same items.

$S i m_{t o t a l} = α * S i m_{I n t r a D i v} + (1 - α) * S i m_{R a t e s} (8)$

where α is a coefficient between 0 and 1 which adjusts the impact of each parameter.

4.2.2. Phase 2: Prediction of the User’s Vote

The last phase predicts the votes of the user for those items which have not been voted on by that user so far. This phase uses the similarity matrix as well as the collaborative filtering algorithm. In this phase, users with the most similarity to the target user are initially chosen. Then the sum of the weighted votes of these selected users for those items which are not voted on by that user are calculated. These values are considered as the prediction of the votes of the target user. Finally, after sorting all the predicted items, L items with the most prediction value are selected as the recommendation list to the user. Below, the proposed algorithm is described step by step.

1) Collect data and clean it to reduce abnormality in the dataset.
2) Calculate the similarity of all movies in the dataset based on their genres as content features using Eq. 3.
3) Calculate desire diversity for all users, based on Eq. 5 using the similarity of rated items by each user, calculated in Step 2.
4) Calculate similarity of user u to all other users:
a) Calculate Pearson similarity of user u to all other users using Eq. 1 (usual collaborative filtering similarity measurement).
b) Calculate similarity of user u to all other users based on their desire diversity using Eq. 7.
c) Calculate the total similarity of user u to other users based on the weighted combination of similarities calculated in Steps 4a and 4b.
5) Calculate the prediction rate P(u, o) for user u and all items o in the dataset using Eq. 2 and the total similarity matrix calculated in step 4c as sim(u, h).

4.3. Structural Analysis of the Proposed Method

An important feature of the recommender systems is the computational complexity of the algorithms. For example, although some algorithms such as singular value decomposition (SVD) (Koren, Bell, & Volinsky, 2009) which are based on matrix factorization have a good performance in most of the evaluation criteria, they need huge calculations in each iteration. Therefore, such complex algorithms are not suitable for e-commerce applications.

The proposed method is an efficient algorithm with a low computational complexity. Evaluating users’ similarity in how diversely they voted just depends on the set of items each user has rated, which is much less than the size of all items, and also this similarity calculation can be done in parallel to users’ conventional similarity of collaborative filtering. As a result, the time complexity of calculating the similarity of the users’ respect to diversity of voted items for users is negligible in comparison with the usual algorithms, such as user-based or item-based collaborative filtering.

5. PERFORMANCE EVALUATION

This section presents the results of the evaluation of the compared algorithms in addition to the proposed method. Matlab programming language is used for all the implementations. Experiments include the results of evaluating item-based collaborative filtering (IICF) (Sarwar, Karypis, Konstan, & Riedl, 2001), user-based collaborative filtering using Pearson similarity (UBCF) (Zhao & Shang, 2010), diverse clustering (Div-Clust) (Eskandanian et al., 2017) as a novel personalized diversity recommender system, and the proposed method, personal collaborative filtering (PCF).

5.1. Dataset

We used the MovieLens dataset1 (Harper & Konstan, 2016) which contains real data corresponding to movie ratings captured on the website of the MovieLens movie recommender. This dataset has been widely used in collaborative filtering research in the last decade. As previously mentioned, the proposed method needs the contents of the items to calculate personal diversity criterion: Eq. 4. Therefore, a dataset including the contents of the items was needed. In this research, the information of the movies in the MovieLens dataset is used. This dataset contains items, users, votes, and items’ contents. Total number of movies, in the dataset, is 4,000, total number of users is 6,000 and number of votes is 1,000,000.

In this dataset, each user has at least 20 ratings. Ratings are made on a 5-star scale. The format of rating file is as Table 1.

Table 1.

Format of rating file

This dataset also includes information on the content of the movies, including the genre of the movies, the director, the country of production, and the actors. Table 2 is an example of a table for content information related to the movie’s genre.

Table 2.

Real content information related to movie’s genre

The first number in each row defines the movie ID number. Hence, we preprocessed the information for each movie and changed the format of information to a set of attributes which is presented as columnar contents, with a value of 1 indicating that the movie has that attribute and 0 indicating the lack of that attribute, as is shown in Table 3. In total, 18 different genres exist for all movies.

Table 3.

An example of the contents of the genres

The abnormal data in the dataset are cleaned as a preprocess step, e.g., some MovieIDs did not correspond to a movie due to accidental duplicate entries and/or test entries.

5.2. Methodology

The offline strategy is considered to evaluate the algorithms. In this strategy, experiments are carried out based on a series of data (a set of the users’ recorded votes for a list of items which also contains information about their contents) collected from past system performance. In this method, the initial data set is first divided into training and testing sets using 5-fold cross validation. In this way, a real test is simulated. To evaluate the results, 80 percent of the datasets were used as training data and the remainder was used as test data. Also, L=20 of the items with the highest prediction value are selected as the recommendation list. Then, the performance of each of the recommender algorithms is evaluated based on the votes recorded in the test dataset.

5.3. Optimal Value of α

In order to find the optimal value for parameter α in Eq. 9, a 4-fold cross-validation method was used, which ultimately considered the value of 0.2 for this parameter. Fig. 1 shows mean absolute error (MAE)—which will be described in Section 5.5—for different values of parameter α.

Fig. 1.

Evaluation results of mean absolute error (MAE) for different values of α.

5.4. Time Complexity

To validate our claim about the low complexity of the proposed method, an execution time experiment was carried out. Fig. 2 shows the execution times of the algorithms. Execution time includes both training and prediction time. Also, time was measured on a PC with a Core i7 Intel processor at 3.2 GHz with 4 GB RAM. SVD algorithm (Koren et al., 2009) is also considered in this experiment.

Fig. 2.

Execution time of algorithms. IICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; PCF, personal collaborative filtering; Div-Clust, diverse clustering; SVD, singular value decomposition.

Fig. 2 indicates that PCF and UBCF have the best execution times and IICF has the second rank, while Div-Clust and SVD have the worst execution times. The reasons for the time efficiency of the proposed method were mentioned in Section 4.3.

5.5. Accuracy Results

One of the most important criterion for evaluating accuracy of the recommender systems is the mean absolute error rate. The significance of this criterion is such that if a recommender system performs well for all other criteria, but the mean of its absolute error is inappropriate, that recommender system is not suitable. MAE is calculated using Eq. 9:

$M A E = \frac{\sum_{i = 1}^{N} | r_{u i} - {\tilde{r}}_{u i} |}{N} (9)$

where r_ui is the actual value of the vote of the user for the item i and ${\tilde{r}}_{u i}$ is the predicted value of the recommender system. The lower the MAE, the closer the predictions to the actual values, and the performance of the system will be better.

Another important criterion for evaluating accuracy of recommender systems is the root mean square error (RMSE). RMSE is similar to MAE but it gives more penalty to large prediction errors. This criterion can be obtained using Eq. 10:

$R M S E = \frac{\sum_{i = 1}^{N} \sqrt{{(r_{u i} - {\tilde{r}}_{u i})}^{2}}}{N} (10)$

It is clear that if the difference of r_ui and ${\tilde{r}}_{u i}$ is great (more than 1), due to the existence of power 2, the effect of this difference value is much more than when the difference value is low (less than 1). Here too, the low value of this criterion indicates that the recommender system is more appropriate.

Figs. 3 and 4 show the results of the experiments for accuracy, MAE and RMSE respectively. The best accuracy rate is achieved by the IICF method and our proposed method has been ranked second. Div-Clust and UBCF methods are in the next ranks, respectively. These figures indicate that our method outperforms the UBCF method. It shows that the new users’ similarity aspect, which is based on diversity of their past ratings, has positive impact on the performance of the recommender method. In other words, since our method finds similar users in different aspects it recognizes users’ interests better, so it can suggest more appropriate items to them.

Fig. 3.

Evaluation results of mean absolute error (MAE). ICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; Div-Clust, diverse clustering; PCF, personal collaborative filtering.

Fig. 4.

Evaluation results of root mean square error (RMSE). IICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; Div-Clust, diverse clustering; PCF, personal collaborative filtering.

5.6. Intra-List Diversity Results

Another criterion which is closely related to the proposed criterion is intra-list diversity. In order to calculate this criterion, a definition of similarity between items is needed. For example, one can count the number of the same tags associated with the items in the list. Clearly, the higher the similarity value of the items, the less the intra-list diversity. The amount of intra-list similarity can be obtained from Eq. 12 (Felfernig et al., 2013):

$I L S = \frac{Σ_{k \in L_{i}} Σ_{j \in L_{i}, j \neq k} c (k, j)}{2} (11)$

c(k, j) is the similarity amount of two items of the list based on a specific predefined feature which has a value between 0 and 1. Eq. 13 can be used to calculate the intra-list diversity amount:

$D = 1 - (^{I L S} / N) (12)$

In some studies, the main goal has been to increase this criterion. Fig. 5 shows the results of the experiments regarding this criterion for different simulated algorithms. Although the main goal of the proposed method was to improve the level of users’ desire diversity, it is obvious that the proposed method has also significantly improved the intra-list diversity of the list. This improvement is also due to preparing a more diverse list for those users who are interested in a high degree of diversity, while suggesting a list of items with less diversity to those who are not fans of diverse items. It is clear that for this criterion the effectiveness of the compared methods is less than for the proposed method.

Fig. 5.

Evaluation results of intra-list diversity. IICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; Div-Clust, diverse clustering; PCF, personal collaborative filtering.

5.7. Precision and Recall

Precision and recall are two important and commonly used recommendation quality criteria which indicate the portion of interesting items for the user from the total number of the recommended list, and the portion of interesting items for the user in the recommended list from the total number of interesting items, respectively. Interesting items are defined based on the formula True Positive, which is as follows:

$T P = {i \in L_{u} | r_{u i} > θ} (13)$

where L_u is the suggested list to user u and θ is the threshold which defines interesting items for users. For example, here we set θ equal to 3, which means all rated items by users with a rate greater than 3 count as an interesting item for that user.

It is desirable to have a recommendation list with high precision and recall. However, precision and recall are usually reversely related. In other words, an increment in one usually causes the other one to decrease. Precision and recall are defined as follows:

$p r e c i s i o n = \frac{1}{U} \sum_{u = 1}^{U} \frac{# {i \in L_{u} | r_{u i} > θ}}{# L} (14)$

$r e c a l l = \frac{1}{U} \sum_{u = 1}^{U} \frac{# {i \in L_{u} | r_{u i} > θ}}{# {i \in L_{u} | r_{u i} > θ} + # {i \in L_{u}^{c} | r_{u i} > θ}} (15)$

where $L_{u}^{c}$ is a complementary set of items in L_u. Table 4 shows the result of comparison of precision and recall of different methods. One can conclude that the PCF method has the best performance on precision and recall, since PCF has the highest precision and recall overall.

Table 4.

Results of comparison of precision and recall

IICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; Div-Clust, diverse clustering; PCF, personal collaborative filtering.

5.8. Users’ Desire Diversity Error Results

The most important goal of this research is to improve the users’ desire diversity. Fig. 6 shows the results of experiments and comparison of different methods. As outlined in Fig. 6, the proposed method has improved the users’ desire diversity criterion and outperforms the others.

Fig. 6.

Evaluation results of personalized diversity. IICF, item-based collaborative filtering; UBCF, user-based collaborative filtering using Pearson similarity; Div-Clust, diverse clustering; PCF, personal collaborative filtering.

Considering all the evaluated criteria together, the proposed method has a proper performance because the proposed method, in addition to improving the personal desire diversity criterion, also has an acceptable performance for accuracy and intra-list diversity criteria compared to other evaluated methods.

6. CONCLUSION AND FUTURE WORK

In this paper, a new perspective of similarity between users which considers diversity of contents of rated items by users is proposed. Combining this similarity criterion with the conventional similarity used in user-based collaborative filtering method, a new method was proposed for recommender systems. The proposed method tried to recommend a list to users in which the diversity of items is almost the same as users’ desire diversity. To achieve this goal, a new criterion called users’ desire diversity was introduced, which evaluates how similar the diversity of the recommended list is comparing to the diversity of rated items by users based on the items’ content.

Considering the performance evaluations of the compared methods, it can be concluded that the proposed method increases the users’ desire diversity as well as intra-list diversity. In addition, since our method finds similar users in different aspects it recognizes users’ interests better, so it can suggest more appropriate items to them. The results of the experiments indicate the effectiveness of the proposed method. Also, a bold property of the proposed method is its low complexity, which can introduce the method as a suitable recommender system algorithm for e-commerce applications.

In future work, the authors are interested to study the degree of users’ interest in how much novelty they prefer. Defining a new similarity perspective based on the novelty of items each user is interested in, and combining this similarity aspect with user based collaborative filtering and our proposed similarity aspect, can be considered. It is expected that the performance of the recommender system will improve by adding new similarity aspects to it.

https://grouplens.org/datasets/movielens/

Ack

We thank the Islamic Azad University, Dariun Branch, for funding, along with Shyong Lam and Jon Herlocker for cleaning up and generating the data set.

References

Ahn, H. J. ((2008)) A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem. Information Sciences, 178(1), 37-51 .

, Bennett, J., & Lanning, S. (2007). , The Netflix prize, . Paper presented at the KDD Cup and Workshop, San Jose, CA, USA., , The Netflix prize., Paper presented at the KDD Cup and Workshop, (2007), San Jose, CA, USA

Novelty and diversity in recommender systems. In F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender systems handbook (Castells, P., Hurley, N. J., & Vargas, S.) ((2015)) New York: Springer Castells, P., Hurley, N. J., & Vargas, S. (2015). Novelty and diversity in recommender systems. In F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender systems handbook (pp. 881-918). New York: Springer. , pp. 881-918

, Di Noia, T., Ostuni, V. C., Rosati, J., Tomeo, P., & Di Sciascio, E. (2014). , An analysis of users’ propensity toward diversity in recommendations, . Paper presented at the 8th ACM Conference on Recommender Systems, Foster City, CA, USA., , An analysis of users’ propensity toward diversity in recommendations., Paper presented at the 8th ACM Conference on Recommender Systems, (2014), Foster City, CA, USA

, Eskandanian, F., Mobasher, B., & Burke, R. (2017). , A clustering approach for personalizing diversity in collaborative recommender systems, . Paper presented at the 25th Conference on User Modeling, Adaptation and Personalization, Bratislava, Slovakia., , A clustering approach for personalizing diversity in collaborative recommender systems., Paper presented at the 25th Conference on User Modeling, Adaptation and Personalization, (2017), Bratislava, Slovakia

Toward the next generation of recommender systems: Applications and research challenges. In G. A. Tsihrintzis, M. V. Lakhmi, & C. Jain (Eds.), Multimedia services in intelligent environments (Felfernig, A., Jeran, M., Ninaus, G., Reinfrank, F., & Reiterer, S.) ((2013)) New York: Springer Felfernig, A., Jeran, M., Ninaus, G., Reinfrank, F., & Reiterer, S. (2013). Toward the next generation of recommender systems: Applications and research challenges. In G. A. Tsihrintzis, M. V. Lakhmi, & C. Jain (Eds.), Multimedia services in intelligent environments (pp. 81-98). New York: Springer. , pp. 81-98

Harper, F. M., & Konstan, J. A. ((2016)) The MovieLens datasets: History and context. ACM Transactions on Interactive Intelligent Systems, 5(4), 19 .

Koren, Y., Bell, R., & Volinsky, C. ((2009)) Matrix factorization techniques for recommender systems. Computer, 42, 30-37 .

Martinez-Cruz, C., Porcel, C., Bernabe-Moreno, J., & Herrera-Viedma, E. ((2015)) A model to represent users trust in recommender systems using ontologies and fuzzy linguistic modeling. Information Sciences, 311, 102-118 .

, McNee, S. M., Riedl, J., & Konstan, J. A. (2006). , Being accurate is not enough: How accuracy metrics have hurt recommender systems, . Paper presented at the CHI’06 Extended Abstracts on Human Factors in Computing Systems, Montreal, Canada., , Being accurate is not enough: How accuracy metrics have hurt recommender systems., Paper presented at the CHI’06 Extended Abstracts on Human Factors in Computing Systems, (2006), Montreal, Canada

, Medina-Moreira, J., Apolinario, O., Luna-Aveiga, H., Lagos-Ortiz, K., Paredes-Valverde, M. A., & Valencia-Garcia, R. (2017). , A collaborative filtering based recommender system for disease self-management, . Paper presented at the International Conference on Technologies and Innovation, Guayaquil, Ecuador., , A collaborative filtering based recommender system for disease self-management., Paper presented at the International Conference on Technologies and Innovation, (2017), Guayaquil, Ecuador

Nguyen, T. T., Harper, F. M., Terveen, L., & Konstan, J. A. ((2018)) User personality and user satisfaction with recommender systems. Information Systems Frontiers, 20(6), 1173-1189 .

Nilashi, M., Ibrahim, O., & Bagherifard, K. ((2018)) A recommender system based on collaborative filtering using ontology and dimensionality reduction techniques. Expert Systems with Applications, 92, 507-520 .

Introduction. In F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender systems handbook (Ricci, F., Rokach, L., & Shapira, B.) ((2011)) New York: Springer Ricci, F., Rokach, L., & Shapira, B. (2011). Introduction. In F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender systems handbook (pp. 1-35). New York: Springer. , pp. 1-35

, Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001). , Item-based collaborative filtering recommendation algorithms, . Paper presented at the 10th International Conference on World Wide Web, Hong Kong., , Item-based collaborative filtering recommendation algorithms., Paper presented at the 10th International Conference on World Wide Web, (2001), Hong Kong

, Tkalcic, M., Kunaver, M., Tasic, J., & Kosir, A. (2009). , Personality based user similarity measure for a collaborative recommender system, . Paper presented at the 5th Workshop on Emotion in Human-Computer Interaction-Real World Challenges, Cambridge, UK., , Personality based user similarity measure for a collaborative recommender system., Paper presented at the 5th Workshop on Emotion in Human-Computer Interaction-Real World Challenges, (2009), Cambridge, UK

, Vargas, S., & Castells, P. (2013). , Exploiting the diversity of user preferences for recommendation, . Paper presented at the 10th Conference on Open Research Areas in Information Retrieval, Lisbon, Portugal., , Exploiting the diversity of user preferences for recommendation., Paper presented at the 10th Conference on Open Research Areas in Information Retrieval, (2013), Lisbon, Portugal

, Wu, W., Chen, L., & He, L. (2013). , Using personality to adjust diversity in recommender systems, . Paper presented at the 24th ACM Conference on Hypertext and Social Media, Paris, France., , Using personality to adjust diversity in recommender systems, Paper presented at the 24th ACM Conference on Hypertext and Social Media, (2013), Paris, France

, Zhao, Z.-D., & Shang, M.-S. (2010). , User-based collaborativefiltering recommendation algorithms on hadoop, . Paper presented at 2010 Third International Conference on Knowledge Discovery and Data Mining, Phuket, Thailand., , User-based collaborativefiltering recommendation algorithms on hadoop., Paper presented at 2010 Third International Conference on Knowledge Discovery and Data Mining, (2010), Phuket, Thailand

Zhou, T., Kuscsik, Z., Liu, J.-G., Medo, M., Wakeling, J. R., & Zhang, Y.-C. ((2010)) Solving the apparent diversityaccuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences, 107(10), 4511-4515 .