1.1. Research Assessment Metrics

There are several research assessment metrics which are often used in bibliometric analysis. Citation count, i.e., the number of citations to a publication, is a document-level measure used to approximate the impact or importance of a paper, whereas publication count, i.e., the number of publications, is an author-level measure that represents an author’s research productivity. Another authorlevel measure that takes into consideration both the impact and productivity of a researcher is h-index (Hirsh, 2005). h-index is computed by sorting the publications of a given author by the descending order of citation count and finding the rank of publication at which the citation count is equal to or greater than the rank. In this way, an author with h-index of k is guaranteed to have k papers with at least k or more citations to each paper. The strength of h-index lies in the fact that it requires many high impact papers to achieve a high score. In other words, neither the authors with many papers that are cited infrequently nor the authors with a few papers that are cited highly will receive high h-index scores.

h-index, however, is not good at differentiating among authors with similar publication and citation patterns but different citation magnitudes. As Table 1 illustrates, authors with a higher number of citation counts at top ranks (e.g., author 1) can get the same h-index as other authors with fewer citations (e.g., author 2) as long as their citation counts near the h-index rank are similar. g-index, proposed by Egghe (2006), compensates for this weakness of

Table 1.

Example: h-index vs. g-index

h-index by using the cutoff rank criteria as the rank at which the accumulative citation counts are equal to or greater than the rank squared, thus taking into consideration the total number of citations for highly cited papers.

Another way to assess the quality of publications other than citation count is to consider the venue of publication. A paper published in a high impact journal can be regarded to be of higher quality than a paper published in a low impact journal. The popular metric for assessing the impact of a journal is given by the impact factor, which is computed by dividing the number of citations in a given year to papers published in a journal during the prior two years by the number of publications in those years (Equation 1). Impact factor, being the average number of citations to a paper for a journal, suffers from the same weakness as the citation counts which estimate the impact of a publication in that they treat all citations to be of equal importance, which can be a gross oversight in reality.

Eigenfactor (Bergstrom, 2007) addresses this shortcoming of impact factor by estimating the importance of citing journals similar to Google’s PageRank approach. As can be seen in Figure 1, d1 and d2 both have two citations but one of the

Fig. 1

Link Propagation Example

papers that cites d1 is much more important (indicated by its size) due to its own citation counts. Eigenfactor captures this property with a recursive link propagation algorithm (Equation 2). Unfortunately, computation of recursive link propagation measures such as Eigenfactor requires a complete set of the citation network, which is impractical if not impossible for most people. The computation, not to mention data collection, needed to apply an eigenfactor-like algorithm at an author- or document- level is prohibitively complex, so it is doubtful whether such measures can be computed in a dynamic fashion even with an inside access to citation databases such as Web of Science and Scopus.

1.2. Challenges for Research Assessment

Like the assessment of beauty, research assessment is an inherently subjective task that strives for objective standards by means of commonality. The input, outcome, and methods for research assessment depend on who is evaluating whom for what purpose and in what context. At the same time, research assessment should be a consistent and methodological process that produces a valid and robust outcome. Consequently the first issue in research assessment is whether to take a qualitative approach, in which subjective criteria suitable for the purpose and context of assessment can be applied to evaluate research performance in a comprehensive manner, or to take the quantitative approach that employs standard evaluation metrics to generate comparable assessment outcomes.

The qualitative approach has the advantages of human judgment, flexibility, and customization, but is a resource-intensive process with a lack of standardized criteria and methodology that can lead to inconsistent or biased results. The quantitative approach is a fairly standardized process which can be applied to evaluate and compare a large amount of data in an efficient manner. However, the evaluative outcome, which is based on only those facets of research performance that are readily quantifiable, is neither as holistic nor personalized as that arrived at by the qualitative approach. Furthermore, the quantitative approach typically does not adequately take into consideration the differences among base units of evaluation (e.g., citations, publications), thus sacrificing the accuracy of assessment for the sake of simplification. On top of these challenges, properly assessing the contribution of each author for a collaborative work is a troublesome undertaking. Estimating author contributions by the order of authorship (i.e. author rank) in a multi-author paper is guess work at best and does not always correspond to the true contributions that authors put forth to the publication in question.

In addition to the challenges inherent in assessment approaches are the sources of evaluation data, especially for the quantitative approach. The publication and citation data that feed into research performance assessment are collected from citation databases such as Web of Science, but citation databases suffer from lack of comprehensive coverage and standard data inclusion criteria that can lead to inconsistent outcomes (Meho & Yang, 2007; Yang et al., 2012). Moreover, citation databases are not yet very user-friendly for research assessment tasks that require more than raw publication and citation counts.

Faced with these challenges for research assessment, meaningful and consistent analysis of bibliometric data is no trivial task. Until the coverage, quality, and usability of citation databases are significantly enhanced, we must keep in mind that bibliometric indicators are only as reliable as their data sources and methods employed to produce them. Although the approach to citation database enhancement is one of the core issues in our project (Yang & Meho, 2011), we focused on examining bibliometric measures for research assessment in the current study. Specifically, we investigated how robust different bibliometric indicators are in assessing research performance.

1.3. Study Design

In order to test the reliability and stability of bibliometric indicators (BI) for research assessment, we compared the rankings of faculty members by various BI scores, such as publication count, citation count, and h-index. By comparing the rankings, we hoped to gain insights into the aspects of research performance measured by BIs and to determine how robust the assessment may be.

3.1 Publication Count vs. Citation Count

We first compared the ranking of the authors by publication count (PC) with the ranking by citation count (CC). Table 2 shows the top 20 faculty members ranked by PC versus the top 20 by CC. It can be seen that in the left table where authors are ranked by PC, large PC and CC rank differences (rows in red) are due to high PC and low CC. In other words,

Table 2.

Top 20 Authors by Publication Count (PC) and Citation Count (CC)

at least 4 out of 20 most-published authors in the study sample have very few citations to their work despite the high numbers of papers that they produced. On the other hand, large rank differences in the right table, where authors are ranked by CC, are caused by a handful of highly cited papers. For example, the top ranked author (P075) had four papers cited 179, 30, 26, and 18 times respectively, and the second ranked author had a paper with 87 citations and another with 45 citations. Similarly, the fourth and fifth ranked authors (P008 and P033) had 29, 20, 19, and 19 citations and 27, 23, and 21 citations respectively, and so on. Another interesting fact is that these highly cited papers are all international publications, whereas most of the papers with few citations in the right table are published in Korean journals.

To ascertain whether PC and CC measure the same or different aspects of research performance, we computed the Spearman’s rank order correlation. For the entire rank of 159, Spearman’s rho showed positive association (p = 0.7045). PC and CC both being the measures of research performance, overall association between two indicators showing positive correlation seemed reasonable since pockets of differences are likely to be hidden when averaged over the entire ranking. To examine the rank differences at a finer grade, we computed the rank correlation for the rank intervals of 20 (Table 3). We can see clearly that PC and CC are not correlated at rank intervals, which indicates that PC and CC measure different aspects of research performance. Table 3 also shows that the strength of association gets weaker at lower rank intervals, which suggests the importance of citation counts for authors with low publication counts.

Table 3.

Spearman’s Rank Order Correlation at Rank Intervals: Publication Count vs. Citation Count

(df=18, α=0.05, CV=0.447)

3.2. Author Rank Effect

Collaborative research projects produce publications with multiple authors. Typically, the first author is the main contributor with co-authors listed in the order of contribution amount. An exception to this format occurs when there is a corresponding author, who may sometimes be listed as the last author but whose contribution can be comparable to the first or the second author. A corresponding author may be someone akin to the principal investigator of a research project, who architected and directed the research that produced the publication, while the first author did the leg work and wrote the bulk of the paper. In such an instance, the contribution of the corresponding author should be counted heavily regardless of the author order. Zhang (2009), for example, assigns an author weight of 1 to both the first author and corresponding author, while remaining co-authors are assigned weights that diminish with the author order.

However, there is no guarantee regarding the contribution of the corresponding author and the order of authorship linearly corresponding to the contribution amount. Co-author contributions will probably vary from case to case, so the most accurate assessment should come from authors themselves. Since such information is impractical to collect in a large scale as well as being subject to personal bias and subjective interpretations, one must turn to readily available evidence, which is the order in which authors are listed in a publication. Based on the assumption that an author’s contribution to a collaborative work should correspond to what we call the “authorRank” (i.e. author order), we compared the raw citation count (CC) with citation counts weighed by their estimated contribution to the paper that is being cited. The first weighting formula (CC2) is a modified version of Zhang’s (2009) co-author weights. Since our study data did not include information on who the corresponding authors are, we assigned the first author the weight of 1 and the co-authors diminishing weights that sum to 1. The formula for CC2 is shown in Equation 4.

We also computed the second co-author weighting formula for citation count (CC3) using 1 over authorRank (Equation 5) and the third formula using 1 over author count (Equation 6). CC3 uses diminishing author weights, the sum of which can exceed the first author weight of 1, while CC4 assigns all co-authors the same weights that can be a small fraction of the first author weight for publications with many authors. An example shown in Table 4 illustrates the differences in the authorRank formulas.

When we compared the rankings of authors by CC, CC2, CC3, and CC4, we observed only small differences

Table 4.

Example: AuthorRank Weights Applied to Citation Counts

Table 5.

Top 20 Authors by Citation Counts, Using AuthorRank Weights

in rankings across authorRank weight formulas (Table 5). We attribute this to the fact that only about half of the study data is co-authored (31% with 2 authors, 16% with 3 or more authors). For the 51% of the single author papers, authorRank weights can have no effect, thus the authorRank effect is muted when averaged out over the entire study data. One notable occurrence in the Spearman’s coefficient table (Table 6) is the low numbers in the CC-CC4 column, which suggests that the raw citation count, which passes the entire impact indicator of a given paper to all authors equally, is quite different from CC4, which passes only a fraction of the impact indicator amount to the co-authors.

Table 6.

Spearman’s Rank Order Correlation at Rank Intervals, Using AuthorRank Weights

(df=18, α=0.05, CV=0.447)

Table 7.

Top 20 Authors by Citation Counts, Using AuthorRank Weights (au_cnt>1)

To isolate the effect of authorRank, we excluded single-author publications and redid the rank comparisons with authorRank. There were 1136 out of 2402 papers that were co-authored, 65% of which were written by 2 authors and 35% with 3 or more authors. As can be seen in Table 7, the rank differences are more pronounced without the single author papers, but Spearman’s coefficients still showed little differences and thus an insignificant effect of authorRank (Table 8). The strength of association, especially in the top rank interval, is weaker than when single-author papers were included, which may be due to a few authors who received many citations as co-authors. Restricting to coauthored papers reduced the total number of authors in the study data from 159 to 125, so the last row in the Spearman’s coefficient table spanned the rank interval of 5 instead of 20. The rho’s in the rank interval of 121-125 turned out to be all statistically insignificant, but we considered the ranking difference

Table 8.

Spearman s Rank Order Correlation at Rank Intervals, Using AuthorRank Weights (au_cnt>1)

(df=18, α=0.05, CV=0.447), (df=3, α=0.05, CV=1.000)

Table 9.

Author Rankings by Citation Counts Weighted by AuthorRank Weights (au_rank>1)

in such a small interval to be not meaningful for gauging differences in bibliometric measures even if such outcomes were not spurious.

To isolate the authorRank effect further to the point of magnification, we excluded all publications from the study data where the faculty members in the study were listed as first authors. The resulting data subset included 107 authors that published 594 papers, where 61% were 2-author papers and 39% were papers with 3 or more authors. As expected, the rank differences became more pronounced with a restricted dataset (Table 9) with smaller rho across

Table 10.

Spearman’s Rank Order Correlation at Rank Intervals, Using AuthorRank Weights (au_rank>1)

(df=18, α=0.05, CV=0.447), (df=5, α=0.05, CV=0.786)

rank intervals (Table 10), which serves as evidence that non-primary author contributions should be treated differently from primary author contributions.

3.3. Publication Count and Citation Count vs. h-index and g-index

In addition to comparing publication count and citation count, which measure quantity and quality of research respectively, we investigated h-index and g-index, which consider both quantity and quality of research. The fact that rankings by publication count and h-index (p-h) and by publication count and g-index (p-g) show significant ranking differences in all rank intervals demonstrates inherent differences in what publication count measures and what h- or g-index measure (Table 11). Rankings by citation count and h-index (c-h) and by publication count and g-index (c-g) show significant ranking

Table 11.

Spearman’s Rank Order Correlation at Rank Intervals, Using PC/CC vs. h-/g-index

differences in all rank intervals except for the very top and very bottom rank intervals, which could reflect the tendency of citations to overwhelm the h- and g-index computations for authors with very high or low citation counts.

Table 12 shows the Spearman’s coefficients for the entire 159 authors with additional comparisons of authorRank weights, where suffixes correspond with authorRank formulas. For instance, h2 is ranking by h-index where citation counts are weighted with the author weight formula (CC2), h3 uses the author rank formula (CC3), and h4 uses the author count formula. The strength of association between publication count and h-/g-index using CC2 are consistently lower than for other ranking comparison pairs, which suggests that CC2 (author weight) formula may be the most robust authorRank weighting formula used in the study.

Table 12.

Spearman’s Rank Order Correlation at Rank Intervals (Overall)