3.1. Citation Performance of Experimental Articles

Table 2 shows citation performance as the average citation index of the experimental articles. Citation performance appears to have a pattern of increment and stabilization after journal publication. The citation performance of papers from 2001 kept a constant level from 2006, the sixth year of publication. Papers in 2004 and 2007 showed an average citation performance maintained from their fifth year, 2008 and 2011, respectively. Interestingly, in more than 10 years of tracking of papers of 2001 and 2004, 2004-issued papers in 2008 (i.e. the fifth year of issue) show a trend of increasing citation performance that exceeds that of 2001-issued papers and then showed a descent below performance of 2001-issued papers in 2012 (i.e. ninth year) and currently show an indiscriminately small increase over the performance of 2001-issued ones. This trend of fluctuating research reference performance of the papers suggests dissemblance in the research field, to which each of the annual journal issue provides contribution.

Table 2.

Citation Performance of Experimental Articles

With constant citation performance for more than 10 years, our target journal Information Systems Research falls into the research field categorization of social science – “Information Science & Library Science,” as well as “Operations Research & Management Science.”

Among the experimental articles, as far as the papers citing for the longest period of time are concerned, are: in 2001, “Research commentary: Technology-mediated learning - A call for greater depth and breadth of research” (2001–2014, 83 cases): in 2004, “IT outsourcing strategies: Universalistic, contingency, and configurational explanations of success” (2004-2014, 91 cases), “Building effective online marketplaces with institution-based trust” (2004-2007, 270 cases): in 2007, “Is the world flat or spiky? Information intensity, skills, and global service disaggregation” (2007-2014, 33 cases): in 2010, “Ex ante information and the design of keyword auctions” (2010-2014, 22 cases), “Quality uncertainty and the performance of online sponsored search markets: An empirical investigation” (2010-2014, 15 cases), and “Information transparency in business-to-consumer markets: Concepts, framework, and research agenda” (2010-2014, 10 cases): while the most-citing paper in 2001 is “Research commentary: Desperately seeking the ‘IT’ in IT research - A call to theorizing the IT artifact” (355 cases, 2004-2014): in 2004, “Building effective online marketplaces with institution-based trust” (270 cases, 2004-2014): in 2007, “Through a glass darkly: Information technology design, identity verification, and knowledge contribution in online communities” (98 cases, 2008-2014): and in 2010, ‘The new organizing logic of digital innovation: An agenda for Information Systems Research” (25 cases, 2012-2014 ).

3.2. Subject Analysis of Experimental Articles

Subject-words (“Author Keywords” field of Web of Science record) of experimental articles used are 697, appearing in 782 cases; by mean, around 6.11 subject-words were used per paper. Major subject-words discovered were “electronic commerce” (9 cases), “information technology” (6 cases), “information systems strategy,” “competitive dynamics,” “decision support systems,” “electronic markets,” “resource-based view” (4 cases), “knowledge management,” “competitive actions,” “structural equation modeling,” “decision support,” “decision making,” “information systems development,” “business value,” “inter-organizational information systems,” “competitive impacts of IS,” “partial least squares,” and “coordination” (3 cases), which are demonstrated in Table 3 as the year-wise cluster of subject-words for the issued journal. The subject-word cluster showed similar result when comparing the subject-word cluster for the entire experimental article to the subject-word cluster for experimental articles with citation performance. It seems because articles with citation performance are 96.9% of the entire articles in the journal.

Table 3.

Major subject-word and subject-word Cluster in Publication Year of Experimental Articles

The number of cluster in earlier years shows a greater subject-word cluster: for the year 2001, 3 clusters and in 2004, 2 clusters; for both 2007 and 2010, 5 clusters. Entire research fields were separated into 100 classes on the basis of ISI Categories, which proved to be problematic in distinguishing subject difference on year of publication, by means of subject-word and subject-word cluster of experimental articles.

4.1. Research Field Cluster of Citing Paper

To analyze the research field of citing papers of experimental articles, clustering and statistical methods are applied to four groups (2001, 2004, 2007, and 2010) of experimental articles to attain the extent of contribution to the field. Each cluster of research field was titled after the research field with the highest weight. There are 8 clusters in total of four experimental groups: #1 (Computer Science), #2 (Business & Economics), #3 (Operations Research & Management Science), #4 (Psychology), #5 (Communication), #6 (Telecommunications), #7 (Oceanography), and #8 (Science & Technology - Other Topics). The aforementioned top 5 fields in research field analysis – “Information Science & Library Science” and “Engineering” did not name the cluster, but instead are included as cluster items: “Information Science & Library Science” is under cluster #1 (Computer Science), and “Engineering” is under cluster #3 (Operations Research & Management Science).

Clustering of citing research field by four groups of publication year of experimental articles, shown in Table 5, disclose the distinct nature of research fields for each group. 2004’s citing field cluster has “Information Science & Library Science” and “Computer Science” clustered together; however, they are differently clustered in other year-groups. Also, “Engineering,” clustered with “Operations Research & Management Science” in 2001, 2004, and 2010, but didn’t cluster with it in 2007.

Table 5.

Research Field Cluster of Citing Paper

#ranking (research filed, weight)

Research field clusters of citing paper for the 4 yeargroups within the three-year unit of experimental paper publication is shown in Table 6. As the time passes, amount of cluster increases. Specifically, as noticed with number of citation in Table 2, number of cluster remains stabilized from six year after the publication.

Table 6.

Number of Research Field Cluster per Publication Year Unit of Citing Paper

However, experimental articles maintaining a stable citation index and research field group alone cannot lead to stability in contribution to research field. To prove this, chronological variation in research field must be considered.

Comparisons of groups of citing papers from 3 consecutive years of experimental article publication were made to conduct comparative analysis. Table 7 displays the cluster in 2001-issued experimental articles on citing fields, where chronological variation in the citing field is apparent. The most citing research fields are 1st “Computer Science,” 2nd “Business & Economics,” 3rd “Information Science & Library Science,” 4th “Operations Research & Management Science,” and 5th “Engineering.” From the citing field of 2001-2003, OP belongs to #1 (Business & Economics) cluster and IS belongs to cluster #2, but from 2004-2006, OP is in cluster #3 and IS is in cluster #1. During a period of stable experimental article citation performance, 2007-2009 and 2010-2012, the citing field cluster number stays at 8 but composition of cluster differed. In the case of ENG, it was placed in #4 (Operations Research & Management Science) in 2007-2009 while forming a new cluster in 2010-2012. IS formed another cluster #3 (Information Science & Library Science) in 2007-2009; however, it fell into #1 (Computer Science) in 2010-2012. Moreover, one interesting note here is that besides major citing field, Psychology, Social Science, Education & Educational Research, and Telecommunications appeared as new clusters and disappeared with time passage.

Table 7.

Citing-Period-Wise Research Field Cluster of 2001-issued Experimental Articles

#ranking (research filed, weight)

4.2. Variation in Citing Field on the Publication Year of Experimental Articles

As mentioned before, 4 groups of experimental articles (2001, 2004, 2007, and 2010) showed varied citing field. In a further step of the study, One-way analysis of variance table was used in statistical interpretation of variation in research field based on publication year, as shown in Table 8. In the top 10 ranking of research field (citing field) are “Computer Science,” “Business & Economics,” “Information Science & Library Science,” “Engineering,” “Operations Research & Management Science,” “Psychology,” “Education & Educational Research,” “Communication,” “Social Sciences - Other Topics,” and “Health Care Sciences & Services,” along with individual clusters of research field from citing field cluster, such as “Telecommunications,” “Public Administration,” “Environmental Sciences & Ecology,” and “Geography.”

Table 8.

One-Way Analysis of Variance for Citing Field of Experimental Articles

* p<0.05, **p<0.01

Citing fields showed significant variation in citing field among the experimental group, except for “Information Science & Library Science,” “Telecommunications,” “Public Administration,” “Environmental Sciences & Ecology,” and “Geography”. In analysis of the field “Business & Economics”, it showed that the higher the year of the experimental group was, the greater the average score was, and inter-group variation by Dunnett T3 appeared the significant difference between the group 2001 and the groups 2004, 2007 and 2010.

“Operations Research & Management Science” and “Health Care Sciences & Services” fields also show a similar pattern of correlation between year and average score. Confirmation testing for “Operations Research & Management Science” proved the group of 2001 and 2004 had significant difference from the group of 2007 and 2010, whereas for “Health Care Sciences & Services” the group of 2001, 2004, and 2007 differed from the group of 2010.

On the contrary, “Education & Educational Research” and “Social Sciences - Other Topics” fields showed an opposite trend of decreasing average as the year of group grew later. The “Education & Educational Research” field group of 2001 showed significant different from the group of 2004, 2007, and 2010, whereas the “Social Sciences - Other Topics” group of 2001 and 2004 differed from the group of 2007 and 2010.

In summary, the target journal papers group in the past few years of time had a high contribution (average score) in the fields of “Business & Economics,” “Operations Research & Management Science,” and “Health Care Sciences & Services,” and a low contribution (average score) in the fields of “Education & Educational Research” and “Social Sciences - Other Topics.” It is known that the study-target journal Information Systems Research, which is classified in fields of “Information Science & Library Science” and “Operations Research & Management Science” in JCR (Journal Citation Report), has been contributing to a higher extent in recent years to the fields of “Business & Economics,” “Operations Research & Management Science” and “Health Care Sciences & Services,” while contribution in the fields of “Education & Educational Research” and “Social Sciences - Other Topics” are reduced in recent years.