Advancing bibliometric assessment of research productivity: an analysis of US Departments of Family Medicine

KEYwords: Professional development; primary health care; health research; bibliometric analysis

‘When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely…advanced to the stage of science’…Sir William Thomson¹

The lifeblood, legitimacy and future of a scientific discipline depends on continual growth of its unique features and body of knowledge through research.^2,3 Family physicians (a synonym we use in this paper for ‘general practitioners’) and other primary care providers depend on a vibrant research enterprise to find answers to the questions relevant to the unique set of health services provided to most patients, most of the time, in all stages of wellness and illness.⁴

There is an expansive literature that captures research productivity across topics, countries and disciplines.^5–11 Used in hiring, annual reviews, promotion and grants, publications are widely recognised as units of productivity. Given the differential impact of publications, researchers sometimes also incorporate into their curriculum vitae citations and the impact factors of journals where the works are published.^12,13

Measurement of publication productivity in family medicine has been reported only sporadically, using different methods and with different purposes in mind. Twenty-five years ago, a review of United States (US) primary care research conducted by family medicine investigators revealed that the majority of studies had been completed without external funding.¹⁴ Using PubMed and hardcopy searches, Pathman et al. identified family medicine authors of research articles published in a wide range of journals and concluded that productivity is growing and higher than previously recognised.¹⁵ Instead of capturing output, Merenstein et al. compared design quality for research papers that had at least one family medicine author and were published in one of 20 journals in 2000 and in 2005. They found that research quality had not improved during this time.¹⁶ Like Pathman et al., Post et al. used PubMed to assess the productivity of Society of Teachers of Family Medicine (STFM) members and found that a higher percentage of members published in 2009 than in 1999.¹⁷ In a 2015 study using Web of Science, another group updated these figures and concluded that US family medicine faculty published 3002 times that year.¹⁸ These studies demonstrate the heterogeneity of data sources, authors, journals and publication types of past reports, highlighting the need for a consistent methodology and population of study to provide a longitudinally coherent account of publication productivity in family medicine.

Family medicine in the US has dedicated significant resources towards building its research enterprise. Family Medicine for America’s Health was launched to explore the future and enhance the visibility of the discipline in six core areas, including research.¹⁹ The Association of Departments of Family Medicine and North American Primary Care Research Group have collaborated to create the Building Research Capacity initiative to support research development in the discipline.²⁰ Starfield Summits have used research to inform policymakers and seeded investigators with new ideas, methods and collaborators.^21,22 In some cases, external agencies have also enabled efforts to enhance scholarship; for example, the family medicine milestones created by the Accreditation Council for Graduate Medical Education established expectations for faculty research productivity.^23,24

In this study, we aimed to compare three methods of capturing productivity in an attempt to identify the advantages and disadvantages of these methods alone or for all possible combinations. This work provides the empirical assessment needed to develop a meaningful and practical methodology for measuring research productivity that could be replicated periodically across all US family medicine departments in academic medical schools in a longitudinal manner, and is likely to be transferable to other countries, allowing for international comparisons.

In this cross-sectional analysis, we compared three methods for capturing the research publication productivity of US departments of family medicine: faculty-to-publications, publications-to-faculty, and department-reported publications. We included 13 family medicine departments with active research programmes (Table 1) for this developmental study. The results of this work informed a broader effort to capture the publications across all US family medicine departments.¹⁸ Ten of these department chairs were members of the Research Development Committee of the Association of Departments of Family Medicine. This Committee comprises 12 members who develop and implement strategic goals and programmes designed to strengthen the research and scholarship capacity of US Departments of Family Medicine. Three additional chairs were recruited to enhance geographic and departmental structure diversity, particularly to include departments with divisions representing other disciplines in addition to family medicine. In 2015, there were 134 family medicine departments in US medical schools, a majority of which were members of the Association of Departments of Family Medicine in 2015.

Table 1. Participating Department characteristics

We first used department websites to identify faculty (as of December 2016) and their research publications using Web of Science, a subscription-based bibliometric database that contains journals covering a wide range of disciplines (including medicine, social sciences and humanities).

Using the author affiliation field in PubMed, we identified publications linked to family medicine departments and then connected these products to commensurate faculty members. In this field, we searched for the institution’s name, the name of the department as listed on the department’s website and ‘Department of Family Medicine’. PubMed is a free database that includes biomedical and health-related life science journals. Because of its more limited scope, it also indexes fewer journals than Web of Science.²⁵

Because this list of publications includes individuals affiliated with each department but not in the core faculty (eg residents, visiting professors), we included only articles from employed faculty listed on the department website or the department-provided list (see below).

In this method, we asked the 13 department chairs to provide a list of their employed faculty (including faculty who did not publish) and their publications.

Through a consensus-driven, iterative process, we collaborated with the Research Development Committee to identify faculty, divisions and publications to include in this analysis. We defined core academic faculty as faculty who have primary employment in the department (excluding faculty contracted with the department, visiting faculty, etc), have primary academic appointment with the department as of 1 January 2015 and whose academic appointment was included in the approved list (Table 2). If a faculty member had conflicting titles (one in the inclusion list and one in the exclusion list), we included the individual in the analysis. We incorporated all divisions within the departments, which included family medicine, community health, occupational medicine, physician assistant, public health and epidemiology. In the US, a subset of family medicine departments consists of multiple divisions. These divisions can emerge organically due to a critical mass of faculty expertise or result from the merger of departments as devised by institutional leadership. Understanding the contribution of these divisions to productivity informs the interpretation of data describing the output of family medicine scholars. In addition, many US family medicine departments employ investigators from disciplines such as psychology, epidemiology, anthropology and biostatistics on their faculty, and are not organised into distinct disciplinary divisions. To be included, publications had to be published in a peer-reviewed, indexed journal. We defined peer-reviewed as an editorial process where manuscripts were reviewed by reviewers, editorial referees or editors, and the journal had to be retrievable through a bibliometric index. We included original research (both primary and secondary data analyses) and narrative publications. We included articles with publication dates between 1 January 2015 and 31 December 2015.

Table 2. Academic appointment inclusion and exclusion criteria

We calculated counts and proportions of the number of faculty members and the number of unique publications by department and by faculty academic rank titles identified through department-reported publications, faculty-to-publications, publications-to-faculty processes separately; department-reported publications and faculty-to-publications combined; department-reported publications and publications-to-faculty combined; faculty-to-publications and publications-to-faculty combined; and all three processes combined. We repeated these processes for the three departments with divisions, merging family medicine, physician assistant and occupation health divisions. We reported these measures individually for the community health, public health and epidemiology divisions.

The American Academy of Family Physicians Institutional Review Board approved this protocol.

We identified 750 faculty members, a mean of 58 per department (Table 3). Eighty-seven percent were found on websites, whereas 89% were reported by the department. The percentage of faculty identified on websites varied across departments, ranging from 53.6% to 98.3%.

Table 3. Counts and proportions of Department Faculty identified by method or a combination of methods

We identified 1052 unique publications by combining the three processes to identify publications. Departments reported 878 publications in 2015, while the faculty-to-publications process identified 616 and the publications-to-faculty process identified 412 (Table 4). The percentage of total publications identified through the faculty-to-publications process ranged from 27% to 89% per department, whereas these figures were 9% to 100% for the publications-to-faculty process.

Table 4. Publication counts and proportions by individual methods and combinations of methods

Across all departments and combining the three methods, we found that 32.0% of faculty had any publications in 2015 and that the mean number of publications per faculty was 1.4 (Table 5). Including only faculty who published, the mean number of publications was 4.4. The percentage of faculty publishing in 2015 ranged from 16.0% to 82.6%. Two departments averaged over four publications per faculty member. Over 90% of Chairs published in 2015, whereas this figure was 66% for Professors. Assistant Professors (often called ‘senior lecturers’ in other countries), Associate Professors, Professors and Chairs accounted for 92% of all publications. Among the three departments with divisions, faculty in the family medicine divisions accounted for one-third of all publications and averaged nearly two publications per faculty.

Table 5. Publications, by Faculty title, Department and Division

This project represents a major step forward in bibliometric measurement of family medicine research productivity. Through this effort, we have brought together expert informants from multiple, leading family medicine departments, the support of a major scholarly organisation, and explored traditional and novel methods for tracking family medicine research productivity. This study advances medical bibliographic research by comparing multiple bibliometric research methods for estimating proportional capture (relative to total publication denominator) identified by the three methods of department-reported publications (84%), faculty-to-publications (59%) and publications-to-faculty (40%). Optimal measurement may require combining all three methods. However, making the assumption that the systematic error leading to missed publications using PubMed (the publications-to-faculty method) is relatively consistent over time, a trend could be established over the 50 years since family medicine became a recognised medical speciality in the US. The more recent development of department websites and Web of Science would not allow the faculty-to-publications method to be useful retrospectively, but may provide an alternative method for prospective monitoring of publications.

Faculty generated a mean of 1.4 publications (4.4 for faculty who have published) and 32% of faculty had at least one publication in 2015. These figures are higher than those reported in the article by Post et al., who found that 21% of STFM members had published in 2009, with a mean of 2.2 publications for members who published, although faculty with qualified academic titles may be overrepresented because the STFM focuses on teaching more than research.¹⁷

The combination of all three methods led to the most comprehensive view of publication productivity, underscoring the errors inherent in each individual method. For instance, departments missed faculty and publications when submitting the departmental reports. Because publications are tracked for promotion and tenure, this method is dependent on the conscientiousness of the department or individual recording an individual’s publications in a given year, and several of these faculty members had more than 20 publications. The faculty-to-publications method is limited by the accuracy of websites, which do not have standardized practices for titles to display, faculty to include and intervals for updating. Once identified, we found that using names to query bibliometric databases was also fraught with imprecision, as website and publication names differed due to the use of nicknames and middle names and the modification of names over time. The faculty-to-publications method was further limited by the universe of journals included in Web of Science, encompassing articles in journals found in PubMed through the publications-to-faculty process and missing articles reported by departments but in journals not indexed in Web of Science. Finally, the publications-to-faculty process was dependent on the accuracy and completeness of the author affiliation field in PubMed. The publications-to-faculty process missed articles where this field was blank, included the institution name only (e.g. the University of Virginia) rather than the institution and department names (e.g. the Department of Family Medicine, University of Virginia) or included a different institutional affiliation (e.g. a cancer centre within the institution or an affiliated hospital). This method also captured authors affiliated with the department but not core faculty, such as family medicine residents. Publishing articles online before print articles also explained discrepancies across the methods, with PubMed capturing both electronic publishing and print dates while Web of Science includes only print dates. Therefore, a publication electronically published in 2015 but printed in 2016 was captured by PubMed but not Web of Science.

Using a robust method to track productivity over time is critical to academic family medicine’s future, and we have established a set of methods that can be used in isolation or combination to yield an approximation of family medicine research productivity. Our discipline should invest resources in improving our research enterprise and assessing the impact by repeating this process across all departments at predictable intervals. Without such data, the discipline generally, and departments specifically, will be unable to track performance against benchmarks, calculate returns on investments in research, and identify positive outliers with effective strategies for success.

One incremental step towards realising this goal would be to create and maintain a database of family medicine faculty members, each with a unique identifier. Because similar unique identifiers exist, they have their own shortcomings; for example, ORCID uniquely identifies researchers, but adoption is not comprehensive for family medicine.²⁶ In contrast, the Association of American Medical Colleges’ Faculty Roster is more comprehensive but not publicly available.²⁷ Combining the faculty-to-publications or publications-to-faculty methods with department reporting is associated with marginal benefits and costs. As we have demonstrated, the inclusion of departmental data will increase the number of publications identified, yet the inclusion of departmental input naturally requires additional resources. Furthermore, the degree to which departments participate may be variable across departments and over time, introducing additional error.

Similar to other bibliometric approaches to measuring research productivity and performance at the department level, several limitations need to be kept in mind. The most frequent criticism of bibliometric analysis is its inability to measure quality accurately. Readers should have reservations about using bibliometrics to rate an individual researcher’s intellectual performance, and whether such quantitative analyses appropriately account for qualitative evaluations. In this analysis, each publication is weighed equally, although publications differ with respect to methodological rigour, journal quality and citations. Our methods do not allow us to measure the ability of the research to improve health or change policy. Second, our department sample was purposefully limited to 13 departments, most of whose Chairs were members of the standing research development committee of a national organisation. Our findings with respect to publication count are not likely to be representative of all family medicine departments. Similarly, the performance of the methods may be different when applied to all departments. Third, we limited our analysis to a specific subset of publications and faculty. As we were interested in the productivity of core faculty, we did not include instructors, adjunct professors and residents, although these faculty members contribute to scholarship within departments. Nor did we include community-based residency programmes where core faculty are not employed by academic departments. Furthermore, we restricted this analysis to research publications and excluded books, curricula and other scholarly works. Finally, because we captured faculty names from websites in December 2016, the website roster may not be an accurate representation of faculty present in 2015.

In summary, nearly 60% of total publications were identified through a method that uses websites and Web of Science alone, whereas only 40% were identified using PubMed alone. Combining these two methods identified 71% of all publications for this study. One of every three faculty in our sample had published one paper in 2015, which is higher than previous estimates, although the sample and methodology make direct comparisons difficult. In addition to future efforts to measure the quality of publications and impact, assess the products of scholarship more broadly defined among all family medicine organisations and compare these results across countries, these findings should be tracked at predictable intervals to determine the return on our discipline’s investment in research and identify strategies for enhancement.

The authors declare that they have no competing interests.

This research received support from the American Board of Family Medicine Foundation and Family Medicine for America’s Health. The funders did not have a role in the design of the study and collection, analysis and interpretation of data. The funders did not have a role in writing or disseminating the manuscript.