Higher education is inundated with data. The assessment of academic work relies heavily on quantification, whether through the criteria utilized by college rankings, the metrics employed by faculty and administrators to evaluate scholarly significance, or the outcomes chosen to measure the value of a college education.
The challenge lies in effectively utilizing this abundance of data. How can it be harnessed, combined, and analyzed to produce accurate and meaningful assessments of institutional quality, scholarly influence, and social impact?
A recently established company, founded in 2020 by Jed Macosko, a Berkeley PhD, professor of physics at Wake Forest University, and president of the emerging startup, proposes a solution through an innovative application of machine learning. This approach aims to evaluate various facets of higher education, such as college and department rankings, scholarly reputations, and publications of significant impact. (I have authored two articles for the company’s website, full disclosure.)
The company’s methodology revolves around an algorithm that tracks the frequency of references to an individual’s work across various databases. For instance, it analyzes the mentions of Richard Phillips Feynman, the Nobel laureate in theoretical physics, in sources like Wikipedia, and in the abstracts, titles, references, and keywords of publications from different platforms. By extrapolating from these data points, the algorithm generates comprehensive assessments of various forms of impact.
Rankings in Higher Education
Among Academic Influences’ initial offerings is a college ranking system that operates on the premise that the quality of an institution is determined by the individuals associated with it.
Utilizing machine learning technology, initially developed with support from the Defense Advanced Research Projects Agency, Academic Influence scours open-source data from three major platforms – Wikipedia, Semantic Scholar, and CrossRef – to compile information on papers, chapters, books, and citations linked to individuals worldwide. These databases collectively house billions of constantly updated data points concerning the achievements of millions of individuals.
Subsequently, an institution’s influence score is computed by aggregating all references to individuals affiliated with it as faculty, administrators, or alumni. This score is then normalized by the school’s total student population to ensure that smaller and mid-sized institutions have an equal opportunity to compete with larger colleges. This normalization process, termed “Concentrated Influence,” enables a smaller school with proportionally more influential faculty to potentially outrank a larger institution with greater absolute influence.
To mitigate potential biases, the data considered are limited to the past decade, and the names of well-known figures from fields like politics, arts, entertainment, and sports are suppressed to prevent them from unduly influencing a school’s ranking. For example, the achievements of Michael Jordan do not artificially boost the University of North Carolina’s ranking, nor does Ashley Judd’s success enhance the standing of the University of Kentucky.
Leveraging its influence calculations, Academic Influence has generated rankings for various types of institutions and programs, including research universities, global establishments, online MBA programs, liberal arts colleges, historically black colleges and universities (HBCUs), religious institutions, and community colleges. Additionally, it identifies the top college in each state.
These rankings serve as a valuable resource for students, parents, and other stakeholders, driving traffic to the company’s website through organic searches. This web traffic is vital to the company’s revenue model, as a significant portion of its income is derived from facilitating college applications by prospective students, with colleges compensating the company for each applicant expressing interest. Another revenue stream involves licensing the proprietary ranking algorithm to external platforms.
Since its website launch in August 2020, Academic Influence has witnessed a substantial increase in traffic, soaring from 50,000 to over 1,000,000 visitors per year, as measured by [ppp1].
Scholarly Influence
Academic Influence extends its ranking methodology to assess the impact of individuals within various academic disciplines, akin to evaluating an academic’s intellectual footprint online.
Currently, the platform provides insights on scholars across 24 disciplines and approximately 300 sub-disciplines. Whether one seeks information on leading economists, psychologists, or sociologists, Academic Influence offers detailed lists on its website. These rankings often feature additional categorizations based on demographics, such as identifying the most influential women in a particular field.
Measuring scholarly impact has become a critical practice in academia. Universities frequently engage services like [ppp2] to gauge the influence of faculty members’ research outputs – encompassing grants, papers, journal articles, and books – incorporating these assessments into faculty promotion and tenure evaluations.
Metrics like the [ppp3], developed by physicist Jorge E. Hirsch, are commonly employed to quantify scholarly impact, combining an author’s publication volume with the extent of their citations to establish a unified measure of influence.
Macosko, serving as the President and Research Director of Academic Influence, asserts that the company’s approach, which considers thousands of data points, represents an advancement over traditional metrics like the h-index, which only accounts for an author’s publication volume and citation count.
He highlights limitations in the h-index, such as its failure to differentiate between highly-cited review articles and empirical studies reporting original data. This distinction is crucial as an author emphasizing review articles might garner a higher h-index than a researcher publishing empirical studies with groundbreaking discoveries.
Moreover, the h-index tends to overlook publications receiving numerous citations, potentially underestimating the impact of exceptionally influential contributions.
In evaluating academic departments and graduate programs, particularly in scientific fields, data-driven impact assessments play a pivotal role. These assessments inform strategic decisions within universities, guiding choices on bolstering specific academic departments, reducing emphasis on others, or identifying research areas with promising prospects for increased funding.
Given the consequential implications of these assessments, faculty members often exhibit caution, if not skepticism, towards relying solely on big data to evaluate the quality of their work. Their reservations warrant serious consideration, as no impact measurement tool, including those utilized by Academic Influence, is devoid of flaws.
For instance, an individual may receive numerous mentions for work that is controversial in the public sphere but lacks fundamental or substantial impact within an academic domain. Jordan Peterson, a prominent clinical psychologist according to Academic Influence’s rankings, serves as an illustration. While widely influential, Peterson might be perceived by many academic psychologists more as a cultural figure or divisive voice rather than a scholarly authority.
When confronted with such objections, Macosko emphasizes that Academic Influence’s approach to measuring influence diverges from conventional metrics. By considering a broader array of data points, the company’s methodology aims to provide a more comprehensive evaluation of an individual’s impact, transcending the limitations of metrics like the h-index.
Academic Influence aspires to revolutionize the landscape of college rankings and academic impact assessments. Its founders contend that their computational methodology minimizes the inherent subjectivity present in other approaches, often referred to as “gameability,” where subjective factors intermingle with numeric indicators of institutional and scholarly influence. However, the extent to which removing human judgment from these processes eliminates noise without losing valuable insights remains a critical question.
As with any ranking system, it is advisable for consumers to view Academic Influence’s rankings as a component of the broader picture rather than an ultimate verdict. Macosko underscores this perspective, highlighting that Academic Influence’s algorithm serves as an additional tool for academic departments to make informed decisions, supplementing traditional vetting processes to identify the most suitable candidates. Ultimately, Academic Influence aims to empower academia with a new tool that accentuates the intrinsic influence of individuals, drawing from a vast array of data points beyond conventional metrics like the h-index, fostering a more nuanced understanding of scholarly impact and institutional quality.