The Second Research, Part II

Institutional Effectiveness Today

One is not trained to be an institutional researcher, but rather becomes an institutional researcher. The Wikipedia page for Institutional Research suggests, in the section labeled, “Becoming an Institutional Researcher,” that “[t]here is no single academic degree that qualifies one to be an institutional researcher….”^[1] Although most degree programs in higher education administration aspire to prepare “future heads of colleges and universities and other administrators… to lead effectively,”^[2] the Wikipedia page seemingly affirms the absence of substantive administrative functions and defined leadership roles for institutional researchers in the literature on higher education and higher education administration. Wikipedia hardly figures as a primary resource for career decisions, but the description of the profession invites one to wonder: who would aspire to become an institutional researcher after reading the entry?

The Association for Institutional Research (AIR) acknowledges that “the current function of IR is not clearly defined, and the future path of IR is unknown,” and seeks through various means an answer to the question. Aided by the largesse of the Bill & Melinda Gates Foundation, the AIR established an initiative, “Improving & transforming institutional research in postsecondary education,” to address “a concern that current IR practices are not adequate for the challenges that face decision makers today.” In response, the AIR is working on statements of aspirational practice for institutional research. The preliminary statements generalize the concept of “Decision Makers” to include nearly every person with the potential to consume institutional data (including students), elevate service to students as the first priority for analysis and reporting, and redefine institutional research as a commonplace capacity and practice that is effectively managed by the functional units of the institution. Each aspirational statement is laudable in its own terms and certainly provide fodder for those who wish to become institutional researchers.

As a whole, however, the preliminary aspirational statements amplify the perception of institutional research outlined in Part I of this brief. The generalized decision maker, far removed from the executives of higher education administration, further solidifies institutional research as a nondisciplinary practice or a non-specialized secondary research activity for the requesters of data. The “student-focused paradigm,” coupled with the expanded definition of decision makers to be non-administrators, portrays the most impactful decisions on student success as those occurring on the “student and faculty calendars… not always align[ed] with administrative cycles.” Lastly, removing “obstacles for institutional studies” posits that functional directors and vice presidents know best how to organize institutional research and “they will expand their unit’s capacity to conduct data studies that align with their own information needs.” No longer required to be the factotum, the institutional research officer of tomorrow now aspires to leave institutional research to the functional units and ferret out information across data silos for those in a better position to make effective decisions for students. If, as brief summary indicates, the statements “are reflections of changes that are already observable,” the institutional research future seems to be always already determined by a repetition of the institutional research past.

Without a discipline, a core learned competency, institutional research functions will continue to languish in ill-defined roles and in uncertain futures dictated by the mundane requests of others (so-called “decision-makers”). As stated in Part I, however, there is another path open to the future of institutional research, also “already observable” in the changing landscape of higher education, that holds the promise of continuous improvement – or, institutional effectiveness today.

Formative Terms

In deference to the preeminence of decision-making in the direction of institutional research, a sense of its origins provides some important context for the future of institutional research. The Harvard Business Review provides a “Brief History of Decision Making” in which the authors note that the term as a managerial device originated in public administration. Subsequently, the business world adopted and substantially transformed its relevance for organizational success during the late twentieth century. Aided by computers and the development of “decision support systems” for executive leadership, technology enabled “‘chief executives to define their own data needs‘” and establish “‘business intelligence'” solutions for “decision makers throughout the organization” who held responsibilities for the direction of their companies. At the end of the century, flush with their success with chief executives, technology companies targeted a new kind of decision maker: “customers themselves.”^[3] As is well known, in recent years, the private sector sought to return decision making to higher education administration in this altered form.

The latter stage in the genealogy of decision-making remains evident in education technology literature today. IBM’s vision for “Big Data” and education asserts, “[t]oday’s students expect their learning environments to mirror the environments in which they grew up and now live ― that is, punctuated by always-on, available-anywhere information and personalized, multichannel learning.” To service students in their natural environment, Tableau’s product provides “self-service analytics” that empowers end-users to “use your natural ability to see patterns, identify trends and discover visual insights in seconds.” There is “no shortage of data,” IBM declares. There is only a shortage of data analysis tools that “work as fast as you do,” Tableau answers. The decision maker-customer-student model conforms to the trajectory of technology-aided decision making over the past 80 years. Every person knows what s/he “needs” and possesses the “natural ability” to read the iconography of data to find the stuff to make decisions. It is common sense, “PhD not required” (Tableau), and never mind the traps.

Institutional research stands at the crossroads of social science research and common sense “data studies”^[4]. In his seminal work, The Structure of Science, the philosopher of science, Ernest Nagel, juxtaposes science and common sense in order to define the “special excellence” (1) and superior contributions “to the acquisition of knowledge” (2) by the former over the latter. Science seeks to understand “why the facts are as alleged” (3) and shape explanations based on “critical tests of their [the explanations’] relevance to the facts” (4) in order to accumulate knowledge systematically. “[T]he organization and classification of knowledge on the basis of explanatory principles… is the distinctive goal of the sciences” (4). In contrast, common sense establishes rules that fail to grasp the limits of their reliability or validity and perpetuates knowledge that is “most adequate in situations in which a certain number of [unexplained] factors remain practically unchanged” (5). Common sense entertains “incompatible and even inconsistent beliefs” borne of “an almost exclusive preoccupation with the immediate consequences and qualities of observed events” (6). Common sense relays its beliefs through imprecise language and terminology that muddles “the distinction between confirming and contradicting evidence for such beliefs” (8). Common sense primarily concerns itself with parochial knowledge of “special value” (10) to the particular condition [i.e., institution] that is not subjected to “systematic scrutiny” to assess “the range of [its] validity” (12).

In a manner of speaking, institutional research may have linked its fortunes to common sense long ago. The myth of the factotum depends on premises that the “needs” and the “natural abilities” of others – whether decision-makers at a local institution or external agencies – provide the necessary and sufficient conditions for the practice of institutional research. In service to common sense “data studies,” institutional research will certainly become more entrenched as an other-directed enterprise and accentuate IR officers’ in/ability to build relationships (i.e., “respond to requests for data”). Big Data and technology-laden solutions correspond well with the institutional research past, but should they be the foundational aspirations for the institutional research future? As an alternative, Nagel suggests, “The conclusions of science are the fruits of an institutionalized system of inquiry…” [ my emphasis] (14) that refines a body of knowledge, unifies a system of explanation, determines language, produces abstractions, and sustains the critique of its own knowledge and practices. Undoubtedly, institutional research exhibits aspects of – and, already is in name – “an institutionalized system of inquiry” to explain student learning outcomes, bolster strategic planning, and champion continuous improvement in higher education. How then do we fully conceive institutional research as a social science?

Exemptions

“Sapere aude! ‘Dare to know!'” As a historian by training, I find Immanuel Kant’s dictum in answer to the question, “What is Enlightenment?” to be the germinal sentiment for institutional research as a social science. Of course, institutional research in the past has been quite daring in its efforts to assimilate through secondary acquisitions the understanding and explanations of others who know what data are needed to run an institution of higher education. Kant’s admonishment, however, stands as a constant reminder that a science require its practitioner to “‘[h]ave courage to use your own reason!'” – to form an understanding independently and acquire knowledge directly. Moreover, the exercise of one’s own reason must take place “as a scholar before the reading public,” not as a person may engage reason privately “in a particular civil post or office [or institution] which is intrusted to him [sic].”^[5] In application to institutional research as a social scientific enterprise, the IR practitioner must engage in primary research to systematically scrutinize explanations resulting from prior acquisition of knowledge and to enhance the understanding of education-related phenomena in order to improve practical problem solving and decision making.^[6] Dare to know: the unassailable prerequisite of inquiry.

To stand as social scientific research, institutional research nonetheless must have a set of phenomena to distinguish its field of inquiry from existing social sciences while also establishing its interrelations with other social sciences. On the first account, the U.S. federal guidelines for the protection of human subjects in research delineates the criteria to distinguish institutional research from the common fields of social scientific research. The National Science Foundation’s (NSF) Common Rule establishes criteria to determine when research is subject to the oversight of a federal department or agency, including research conducted by social and behavioral scientists. To administer its policy and regulations. the NSF provides rules for the constitution of Institutional Review Boards, typically at the local institution, empowered “to review and have authority to approve, require modifications in (to secure approval), or disapprove all research activities covered by this policy.” Of direct importance to institutional research, though, are the exemptions to the policy for certain areas of human subjects research, and specifically the first exemption: “Research conducted in established or commonly accepted educational settings, involving normal educational practices….” In effect, institutional research (and assessment) is singled out as an area of human subjects research that is not subject to the authority and oversight of an Institutional Review Board.

Erroneously, the exemption may be construed as an indication that institutional research studies “are not normally deemed ‘research.’” To the contrary, the NSF document records definitions for research and human subjects that clearly include a group of researchers who systematically collect and report statistics on the educational outcomes of living individuals. The U.S. Department of Health and Human Services in its Human Subjects Regulations Decision Charts^[8] affirms the designation in its decision tree. Considering only institutional reporting of educational statistics to the federal government to advance the work of the National Center for Educational Statistics, is institutional research “a systematic investigation designed to develop or contribute to generalizable knowledge?” Yes. Does institutional research “involve obtaining information about living individuals?” Yes. Does institutional research involve intervention or interaction with the individuals? Yes.^[7] Decision: “Activity is research involving human subjects.” Charts 2 and 3 then determine, “Is the research only conducted in established or commonly accepted educational settings, involving normal educational practices?” Yes. Thus, the HHS decision tree does not imply that research studies conducted in a college to study educational outcomes categorically fall short of the designation of research or a system of inquiry.^[9]

The NSF and HHS guidance on the exemption to the federal policy and regulations for human subjects research in practice endows institutional research in the United States with unique responsibilities in a distinct setting for its field of study. The officer responsible for institutional research must adhere, without oversight, to the “ethical principles and guidelines for the protection of humans subjects of research” outlined in the Belmont Report and conduct research in a manner consistent with all six exemptions to the Common Rule of the NSF. By singling out “established or commonly accepted educational settings,” the U.S. guidelines further suggest that educational institutions, e.g., colleges and universities, that engage in “normal educational practices” represent a discrete area of research for a system of inquiry. At the same time, the key words – “setting,” “normal,” and “practices” – signify that institutional research shares interests with the fields of study for institutions and social groups (economics, sociology), customs to signify maturation and social class (anthropology, history), and interventions in human behavior and learning (psychology, education). In a formal respect for a social science, institutional research even at its most basic level – institutional reporting of statistics to the federal government – qualifies as a system of inquiry designed to develop or contribute to generalizable knowledge about human society and relationships in educational settings. The more pertinent question is therefore: how best to institutionalize institutional research with the “special excellence” of a social science?

Structures

One indisputable fact of institutional research is that the vast majority of its practitioners are administrators and professional employees of higher education who are rarely teaching or research faculty. In the United States, as the first professional file of the AIR indicates, the function originated from the inward-looking exercise of self-study – wherein the particular college or university entrusted to the researchers served as its own private object of study (p. 1). Likewise, an early professional file on career development emphasizes that the skills of institutional research are “the skills of self-study” (No. 12, p. 1). Additionally, the institutionalization of institutional research as an administrative function is likely associated with the sheer number, diversity, and dispersion of higher education institutions in the United States – an academic department of institutional research faculty would have been cost-prohibitive and inefficient. Consequently, the administrative organization likely obscured how institutional research could take shape as a discipline guided by the significant questions and theories of a social science. Two of the earliest AIR professional files on career development note that the lack of faculty experience, possession of nontraditional degrees, and inadequate record of publications constrained the career opportunities of IR professionals (no. 12, p. 2; No. 13, p. 5).^[10] In short, the traditional values and attributes of a social science as organized in academic disciplines did not seem to apply readily to institutional research and institutional research as an administrative function developed as if wholly determined by the needs of requesters and the demands of external agencies.

In 2002, the Committee on Scientific Principles for Education Research for the National Research Council [NRC] (U.S.) released a report “to clarify the nature of scientific inquiry in education.”^[11] The introduction suggests that institutional researchers’ struggle to gain legitimacy and stature in higher education administration corresponds with higher education academicians’ efforts to overcome “skepticism concerning the value and validity of developing a ‘science of education.'” (p. 13) Perhaps compounding the plight of administrative institutional research as an unacknowledged form of social science research in educational settings, education researchers themselves split along lines of qualitative vs. quantitative methods and basic vs. applied research, with schools of education favoring basic research and qualitative methods, “often at the expense of quantitative methods…” (19)^[12] To improve the quality of scientific research in education, the NRC report endorses several positions that open the door for administrative institutional research to be recognized as social scientific endeavor, two of which are most relevant. First, the report acknowledges that education research is “an applied field” (p. 83), but that makes it no less scientific than basic research: “What makes research scientific is not the motive for carrying it out, but the manner in which it is carried out” (p.20). Second, the report states that “the research question drives the design, not vice versa,” and specifies three types of interrelated questions: descriptive, cause, and process or mechanism (p.99). To emphasize the second point, the inclusion of questions that produce descriptive questions ostensibly includes the research performed by institutional research professionals in response to the submission of statistics to the federal government to support NCES publications, the Delta Cost Project, and a myriad of publications by academics, researchers, etc., utilizing the resources of the Integrated Postsecondary Education Data System (IPEDS).

To be clear, what this means is that an administrative institutional research office can be institutionalized along social scientific principles 1) by virtue of the manner in which it carries out its work and 2) by the rigor and formalism with which it formulates its questions. In a sense, institutional research was untimely. Increasingly, as the Organisation for Economic Co-operation and Development (OECD) reported ten years ago, “the prominence of the business sector has sharpened” in the funding of academic research and development expenditures internationally,^[13] and university-corporate partnerships in many scientific disciplines surely will not abate in future years. Institutional research, as a contributor to scientific research in education, fundamentally plays the same role in the higher education industry. Every institution directly funds from its non-academic support expenditures the research activity that supports the scholarship of academicians in research universities and the publications of the statisticians employed by the federal government. In effect, institutions of higher education in the United States have underwritten a substantial portion of the costs for scientific research in education in the United States – and, in return, each has received only indirect benefits to the extent that the higher education literature has produced relevant insights for administrators and the federal government has published relevant reports on the local markets. In other words, the partnership between universities’ administrative research and universities’ academic research, or the federal governments’ research, has been quite lopsided.

A third conclusion is thus inferred from the NRC’s report on scientific research in education: institutional research functions can be organized administratively to perform more significant research than the production and submission of statistics to support the descriptive research of the federal government and the basic research of academics with no administrative responsibility within higher education institutions. The NRC report in its final chapter provides “design principles for fostering science in a federal education research agency.”^[14] No doubt, a federal agency “to promote and protect the integrity of scientific research in education with the goal of generating knowledge that can inform policy and practice decisions” requires a legion of higher education institutions prepared to practice institutional research with equal integrity and rigor to achieve the same goals. Therefore, these six design principles provide a framework applicable to the institutionalization of institutional research as a social scientific enterprise in an educational setting (paraphrased for applied research in an administrative office):

1) Staff the office with people skilled in science, leadership, and management
2) Create structures to guide agenda, inform funding decisions, and monitor work
3) Insulate the office from inappropriate political interference
4) Develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to institutional policy and practice
5) Adequately fund the department
6) Invest in research infrastructure

The choice for higher education executives is whether to staff and fund their institutional research offices to support descriptive social science (reporting), causal social science (planning), or process / mechanism social science (effectiveness). The first will permit an institution of higher education to skirt the fines associated with non-reporting of statistics to the federal government or the shame doled out for “misreporting data” to the U.S. News and World Report. The second will provide a modicum of assurance that the institution’s current five year strategic plan will not result in institutional closure or acrimony. But, only the third will enable an institutions to receive U.S. federal grant dollars to advance scientific research in education, to understand how best to fulfill its mission for students, and to become a substantial competitor in the the global market to deliver what works in higher education.

Institutional Research Unbound

Each of the six design principles for scientific research in education factor into the solutions outlined in the h|r portfolio of institutional research: an institutionalized system of inquiry to foster continuous improvement in higher education.

The 50 years since the founding of the Association for Institutional Research has witnessed a manifold increase in the phenomena of an educational setting now accessible to institutional researchers. Student information systems record tens of thousands of transactions and records regarding the normal educational practices of postsecondary students: academic preparedness, institutional fit, socioeconomic background, registration patterns, grade outcomes, behavior issues, group affiliations, absenteeism, counseling, residential life, etc., etc. etc. — all of which are supplemented by i) student survey research on intentions, attitudes, engagement, and outcomes; ii) assessment of student learning outcomes; iii) student tracker outcomes from the National Student Clearinghouse, and iv) etc., etc., etc., of collectible data at the institution. The educational settings of colleges and universities, moreover, are much much more than its students can expect: global providers of higher education, institutions in national systems for higher education, education providers in state and local educational markets, and small communities united by mission, vision, and values. As communities, the educational setting certainly encompasses students, but also the faculty, non-exempt employees, salaried professionals, administrators, alumni, the Board, the immediate community surrounding the institution, and many others with a perceived stake in the outcomes of the institution.

Institutional researchers sit atop a mountain of recorded phenomena in the educational setting of colleges and universities – to which academic researchers and functional unit leaders do not have the same access. In the United States, as well as other nations, the policies for human subjects research have endowed institutions with the power and responsibility to conduct research to determine how best to educate students in higher education. And, yet, the typical structure and operation of an institutional research function in higher education is designed to respond to informal questions (“needs,” “requests,” etc.) from administrators with no required scientific background, no defined agenda, no discernible oversight, no assurance of political interference, and no focused intention beyond the immediate concerns of an ad-hoc project. Institutional research, in this respect, has been too closely bound to self-study and too long burdened by the private exercise of reason for particular institutions. As applied research, institutional research officers obviously must formulate solutions and produce results that inform decision-making, but the conditions under which they design and conduct research matters:

The utilitarian attitude is especially disastrous in studies and research that call for a modicum of inventiveness and originality, and some lucky little discoveries. Discussions, extracurricular reading, a walk with the mind freely wandering, can be much more profitable than the dull compilation of involved points of [data].

If the institutional research future presents a choice between technical service to the “common sense” insights of data studies and institutionalized system of inquiry for the “special excellence” of social science, what more can we say: Dare to know.

NOTE: This post is subject to unrecorded edits until July 31, 2015.