Here's what we have on this site

Home
News
Online Publications
Publications List
JD's Background
Links
Feedback to JD
Search this site

 

SCIENTIFIC AND TECHNICAL INFORMATION TRANSFER IN AEROSPACE:

A SURVEY OF U.S. AEROSPACE SCIENTISTS AND ENGINEERS

 

J.D. EVELAND

THOMAS PINELLI

JOHN KENNEDY

 

WITH

DIANE M. MANUEL

KIMBERLY BISHOP

 

CLAREMONT GRADUATE SCHOOL

CLAREMONT CA

 

10/22/91

 

 

CONTENTS

Background to the Study

Research Questions

The Current Study

Portrait of the User Group

Portrait of Information Sources and their Use

Factors Influencing STI Acquisition and Use

Information Use as a Multidimensional Phenomenon

Conclusions and Further Research

Statistical Note

[NOTE:  Figures and Tables to be added shortly]

 

BACKGROUND TO THE STUDY:  WHY DO WE CARE?

                In recent years there has been an increasing degree of attention given to issues of the generation, management, and use of scientific and technical information[1].  In part this attention can be attributed simply to the ever-increasing volume of data emerging from the nation's laboratories, libraries, and field sites, and to the amount of public and private resources going in to these facilities.  In part, it results from a simultaneously growing uneasy feeling that we are failing to make the best use of this rapidly developing and changing resource.  As we contemplate the increasing disadvantages experienced by American manufacturers in field after field of both low and high technology development, we are forced to ask ourselves why we seem to be unable to leverage our scientific expertise into  equally significant economic payoffs (Griffiths et al., 1991).

                There is no shortage of explanations -- or prescriptions -- to be offered.  Those of an economic bent tend to stress the role of tax incentives, of regulatory influence and uncertainty, and problems associated with capital formation and deployment (references).  Those of a more managerial turn of mind criticize the emphasis in American companies on short-term performance and limited financial planning criteria (e.g., Hayes and Abernathy, 1980).  Some look to political solutions, such as the creation of "enterprise zones" or subsidies for small high-technology businesses.  Others look to the development and expansion of industry/university consortia, either with or without public participation.

                The area of aerospace science and engineering has been of particular interest within this overall context of STI utilization.[2]  For one, it has been on balance one of the more successful fields in meeting current challenges.  As Mowery and Rosenberg (1982) note,

"Judged against almost any criterion of performance -- growth in output, exports, productivity, or innovation -- the U.S. aerospace industry, in particular the commercial aviation sector, must be considered a star performer in the American economy."

The aerospace industry continues to be the leading positive contributor to the U.S. balance of trade, even surpassing agriculture (U.S. Department of Commerce, 1990).  It is a particularly information-intensive industry, leading all other industries in expenditures for R&D -- $24 billion in 1988 -- with an extremely high proportion of that research funding provided directly or indirectly by the Federal government.  This funding is channelled through a wide variety of agencies, including NASA, the Department of Defense, the National Science Foundation, and a host of others.

                The aerospace sector is closely linked to many other scientific and technical areas such as metallurgy, chemicals, computing, and materials science.  Moreover, it deals with highly complex products and a complicated marketing environment in which new sets of international arrangements and competition are increasing the pressures on all phases of the research, development, and production cycle.  In short, aerospace is an area where an exploration of the dynamics of interaction of STI, R&D, and productivity can be particularly informative (National Academy of Engineering, 1985).  If we lose the competitive edge in aerospace, we will be in trouble indeed.

                One common denominator in most analyses of the relationship of science and technology -- and STI in particular -- to economic performance and competitiveness is a perception that the mechanisms for moving information from place to place in the overall system of knowledge generation and application are functioning at something less than an optimal level of efficiency and effectiveness (Bikson et al., 1984; Tornatzky and Fleischer, 1990).  It is clear from a wide variety of studies (Allen, 1970; Hall and Ritchie, 1975; Rubenstein et al., 1971; Smith, 1970) that communication of STI and technical performance are closely linked at both the individual and organizational level.  As Fischer (1980) notes, "...scientific and technical communication is central to the success of the innovation process in general, and the management of R&D activities in particular."

                While there have been no shortage of initiatives aimed at improving these arrangements in the last 25 years, there has never emerged a coherent or defined paradigm of proven accomplishment. Rather, the effect has been the creation of isolated "points of light" that function for a while and then fade into insignificance. However, aerospace science and engineering has had more than its share of successes in this STI dissemination game over the years. Pinelli (1991) has outlined a wide range of efforts, dating back to the creation of the National Advisory Committee for Aeronautics in 1915, to integrate knowledge transfer and utilization into the framework of Federally supported research and development.  From its inception, NASA has had a firm organizational commitment to knowledge sharing and use.  Yet it remains a testimony to the intractibility of the systemic problems of STI dissemination that despite the best efforts of vast numbers of highly capable individuals and the commitment of significant organizational resources to the problems, the issue of effective STI diffusion remains as critical as ever, and nearly as difficult to resolve.

                It is clear that whatever resolutions to these problems that may exist must be found as much in institutional relationships and organizational processes as in the technologies of information or the dynamics of individual information utilization.  As Eveland (1991:3) has noted,

"...the nature of the processes of organizational management of information creates some predictable barriers to effective interpenetration.  These barriers are of several types:

bullet

Structural:  Those barriers posed by organizational arrangements and the need to achieve internal organizational maintenance criteria.

bullet

Cultural: Those barriers posed by the basic frame of reference of the parties involved.  They may involve general cultures (e.g., American firms trying to do business in Japan and encountering unfamiliar practices), or the professional and technical cultures either within or across organizational lines (e.g., manufacturing trying to talk to R&D; university professor talking to industrial researcher; economist talking to anyone else).

bullet

Geographical:  Those barriers posed by separation in space or time; despite the advances of information technology in helping achieve asynchronous communication, the fact remains that it is still a lot easier to share a lot of information with those close by than those farther away.

bullet

Procedural:  Those barriers posed by different ways of defining and conducting operations.  Frequently what this amounts to is a failure to appreciate that words have different meanings in different contexts, and what seems normal and logical to one organization in terms of procedures may not seem equally logical to everyone else.

                The key point here is that all these "barriers" are in fact not physical or even organizational, but cognitive.  That is, they are created by people operating within their own contexts for reasons that make sense within that context.  By the same token, they can by modified or removed by the same cognitive processes that brought them into being.  The critical dimension is not who people are, or even where they are -- it is how they think and feel."

                A convenient model for examining how individual, group, and organizational factors come together to affect STI transfer in aerospace has been developed by Kennedy and Pinelli (1990).  Within this framework (Figure 1), two main components can be distinguished: the informal, relying on interpersonal and collegial networks, and the formal, relying on surrogates, information products, and information intermediaries.  Surrogates include clearinghouses and technical report repositories such as the NASA Scientific and Technical Information Facility (STIF), the Defense Technical Information Center (DTIC), and the National Technical Information Service (NTIS), together with the diverse group of interfaces they have created to make themselves accessible to the user communities.  Information intermediaries include librarians and other technical information specialists who function as "knowledge brokers" or "linking agents" (McGowan and Loveless, 1981; Havelock and Eveland, 1984; Larsen, 1986).

                Over the years, the Federal government has experimented with a vast range of STI dissemination and utilization arrangements combining surrogate and intermediary services in different combinations.        The degree of enthusiasm of the government for each of these approaches has waxed and waned as political fashions have changed.  Williams and Gibson (1990) have outlined an interesting framework that categorizes these initiatives in terms of three models of increasing interactivity:

bullet

Appropriability models:  Under this approach, research knowledge is generally treated as a commodity, to be procured/purchased by a user in accordance with his/her judgments of its utility.  Good research will sell itself; the "better mousetrap" will find utilization.  Surrogates predominate; transfer mechanisms emphasize publication in the research literature and, to a lesser extent, direct interpersonal interaction initiated by the user.  The user bears the primary responsibility for utilization.

bullet

Dissemination models:  Under this approach, technical experts have a responsibility to identify good ideas and bring them to the attention of potential users.  The experts, who may be either the producers of the knowledge or third parties, establish linkages that presumably ensure a continuous flow of ideas.  Intermediaries may play a greater or lesser role; responsibility for use is thus shared between producer and user.

bullet

Knowledge utilization models:  Under this approach, the flow of ideas moves toward bidirectionality.  That is, the user accepts a responsibility to interact creatively with the producer to ensure a more precise targeting of the knowledge to real problems.  Information exchange becomes more of a transaction and less a linear flow.  Both effective and interactive surrogates and highly proactive intermediaries are required, as are informed and fully participative producers and users.[3]

                At present, it is clear that most Federal efforts in the STI area emphasize dissemination strategies, with a substantial residue of appropriability-based programs and processes.  There are as yet relatively few examples of true knowledge utilization strategies, although there is general consensus among the informed STI community that this is the way to go.   Even the better dissemination systems frequently feature "retrofitting" of user requirements onto what are basically passive and product-oriented arrangements (Adam, 1975).  Such systems will have limited success meeting the demand for truly user-oriented information (Bikson et al., 1984).  Moreover, what constitutes effective performance of the intermediary role is as yet only dimly understood (Beyer and Trice, 1982).

                In any event, it is clear that STI issues involve widely diverse populations of knowledge producers, knowledge users, and institutional intermediaries in a complex dance of transactions. These issues will not resolve themselves through inattention or neglect; rather, given the increasing complexity of the information itself and the rapidly changing nature of the information technologies available today and coming down the pike, they stand to become increasingly unmanageable without informed and careful strategizing by policy makers in both public and private sectors.

               

RESEARCH QUESTIONS

                Given what is currently known -- and not known -- about STI utilization dynamics within the aerospace community in particular and across all fields of science and technology generally, we can pose a number of questions that need to be addressed:

  1. Who are the players in the aerospace STI utilization game? What do they do?  What institutional contexts do they operate within?

  2. What patterns of STI acquisition and use appear to operate? Do all aerospace scientists and engineers search for and acquire STI similarly, or are there different patterns that can be distinguished?  If there are patterns, how easy is it to predict who might be expected to show a particular pattern?

  3. How are different kinds of information sources used?  What is the balance of the roles played by surrogates and intermediaries?  Are scientists and engineers influenced in their approach to STI by the same general factors, or are there distinct patterns? 

  4. Are there patterns in the use of the new electronic information technologies?  Are there any differences in how electronic information technologies are used as opposed to more traditional sources? 

  5. What are the "dynamics of STI utilization"?  Are there distinct patterns of STI use?  If so, can we predict who uses what kinds of information, and for what?

  6. To what degree is expense a significant barrier to access and use of STI?  Would interventions aimed at changing the costing of STI affect utilization patterns?

  7. What are the implications of any patterns that might be uncovered in STI acquisition and use for the development of Federal and non-Federal strategies for more effective STI dissemination and application?

 

THE CURRENT STUDY

                A significant part of the difficulty in evolving new approaches to STI use, particularly in the aerospace area, has been a shortage of basic information on just what STI really is, and how it is accessed and used.  In 1987,  NASA and DOD launched the NASA/DOD Aerospace Knowledge Diffusion Research Project, aimed at systematically investigating how the results of NASA and DOD research find homes in the wider aerospace R&D process (Kennedy and Pinelli, 1990). 

                The Project was originally planned in four phases.  Phase 1 was a study of the information-seeking habits of U.S. aerospace scientists and engineers.  Phase 2 was to be concerned primarily with the transfer of STI among industry and government, and particularly with the role of formal information intermediaries such as librarians and technical information specialists in that process.  Phase 3 was to look at the use and transfer of STI within the academic community, while Phase 4 was to examine the international dimensions of the STI dissemination process.  This report describes the results from Phase 1 of the overall project.

The Research Samples

                In 1989, three samples of aerospace scientists and engineers were drawn to participate in Phase 1.  The sample was drawn from the membership lists of the American Institute of Aeronautics and Astronautics (AIAA).  Of the nearly 20,000 AIAA members, 6,741 were selected as initial respondents.  Three separate questionnaires were prepared:

bullet

NASA1, the major instrument, focused on four main categories of information sources:  conference and meeting papers, journal articles, in-house technical reports, and Government technical reports generally.  It also asked about how STI was used in the resolution of technical problems, and about some barriers to and facilitators of access to STI generally.  The final sample numbered 3,298, with 2,016 usable responses (an adjusted rate of 66.2%).[4]

bullet

NASA2 was a shorter questionnaire aimed specifically at a set of particular information sources:  technical translations, AGARD reports, DoD technical reports, and NASA technical reports.  Questions emphasized judgments about various qualities of the reports and barriers to their access and use. The sample numbered 1,735, with 977 (an adjusted rate of 67.1%) responding.

bullet

NASA3 sought information specifically about a number of STI bibliographies and databases:  STAR, NASA SP-7037, CAB, GRA&I, RECON, DROLS, and NTIS File.[5]  Again, questions emphasized judgments about the reports and accessibility issues.  The sample size was 1,705, with 955 responses (an adjusted rate of 69.0%)

                Each questionnaire also included some basic demographic information about the respondents (education, training, organization, tenure, AIAA specialization, and professional duties.

Analysis and Interpretation

                The data were originally received and tabulated by Indiana University's Survey Research Center.  Several preliminary reports reporting basic frequencies and some initial analyses have been issued to date by IU and by NASA (see References for the complete list).  This report results from further analysis undertaken at the Claremont Graduate School, in collaboration with NASA and IU.  Its aim is not to duplicate previous reports, but rather to supplement and extend their approach.

 

PORTRAIT OF THE USER GROUP

                As noted, all three Phase 1 surveys included questions on basic demographic information about the respondents.  Appendix B, Questions 56 to 64, summarize these data.[6]  

                Respondents were generally well educated, with over 70% having some graduate education and over 30% doctorates.[7]  83% reported being originally trained as engineers; 12% reported training as scientists.[8]  By contrast, only 67% reported currently working as engineers, with 9% scientists and 23% other positions.  Of those trained as engineers, 67% continued to work as engineers, with 3% converting to scientists and the rest to other occupations (presumably largely management positions).  Of those trained as scientists, only 46% continue to work as scientists, with 33% converting to engineers and the rest to other specialties.

                Respondents are also relatively senior as a group, with 44% reporting more than 25 years with their present organization and another 33% reporting between 10 and 25 years.  About 13% work in academic institutions, with 10% each in NASA and DoD facilities. 52% of the total work in industry; the rest are scattered in other categories.  In terms of professional duties, about 10% teach and 16% do research; 31% do technical management, 6% do administrative management, and 28% do design/development.  Regarding specialization, 35% report working in aeronautics/astronautics and another 40% report working in general engineering; the rest are broadly scattered.  85% report that the Federal government funds all or part of their work, with 75% indicating that the government is the single largest supporter of their work.

                These characteristics interact in interesting ways.[9] Industrial employees tend to have been employed longer; employees of academic and research organizations tend to have shorter tenures. Scientists tend to be shorter in tenure; engineers, longer.  Scientists are more likely to have graduate degrees than are engineers.  Those carrying out education and research duties are more likely to have graduate degrees, while technical managers are less likely to have them.  Some non-relationships are also interesting.  For example, there is no relationship between scientist/engineer training and the kind of duties being performed, or between training and the kinds of organizations worked for.

                While there were few one-to-one matches between the demographic variables, it is clear that there are some notably different patterns present in the data, most importantly the distinction between (at the one end) younger academic scientists and, at the other, older technical and managerial engineers. Accordingly, an index was constructed in which points in one direction were given for being a scientist, employed a shorter time, in an academic or research organization, and doing academic or research tasks.  Points in the other direction were given for being an engineer, employed longer, and working in management or industry.  Figure 2 shows the distribution of this index.  The interesting point is that there is quite a distribution of values, with relatively fewer people getting extreme scores in either direction (more in the older/engineer direction).  Below, we will find this index interestingly related to a number of outcome dimensions.

                The NASA1 questionnaire included a series of questions relating to information gathering behavior in relation to a recent "technical problem, project, or task" completed by the respondent. Respondents were asked to rank in order the steps they went through to gather information relative to it.  The nine possible information gathering modes fell into three general classes:

bullet

 Formal data sources:  Searching databases, consulting library sources, etc.

bullet

Information intermediary sources:  Consulting with librarians and technical information specialists

bullet

Informal network sources:  Consultation with colleagues, supervisors, "gatekeepers", etc.

                Ranking nine steps is inherently a rather difficult task, and it is hard to be fully confident in the rankings at later stages. However, it is probably that at least the first few steps would be recalled with some precision.  Accordingly, respondents were grouped in terms of whether they employed data, information specialist, or network sources at the first and second steps of the process.[10]  Table 1 gives the frequencies for these usage patterns. In the analyses that follow, these groups will be referred to as "Data-to-Data" (D/D), "Data-to-Network" (D/N), "Network-to-data" (N/D), and "Network-to-Network" (N/N) depending on which source they took first and which they subsequently resorted to.[11]

                There are some interesting differences between the four groups in terms of composition.  The scientists in the sample are disproportionally D/D's (33% vs 24% across the sample) and low in N/N's; however, the engineers distribute themselves across all four groups more or less in proportion.  Perhaps as a consequence of this concentration, the D/D's tend to be better educated (an average of one more degree than the other groups), and to be employed in education and research; the N/N group is disproportionately higher among the administrators. 

 

PORTRAIT OF INFORMATION SOURCES AND THEIR USE

                Across the three questionnaires, reference was made to a highly varied group of information sources.  NASA1 included questions about papers, articles and technical reports generally, and also contained questions regarding the use of libraries (including barriers to library use) and information technologies. There were also a series of questions relating to specific ways of using government technical reports.  NASA2 included questions about a range of specific named technical reports, while NASA3 asked about databases and bibliographies, some of which are printed and some of which are on-line.  In this section, we report some issues relating to these different sources and how they are received.

Library Services

                There were no significant differences among the groups in terms of the availability of library resources (over 90% across the board), although in absolute terms over 100 industry respondents indicated that they did not have such access, while only 30 of the academic and government respondents so indicated.  In general, it seems clear that access to basic library services is not a significant problem.

                Respondents were asked about the relative frequency with which they interacted with their library in seven specific ways ranging from simple visits to requesting various types of information. These frequencies ranged from zero to greater than 100 (Table 2), strongly skewed toward the low end of the scale.  There were no differences between scientists and engineers in terms of how often they visited the library, although academics and those working in academic organizations did tend to have about half again the overall level of use (averaging over 30 visits in the last six month).

                These seven items were combined together to obtain a profile of how respondents used different services at libraries and technical centers, two groups of behaviors were identified.[12]  The first group comprised use of the library in an active fashion, with the assistance of library staff.  They included physically visiting the library or technical center, requesting assistance from a staff member, or being offered help from an employee at the library.           The second group of behaviors were much more passive.  They included making requests in writing, electronically, by telephone, through a proxy, or having the library request something.

                While there were few differences in the relative frequency of these library behaviors attributable to any of the main demographic variables, there were interesting differences that seemed to be a function of the information-gathering style characteristic described earlier.  Data-to-data people (D/D's) were, not surprisingly, more likely to have visited the library in the past six months (95% vs. 80% for network-to network people (N/N's), and to have sought help from a specialist (85% vs 67%).  On average, D/D's rated the library one full point higher on a 5-point scale of importance than did N/N's; 63% rated it as "very important", as opposed to only 25% of N/N's.

                The four groups were compared in terms of scores representing each of the two general types of library behavior (active or passive).  It appears that D/D people are rather different in their use of library resources from members of the other three groups. D-D people were more likely to use both approaches -- not only a variety of resources in the data retrieval process, but assistance from people and use other traditional search techniques.

                Respondents were also asked about why they might NOT have used their library or technical center.  A factor analysis of these reasons identified four broad categories from the eleven specific reasons queried, as follows: 

  1. The library is of no help (i.e. not cooperative, does not understand my needs, or does not have the information I need);

  2. Administrative barriers (i.e. the library is too slow in acquiring the information I need, I have to pay to use the library, we are discouraged from using the library); 

  3. Other sources (I have my own personal library, the library is too far away, my information needs were met elsewhere);

  4. No need (I had no information needs). 

                 Again, none of the main demographic factors is related to library use or non-use.  However, information-gathering style does seem to matter; D/D's tended to react most strongly to administrative barriers, while N/N's tended simply to have no information needs or to have other sources of information. 

                In general, then, libraries are widely available sources of STI, and function broadly across the spectrum of users.  Those who are already prone to look for solutions in formal data sources tend to use these resources more heavily than those who prefer informal networks.  But in general there seem to be few institutional or structural barriers to library use revealed in these data.

Information Technologies

                Respondents were asked about their degree of use or non-use of fifteen specific information technologies (See Appendix B, Questions 47a to 47o).  These technologies appear to comprise four distinct categories:[13]

  1. electronic sources (data bases, CD-ROM, electronic networks, electronic bulletin boards, and electronic mail); 

  2.  film or audio media (micrographics/ microfilms, video tape, motion picture film, and audio tapes/cassettes);

  3.  communication technologies (teleconferencing, videoconferencing, and fax/telex);

  4. computer sources (computer cassette/cartridge tapes, floppy disks, and desk-top/electronic publishing).                                          

Individuals were given scores reflecting their degree of use of each of these four categories of sources, and analysis of variance was used to examine relationships between patterns of information technology use and individual descriptors.[14]  Tables 3-5 show the relationships of these different types of electronic technology use to the demographic variables noted below.

                For the use of electronic sources, organization type appears to affect the level;  individuals employed in academic organizations are high users of electronic sources, whereas individuals employed in industrial settings are low users.              

              The type of problem or task identified as focal in the past six months also produced a significant effect: electronic sources are most likely to be used by those working on research related tasks and least likely to be used by individuals engaged in educational, design/development, or production type tasks.  In addition, electronic sources are more likely to be used by scientists than by engineers, and by D/D information gatherers.

                For film/audio sources, organization type again was significant; those respondents employed in academic or not-for-profit organizations are high users of this information source, whereas incumbents of government or industry are low users.  These sources are more frequently used by engineers as compared to scientists, and by D/D individuals.

                For communication sources, organization type again mattered. People at academic organizations are high users of this source, as compared to those employed in government, industry, or not-for-profit organizations.  Further, individuals who hold a graduate degree are higher users of communication sources relative to those who do not possess a graduate degree.  Problem type is also significant; people working on research or educational related tasks utilize communication sources more frequently than those individuals working on design/development tasks or manufacturing/production tasks.  N/N individuals have a higher level of use of these tools as well.

                For computer sources, individuals employed in academic organizations reported high use relative to employees in industrial settings.  In addition, individuals employed in academic organizations are higher users of this source than are people in government organizations.  A significant effect is also noted for problem type.  Individuals working on research related tasks are higher users of computer sources than are those engaged in manufacturing/production work.

                The fifteen information technology variables were additionally put through a multidimensional scaling procedure in a search for underlying patterns of relationships among them.  The result of this procedure is a sort of "map" in which the positions of the technologies reflect their similarity or dissimilarity in use, and basic underlying dimensions can be identified.[15]  The result is Figure 3.  On this "map", the two basic dimensions appear to be (1) accessibility, with fax and floppy discs highest and film lowest; and (2) interactivity, with teleconferencing highest and micrographics and tape lowest.  The map suggests that there may be a degree of substitutability among different kinds of information technologies, a point suggested earlier by the factor analysis. 

 

FACTORS INFLUENCING STI ACQUISITION AND USE

Use of Specific Media

                The NASA1 questionnaire contained parallel sets of questions relating to factors influencing the use of four different STI sources (papers, journals, in-house reports, and government reports).  Seven dimensions were identified for each source: accessibility, ease of use, expense, familiarity, quality, comprehensiveness, and relevance. 

                These 28 items were subjected to both factor analysis and cluster analysis, with the same general patterns emerging from each technique.[16]  Six overall indices emerged:

bullet

Quality, comprehensiveness, and relevance of papers and journals;

bullet

Quality, comprehensiveness, and relevance of in-house and government technical reports;

bullet

Accessibility and ease of use of papers, journals, and government reports;

bullet

Accessibility and ease of use of in-house reports;

bullet

Familiarity, across all four sources;

bullet

Expense, across all four sources.

Figure 4 shows the derivation of the six clusters which underlie the indices.  Each of these indices had a Cronbach's alpha of between .71 and .86.  Accordingly, all were retained for analysis (see Table 6 for descriptive measures on these indices).

                These indices can be interpreted as representing six different sets of factors influencing STI use.  They are interrelated, but not for the most part heavily (Table 7); the highest intercorrelation is r=.52, between the two quality indices.   It is reasonable to suppose that scientists and engineers in different situations might be expected to be influenced by different factors.

             Overall, relatively few differences were found, although those that do emerge are interesting.[17]  Those without graduate degrees are more influenced by the quality of in-house reports than are those with graduate degrees.  Scientists are in general more influenced by familiarity than are engineers; engineers tend to be more influenced by the quality of in-house reports than are scientists, as are those working in government organizations.  Those working in academic institutions are more influenced by the quality of papers and journals than are those in other organizations.  Those whose duties involve education tend to be more influenced by expense across the board; those doing research duties tend to be less influenced by expense.  Those doing technical management tend to be more influenced by accessibility and ease of use.

                While the first five indices tend to be generally normally distributed, the expense factor is unique in having a surprisingly large number of respondents (216, or about 20% of those answering these items) totally uninfluenced by the item.  This group contrasts with the others in the sample in interesting ways.  They tend to be:

bullet

involved in research problems (none are in education)

bullet

somewhat lower in average tenure, and somewhat more heavily degreed

bullet

scientists rather than engineers

bullet

in government rather than other organizations (none are in industry).

So this group to whom expense issues are irrelevant appears to be composed largely of young research scientists in government installations.[18]  Aside from this group, the importance of expense -- positive or negative -- is largely unrelated to situational factors.

Specific STI Sources

                While NASA1 was addressed to broad issues of STI access and use, NASA2 and NASA3 took up the issues of how respondents reacted to specific characteristics about the access, use and barriers of various particular reports and information sources.  In this section, we report some conclusions from these data.

                Reports considered in NASA2 included Technical Translations, AGARD Technical Reports, DOD Technical Reports and NASA Technical Reports.    Technical Translations were primarily used for research and, to a lesser degree, to investigate management issues (Appendix c, Question 3).  The barriers to the use of Technical Translations can generally be categorized into three areas: (1) Inaccuracy based on either language or technical issues (which was not a concern); (2) Availability and timeliness; and (3) Relevancy of the information.                  The technical accuracy and language of Technical Translations did not seem to be a problem, with greater than 80% of the respondents indicating no problem with the reliability of the reports.  The information comprising Technical Translations was felt to be current, but about 50% of the respondents indicated that it takes too long to get the report (Appendix B, Question 3).  The predominant issue with non-use of Technical Translations is that the information was not relevant to an individual's discipline or research; at least 50% indicated that this was their reason for not using this report.  In fact, relevancy and accessibility are the major characteristics influencing the non-use of Technical Translations.

                AGARD Technical Reports are also primarily used for research and management.  Of those persons indicating that they did not use AGARD, 50% cited the lack of availability (Appendix B, Question 5). Respondents also indicated that AGARD is not relevant to their research or discipline.  By contrast, almost 100% indicated that AGARD reports were reliable and accurate.  Based on the grouping of questions regarding the barriers to the use of AGARD, two basic categories emerged:  quality and relevancy (intercorrelations are provided in Table 8).  Quality included such issues as timeliness of the information and relevancy encompassed the use of the information in the respondents' disciplines.

                The awareness by respondents of AGARD reports was the result of four different types of techniques for information searches. Individuals usually discovered AGARD through (1) deliberate searches; (2) through passive investigations (i.e. a referral from a colleague); (3) through public announcements; and (4) through an information intermediary (i.e. library).  Getting physical access to AGARD usually was the result of direct means (i.e. someone sent the report) or through an indirect manner (i.e. AGARD is requested from an intermediary).  The latter method was the most popular, with many of the respondents indicating that they received AGARD from the library or from a colleague (Appendix B, Question 6).  The overall rating for AGARD was fair, with the quality of the report being the major characteristic influencing use (Figure 5)[19]. Accessibility and expense did not provide much explanation regarding the use of AGARD.

                Department of Defense (DOD) Technical Reports were used by many more of the respondents than were Technical Translations and AGARD and primarily for the same purposes:  research and management (Appendix B, Question 10).  Again, quality and relevancy were the major issues regarding the use of the reports (Table 9).  Unlike AGARD reports, respondents that indicated that they did not use DOD reports because of a lack of relevancy to their area of research; availability was not an issue.  Individuals seemed to be aware of the publication but have no need to use it.  Availability was also associated with timeliness; if the report is not timely, it was not considered available.

                The respondents were split regarding the availability of DOD reports.  70% indicated that DOD reports were not used in their area of research, but 60% responded that DOD reports were used in their specific discipline (Appendix B, Question 10).  These number seem to indicate that people are aware of DOD reports and used them when they could provide assistance.  The question of concern is to what degree the information in these reports were actually of use to the respondents.

                As with AGARD reports, respondents found out about DOD reports through deliberate searches, passive investigations, public announcements and information intermediaries (Appendix B, Question 11).  More specifically, respondents learned of DOD reports through a citation in a report or journal or from a colleague.  After they discovered the existence of the report, respondents were most likely to request a copy from a information intermediary, such as the library or NTIS.  The primary factors influencing the use of DOD reports were their accessibility, familiarity, and relevancy. To a great extent, it seems that DOD reports were quite popular and are quite handy to use and can easily be obtained from one's local library.  A composite scale for the overall rating of DOD reports indicated that it was rated above average (Figure 6).

                NASA Technical Reports (NTR) were used by more respondents than any of the other technical reports.  As with DOD reports, the major barrier to usage of NTRs was the degree of relevancy of the document.  Though 70% of the respondents indicated that NTRs were relevant to their research, only about half indicated that the report was used in their discipline (Appendix B, Question 15). 

                The source of NTRs were the same four as found for AGARD and DOD reports, with differences being in the sources from which the report was received.  People most often learned of NTRs through their colleagues, either directly through a referral or through a citation in a journal or other written document (Appendix B, Question 16).  As with the retrieval process for AGARD reports, respondents were most likely to request a copy of an NTR from a library.  The second most popular means of retrieval was from a colleague. 

                The overall rating of the quality of NTRs was only fair (Figure 7). To predict the overall ratings of NTRs, the quality, accessibility, and expense of the reports were analyzed.  Both quality and accessibility were found to be significant predictors of the overall ratings for NTRs.  As with the other reports, the quality of the report, along with its accessibility were strong determinants of the overall rating of the report (Table 10).

                Across all these sets of reports, some consistent findings emerge.  Though some reports were used more than others (such as DOD and NASA Technical Reports), the barriers to use of all reports seemed to be based on the significance of the document to a specific discipline.  If the document was viewed as irrelevant to a concentration or specialty, it received little use.  In fact, many respondents recognized the availability of these reports but did not use them because of the lack of a connection with a specific area of interest.  Additionally, the quality and accessibility of the reports were major points influencing the use of any of the documents, with quality being the driving force.

                Another interesting point was the lack of importance of expense to any of the respondents.  One explanation for this may be the fact that there may be no costs for the reports or that the costs were hidden because they were requested from an intermediary, such as the library or a colleague. 

                In order to test the commonality of factors influencing use across all the four types of reports, a common cluster analysis was undertaken on the barrier factors for all reports, a procedure similar to that employed above to create the general "influence type" variables.[20]  With the NASA2 data, the analysis was not successful; it appears that each type of report is unique in terms of what factors influence its use.  The only factors that clustered together regardless of the type of report were the expense variables (as in the more general case). Again, this may be a result of expense issues not being a major concern for the respondents; when they needed to get information it was neither difficult nor expensive to get.

                The NASA3 questionnaire asked similar questions about a range of databases and bibliographic resources for STI.  Despite the relatively large sample size, this questionnaire produced relatively little useful information.  The main problem is the generally low level of use of any of these sources (see Appendix D, Questions 1,7,13,19,25,31, and 37).  The most used source is STAR, which is only used even "sometimes" by less than 15% of respondents.  The others are significantly less used.  Accordingly, extremely small N's and the probability that those who did answer were significantly different from those who did not makes analysis of these data very problematic.

                One set of tests that did prove interesting was a cluster analysis of factors affecting use, similar to that performed with the NASA2 sources.  Results here showed both similarities and differences with the other sources and within this group as well. For STAR, SP-7037, RECON, DROLS, and NTIS File, the factor grouping associated accessibility/ease/familiarity, quality/comprehensiveness/reliability,  and expense.  For CAB, the grouping was accessibility/ease/familiarity, comprehensiveness/relevance, quality, and expense.  For GRA&I, the association was accessibility/ease/familiarity, quality/comprehensiveness, relevance, and expense.

                How much of these differences are real and how much are statistical artifacts remains to be seen.  It does seem that the factors of accessibility, ease of use, and familiarity are generally associated with one another across lots of different media, and that consideration of expense is generally not related to other considerations.  Beyond that, it is hard to speculate.

Summary

                In this section, we have reviewed a wide range of findings regarding STI sources and how their use is related to background and situational factors.  In general, it is difficult to make unequivocal generalizations about what kinds of people use what kinds of information, and why.  There are complex interactions that the NASA questionnaire data only partially tap, and the range of variance on almost any dimension is significantly large.  It appears that we are dealing with both an extremely diverse clientele and a diverse population of sources linked in shifting and subtle ways.

 

INFORMATION UTILIZATION AS A MULTIDIMENSIONAL PHENOMENON

                The key outcomes for which this study sought to account are various dimensions of the use of scientific and technical information by aerospace scientists and engineers.  In the NASA1 questionnaire, the issue of utilization was approached from several different angles.  In this section, we review the different measures employed and construct some aggregated variables reflecting underlying dimensions of use.

The Overall Measures

                Question 1 of NASA1 was a simple yes/no inquiry asking if each of four types of STI were used by respondents in their professional duties:  (a) conference/meeting papers, (b) journal articles, (c) in-house technical reports, and (d) government technical reports. The proportions answering positively were between 85% and 87% for all four types of STI sources (Appendix B, Questions 1c to 1g). Interestingly, however, the intercorrelations among these answers are quite low (Table 11); the highest relationship, that between use of conference papers and journal articles, is only r=.45. 

                Question 3 was similar in format, but asked instead about numbers of times each source was used in the last six months. Answers ranged from "1" to "over 300", and in general are strongly skewed toward the lower ends of the scales.  While the intercorrelations of these variables are somewhat higher than those for the "yes/no" measures, they are still generally low, with the highest, again being between papers and articles, being r=.74 (Table 12). 

                These findings suggest that none of these overall variables is necessarily the best indicator of utilization.  The highly skewed distributions make normal inferential statistics suspect.  In addition, the low intercorrelations among the different information source usage rates suggest that use of the different sources may represent rather different underlying phenomena.  However, in the interests of overall indicators of STI use, summative indices were constructed across both Questions 1 and 3.  Cronbach's alpha for the Question 1 index was only .57; accordingly, this index was dropped from further consideration.  Alpha for the Question 3 index was .83, an acceptable level, and this index is used below as the indicator of "overall STI use".

The Percentage Measures

                Questions 37 to 41 of the NASA1 questionnaire asked a series of questions about the amounts of five different types of information gained from different sources in the past six months: (a) basic science and technology, (b) in-house technical data, (c) computer programs, (d) technical specifications, and (e) product and performance characteristics.  The sources were, again, papers, articles, in-house reports, and government reports. 

                These 20 answers (five kinds of information by four sources) were subjected to a principal components factor analysis.[21]  Five factors were identified, accounting for 74% of the variance of the original data (Table 13 presents the factor loadings).  Four of these factors represented the four sources across four of the five issue areas (a,b,d, and e).  The fifth factor represented all four sources in relation to the issue area of computer programs.  The rotated factor loadings are quite high and quite distinctive, suggesting a strong and stable factor structure reflecting distinctively different patterns of STI acquisition.

                Accordingly, five indices based on these factors were constructed.[22]  Cronbach alphas for these indices ranged from .79 to .85, confirming the validity of the index construction process. These indices correlate with the overall indices reflecting the same sources in varying measure (Table 14), ranging about r=.7 for the overall use measures; correlations with amount of use are much lower (only the journal measure is significant).  Intercorrelations among these indices of amount of information utilization range from r=.56 for journals and papers, to r=.19 for programs and papers. Some relationships are worth noting.  Utilization of papers and articles is related strongly, as are use of the two types of technical reports.  Use of programs is strongly related only to use of in-house reports.  Based on both the factor analysis and the relationship patterns among the indices, there is ample evidence that the five indices reflect distinctly different kinds of STI utilization, and should be analyzed separately.

Utilization and Situational Factors

                As noted earlier, the NASA1 questionnaire contained five major personal descriptors: (a) level of education; (b) type of education (scientist vs. engineer); (c) years of experience; (d) type of organization; and (e) primary professional duty.  Each of these variables was tested against the five types of utilization, using a oneway analysis of variance, to determine the degree to which background factors alone might account for differential STI use.[23]          

               In general, relatively few relationships were found (Tables 16 to 19).  Level of education was related to use of journals and papers (those with graduate degrees reported higher use), but not to the use of technical reports or programs.  Type of educational preparation was unrelated to any of the specific types of use, although scientists did report somewhat higher overall levels of use across types than did engineers.  Those reporting higher levels of technical report use (of both types) tended to have somewhat longer tenure in their organizations; however, this did not hold for use of papers, journals, or programs.  The only kind of organization that appeared to matter is the university.  Those in academic institutions tended to report higher levels of journal use and lower levels of in-house report use; they also tended to be higher overall users.  Type of professional duty was largely unrelated to utilization; the only exception was the relatively lower rate of in-house report use by those in education.

                A nondemographic but equally situational factor is represented in the question asking for a description and a categorization of a major technical problem or task the respondent worked on the last six months.  The original categories were collapsed into 5 major categories:  (a) education, (b) research, (c) design/development, (d) manufacturing/production, and (e) other.  Analysis of variance (see Table 20 for the means) using these categories as predictors showed that:

bullet

those involved in education and research problems tended to have higher levels of utilization of papers and journals, while those involved in production tended to have low levels for these sources

bullet

those involved in research tended to have higher levels of use of government reports, while those involved in development tend to have higher levels of use of in-house reports;

bullet

those involved in research problems tended to have the highest overall levels of STI use, while those involved in production problems tended to have the lowest overall levels.

No other differences were statistically significant.

Utilization and Influence

                The intercorrelations between the influence factors and the utilization measures show that rather different influences appear to be associated with different kinds of use (Table 21).  Use of conference papers and journals is positively associated with emphasis on quality, comprehensiveness, and relevance of papers and journals, but not with accessibility/ease of use or familiarity. For use of in-house reports, what seems to matter are emphasis on quality and on accessibility/ease of such reports.  Use of government reports is associated with emphasis on familiarity and accessibility/ease.  There is no association in these data between emphasis on expense and use of any of the different kinds of information sources.  None of the influence categories appears to strongly predict use of programs.

                As relatively sparse as they are, these findings can be combined with our previously reported findings on demographic interrelationships to suggest a pattern of differential STI use. What we appear to have is a situation where those more toward the basic science end of the R&D continuum tend to rely more on papers and journals, and to be more impressed by traditional scientific criteria, while those more toward the development end tend to rely more on technical reports, and to be more impressed by factors such as ease of use and accessibility.

Utilization and Other Barriers

    The degrees of utilization of the four written information sources might be expected to be related to other variables described earlier, particularly barriers to library use and the utilization of other information technologies.  Interestingly, the library barrier factors show no relationship to utilization for any of the different media.  However, several significant relationships emerged for the criteria pertaining use of other information technologies (Table 22).        

                For journal articles, high users are also high users of electronic sources (r=.10), film or audio sources, (r=.09), and communication sources (r=.14).  High users of conference papers are also high users of film or audio media (r=.22) as well as high users of communication sources (r=.09).   High users of in-house technical reports are low users of film or audio media (r=-.13) as well as low users of communication sources (r=-.15).  High users of Government technical reports are also low users of communication sources (r=-.12).  High users of programs are also high users of electronic sources (r=.07) and low users of communication sources (r-.06). 

                In Section 2, we described an overall index that we constructed from several of the demographic variables to reflect the degree of research/basic science vs. engineering/production emphasis on the part of individuals.  This index was correlated with both the utilization and the influence variables (Table 23). The only significant relationship with utilization factors is as might be predicted; individuals who score heavily on the basic science end of the index tend to be low users of in-house reports, though not necessarily higher users of anything else.  For the influence factors, high-basic science types tend to be negatively influenced by all the factors relating to in-house and government reports, and, interestingly, by familiarity generally.  They are positively influenced by accessibility/ease of papers and journals, and unaffected by journal quality or expense.

                The NASA1 questionnaire also asked some questions specifically about Government technical reports used in completion of the project or task identified earlier as focal.  63% reported that they were used; there appear to be no relationships between this usage and overall sample demographic factors.  However, those who reported such usage did tend to be more influenced by most of the factors dealing with reports -- that is, their quality, accessibility and ease, and familiarity (though not their expense).

                There has been interesting speculation in the literature (e.g., Rogers, 1984) that use and kinds of information may vary depending upon when in the R&D process it is used.  Respondents were asked to identify whether the Government reports were used at the beginning, middle, or end of the focal project, or throughout its conduct (Appendix B, Questions 53a to 53d).  82% reported use throughout; of those who indicated predominant use at one or more specific stages, 69% indicated primary use at the beginning, with progressively less use as the project proceeded (Table 24).  There are no significant relationships between stage of predominant use and any of the demographic or situational variables, including problem type.  There is a relatively weak relationship with the satisfaction items (Table 24); those who use at two stages rather than just one tend to be more satisfied with both effectiveness and efficiency, and those who use only at the end tend to be the least satisfied.  The directionality of these influences cannot be estimated from these data.  Within this group and the context of this question, there thus appears to be little support for a hypothesis of differential usage depending on stage.

                None of these relationships is enormously large, but taken together the picture is consistent with what we have noted earlier; that is, one group of individuals in the research community who are heavily affected by traditional academic media, and another more heavily impacted by technical reports.  In between these extremes, there is a vast spectrum of individuals who combine these approaches and interests in various ways.

 

CONCLUSIONS AND FURTHER RESEARCH

Findings from the Study

                At the beginning of this report, we posed a series of specific questions relating to STI acquisition and use.  In varying measure, these questions have been resolved:

  1. The players in the STI game differ widely in terms of specialty, tenure, organizational context, and training. There are no clearly defined subgroups who share consistent and exclusive criteria of membership.  Actually, there is a broad continuum on most descriptive measures such that knowing any given piece or two of information about an individual is of little help in predicting any other data. This implies that we are not dealing with an easily segmented market for STI, but rather a highly diffuse and uncoordinated one.

  2. Although we cannot easily categorize individuals, it does appear that there are distinctly different patterns of STI acquisition and use practiced by different people.  Some of these are associated particularly with specific groups; some are not.  Different factors influence the use of media such as conference papers and journals than are typically applied to evaluating technical reports.  However, it does appear that relevance to one's discipline and research/engineering interests is a relatively constant influence.

  3. Different kinds of information sources are used differently.  There are a significant number of individuals who tend to rely more on their interpersonal networks than on formal data sources for access to STI.  These networks appear to be largely outside the structure of surrogates and intermediaries discussed earlier.  These patterns of data acquisition are largely independent of personal characteristics.

  4. Electronic information technologies fall into distinguishable classes, varying in terms of the general dimensions of accessibility and interactivity.  Different kinds of sources tend to be used rather differently by different groups, although in general access to such technologies is less than for traditional media.  The greater degree of use of these tools by academic researchers is worthy of note; the data do not allow us to infer at this point whether lower levels of industry use result from lack of accessibility, inutility, or simply indifference or unfamiliarity.  It may well be that further attention to targeting such technologies at non-academic audiences might be warranted.

  5. There are distinctly different patterns of STI use that revolve primarily around media rather than purposes.  That is, people tend to go to particular media for a range of STI rather than to go to different media for different kinds of information.  These patterns of use are only weakly related to individual characteristics.  They may, however, be a function of professional discipline and specialization, although this is hard to determine with precision from these data.

  6. Expense appears to be a very minor issue, often no issue at all.  While initially somewhat surprising, this finding makes excellent sense in retrospect. The key is to think of organizational contexts in relation to usage patterns. Academic researchers have access to university libraries with large stocks of their preferred media -- journals and papers.  Government scientists and engineers likewise have wide access to technical reports through their library systems.  Industry personnel have somewhat less library access, but their preferred media are in fact in-house reports, where presumably cost is not accounted for.  While there is no way from these data to infer a causal order to cost/access, media preferences and organizational location, it is clear that financial consideration would be at best a rather minor lever for future policy. 

  7. The general implication of these findings is that the original model with which we started (Figure 1) is accurate as far as its representation of the interactions of individual, context, and sources, but perhaps underemphasizes just how complex these interactions are and how many different kinds of transactions are represented within them.  This in turn suggests the need for a broad range of STI strategies capable of coping with the equally broad range of challenges they face.

 

Long-Term Issues

                We have suggested earlier in this paper that there is potentially an enormous range of arrangements that might be considered for knowledge transfer systems, varying along several different dimensions.  Some are susceptible to formal creation and management; others are simply a matter of not getting in the way of something that is working.  All have met some needs at some places and some times, for some people.

                We began this report by noting the vital importance of the aerospace science and engineering community to U.S. competitiveness and economic viability in a complex and changing world of science and technology.  In this context, one of the major contributions of this survey is to highlight that this "community" is in fact an extremely diverse and largely amorphously structured group, with many different kinds of needs and work patterns.  What serves one component of this community well may do nothing for another, and might actually impede still others.  We must recognize that this overall enterprise will not be shifted one way or another in lock step.  Rather, locally targeted solutions, designed within the context of the overall situation, are to be preferred to large- scale exercises.

                From these data, there can be no way of inferring what strategy is "best"; indeed, given the fact that the respondents were all presumably well qualified professionals, the data tend to call into serious question the idea that any one model or set of surrogate/intermediary relationships  might meet the needs of more than a distinct minority of possible users.  Thus, we have empirical reinforcement for the idea of the value of diversity in knowledge transfer strategies.

                One point that does emerge loudly and clearly from these data is that the traditional strategy of essentially passive information distribution through formal surrogate channels -- under an appropriability or even a dissemination model -- appears to be the preferred approach of only about one-quarter of this large and diverse population of users.  That is, over three-quarters of the respondents preferred to use a networking approach early in their information gathering process, rather than relying on the surrogate data and information intermediary systems to produce what they needed. 

                Surely this constitutes an argument for a movement toward a more comprehensive information utilization model, in which formal sources of data can be used in creative combination with interpersonal and interactive media to produce a more situation- and person-responsive operation.  Such an approach, which one might call an "interactive information intermediary" system, would lend itself to effective use by a significantly higher number of individuals than are now comfortable with any one component of our present highly disaggregated and generally reactive arrangements for knowledge transfer.

                It is clear that the technological infrastructure to support such a system, if not wholly developed, is at least feasible.  The data indicate quite high overall levels of use of a significant number of the interactive information technologies that would be required by this approach.  With some additional augmentation -- for example, expert system tools to assist in literature search, or object-oriented databases that link text, graphics, and audio in searchable patterns -- existing knowledge transfer systems would find themselves reaching vastly more individuals, and vastly better.

                Attention to the technology of transfer should not lead us to forget, however, that the underlying issues of quality and utility of data are paramount.  On-line data retrieval is paced by the ability of the searcher to bound the problem; computer or video conferencing is no better than the quality of the participants and the time they can afford to devote to the exchange.  The "new media" (Rice et al., 1984) can best be seen as "multipliers", affecting the power and magnitude of the exchanges they facilitate rather than their basic nature.

                We have by no means exhausted the research needed to understand this problem, even within the limited compass of NASA/DOD research publications and the aerospace community.  For one thing, it would be particularly interesting to have follow-up data that reflect more directly the networks of relationships among information providers, intermediaries, and users.  The present data, while extremely informative, do not allow us to link, for example, the opinions of users and those of the intermediaries that serve them.  More systematic attention to the patterns of interaction that characterize this extremely diverse and heterogenous community of participants in the research process would be extremely helpful in estimating the need that remain to be met by such an interactive strategy.

                                In sum, the evidence to date appears to reinforce the concept that individuals' "information environments" take many different shapes, and interact with each other and with formal data transmission sources in many different and equally valuable ways. Any overall strategy for improving the effectiveness and efficiency of scientific and technical information sharing must take this divergence into account, and work toward the creation of systems that reinforce true interactive knowledge utilization rather than simply "disseminating" data.  We have a long way to go before we can specify what such a strategy would look like, but studies such as this can help point the way.


STATISTICAL NOTE

                This Note outlines statistical procedures applied in the course of this analysis, and the varieties of techniques employed. The NASA questionnaire data pose a number of complex analytical issues that deserve attention, but that would have only served to distract the reader in the course of the report itself.

                One major issue is that of the sizes of the samples alone. These three surveys employed rather large samples (over 200 for NASA1 to over 800 for NASA3), and contained highly varied items. This poses two somewhat contradictory problems.  On the one hand, for those items that were answered by relatively large numbers of the respondents, the large N for statistical analyses means that relationships between variables that may be of even very small magnitudes are going to be "statistically significant" at usually accepted levels of significance such as .01 or .001.  With an N of 1000, a correlation coefficient as small as .06 will be significant at the .05 level, and a coefficient of .09 will be significant at the .001 level.  All "significance" means, of course, is that the reported relationship is not zero.  The effect of these sample sizes is that we need to shift our focus of statistical inference from the usual criterion of significance to one emphasizing magnitude of relationships.

                The related problem is that for many of the variables, particularly those in the NASA2 and NASA3 surveys, there are very large amounts of missing or negative data.  For the items dealing with specific kinds of reports, there are often very few respondents who actually use the report.  This makes comparisons across reports somewhat problematical, since one is in fact dealing with very different and distinct subpopulations.  Small and diverse subpopulations often result in unstable findings.

                The analyses reported in this paper employed a number of standard statistical procedures.  Here, we review the major procedures and the ways in which each were used.

Analysis of Categorical Data

                Many of the variables used in these questionnaires, particularly the demographic items, were categorical in nature. The general procedure was to reduce items with large numbers of categorical alternatives (which invariably had some categories with very small numbers of respondents) to recoded items with no more than four or five categories to provide large enough aggregates for comparisons.  For analyses including only categorical variables, two varieties of tests were used.  First, chi-square statistics were examined for the table as a whole, to determine if there were overall significant deviations from randomness.  If the overall statistic was significant, individual cell deviations from expected values were examined to identify from where the effects might be coming.

Analysis of Variance

                For relationships involving one categorical variable and one intervally scaled variable, oneway analysis of variance was employed.  All relationships reported as "significant" under this test were significant at the .001 level or better.  To test the significance of differences among individual group means, Tukey's "honestly significant difference" test was employed, with alpha set at .05.

Regression and Correlation

                For relationships between intervally-scaled variables, Pearson's r was the correlation coefficient employed.  As noted, values of r that are quite small in terms of real relationships will be "significant" with large enough N's.  In fact, for most of the correlations reported as significant here, the percent of variance in one variable accounted for by the other is rather small.  To test a few relationships, stepwise linear regression was employed; the chosen level of significance was .001.

Factor Analysis and Index Construction

                Since the questionnaires contained numerous different specific questions that could readily be interpreted as relating to similar underlying constructs, factor analysis was employed at several points to search for latent structure.  Specifically, we employed principal components analysis with varimax rotation and an eigenvalue criterion of 1 for factor retention.  In general, factor loadings less than .6 were discounted.

                Two different varieties of index construction were employed. Where there was reason to believe that the factor structure was clear and unambiguous, simple additive indices were employed to weight each incoming variable equally.  This procedure was used with the utilization and influence indices.  Where factor structures were less clearly defined, factor scores were employed instead to preserve the full efficiency of the factor analysis.

                Indices were checked for reliability by the use of Cronbach's alpha.  Alpha levels of .7 or higher were taken to reflect reliability of the scale.  Where alphas were lower than .7, item‑ to-total correlations were checked to determine if one or more items could be deleted to produce an acceptable scale.

Cluster Analysis

                To aggregate the influence variables in particular, cluster analysis was employed.  Clustering techniques have the advantage of grouping variables less ambiguously than factor analyses, particularly when there may be a rather complex factor structure with several underlying dimensions.  The clustering in this study was accomplished using a squared euclidean distance metric with Ward's Method as an algorithm; this algorithm produces well defined and separable clusters.  The solutions obtained through cluster analysis were checked against factor analyses of the same variables, and the two approaches confirmed each other.

"Mapping" with Multidimensional Scaling

                To examine relationships among a large number of parallel variables -- here, the information technology variables -- nonmetric multidimensional scaling was employed to construct a "map" reflecting underlying dimensionality.  The algorithm employed is ALSCAL, using a euclidean distance model.  The Kruskal's stress coefficient for the final 2-dimensional solution was less than .03.

ENDNOTES

[1] Henceforth, we will generally user the acronym "STI". Obviously, this is an extremely complex composite of many different kinds of data, research results, interpretations, basic science, applied engineering, and the like.  The essential "STI-ness" of all these diverse kinds of information is made up of the nature of the intellectual community within which they move.  We hope that the oversimplifications inherent in a concept like "STI" will be compensated by an ability to see where generic solutions might be valid and applicable.

 [2] "Aerospace" includes aeronautics, space science, space technology, and related fields.

[3] It is useful to note here that not infrequently the "producer" and the "user" of the information amy be essentially the same person at different times and in different contexts.  Producer and user are roles, not fixed categories of personnel.

[4] For each of the three surveys in Phase 1, a supplemental mailing was undertaken to increase the proportion of academic respondents, who were found to be systematically underrepresented in the original sample.  Reported figures cover both the original and supplementary questionnaires.  See Project Report #  __ for full details on the questionnaire construction and mailing processes.

 [5] All acronyms are defined in Appendix A.

 [6] Appendices B, C, and D are the actual questionnaires with the univariate frequencies provided.  They are provided in lieu of separate univariate tables in most cases.

 [7] In the discussion that follows, the percentages described are in fact those for NASA1 rather than for the other two questionnaires.  However, as a comparison of Appendices B-D shows, the relative frequencies of particular groups of respondents were almost always with a couple of percentage points of each other. Thus, all three surveys can be considered to refer to essentially the same kind of population.

 [8] Here, and in the future, any sets of percentages that do not add to 100% may be considered as having the residue consisting of "other" or "don't know" responses.

 [9] All relationships described here are significant at the .01 level or less using standard chi-square tests on crosstabulations. In view of the multitude of relationships possible among these variables, detailed tables have been omitted in favor of summary description.  See the Statistical Note for further elaboration.

 [10] Since there was relatively little reported use of information intermediary sources at these stages, they were combined with "data" sources for the rest of this analysis.  Thus, references to "data people" should be interpreted as referring to both those who chose formal data sources and those who relied on information intermediaries.

[11] Of the 416 D/D's, 210 were what might be called "hardcore D/D's", in that they remained with data sources even to the third iteration.  The behavior of this hardcore group seldom differs from that of the rest of the D/D group significantly, though they do tend to exhibit the D/D properties with a bit more strength than do the "softcore" D/D's.

[12] These groupings were obtained by a factor analysis on the library interaction variables.  See Statistical Note for further details.

[13] Again, factor analysis was used to group the technologies into these categories.

[14] See Statistical Note for details of these tests and methods of determining significance.

[15] See Statistical Note for further details of this procedure.

[16] See Statistical Note for these techniques.

[17] These differences were detected through analysis of variance.  See Statistical Note.

[18] Since this group works almost entirely in government installations, there is not a heavy overlap with the "basic science" group identified earlier through the index construction process -- although they are otherwise alike in many ways.

[19] The overall quality ratings are a summative index of five specific quality dimensions asked about each report.  Factor analysis revealed no sub-structure to these questions; alphas on the resulting additive indices range from .71 to .83.

[20] See Statistical Note for procedures.

[21] See Statistical Note for a description of the procedures used.

[22] Again, see the Statistical Note for details on index construction methods.

[23] See Statistical Note for information regarding ANOVA procedures.