Our business

is evaluation of research through publication metrics.

The company

Our MD is Dr. Grant Lewison. Grant Lewison's background is in engineering but since 1981 his research interests have been mainly in science policy. He has been practising the new science of bibliometrics since 1988 and was appointed to a visiting professorship at City University, London, in 2001, following eight years as a Senior Policy Adviser at the Wellcome Trust.

Citation indexing is reckoned to be fifty years old. Eugene Garfield's original paper appeared in "Science" in 1955. When the journal, Current Science, published a special section (Vol. 89 No.9, 19 November 2005) to mark its the fiftieth anniversary Grant was invited to contribute an article. It is instructively called "Beyond SCI citations - New ways to evaluate research." (SCI is the Science Citation Index, now © Thomson Scientific.)

Evaluametrics Ltd. was founded as a company at that time. Although it is new, it brings together many years of experience of evaluation. Another director, Dr. Philip Roe FRAS, whose background is in theoretical astronomy, has several years' experience of evaluating large-scale UK and European programmes.

Techniques

Information retrieval and analysis: research papers

The scale of world-wide scientific output makes the task of information retrieval non-trivial, even from well-organised data bases of research papers such as the SCI. This is not the only source of bibliographic data on research papers, but it is the one most commonly used for evaluation because it is multi-disciplinary and lists all authors and all addresses. Evaluametrics has several important techniques for the compilation and analysis of groups of research papers. The first of these is the ability to "filter" an electronic database so as to select papers in a particular subject, especially in the biomedical field. Filters are often developed specially for a client, but we now have almost 100 available for use. They are based on the names of journals and words in the titles of papers, and can give values of precision and recall above 90%.

A second resource is the ability to record the financial acknowledgements on research papers, using a special thesaurus of over 10,000 funding bodies, both in the UK and world-wide. (This was originally developed to enable The Wellcome Trust to identify the papers that it had supported.) Inspection of papers and the recording of these acknowledgements is by no means simple, but we now have 13 years of experience of doing this.

A third resource is a means of classifying biomedical papers by their research level (on a spectrum from clinical to basic), based not only on the journal in which they are published, but on their individual characteristics. This is of great utility in making a fair evaluation of the output of a research team or a country, as it is known that on many indicators, basic research scores much better than clinical work, and it is therefore necessary to allow for this.

Finally, Evaluametrics possesses a set of special programs which enable many analyses to be carried out quickly and easily. They include:

There is more to this than simple filtering. Are John Jones, J Jones and JG Jones one person or two or three? Cross referencing, citation, a study of addresses etc.are among the methods a smart program can use. (Philip Roe, who wrote many of these programs, has also written information retrieval programs for commercial insurance companies. Philip has written other things too. One national newspaper uses a program of his to check that the sudoku puzzle is valid and at the right level of difficulty.)

Other information sources: references to research

Grant's paper cited above describes five other sources of citing information that can be used for research evaluation: international standards, government policy documents, clinical guidelines, textbooks and newspapers. Of these, clinical guidelines are obviously most relevant in medical research and largely cite clinical papers. International standards govern much commercial activity, and are usually devised by expert committees and have extensive underpinning references. Textbooks increasingly read like research reviews and also have long lists of references, mostly to recent research. Newspapers are a source of information not so much about research as about the public perception of research. If, for example, we need to know whether the space program has led to more funding for astronomical research, we need to see how the public (whose votes ultimately allocate the funding in a democracy) perceive, or are encouraged to perceive, the outputs of such research.

Techniques for studying these various sources are available to Evaluametrics. The citations can be identified with research papers and their characteristics compared with those of a norm group. Sometimes the simplest methods are best. When we need to know the source of funding acknowledged in scientific journals there is no substitute for getting a reliable, conscientious person to do a great deal of looking up. The evaluametrics team includes such people.

Past projects

Before evaluametrics was founded as a company Grant Lewison and his colleagues had completed a number of projects.at City University, the Wellcome Trust and earlier. These included the following:

Other recent clients have included WHO, Geneva, National Cancer Research Institute, Royal Society of Chemistry, Association of Medical Research Charities, British Heart Foundation, Global Forum for Health Research, Geneva, Lund University, Sweden, Hong Kong Science Park Association, Goteborg University, Sweden, Arthritis Research Campaign, and Diabetes UK

Current projects

Currently Evaluametrics Ltd is conducting projects for Uk clients:the Multiple Sclerosis Society, Cancer Research UK and NHS Public Health Resource Unit. It is also providing support to a US company, WTEC Inc. Baltimore,. who are organizing evaluation panels on behalf of consortia of US government agencies in two areas of new technology (brain-computer interaction and nanocatalysis) which will be undertaking study tours in Europe and the Far East. (Evaluametric staff between them have a working knowledge of most of the common Market languages and of Japanese.)

Further information about evaluametrics is available by e-mail: information@evaluametrics.co.uk