About | HeinOnline Law Journal Library | HeinOnline Law Journal Library | HeinOnline

LOG IN
LOG IN


5 A.I. & L. 1 (1997)

handle is hein.journals/artinl5 and id is 1 raw text is: Artificial Intelligence and Law 5: 1-74, 1997. 1 © 1997 Kluwer Academic Publishers. Printed in the Netherlands. Evaluating a Legal Argument Program: The BankXX Experiments* EDWINA L. RISSLAND, DAVID B. SKALAK and M. TIMUR FRIEDMAN Department of Computer Science, University of Massachusetts, Amherst, MA 01003, U.S.A. (E-mail: rissland@cs.umass.edu) Abstract. In this article we evaluate the BankXX program from several perspectives. BankXX is a case-based legal argument program that retrieves cases and other legal knowledge pertinent to a legal argument through a combination of heuristic search and knowledge-based indexing. The program is described in detail in a companion article in Artificial Intelligence and Law 4: 1-71, 1996. Three perspectives are used to evaluate BankXX: (1) classical information retrieval measures of precision and recall applied against a hand-coded baseline; (2) knowledge-representation and case-based reasoning, where the baseline is provided by the functionality of a well-known case-based argument program, HYPO (Ashley, 1990); and (3) search, in which the performance of BankXX run with various parameter settings, for instance, resource limits, is compared. In this article we report on an extensive series of experiments performed to evaluate the program. We also describe two additional experiments concerning (1) the program's search behavior; and (2) the use of a modified form of precision and recall based on case similarity. Finally we offer some general conclusions that might be drawn from these particular experiments. Key words: case-based reasoning, legal argument, information retrieval, search, evaluation, bank- ruptcy Part I: Introduction 1. Introduction: The Problem of Evaluating Arguments Evaluation of argument is a longstanding problem. It is also a complex one. Perel- man and Olbrechts-Tyteca (1969) have noted, in fact, In view of the complexity of the factors to be taken into consideration even just to judge whether an argument has any strength at all, it is curious that the writers of treatises on rhetoric should so glibly state, almost incidentally, that the strength of arguments is common knowledge and that they should base their advice regarding the order of discourse and the sequence of replies, on the degree of conviction that the arguments must have produced, which it is not hard for us to know, because we know what ordinarily brings this about. (Perelman and Olbrechts-Tyteca, 1969, p. 462) * This research supported in part by grant No. 90-0359 from the Air Force Office of Scientif- ic Research and NSF grant No. EEC-9209623 State/University/Industry Cooperative Research on Intelligent Information Retrieval.