Knowledge engineering is the process used to develop an expert system. Knowledge engineering is an example of a software development methodology. Such a methodology defines the artifacts that must be created as part of building a software application (e.g., requirements, designs, code, and documentation) as well as the various tasks that must be completed, their dependencies, etc.
Background: Expert Systems
An expert system is an Artificial Intelligence application that captures the knowledge of a highly skilled expert such as a doctor or engineer and partially automates one or more critical tasks that the expert must perform. One of the first examples of an expert system was MYCIN, an application to perform medical diagnosis. In the MYCIN example the domain experts were medical doctors and the knowledge represented was their expertise in diagnosis.
Expert systems were first developed in artificial intelligence laboratories as an attempt to understand complex human decision making. Based on positive results from these initial prototypes the technology was adopted by the US business community (and later world wide) in the 1980's. The Stanford heuristic programming projects led by Edward Feigenbaum was one of the leaders in defining and developing the first expert systems.
In the earliest "cowboy" days of expert systems there was little or no formal process for the creation of the software. Researchers just sat down with domain experts and started programming, often developing the required tools (e.g. inference engines) at the same time as the applications themselves. As expert systems moved from academic prototypes to deployed business systems it was realized that a methodology was required to bring predictability and control to the process of building the software. There were essentially two approaches that were attempted:
- Use conventional software development methodologies
- Develop special methodologies tuned to the requirements of building expert systems
Many of the early expert systems were developed by large consulting and system integration firms such as Andersen Consulting. These firms already had well tested conventional waterfall methodologies (e.g. Method/1 for Andersen) that they trained all their staff in and that were virtually always used to develop software for their clients. One trend in early expert systems development was to simply apply these waterfall methods to expert systems development.
However, there were significant issues with trying to apply these conventional methods to the development of expert systems. Many of the design modeling techniques (e.g. flow charts, dataflow diagrams) that were essential to these methods were of minimal value to designing expert systems. For example, an inference engine (the tool used by expert systems to represent and utilize expert knowledge coded as rules) attempts to abstract away from things like explicit flow of control. The control flow for an expert system can be very hard to predict because for each example the system will be driven by the particular rules that have fired. This can be a very powerful mechanism, allowing engineers to define knowledge via rules that are independent of specific programs. However, trying to specify such rules and their control flow via diagrams that must specify predefined flow of control will be very difficult. Indeed one of the goals of expert systems was to abstract away from specific programming. It was often claimed that expert system shells allowed the experts to become programmers and to do away with professional programmers. This was seldom true in reality but it reflects the core idea that expert system shells tried to move the definition of the system to a higher level of abstraction than conventional code. Thus, the whole need for detailed design before programming was mostly ameliorated. Note that while design diagrams could often be minimized for the expert system itself as expert systems began to take off the requirement for them to integrate with existing systems and legacy databases was significant and design of such integration was a critical part of the complete knowledge engineering process.
Another issue with using conventional methods to develop expert systems was that due to the unprecedented nature of expert systems they were one of the first applications to adopt rapid application development methods that feature iteration and prototyping as well as or instead of detailed analysis and design. In the 1980's few conventional software methods supported this type of approach.
The final issue with using conventional methods to develop expert systems was the need for knowledge acquisition. Knowledge acquisition refers to the process of gathering expert knowledge and capturing it in the form of rules and ontologies. Knowledge acquisition has special requirements beyond the conventional specification process used to capture most business requirements.
These issues led to the second approach to knowledge engineering: development of custom methodologies specifically designed to build expert systems. One of the first and most popular of such methodologies custom designed for expert systems was the Knowledge Acquisition and Documentation Structuring (KADS) methodology developed in Europe. KADS had great success in Europe and was also used in the United State.
- Feigenbaum, Edward; McCorduk, Pamela (1983). The Fifth Generation (1st ed.). Reading, MA: Addison-Wesley. ISBN 978-0-201-11519-2. OCLC 9324691.
- Schreiber, August Th.; Akkermans, Hans; Anjewierden, Anjo; Dehoog, Robert; Shadbolt, Nigel; Vandevelde, Walter; Wielinga, Bob (2000), Knowledge engineering and management: the CommonKADS methodology (1st ed.), Cambridge, MA: The MIT Press, ISBN 978-0-262-19300-9
- Knowledge representation
- Knowledge retrieval
- Knowledge management
- Knowledge level modeling
- Knowledge acquisition
- Knowledge tagging
- Expert systems
- Data & Knowledge Engineering – Elsevier Journal
- Knowledge Engineering Review, Cambridge Journal
- The International Journal of Software Engineering and Knowledge Engineering – World Scientific
- IEEE Transactions on Knowledge and Data Engineering
- Expert Systems: The Journal of Knowledge Engineering – Wiley-Blackwell