 Coordinator:
Martin
O'CONNOR
|
|
Quality Assessment (QA)
|
WP3 Transversal Elements
|
Allocation of Person-Months per Partner
|
|
|
P1
|
SC-1
|
P2
|
P3
|
P4
|
P5
|
P6
|
P7
|
P8
|
P9
|
P10
|
Totals
|
|
3.A
Extended Peer Review
|
1.0
|
|
2.0
|
|
2.0
|
1.0
|
2.0
|
4.0
|
4.5
|
2.0
|
1.0
|
19.5
|
|
3.B
Scientific Peer Review
|
1.0
|
|
2.0
|
|
2.0
|
2.0
|
1.0
|
|
|
0.5
|
1.0
|
9.5
|
|
3.C
Technology QA
|
1.0
|
2.0
|
1.0
|
1.0
|
2.0
|
|
|
|
0.5
|
|
|
7.5
|
| Deliverables |
Notes: (1) The Work Package
lead partners are indicated by the highlighted box in the matrix. (2) Much
of the work package leadership and work has been supported by tenured academics
of research institutions, hence this does not appear in the person-months
specified above.
Objectives
The work package set provided for the reviewing on a formal and informal
basis to ensure the quality of the ICT prototypes to be used by individuals,
in decision situations and policy formulation about sustainable relationships
with ecosystems and living resources. This consisted not only of checks
on the technical quality (WP3.C) and the scientific validity and robustness
(WP3.B) of the ICT tools to be designed and developed in WP4, but also
of an extended review by potential users (WP3.A) who, most often, are
representatives of other knowledges and interests which are not accommodated
in a purely scientific review. For the VIRTU@LIS work, knowledge quality
evaluation entails:
-
the classical scientific
considerations of rigour, coherence, measurement validation and sensitivity
testing for the sequences of data transformation, aggregation and
modelling;
-
the user-oriented considerations
of pertinence for framing a decision problem and for supporting a
multi-user learning activity.
These criteria have been carried
over into the VR domain, so as to lead to a unique brand of learning environments
including simulations and "games" that have the added quality
of being "reality checked" rather than arbitrary constructions.
The crucial KQA task within the VIRTU@LIS project is to show how this
quality assurance can be implemented and, as such, provide protocols for
pedagogic applications that are not just entertaining but also of high
pertinence for citizens' information and in empirical teaching situations.
The final deliverables from this Work Package included Guidelines
for scientific quality assessment procedures appropriate to ICT-based
learning tools representing environmental systems and sustainable resource
management problems.
| Box
3.1: 'Tuning Contexts': The tuning of the modelling and software tools
developed during this work package set was done by setting up
'tuning contexts' - that mean a group of 'informal' methods that aim
at incorporating knowledges other than the technical and scientific
one into a modelling or an assessment framework. This is important
where the use is intended to be decision or policy formulation situations.
So, the 'Tuning contexts' are the forums where appropriate debate
over the design of Information & Communication Technology can
be developed and documented. |
In practice, during group sessions (focus groups, e.g.) or through Internet
linkages, participants was invited to evaluate the ICT tools they
are interacting with, based on a set of criteria given by the moderators
that are related to user and societal needs, system functionality and
operability, quality and reliability of the provided information, user
interface, etc. The users were also asked to make recommendations for
modifications. These recommendations were complemented by the moderators
based on the observation of the user's interaction with the applications.
Work description:
WP3.A "Extended Peer Review" - Complementary to technology
quality assessment and scientific evaluation (see WP3.C and WP3.B, below),
there needs to be "stakeholder review" of the ICT. In effect,
while user requirements was at the heart of initial prototype conception,
these requirements are reassessed through "tuning". For this,
participatory contexts were designed and set for bringing together
stakeholders' perspectives into a non-scientific review process in order
to search for a quality assured technology. This implied a process that
has enabled the integration of perspectives other than those from the
scientific communities (see box 3.1).
This WP will be led by Jim Ewing of Dundee University (Dundee, Partner
7), specialist in the assessment of learning effectiveness in formal and
wider societal contexts, and supported by Partners 1, 4, 5, 6, 7, 8, 9,
10 (that is, all partners except ISIS and YDREAMS, the specialist ICT
developers). The testing of the Virtual Worlds and of the various individual
ICT products has consisted of running a number of sessions with groups.
Discussion and evaluation of these tools becomes a value-articulation
process within the user community, and a dynamic contribution to setting
up specifications from the users' points of view, indicating the benefits
and drawbacks of using a specific tool in the context it was designed
for. Performance and efficacy of the software interface features (ease
of use, clarity, reliability, etc.) has been evaluated during this process.
This is, in effect, a process of quality assurance through knowledge sharing.
This work was carried out in three ways.
First, groups of secondary school and university students will be engaged;
this were done in three different countries - Italy (University of
Milan, Partner 9), France (C3ED, Partner 1 and the GRIC, Partner 8) and
the United Kingdom (Cranfield, Partner 4 and Dundee University, Partner
7). However, student populations were not representative of full citizen
diversity.
Second, therefore, wide-spectrum groups representing a cross-section of
socio-economic classes, has been recruited to offer a spontaneous appraisal
of the VIRTU@LIS prototypes in structured 'focus group' settings. This
category of work has been carried out in the United Kingdom (CoMPLEX, Partner
10, drawing on existing networks), in France (C3ED, Partner 1, in close
co-operation with the environmental section of one or several large territorial
authorities) and in Italy (University of Milan, Partner 9).
Third, in the VIRTU@LIS Workshop I (Month 17), panels were invited
to experiment with the prototypes and offer the comments and design "tuning"
suggestions on the modelling and visualisation rationales used to represent
the physical phenomena related to a particular human impact to the living
resource.
This extended peer appraisal work was thus conducted, overall, in
three phases:
-
A preliminary round, based
mostly on student populations, has taken place between months 7-10,
in order to guide the prototype developments.
-
An in-depth testing process,
engaging school students, university populations and citizens at large,
was undertaken during Months 17-20, once fully-fledged prototypes
are available; this includes the VIRTU@LIS Workshop 1.
Finally, after revisions or "tuning" are
carried out, a third phase has been undertaken involving university populations
and citizens at large. This last phase has been simultaneous with work WP5
on demonstrations to potential institutional users (WP5.C) and as an input
to the activities of diffusion via Internet site design and distance learning
developments (WP5.A and WP5.B).
Box 3.2: The NUSAP scheme for Knowledge
Quality Assessment was designed to provide a robust system of notations
for expressing and communicating uncertainties in quantitative information.
The name is an acronym for the categories Numeral, Unit, Spread, Assessment
and Pedigree. This enables us to make the distinction between the
sources and the sorts of uncertainty that we can find in policy-relevant
we are able to effectively relate uncertainty and quality. The NUSAP
approach is thus not merely a notational system; but it provides awareness
to the issue of quality of scientific information, and a commitment
to find operational solutions to its problems. It operates by clarifying
the critical distinctions used in each field of practice. It is an
extension of standard scientific notation, and can be easily mastered
by anyone with a concern for quality of information.
- Original Development of NUSAP : Uncertainty and Quality in Science
for Policy, by S.O. Funtowicz and J.R. Ravetz, Kluwer 1990. |
WP3.B "Scientific Knowledge Quality
Assurance"
The complexity of natural systems, the time gaps between exploitation
pressure or contamination and cumulative environmental response, and the
consequent uncertainties concerning management risks and requirements
for long-term stability, pose special problems for scientific quality
assurance of any information system. The variety of problems of error
and uncertainty (in measurement, classification, aggregation (etc.) associated
with measurement and ecosystem description are well known. The use of
complex models brings the additional risk of 'bugs' and errors, viz.,
that the algorithms etc. do not actually do what the design intends of
them. This WP3.B was led by Silvio Funtowicz of ISIS (Partner 2).
First, small groups was organised during Months 12-15 where selected
"experts" (from 1 to 4 at a time) will be invited to evaluate
the way in which the prototype tools present the specific problematique
it addresses and the way it triggers the process of enhancing science
inputs into resource management and governance.
Second, in the VIRTU@LIS Workshop I (Month 17), panels of experts were invited to experiment with the prototypes and offer the comments and
design "tuning" suggestions on the modelling and visualisation
rationales used to represent the physical phenomena and human impact to
the living resource.
Third, the ISIS (S. Funtowicz, A. Pereira, S. Corral) and the C3ED (M.
O'Connor) was co-operated to provide scientific knowledge assessment (KA)
guidelines to be applied to the model components of the VIRTU@LIS prototypes.
These guidelines was based on the NUSAP framework which addresses
issues of measurement, quantification and spread (error bars, confidence
intervals, fundamental uncertainty, etc.) and also the significance of
underlying theoretical frameworks and quantification conventions for the
robustness, pertinence and acceptance of results by the scientific and
stakeholder communities (see Box 3.2).
Fourth, based on the three preceding elements, scientific partners (C3ED
for climate change; C3ED and the OpenUni for agriculture; Cranfield and
NIMR for fisheries; Cranfield and C3ED for water) have worked with the ISIS
experts to document scientific quality considerations in ways pertinent
to envisaged prototype users. Selected results of this appraisal was included in Virtual Library components.
WP3.C "Independent Technology Assessment"
In addition to the normal checks on software functioning, programming
(etc.) carried out by the ICT development teams, a specialised subcontractor,
FUTUREtec (SC1), made an independent technical 'audit' of the prototypes
in order to test for bugs, provide feedback to the developers about design
and documentation of linkages between data sets, model components, etc.
. This technological 'audit' was conducted prior to the final "tuning"
work on the domain prototypes. Preparations for this audit began
in Month 15, and the main work, based on submitted prototypes and documentation
from consortium partners, took place during Months 18-24, as the
integration of component prototypes takes place.
|
|
|
International
Workshop "Interfaces between Sciences&Society"
Milan, 27-28 Nov. 2003
