More than 80 % of all information in an organization is unstructured, created by knowledge workers engaged in peer-to-peer networks of expertise to share knowledge across organizational boundaries. Enterprise Information Systems (EIS) do not integrate unstructured information. At best, they integrate links to unstructured information connected with structured information in their databases. The amount of unstructured information is rising quickly. Ensuring the quality of this unstructured information is difficult. It is often inaccessible, unavailable, incomplete, irrelevant, untimely, inaccurate, and/or incomprehensible. It becomes problematic to reconstruct what has happened in organizations. When used for organizational policies, decisions, products, actions and transactions, structured and unstructured information are called records. They are an entity of information, consisting out of an information object (structured or unstructured) and its metadata. They are important for organizational accountability and business process performance, for without them reconstruction of past happenings and meaningful production become an impossibility. Organization-wide management of records is not a common functionality for EIS, resulting in [1] a fragmentation in the management of records, where structured and unstructured information objects are stored in a variety of systems, unconnected with their metadata; [2] a fragmentation in metadata management, leading to a loss of contextuality because metadata are separated from their information objects; and [3] a declining quality or records, because their provenance, integrity, and preservation are in peril. Organizational accountability is based on records and their context to reconstruct the past. Because records are not controlled by EIS, they can only marginally be used for accountability. The challenge for organizational accountability is to generate trusted records, fixed and contextual information objects inseparately linked with metadata that capture context to regain evidential value and to allow for the reconstruction of the past. The research question of this paper is how to capture records and their context within EIS to regain the evidential value of records to allow for a more robust organizational accountability. To find an answer, we need to pay attention to the concept of context, on how to capture context in metadata, and how to embed and manage records in EIS.
DOCUMENT
Many organizations have undergone substantial reorganization in the last decade. They re-engineered their business processes and exchanged proprietary, not integrated applications for more standard solutions. Integration of structured data in relational data bases has improved documentation of business transactions and increased data quality. But almost 90% of the information that organizations manage is unstructured, can not easily be integrated into a traditional database. Just like structured data, unstructured information in organizations are records, meant and used as evidence for organizational actions and transactions. Governments, courts and other stakeholders are making increasing demands for the trustworthiness of records. This is part of a long-term trend toward defining what accountability means in a digital era. An analysis of literature of information science, organization science and archival science illustrates that for accountability, reconstruction of the past is essential. Hypothesis of this paper is that for the reconstruction of the past each organization needs (at least) a combination of three mechanisms: enterprise records management, organizational memory and records auditing. Enterprise records management ensures that records meet the for accountability necessary quality requirements: integrity, authenticity, controllability and historicity. These requirements ensure records that can be trusted. Trusted records enhance the possibility of reconstructing the past. The organizational memory ensures that trusted records are preserved for as long as is necessary to comply to accountability regulations. It provides an information and communication technology infrastructure to (indefinitely) store those records and to keep them accessible. Records auditing audits enterprise records management and organizational memory to assess the possibility to reconstruct past organizational actions and transactions. These mechanisms ensure that organizations have a documented understanding of: the processing of actions and transactions within business processes; the dissemination of trusted records; the way the organization accounts for the actions and transactions within its business processes; and the reconstruction of actions and transactions from business processes over time. This understanding is important for the reconstruction of the past in digitized organizations and improve organizational accountability.
DOCUMENT
Multilevel models (MLMs) are increasingly deployed in industry across different functions. Applications usually result in binary classification within groups or hierarchies based on a set of input features. For transparent and ethical applications of such models, sound audit frameworks need to be developed. In this paper, an audit framework for technical assessment of regression MLMs is proposed. The focus is on three aspects: model, discrimination, and transparency & explainability. These aspects are subsequently divided into sub-aspects. Contributors, such as inter MLM-group fairness, feature contribution order, and aggregated feature contribution, are identified for each of these sub-aspects. To measure the performance of the contributors, the framework proposes a shortlist of KPIs, among others, intergroup individual fairness (DiffInd_MLM) across MLM-groups, probability unexplained (PUX) and percentage of incorrect feature signs (POIFS). A traffic light risk assessment method is furthermore coupled to these KPIs. For assessing transparency & explainability, different explainability methods (SHAP and LIME) are used, which are compared with a model intrinsic method using quantitative methods and machine learning modelling.Using an open-source dataset, a model is trained and tested and the KPIs are computed. It is demonstrated that popular explainability methods, such as SHAP and LIME, underperform in accuracy when interpreting these models. They fail to predict the order of feature importance, the magnitudes, and occasionally even the nature of the feature contribution (negative versus positive contribution on the outcome). For other contributors, such as group fairness and their associated KPIs, similar analysis and calculations have been performed with the aim of adding profundity to the proposed audit framework. The framework is expected to assist regulatory bodies in performing conformity assessments of AI systems using multilevel binomial classification models at businesses. It will also benefit providers, users, and assessment bodies, as defined in the European Commission’s proposed Regulation on Artificial Intelligence, when deploying AI-systems such as MLMs, to be future-proof and aligned with the regulation.
DOCUMENT
