Objective: To annotate a corpus of randomized controlled trial (RCT) publications with the checklist items of CONSORT reporting guidelines and using the corpus to develop text mining methods for RCT appraisal. Methods: We annotated a corpus of 50 RCT articles at the sentence level using 37 fine-grained CONSORT checklist items. A subset (31 articles) was double-annotated and adjudicated, while 19 were annotated by a single annotator and reconciled by another. We calculated inter-annotator agreement at the article and section level using MASI (Measuring Agreement on Set-Valued Items) and at the CONSORT item level using Krippendorff's α. We experimented with two rule-based methods (phrase-based and section header-based) and two supervised learning approaches (support vector machine and BioBERT-based neural network classifiers), for recognizing 17 methodology-related items in the RCT Methods sections. Results: We created CONSORT-TM consisting of 10,709 sentences, 4,845 (45%) of which were annotated with 5,246 labels. A median of 28 CONSORT items (out of possible 37) were annotated per article. Agreement was moderate at the article and section levels (average MASI: 0.60 and 0.64, respectively). Agreement varied considerably among individual checklist items (Krippendorff's α= 0.06–0.96). The model based on BioBERT performed best overall for recognizing methodology-related items (micro-precision: 0.82, micro-recall: 0.63, micro-F1: 0.71). Combining models using majority vote and label aggregation further improved precision and recall, respectively. Conclusion: Our annotated corpus, CONSORT-TM, contains more fine-grained information than earlier RCT corpora. Low frequency of some CONSORT items made it difficult to train effective text mining models to recognize them. For the items commonly reported, CONSORT-TM can serve as a testbed for text mining methods that assess RCT transparency, rigor, and reliability, and support methods for peer review and authoring assistance. Minor modifications to the annotation scheme and a larger corpus could facilitate improved text mining models. CONSORT-TM is publicly available at https://github.com/kilicogluh/CONSORT-TM.
DOCUMENT
Objective:Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications.Methods:We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale.Results:Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance ().Conclusion:The model could support automated screening tools which can be used by journals to draw the authors’ attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.
MULTIFILE
Energy policies are vital tools used by countries to regulate economic and social development as well as guarantee national security. To address the problems of fragmented policy objectives, conflicting tools, and overlapping initiatives, the internal logic and evolutionary trends of energy policies must be explored using the policy content. This study uses 38,277 energy policies as a database and summarizes the four energy policy objectives: clean, low-carbon, safe, and efficient. Using the TextCNN model to classify and deconstruct policies, the LDA + Word2vec theme conceptualization and similarity calculations were compared with the EISMD evolution framework to determine the energy policy theme evolution path. Results indicate that the density of energy policies has increased. Policies have become more comprehensive, barriers between objectives have gradually been broken, and low-carbon objectives have been strengthened. The evolution types are more diversified, evolution paths are more complicated, and the evolution types are often related to technology, industry, and market maturity. Traditional energy themes evolve through inheritance and merger; emerging technology and industry themes evolve through innovation, inheritance, and splitting. Moreover, this study provides a replicable analytical framework for the study of policy evolution in other sectors and evidence for optimizing energy policy design
DOCUMENT
Objective: To automatically recognize self-acknowledged limitations in clinical research publications to support efforts in improving research transparency.Methods: To develop our recognition methods, we used a set of 8431 sentences from 1197 PubMed Central articles. A subset of these sentences was manually annotated for training/testing, and inter-annotator agreement was calculated. We cast the recognition problem as a binary classification task, in which we determine whether a given sentence from a publication discusses self-acknowledged limitations or not. We experimented with three methods: a rule-based approach based on document structure, supervised machine learning, and a semi-supervised method that uses self-training to expand the training set in order to improve classification performance. The machine learning algorithms used were logistic regression (LR) and support vector machines (SVM).Results: Annotators had good agreement in labeling limitation sentences (Krippendorff's α = 0.781). Of the three methods used, the rule-based method yielded the best performance with 91.5% accuracy (95% CI [90.1-92.9]), while self-training with SVM led to a small improvement over fully supervised learning (89.9%, 95% CI [88.4-91.4] vs 89.6%, 95% CI [88.1-91.1]).Conclusions: The approach presented can be incorporated into the workflows of stakeholders focusing on research transparency to improve reporting of limitations in clinical studies.
DOCUMENT
DOCUMENT
The Confederation of European Probation detects a clear need for a central record of training and qualification requirements, standards or curricula for probation and community supervision practitioners in EU Member States. Hogeschool Utrecht (UAS) has been commissioned to conduct a survey to collect an overview of qualification requirements and training of probation workers in order to set up a network to share knowledge, competencies and expertise among members’ states. The research question is: What kind of formation precedes the start of a career of a probation worker in the various EU Member states and how does it develop throughout his or her career? In order to answer this question, an online survey with closed and open questions has been send out among CEP members that provide or are responsible for probation services. Twenty of them filled out the questionnaire covering 18 different countries.
DOCUMENT
Background: Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers’ comments. Methods: A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January – 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports. Results: A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes “writing criteria” (dimension II) is the top ranking theme, followed by comments in relation to the “methods” (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts. Conclusions: This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called “Degree of data transformation” to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.
MULTIFILE
From the introduction: "There are two variants of fronto-temporal dementia: a behavioral variant (behavioral FTD, bvFTD, Neary et al. (1998)), which causes changes in behavior and personality but leaves syntax, phonology and semantics relatively intact, and a variant that causes impairments in the language processing system (Primary Progessive Aphasia, PPA (Gorno-Tempini et al., 2004). PPA can be subdivided into subtypes fluent (fluent but empty speech, comprehension of word meaning is affected / `semantic dementia') and non-fluent (agrammatism, hesitant or labored speech, word finding problems). Some identify logopenic aphasia as a FTD-variant: fluent aphasia with anomia but intact object recognition and underlying word meaning."
MULTIFILE
Research into automatic text simplification aims to promote access to information for all members of society. To facilitate generalizability, simplification research often abstracts away from specific use cases, and targets a prototypical reader and an underspecified content creator. In this paper, we consider a real-world use case – simplification technology for use in Dutch municipalities – and identify the needs of the content creators and the target audiences in this scenario. The stakeholders envision a system that (a) assists the human writer without taking over the task; (b) provides diverse outputs, tailored for specific target audiences; and (c) explains the suggestions that it outputs. These requirements call for technology that is characterized by modularity, explainability, and variability. We argue that these are important research directions that require further exploration
MULTIFILE
Background: Early detection and remediation of language disorders are important in helping children to establish appropriate communicative and social behaviour and acquire additional information about the world through the use of language. In the Netherlands, children with (a suspicion of) language disorders are referred to speech and hearing centres for multidisciplinary assessment. Reliable data are needed on the nature of language disorders, as well as the age and source of referral, and the effects of cultural and socioeconomic profiles of the population served in order to plan speech and language therapy service provision. Aims: To provide a detailed description of caseload characteristics of children referred with a possible language disorder by generating more understanding of factors that might influence early identification. Methods & Procedures: A database of 11,450 children was analysed consisting of data on children, aged 2–7 years (70% boys, 30% girls), visiting Dutch speech and hearing centres. The factors analysed were age of referral, ratio of boys to girls, mono‐ and bilingualism, nature of the language delay, and language profile of the children. Outcomes & Results:Results revealed an age bias in the referral of children with language disorders. On average, boys were referred 5 months earlier than girls, and monolingual children were referred 3 months earlier than bilingual children. In addition, bilingual children seemed to have more complex problems at referral than monolingual children. They more often had both a disorder in both receptive and expressive language, and a language disorder with additional (developmental) problems. Conclusions & Implications: This study revealed a bias in age of referral of young children with language disorders. The results implicate the need for objective language screening instruments and the need to increase the awareness of staff in primary child healthcare of red flags in language development of girls and multilingual children aiming at earlier identification of language disorders in these children.
DOCUMENT
