Het platform voor open praktijkgericht onderzoek

product

Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study

Background: In their research reports, scientists are expected to discuss limitations that their studies have. Previous research showed that often, such discussion is absent. Also, many journals emphasize the importance of avoiding overstatement of claims. We wanted to see to what extent editorial handling and peer review affects self-acknowledgment of limitations and hedging of claims.Methods: Using software that automatically detects limitation-acknowledging sentences and calculates the level of hedging in sentences, we compared the submitted manuscripts and their ultimate publications of all randomized trials published in 2015 in 27 BioMed Central (BMC) journals and BMJ Open. We used mixed linear and logistic regression models, accounting for clustering of manuscript-publication pairs within journals, to quantify before-after changes in the mean numbers of limitation-acknowledging sentences, in the probability that a manuscript with zero self-acknowledged limitations ended up as a publication with at least one and in hedging scores.Results: Four hundred forty-six manuscript-publication pairs were analyzed. The median number of manuscripts per journal was 10.5 (interquartile range 6-18). The average number of distinct limitation sentences increased by 1.39 (95% CI 1.09-1.76), from 2.48 in manuscripts to 3.87 in publications. Two hundred two manuscripts (45.3%) did not mention any limitations. Sixty-three (31%, 95% CI 25-38) of these mentioned at least one after peer review. Changes in mean hedging scores were negligible.Conclusions: Our findings support the idea that editorial handling and peer review lead to more self-acknowledgment of study limitations, but not to changes in linguistic nuance.

PDF

Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study

product

Learning about racism: A week-by-week qualitative exploration of two white Dutch primary student teachers’ emotional responses during a Critical Race Theory based course

This qualitative study explored emotional responses of two white Dutch student teachers during a Critical Race Theory (CRT) based course. Following Plutchik's (2001) classification of 32 emotions, the analysis of their weekly diaries resulted in the identification of 16 emotions. In both diaries similar emotional responses were identified. However, the analysis did not reveal a straightforward path these students emotionally went through. The number and types of emotional responses, both comfortable and uncomfortable, fluctuated weekly and occurred simultaneously in various combinations. Even when similar emotional responses were identified, students connected differently to the course content. This could be explained by different starting points both students had when entering the course. The findings add to past work by identifying a variety and complexity of emotional responses of white student teachers during a CRT based course and can be used to create course conditions to prepare teachers for contributing to anti-racist education.

PDF

Learning about racism: A week-by-week qualitative exploration of two white Dutch primary student teachers’ emotional responses during a Critical Race Theory based course

product

Co-Adaptation processes of syntactic complexity in real-time kindergarten teacher-student interactions

Under the premise that language learning is bidirectional in nature, this study aimed to investigate syntactic coordination within teacher-student interactions by using cross-recurrence quantification analysis (CRQA). Seven teachers’ and a group of their students’ interactions were repeatedly measured in the course of an intervention in early science education. Results showed changes in the proportion of recurrent points; in case of simple sentences teachers and students became less coordinated over time, whereas in case of complex sentences teachers and students showed increasing coordination. Results also revealed less rigid (more flexible) syntactic coordination, although there were no changes in the relative contribution of teacher and students to this. In the light of the intervention under investigation this is an important result. This means that teachers and students learn to use more complex language and coordinate their language complexity better in order to co-construct science discourse. The application of CRQA provides new insights and contributes to better understanding of the dynamics of syntactic coordination.

PDF

Co-Adaptation processes of syntactic complexity in real-time kindergarten teacher-student interactions

product

Corporate Reputation of Companies on Twitter Seen from a Sustainability Perspective

Corporate reputation is becoming increasingly important for firms; social media platforms such as Twitter are used to convey their message. In this paper, corporate reputation will be assessed from a sustainability perspective. Using sentiment analysis, the top 100 brands of the Netherlands were scraped and analyzed. The companies were registered in the sustainable industry classification system (SICS) to perform the analysis on an industry level. A semantic search tool called Open Semantic Desktop Search was used to filter through the data to find keywords related to sustainability and corporate reputation. Findings show that companies that tweet more often about corporate reputation and sustainability receive overall a more positive sentiment from the public.

PDF

Corporate Reputation of Companies on Twitter Seen from a Sustainability Perspective

product

Causes of reporting bias: a theoretical framework [version 2; peer review: 2 approved]

Reporting of research findings is often selective. This threatens the validity of the published body of knowledge if the decision to report depends on the nature of the results. The evidence derived from studies on causes and mechanisms underlying selective reporting may help to avoid or reduce reporting bias. Such research should be guided by a theoretical framework of possible causal pathways that lead to reporting bias. We build upon a classification of determinants of selective reporting that we recently developed in a systematic review of the topic. The resulting theoretical framework features four clusters of causes. There are two clusters of necessary causes: (A) motivations (e.g. a preference for particular findings) and (B) means (e.g. a flexible study design). These two combined represent a sufficient cause for reporting bias to occur. The framework also features two clusters of component causes: (C) conflicts and balancing of interests referring to the individual or the team, and (D) pressures from science and society. The component causes may modify the effect of the necessary causes or may lead to reporting bias mediated through the necessary causes. Our theoretical framework is meant to inspire further research and to create awareness among researchers and end-users of research about reporting bias and its causes.

PDF

Causes of reporting bias: a theoretical framework [version 2; peer review: 2 approved]

product

Automatic recognition of self-acknowledged limitations in clinical research literature

Objective: To automatically recognize self-acknowledged limitations in clinical research publications to support efforts in improving research transparency.Methods: To develop our recognition methods, we used a set of 8431 sentences from 1197 PubMed Central articles. A subset of these sentences was manually annotated for training/testing, and inter-annotator agreement was calculated. We cast the recognition problem as a binary classification task, in which we determine whether a given sentence from a publication discusses self-acknowledged limitations or not. We experimented with three methods: a rule-based approach based on document structure, supervised machine learning, and a semi-supervised method that uses self-training to expand the training set in order to improve classification performance. The machine learning algorithms used were logistic regression (LR) and support vector machines (SVM).Results: Annotators had good agreement in labeling limitation sentences (Krippendorff's α = 0.781). Of the three methods used, the rule-based method yielded the best performance with 91.5% accuracy (95% CI [90.1-92.9]), while self-training with SVM led to a small improvement over fully supervised learning (89.9%, 95% CI [88.4-91.4] vs 89.6%, 95% CI [88.1-91.1]).Conclusions: The approach presented can be incorporated into the workflows of stakeholders focusing on research transparency to improve reporting of limitations in clinical studies.

PDF

product

Toward assessing clinical trial publications for reporting transparency

Objective: To annotate a corpus of randomized controlled trial (RCT) publications with the checklist items of CONSORT reporting guidelines and using the corpus to develop text mining methods for RCT appraisal. Methods: We annotated a corpus of 50 RCT articles at the sentence level using 37 fine-grained CONSORT checklist items. A subset (31 articles) was double-annotated and adjudicated, while 19 were annotated by a single annotator and reconciled by another. We calculated inter-annotator agreement at the article and section level using MASI (Measuring Agreement on Set-Valued Items) and at the CONSORT item level using Krippendorff's α. We experimented with two rule-based methods (phrase-based and section header-based) and two supervised learning approaches (support vector machine and BioBERT-based neural network classifiers), for recognizing 17 methodology-related items in the RCT Methods sections. Results: We created CONSORT-TM consisting of 10,709 sentences, 4,845 (45%) of which were annotated with 5,246 labels. A median of 28 CONSORT items (out of possible 37) were annotated per article. Agreement was moderate at the article and section levels (average MASI: 0.60 and 0.64, respectively). Agreement varied considerably among individual checklist items (Krippendorff's α= 0.06–0.96). The model based on BioBERT performed best overall for recognizing methodology-related items (micro-precision: 0.82, micro-recall: 0.63, micro-F1: 0.71). Combining models using majority vote and label aggregation further improved precision and recall, respectively. Conclusion: Our annotated corpus, CONSORT-TM, contains more fine-grained information than earlier RCT corpora. Low frequency of some CONSORT items made it difficult to train effective text mining models to recognize them. For the items commonly reported, CONSORT-TM can serve as a testbed for text mining methods that assess RCT transparency, rigor, and reliability, and support methods for peer review and authoring assistance. Minor modifications to the annotation scheme and a larger corpus could facilitate improved text mining models. CONSORT-TM is publicly available at https://github.com/kilicogluh/CONSORT-TM.

PDF

product

Sentiment polarity classification of corporate review data with a bidirectional Long-Short Term Memory (biLSTM) neural network architecture

A considerable amount of literature has been published on Corporate Reputation, Branding and Brand Image. These studies are extensive and focus particularly on questionnaires and statistical analysis. Although extensive research has been carried out, no single study was found which attempted to predict corporate reputation performance based on data collected from media sources. To perform this task, a biLSTM Neural Network extended with attention mechanism was utilized. The advantages of this architecture are that it obtains excellent performance for NLP tasks. The state-of-the-art designed model achieves highly competitive results, F1 scores around 72%, accuracy of 92% and loss around 20%.

PDF

Sentiment polarity classification of corporate review data with a bidirectional Long-Short Term Memory (biLSTM) neural network architecture

product

Automatic categorization of self-acknowledged limitations in randomized controlled trial publications

Objective:Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications.Methods:We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale.Results:Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance ().Conclusion:The model could support automated screening tools which can be used by journals to draw the authors’ attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.

MULTIFILE

Zoekresultaten

Producten 114

Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study

Learning about racism: A week-by-week qualitative exploration of two white Dutch primary student teachers’ emotional responses during a Critical Race Theory based course

Co-Adaptation processes of syntactic complexity in real-time kindergarten teacher-student interactions

Corporate Reputation of Companies on Twitter Seen from a Sustainability Perspective

Causes of reporting bias: a theoretical framework [version 2; peer review: 2 approved]

Automatic recognition of self-acknowledged limitations in clinical research literature

Toward assessing clinical trial publications for reporting transparency

Sentiment polarity classification of corporate review data with a bidirectional Long-Short Term Memory (biLSTM) neural network architecture

Automatic categorization of self-acknowledged limitations in randomized controlled trial publications

Navigeer naar

Categorieën

Filters

Producten 114

Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study

Learning about racism: A week-by-week qualitative exploration of two white Dutch primary student teachers’ emotional responses during a Critical Race Theory based course

Co-Adaptation processes of syntactic complexity in real-time kindergarten teacher-student interactions

Corporate Reputation of Companies on Twitter Seen from a Sustainability Perspective

Causes of reporting bias: a theoretical framework [version 2; peer review: 2 approved]

Automatic recognition of self-acknowledged limitations in clinical research literature

Toward assessing clinical trial publications for reporting transparency

Sentiment polarity classification of corporate review data with a bidirectional Long-Short Term Memory (biLSTM) neural network architecture

Automatic categorization of self-acknowledged limitations in randomized controlled trial publications