Telescope Reference and Astronomy Categorization Shared task (TRACS)

Quicklinks:

  • dataset and competition on Kaggle
  • registration for ICNLP-AACL here
  • registration for TRACS here
  • ACL LaTeX template on Github
  • submission link on OpenReview.
  • mirror link of the dataset on Huggingface (different formating, for archival purposes)

Introduction and Motivation

A well-established way to assess the scientific impact of an observational facility in astronomy is the quantitative analysis of the studies published in the literature which have made use of the data taken by the facility. A requirement of such analysis is the creation of bibliographies which annotate and link data products with the literature, thus providing a way to use bibliometrics as an impact measure for the underlying data. Creating such links and bibliographies is a laborious process which involves specialists searching the literature for names, acronyms and identifiers, and then determining how observations were used in those publications, if at all Observatory Bibliographers Collaboration, 2024.
The creation of such links represents more than just a useful way to generate metrics: doing science with archival data depends on being able to critically review prior studies and then locate the data used therein, a basic tenet behind the principle of scientific reproducibility. From the perspective of a research scientist, the data-literature connections provide a critical path to data discovery and access. Thus, by leveraging the efforts of librarians and archivists, we can make use of telescope bibliographies to support the scientific inquiry process. We wish to make the creation of such bibliographies simpler and more consistent by using AI technologies to support the efforts of data curators.

Typical Curation Process

While different groups use different approaches and criteria to the problem of bibliography creation and maintenance, the steps involved typically consist of the following:

  1. Use a set of full-text queries to the ADS bibliographic database in order to find all possible relevant papers. This first step aims to identify articles that contain mention of the telescope/instrument of interest so that they can be further analyzed. For instance, the set of query terms used to find papers related to the Chandra X-Ray telescope may be “Chandra,” “CXC,” “CXO,” “AXAF,” etc.
  2. Analyze the text containing mentions of the telescope/instrument and its variations in order to disambiguate the use of the terms of interest. For the Chandra example, this includes teasing apart the different entities associated with “Chandra,” which may correspond to a person, a ground-based telescope, or a space-based telescope.
  3. Identify whether the paper in question shows evidence of the use of datasets generated by the telescope or hosted by the archive of interest. The mention of data use may be explicit (e.g. the listing of dataset identifiers), or implied in the text (e.g. mention of analysis and results without identification of the actual dataset). Whenever dataset ids are used, they should be extracted and identified.
  4. In some cases, additional classification of the dataset may be collected, such as the instrument used in the observations. This information is also correlated with the kind of data that was used (e.g. image vs. spectra vs. catalog) and its characteristics. In the case of Chandra, there are 7 different instruments that can be used for the data collection (ACIS, HRC, HETG, LETG, HRMA, PCAD, EPHIN), and their use, if explicitly mentioned in the paper, should be reported.
  5. For some bibliographies, additional information is collected, such as the relevance of the paper to the scientific use of the data archive. For example, for the Chandra bibliography, the following categories are defined:
    1. Direct use of Chandra data
    2. Refers to published results
    3. Predicts Chandra results
    4. Paper on Chandra software, operations, and/or instrumentation
    5. General reference to Chandra

Task

An automated assistant able to emulate the supervised curation activities listed in the steps 2-5 above would provide a valuable contribution to the human effort involved. LLMs have shown flexibility in interpreting and classifying scientific articles which are the basis for this curation activity. They have also been successfully used for information extraction tasks, which would help identify the specific datasets mentioned in the papers. This shared task aims at improving the state of the art technologies to support these curation efforts.

We at WASP-2025 challenge the community to build such automated assistants, especially if they can be applied to new telescopes.
(see Instructions for Participants for a precise description of the task)

Dataset

We provide a dataset of scientific papers from SciX annotated with their associated telescope, categorization, and metadata.
The dataset and competition are on Kaggle here.

Dataset Description

The dataset entries consists of the following features:

  • "bibcode": unique string that identifies the entry in the SciX database
  • "telescope": the telescope referenced in the entry
  • "science", instrumentation, "mention", "not_telescope": boolean labels for the entry
  • "author", "year": metadata for the entry
  • "title", "abstract", "body", "acknowledgments", "grants": the relevant textual information for the entry.

On Kaggle, an additional Id column is present.

The "science", instrumentation, "mention", "not_telescope" labels describe how the paper use the data from the telescope. Broadly, "science" papers use data from the designated telescope to obtain new results; instrumentation papers describe the technical aspects of the telescope; "mention" papers do reference the designated telescope but do not produce new scientific results; and "not_telescope" are papers that include a reference that might otherwise be confused with the designated telescope, i.e. false positives. For a more detailed description of each label, see here.

This dataset was created with the collaboration of:

  • Jenny Novacescu, Chief Librarian at the STSCi who helped with the data from the HST and JWST
  • Erin Scott, Astrophysicist from the CfA who provided the CHANDRA data
  • Anjali Pandiri, of Florida State University who helped curate the data

Challenge

Because new telescopes are deployed each year, we are especially interested in methods that can generalize and be applied to papers about telescopes not present in the training data.
The task for participants is thus :

  1. Identifying if a paper relates to a telescope
  2. Grouping papers by identified telescope
  3. Labeling the papers as one or more of "science", instrumentation, "mention", "not_telescope"

Instructions for Participants

  1. Participants should register for ICNLP-AACL here.
  2. Particpants should register for TRACS on this google form here.
  3. Participants should join the competition on Kaggle here, where they can find the latest version of the dataset, and score their submissions.
  4. Participants should format their papers using the ACL LaTeX template on Github here, and submit them to OpenReview here.

Participants are to used the data from the "title", "abstract", "body", "acknowledgments", "grants" features to build models.
These models should then make predictions and submit them to Kaggle in for the a .csv file with the following columns: Id,telescope,science,instrumentation,mention,not_telescope.
For example, a sample_submission.csv file could start with:

Id,telescope,science,instrumentation,mention,not_telescope
2012A&A...537A..18M_CHANDRA,JWST,True,False,False,False
1998SPIE.3356.1078P_CHANDRA,HST,False,False,True,False
2022arXiv220915324P_CHANDRA,HST,True,False,False,False

Evaluation

Submission are ranked by macro f1-score across the "telescope" feature and the type of label of the telescope ("science", instrumentation, "mention", "not_telescope").
The code for scoring is available here.
However we highly encourage participants to propose their own evaluation metrics and use those to highlight the strength of their models.

Baseline

(more details to come)

Registration for TRACS

Please register for ICNLP-AACL 2025 here and to TRACS here to participate.
You should also register for Kaggle here.

Timeline

Timeline Date
1st CfP + Registration starts August 27, 2025
Train and Validation Data Release August 31, 2025
Test Set Release August 31, 2025
Registration Ends October 10, 2025
System Run and Output Submission October 13, 2025
System Paper Submisison October 20, 2025
Result Announcement October 22, 2025
Notification November 3, 2025
Camera Ready Submission November 11, 2025

Contact

for enquiries, contact Felix Grezes felix.grezes@cfa.harvard.edu