A Study Based on Large Language Model (2024)

2.1 Hierarchical Data Transformation

Data transformation in the ConferenceQA dataset involves standardizing the diverse formats of academic conference data sourced from official conference websites into a unified tree structure. Each conference page combines unstructured text, like conference introductions and paper submission guidelines, with structured data such as payment and schedule details. To manage this format variability, we employ a semi-automated method to create tree-structured data for each conference.

Specifically, the automated component converts structured table data into a tree format using ChatGPT, as shown in Fig.1, where registry information is transformed. For other structured data, such as accepted papers with consistent schemas (title, authors, abstract), we employ web crawlers to fetch HTML pages and convert them into corresponding tree-structured data based on the HTML tags. The manual component involves annotating inter-page relationships. Annotators assign page titles to tree nodes based on the linkage among pages, evident in navigation bars and subpage links like ‘calls’, ‘proceedings’ and ‘programs’. Additionally, subtitles within pages are identified and designated as child nodes under the relevant page titles. These manual steps are essential to maintain the dataset’s quality and coherence.

Ultimately, we obtain seven conference datasets organized in a tree-structured format. They are served as accurate and rigorous knowledge sources.

2.2 QA Pair Generation

This step involves generating reliable question-answer pairs through role creation, LLM-generated questions, and manual annotation. For each conference, we utilize ChatGPT to simulate the roles of conference participants, generating relevant questions which are then manually filtered and annotated with answers and their sources to ensure realism and reliability.

We use ChatGPT to create 20 roles characterized by specific attributes such as age, research direction, position, publication history, and conference attendance experience, mimicking real-life researchers with diverse backgrounds interested in the conferences.With these roles, we prompt ChatGPT to engage in role-playing scenarios, generating five varied questions per conference. These questions cover different areas of interest or uncertainty relevant to the roles’ diverse backgrounds. To avoid redundancy and enhance question diversity, we iteratively prompt the model. Specifically, we use the results generated by the ChatGPT as examples for the next iteration and encourage the ChatGPT to generate more diverse questions. In the final step, we manually review and filter the questions to eliminate duplicates and unrealistic queries. We then annotate the answers based on our tree-structured data, ensuring the reliability of the dataset by documenting the source of each answer within the constructed academic conference data.

2.3 Question Classification

To assess the model’s capability in handling questions of varying difficulty, we design a scheme to classify the question-answer pairs based on two criteria: the method used to generate the answer and the complexity of paths required to arrive at the correct answer.

Extraction vs. Reasoning: This category evaluates the process of answer generation. Answers directly pulled from the dataset are labeled as extraction, whereas answers that necessitate reasoning beyond the dataset content are labeled as reasoning. Reasoning questions are more challenging than extraction questions because, unlike direct extraction, reasoning questions require the model to have the capability to infer the relationship between the retrieved paths and the question.

Atomic vs. Complex: This category assesses the complexity of paths needed to generate the answer. Answers that depend on a single path are termed atomic, while those requiring multiple paths are termed complex. Complex questions are more difficult than atomic questions because, instead of a single path, complex questions require recalling multiple paths to derive an answer.

Combining these dimensions results in four levels of difficulty: extraction-atomic, extraction-complex, reasoning-atomic, and reasoning-complex. This classification is vital for analyzing the model’s performance across different complexities and reasoning demands.

2.4 Dataset Validation

3.1 Tree-structured Data Processing

The tree-structured data is hierarchically arranged, with each node representing a page or a section heading, and each leaf node corresponding to its specific content. For retrieval, we pair each leaf node with its root node to provide additional context to the LLM. Paths in the tree use the ‘>>’ field to denote hierarchical relationships and contain both structural and semantic information. An example path is: WWW2023>>Attendees>>Registration>>Register Fee>>Virtual Conference>>ACM Members>>$300.After the tree-structured data processing, the knowledge source for retrieval could be represented as a set of paths that $\mathcal{P}=\{p_{1},p_{2},...,p_{m}\}$ where $m$ is the number of paths in the dataset.

3.2 Path Retrieval

Upon receiving a query input $q$, the retriever selects a subset of paths from $\mathcal{P}=\{p_{1},p_{2},...,p_{m}\}$ that are relevant to $q$.Following established methods [20], we use a dense retriever based on a dual encoder framework. This framework employs an encoder to transform both the query $q$ and each path$p\in\mathcal{P}$ into embeddings. The similarity between the query and path embeddings is assessed using cosine similarity, and the top-$k$ paths with the highest similarity scores are retrieved, as expressed in (1), where E denotes the embedding function.

3.3 Structure-aware Retrieval

The limitation of treating a single path as the retrieval object is that it disconnects the structural relationships among paths. For example, the relationship between an author’s name and their affiliated institution is lost when paths are retrieved independently.

To overcome this, we introduce a novel method called STAR (STructure-Aware Retrieval). As shown in Fig.2, STAR employs ChatGPT to iteratively generate textual descriptions for each path $des_{p}$, from the root to individual nodes, in a top-down manner. We enhance the retrieval process by incorporating structural information in the user input, which includes siblings, parent path descriptions, and the query path itself. This approach helps maintain the contextual relevance of each path, which is crucial for recognizing relationships like those between an author and their institution. For instance, when generating path descriptions, we not only consider the node’s immediate context but also integrate the structural significance of related nodes. This includes the siblings of a node and their parent nodes, ensuring a comprehensive representation of each path’s context. To avoid the loss of information about the siblings of leaf nodes, we append the text of their parent node to each sibling of the leaf nodes. Ultimately, this method effectively preserves and utilizes structural relationships, enhancing the retrieval process.

4.1 Experimental Details

Based on the constructed ConferenceQA, we use currently popular LLMs, including Bloom (7B) [31], GPT-J (6B) [30], Flan-T5 (xl and xxl) [29], LLaMA2 (7B and 13B) [7], Mistral (7B) [25] and ChatGPT, as the main evaluation backbone to assess the performance of mainstream LLMs. For ChatGPT, we employ GPT-3.5-turbo and access it via API¹¹1 from https://api.openai.com/.We employ BM25[1], SentenceBert[26], DPR[27], ANCE[28] and text-embedding-ada-002 as our retriever.In addition, we use Chroma²²2https://github.com/chroma-core/chroma as our vector database and employ cosine similarity for matching. In all experiments, we select the top 5 paths retrieved.

4.2 Evaluation Metrics

In line with prior studies, we assess the QA capabilities of LLMs using the F1 score and the exact match (EM) score. Specifically, we employ GPT-4 to compute the EM, referred to as EM-GPT4.

The F1 score quantifies the overlap between the predicted and correct answers by calculating the harmonic mean of precision and recall.

The EM-GPT4 score evaluates the proportion of instances where the LLM’s predicted answer exactly matches the correct answer. Given the generative nature of LLMs, slight textual variations in responses might still represent the same answer. We use GPT-4, a highly advanced LLM known for its semantic understanding capabilities, to precisely assess if the LLM’s response matches the golden answers.

4.3 Experimental Results Analysis

Effect of Different LLMs We analyzed the performance of various LLMs on different types of questions to understand their perception capabilities and limitations. The results, shown in Table 2, provide several insights: (1) Our STAR method significantly improves the answering performance across various LLMs. For instance, on models like Bloom-7B1, GPT-J-6B, and GPT-3.5-turbo, F1 scores increased by 4%, 14.9%, and 25.04% respectively, while EM-GPT4 scores improved by 0.04%, 24.65%, and 24.94%. The least improvement was on Bloom-7B1, suggesting its inherent limitations. However, substantial gains on other models demonstrate our method’s effectiveness. (2) There is an inconsistency between F1 and EM-GPT4 scores; lower F1 scores sometimes align with higher EM-GPT4 scores. This may be due to LLMs generating longer textual responses, affecting F1 accuracy but not EM-GPT4, which better evaluates semantic similarity. (3) The complexity of question types affects performance; atomic questions are simpler than complex ones. Atomic questions, akin to single-hop queries, generally show higher accuracy than multi-hop complex questions. Despite this, LLMs perform comparably or better on reasoning questions than on extraction, likely due to their robust contextual learning and reasoning capabilities. (4) Different LLMs show varied understanding of paths. For example, under the same retrieval conditions, Mistral-7B outperforms GPT-3.5-turbo. Generally, models with more parameters, like LLama2-13B and Flan-T5-xxl, achieve higher accuracy, supporting the notion that larger LLMs perform better.

Effect of Different Retrievers We evaluated four retrievers—BM25 [1], SentenceBert [26], DPR [27], and ANCE [28]—using the gpt-3.5-turbo generator across four question types within the ConferenceQA dataset. The results, detailed in Table 3, reveal: (1) BM25 showed weak performance, especially with extraction-atomic and reasoning-complex questions. In contrast, dense retrievers like SentenceBERT, DPR, and ANCE significantly outperformed BM25, underscoring the advantages of dense retrieval methods. (2) Performance varied among dense retrievers: SentenceBERT was effective in extraction-atomic questions but less so in reasoning-atomic questions. DPR excelled in reasoning-atomic questions, while ANCE showed consistent performance across all question types. This indicates that selecting an appropriate retriever can significantly impact question-answering effectiveness. (3) While STAR occasionally had negative effects in some configurations, it generally enhanced performance across most settings, demonstrating its utility and reliability.

Effect of Different Conference Fig.3 shows the performance across various conferences using text-embedding-ada-002 and gpt-3.5-turbo as the retriever and generator, respectively. Key observations include: (1) There is notable variability in question difficulty across conferences, highlighting the diversity of our dataset. (2) Significant differences in difficulty are apparent between conferences; for example, the average EM-GPT4 score at ICML is 94.9% higher than at ACL,underscoring the importance of accounting for conference-specific characteristics in question-answering research.(3) Except for reasoning-atomic questions at SIGMOD and reasoning-complex questions at ISWC, our STAR method consistently outperforms traditional path retrieval, demonstrating its versatility and effectiveness across different conferences and question types.