SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty.

SpreadsheetBench comprising 912 instructions and 2,729 test cases, with an average of three test cases per instruction. The instructions in our benchmark cover a broad spectrum of spreadsheet manipulation types, including find, extract, sum, highlight, remove, modify, count, delete, calculate, and display. The spreadsheet files in our benchmark contain tabular data with various row size, column size, number of table and table formats.

Table 1 compares SpreadsheetBench to other spreadsheet manipulation benchmarks. Our questions are sourced exclusively from real-world data and exhibits a higher average word count per instruction. Our spreadsheet files contain multiple sheets with non-standard relational tables and multiple tables within a single sheet. Real-world questions often involve additional explanations within the spreadsheet, a characteristic not present in previous benchmarks. Furthermore, we employ OJ-style evaluation metrics with three test cases per instruction.

SpreadsheetBench incorporates complex questions based on real-world scenarios and diverse types of tables in spreadsheet files, compared to previous benchmarks. We present four data examples in SpreadsheetBench to illustrate the attributes of real-world problems and the challenging of our benchmark.

Example 1: A cell-level manipulation example question involves manipulating a non-standard relational table (missing header on column D and column E).

This example shows a cell-level manipulation data example which aims to extract a specific part of a text string from a column of cells. The instruction contains the demand of the user and an example manipulating action of the question, which rarely occur in synthetic instructions of the previous benchmarks. Furthermore, the table within the spreadsheet file is a non-standard relational table that lacks a complete table header. The final result is required to be filled in the cells from B3 to B14, which ensure the uniqueness of the answer.

Example 2: A sheet-level manipulation example question involves manipulating multiple tables in one sheet (first table from A1 to B22; second table from E1 to T22).

This example shows a sheet-level manipulation data example aimed at creating a grid for a duty assignment schedule. The instruction contains the demand of the user, the current formula solution and the error encountered by the user. Besides, the spreadsheet file contains two relational tables in one sheet. The final result should be filled in a two dimensional position (i.e., F2 to T22).

Example 3: A sheet-level manipulation example question involves manipulating multiple tables on multiple sheets (first table in sheet named BUYING, from A1 to J17; second table in sheet named SH2, from A4 to D10).

This example shows a sheet-level manipulation data example aims to search across sheets based on a sheet name and two dates (i.e., first date and last date). The instruction contains a complex requirements, a detailed explanation, a manipulation example and the error encountered by the user. Moreover, the spreadsheet file contains multiple sheets, including the table to be manipulated (e.g., sheets named BUYING or SALES) and examples of possible answers (e.g., sheets named CASE1 or CASE2). The final result should be inserted into a two-dimensional location on the sheet named SH2.

Example 4: A test case modified from the question in Example 3.

This example shows a test case modified from the question in Example 3. This test case is constructed following the user's requirements in the instruction. The user expects to obtain a result that will search within a specific sheet name (cell G2). Only the rows with a date falling between the first date (cell C2) and last date (Cell E2) should be returned. Therefore, we modify the value of the sheet name (cell G2) to construct a new test case, as highlighted in yellow. This test case yields search results in the SALES sheet, rather than the original case that searches in the BUYING sheet. The values in the answer position also change, as highlighted in red. By modifying the sheet name, we can alter the sheet that is being searched, enabling a more comprehensive assessment of the solution's effectiveness.

We evaluate LLMs across five categories: (1) TableQA models (e.g., Binder), (2) Open-source code models (e.g., DeepseekCoder), (3) Open-source general models (e.g., Llama 3), (4) Close-source models (e.g., GPT-4), and (5) Spreadsheet-specific methods or products (e.g., SheetCopilot).

We evaluate LLMs under two distinct settings: 1. Single Round: In this mode, we present the model with the initial few rows of spreadsheet files within the prompt, allowing for only one inference. 2. Multi-Round: Building on the single-round prompt setting, we incorporate additional prompt that utilizes the ReAct technique and code execution feedback to enhance the accuracy of code solutions produced by LLMs over multi-round conversation.

The results shown in Table 2 indicate that current LLMs and spreadsheet agents are inadequate in managing complex spreadsheet manipulation tasks as required by real-world scenarios. Even the most advanced spreadsheet agent, Copilot in Excel, only achieves an accuracy of roughly 20%. GPT-4o, the SOTA LLM, scores around 17% in accuracy, aligning with Copilot in Excel's performance. Open-source LLMs significantly underperform compared to the SOTA model, likely due to their limited comprehension and coding proficiency. Overall, there is a substantial gap between existing LLMs or products and human performance produced by Excel experts, emphasizing the critical need for advancement in LLMs tailored for spreadsheet manipulation.