Abstract

Benchmarks play a crucial role in evaluating system components, comparing algorithmic advancements, and assessing software implementations. While well‑established benchmark suites exist for various domains, modern memory technologies, such as high‑bandwidth memory and near‑memory computing, lack standardized benchmarking methodologies. Researchers often resort to custom benchmarks or adapt existing ones, making comparative analyses across different technologies challenging. This survey provides a structured overview of benchmarks used in modern memory architectures, catering to both newcomers and experienced researchers. It builds on top of a structured literature review of 1937 benchmark‑related publications obtained via Google Scholar in 2024, out of which 940 were within the scope of this survey, covering a total of 834 unique benchmarks. Out of these, we categorize and analyze 17 proposed and 739 existing benchmarks by manual analysis, identify trends in their adoption, and highlight gaps in the benchmarking landscape. By offering insights into benchmarking methodologies and their evolution, this survey aims to guide researchers in selecting appropriate benchmarks for memory system evaluations, fostering consistency and comparability in future research.

Benchmarks play a crucial role in evaluating system components, comparing algorithmic advancements, and assessing software implementations. While well‑established benchmark suites exist for various domains, modern memory technologies, such as high‑bandwidth memory and near‑memory computing, lack standardized benchmarking methodologies. Researchers often resort to custom benchmarks or adapt existing ones, making comparative analyses across different technologies challenging. This survey provides a structured overview of benchmarks used in modern memory architectures, catering to both newcomers and experienced researchers. It builds on top of a structured literature review of 1937 benchmark‑related publications obtained via Google Scholar in 2024, out of which 940 were within the scope of this survey, covering a total of 834 unique benchmarks. Out of these, we categorize and analyze 17 proposed and 739 existing benchmarks by manual analysis, identify trends in their adoption, and highlight gaps in the benchmarking landscape. By offering insights into benchmarking methodologies and their evolution, this survey aims to guide researchers in selecting appropriate benchmarks for memory system evaluations, fostering consistency and comparability in future research.

Abstract

The non-excludability and non-rivalry characteristics of data pose challenges to Coase’s property rights theory. Based on Ostrom’s Common Pool Resources (CPR) Governance Theory, this study proposes defining the data factor as a six-tuple and designing a computational environment that supports the protection of data factor property rights. This definition integrates the original dataset, transaction entities, commitments, supervision, and punishment into a unified whole, specifically referred to as data factor. Similar to the intellectual property protection system, the six-tuple first endows data factors with excludability in a contractual sense. Secondly, to effectively support such exclusivity, this study proposes a distributed property rights protection computing environment with the capabilities of “supervision-punishment-commitment”, adopting the “terminal-pipe-cloud” architectural template. This computing environment better matches the self-organizing nature of data trading activities. We use evolutionary game theory to verify how the new data factor definition promotes the development of trading markets. To provide practical reference, the simulation focuses on two key aspects: market stability under the coexistence of multiple strategies, and the impact of the initial conditions of the institutional environment on participants’ strategy choices.

The non-excludability and non-rivalry characteristics of data pose challenges to Coase’s property rights theory. Based on Ostrom’s Common Pool Resources (CPR) Governance Theory, this study proposes defining the data factor as a six-tuple and designing a computational environment that supports the protection of data factor property rights. This definition integrates the original dataset, transaction entities, commitments, supervision, and punishment into a unified whole, specifically referred to as data factor. Similar to the intellectual property protection system, the six-tuple first endows data factors with excludability in a contractual sense. Secondly, to effectively support such exclusivity, this study proposes a distributed property rights protection computing environment with the capabilities of “supervision-punishment-commitment”, adopting the “terminal-pipe-cloud” architectural template. This computing environment better matches the self-organizing nature of data trading activities. We use evolutionary game theory to verify how the new data factor definition promotes the development of trading markets. To provide practical reference, the simulation focuses on two key aspects: market stability under the coexistence of multiple strategies, and the impact of the initial conditions of the institutional environment on participants’ strategy choices.

Abstract

We introduce ContextFidelity‑Bench, a reproducible benchmark and evaluation framework for studying hallucination‑related behavior in multi‑turn language model conversations. The framework comprises three paradigms: numeric accuracy across a 20‑turn financial analysis that ends with a data correction (P1), constraint satisfaction in a 20‑turn scheduling task whose requirements accumulate, are revised, and may become jointly unsatisfiable (P2), and source fidelity in a multi‑document synthesis with planted contradictions, information gaps, and authority manipulations (P3). The released artifact includes scenario generators with pre‑computed ground truth and verified feasibility labels, scoring scripts with a documented protocol version, prompt templates, and 550 conversations from five API‑accessible models, of which 4,250 P1 answers and 800 P2 checkpoints are scored automatically and 100 P3 syntheses are judged. An initial case study illustrates the framework’s diagnostic use. Numeric accuracy and feasible‑plan success rank the five models similarly (Spearman \(\rho=0.90\)), whereas source‑fidelity abstention ranks them differently: the model with the lowest numeric accuracy (0.880) has the highest gap abstention (0.975) and no fabrication, while the model that produces complete valid plans most often (24 of 29 feasible post‑revision checkpoints) has the joint‑lowest gap abstention (0.875). When requirements are jointly unsatisfiable, models differ in whether they say so and in whether they still supply a plan; when a new requirement merely appears to conflict, some models raise false alarms. These observations come from a five‑model, single‑run snapshot and are presented as hypothesis‑generating; the framework is designed for replication and extension.

We introduce ContextFidelity‑Bench, a reproducible benchmark and evaluation framework for studying hallucination‑related behavior in multi‑turn language model conversations. The framework comprises three paradigms: numeric accuracy across a 20‑turn financial analysis that ends with a data correction (P1), constraint satisfaction in a 20‑turn scheduling task whose requirements accumulate, are revised, and may become jointly unsatisfiable (P2), and source fidelity in a multi‑document synthesis with planted contradictions, information gaps, and authority manipulations (P3). The released artifact includes scenario generators with pre‑computed ground truth and verified feasibility labels, scoring scripts with a documented protocol version, prompt templates, and 550 conversations from five API‑accessible models, of which 4,250 P1 answers and 800 P2 checkpoints are scored automatically and 100 P3 syntheses are judged. An initial case study illustrates the framework’s diagnostic use. Numeric accuracy and feasible‑plan success rank the five models similarly (Spearman \(\rho=0.90\)), whereas source‑fidelity abstention ranks them differently: the model with the lowest numeric accuracy (0.880) has the highest gap abstention (0.975) and no fabrication, while the model that produces complete valid plans most often (24 of 29 feasible post‑revision checkpoints) has the joint‑lowest gap abstention (0.875). When requirements are jointly unsatisfiable, models differ in whether they say so and in whether they still supply a plan; when a new requirement merely appears to conflict, some models raise false alarms. These observations come from a five‑model, single‑run snapshot and are presented as hypothesis‑generating; the framework is designed for replication and extension.