Author: Bakhmat M.
Evaluating the performance and capabilities of quantum computers is essential for understanding their progress, comparing different systems, and determining their potential for solving real-world problems. However, benchmarking quantum systems presents unique challenges compared to classical computers.
Challenges in Benchmarking
Traditional benchmarks focused on hardware limits provide limited insight into real-world application performance. The current focus often struggles to convey clear business value to non-quantum-literate industry leaders. There is a need for benchmarks that assess the entire quantum stack, including the interplay of hardware, software, and cloud services. Benchmarks also need to be linked to application performance and presented in a way that is easier to understand in a real-life context.
Proposed Benchmarking Approaches
Industry experts propose the establishment of application-based benchmarks that are independent of specific quantum stack providers. Such benchmarks would use:
- Standardized metrics that consider hidden costs between the quantum and classical layers.
- Real-world use-case-based datasets across various industries.
- Reproducible benchmarking methods.
- Results digestible for business leads with little to no quantum knowledge.
Key metrics for benchmarking could include:
- User-friendliness: Assessing ease of learning, expressibility (range of problems addressed), modulation level (packaging of subroutines), and transparency (documentation).
- Execution time: The time needed to run an algorithm, including repetitions.
- Time-to-Solution (TTS): A metric that accounts for mapping, solving, and processing the solution.
- Validity and Efficiency: Measuring if the solution conforms to constraints and how effective it is.
- Cost: Financial cost per job or energy cost.
Current Status and Future Outlook
Hardware benchmarking currently highlights the engineering challenges in achieving full-scale, fault-tolerant quantum computers across various qubit technologies. Companies report physical qubit counts, but achieving reliable logical qubits is the critical next step.
IBM has set challenges, such as the 100×100 Challenge, aiming to demonstrate the ability to calculate unbiased observables of circuits with 100 qubits and depth-100 gate operations in a reasonable runtime by 2024. They are confident this is achievable with processors like Heron if certain error rate thresholds are met. Such systems would be able to run circuits with complexity beyond the capabilities of the best classical computers today.
Effective benchmarking that moves beyond purely technical metrics to focus on practical application performance is crucial for the industry’s maturity and for guiding businesses in leveraging quantum technology.
