Benchmark

non-incremental/QF_BV/tacas07/s-40-50-bv.smt2 

This directory contains benchmarks used in the experimental results
reported in the following paper:

Title: Deciding Bit-Vector Arithmetic with Abstraction
Authors: R. E. Bryant, D. Kroening, J. Ouaknine, S. A. Seshia, O. Strichman, B. Brady

If you use these benchmarks, we will appreciate hearing from you by 
e-mail to sseshia@eecs.berkeley.edu. Additionally, the following sources 
must be acknowledged:

1) Y86_std.smt, Y86_btnft.smt: 
   Generated from a UCLID model created by R. E. Bryant, as described
   in the following paper:
    http://www.cs.cmu.edu/~bryant/pubdir/CMU-CS-05-195.pdf

2) s-40-50-bv.smt: 
   Generated by V. Ganapathy for finding format-string exploits in
   the WiSA project (http://www.cs.wisc.edu/wisa)
 
3) BBB-32.smt, rfunit_flat-64.smt:
   Generated from benchmarks generated by Z. Hanna of Intel.

All benchmarks are in the SMT-LIB format, as described at:
http://combination.cs.uiowa.edu/smtlib/
Benchmark
Size50764
Compressed Size4519
License Creative Commons Attribution 4.0 International (CC-BY-4.0)
Categoryindustrial
First Occurrence2007-07-03
Generated By—
Generated On—
Generator—
Dolmen OK1
strict Dolmen OK1
check-sat calls1
Query 1
Status sat
Inferred Status sat
Size 50756
Compressed Size4533
Max. Term Depth306
Asserts 1
Declared Functions0
Declared Constants101
Declared Sorts 0
Defined Functions0
Defined Recursive Functions 0
Defined Sorts0
Constants0
Declared Datatypes0

Symbols

not338 or693 and475 =1045
let7 extract43 bvneg1 bvadd132
bvmul1 bvsub4 bvult5

Evaluations

Evaluation Rating Solver Variant Result Wallclock CPU Time
SMT-COMP 2007 Spear Spear v1.9 (fh-1-2) sat ✅ 2.90000 —
Spear v1.9 (sw-v) sat ✅ 10.40000 —
Yices Yices 1.0.10 sat ✅ 175.30000 —
Z3 Z3 0.1 sat ✅ 3.20000 —
Z3 0.1 (fixed BV) sat ✅ 3.10000 —
SMT-COMP 2008 0.14 (6/7) Beaver Beaver-1.0 sat ✅ 3.80000 —
Boolector Boolector sat ✅ 13.50000 —
CVC3 CVC3-1.5 unknown ❌ 1801.00000 —
MathSAT MathSAT-4.2 sat ✅ 209.50000 —
Spear Spear sat ✅ 6.90000 —
Spear v1.9 (fh-1-2) sat ✅ 3.30000 —
SWORD SWORD v0.2 sat ✅ 26.50000 —
Z3 Z3.2 sat ✅ 5.80000 —
SMT Evaluation 2013 4Simp 4Simp-SMT-COMP-2012 default sat ✅ 2.05472 —
Abziz abziz_portfolio_2011_minfeatures_2 default sat ✅ 3.59880 —
abziz_portfolio_2011_solvers_2 default unknown ❌ — —
abziz_portfolio_2012_minfeatures default sat ✅ 3.59815 —
abziz_portfolio_2012_solvers default unknown ❌ — —
AbzizPortfolio-SMT-COMP-2012 default unknown ❌ — —
Boolector Boolector-1.5.118-SMT-EVAL-2013 default sat ✅ 6.91793 —
Boolector-SMT-COMP-2011 default unknown ❌ — —
Boolector-SMT-COMP-2012 default sat ✅ 2.12370 —
CVC3 CVC3-SMT-COMP-2010 default unknown ❌ — —
CVC3-SMT-COMP-2011 default sat ✅ 8.18646 —
CVC3-SMT-COMP-2012 default sat ✅ 8.48942 —
CVC4 CVC4-SMT-COMP-2012-Resubmission default sat ✅ 0.45356 —
CVC4-SMT-EVAL-2013 default unknown ❌ — —
MathSAT MathSAT5-5.2.6-SMT-EVAL-2013 default sat ✅ 0.52907 —
MathSAT5-HeavyBV-SMT-COMP-2012 default unknown ❌ — —
MathSAT5-SMT-COMP-2011 default sat ✅ 4.22868 —
MathSAT5-SMT-COMP-2012 default sat ✅ 0.51915 —
SONOLAR SONOLAR-2013-05-15-SMT-EVAL-2013 default sat ✅ 1.63886 —
SONOLAR-SMT-COMP-2010 default unknown ❌ — —
SONOLAR-SMT-COMP-2011 default sat ✅ 1.53730 —
SONOLAR-SMT-COMP-2012 default sat ✅ 1.83822 —
STP STP2-SMT-COMP-2011 default sat ✅ 2.04104 —
STP2-SMT-COMP-2012 default unknown ❌ — —
simplifyingSTP-SMT-COMP-2010 default sat ✅ 2.93605 —
Z3 Z3-4.3.2.a054b099c1d6-x64-debian-6.0.6-SMT-EVAL-2013 default sat ✅ 3.37766 —
Z3-SMT-COMP-2011 default sat ✅ 3.55776 —
SMT-COMP 2014 4Simp 4Simp - 2014 default sat ✅ 1.74009 1.73173
Abziz abziz_portfolio_all_features default sat ✅ 8.67086 8.65868
abziz_portfolio_min_features default sat ✅ 8.60315 8.59269
Boolector Boolector boolector sat ✅ 21.89920 21.89370
CVC4 CVC4 f7118b2 default sat ✅ 2.61867 5.18821
MathSAT MathSAT-5.2.12-Main default sat ✅ 4.44947 4.42933
SONOLAR sonolar_smtcomp-2014 default sat ✅ 1.74369 1.73973
STP stp-cryptominisat4 default sat ✅ 2.51309 2.50562
Yices2 Yices-2.2.1-smtcomp2014 default sat ✅ 0.82122 0.81088
Z3 Z3-4.3.2.a054b099c1d6-x64-debian-6.0.6-SMT-COMP-2014 default sat ✅ 4.61344 4.60830
SMT-COMP 2015 Boolector Boolector SMT15 QF_BV final boolector_qf_bv sat ✅ 14.50730 14.51180
CVC4 CVC4-master-2015-06-15-9b32405-main default sat ✅ 1.78663 3.53446
CVC4-experimental-2015-06-15-ff5745a-main default sat ✅ 3.29646 6.56300
MathSAT MathSat 5.3.6 main smtcomp2015_main sat ✅ 2.15962 2.15967
SMT-RAT SMT-RAT-final default sat ✅ 74.39160 74.40870
STP stp-cryptominisat4 default sat ✅ 2.62190 2.62160
stp-cmsat4-v15 default sat ✅ 3.01899 3.01954
stp-cmsat4-mt-v15 default unknown ❌ 0.52951 0.52892
stp-minisat-v15 default sat ✅ 1.62464 1.62375
Yices2 Yices default sat ✅ 0.22287 0.22196
Z3 z3 4.4.0 default sat ✅ 3.91882 3.92040
SMT-COMP 2016 0.10 (9/10) ABC ABC_default default_abc sat ✅ 29.32160 29.33780
ABC_glucose glucose sat ✅ 12.78590 12.77470
Boolector Boolector boolector sat ✅ 4.97540 4.97831
Boolector preprop boolector sat ✅ 5.07110 5.07340
CVC4 CVC4-master-2016-05-27-cfef263-main default sat ✅ 3.28713 6.55894
MapleSTP MapleSTP default sat ✅ 4.64734 4.65038
MapleSTP-mt default sat ✅ 5.79493 21.14590
MathSAT mathsat-5.3.11-linux-x86_64-Main default sat ✅ 1.53561 1.53830
MinkeyRink Minkeyrink 2016 default sat ✅ 6.34721 11.86730
Q3B Q3B default unknown ❌ 2400.13000 7203.73000
STP stp-cms-exp-2016 default sat ✅ 3.99683 3.99912
stp-cms-mt-2016 default sat ✅ 3.01902 10.30730
stp-cms-st-2016 default sat ✅ 3.87567 3.87792
stp-minisat-st-2016 default sat ✅ 22.15980 22.17260
Yices2 Yices-2.4.2 default sat ✅ 0.24379 0.24386
Z3 z3-4.4.1 default sat ✅ 5.03904 5.04322
SMT-COMP 2017 0.12 (7/8) Boolector Boolector+CaDiCaL SMT17 final boolector sat ✅ 5.56209 5.55888
Boolector SMT17 final boolector sat ✅ 5.99282 5.99171
CVC4 CVC4-smtcomp2017-main default sat ✅ 3.43057 6.84094
MathSAT mathsat-5.4.1-linux-x86_64-Main default sat ✅ 1.31514 1.31518
MinkeyRink MinkeyRink 2017.3a default sat ✅ 2.29133 2.29110
Q3B Q3B default unknown ❌ 600.03000 1631.75000
STP stp_st default sat ✅ 3.34743 3.34563
stp_mt default sat ✅ 2.84297 9.24376
Yices2 Yices2-Main default sat ✅ 0.16024 0.15926
Z3 z3-4.5.0 default sat ✅ 4.95164 4.95074
SMT-COMP 2018 Boolector Boolector_default sat ✅ 4.15103 4.15066
CVC4 master-2018-06-10-b19c840-competition-default_default sat ✅ 1.94774 1.94764
MathSAT mathsat-5.5.2-linux-x86_64-Main_default sat ✅ 1.99166 1.99253
MinkeyRink Minkeyrink MT_mt sat ✅ 0.99176 2.22815
Minkeyrink ST_st sat ✅ 1.04510 1.04522
STP STP-CMS-st-2018_default-no-stderr sat ✅ 1.44592 1.44599
STP-CMS-mt-2018_multicore-no-stderr sat ✅ 1.78187 5.27052
STP-Riss-st-2018_riss-no-stderr sat ✅ 3.28308 3.28282
Yices2 Yices 2.6.0_default sat ✅ 0.41686 0.41669
Z3 z3-4.7.1_default sat ✅ 5.05291 5.05276
SMT-COMP 2019 Boolector Boolector-wrapped-sq_default sat ✅ 2.18979 2.18983
Poolector-wrapped-sq_default sat ✅ 2.50034 9.35000
Boolector-ReasonLS boolector-ReasonLS-wrapped-sq_default sat ✅ 5.11800 5.11755
CVC4 CVC4-2019-06-03-d350fe1-wrapped-sq_default sat ✅ 0.95017 0.95039
MinkeyRink MinkeyRink MT-wrapped-sq_default sat ✅ 0.71167 1.30027
MinkeyRink ST-wrapped-sq_default sat ✅ 1.30544 1.30544
Par4 Par4-wrapped-sq_default sat ✅ 0.08152 0.00591
STP STP-2019-wrapped-sq_default sat ✅ 0.83525 0.83558
stp-mergesat-fixed-wrapped-sq_default sat ✅ 21.23680 21.23610
stp-minisat-wrapped-sq_default sat ✅ 2.03534 2.03566
stp-mt-wrapped-sq_default sat ✅ 0.97241 2.01788
stp-portfolio-fixed-wrapped-sq_default sat ✅ 21.17920 23.21200
stp-riss-wrapped-sq_default unknown ❌ 0.62578 0.61021
Yices2 Yices 2.6.2-wrapped-sq_default sat ✅ 0.09450 0.09451
Yices 2.6.2 Cadical-wrapped-sq_default sat ✅ 2.67457 2.67420
Yices 2.6.2 Cryptominisat-wrapped-sq_default sat ✅ 2.44220 2.43929
Yices 2.6.2 MCSAT BV-wrapped-sq_default sat ✅ 1.52740 1.52726
Yices 2.6.2 new bvsolver-wrapped-sq_default sat ✅ 0.32171 0.32172
Z3 z3-4.8.4-d6df51951f4c-wrapped-sq_default sat ✅ 1.86441 1.86425
SMT-COMP 2021 Bitwuzla Bitwuzla-fixed_default sat ✅ 2.30112 2.30092
MathSAT mathsat-5.6.6_default sat ✅ 1.61139 1.61111
STP STP 2021.0_default sat ✅ 2.59376 8.70800
Z3 z3-4.8.11_default sat ✅ 4.08290 4.08274
SMT-COMP 2022 Bitwuzla Bitwuzla-fixed_default sat ✅ 2.36195 2.36180
Bitwuzla-wrapped_default sat ✅ 2.72609 2.72589
cvc5 cvc5-default-2022-07-02-b15e116-wrapped_sq sat ✅ 0.97293 0.97346
MathSAT MathSAT-5.6.8_default sat ✅ 0.58200 0.58188
STP STP 2022.4_default sat ✅ 3.07142 3.06783
Yices2 Yices 2.6.2 for SMTCOMP 2021_default sat ✅ 1.10059 1.10053
Z3++BV z3++bv_0702_default sat ✅ 1.15109 1.15088
Z3 z3-4.8.17_default sat ✅ 7.30806 7.30894
SMT-COMP 2023 Bitwuzla Bitwuzla-fixed_default sat ✅ 2.31479 2.31346
cvc5 cvc5-default-2023-05-16-ea045f305_sq sat ✅ 1.94913 1.94968
STP STP 2022.4_default sat ✅ 1.30850 1.30838
STP 2022.4_default sat ✅ 1.31129 1.31133
UltimateEliminator UltimateIntBlastingWrapper+SMTInterpol_default sat ✅ 98.21150 133.62700
Yices2 Yices 2 for SMTCOMP 2023_default sat ✅ 0.31411 0.31399
Z3-Owl z3-Owl-Final_default sat ✅ 21.56020 21.12450
z3-Owl-Final_default sat ✅ 20.79390 20.78510
SMT-COMP 2025 Bitwuzla Bitwuzla sat ✅ 0.64569 0.52706
Bitwuzla-MachBV-base sat ✅ 1.18430 1.06954
Bitwuzla-MachBV Bitwuzla-MachBV sat ✅ 1.22196 1.10384
BVDecide bv_decide sat ✅ 12.04246 11.87892
bv_decide-nokernel sat ✅ 11.93812 11.80134
cvc5 cvc5 sat ✅ 0.68616 0.56017
SMTInterpol SMTInterpol sat ✅ 7.10153 19.15669
Yices2 Yices2 sat ✅ 1.68407 1.56286
Z3alpha Z3-alpha sat ✅ 2.28042 7.65874
Z3 Z3-alpha-base sat ✅ 1.38654 1.26129
Z3-Owl-base sat ✅ 4.75026 4.62787
z3siri-base sat ✅ 1.36898 1.24826
Z3-Owl Z3-Owl sat ✅ 8.73434 8.61146