Evaluate Collaboratory - benchmarks

Evaluate Collaboratory - benchmarks http://evaluate.inf.usi.ch/taxonomy/term/74/0 en The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations http://evaluate.inf.usi.ch/node/380 <span class="biblio-authors"><a href="/biblio/author/4">Blackburn, SM</a>, <a href="/biblio/author/10">Diwan A</a>, <a href="/biblio/author/26">Hauswirth M</a>, <a href="/biblio/author/27">Sweeney PF</a>, <a href="/biblio/author/274">Amaral J N</a>, <a href="/biblio/author/275">Brecht T</a>, <a href="/biblio/author/276">Bulej L\'ır</a>, <a href="/biblio/author/277">Click C</a>, <a href="/biblio/author/3">Eeckhout L</a>, <a href="/biblio/author/278">Fischmeister S</a> et al.</span>.  2016.  <span class="biblio-title"><a href="/biblio/view/380">The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations</a>. </span>ACM Trans. Program. Lang. Syst.. 38:15:1–15:20.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.atitle=The+Truth%2C+The+Whole+Truth%2C+and+Nothing+But+the+Truth%3A+A+Pragmatic+Guide+to+Assessing+Empirical+Evaluations&rft.title=ACM+Trans.+Program.+Lang.+Syst.&rft.issn=0164-0925&rft.date=2016&rft.volume=38&rft.spage=15%3A1%E2%80%9315%3A20&rft.aulast=Blackburn&rft.aufirst=Stephen&rft.pub=ACM&rft.place=New+York%2C+NY%2C+USA&rft_id=info%3Adoi%2F10.1145%2F2983574"></span> http://evaluate.inf.usi.ch/node/380#comments accuracy benchmarks metrics reproducibility statistics Wed, 31 May 2017 11:23:23 +0000 Matthias Hauswirth 380 at http://evaluate.inf.usi.ch Engineering Academic Software (Dagstuhl Perspectives Workshop 16252) http://evaluate.inf.usi.ch/node/377 <span class="biblio-authors"><a href="/biblio/author/245">Allen, A</a>, <a href="/biblio/author/246">Aragon C</a>, <a href="/biblio/author/247">Becker C</a>, <a href="/biblio/author/248">Carver J</a>, <a href="/biblio/author/249">Chis A</a>, <a href="/biblio/author/250">Combemale B</a>, <a href="/biblio/author/251">Croucher M</a>, <a href="/biblio/author/252">Crowston K</a>, <a href="/biblio/author/253">Garijo D</a>, <a href="/biblio/author/254">Gehani A</a> et al.</span>.  2017.  <span class="biblio-title"><a href="/biblio/view/377">Engineering Academic Software (Dagstuhl Perspectives Workshop 16252)</a>. </span>Dagstuhl Manifestos. 6:1–20.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.atitle=Engineering+Academic+Software+%28Dagstuhl+Perspectives+Workshop+16252%29&rft.title=Dagstuhl+Manifestos&rft.issn=2193-2433&rft.date=2017&rft.volume=6&rft.spage=1%E2%80%9320&rft.aulast=Allen&rft.aufirst=Alice&rft.pub=Schloss+Dagstuhl%E2%80%93Leibniz-Zentrum+fuer+Informatik&rft.place=Dagstuhl%2C+Germany&rft_id=info%3Adoi%2F10.4230%2FDagMan.6.1.1"></span> http://evaluate.inf.usi.ch/node/377#comments benchmarks reproducibility Wed, 31 May 2017 11:13:45 +0000 Matthias Hauswirth 377 at http://evaluate.inf.usi.ch Rethinking Experimental Methods in Computing (Dagstuhl Seminar 16111) http://evaluate.inf.usi.ch/node/376 <span class="biblio-authors"><a href="/biblio/author/239">Delling, D</a>, <a href="/biblio/author/240">Demetrescu C</a>, <a href="/biblio/author/241">Johnson DS</a>, <a href="/biblio/author/101">Vitek J</a></span>.  2016.  <span class="biblio-title"><a href="/biblio/view/376">Rethinking Experimental Methods in Computing (Dagstuhl Seminar 16111)</a>. </span>Dagstuhl Reports. 6:24–43.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.atitle=Rethinking+Experimental+Methods+in+Computing+%28Dagstuhl+Seminar+16111%29&rft.title=Dagstuhl+Reports&rft.issn=2192-5283&rft.date=2016&rft.volume=6&rft.spage=24%E2%80%9343&rft.aulast=Delling&rft.aufirst=Daniel&rft.pub=Schloss+Dagstuhl%E2%80%93Leibniz-Zentrum+fuer+Informatik&rft.place=Dagstuhl%2C+Germany&rft_id=info%3Adoi%2F10.4230%2FDagRep.6.3.24"></span> http://evaluate.inf.usi.ch/node/376#comments accuracy benchmarks human factors metrics negative results observational studies reproducibility statistics Wed, 31 May 2017 11:10:51 +0000 Matthias Hauswirth 376 at http://evaluate.inf.usi.ch Rigorous Benchmarking in Reasonable Time http://evaluate.inf.usi.ch/node/364 <span class="biblio-authors"><a href="/biblio/author/102">Kalibera, T</a>, <a href="/biblio/author/211">Jones R</a></span>.  2013.  <span class="biblio-title"><a href="/biblio/view/364">Rigorous Benchmarking in Reasonable Time</a>. </span>Proceedings of the 2013 International Symposium on Memory Management. :63–74.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=Rigorous+Benchmarking+in+Reasonable+Time&rft.series=ISMM+%26%23039%3B13&rft.isbn=978-1-4503-2100-6&rft.date=2013&rft.spage=63%E2%80%9374&rft.aulast=Kalibera&rft.aufirst=Tomas&rft.pub=ACM&rft.place=New+York%2C+NY%2C+USA&rft_id=info%3Adoi%2F10.1145%2F2464157.2464160"></span> http://evaluate.inf.usi.ch/node/364#comments accuracy benchmarks reproducibility statistics Mon, 14 Mar 2016 21:54:50 +0000 Matthias Hauswirth 364 at http://evaluate.inf.usi.ch Languages and performance engineering: method, instrumentation, and pedagogy http://evaluate.inf.usi.ch/node/274 <span class="biblio-authors"><a href="/biblio/author/172">Lea, D</a>, <a href="/biblio/author/173">Bacon DF</a>, <a href="/biblio/author/174">Grove D</a></span>.  2008.  <span class="biblio-title"><a href="/biblio/view/274">Languages and performance engineering: method, instrumentation, and pedagogy</a>. </span>SIGPLAN Not.. 43:87–92.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.atitle=Languages+and+performance+engineering%3A+method%2C+instrumentation%2C+and+pedagogy&rft.title=SIGPLAN+Not.&rft.issn=0362-1340&rft.date=2008&rft.volume=43&rft.spage=87%E2%80%9392&rft.aulast=Lea&rft.aufirst=Doug&rft.pub=ACM&rft.place=New+York%2C+NY%2C+USA&rft_id=info%3Adoi%2F10.1145%2F1480828.1480848"></span> http://evaluate.inf.usi.ch/node/274#comments accuracy benchmarks metrics Wed, 25 Apr 2012 13:37:58 +0000 Matthias Hauswirth 274 at http://evaluate.inf.usi.ch Pitfalls in parallel job scheduling evaluation http://evaluate.inf.usi.ch/node/251 <span class="biblio-authors"><a href="/biblio/author/171">Frachtenberg, E</a>, <a href="/biblio/author/44">Feitelson DG</a></span>.  2005.  <span class="biblio-title"><a href="/biblio/view/251">Pitfalls in parallel job scheduling evaluation</a>. </span>Proceedings of the 11th international conference on Job Scheduling Strategies for Parallel Processing. :257–282.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=Pitfalls+in+parallel+job+scheduling+evaluation&rft.series=JSSPP%26%23039%3B05&rft.isbn=3-540-31024-X%2C+978-3-540-31024-2&rft.date=2005&rft.spage=257%E2%80%93282&rft.aulast=Frachtenberg&rft.aufirst=Eitan&rft.pub=Springer-Verlag&rft.place=Berlin%2C+Heidelberg&rft_id=info%3Adoi%2F10.1007%2F11605300_13"></span> http://evaluate.inf.usi.ch/node/251#comments benchmarks metrics Fri, 06 Apr 2012 14:39:00 +0000 Matthias Hauswirth 251 at http://evaluate.inf.usi.ch Supporting Experimental Computer Science http://evaluate.inf.usi.ch/node/229 <span class="biblio-authors"><a href="/biblio/author/158">Keahey, K</a>, <a href="/biblio/author/159">Desprez F</a></span>.  2012.  <span class="biblio-title"><a href="/biblio/view/229">Supporting Experimental Computer Science</a>. </span><span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=Supporting+Experimental+Computer+Science&rft.date=2012&rft.aulast=Keahey&rft.aufirst=Kate"></span> http://evaluate.inf.usi.ch/node/229#comments accuracy benchmarks metrics reproducibility statistics Thu, 15 Mar 2012 07:47:40 +0000 Matthias Hauswirth 229 at http://evaluate.inf.usi.ch Measuring Computer Performance: A Practitioner's Guide http://evaluate.inf.usi.ch/node/177 <span class="biblio-authors"><a href="/biblio/author/81">Lilja, DJ</a></span>.  2005.  <span class="biblio-title"><a href="/biblio/view/177">Measuring Computer Performance: A Practitioner's Guide</a>. </span><span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.title=Measuring+Computer+Performance%3A+A+Practitioner%26%23039%3Bs+Guide&rft.isbn=0521646707&rft.date=2005&rft.aulast=Lilja&rft.aufirst=David&rft.pub=Cambridge+University+Press"></span> http://evaluate.inf.usi.ch/node/177#comments accuracy benchmarks metrics statistics Mon, 06 Jun 2011 13:31:42 +0000 Matthias Hauswirth 177 at http://evaluate.inf.usi.ch The Art of Computer Systems Performance Analysis: techniques for experimental design, measurement, simulation, and modeling http://evaluate.inf.usi.ch/node/176 <span class="biblio-authors"><a href="/biblio/author/80">Jain, R</a></span>.  1991.  <span class="biblio-title"><a href="/biblio/view/176">The Art of Computer Systems Performance Analysis: techniques for experimental design, measurement, simulation, and modeling</a>. </span><span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.title=The+Art+of+Computer+Systems+Performance+Analysis%3A+techniques+for+experimental+design%2C+measurement%2C+simulation%2C+and+modeling&rft.date=1991&rft.aulast=Jain&rft.aufirst=Raj&rft.pub=Wiley"></span> http://evaluate.inf.usi.ch/node/176#comments accuracy benchmarks metrics statistics Mon, 06 Jun 2011 13:20:03 +0000 Matthias Hauswirth 176 at http://evaluate.inf.usi.ch Java performance evaluation through rigorous replay compilation http://evaluate.inf.usi.ch/node/134 <span class="biblio-authors"><a href="/biblio/author/1">Georges, A</a>, <a href="/biblio/author/3">Eeckhout L</a>, <a href="/biblio/author/2">Buytaert D</a></span>.  2008.  <span class="biblio-title"><a href="/biblio/view/134">Java performance evaluation through rigorous replay compilation</a>. </span>Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications. :367–384.<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Adc&rft.title=Java+performance+evaluation+through+rigorous+replay+compilation&rft.series=OOPSLA+%26%23039%3B08&rft.isbn=978-1-60558-215-3&rft.date=2008&rft.spage=367%E2%80%93384&rft.aulast=Georges&rft.aufirst=Andy&rft.pub=ACM&rft.place=New+York%2C+NY%2C+USA&rft_id=info%3Adoi%2Fhttp%3A%2F%2Fdoi.acm.org%2F10.1145%2F1449764.1449794"></span> http://evaluate.inf.usi.ch/node/134#comments accuracy benchmarks reproducibility statistics Mon, 21 Feb 2011 20:41:00 +0000 Matthias Hauswirth 134 at http://evaluate.inf.usi.ch