[Ur] Drop of several orders of magnitude in Techempower benchmarks

[Adam Chlipala]

I'm glad you brought this up, Oisín.  I was already thinking of 
appealing to this mailing list, in hopes of finding an eager detective 
to hunt down what is going on!  I can say that I can achieve much better 
performance with the latest code on my own workstation (similar profile 
to /one/ of the several machines used by TechEmpower), which leads me to 
think something basic is getting in the way of proper performance in the 
benchmarking environment.

On 7/31/19 8:06 PM, Oisín Mac Fhearaí wrote:
> I've noticed that Ur/web's performance benchmarks on Techempower have 
> changed significantly between round 16 and 17.
>
> For example, in round 16, Urweb measured 323,430 responses per second 
> to the "Fortunes" benchmark.
> In round 17 (and beyond), it achieved 4,024 RPS with MySQL and 2,544 
> RPS with Postgres.
>
> What could explain such a drastic drop in performance? The blog entry 
> for round 17 mentioned query pipelining as an explanation for some of 
> the frameworks getting much faster, but I don't see why Urweb's RPS 
> would drop by a factor of 100x, unless perhaps previous rounds had 
> query caching enabled and then round 17 disabled them.
>
> Can anyone here shed light on this? I built a simplified version of 
> the "sql" demo with the 2016 tarball version of Ur (used by the round 
> 16 benchmarks) and a recent snapshot, and they both perform at similar 
> speeds on my laptop.
>
> Oddly, the load testing tool I used (a Go program called "hey") seems 
> to have one request that takes 5 seconds if I set it to use more 
> concurrent threads than the number of threads available to the Ur/web 
> program. Otherwise, the longest request takes about 0.02 seconds. This 
> seems unrelated to the performance drop on the Techempower benchmarks, 
> since the max latency is quite low there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.impredicative.com/pipermail/ur/attachments/20190801/fadb00c5/attachment.html>