At some point in the life of every Rails developer you are bound to hit a memory leak. It may be tiny amount of constant memory growth, or a spurt of growth that hits you on the job queue when certain jobs run.

Sadly, most Ruby devs out there simply employ monit , inspeqtor or unicorn worker killers. This allows you to move along and do more important work, tidily sweeping the problem snugly under the carpet.

Unfortunately, this approach leads to quite a few bad side effects. Besides performance pains, instability and larger memory requirements it also leaves the community with a general lack of confidence in Ruby. Monitoring and restarting processes is a very important tool for your arsenal, but at best it is a stopgap and safeguard. It is not a solution.

We have some amazing tools out there for attacking and resolving memory leaks, especially the easy ones - managed memory leaks.

Are we leaking ?

The first and most important step for dealing with memory problems is graphing memory over time. At Discourse we use a combo of Graphite , statsd and Grafana to graph application metrics.

A while back I packaged up a Docker image for this work which is pretty close to what we are running today. If rolling your own is not your thing you could look at New Relic, Datadog or any other cloud based metric provider. The key metric you first need to track is RSS for you key Ruby processes. At Discourse we look at max RSS for Unicorn our web server and Sidekiq our job queue.

Discourse is deployed on multiple machines in multiple Docker containers. We use a custom built Docker container to watch all the other Docker containers. This container is launched with access to the Docker socket so it can interrogate Docker about Docker. It uses docker exec to get all sorts of information about the processes running inside the container.
Note: Discourse uses the unicorn master process to launch multiple workers and job queues, it is impossible to achieve the same setup (which shares memory among forks) in a one container per process world.

With this information at hand we can easily graph RSS for any container on any machine and look at trends:

Pasted image738x293 32.7 KB

Long term graphs are critical to all memory leak analysis. They allow us to see when an issue started. They allow us to see the rate of memory growth and shape of memory growth. Is it erratic? Is it correlated to a job running?

When dealing with memory leaks in c-extensions, having this information is critical. Isolating c-extension memory leaks often involves valgrind and custom compiled versions of Ruby that support debugging with valgrind. It is tremendously hard work we only want to deal with as a last resort. It is much simpler to isolate that a trend started after upgrading EventMachine to version 1.0.5.

Managed memory leaks

Unlike unmanaged memory leaks, tackling managed leaks is very straight forward. The new tooling in Ruby 2.1 and up makes debugging these leaks a breeze.

Prior to Ruby 2.1 the best we could do was crawl our object space, grab a snapshot, wait a bit, grab a second snapshot and compare. I have a basic implementation of this shipped with Discourse in MemoryDiagnostics, it is rather tricky to get this to work right. You have to fork your process when gathering the snapshots so you do not interfere with your process and the information you can glean is fairly basic. We can tell certain objects leaked, but can not tell where they were allocated:

3377 objects have leaked
Summary:
String: 1703
Bignum: 1674

Sample Items:
Bignum: 40 bytes
Bignum: 40 bytes
String:
Bignum: 40 bytes
String:
Bignum: 40 bytes
String:

If we were lucky we would have leaked a number or string that is revealing and that would be enough to nail it down.

Additionally we have GC.stat that could tell us how many live objects we have and so on.

The information was very limited. We can tell we have a leak quite clearly, but finding the reason can be very difficult.

Note: a very interesting metric to graph is GC.stat[:heap_live_slots] with that information at hand we can easily tell if we have a managed object leak.

Managed heap dumping

Ruby 2.1 introduced heap dumping, if you also enable allocation tracing you have some tremendously interesting information.

The process for collecting a useful heap dump is quite simple:

Turn on allocation tracing

require 'objspace'
ObjectSpace.trace_object_allocations_start

This will slow down your process significantly and cause your process to consume more memory. However, it is key to collecting good information and can be turned off later. For my analysis I will usually run it directly after boot.

When debugging my latest round memory issues with SideKiq at Discourse I deployed an extra docker image to a spare server. This allowed me extreme freedom without impacting SLA.

Next, play the waiting game

After memory has clearly leaked, (you can look at GC.stat or simply watch RSS to determine this happened) run:

io=File.open("/tmp/my_dump", "w")
ObjectSpace.dump_all(output: io);
io.close

Running Ruby in an already running process

To make this process work we need to run Ruby inside a process that has already started.

Luckily, the rbtrace gem allows us to do that (and much more). Moreover it is safe to run in production.

We can easily force SideKiq to dump a its heap with:

bundle exec rbtrace -p $SIDEKIQ_PID -e 'Thread.new{GC.start;require "objspace";io=File.open("/tmp/ruby-heap.dump", "w"); ObjectSpace.dump_all(output: io); io.close}'

rbtrace runs in a restricted context, a nifty trick is breaking out of the trap context with Thread.new

We can also crack information out of the box with rbtrace, for example:

bundle exec rbtrace -p 6744 -e 'GC.stat'
/usr/local/bin/ruby: warning: RUBY_HEAP_MIN_SLOTS is obsolete. Use RUBY_GC_HEAP_INIT_SLOTS instead.
*** attached to process 6744>> GC.stat
=> {:count=>49, :heap_allocated_pages=>1960, :heap_sorted_length=>1960, :heap_allocatable_pages=>12, :heap_available_slots=>798894, :heap_live_slots=>591531, :heap_free_slots=>207363, :heap_final_slots=>0, :heap_marked_slots=>335775, :heap_swept_slots=>463124, :heap_eden_pages=>1948, :heap_tomb_pages=>12, :total_allocated_pages=>1960, :total_freed_pages=>0, :total_allocated_objects=>13783631, :total_freed_objects=>13192100, :malloc_increase_bytes=>32568600, :malloc_increase_bytes_limit=>33554432, :minor_gc_count=>41, :major_gc_count=>8, :remembered_wb_unprotected_objects=>12175, :remembered_wb_unprotected_objects_limit=>23418, :old_objects=>309750, :old_objects_limit=>618416, :oldmalloc_increase_bytes=>32783288, :oldmalloc_increase_bytes_limit=>44484250}
*** detached from process 6744

Analyzing the heap dump

With a rich heap dump at hand we can start analysis, a first report to run is looking at the count of object per GC generation.

When trace object allocation is enabled the runtime will attach rich information next to all allocations. For each object that is allocated while tracing is on we have:

1. The GC generation it was allocated in
2. The filename and line number it was allocated in
3. A truncated value
4. bytesize
5. ... and much more

The file is in json format and can easily be parsed line by line. EG:

{"address":"0x7ffc567fbf98", "type":"STRING", "class":"0x7ffc565c4ea0", "frozen":true, "embedded":true, "fstring":true, "bytesize":18, "value":"ensure in dispatch", "file":"/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/activesupport-4.1.9/lib/active_support/dependencies.rb", "line":247, "method":"require", "generation":7, "memsize":40, "flags":{"wb_protected":true, "old":true, "long_lived":true, "marked":true}}

A simple report that shows how many objects are left around from which GC generation is a great start for looking at memory leaks. Its a timeline of object leakage.

require 'json'
class Analyzer
  def initialize(filename)
    @filename = filename
  end

  def analyze
    data = []
    File.open(@filename) do |f|
        f.each_line do |line|
          data << (parsed=JSON.parse(line))
        end
    end

    data.group_by{|row| row["generation"]}
        .sort{|a,b| a[0].to_i <=> b[0].to_i}
        .each do |k,v|
          puts "generation #{k} objects #{v.count}"
        end
  end
end

Analyzer.new(ARGV[0]).analyze

For example this is what I started with:

generation  objects 334181
generation 7 objects 6629
generation 8 objects 38383
generation 9 objects 2220
generation 10 objects 208
generation 11 objects 110
generation 12 objects 489
generation 13 objects 505
generation 14 objects 1297
generation 15 objects 638
generation 16 objects 748
generation 17 objects 1023
generation 18 objects 805
generation 19 objects 407
generation 20 objects 126
generation 21 objects 1708
generation 22 objects 369
...

We expect a large number of objects to be retained after boot and sporadically when requiring new dependencies. However we do not expect a consistent amount of objects to be allocated and never cleaned up. So let's zoom into a particular generation:

require 'json'
class Analyzer
  def initialize(filename)
    @filename = filename
  end

  def analyze
    data = []
    File.open(@filename) do |f|
        f.each_line do |line|
          parsed=JSON.parse(line)
          data << parsed if parsed["generation"] == 18
        end
    end
    data.group_by{|row| "#{row["file"]}:#{row["line"]}"}
        .sort{|a,b| b[1].count <=> a[1].count}
        .each do |k,v|
          puts "#{k} * #{v.count}"
        end
  end
end

Analyzer.new(ARGV[0]).analyze

generation 19 objects 407
/usr/local/lib/ruby/2.2.0/weakref.rb:87 * 144
/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/therubyracer-0.12.1/lib/v8/weak.rb:21 * 72
/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/therubyracer-0.12.1/lib/v8/weak.rb:42 * 72
/var/www/discourse/lib/freedom_patches/translate_accelerator.rb:65 * 15
/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/i18n-0.7.0/lib/i18n/interpolate/ruby.rb:21 * 15
/var/www/discourse/lib/email/message_builder.rb:85 * 9
/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/actionview-4.1.9/lib/action_view/template.rb:297 * 6
/var/www/discourse/lib/email/message_builder.rb:36 * 6
/var/www/discourse/lib/email/message_builder.rb:89 * 6
/var/www/discourse/lib/email/message_builder.rb:46 * 6
/var/www/discourse/lib/email/message_builder.rb:66 * 6
/var/www/discourse/vendor/bundle/ruby/2.2.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:515 * 5

Further more, we can chase down the reference path to see who is holding the references to the various objects and rebuild object graphs.

The first thing I attacked in this particular case was code I wrote, which is a monkey patch to Rails localization.

Why we monkey patch rails localization?

At Discourse we patch the Rails localization subsystem for 2 reasons:

1. Early on we discovered it was slow and needed better performance
2. Recently we started accumulating an enormous amount of translations and needed to ensure we only load translations on demand to keep memory usage lower. (this saves us 20MB of RSS)

Consider the following bench:

ENV['RAILS_ENV'] = 'production'
require 'benchmark/ips'

require File.expand_path("../../config/environment", __FILE__)

Benchmark.ips do |b|
  b.report do |times|
    i = -1
    I18n.t('posts') while (i+=1) < times
  end
end

Before monkey patch:

sam@ubuntu discourse % ruby bench.rb
Calculating -------------------------------------
                         4.518k i/100ms
-------------------------------------------------
                        121.230k (±11.0%) i/s -    600.894k

After monkey patch:

sam@ubuntu discourse % ruby bench.rb
Calculating -------------------------------------
                        22.509k i/100ms
-------------------------------------------------
                        464.295k (±10.4%) i/s -      2.296M

So our localization system is running almost 4 times faster, but ... it is leaking memory.

Reviewing the code I could see that the offending line is:

Turns out, we were sending a hash in from the email message builder that includes ActiveRecord objects, this later was used as a key for the cache, the cache was allowing for 2000 entries. Considering that each entry could involve a large number of AR objects memory leakage was very high.

To mitigate, I changed the keying strategy, shrunk the cache and completely bypassed it for complex localizations:

One day later when looking at memory graphs we can easily see the impact of this change:

Pasted image791x295 44.8 KB

This clearly did not stop the leak but it definitely slowed it down.

therubyracer is leaking

At the top of our list we can see our JavaScript engine therubyracer is leaking lots of objects, in particular we can see the weak references it uses to maintain Ruby to JavaScript mappings are being kept around for way too long.

To maintain performance at Discourse we keep a JavaScript engine context around for turning Markdown into HTML. This engine is rather expensive to boot up, so we keep it in memory plugging in new variables as we bake posts.

Since our code is fairly isolated a repro is trivial, first let's see how many objects we are leaking using the memory_profiler gem:

ENV['RAILS_ENV'] = 'production'
require 'memory_profiler'
require File.expand_path("../../config/environment", __FILE__)

# warmup
3.times{PrettyText.cook("hello world")}

MemoryProfiler.report do
  50.times{PrettyText.cook("hello world")}
end.pretty_print

At the top of our report we see:

retained objects by location
-----------------------------------
       901  /home/sam/.rbenv/versions/2.1.2.discourse/lib/ruby/2.1.0/weakref.rb:87
       901  /home/sam/.rbenv/versions/2.1.2.discourse/lib/ruby/gems/2.1.0/gems/therubyracer-0.12.1/lib/v8/weak.rb:21
       600  /home/sam/.rbenv/versions/2.1.2.discourse/lib/ruby/gems/2.1.0/gems/therubyracer-0.12.1/lib/v8/weak.rb:42
       250  /home/sam/.rbenv/versions/2.1.2.discourse/lib/ruby/gems/2.1.0/gems/therubyracer-0.12.1/lib/v8/context.rb:97
        50  /home/sam/.rbenv/versions/2.1.2.discourse/lib/ruby/gems/2.1.0/gems/therubyracer-0.12.1/lib/v8/object.rb:8

So we are leaking 54 or so objects each time we cook a post, this adds up really fast. We may also be leaking unmanaged memory here which could compound the problem.

Since we have a line number it is easy to track down the source of the leak

require 'weakref'
class Ref
  def initialize(object)
    @ref = ::WeakRef.new(object)
  end
  def object
    @ref.__getobj__
  rescue ::WeakRef::RefError
    nil
  end
end

class WeakValueMap
   def initialize
      @values = {}
   end

   def [](key)
      if ref = @values[key]
        ref.object
      end
   end

   def []=(key, value)
     @values[key] = V8::Weak::Ref.new(value)
   end
end

This WeakValueMap object keeps growing forever and nothing seems to be clearing values from it. The intention with the WeakRef usage was to ensure we allow these objects to be released when no-one holds references. Trouble is that the reference to the wrapper is now kept around for the entire lifetime of the JavaScript context.

A fix is pretty straight forward:

class WeakValueMap
  def initialize
    @values = {}
  end

  def [](key)
    if ref = @values[key]
      ref.object
    end
  end

  def []=(key, value)
    ref = V8::Weak::Ref.new(value)
    ObjectSpace.define_finalizer(value, self.class.ensure_cleanup(@values, key, ref))

    @values[key] = ref
  end

  def self.ensure_cleanup(values,key,ref)
    proc {
      values.delete(key) if values[key] == ref
    }
  end
end

We define a finalizer on the wrapped object that ensures we clean up the all these wrapping objects and keep on WeakValueMap small.

The effect is staggering:

ENV['RAILS_ENV'] = 'production'
require 'objspace'
require 'memory_profiler'
require File.expand_path("../../config/environment", __FILE__)

def rss
 `ps -eo pid,rss | grep #{Process.pid} | awk '{print $2}'`.to_i
end

PrettyText.cook("hello world")

# MemoryProfiler has a helper that runs the GC multiple times to make
#  sure all objects that can be freed are freed.
MemoryProfiler::Helpers.full_gc
puts "rss: #{rss} live objects #{GC.stat[:heap_live_slots]}"

20.times do

  5000.times { |i|
    PrettyText.cook("hello world")
  }
  MemoryProfiler::Helpers.full_gc
  puts "rss: #{rss} live objects #{GC.stat[:heap_live_slots]}"

end

Before

rss: 137660 live objects 306775
rss: 259888 live objects 570055
rss: 301944 live objects 798467
rss: 332612 live objects 1052167
rss: 349328 live objects 1268447
rss: 411184 live objects 1494003
rss: 454588 live objects 1734071
rss: 451648 live objects 1976027
rss: 467364 live objects 2197295
rss: 536948 live objects 2448667
rss: 600696 live objects 2677959
rss: 613720 live objects 2891671
rss: 622716 live objects 3140339
rss: 624032 live objects 3368979
rss: 640780 live objects 3596883
rss: 641928 live objects 3820451
rss: 640112 live objects 4035747
rss: 722712 live objects 4278779
/home/sam/Source/discourse/lib/pretty_text.rb:185:in `block in markdown': Script Timed Out (PrettyText::JavaScriptError)
	from /home/sam/Source/discourse/lib/pretty_text.rb:350:in `block in protect'
	from /home/sam/Source/discourse/lib/pretty_text.rb:348:in `synchronize'
	from /home/sam/Source/discourse/lib/pretty_text.rb:348:in `protect'
	from /home/sam/Source/discourse/lib/pretty_text.rb:161:in `markdown'
	from /home/sam/Source/discourse/lib/pretty_text.rb:218:in `cook'
	from tmp/mem_leak.rb:30:in `block (2 levels) in <main>'
	from tmp/mem_leak.rb:29:in `times'
	from tmp/mem_leak.rb:29:in `block in <main>'
	from tmp/mem_leak.rb:27:in `times'
	from tmp/mem_leak.rb:27:in `<main>'

After

rss: 137556 live objects 306646
rss: 259576 live objects 314866
rss: 261052 live objects 336258
rss: 268052 live objects 333226
rss: 269516 live objects 327710
rss: 270436 live objects 338718
rss: 269828 live objects 329114
rss: 269064 live objects 325514
rss: 271112 live objects 337218
rss: 271224 live objects 327934
rss: 273624 live objects 343234
rss: 271752 live objects 333038
rss: 270212 live objects 329618
rss: 272004 live objects 340978
rss: 270160 live objects 333350
rss: 271084 live objects 319266
rss: 272012 live objects 339874
rss: 271564 live objects 331226
rss: 270544 live objects 322366
rss: 268480 live objects 333990
rss: 271676 live objects 330654

Looks like memory stabilizes and live object count stabilizes after the fix.

Summary

The current tooling offered in Ruby offer spectacular visibility into the Ruby runtime. The tooling around this new instrumentation is improving but still rather rough at spots.

As a .NET developer in a previous lifetime I really missed the excellent memory profiler tool, luckily we now have all the information needed to be able to build tools like this going forward.

Good luck hunting memory leaks, I hope this information helps you and please think twice next time you deploy a "unicorn OOM killer".

A huge thank you goes to Koichi Sasada and Aman Gupta for building us all this new memory profiling infrastructure.

PS: another excellent resource worth reading is Oleg Dashevskii's "How I spent two weeks hunting a memory leak in Ruby"

Recently, I discovered a performance regression on a very common page on Discourse. I spent a fair amount of time debugging and optimizing. I follow a certain methodology while I do this kind of work.

This post is a breakdown on the specific issue I faced with some points you can take back and apply to your next performance debugging session.

Pick your fights

The first and most important point to take is that you should pick your battles. Discourse has hundreds of routes, however the vast majority of the server cost is incurred by a handful.

Pasted image792x576 22 KB

The most important 3 routes for us are "topics/show", "list/latest" and "categories/index". They are the heart of the site and lion share of foreground routes.

"topic/timings", "user avatars" and "drafts" are all background routes, we still want to minimize work on them so servers work less hard and we can host more sites, however slowness there is usually not observed by end users.

I always try to focus first on the most active routes foreground routes, those are the spots where I will invest the most amount of effort optimizing.

To get a good picture of our traffic patterns we use Kibana. To answer the same question you may use Google Analytics, New Relic or some other tool.

Start with a baseline and a goal

We have a Grafana dashboard keeping an eye on our 2 most important routes for every site we run. I visit the dashboard regularly to see how performance is on those routes. Is displaying topics getting faster or slower? It is very important to have long term trends so you can isolate when stuff starts playing up.

Recently I discovered this:

Pasted image971x271 56.7 KB

Showing topics on 2 particular sites (one is shown) got much slower. Having this information is golden.

This graph is a visible report card on my work towards improving performance. When I see a graph like this my immediate goal becomes restoring old performance characteristics. This is particularly important here since this is our most important route.

Know your tools and their limitations / strengths

During my performance work I generally use rack-mini-profiler and flamegraphs.

rack-mini-profiler is great at getting a good overall picture of activity on a page. In particular, I find it very effective at isolating slow SQL and minimizing SQL calls.

Flamegraphs are awesome at looking underneath the surface and finding hidden costs.

Profile in production mode

A huge pitfall many can fall into with Rails is doing performance tuning in development mode. There is huge amount of noise and irrelevant costs that are absent in production.

Pasted image1274x341 39.1 KB

In production a lot less work happens, additionally there is high risk that if I debug performance in development mode I may work on something that has no impact in production... like improving the rendering time of common/_discourse_stylesheet

Use a real customer database

During my dev work I will cycle between various customer databases, this allows me to have a good feel of performance real customers feel. When debugging a performance problem it is critical to have an exact copy of the database exhibiting the issue, the regression in performance may be due to a customer setting, data anomaly and so on. Doing performance work on a stock development database is like performing an operation with a blindfold.

Do less work

I listened once to a talk by Charlie Nutter where he was talking about performance. He said the key to fast programs is doing less work. I completely agree with this analysis.

Side note: when I am doing a round of optimizations on a key route I tend to go a little crazy. Even though in itself each optimization may only save a millisecond, add up 1000 and you save a second. You don't have to follow all these tips, or even agree with all of them, nonetheless I feel it is instructive to talk through the reasoning.

Here are some specific examples of commits I made while debugging this issue.

• obj && obj.prop is faster than obj.try(:prop). By using the first "rubyish" way we are avoiding a respond_to? call. (commit)

• Don't run a query if you know it will return no results: (commit)

• Often superfluous queries running are simply a result of bugs: (commit)

• Instead of performing 5 redis calls in a sequence, do them in a batch: (commit)

Pasted image429x738 25 KB

The database is usually the bottleneck on the server

A typical topic show page on Discourse will spend half the time in SQL.

A typical breakdown for a page being rendered at Discourse is

• 50% time waiting for SQL
• 30% ActiveRecord overhead
• 10% json serialization and view generation
• 10% other

Pasted image624x790 50.9 KB

By running less SQL statements not only do we save the time in SQL, we also save on the Active Record overhead. A huge amount of the time I spend optimizing performance looks at improving SQL, or eliminating SQL calls.

A recent example here would be memoizing a property on an ActiveRecord object to avoid N+1 (commit)

The SQL view in rack-mini-profiler is key at finding SQL to eliminate:

Pasted image835x307 11.1 KB

Can these queries be done in once go?

Sometimes the majority of the cost boils down to a single SQL statement, a specific example may be the query we use to track what unread and new posts a user has, this particular query has gone through at least 5 rewrites: (commit)

Profile for the common case

When profiling I always prefer to profile the common case. For most our sites the most common case is "anonymous" visiting a particular topic.

I usually cheat to enable rack-mini-profiler, run it in production and add a return true in my authorization method:

def mini_profiler_enabled?
   return true
   defined?(Rack::MiniProfiler) && guardian.is_developer?
end

By profiling the common case you can find optimizations that may be less obvious.

Cache the common case

A classic example is more aggressive caching for anonymous, during my debugging I discovered that a huge portion of the time generating our pages was serializing our view of what a "site" is. This construct lists the categories a user has access to, the groups, site settings an so on.

This particular construct is highly cachable for anonymous, and partially cachable for logged on users.

I introduced a fragment cache to cache parts of it: (commit , commit)

I introduced a full cache for the anonymous case to avoid a huge chunk of work: (commit)

A commit is usually to blame but not always

Usually we assume that a commit caused a performance regression, though often the case, this is not always the case.

For example, I recently discovered this issue:

We have this site setting called: apple touch icon url, this allows users to add a url for an icon that is included in a meta tag like so

<link rel="apple-touch-icon" type="image/png" href="https://example.com/awesome.png"><link rel="icon" sizes="144x144" href="https://example.com/awesome.png">

When I was debugging one of the slow sites I noticed a rather odd behavior. When I hit refresh on a page a web request was made to get the page. After the page was loaded a second GET request was made the same URL. This stumped me.

After a long round of debugging I discovered that the link tags were:

<link rel="apple-touch-icon" type="image/png" href=""><link rel="icon" sizes="144x144" href="">

The customer set apple icon to nothing, it was their way of communicating they do not care for this feature. But we were rendering a blank href. Chrome and Firefox were dutifully attempting to grab this icon from the relative URL "", aka. this page.

This was fixed in (commit)

Take a walk

The nastier the performance problem I hit the more likely I will not arrive at an efficient solution on first go. It is often tempting to just sit at the computer for 10 hours straight nutting out an issue. Unfortunately this is often counter productive. Taking a walk lets me clear my head, de-stress and come up with new novel solutions.

Don't give up

Traditionally people may approach a performance regression with the mindset of:

Something made this slower, let's work hard to find out what

Instead I prefer to take the approach of

What can I do to make this slow piece of code faster

I usually discover what it is that made the code slow during my process of optimizing. Making code fast leads to a very intimate understanding of its performance traits. As a side effect of my latter approach I leave the slow code faster than the baseline when I am done.

Sometimes when strapped for time I will just bisect, hope for the best and move on when I find the hole, but I find it so much more fun to leave the code faster than it was before it regressed.

This approach though takes a lot of time and patience. Not giving up is the key to success.

Pasted image956x276 60.4 KB

Good luck debugging

When developers hear that WebSockets are going to land in the near future in Rails they get all giddy with excitement.

But your users don't care if you use WebSockets:

• Users want "delightful realtime web apps".

• Developers want "delightfully easy to build realtime web apps".

• Operations want "delightfully easy to deploy, scale and manage realtime web apps".

If WebSockets get us there, great, but it is an implementation detail that comes at high cost.

Do we really need ultra high performance, full duplex Client-Server communication?

WebSockets provides simple APIs to broadcast information to clients and simple APIs to ship information from the clients to the web server.

A realtime channel to send information from the server to the client is very welcome. In fact it is a part of HTTP 1.1.

However, a brand new API for shipping information to the server from web browsers introduce a new decision point for developers:

• When a user posts a message on chat, do I make a RESTful call and POST a message or do I bypass REST and use WebSockets?

• If I use the new backchannel, how do I debug it? How do I log what is going on? How do I profile it? How do I ensure it does not slow down other traffic to my site? Do I also expose this endpoint in a controller action? How do I rate limit this? How do I ensure my background WebSocket thread does not exhaust my db connection limit?

If an API allows hundreds of different connections concurrent access to the database, bad stuff will happen.

Introducing this backchannel is not a clear win and comes with many caveats.

I do not think the majority of web applications need a new backchannel into the web server. On a technical level you would opt for such a construct if you were managing 10k interactive console sessions on the web. You can transport data more efficiently to the server, in that the web server no longer needs to parse HTTP headers, Rails does not need to do a middleware crawl and so on.

But the majority of web applications out there are predominantly read applications. Lots of users benefit from live updates, but very few change information. It is incredibly rare to be in a situation where the HTTP header parsing optimisation is warranted; this work is done sub millisecond. Bypassing Rack middleware on the other hand can be significant, especially when full stack middleware crawls are a 10-20ms affair. That however is an implementation detail we can optimise and not a reason to rule out REST for client to server communications.

For realtime web applications we need simple APIs to broadcast information reliably and quickly to clients. We do not need new mechanisms for shipping information to the server.

What's wrong with WebSockets?

WebSockets had a very tumultuous ride with a super duper unstable spec during the journey. The side effects of this joyride show in quite a few spots. Take a look at Ilya Grigorik's very complete implementation. 5 framing protocols, 3 handshake protocols and so on.

At last, today, this is all stable and we have RFC6455 which is implemented ubiquitously across all major modern browsers. However, there was some collateral damage:

• IE9 and earlier are not supported

• Many libraries – including the most popular Ruby one – ship with multiple implementations, despite Hixie 75 being flawed.

I am confident the collateral damage will, in time, be healed. That said, even the most perfect implementation comes with significant technical drawbacks.

1) Proxy servers can wreak havoc with WebSockets running over unsecured HTTP

The proxy server issue is quite widespread. Our initial release of Discourse used WebSockets, however reports kept coming in of "missing updates on topics" and so on. Amongst the various proxy pariahs was my mobile phone network Telstra which basically let you have an open socket, but did not let any data through.

To work around the "WebSocket is dead but still appears open problem" WebSocket implementers usually introduce a ping/pong message. This solution works fine provided you are running over HTTPS, but over HTTP all bets are off and rogue proxies will break you.

That said, "... but you must have HTTPS" is the weakest argument against WebSocket adoption, I want all the web to be HTTPS, it is the future and it is getting cheaper every day. But you should know that weird stuff will definitely happen you deploy WebSockets over unsecured HTTP. Unfortunately for us at Discourse dropping support for HTTP is not an option quite yet, as it would hurt adoption.

2) Web browsers allow huge numbers of open WebSockets

The infamous 6 connections per host limit does not apply to WebSockets. Instead a far bigger limit holds (255 in Chrome and 200 in Firefox). This blessing is also a curse. It means that end users opening lots of tabs can cause large amounts of load and consume large amounts of continuous server resources. Open 20 tabs with a WebSocket based application and you are risking 20 connections unless the client/server mitigates.

There are quite a few ways to mitigate:

• If we have a reliable queue driving stuff, we can shut down sockets after a while (or when in a background tab) and reconnect later on and catch up.

• If we have a reliable queue driving stuff, we can throttle and turn back high numbers of TCP connections at our proxy or even iptables, but it is hard to guess if we are turning away the right connections.

• On Firefox and Chrome we can share a connection by using a shared web worker, which is unlikely to be supported on mobile and is absent from Microsoft's offerings. I noticed Facebook are experimenting with shared workers (Gmail and Twitter are not).

• MessageBus uses browser visibility APIs to slow down communication on out-of-focus tabs, falling back to a 2 minute poll on background tabs.

3) WebSockets and HTTP/2 transport are not unified

HTTP/2 is able to cope with the multiple tab problem much more efficiently than WebSockets. A single HTTP/2 connection can be multiplexed across tabs, which makes loading pages in new tabs much faster and significantly reduces the cost of polling or long polling from a networking point of view. Unfortunately, HTTP/2 does not play nice with WebSockets. There is no way to tunnel a WebSocket over HTTP/2, they are separate protocols.

There is an expired draft to unify the two protocols, but no momentum around it.

HTTP/2 has the ability to stream data to clients by sending multiple DATA frames, meaning that streaming data from the server to the client is fully supported.

Unlike running a Socket server, which includes a fair amount of complex Ruby code, running a HTTP/2 server is super easy. HTTP/2 is now in NGINX mainline, you can simply enable the protocol and you are done.

4) Implementing WebSockets efficiently on the server side requires epoll, kqueue or I/O Completion ports.

Efficient long polling, HTTP streaming (or Server Sent Events) is fairly simple to implement in pure Ruby since we do not need to repeatedly run IO.select. The most complicated structure we need to deal with is a TimerThread

Efficient Socket servers on the other hand are rather complicated in the Ruby world. We need to keep track of potentially thousands of connections dispatching Ruby methods when new data is available on any of the sockets.

Ruby ships with IO select which allows you to watch an array of sockets for new data, however it is fairly inefficient cause you force the kernel to keep walking big arrays to figure out if you have any new data. Additionally, it has a hard limit of 1024 entries (depending on how you compiled your kernel), you can not select on longer lists. EventMachine solves this limitation by implementing epoll (and kqueue for BSD).

Implementing epoll correctly is not easy.

5) Load balancing WebSockets is complicated

If you decide to run a farm of WebSockets, proper load balancing is complicated. If you find out that your Socket servers are overloaded and decide to quickly add a few servers to the mix, you have no clean way of re-balancing current traffic. You have to terminate overloaded servers due to connections being open indefinitely. At that point you are exposing yourself to a flood of connections (which can be somewhat mitigated by clients). Furthermore if "on reconnect" we have to refresh the page, restart a socket server and you will flood your web servers.

With WebSockets you are forced to run TCP proxies as opposed to HTTP proxies. TCP proxies can not inject headers, rewrite URLs or perform many of the roles we traditionally let our HTTP proxies take care of.

Denial of service attacks that are usually mitigated by front end HTTP proxies can not be handled by TCP proxies; what happens if someone connects to a socket and starts pumping messages into it that cause database reads in your Rails app? A single connection can wreak enormous amounts of havoc.

6) Sophisticated WebSocket implementations end up re-inventing HTTP

Say we need to be subscribed to 10 channels on the web browser (a chat channel, a notifications channel and so on), clearly we will not want to make 10 different WebSocket connections. We end up multiplexing commands to multiple channels on a single WebSocket.

Posting "Sam said hello" to the "/chat" channel ends up looking very much like HTTP. We have "routing" which specifies the channel we are posting on, this looks very much like HTTP headers. We have a payload, that looks like HTTP body. Unlike HTTP/2 we are unlikely to get header compression or even body compression.

7) WebSockets give you the illusion of reliability

WebSockets ship with a very appealing API.

• You can connect
• You have a reliable connection to the server due to TCP
• You can send and receive messages

But... the Internet is a very unreliable place. Laptops go offline, you turn on airplane mode when you had it with all the annoying marketing calls, Internet sometimes does not work.

This means that this appealing API still needs to be backed by reliable messaging, you need to be able to catch up with a backlog of messages and so on.

When implementing WebSockets you need to treat them just as though they are simple HTTP calls that can go missing be processed at the server out-of-order and so on. They only provide the illusion of reliability.

WebSockets are an implementation detail, not a feature

At best, WebSockets are a value add. They provide yet another transport mechanism.

There are very valid technical reasons many of the biggest sites on the Internet have not adopted them. Twitter use HTTP/2 + polling, Facebook and Gmail use Long Polling. Saying WebSockets are the only way and the way of the future, is wrongheaded. HTTP/2 may end up winning this battle due to the huge amount of WebSocket connections web browsers allow, and HTTP/3 may unify the protocols.

• You may want to avoid running dedicated socket servers (which at scale you are likely to want to run so sockets do not interfere with standard HTTP traffic). At Discourse we run no dedicated long polling servers, adding capacity is trivial. Capacity is always balanced.

• You may be happy with a 30 second delays and be fine with polling.

• You may prefer the consolidated transport HTTP/2 offers and go for long polling + streaming on HTTP/2

Messaging reliability is far more important than WebSockets

MessageBus is backed by a reliable pub/sub channel. Messages are globally sequenced. Messages are locally sequenced to a channel. This means that at any point you can "catch up" with old messages (capped). API wise it means that when a client subscribes it has the option to tell the server what position the channel is:

// subscribe to the chat channel at position 7
MessageBus.subscribe('/chat', function(msg){ alert(msg); }, 7);

Due to the reliable underpinnings of MessageBus it is immune to a class of issues that affect pure WebSocket implementations.

This underpinning makes it trivial to write very efficient cross process caches amongst many other uses.

Reliable messaging is a well understood concept. You can use Erlang, RabbitMQ, ZeroMQ, Redis, PostgreSQL or even MySQL to implement reliable messaging.

With reliable messaging implemented, multiple transport mechanisms can be implemented with ease. This "unlocks" the ability to do long-polling, long-polling with chunked encoding, EventSource, polling, forever iframes etc in your framework.

When picking a realtime framework, prefer reliable underpinnings to WebSockets.

Where do I stand?

Discourse does not use WebSockets. Discourse docker ships with HTTP/2 templates.

We have a realtime web application. I can make a realtime chat room just fine in 200 lines of code. I can run it just fine in Rails 3 or 4 today by simply including the gem. We handle millions of long polls a day for our hosted customers. As soon as someone posts a reply to a topic in Discourse it pops up on the screen.

We regard MessageBus as a fundamental and integral part of our stack. It enables reliable server/server live communication and reliable server/client live communication. It is transport agnostic. It has one dependency on rack and one dependency on redis, that is all.

When I hear people getting excited about WebSockets, this is the picture I see in my mind.

In a world that already had HTTP/2 it is very unlikely we would see WebSockets being ratified as it covers that vast majority of use cases WebSockets solves.

Special thank you to Ilya, Evan, Matt, Jeff, Richard and Jeremy for reviewing this article.

Chrome recently pushed out support for Brotli, in this post I will cover what this means to you.

In late May 2016 Chrome pushed out Chrome 51, unlike many releases of Chrome which are complete non-events, this release has an enormous impact. Google turned on Brotli support– and they promptly backported it into Chrome 50.

Pasted image1022x761 41.9 KB

Firefox added support for Brotli in September 2015. They even blogged about it. 8 months later, thanks to Google, Brotli went from a compression format supported in less than 10% of global browsers to nearly 50% global adoption!

Brotli is HTTPS only

If you visit a site over HTTP your browser will not accept the br encoding. The reasoning for this is documented at the end of the chromium issue.

pdansk

I have a question. What does intermediates refer to in the reason to restrict this feature to HTTPS only? Caches? If so, this may or may not be solved by adding no-transform to Cache-Control when you send Content-Encoding: br. I don't know.

I think that restricting to HTTPS is regrettable, as what you save with brotli you lose double by not having caching.

kenjibaheux@chromium

Intermediaries (or "middle boxes") refers to companies/infra/software meddling with the data transfer between you (the user) and the webserver.

One example from SDCH that was mentioned to me (the name of the company is not relevant to the discussion so I'm hiding it):

"The most extreme case of middle box was Company AcmeTelecom (fictitious name but true story), that tried to make things better when faced with unknown content encodings, by doing the following things:

a) Remove the unrecognized content encoding
b) Pass the (already compressed!) content through another gzip encoding pass
c) Claim that the content was merely encoded as gzip"

Lack of Brotli support over unencrypted HTTP is no mistake. The world is full of terribly non-compliant HTTP proxies:

Brotli is yet another reason for you to push for that HTTP/2 change you have been holding out on.

Brotli: The New Pied Piper?

When I first heard of Brotli, there were a lot of stats being published about how much better the compression is. The Next Web said it is 26% better. and Akamai blogged about its huge potential. I was thinking to myself… did we just uncover a real world pied piper?

In my measurements, the best case compression for Brotli really does blow away the competition:

Pasted image1370x951 10.3 KB

Pasted image1113x262 14.5 KB

(percentages are expressed as savings compared to gzip -9)

• Even at level 5, Brotli is better than gzip -9
• Worst case savings is 9.29% (for jQuery)
• Best case savings is 29% (for our Discourse application js)
• The larger the file, the better Brotli fares.
• You can get a free 3-4% savings by using zopfli which is compatible with gzip decompression.

Comparing Brotli compression levels

Brotli, like other compression formats, offers a choice of compression levels, from 1 all the way to 11. Level 5 is where Brotli starts using context modelling, one of the more advanced features of the format.

Pasted image1076x759 9.4 KB

Pasted image1233x816 10.2 KB

Pasted image760x262 8.03 KB

Compressing Brotli at the highest setting is so slow gzip -9 is practically invisible from the graph!

• Brotli at level 11 is not feasible for dynamic content
• Brotli at level 5 is competitive with gzip and produces files that are even smaller than gzip -9
• Zopfli is really slow, for large files it can take 4× the time of Brotli 11.

Brotli is not easy to adopt today

Another complication with Brotli is that it is missing-in-action.

Ubuntu 16.04 is the first Ubuntu distro with apt-get install brotli, if you want to play around on earlier distros you have to find a third party apt repo or compile it yourself.

NGINX has a Brotli extension, this is not distributed with the official NGINX package, forcing you to compile NGINX by yourself if you want brotli today.

sidenote: nginx -VV makes it significantly easier to self-compile NGINX without getting into a big mess.

Apache only recently got a Brotli extension

When it comes to the various asset pipelines out there, none of them support pre-compressing Brotli assets, so you have to bolt it in yourself.

Even with all these rather big hurdles you can have it today, and it does work!

Pasted image860x514 56.6 KB

Brotli decompression is fast

I did not include any graphs or stats for gzip and brotli decompression, they are simply so fast that I would need to build a rather elaborate test harness to get an accurate measure of the difference. It is simply so fast that it does not really matter. I found that the IO of writing the file is the bottleneck not the decompression.

CDN support for Brotli is uneven

Of the 4 CDNs I checked:

Fastly and MaxCDN had stripped out accept encoding for Brotli, falling back to gzip, CDN77 appears to be letting Brotli through. KeyCDN do not allow Brotli through (thouge they do provide you with a tool to check if your current CDN is allowing Brotli).

Fastly have a documented workaround, but support is not built in.

This uneven support is understandable. CDNs usually normalize the Accept-Encoding header. This happens cause there is a huge matrix of Accept-Encoding in the wild, if this is not normalized the same gzipped resource can be stored 44 times.

If a CDN allows brotli through it can quite easily double all the requests to your website for the origin resource. It has to check if you have both Brotli and gzip for every asset.

Considering that the vast amount of resources you are serving are probably not Brotli compressed, this can become very wasteful, especially if the first request to the asset comes from a browser that does not accept the br encoding like Safari or IE.

Ideally, CDNs should provide you with a way to whitelist some (or all) assets to opt out of Accept Encoding normalization.

Brotli is about to become an RFC

The magic date, May 25, 2016, was not randomly picked by Google for enabling Brotli in Chrome. It was the date IANA finally stamped the ietf draft with RFC Ed Queue. That is the final state the document enters prior to becoming an RFC and getting an RFC number. The approval announcement is already out.

https://www.ietf.org/mail-archive/web/ietf-announce/current/msg15482.html

The draft's authors submitted it to the Independent Stream in April 2014. It took the ISE rather a long time to find suitable reviewers for a new compression technique, but, it finally got a brief review from Ulrich Speidel(U Auckland, a coding theory researcher), and a more thorough review from Jean-Loup Gailly (the author of gzip). The authors made changes based on those reviews, and it is now ready for publication.

Google did the right thing in not rushing Brotli to the public, it is excellent that Jean-loup Gailly and Mark Adler the original authors of the gzip format had a chance to properly review the draft.

What can we do?

Proxy and web server vendors should make it easier to experiment with Brotli

I would like to see NGINX, Apache, HAProxy etc. make it easier for people to experiment with Brotli. For NGINX this means shipping with Brotli support in an optional official extension.

The biggest challenge ahead is figuring out a smart heuristic for figuring out which compression level to use balancing compression with CPU cost and current load on dynamic content. This is very critical for Brotli adoption, there is huge variance between maximum and minimum compression for both size and time. This was never the case for gzip.

Microsoft's IIS has special settings that allow it to disable gzip compression if CPU goes higher than a certain amount. Ideally NGINX brotli extensions can learn from it and enable a setting that backs off on compression if load is too high due dynamic assets compression.

Overall, dealing with brotli compression requires that our web servers get smarter and more understanding of the huge variance CPU with between the various brotli compression levels.

CDNs should advertise their Brotli support

CDNs these days publish support for HTTP/2, usually on the front page. This is excellent and I am impressed that so many CDNs these days ship with H2 support.

Brotli support on the other hand is a completely hidden item. It is not advertised anywhere. Given the huge compression benefits for the web, I hope CDNs can begin address advertising their Brotli support soon. I hope the community can start collating a list, say on http://cdncomparison.com/ where people shopping for CDNs can decide based on Brotli support.

Additionally there is a big untapped market in the CDN world for adding Brotli compression to assets, like many CDNs do with gzip.

You can experiment with Brotli support for static assets today

At Discourse we now allow you to optionally precompile all your static assets as .br files. You can see this in action when you visit https://discuss.samsaffron.com on latest Chrome.

These are assets that never change between deploys and are always consumed on the front page. You are very likely to also have these kind of assets; your common css file and common js file.

If you have control over your asset pipeline adding support is quite simple. NGINX support though slightly more complicated is straight forward. There are huge wins to be had with the new format and adopting it for "precompiled" assets is a huge win, especially on Android mobile.

If you can perform brotli 11 compression upfront on all your most common static assets you will see very large network performance gains on supporting browsers.

Hold off on fighting with Brotli support for dynamically generated requests unless you fit the correct profile

On-the-fly Brotli compression is an area that needs a lot of experimentation. It is not enough to flick the switch at level 5 and forget about it.

There is a clear win enabling brotli level 1 if you had gzip level 1 enabled.

There is a clear win enabling brotli level 5 if you were comfortable with the CPU cost of gzip level 9 on dynamic assets and have capacity. (in this case you can expect a 5-10% win size wise)

In some cases there is no clear win enabling brotli on dynamic content.

I will follow up this article with another properly addressing the question of on-the-fly brotli compression.

Lobby Microsoft and Apple to add Brotli support

It is unclear when Microsoft and Apple plan to adopt Brotli in their browsers. This is particularly important for mobile browsers where every byte count quite a lot!

Brotli is an exciting technology, which you can start adopting today, big thanks to Google, Jyrki Alakuijala and Zoltán Szabadka!

Big thank you to Zoltán, Ilya and Jeff for reviewing this this article.

What is the fastest and most elegant way to instrument a method in Ruby?

Lately I needed to add some instruments into Discourse. I wanted every request to measure:

• How many SQL statements and Redis commands are executed?

• How much time was spent in those sections?

This can easily be abstracted to a more generalize problem of

class ExternalClass
   def method_to_measure
   end
end

Solving this problem breaks down into a few particular sub-problems

1. What is the fastest way of measuring elapsed time in a method?

2. What is the fastest patch to a method that can add instrumentation?

3. How does one account for multithreading?

Fastest way to measure a method

The first and simplest problem to solve is pretending we do not have to patch anything.

If we had access to all the source and could add all the instruments we wanted to, what is the quickest way of measuring elapsed time?

There are two techniques we can use in Ruby.

1. Create a Time object before and after and delta.

2. Use Process.clock_gettime to return a Float before and after and delta.

Let's measure:

require 'benchmark/ips'

class Test
  attr_reader :count, :duration

  def initialize
    @count = 0
    @duration = 0.0
  end

  def work
  end

  def method
    work
  end

  def time_method
    @count += 1
    t = Time.now
    work
  ensure
    @duration += Time.now - t
  end

  def process_clock_get_time_method
    @count += 1
    t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    work
  ensure
    @duration += Process.clock_gettime(Process::CLOCK_MONOTONIC) - t
  end
end


t = Test.new

Benchmark.ips do |b|
  b.report "method" do |times|
    i = 0
    while i < times
      t.method
      i += 1
    end
  end

  b.report "time_method" do |times|
    i = 0
    while i < times
      t.time_method
      i += 1
    end
  end

  b.report "process_clock_get_time_method" do |times|
    i = 0
    while i < times
      t.process_clock_get_time_method
      i += 1
    end
  end

end

# Calculating -------------------------------------
#              method     19.623M (± 3.5%) i/s -     98.227M in   5.012204s
#         time_method      1.596M (± 1.2%) i/s -      8.061M in   5.050321s
# process_clock_get_time_method
#                          4.972M (± 1.7%) i/s -     24.908M in   5.011634s

As expected Process.clock_gettime(Process::CLOCK_MONOTONIC) is fastest cause no ruby object is allocated and later garbage collected, it also happens to be the most accurate way of measuring duration.

So let's use that from now.

A note on "not doing work"

Our method above was doing no work, if we start doing work, the difference will be much less observable.

For example, if I replace work with $redis.get('x') which is pretty much the shortest time I want to instrument I will get:

              method     17.567k (± 3.0%) i/s -     88.230k in   5.027068s
         time_method     16.730k (± 3.2%) i/s -     84.864k in   5.077868s
process_clock_get_time_method
                         17.112k (± 2.7%) i/s -     85.884k in   5.022470s

So we have 17.5 vs 17.11 vs 16.7 thousand redis gets a second. Fastest being uninstrumented, slowest being using Time.now.

This is not a HUGE difference like the original benchmark, but you can not argue that Time.now is a tax you want to pay when an faster and safer alternative exists.

Fastest way to "patch" a method

When instrumenting code we do not own, we do not have the luxury of making source level changes.

This leaves us with 3 alternatives:

• Using prepend to prepend a module in the class and override the original method
• Using alias_method and chaining our new method around the original
• Replacing the method with a "patched" version of the method

Replacing the method is fraught with risk, cause once the original method in a 3rd party library changes we need to fix our replacement. It is unmaintainable long term. An interesting option that may pop up later is outputting the instruction sequence using RubyVM::InstructionSequence#of , patching the instruction sequence and then loading up the method. This will clearly be the fastest way of patching, but is not quite available yet cause instruction sequence manipulation is still a bit of a dark art.

So let's compare prepend with alias_method, what is going to be faster?

require 'benchmark/ips'
#require 'redis'

$redis = Redis.new

class Test
  attr_reader :count

  def initialize
    @count = 0
  end

  def work
  #  $redis.get 'X'
  end

  def method
    work
  end

  def method_with_prepend
    work
  end

  def method_with_alias_method
    work
  end

  def method_with_manual_instrument
    @count += 1
    work
  end
end

module PrependInstrument
  def method_with_prepend
    @count += 1
    super
  end
end

class Test; prepend PrependInstrument; end

class Test
  alias_method :method_with_alias_method_orig, :method_with_alias_method

  def method_with_alias_method
    @count += 1
    method_with_alias_method_orig
  end
end

t = Test.new

Benchmark.ips do |b|
  b.report "method" do |times|
    i = 0
    while i < times
      t.method
      i += 1
    end
  end

  b.report "method_with_prepend" do |times|
    i = 0
    while i < times
      t.method_with_prepend
      i += 1
    end
  end

  b.report "method_with_alias_method" do |times|
    i = 0
    while i < times
      t.method_with_alias_method
      i += 1
    end
  end

  b.report "method_with_manual_instrument" do |times|
    i = 0
    while i < times
      t.method_with_manual_instrument
      i += 1
    end
  end
end


#               method     20.403M (± 1.6%) i/s -    102.084M in   5.004684s
#  method_with_prepend     10.339M (± 1.5%) i/s -     51.777M in   5.009321s
# method_with_alias_method
#                         13.067M (± 1.8%) i/s -     65.649M in   5.025786s
# method_with_manual_instrument
#                         16.581M (± 1.6%) i/s -     83.145M in   5.015797s

So it looks like the very elegant prepend method of patching methods is slower than the old school alias method trick.

However once you introduce work the difference is far more marginal, this is the same bench with $redis.get "x" as the unit of work

              method     17.259k (± 3.1%) i/s -     86.892k in   5.039493s
 method_with_prepend     17.056k (± 2.8%) i/s -     85.782k in   5.033548s
method_with_alias_method
                         17.464k (± 2.4%) i/s -     87.464k in   5.011238s
method_with_manual_instrument
                         17.369k (± 2.8%) i/s -     87.204k in   5.024699s

Performance is so close it is hard to notice anything.

Accounting for threading

One big real world concern I had for my use case is ensuring I only measure work from my current thread.

In Discourse some background threads could be running doing deferred work and I do not want that to interfere with the measurements on the thread servicing the web. In a Puma based environment this is supercritical cause lots of threads could be accessing the same global thread safe Redis connection, you may also want to "enable" instrumenting prior to having a connection open.

So we can start by measuring the impact of thread safety.

require 'benchmark/ips'
require 'redis'

$redis = Redis.new

class Test
  attr_reader :count

  def initialize
    @count = 0
    @thread = Thread.current

    data = Struct.new(:count).new
    data.count = 0

    Thread.current["prof"] = data
  end

  def work
    # $redis.get 'X'
  end

  def method
    work
  end

  def method_with_safety
    @count += 1 if Thread.current == @thread
    work
  end

  def method_with_thread_storage
    if data = Thread.current["count"]
      data.count += 1
    end
    work
  end

end


t = Test.new

Benchmark.ips do |b|
  b.report "method" do |times|
    i = 0
    while i < times
      t.method
      i += 1
    end
  end

  b.report "method_with_safety" do |times|
    i = 0
    while i < times
      t.method_with_safety
      i += 1
    end
  end

  b.report "method_with_thread_storage" do |times|
    i = 0
    while i < times
      t.method_with_thread_storage
      i += 1
    end
  end

end

There are 2 mechanisms we can account for thread safety:

1. Only run instrumentation if Thread.current is the thread being profiled

2. Store instrumentation data on Thread.current (note an external registry is no faster)

We can see the faster method is using a Thread.current comparison. However, once you start doing work the difference is quite marginal. Safety comes at a cost but it is not a ridiculous cost.

### NO WORK
              method     20.119M (± 3.1%) i/s -    100.579M in   5.004172s
  method_with_safety     11.254M (± 3.0%) i/s -     56.509M in   5.025966s
method_with_thread_storage
                          6.105M (± 3.5%) i/s -     30.559M in   5.011887s

With $redis.get 'x'

              method     18.007k (± 4.8%) i/s -     91.052k in   5.069131s
  method_with_safety     18.081k (± 2.9%) i/s -     91.156k in   5.045869s
method_with_thread_storage
                         18.042k (± 2.8%) i/s -     90.532k in   5.021889s

A generic method profiler

So now, we can piece all of the bits together use the fastest micro benches and build up a general method profiler we can use to gather metrics.

# called once to patch in instrumentation
MethodProfiler.patch(SomeClass, [:method1, :method2], :name)

SomeClass.new.method1
SomeClass.new.method2

# called when starting profiling
MethodProfiler.start
result = MethodProfiler.stop

{
   total_duration: 0.2,
   name: {duration: 0.111, calls: 12}
}

Which can be implemented with:

require 'benchmark/ips'
require 'redis'
class MethodProfiler
  def self.patch(klass, methods, name)
    patches = methods.map do |method_name|<<~RUBY
      unless defined?(#{method_name}__mp_unpatched)
        alias_method :#{method_name}__mp_unpatched, :#{method_name}
        def #{method_name}(*args, &blk)
          unless prof = Thread.current[:_method_profiler]
            return #{method_name}__mp_unpatched(*args, &blk)
          end
          begin
            start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
            #{method_name}__mp_unpatched(*args, &blk)
          ensure
            data = (prof[:#{name}] ||= {duration: 0.0, calls: 0})
            data[:duration] += Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
            data[:calls] += 1
          end
        end
      end
      RUBY
    end.join("\n")

    klass.class_eval patches
  end

  def self.start
    Thread.current[:_method_profiler] = {
      __start: Process.clock_gettime(Process::CLOCK_MONOTONIC)
    }
  end

  def self.stop
    finish = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    if data = Thread.current[:_method_profiler]
      Thread.current[:_method_profiler] = nil
      start = data.delete(:__start)
      data[:total_duration] = finish - start
      data
    end
  end
end

$redis = Redis.new

class Test
  def work
    #$redis.get 'X'
  end

  def method
    work
  end

  def method_unpatched
    work
  end
end


MethodProfiler.patch(Test, [:method], :a_thing)
MethodProfiler.start

t = Test.new

Benchmark.ips do |b|
  b.report "method" do |times|
    i = 0
    while i < times
      t.method
      i += 1
    end
  end

  b.report "method_unpatched" do |times|
    i = 0
    while i < times
      t.method_unpatched
      i += 1
    end
  end

end

Link to Gist

Performance wish there is a definite impact for the "no work scenario"

(while profiling)

              method      2.247M (± 1.6%) i/s -     11.370M in   5.061768s
    method_unpatched     20.261M (± 3.5%) i/s -    101.226M in   5.002638s

(without profiling)

              method      6.015M (± 2.9%) i/s -     30.156M in   5.017655s
    method_unpatched     20.211M (± 2.1%) i/s -    101.288M in   5.013976s

However once we actually start doing work with say $redis.get 'x'

(while profiling)
              method     17.419k (± 3.5%) i/s -     87.108k in   5.006821s
    method_unpatched     17.627k (± 7.3%) i/s -     88.740k in   5.063757s


(without profiling)

              method     17.293k (± 3.1%) i/s -     87.672k in   5.074642s
    method_unpatched     17.478k (± 2.4%) i/s -     88.556k in   5.069598s

Conclusion

Building on all the micro benchmarks we can design a generic class for profiling methods quite efficiently in Ruby with very minimal runtime impact.

Feel free to borrow any of the code in this article with or without attribution and build upon it.

How do you go about debugging high CPU usage in a production Rails system?

Today I noticed one of our customer containers was running really high on CPU.

# top -bn1

190 discour+  20   0 2266292 205128  15656 S  86.7  0.3   9377:35 ruby

# ps aux

discour+   190 19.4  0.3 2266292 207096 ?      Sl    2017 9364:38 sidekiq 5.0.5 discourse [1 of 5 busy]

Looks like sidekiq is stuck on a job.

Where is it stuck?

Usually, this is where the story ends and another series of questions start

• Can we reproduce this on staging or development?

• What code changed recently?

• Why is perf trace not giving me anything I can work with?

• How awesome is my Sidekiq logging?

• Where is my divining rod?

Julia Evans is working on a fantastic profiler that will allow us to answer this kind of question real quick. But in the meantime, is there anything we can do?

rbtrace + stackprof

At Discourse we include rbtrace and stackprof in our Gemfile.

gem 'rbtrace'
gem 'stackprof', require: false

We always load up rbtrace in production, it allows us a large variety of production level debugging. stackprof is loaded on-demand.

In this particular case I simply run:

# rbtrace -p 190 -e 'Thread.new{ require "stackprof"; StackProf.start(mode: :cpu); sleep 2; StackProf.stop;

This injects a new thread that enables stackprof globally and finally writes the performance data to /tmp/perf

This dump can easily be analyzed:

StackProf.results("/tmp/perf") }'
# stackprof /tmp/perf
==================================
  Mode: cpu(1000)
  Samples: 475 (0.63% miss rate)
  GC: 0 (0.00%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
       473  (99.6%)         473  (99.6%)     #<Class:0x00007fac08218020>::Typography.to_html_french
         4   (0.8%)           1   (0.2%)     #<Module:0x00007fac080c3620>.reap_connections

In fact, we can even collect backtraces and generate flamegraphs in stackprof, so I strongly recommend reading through the readme.

So there you have it, my culprit is:

text.gsub(/(\s|)+([!?;]+(\s|\z))/, ' \2\3')

A regular expression consuming 100% CPU in production, this has got to be a first

How we instrument Rails at Discourse and how you can, too.

People following me have occasionally seen me post graphs like this:

image.png1400x391 74.2 KB

Usually people leave this type of instrumentation and graphing to NewRelic and Skylight. However, at our scale we find it extremely beneficial to have instrumentation, graphing and monitoring local cause we are in the business of hosting, this is a central part of our job.

Over the past few years Prometheus has emerged as one of the leading options for gathering metrics and alerting. However, sadly, people using Rails have had a very hard time extracting metrics.

Issue #9 on the official prometheus client for Ruby has been open 3 years now, and there is very little chance it will be “solved” any time soon.

The underlying fundamental issue is that Prometheus, unlike Graphite/Statsd is centered around the concept pulling metrics as opposed to pushing metrics.

This means you must provide a single HTTP endpoint that collects all the metrics you want exposed. This ends up being particularly complicated with Unicorn/Puma and Passenger who usually will run multiple forks of a process. If you simply implement a secured /metrics endpoint in your app, you have no guarantees over which forked process will handle the request, without “cross fork” aggregation you would just report metrics for a single, random, process. Which is less than useful.

Additionally, knowing what to collect and how to collect it is a bit of an art, it can easily take multiple week just to figure out what you want.

Having solved this big problem for Discourse I spent some time extracting the patterns.

Introducing prometheus_exporter

The prometheus_exporter gem is a toolkit that provides all the facilities you need.

1. It has an extensible collector that allows you to run a single process to aggregate metrics for multiple processes on one machine.

2. It implements gauge, counter and summary metrics.

3. It has default instrumentation that you can easily add to your app

4. It has a very efficient and robust transport channel between forked processes and master collector. The master collector gathers metrics via HTTP but reduces overhead by using chunked encoding so a single session can gather a very large amount of metrics.

5. It exposes metrics to prometheus over a dedicated port, HTTP endpoint is compressed.

6. It is completely extensible, you can pick as much or as little as you want.

A minimal example implementing metrics for your Rails app

In your Gemfile:

gem 'prometheus_exporter'

# in config/initializers/prometheus.rb
if Rails.env != "test"
  require 'prometheus_exporter/middleware'

  # This reports stats per request like HTTP status and timings
  Rails.application.middleware.unshift PrometheusExporter::Middleware
end

At this point, your web is instrumented, every request will keep track of SQL/Redis/Total time (provided you are using PG)

image.png1402x390 105 KB

You may also be interested in per-process stats, like:

image.png1414x402 59.7 KB

and

image.png1409x411 144 KB

# in config/initializers/prometheus.rb
if Rails.env != "test"
  require 'prometheus_exporter/instrumentation'

  # this reports basic process stats like RSS and GC info, type master
  # means it is instrumenting the master process
  PrometheusExporter::Instrumentation::Process.start(type: "master")
end

# in unicorn/puma/passenger be sure to run a new process instrumenter after fork
after_fork do
  require 'prometheus_exporter/instrumentation'
  PrometheusExporter::Instrumentation::Process.start(type:"web")
end

Also you may be interested in some Sidekiq stats:

image.png1408x509 214 KB

Sidekiq.configure_server do |config|
   config.server_middleware do |chain|
      require 'prometheus_exporter/instrumentation'
      chain.add PrometheusExporter::Instrumentation::Sidekiq
   end
end

FInally, you may want to collect some global stats across all processes, like:

image.png1411x421 42.3 KB

To do so we can introduce a “type collector”:

# lib/global_type_collector.rb
unless defined? Rails
  require File.expand_path("../../config/environment", __FILE__)
end

require 'raindrops'

class GlobalPrometheusCollector < PrometheusExporter::Server::TypeCollector
  include PrometheusExporter::Metric

  def initialize
    @web_queued = Gauge.new("web_queued", "Number of queued web requests")
    @web_active = Gauge.new("web_active", "Number of active web requests")
  end

  def type
    "app_global"
  end

  def observe(obj)
    # do nothing, we would only use this if metrics are transported from apps
  end

  def metrics
    path = "/var/www/my_app/tmp/sockets/unicorn.sock"
    info = Raindrops::Linux.unix_listener_stats([path])[path]
    @web_active.observe(info.active)
    @web_queued.observe(info.queued)

    [
      @web_queued,
      @web_active
    ]
  end
end

After all of this is done you need to run the collector (in a monitored process in production) using runit ,supervisord, systemd or whatever your poison is (mine is runit).

bundle exec prometheus_exporter -t /var/www/my_app/lib/global_app_collector.rb

Then you follow the various guides online and setup Prometheus and the excellent Grafana and you too can have wonderful graphs.

For those curious, here is an partial example of how the raw metric feed looks for an internal app we use that I instrumented yesterday: https://gist.github.com/SamSaffron/e2e0c404ff0bacf5fbca80163b54f0a4

I hope you find this helpful, good luck instrumenting all things!

It is very likely your Rails application is full of duplicate strings, here are some tricks you can use to get rid of them.

One very common problem Ruby and Rails have is memory usage. Often when hosting sites the bottleneck is memory not performance. At Discourse we spend a fair amount of time tuning our application so self hosters can afford to host Discourse on 1GB droplets.

To help debug memory usage I created the memory_profiler gem, it allows you to easily report on application memory usage. I highly recommend you give it a shot on your Rails app, it is often surprising how much low hanging fruit there is. On unoptimized applications you can often reduce memory usage by 20-30% in a single day of work.

Memory profiler generates a memory usage report broken into 2 parts:

Allocated memory

Memory you allocated during the block that was measured.

Retained memory

Memory that remains in used after the block being measure executed.

So, for example:

def get_obj
   allocated_object1 = "hello "
   allocated_object2 = "world"
   allocated_object1 + allocated_object2
end

retained_object = nil

MemoryProfiler.report do
   retained_object = get_obj
end.pretty_print 

Will be broken up as:

[a lot more text]
Allocated String Report
-----------------------------------
         1  "hello "
         1  blog.rb:3

         1  "hello world"
         1  blog.rb:5

         1  "world"
         1  blog.rb:4


Retained String Report
-----------------------------------
         1  "hello world"
         1  blog.rb:5

As a general rule we focus on reducing retained memory when we want our process to consume less memory and we focus on reducing allocated memory when optimising hot code paths.

For the purpose of this blog post I would like to focus on retained memory optimisation and in particular in the String portion of memory retained.

How you can get memory profiler report for your Rails app?

We use the following script to profile Rails boot time:

if ENV['RAILS_ENV'] != "production"
  exec "RAILS_ENV=production ruby #{__FILE__}"
end

require 'memory_profiler'

MemoryProfiler.report do
  # this assumes file lives in /scripts directory, adjust to taste...
  require File.expand_path("../../config/environment", __FILE__)

  # we have to warm up the rails router
  Rails.application.routes.recognize_path('abc') rescue nil

  # load up the yaml for the localization bits, in master process
  I18n.t(:posts)

  # load up all models so AR warms up internal caches
  (ActiveRecord::Base.connection.tables - %w[schema_migrations versions]).each do |table|
    table.classify.constantize.first rescue nil
  end
end.pretty_print

You can see an example of such a report here:

Very early on in my journey of optimizing memory usage I noticed that Strings are a huge portion of the retained memory. To help cutting down on String usage memory_profiler has a dedicated String section.

For example in the report above you can see:

Retained String Report
-----------------------------------
       942  "format"
       940  /home/sam/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/nodes/node.rb:83
         1  /home/sam/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/actionpack-5.1.4/lib/action_controller/log_subscriber.rb:3
         1  /home/sam/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/activemodel-5.1.4/lib/active_model/validations/validates.rb:115

       941  ":format"
       940  /home/sam/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/actionpack-5.1.4/lib/action_dispatch/journey/scanner.rb:49
         1  /home/sam/.rbenv/versions/2.5.0/lib/ruby/gems/2.5.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292
... a lot more ...

We can see that there are 940 copies of the string "format" living in my Ruby heaps. These strings are all “rooted” so they just sit there in the heap and never get collected. Rails needs the 940 copies so it can quickly figure out what params my controller should get.

In Ruby RVALUEs (slots on the Ruby heap / unique object_ids) will consume 40 bytes on x64. The string “format” is quite short so it fits in a single RVALUE without an external pointer or extra malloc. Still, this is 37,600 bytes just to store the single string “format”. That is clearly wasteful, we should send a PR to Rails.

It is wasteful on a few counts:

1. Every object in the Ruby heap is going to get scanned every time a full GC runs, from now till the process eventually dies.

2. Small chunks of memory do not fit perfectly into your process address space, memory fragments over time and the actual impact of a 40 byte RVALUE may end up being more due to gaps between RVALUE heaps.

3. The larger your Ruby heaps are the faster they grow (out-of-the-box): https://bugs.ruby-lang.org/issues/12967

4. A single RVALUE in a Ruby heap that contains 500 or so RVALUEs can stop it from being reclaimed

5. More objects means less efficient CPU caching, more chances of hitting swap and so on.

Techniques for string deduplication

I created this Gist to cover quite a bit of the nuance around the techniques you can use for string deduplication in Ruby 2.5 and up, for those feeling brave, I recommend you spend some time reading it carefully:

For those who prefer words, well here are some techniques you can use:

Use constants

# before
def give_me_something
   "something"
end

# after
SOMETHING = "something".freeze

def give_me_something
   SOMETHING
end

Advantages:

• Works in all versions of Ruby

Disadvantages:

• Ugly and verbose
• If you forget the magic “freeze” you may not reuse the string properly Ruby > 2.3

Use the magic frozen_string_literal: true comment

# before
def give_me_something
   "something"
end

# after

# frozen_string_literal: true
def give_me_something
   "something"
end

Ruby 2.3 introduces the frozen_string_literal: true pragma. When the comment # frozen_string_literal: true is the first line of your file, Ruby treats the file differently.

Every simple string literal is frozen and deduplicated.

Every interpolated string is frozen and not deduplicated. Eg x = "#{y}" is a frozen non deduplicated string.

I feel this should be the default for Ruby and many projects are embracing this including Rails. Hopefully this becomes the default for Ruby 3.0.

Advantages:

• Very easy to use
• Not ugly
• Long term this enables fancier optimisations

Disadvantages:

• Can be complicated to apply on existing files, a great test suite is highly recommended.

Pitfalls:

There are a few cliffs you can fall which you should be careful about. Biggest is the default encoding on String.new

buffer = String.new
buffer.encoding => Encoding::ASCII-8BIT

# vs 

# String @+ is new in Ruby 2.3 and up it allows you to unfreeze
buffer = +""
buffer.encoding => Encoding::UTF-8

Usually this nuance will not matter to you at all cause as soon as you append to the String it will switch encoding, however if you are passing refs to 3rd party library of the empty string you created havoc can ensue. So, "".dup or +"" is a good habit.

Dynamic string deduplication

Ruby 2.5 introduces a new techniques you can use to deduplicate strings. It was introduced in https://bugs.ruby-lang.org/issues/13077 by Eric Wong.

To quote Matz

For the time being, let us make -@ to call rb_fstring.
If users want more descriptive name, let’s discuss later.
In my opinion, fstring is not acceptable.

So, String’s @- method will allow you to dynamically de-duplicate strings.

a = "hello"
b = "hello"
puts ((-a).object_id == (-b).object_id) # I am true in Ruby 2.5 (usually) 

This syntax exists in Ruby 2.3 and up, the optimisation though is only available in Ruby 2.5 and up.

This technique is safe, meaning that string you deduplicate still get garbage collected.

It relies on a facility that has existed in Ruby for quite a while where it maintains a hash table of deduplicated strings:

The table was used in the past for the "string".freeze optimisation and automatic Hash key deduplication. Ruby 2.5 is the first time this feature is exposed to the general public.

It is incredibly useful when parsing input with duplicate content (like the Rails routes) and when generating dynamic lookup tables.

However, it is not all s

Firstly, some people’s sense of aesthetics is severely offended by the ugly syntax. Some are offended so much they refuse to use it.

Additionally this technique has a bunch of pitfalls documented in extreme levels here.

Until https://bugs.ruby-lang.org/issues/14478 is fixed you need to “unfreeze” strings prior to deduping

yuck = "yuck"
yuck.freeze
yuck_deduped = -+yuck

If a string is tainted you can only “partially” dedupe it

This means the VM will create a shared string for long strings, but will still maintain the RVALUE

love = "love"
love.taint
(-love).object_id == love.object_id 

# got to make a copy to dedupe
deduped = -love.dup.untaint

Trouble is lots of places that want to apply this fix end up trading in tainted strings, a classic example is the postgres adapter for Rails that has 142 copies of "character varying" in the Discourse report from above. In some cases this limitation means we are stuck with an extra and pointless copy of the string just cause we want to deduplicate (cause untainting may be unacceptable for the 3 people in the universe using the feature).

Personally, I wish we just nuked all the messy tainting code from Ruby’s codebase , which would make it both simpler, safer and faster.

If a string has any instance vars defined you can only partially dedupe

# html_safe sets an ivar on String so it will not be deduplicated
str = -"<html>test</html>".html_safe 

This particular limitation is unavoidable and I am not sure there is anything Ruby can do to help us out here. So, if you are looking to deduplicate fragments of html, well, you are in a bind, you can share the string, you can not deduplicate it perfectly.

Additional reading:

Good luck, reducing your application’s memory usage, I hope this helps!

How we manage schema changes at Discourse minimizing downtime

At Discourse we have always been huge fans of continuous deployment. Every commit we make heads to our continuous integration test suite. If all the tests pass (ui, unit, integration, smoke) we automatically deploy the latest version of our code to https://meta.discourse.org.

This pattern and practice we follow allows the thousands self-installers out there to safely upgrade to the tests-passed version whenever they feel like it.

Because we deploy so often we need to take extra care not to have any outages during deployments. One of the most common reasons for outages during application deployment is database schema changes.

The problem with schema changes

Our current deployment mechanism roughly goes as follows:

• Migrate database to new schema
• Bundle up application into a single docker image
• Push to registry
• Spin down old instance, pull new instance, spin up new instance (and repeat)

If we ever create an incompatible database schema we risk breaking all the old application instances running older versions of our code. In practice, this can lead to tens of minutes of outage!

In ActiveRecord the situation is particularly dire cause in production the database schema is cached and any changes in schema that drop or rename columns very quickly risk breaking every query to the affected model raising invalid schema exceptions.

Over the years we have introduced various patterns to overcome this problem and enable us to deploy schema changes safely, minimizing outages.

Tracking rich information about migrations

ActiveRecord has a table called schema_migrations where is stores information about migrations that ran.

Unfortunately the amount of data stored in this table is extremely limited, in fact it boils down to:

connection.create_table(table_name, id: false) do |t|
  t.string :version, version_options
end

The table has a lonely column storing the “version” of migrations that ran.

1. It does not store when the migration ran
2. It does not store how long it took the migration to run
3. It has nothing about the version of Rails that was running when the migration ran

This lack of information, especially, not knowing when stuff ran makes creating clean systems for dealing with schema changes hard to build. Additionally, debugging strange and wonderful issues with migrations is very hard without rich information.

Discourse, monkey patches Rails to log rich information about migrations:

Our patch provides us a very rich details surrounding all the migration circumstances. This really should be in Rails.

Defer dropping columns

Since we “know” when all previous migrations ran due to our rich migration logging, we are able to “defer drop” columns.

What this means is that we can guarantee we perform dangerous schema changes after we know that the new code is in place to handle the schema change.

In practice if we wish to drop a column we do not use migrations for it. Instead our db/seed takes care of defer dropping.

These defer drops will happen at least 30 minutes after the particular migration referenced ran (in the next migration cycle), giving us peace of mind that the new application code is in place.

If we wish to rename a column we will create a new column, duplicate the value into the new column, mark the old column readonly using a trigger and defer drop old column.

If we wish to drop or rename a table we follow a similar pattern.

The logic for defer dropping lives in ColumnDropper and TableDropper.

Not trusting ourselves

A big problem with spectacular special snowflake per-application practices is enforcement.

We have great patterns for ensuring safety, however sometimes people forget that we should never drop a column or a table the ActiveRecord migration way.

To ensure we never make the mistake of committing dangerous schema changes into our migrations, we patch the PG gem to disallow certain statements when we run them in the context of a migration.

Want to DROP TABLE? Sorry, an exception will be raised. Want to DROP a column, an exception will be raised.

This makes it impractical to commit highly risky schema changes without following our best practices:

== 20180321015226 DropRandomColumnFromUser: migrating =========================
-- remove_column(:categories, :name)

WARNING
-------------------------------------------------------------------------------------
An attempt was made to drop or rename a column in a migration
SQL used was: 'ALTER TABLE "categories" DROP "name"'
Please use the deferred pattrn using Migration::ColumnDropper in db/seeds to drop
or rename columns.

Note, to minimize disruption use self.ignored_columns = ["column name"] on your
ActiveRecord model, this can be removed 6 months or so later.

This protection is in place to protect us against dropping columns that are currently
in use by live applications.
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Attempt was made to rename or delete column
/home/sam/Source/discourse/db/migrate/20180321015226_drop_random_column_from_user.rb:3:in `up'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

This logic lives in safe_migrate.rb. Since this is a recent pattern we only enforce it for migrations after a certain date.

Alternatives

Some of what we do is available in gem form and some is not:

Strong Migrations offers enforcement. It also takes care of a bunch of interesting conditions like nudging you to create indexes concurrently in postgres. Enforcement is done via patching active record migrator, meaning that if anyone does stuff with SQL direct it will not be caught.

Zero downtime migrations very similar to strong migrations.

Outrigger allows you to tag migrations. This enables you to amend your deploy process so some migrations run pre-deploy and some run post-deploy. This is the simplest technique for managing migrations in such a way that you can avoid downtimes during deploy.

Handcuffs: very similar to outrigger, define phases for your migrations

What should you do?

Our current pattern for defer dropping columns and tables works for us, but is not yet ideal. Code that is in charge of “seeding” data now is also in charge of amending schema and timing of column drops is not as tightly controlled as it should be.

On the upside, rake db:migrate is all you need to run and it works magically all the time. Regardless of how you are hosted and what version your schema is at.

My recommendation though for what I would consider best practice here is a mixture of a bunch of ideas. All of it belongs in Rails.

Enforcement of best practices belongs in Rails

I think enforcement of safe schema changes should be introduced into ActiveRecord. This is something everyone should be aware of. It is practical to do zero downtime deploys today with schema changes.

class RemoveColumn < ActiveRecord::Migration[7.0]
  def up
     # this should raise an error
     remove_column :posts, :name
  end
end

To make it work, everyone should be forced to add the after_deploy flag to the migration:

class RemoveColumn < ActiveRecord::Migration[7.0]
  after_deploy! # either this, or disable the option globally 
  def up
     # this should still raise if class Post has no ignored_columns: [:name]
     remove_column :posts, :name
  end
end

class RemoveColumn < ActiveRecord::Migration[7.0]
  after_deploy!(force: true)
  def up
     # this should work regardless of ignored_columns
     remove_column :posts, :name
  end
end

I also think the ideal enforcement is via SQL analysis, however it is possible that this is a bit of a can-of-worms at Rails scale. For us it is practical cause we only support one database.

rake db:migrate should continue to work just as it always did.

For backwards compatibility rake db:migrate should run all migrations including after_deploy migrations. Applications who do not care about “zero downtime deploys” should also be allowed to opt out of the safety.

New post and pre migrate rake tasks should be introduced

To run all the application code compatible migrations you would run:

rake db:migrate:pre
# runs all migrations without `after_deploy!`

To run all the destructive operations you would run:

rake db:migrate:post
# runs all migrations with `after_deploy!`

Conclusion

If you are looking to start with “safe” zero downtime deploys today I would recommend:

1. Amending build process to run pre deploy migrations and post deploy migrations (via Outrigger or Handcuffs)

2. Introduce an enforcement piece with Strong Migrations

Current patterns in Active Record lead to enormous amounts of resource usage. Here is an analysis of Rails 5.2

One of the very noble goals the Ruby community which is being spearheaded by Matz is the Ruby 3x3 plan. The idea is that using large amounts of modern optimizations we can make Ruby the interpreter 3 times faster. It is an ambitious goal, which is notable and inspiring. This “movement” has triggered quite a lot of interesting experiments in Ruby core, including a just-in-time compiler and action around reducing memory bloat out-of-the-box. If Ruby gets faster and uses less memory, then everyone gets free performance, which is exactly what we all want.

A big problem though is that there is only so much magic a faster Ruby can achieve. A faster Ruby is not going to magically fix a “bubble sort” hiding deep in your code. Active Record has tons of internal waste that ought to be addressed which could lead to the vast majority of Ruby applications in the wild getting a lot faster. Rails is the largest consumer of Ruby after all and Rails is underpinned by Active Record.

Sadly, Active Record performance has not gotten much better since the days of Rails 2, in fact in quite a few cases it got slower or a lot slower.

Active Record is very wasteful

I would like to start off with a tiny example:

Say I have a typical 30 column table containing Topics.

If I run the following, how much will Active Record allocate?

a = []
Topic.limit(1000).each do |u|
   a << u.id
end

Total allocated: 3835288 bytes (26259 objects)

Compare this to an equally inefficient “raw version”.

sql = -"select * from topics limit 1000"
ActiveRecord::Base.connection.raw_connection.async_exec(sql).column_values(0)

Total allocated: 8200 bytes (4 objects)

This amount of waste is staggering, it translates to deadly combo:

• Extreme levels of memory usage

and

• Slower performance

But … that is really bad Active Record!

An immediate gut reaction here is that I am “cheating” and writing “slow” Active Record code, and comparing it to mega optimized raw code.

One could argue that I should write:

a = []
Topic.select(:id).limit(1000).each do |u|
  a << u.id
end

In which you would get:

Total allocated: 1109357 bytes (11097 objects)

Or better still:

Topic.limit(1000).pluck(:id) 

In which I would get

Total allocated: 221493 bytes (5098 objects)

Time for a quick recap.

• The “raw” version allocated 4 objects, it was able to return 1000 Integers directly which are not allocated indevidually in the Ruby heaps and are not subject to garbage collection slots.

• The “naive” Active Record version allocates 26259 objects

• The “slightly optimised” Active Record version allocates 11097 objects

• The “very optimised” Active Record version allocates 5098 objects

All of those numbers are orders of magnitude larger than 4.

How many objects does a “naive/lazy” implementation need to allocate?

One feature that Active Record touts as a huge advantage over Sequel is the “built-in” laziness.

ActiveRecord will not bother “casting” a column to a date till you try to use it, so if for any reason you over select ActiveRecord has your back. This deficiency in Sequel is acknowledged and deliberate:

This particular niggle makes it incredibly hard to move to Sequel from ActiveRecord without extremely careful review, despite Sequel being so incredibly fast and efficient.

We have no “fastest” example out there of an efficient lazy selector. In our case we are consuming 1000 ids so we would expect the mega efficient implementation to allocate 1020 or so objects cause we can not get away without allocating a Topic object. We do not expect 26 thousand.

Here is a quick attempt at such an implementation: (note this is just proof of concept of the idea, not a production level system)

$conn = ActiveRecord::Base.connection.raw_connection

class FastBase

  class Relation
    include Enumerable

    def initialize(table)
      @table = table
    end

    def limit(limit)
      @limit = limit
      self
    end

    def to_sql
      sql = +"SELECT #{@table.columns.join(',')} from #{@table.get_table_name}"
      if @limit
        sql << -" LIMIT #{@limit}"
      end
      sql
    end

    def each
      @results = $conn.async_exec(to_sql)
      i = 0
      while i < @results.cmd_tuples
        row = @table.new
        row.attach(@results, i)
        yield row
        i += 1
      end
    end

  end

  def self.columns
    @columns
  end

  def attach(recordset, row_number)
    @recordset = recordset
    @row_number = row_number
  end

  def self.get_table_name
    @table_name
  end

  def self.table_name(val)
    @table_name = val
    load_columns
  end

  def self.load_columns
    @columns = $conn.async_exec(<<~SQL).column_values(0)
      SELECT COLUMN_NAME FROM information_schema.columns
      WHERE table_schema = 'public' AND
        table_name = '#{@table_name}'
    SQL

    @columns.each_with_index do |name, idx|
      class_eval <<~RUBY
        def #{name}
          if @recordset && !@loaded_#{name}
            @loaded_#{name} = true
            @#{name} = @recordset.getvalue(@row_number, #{idx})
          end
          @#{name}
        end

        def #{name}=(val)
          @loaded_#{name} = true
          @#{name} = val
        end
      RUBY
    end
  end

  def self.limit(number)
    Relation.new(self).limit(number)
  end
end

class Topic2 < FastBase
  table_name :topics
end

Then we can measure:

a = []
Topic2.limit(1000).each do |t|
   a << t.id
end
a

Total allocated: 84320 bytes (1012 objects)

So … we can manage a similar API with 1012 object allocations as opposed to 26 thousand objects.

Does this matter?

A quick benchmark shows us:

Calculating -------------------------------------
               magic    256.149  (± 2.3%) i/s -      1.300k in   5.078356s
                  ar     75.219  (± 2.7%) i/s -    378.000  in   5.030557s
           ar_select    196.601  (± 3.1%) i/s -    988.000  in   5.030515s
            ar_pluck      1.407k (± 4.5%) i/s -      7.050k in   5.020227s
                 raw      3.275k (± 6.2%) i/s -     16.450k in   5.043383s
             raw_all    284.419  (± 3.5%) i/s -      1.421k in   5.002106s

Our new implementation (that I call magic) does 256 iterations a second compared to Rails 75. It is a considerable improvement over the Rails implementation on multiple counts. It is both much faster and allocates significantly less memory leading to reduced process memory usage. This is despite following the non-ideal practice of over selection. In fact our implementation is so fast, it even beats Rails when it is careful only to select 1 column!

This is the Rails 3x3 we could have today with no changes to Ruby!

Another interesting data point is how much slower pluck, the turbo boosted version Rails has to offer, is slower that raw SQL. In fact, at Discourse, we monkey patch pluck exactly for this reason. (I also have a Rails 5.2 version)

Why is this bloat happening?

Looking at memory profiles I can see multiple reasons all this bloat happens:

1. Rails is only sort-of-lazy… I can see 1000s of string allocations for columns we never look at. It is not “lazy-allocating” it is partial “lazy-casting”

2. Every row allocates 3 additional objects for bookeeping and magic. ActiveModel::Attribute::FromDatabase, ActiveModel::AttributeSet, ActiveModel::LazyAttributeHash . None of this is required and instead a single array could be passed around that holds indexes to columns in the result set.

3. Rails insists on dispatching casts to helper objects even if the data retrieved is already in “the right format” (eg a number) this work generates extra bookkeeping

4. Every column name we have is allocated twice per query, this stuff could easily be cached and reused (if the query builder is aware of the column names it selected it does not need to ask the result set for them)

What should to be done?

I feel that we need to carefully review Active Record internals and consider an implementation that allocates significantly less objects per row. We also should start leveraging the PG gem’s native type casting to avoid pulling strings out of the database only to convert them back to numbers.

You can see the script I used for this evaluation over here:

In this post I would like to cover a severe, extremely hard to debug vector for memory bloat in Ruby which can be triggered by the PG gem.

I have blogged a bit about the Ruby GC previously and covered some basics about malloc triggering GC runs. Over the years much in that blog post has been addressed in Ruby including dynamically growing malloc limits that mean we very rarely would need to amend malloc related GC vars.

As an aside, the only GC var Discourse still overrides is RUBY_GLOBAL_METHOD_CACHE_SIZE for reasons that are specified in the Shopify blog post by Scott Francis.

The GC in Ruby can be triggered by 2 different types of conditions.

1. We are out of space in our managed heaps.

2. We detected that data associated with Ruby objects via malloc calls has grown beyond a certain threshold.

In this blog post I am covering (2) and demonstrating what happens when Ruby is not aware of malloc calls.

Why malloc calls can trigger a GC?

When reading through GC.stat we may be a bit surprised to see the amount of malloc related accounting:

malloc_increase_bytes
malloc_increase_bytes_limit
oldmalloc_increase_bytes
oldmalloc_increase_bytes_limit

We keep track of the amount of memory allocated using malloc, if it hits the malloc_increase_bytes_limit we will trigger a minor GC.

When we promote an object to the old generation we also try to estimate how much malloc increased since the last major GC. This way when we promote large objects from a young heap to an old heap we have a chance to GC as soon oldmalloc_increase_bytes_limit is hit.

The oldmalloc_increase_bytes_limit and malloc_increase_bytes_limit dynamically size themselves growing as we hit GCs due to malloc limits.

Seeing this in action

Having this in place allows us to run code like this without bloating memory:

def count_malloc(desc)
  start = GC.stat[:malloc_increase_bytes]
  yield
  delta = GC.stat[:malloc_increase_bytes] - start
  puts "#{desc} allocated #{delta} bytes"
end

def process_rss
  puts 'RSS is: ' + `ps -o rss -p #{$$}`.chomp.split("\n").last
end

def malloc_limits
  s = GC.stat
  puts "malloc limit #{s[:malloc_increase_bytes_limit]}, old object malloc limit #{s[:oldmalloc_increase_bytes_limit]}"
end

puts "start RSS/limits"
process_rss
malloc_limits

count_malloc("100,000 byte string") do
  "x" * 100_000
end

x = []
10_000.times do |i|
  x[i%10]  = "x" * 100_000
end

puts "RSS/limits after allocating 10k 100,000 byte string"
malloc_limits
process_rss

Result is:

start RSS/limits
RSS is: 11692

malloc limit 16777216, old object malloc limit 16777216
100,000 byte string allocated 103296 bytes

RSS/limits after allocating 10k 100,000 byte string
malloc limit 32883343, old object malloc limit 78406160

RSS is: 42316

The key figures to watch here is.

1. malloc_increase_bytes_limit starts at 16MB and moves up to 32MB

2. oldmalloc_increase_bytes_limit starts at 16MB and moves up to 78MB

3. RSS moves up from 11MB to 42MB

To recap this is a fairly well behaved non bloated process, despite allocating pretty gigantic objects (strings that have 100,000 bytes in them) and retaining a handful (10).

This is what we want and it gets a stamp of approval!

Where malloc accounting falls over!

Ruby does not “monkey patch” the libc malloc function to figure out how much memory got allocated.

It requires c extension authors to be very careful about how they allocate memory, in particular extension authors are expected to use all sorts of helper macros and functions when allocating and converting memory that will be tied to Ruby objects.

Unfortunately, some gems that package up c libraries do not use the helpers in some cases. This is often nobody’s explicit fault, but a culmination of a very sad series of coincidences.

I have been looking at improving Active Record performance recently and was very surprised to see this pattern everywhere:

Every time we are running a piece of SQL and getting a perfectly good PG::Result back we convert it to an array of arrays that is 100% materialized and manually discard the PG::Result object. Why is this?

Turns out, this is there for a very good reason ™

If we adapt our sample to use the PG gem to allocate the strings we see this:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'pg'
end

require 'pg'

conn = PG.connect(dbname: 'test_db')
sql = "select repeat('x', $1)"

# simulate a Rails app by long term retaining 400_000 objects

puts "start RSS/limits"
process_rss
malloc_limits

count_malloc("100,000 bytes PG") do
  conn.exec(sql, [100_000])
end

x = []
10_000.times do |i|
  r = x[i%10] = conn.exec(sql, [100_000])
  r.clear
end

puts "RSS/limits after allocating 10k 100,000 byte strings in libpq (and clearing)"
malloc_limits
process_rss

10_000.times do |i|
  x[i%10] = conn.exec(sql, [100_000])
end

puts "RSS/limits after allocating 10k 100,000 byte strings in libpq (and NOT clearing)"
malloc_limits
process_rss

We get this:

start RSS/limits
RSS is: 27392
malloc limit 16777216, old object malloc limit 16777216
100,000 bytes PG allocated 960 bytes
RSS/limits after allocating 10k 100,000 byte strings in libpq (and clearing)
malloc limit 16777216, old object malloc limit 16777216
RSS is: 27636
RSS/limits after allocating 10k 100,000 byte strings in libpq (and NOT clearing)
malloc limit 16777216, old object malloc limit 16777216
RSS is: 295500

our RSS just jumped to 295MB when we forgot to run #clear on the results PG gave us!!!

Further more we can make the problem WAY worse if we simulate a Rails App by growing our Ruby heaps first with:

$long_term = []
400_000.times do
  $long_term << +""
end

If we run that code first we reach 1GB of RSS after “forgetting” to clear our PG::Result object!

We can see PG allocated 100,000 bytes but Ruby was only aware of 960.

Aaron Patterson has been aware of this issue for many years, in fact he has attempted to patch libpq the library that powers the PG gem so it can handle this exact case gracefully.

See: PostgreSQL: Custom allocators in libpq

So where does this leave us?

At Discourse we notice occasional bloat in our Sidekiq process. This is despite being extremely careful to run a specific version of jemalloc that tames memory quite a bit.

Now that I am aware of this vector I do have my suspicion that some “Raw SQL” helpers we have lurking in Discourse can cause this issue. In particular we have places that return results directly in a PG::Result object. In Sidekiq, under heavy concurrency with a very large heap these objects can sneak into the old generation and be retained for way too long leading to process bloat.

This thorn also makes it very hard for us to tame Active Record memory usage cause we are stuck relying on copying entire result sets so we can stay safe, which is a very high priority for Rails.

That said, I have not given up quite yet and see quite a few paths forward. (none of which conflict):

It would be nice to drive Aaron’s patch home, if libpq provided better hooks for memory allocation we could nip this problem at the bud.

Advantages

• This would resolve the problem at the source

Disadvantages

• Even if this is accepted today it will be many years till people can lean on this, requires a new version of libpq many people run 5 year old versions of it.

It would be nice to have an API in libpq that allows us to interrogate how many bytes are allocated to a result it returns.

Advantages

• This would resolve the problem at the source.
• A much easier patch to land in libpq.
• Ruby 2.4 and up have rb_gc_adjust_memory_usage, per #12690, so it is simple to make this change. (Thanks Eric for the tip)

Disadvantages

• Same as above, will take many years till people can use it.

PG gem can add a Lazy results object.
In this case we simply extend the PG gem API to return a copy of the results provided by libpq that allocates significantly less Ruby objects. Then once we have the copy we can clear the result we get from libpq.

For example:

r = pg.exec('select * from table')
rs = r.result_set
r.clear

# at this point only 2 RVALUEs are allocated. 
# the new ResultSet object has internal c level storage
# pointing at an array of strings, and an API for access where it defer creates
# objects

row = rs[1]

### ResultSetRow is allocated, it also only allocates 1 RVALUE

row["abc"] # allocates a new RVALUE or returns a cached internal instance 
row[1] # same

rs.get(1,100) # same as above

Advantages

• This drops in to ActiveRecord and other ORMs as the best practice for grabbing data if #clear is not guaranteed

• Reasonably efficient, only allocates a very minimal number of Ruby objects

• We can start using this very soon

Disadvantages

• We are forced to make memory copies of results returned via PG, this has a non zero cost (I suspect it is not too high though compared to 1000s of Ruby objects that need to be garbage collected with #values calls)

Build tooling to detect this problem in production apps! It would be amazing if when we saw a Ruby app that is bloated in memory we could run a simple diagnostic on it to figure out where the bloat is coming from.

• Is the bloat there due to glibc arenas?

• Is the bloat there cause Ruby is not aware of a bunch of allocated memory?

• Is the bloat there due to a simple managed leak, eg: an ever growing array?

It is a hard problem to solve though. jemalloc does provide a lot of internal diagnostics, so we could look at the delta between what jemalloc has allocated and what Ruby knows about!

Advantages

• Would increase visibility of this problem and the family of related problems and allow us to alert various gem authors if they are impacted by it.

Disadvantages

• Hard to build and may require a custom startup.

What we are doing?

I have invested many hours investigating these issues. Discourse is actively investing in improving the memory story in Ruby. Together with Shopify and Appfolio we are sponsoring Eric Wong to experiment and improve Ruby for the next few months.

Discourse are also looking to throw more dollars behind a project to heavily improve Active Record for the 6.0 release which I plan to blog about soon. We also plan to extract, improve, formalize and share our built in raw SQL helpers.

I hope you found this helpful and as always, enjoy!

Often in development we have an annoying message in our console that we simply can not find the source of, here is a little trick you can use to hunt messages like this down.

Recently, we have been experiencing “stalls” in the Puma web server in development, this means that quite often during our dev cycle we would hit CTRL-C and be stuck waiting many many seconds for Puma to stop. Sometimes needing to fallback to kill -9 on the Puma process.

We definitely want this Puma issue fixed, however our “web application server of choice” is Unicorn not Puma. It makes little sense for us to run Puma in development. Our Unicorn configuration is very mature and handles all sorts of magic including automatic forking of our Sidekiq job scheduler which is awesome in dev.

A major problem though is that when we run Puma in dev our console is pristine, run Unicorn in dev and it is noise central.

127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/pretty-text-bundle.js?1533620338.6222095 HTTP/1.1" 200 112048 0.0481
127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/plugin.js?1533620338.6222444 HTTP/1.1" 200 146176 0.0726
127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/plugin-third-party.js?1533620338.6222594 HTTP/1.1" 200 3364 0.0569
127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/application.js?1533620338.6222193 HTTP/1.1" 200 3039095 0.2049
127.0.0.1 - - [07/Aug/2018:15:38:59 +1000] "GET /assets/fontawesome-webfont.woff2?http://l.discourse&2&v=4.7.0 HTTP/1.1" 304 - 0.0016

I am a puts debugger and being barred from being a puts debugger in development is a blocking feature for me.

So, how do we find where these messages are coming from?

Before we start the little tip here first… if you have not yet… take a break and read _why’s classic seeing metaclasses clearly.

Now that you know about metaclasses, time to have some fun, let’s reopen STDERR and glue a little debug method to it that will output caller locations when we invoke write on STDERR (note this will work on STDOUT as well if you want):

class << STDERR
  alias_method :orig_write, :write
  def write(x)
    orig_write(caller[0..3].join("\n"))
    orig_write(x)
  end
end

/home/sam/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rack-2.0.5/lib/rack/common_logger.rb:61:in `log'
/home/sam/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rack-2.0.5/lib/rack/common_logger.rb:35:in `block in call'
/home/sam/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rack-2.0.5/lib/rack/body_proxy.rb:23:in `close'
/home/sam/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rack-2.0.5/lib/rack/chunked.rb:34:in `close'
127.0.0.1 - - [07/Aug/2018:15:44:57 +1000] "POST /mini-profiler-resources/results HTTP/1.1" 200 - 0.0109

So, there you have it this line is coming from CommonLogger.

However… Discourse does not use the Rack::CommonLogger middleware… a little bit more hunting we can find out that unicorn will always load Rack::CommonLogger, Rack::ShowExceptions, and Rack::Lint middleware in development and it has a little command line option of -N or --no-default-middleware to disable this behavior.

This tip is handy for a large number of issues you can encounter, be it stray messages in your test suite or leftover puts in some gem you upgraded. And as always, enjoy.

I have always been somewhat fascinated with logs. I tend to see the warning and error logs in production as a valuable heartbeat of an application. Proper handling of error logs is a very strong complement to a robust test suite. It shows us what really happens when real world data meets our application.

9 years ago, at Stack Overflow we had a daily ritual where we would open up our fork of ELMAH every morning and fish through our logs for problems. This had a dramatic positive effect on Stack Overflow.

Almost 7 years into our journey building Discourse, every single week we find and fix issues in our application thanks to our error logs and Logster. Error logs are the pulse of our application, they let us know immediately if there are any urgent issues and where. Since we host more than 1500 sites running many different code branches, we needed to evolve a sane and robust set of practices and tools.

Top level structure of logging and monitoring at Discourse

We have lots of logs at Discourse and many systems for dealing with them.

• We keep raw Docker, Postgres, Redis, NGINX, Rails and HAProxy and so on in Elastic Search and use Kibana for business intelligence.

• We have a monitoring system built on alertmanager and Prometheus, with business intelligence in Grafana and alert escalation in our internal Discourse instance and opsgenie.

• We have logster which we use for web application aka. “Rails / Sidekiq” warnings and errors.

I would like to focus on logster and our Rails / Sidekiq portion for this blog post, but think it is worth mentioning other mechanisms cause I don’t want people to think we are not good data hoarders and only have very limited visibility into our systems.

About Logster

At Discourse we developed a log viewer called logster.

Logster is a free and open source tool you can embed into any Ruby on Rails or Rack application in production and development. It runs as Rack middleware and uses Redis as its backend for log storage and analysis.

It operates in two different modes:

• In production mode it aggregates similar errors by fingerprinting backtraces listening for warnings/errors and fatal messages. The intention is to display a list of open application problems that can somehow be resolved.

• In development mode it provides a full fire-hose of all logs produced by Rails. (debug and up). This has significant advantages over console as you have proper access to backtraces for every log line.

Here are a few screenshots from logs on this very blog (accessible to admins at https://discuss.samsaffron.com/logs):

image.png1930×1418 359 KB

Each error log has a full backtrace

image.png1928×1414 336 KB

Web requests have extensive environment info, including path, ip address and user agent.

Logster has accumulated a large amount of very useful features over the years, including:

• The ability to suppress errors from the logs until the application is upgraded. (The solve button)

• The ability to protect certain log messages so they are not purged when clear all is clicked.

• Advanced filtering, including regex and reverse regex search

• Custom environment (ability to tag current thread with arbitrary metadata)

• JavaScript error and backtrace support

• Rich API allowing you to suppress patterns, ship errors from other instances, integrate automatically into Rails and so on.

The Logter project is still very much alive, recently our part time developer Osama added a mobile view and upgraded the Ember frontend to latest Ember. We have many exciting new features planned for 2019!

Giving up on tail -f logs/development.log

I do not remember the last time I tailed logs in development. There are a few reasons this does not happen anymore.

• Most of the time when building stuff I use TDD, using our rake autospec tool. I will focus on one broken test. Every time I save a file it automatically triggers the test to re-run, if I need extra diagnostics I sprinkle puts statements.

• If I am dealing with a specific error on a page I often find working with better_errors far more effective than reading logs.

• If I need access to logs I will always prefer using logster in development. It allows me to filter using a text pattern or log level which is a huge time saver. It also provides information that is completely absent from the Rails logs on a per-line basis (environment and backtrace).

image.png1938×1120 303 KB

I sprinkled Rails.logger.warn("someone called featured users, I wonder who?") and filtered on “featured”

Death by 10,000 log messages in production

Logster attempts to provide some shielding against log floods by grouping based off stack traces. That said, we must be very diligent to keep our logs “under control”.

For the purpose of our Logster application logs usage we like to keep the screens focused on “actionable” errors and warnings. Many errors and warnings that get logged by default have no action we can take to resolve. We can deal with these elsewhere (offending IPs can be blocked after N requests and so on).

Here are a non exhaustive example of some “errors” that we really have no way of dealing with so they do not belong in Logster.

• A rogue IP making a web request with corrupt parameter encoding

• A 404 to index.php which we really do not care about

• Rate limiting … for example a user posting too fast or liking too fast

• Rogue users making a requests with an unknown HTTP verbs

Another interesting point about our use of Logster is that not all errors that float into our logs mean that we have a broken line of code in our application that needs fixing. In some cases a backup redis or db server can be broken so we will log that fact. In some cases there is data corruption that the application can pick up and log. Sometimes transactions can deadlock.

Keeping our Logster logs useful is extremely important. If we ignore in-actionable errors for long enough we can end up with a useless error log where all we have is noise.

Proactively logging issues

Given we have a high visibility place to look at errors. We will sometimes use our error logs to proactively report problems before a disaster hits.

In this case we are watching our “defer” queue, which is a special thread we have for light-weight jobs that run between requests on our web workers in a background thread. We need this queue to be serviced quickly if it is taking longer than 30 seconds per job we have a problem… but not necessarily a disaster. By reporting about this early we can correct issues in the job queue early, rather than dealing with the much more complex task of debugging “queue starvation” way down the line. (which we also monitor for)

The logs hot potato game

Half a year ago or so we introduced a fantastic game within our development team. The idea is very simple. Every developer attempts to correct an issue raised in our error logs and then assigns to the next person on the list.

We attempted many other patterns in the past, including:

• Having our internal Discourse instance raise a big warning when too many errors are in the logs (which we still use)

• Having “log parties” where a single team member triages the logs and assigns issues from the logs to other team members.

• Having arbitrary triage and assign.

The “logs game” has proven the most effective at resolving a significant number of issues while keeping the entire team engaged.

We structure the game by having a dedicated Discourse topic in our internal instance with a list of names.

When we resolve issues based on log messages we share the resolution with the team. That way as the game progresses more people learn how to play it and more people learn about our application.

Once resolved, the team member hands the torch to the next person on the list. And so it goes.

This helps all of us get a holistic picture of our system, if logs are complaining that our backup redis instance can not be contacted, this may be a provisioning bug that needed fixing. For the purpose of the “logs game” fixing system issues is also completely legitimate, even though no line of code was committed to Discourse to fix it.

Should my Ruby web app be using Logster?

There are many other products for dealing with errors in production. When we started at Discourse we used errbit these days you have many other options such as sentry, airbrake or raygun.

One big advantage Logster has is that it can be embedded so you get to use the same tool in development and production with a very simple setup. Once you add it to your Gemfile you are seconds away from accessing logs at /logs.

On the other hand the for-pay dedicated tools out there have full time development teams building them with 100s of amazing features.

Logster is designed so it can work side-by-side with other tools, if you find you need other features you could always add an additional error reporter (or submit a PR to Logster).

Regardless of what you end up choosing, I recommend you choose something, there is enormous value in regular audits of errors and better visibility of real world problems your customers are facing.

I made this tweet that got reasonably popular:

We benchmarked how long it takes to run the Ruby test suite for Discourse across our various dev machines. I can not believe what a crazy tax I have paid over the years insisting on sticking with Windows, highlighted results mine.

image.png824×706 73.7 KB

https://twitter.com/samsaffron/status/1111511735851081728

This evoked a lot of extremely strong emotions from various people out there. Ranging from “Sam is a fool what kind of insane benchmark is this”, “the real story is MacOS has bad Ruby perf” to a general “Oh no”.

The core point I was trying to make was that I was paying a pretty high tax for deciding to “stick with with Windows”. There are a bunch of other points hiding here that are also worth discussing.

Why are you running sticking with Windows to run Linux in a VM?

What I did not know is the extent of the VM tax I was paying regularly. I never dual booted my computer so I had no proper anchoring point of reference.

I very strongly believe that many Ruby/Rust/Go/Elixir/Scala and even some Node developers who end up doing the WSL dance or run Linux in a VM for development, or use Linux Docker for dev on Windows are not aware of the full extent of the tax.

On my machine the price of admission for using WSL was 25% slowdown in my day to day running of specs. And a 38% slowdown for using a VMware based VM.

I am not alone here… other team members have experienced similar slowdowns. Other people out there also experience similar slowdowns.

What I thought was inside my wonderful wish hat was that the performance hit was minor:

Yes. But that is not the question. The difference is normally negligible (1% to 5%).

If you Google, well that is the general answer you get. VMs are more or less the same as no VM, 5-10% perf tax. My reality was different. Maybe I was missing magic bios turbo settings, maybe I needed to direct mount a volume instead of using a vmdk on my dedicated NVMe Samsung 960 pro, maybe there is some magic I could do to get to this magic 1-5% number. Maybe Hyper-V is better I am not sure. All I know is that I am not alone here.

WSL is not an option for me cause Ruby likes lots of small files and lots of stats calls, WSL has terrible lots of small file performance as documented by the WSL team. How terrible you ask? As a baseline just running a rails c console without bootsnap was taking me upwards of 30 seconds. Same operations takes 4-5 seconds on Linux without bootsnap. Even with all the workarounds we could place this bad IO perf was something that I just noticed too much. In fact I preferred the 38% slowdown cause at least stuff was consistent and not wildly off balance like WSL is. Being able to launch a console or web server quickly is critical during dev. Fuse does not appear to be happening any time soon so you can not work around this using ninja tricks of block mounting a device.

So, I stuck with a VM cause it was nice not to have to constantly reboot my computer and thought the price I was paying was not that high.

I like the Windows 10 font rendering, I like the HiDPI support, I like using Lightroom on Windows and playing Rocksmith on Windows. I like the out-of-the-box experience and minimal amount of tweaking needed. I like being able to launch Skype without it segfaulting cause I was LD_PRELOADing jemalloc. I feel Windows 10 as a window manager is on par (for my usage) to my Macbook Pro running MacOS.

Dual booting is a compromise for me, some stuff I have works best in Windows. I thought the compromise I was making performance wise was worth the comfort of living in a “known OS” that I like.

I felt that if I start booting to Linux I am going to have to fight with drivers, have stability issues, not have a complete toolset and so on.

I felt comfortable at home and moving is one of the most stressful life events.

Is 2019 the year of Linux on the Desktop?

The joke goes like this. Every year a bunch of people joke about how LOL this will be the year of Linux on the Desktop. It happens every year. It starts cause someone says “hi Linux is quite good these days, could this be the year of Linux on the Desktop?”. And then a bunch of happy and well meaning trolls, say ha ha … as always you are wrong… this is not the year of Linux on the Desktop.

And so it goes…

This banter is usually well meaning and grounded in reality. However it has a very big side effect, which impacts developers in a significant manner. Developers who do not use Linux on the desktop are scared of Linux. They are scared even if their production code only deploys on Linux (and not MacOS or Windows)

I felt super scared to go down the path of Linux cause I was terrified … about drivers … font rendering… HiDPI support… multi monitor support and the list goes on.

In fact, I was not wrong to be scared. It is fiddly to get Linux going. I almost gave up after my first 4-8 hours cause Firefox on Linux is still stuck on a very sad default there is no hardware acceleration out of the box, so scrolling is mega jerky. This very simply rectifiable behavior was a deal breaker for me. If I could not get scrolling a web page to be smooth, I am out of here, not going to use Linux. Luckily the issue was resolved after tweaking 1 value in about:config.

NVIDIA does not have a great story as well, the future of Desktop on Linux is Wayland. The windows manager I wanted to try, sway, only works properly if you use the open source community provided nouveau driver. Even getting NVIDIA to work nicely involves enabling hardware compositing and fiddling with X11 config.

My position is not that Linux is poised to take over the world in a storm this year. It is a far more humble position. If you want to get the best bang for buck and want to get the best possible performance developing Discourse, or any Ruby on Rails application Linux on the Desktop/Laptop with no VM is your best bet.

It is also important to note that I opted for medium hard mode when I moved to Linux. I am only 2 neck beards away from installing Linux from scratch.

source

My colleagues who went through similar exercises of shifting from Windows/Mac to Linux stuck with Ubuntu and Linux Mint, they tell me they had a very smooth ride.

Have you tried running Ruby on Windows?

Avdi triggered quite a discussion about this a few days ago:

The point he is trying to make is that a Ruby that works well on native Windows will help Ruby adoption a lot and eliminate drain to other frameworks. Installing a whole brand new OS is just too much of a barrier. Just install Linux is not a solution.

The reality is that running MRI Ruby native on Windows hits 2 big fundamental problems:

1. Filesystem performance characteristics of NTFS on Windows are badly suited to the current Ruby design. We love lots of small files, we love lots of stats calls.

2. It is a gigantic effort porting various dependencies to Windows native (and maintaining them), as it stands many of the Discourse dependencies simply do not work on Windows. The gems simply will not install. The fundamental issue is that if you are writing a c extension in a gem it is extra work to get it to work on Windows. Getting stuff to work on MacOS and Linux is no extra work vast majority of the time.

(2) is a tractable problem, but I wonder if it is worth any kind of effort given WSL has far wider compatibility and should offer reasonable performance once a workaround exists for the filesystem problem (which is fundamental and not going to change on Windows native). Discourse works fine on WSL (provided you skip using unicorn) Discourse does not work at all on Ruby on Windows native. The Apple tax is similar in cost to the Windows WSL tax (except for filesystem perf). I feel that once WSL gets a bit more polish and fixes it will be competitive with the current Mac experience.

The Apple performance tax

One pretty obvious thing from the chart I provided was showing there is a pretty severe Apple performance tax as well.

When looking at user benchmarks per: UserBenchmark: Intel Core i7-8559U vs i7-8750H. We expect an 8559U to have faster single core performance (thermal locking withstanding) than the 8750H. Yet this Linux 8750H laptop is clocking a spectacular 9m13s compared to the Macbook Pro 15m16s. We are seeing poor MacOS performance across the board. And we are not alone:

It appears that people insisting on the native MacOS experience are paying a significant tax for developing Ruby on Rails on a Mac.

I know that DHH loves his iMac Pro and recommends it enormously.

Yes, the hardware is real nice, the screen is beautiful, the machine is wonderfully put together. The Window manager is nice. Zero driver problems. However, sadly, there is a significant OS tax being paid sticking on MacOS for Ruby on Rails development.

I think the Ruby community should explore this problem, document the extent of this problem and see if anything can be done to bring Darwin closer to the numbers the same machine does with Linux. Is this problem rooted in the filesystem? The OS? The llvm compile of Ruby? Security features in MacOS? Something about how Spectre+Meltdown (which is already patched in my Linux)? It is very unclear.

As it stands I would not be surprised at all if you dual booted a Mac with Windows, installed WSL and got better performance running the Discourse test suite on Mac+Windows+WSL. In fact I am willing to take bets you would.

So, to all those people who say… oh there is an alternative … just hackintosh your way out of this mess. Not only are you stuck playing Russian roulette every MacOS update, you are also paying a tax which is similar to the tax you are paying on Windows already.

What about parallel testing?

Rails 6 is just around the corner. This is the first time Rails is going to ship with officially supported and sanctioned parallel testing. When I run the Discourse spec suite on my Linux system CPU barely scratches the 10% mark for the whole 8 minutes the test suite is running, IO is not saturated.

Here I am freaking out about a measly 38% perf hit when I could be running stuff concurrently and probably be able to run our entire test suite in 2 minutes on my current machine on Windows in a VM.

It may feel a bit odd to be making such a big deal prior to taking care of the obvious elephant in the room.

I completely agree, parallel testing is an amazing thing for Rails, this is going to make a lot of developers extremely happy.

Also, profiling our test suite, eliminating and improving slow tests is super important.

We are going to adopt parallel testing for our dev environments this year.

But I guess this was not my point here. My issue is that we I was driving with the hand break on. Even when our test suite gets faster, the hand break will remain on.

Where am I headed?

I am feeling significantly happier in my Arch Linux home. In a pretty surprising turn of events not only is stuff much faster for me, I also feel significantly more productive at work due to having a windows manager that works much better for me than my Mac or Windows setups ever did. Yes there are compromises, I need to get my hands far dirtier than I had to in the past. However the payoff has been huge.

I have been a long time I3wm user, however I never got the proper experience being straddled in the middle of 2 windows managers. Now that i3 is my only windows manager I am unlocking tremendous amount of value out of it.

Why, you ask? Well I plan to write a bit about my experience over the next few weeks. My plan is to try out a different tiling windows manager every month for the next few months to find the flavor that fits me best.

I stuck with Windows for 6 years developing an application that works best on Linux because I was comfortable in Windows. Habits are incredibly hard to break. I was not fully aware what price I was paying. I can also assure you many other developers are in the same boat as I was.

If I have one piece of advice here, it is … don’t be afraid to experiment. Linux on the desktop is getting better, it is reasonably straight forward to re-partition a drive and setup a dual booting system. If you are in the same boat as I was, living between 2 worlds, especially if you are on a desktop and not a laptop, take a break and experiment.

Please feel free to post any of your experiences or benchmarks here, I will try to answer every post on my blog carefully. I am curious to see more benchmarks from more people comparing MacOS to Linux on the same computer or Windows+WSL / VM and Linux.

And as always … enjoy.

I have been a long time i3 window manager user. But not really.

My old Windows 10 based setup involved doing all my console work in an Ubuntu VM running i3. However, the lion’s share of the non console work was still done in Windows, including browsing and more.

For multiple years now I only partially experienced i3, it showed. My i3 setup was almost vanilla.

My move to Arch Linux changed everything.

This move completely shifted the way I think about my relationship with my desktop environment. Previously, my relationship with Windows was very simplistic. Windows works the way it works, I simply adapted to that. Sometimes I learned a new shortcut, but the majority of my Windows day-to-day involved dragging windows around, reaching Firefox window and tab saturation, closing windows with the mouse and so on.

I am not a great example of a Windows ninja some users go down a far more custom path. I do feel I am pretty typical though of a developer using Windows or Mac. I was given a menu, I learned a tiny bit of it, then I simply threw away the menu and reached for the mouse.

In this blog post I would like to talk about what my 3.5 week adventure has looked like and where I am today!

Opening moves

When I moved to Linux I did not know much of the current state of Linux on the desktop but I did know 2 things:

1. I would be using Arch Linux
2. I would be using the i3 tiling window manager

I opted for Arch cause I love not having to worry about upgrading my system every 6-12 months to another major release, I think pacman and the package library on Arch is amazing, if I ever am missing tiny bits from the official library it is trivial for me to just grab a package from the very comprehensive AUR. I also think the documentation in the Arch wiki is fantastic and it helped me enormously.

I opted for i3 cause I wanted to fully experience the window manager, not treat it as a glorified tmux like I was for years.

A day or so into my move I was uncomfortable with the way my stock install looked and acted, I quickly learned about the r/unixporn reddit and this movement called “Ricing”.

During the first few days I watched a fair bit of youtube to see what others are doing.

I can recommend:

• The 3 part series by code cast on i3. Especially the last one.
• Ethan Schoonover excellent xmonad demo - If you have not seen this yet I recommend you stop reading and go watch it.
• Luke Smith has interesting videos about his Linux usage including this one on his i3 usage
• Aline Abler course on tiling window managers

My basic ricing

I totally get that lots of people love dmenu people get it to do all sorts of amazing things like mount drives, select monitors and pick files. It is a very powerful and in true suckless fashion minimal tool.

I opted to swap out my dmenu with rofi which I seem to like a bit more. It looks like this:

image.png1919×407 31.4 KB

I prefer the positioning and really like the combi menu that allows me to also navigate through my open windows. rofi works in a dmenu mode as well so I can just use it interchangeably.

I also used LXApperance for some very rudimentary themeing in particular I do like the Apple San Fransico font that I use for my window titles:

I also set up a basic gruvbox theme for my urxvt terminal and was careful to grab the fork with 24 bit color support so everything looks just right. Initially I tried out terminator but find urxvt a bit “lighter” that said, I may try out st next.

image.png1273×659 75.8 KB

Finally I swapped out i3status with i3status-rust. It shows me weather, volume, network and cpu speed and pending update count. I really enjoy it.

image.png1959×50 13.5 KB

My ricing is very basic, I don’t like wallpapers, I don’t like transparency and am unsure if I would even like to try gaps or not.

A tiny note on mod keys

A large amount of i3 configuration relies on using a mod key. The mod key is mapped by end users to an appropriate key that does not get in the way with other keyboard bindings other programs use.

In my case I map mod to both the Windows key and the right Menu key. I do the menu key mapping by running exec_always --no-startup-id xmodmap -e "keysym Menu = Super_R" in my config file.

The tool I used for displaying keys on this blog post (the amazing screenkey) calls the Windows key Super which is the Linuxey name. I can rename it to mod, but I am already multiple screenshots in.

For the purpose of this blog post. Mod == Super == Windows Keyboard key. I will be calling this key Super from here downwards.

Easy editing of Discourse

When I moved to i3 proper I set myself the goal to eliminate trivialities. I observed things that I kept on doing inefficiently and optimized my setup.

I found that in the past every time I wanted to hack on Discourse I would

• Open a terminal
• cd Source/discourse
• open nvim
• split the window
• open nerdtree

This flow involved lots of steps which can easily be automated:

image.png3840×2160 79.9 KB

I now hit Super + Shift + D and tada Discourse opens.

This is done by adding this to my i3 config:

bindsym $mod+Shift+d exec "i3-sensible-terminal -e '/home/sam/.i3/edit_discourse'"

And this tiny shell script

sam@arch .i3 % cat edit_discourse 
#!/bin/zsh

cd /home/sam/Source/discourse
exec nvim -c ':NERDTree|:wincmd w|:vsplit'

Smart window centering

Even though i3 is a “tiled” window manager. Some windows… I prefer in floating mode. In particular I like having Firefox in floating mode.

I like having Firefox in the middle of my center monitor at very particular dimensions. I do sometimes drag it around but it is nice to “reset” the position.

image.png3840×2160 34.8 KB

image.png3840×2160 45.5 KB

Sometimes I like it a bit wider, so I hit Super + c again.

image.png3840×2160 44.8 KB

And sometimes I like it a tiny bit wider, so I hit Super+c again.

image.png3840×2160 45 KB

If I hit Super + c yet again it is back to square one and window is small centered.

I achieve this by having this in my i3 file.

bindsym $mod+c exec "/home/sam/.i3/i3-plus smart_center 1830x2100,2030x2100,2230x2100"

The little i3-plus utility is a work-in-progress Ruby utility I have that interacts with the i3 IPC so it can make smart decisions about what to do. You can find the source for it in my dotfiles.

The basic logic being:

This config also allows me to quickly zoom a tiled panel to the middle of the screen, size it right and once I am done with focused work I can ship it back to the tile with Super+Shift+Enter

Easy terminal arrangement

One issue I had with i3 for quite a while was needing to remember to flip the way I split windows in tiling mode. I would hit Super+Enter to open a terminal, then hit it again and open a terminal to the right.

And then I hit a problem.

My brain simply did not consistently remember if I had to hit Super+v for a vertical split or Super + h for a horizontal split. Is splitting vertically splitting the vertical window in half or is splitting horizontally splitting tall window at the horizon.

Clearly, I could work around my brain glitch by using a different shortcut that was easier for me to associate. Or just tough it up and train myself properly. But what I observed here is that I was just repeating a pointless task.

I like my tiled windows arranged “just so” and in a specific order. i3 by design is not a “just so” tiler, all tiling is manual not automatic like xmonad and dwm. This is an explicit design goal of the project.

Michael Stapelberg explains:

Actually, now that I think of it, what you describe is how automatic tiling WMs work (like DWM, awesome, etc.). If you really need that, you might be better off using one of these. i3 is (and will stay) a manual tiling WM.

That said… this is my Window Manager, and I can make it do what I want. Unlike my life in Windows and Mac, when I dislike a behavior I can amend it. I am encouraged to amend it. i3 will not merge in dynamic tiling which is a way they manage bloat and complexity, but I can have a bodged up dynamic tiling system that works for my workflows with i3.

So, I have this standard behavior:

image.png3840×2160 30 KB

Followed by this … non standard behavior. (notice how I never had to hit Super+v

image.png3840×2160 34.9 KB

What more it gets better cause then next Super+enter switches panels, no matter what terminal I am on.

image.png3840×2160 36.6 KB

My system is somewhat glitchy, I have only been doing this for a few weeks, but it scratches my itch big time.

As an added bonus I made it so when I am on my right most monitor I start tiling vertically in the left column instead of right.

My work in progress code to make this happen is at my i3-plus file in my dotfiles.

At the moment layout is hardcoded and I simply run:

bindsym $mod+Return exec /home/sam/.i3/i3-plus layout_exec i3-sensible-terminal

Tweaks to multi monitor focus

I tend to keep a floating window around on my left monitor for chat. I found that I tended to get trapped on my left monitor after hitting Super + Left. i3 has a behavior where it cycles between floating windows on a monitor. This got in the way of my workflows.

After raising raising this at GitHub airblader fairly concluded that this is a minor annoyance with a clear workaround but was worried about adding any more complexity to focus behavior. This is fair.

But… this is my Window Manager and I get to call the shots on my computer.

So now my focus never gets trapped on a monitor. My Super + Right key works the way I want it to.

Tweaks to move

Out-of-the-box i3s example file binds Super + Shift + Right/Left to the i3 command move.

What this does is:

• In tiled mode moves the tile to left or right
• In floating mode moves the window a few pixels to the left or right.

The behavior in tiled mode worked for me, but I found that I am not really into positioning floating windows using arrows and instead find it far more useful to “throw” a floated window to the right or left monitor.

From what I can tell (and I may be wrong) I could not find a way to tell i3 to run a certain command in floating mode and another in tiled mode. However using the ipc interface this was trivial:

  def move(dir)
    if is_floating?
      @i3.command("mark _last")
      @i3.command("move to output #{dir}")
      @i3.command('[con_mark="_last"] focus')
    else
      @i3.command("move #{dir}")
    end
  end

A keyboard friendly exit

The i3 sample config spins up a nagbar prior to attempting to exit the windows manager. I found the position of this nagbar not ideal and did not like that I needed to reach for the mouse. I am not alone here but this is really only a common problem when you are heavily tweaking stuff.

That said I came across this wonderful idea somewhere, which I would love to share:

mode "exit: [l]ogout, [r]eboot, [s]hutdown" {
  bindsym l exec i3-msg exit
  bindsym r exec systemctl reboot
  bindsym s exec systemctl shutdown
  bindsym Escape mode "default"
  bindsym Return mode "default"
}

bindsym $mod+x mode "exit: [l]ogout, [r]eboot, [s]hutdown"

I now use Super + x to enter my “exit i3 mode”, which gives me all the goodies I need with a nice UX.

I love screenshots

During my day I tend to take a lot of screenshots. I always struggled with this for a degree. I never had the “right” tool for the job in my Windows days. Now I do.

I set up my screenshot hotkeys as:

1. PrtScn : take a screenshot of a selection

2. Super + PrtScn : take a 3 second delayed screenshot of a selection

3. Super + Shift + PrtScn: take a screenshot of the current desktop + pass it through pngquant and add to clipboard.

(1) in the list here was really easy. I used the flameshot tool and simply bound prtscn to it:

exec --no-startup-id flameshot
bindsym Print exec "flameshot gui"

It works a treat. Highly recommend.

Delayed screenshots (2) is where stuff got tricky.

Flameshot has a delay option, even if it did not it is trivial to exec sleep 2 && flameshot gui. However, I like having a visible reminder on the screen that this thing will happen:

To implement this I adapted the countdown script from Jacob Vlijm

My adaptation is here..

In i3 I have:

bindsym $mod+Print exec "/bin/bash -c '/home/sam/.i3/countdown 3 && sleep 0.2 && flameshot gui'"

Full screen screenshots (3), like the ones further up this blog post was a bit tricky.

Xwindows screenshot tools like treating all my 3 4k screens as one big panel, not too many tools out there can figure out current focused monitor let alone split up the enormous image.

To achieve this I rolled my own script that uses the i3 IPC to figure out what display has focus and then tells ImageMagick to capture and crop correctly and finally passes throw pngquant and back on to the clipboard in a web past friendly format using CopyQ.

This simple binding then takes care of it for me.

bindsym $mod+Shift+Print exec "/home/sam/.i3/i3-plus screenshot"

Scratchpad

i3 has a special desktop that is not visible called “the scratchpad”. If you want to get rid of a window temporarily you can always just ship it there and recall it from there. I like to use it for a couple of things.

1. I bind Super + b to toggle my browser in and out of the scratchpad. No matter which monitor I am on I can summon my browser with this hotkey (and make it go away)

2. I bind Super + p to toggle a dedicated terminal. I like to use this dedicated terminal to run stuff like pacman -Syu that can take a bit, look at a calendar, run a quick calculation and so on.

3. Similar to both above I like Super + y to bring up my yubico authenticator. (I highly recommend a Yubikey for devs it is a big time saver)

# terminal that pops up on demand
exec urxvt -name scratch-term
for_window [instance="scratch-term"] floating enable, move to scratchpad
bindsym $mod+p [instance="scratch-term"] scratchpad show

exec firefox
for_window [class="Firefox"] floating enable, move to scratchpad, scratchpad show
bindsym $mod+b [class="Firefox"] scratchpad show

exec yubioath-desktop
for_window [class="Yubico Authenticator"] floating enable, move to scratchpad
bindsym $mod+y [class="Yubico Authenticator"] scratchpad show

Other bits and pieces

My current .xinitrc looks like this:

eval $(dbus-launch -sh-syntax --exit-with-session)
dbus-update-activation-environment --systemd DISPLAY

xrdb -merge ~/.Xresources

xrandr --output DVI-D-0 --off --output HDMI-1 --off --output HDMI-0 --mode 3840x2160 --pos 0x0 --rotate normal --output DP-3 --off --output DP-2 --primary --mode 3840x2160 --pos 3840x0 --rotate normal --output DP-1 --off --output DP-0 --mode 3840x2160 --pos 7680x0 --rotate normal

eval $(/usr/bin/gnome-keyring-daemon --start --components=gpg,pkcs11,secrets,ssh)
export GNOME_KEYRING_CONTROL GNOME_KEYRING_PID GPG_AGENT_INFO SSH_AUTH_SOCK

exec i3

I am not a fan of using Gnome Display Manager as I feel it introduces more complexity into my setup. Instead, I just run startx after logging in.

The two trips here is that I needed a dbus session so gnome type apps work (like skype for example) and needed to spin up my keyring (which skype needed as well)

Do I actually get any work done?

The i3 sample config file has a wonderful comment at the top.

# This file has been auto-generated by i3-config-wizard(1).
# It will not be overwritten, so edit it as you like.

My i3 setup is my setup. It is tailored for my use cases.

I love that i3 has a single config file, it is very easy to reason about my current desktop environment. If I don’t ever use a shortcut I can remove it. If I need a new shortcut I can add it. If I forget what is “on the menu” I can read the reasonably small file to figure out!

All of this tweaking does sound like it could be a full time job for multiple weeks but it really was not at all. I hit barriers in my workflow, unblocked them and then moved on. Each barrier I removed made me more efficient.

The end result has been that I can now jump on a problem and solve it with significantly more focus. My window manager is working for me, I am no longer its slave.

In my previous blog post I talked about leaving Windows. The catalyst was performance. What I did not know was what a wonderful experience I would have in my new Linux home.

Sure, I have the usual niggles of needing to run a compositor fight with Nvidia drivers and deal with finding Linux alternatives for Windows tools I was used to using. However, on a fundamental level I am just so much happier now. I feel like I relinquished control over my computer for too long.

What you can do?

If you wish to do a Linux experiment you can choose the hard mode or the easy mode, there are plenty of alternatives out there. If you want to try out tiling, you can even just pick up a full pre-configured stack from Luke Smith. You can start from blank and amend the defaults to suite.

As a programmer in any terminal dominated technology stack (like Ruby/Rust/Golang and so on) I strongly recommend trying out a tiling window manager.

From all my research i3 is the perfect first choice for trying out a tiling window manager, it comes with very sane and complete defaults, the config file is very easy to reason about and it works great!

If you have any questions or would like any tips feel free to post here and I will reply, but be warned, I am no expert I am just learning.

Big thank you to Michael Stapelberg for creating i3, and the very active community (Airblader, Orestis and others) for maintaining i3. Big thank you to all you people putting excellent content out there and making my ride into the Linux world easy.

wolf.jpg695×564 286 KB

A liar will not be believed, even when he speaks the truth. : Aesop

Once you have a project that is a few years old with a large test suite an ugly pattern emerges.

Some tests that used to always work, start “sometimes” working. This starts slowly, “oh that test, yeah it sometimes fails, kick the build off again”. If left unmitigated it can very quickly snowball and paralyze an entire test suite.

Most developers know about this problem and call these tests “non deterministic tests”, “flaky tests”,“random tests”, “erratic tests”, “brittle tests”, “flickering tests” or even “heisentests”.

Naming is hard, it seems that this toxic pattern does not have a well established unique and standard name. Over the years at Discourse we have called this many things, for the purpose of this article I will call them flaky tests, it seems to be the most commonly adopted name.

Much has been written about why flaky tests are a problem.

Martin Fowler back in 2011 wrote:

Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite.

To this I would like to add that flaky tests are an incredible cost to businesses. They are very expensive to repair often requiring hours or even days to debug and they jam the continuous deployment pipeline making shipping features slower.

I would like to disagree a bit with Martin. Sometimes I find flaky tests are useful at finding underlying flaws in our application. In some cases when fixing a flaky test, the fix is in the app, not in the test.

In this article I would like to talk about patterns we observed at Discourse and mitigation strategies we have adopted.

Patterns that have emerged at Discourse

A few months back we introduced a game.

We created a topic on our development Discourse instance. Each time the test suite failed due to a flaky test we would assign the topic to the developer who originally wrote the test. Once fixed the developer who sorted it out would post a quick post morterm.

image.png1245×1779 203 KB

This helped us learn about approaches we can take to fix flaky tests and raised visibility of the problem. It was a very important first step.

Following that I started cataloging the flaky tests we found with the fixes at: https://review.discourse.org/tags/heisentest

Recently, we built a system that continuously re-runs our test suite on an instance at digital ocean and flags any flaky tests (which we temporarily disable).

Quite a few interesting patterns leading to flaky tests have emerged which are worth sharing.

Hard coded ids

Sometimes to save doing work in tests we like pretending.

user.avatar_id = 1
user.save!

# then amend the avatar
user.upload_custom_avatar!

# this is a mistake, upload #1 never existed, so for all we know
# the legitimate brand new avatar we created has id of 1. 
assert(user.avatar_id != 1)  

This is more or less this example here.

Postgres often uses sequences to decide on the id new records will get. They start at one and keep increasing.

Most test frameworks like to rollback a database transaction after test runs, however the rollback does not roll back sequences.

ActiveRecord::.transaction do
   puts User.create!.id
   # 1
   raise ActiveRecord::Rollback
puts 

puts User.create!.id
# 2

This has caused us a fair amount of flaky tests.

In an ideal world the “starting state” should be pristine and 100% predictable. However this feature of Postgres and many other DBs means we need to account for slightly different starting conditions.

This is the reason you will almost never see a test like this when the DB is involved:

t = Topic.create!
assert(t.id == 1)

Another great, simple example is here.

Random data

Occasionally flaky tests can highlight legitimate application flaws. An example of such a test is here.

data = SecureRandom.hex
explode if data[0] == "0"

Of course nobody would ever write such code. However, in some rare cases the bug itself may be deep in the application code, in an odd conditional.

If the test suite is generating random data it may expose such flaws.

Making bad assumptions about DB ordering

create table test(a int)
insert test values(1)
insert test values(2)

I have seen many times over the years cases where developers (including myself) incorrectly assumed that if you select the first row from the example above you are guaranteed to get 1.

select a from test limit 1

The output of the SQL above can be 1 or it can be 2 depending on a bunch of factors. If one would like guaranteed ordering then use:

select a from test order by a limit 1

This problem assumption can sometimes cause flaky tests, in some cases the tests themselves can be “good” but the underlying code works by fluke most of the time.

An example of this is here another one is here.

A wonderful way of illustrating this is:

[8] pry(main)> User.order('id desc').find_by(name: 'sam').id
  User Load (7.6ms)  SELECT  "users".* FROM "users" WHERE "users"."name" = 'sam' ORDER BY id desc LIMIT 1
=> 25527
[9] pry(main)> User.order('id').find_by(name: 'sam').id
  User Load (1.0ms)  SELECT  "users".* FROM "users" WHERE "users"."name" = 'sam' ORDER BY id LIMIT 1
=> 2498
[10] pry(main)> User.find_by(name: 'sam').id
  User Load (0.6ms)  SELECT  "users".* FROM "users" WHERE "users"."name" = 'sam' LIMIT 1
=> 9931

Even if the clustered index primary key is on id you are not guaranteed to retrieve stuff in id order unless you explicitly order.

Incorrect assumptions about time

My test suite is not flaky, excepts from 11AM UTC till 1PM UTC.

A very interesting thing used to happen with some very specific tests we had.

If I ever checked in code around 9:50am, the test suite would sometimes fail. The problem was that 10am in Sydney is 12am in UTC time (daylight savings depending). That is exactly the time that the clock shifted in some reports causing some data to be in the “today” bucket and other data in the “yesterday” bucket.

This meant that if we chucked data into the database and asked the reports to “bucket” it the test would return incorrect numbers at very specific times during the day. This is incredibly frustrating and not particularly fair on Australia that have to bear the brunt.

An example is here (though the same code went through multiple iterations previously to battle this).

The general solution we have for the majority of these issues is simply to play pretend with time. Test pretends it is 1PM UTC in 2018, then does something, winds clock forward a bit and so on. We use our freeze time helper in Ruby and Sinon.JS in JavaScript. Many other solutions exist including timecop, the fascinating libfaketime and many more.

Other examples I have seen are cases where sleep is involved:

sleep 0.001
assert(elapsed < 1) 

It may seem obvious that that I slept for 1 millisecond, clearly less than 1 second passed. But this obvious assumption can be incorrect sometimes. Machines can be under extreme load causing CPU scheduling holdups.

Another time related issue we have experienced is insufficient timeouts, this has plagued our JS test suite. Many integration tests we have rely on sequences of events; click button, then check for element on screen. As a safeguard we like introducing some sort of timeout so the JS test suite does not hang forever waiting for an element to get rendered in case of bugs. Getting the actual timeout duration right is tricky. On a super taxed AWS instance that Travis CI provides much longer timeouts are needed. This issue sometimes is intertwined with other factors, a resource leak may cause JS tests to slowly require longer and longer time.

Leaky global state

For tests to work consistently they often rely on pristine initial state.

If a test amends global variables and does not reset back to the original state it can cause flakiness.

An example of such a spec is here.

class Frog
   cattr_accessor :total_jumps
   attr_accessor :jumps

   def jump
     Frog.total_jumps = (Frog.total_jumps || 0) + 1
     self.jumps = (self.jumps || 0) + 1
   end
end

# works fine as long as this is the first test
def test_global_tracking
   assert(Frog.total_jumps.nil?)
end

def test_jumpy
   frog = Frog.new
   frog.jump
   assert(frog.jumps == 1)
end 

Run test_jumpy first and then test_global_tracking fails. Other way around works.

We tend to hit these types of failures due to distributed caching we use and various other global registries that the tests interact with. It is a balancing act cause on one hand we want our application to be fast so we cache a lot of state and on the other hand we don’t want an unstable test suite or a test suite unable to catch regressions.

To mitigate we always run our test suite in random order (which makes it easy to pick up order dependent tests). We have lots of common clean up code to avoid the situations developers hit most frequently. There is a balancing act, our clean up routines can not become so extensive that they cause major slowdown to our test suite.

Bad assumptions about the environment

It is quite unlikely you would have a test like this in your test suite.

def test_disk_space
   assert(free_space_on('/') > 1.gigabyte)
end

That said, hidden more deeply in your code you could have routines that behaves slightly differently depending on specific machine state.

A specific example we had is here.

We had a test that was checking the internal implementation of our process for downloading images from a remote source. However, we had a safeguard in place that ensured this only happened if there was ample free space on the machine. Not allowing for this in the test meant that if you ran our test suite on a machine strained for disk space tests would start failing.

We have various safeguards in our code that could depend on environment and need to make sure we account for them when writing tests.

Concurrency

Discourse contains a few subsystems that depend on threading. The MessageBus that powers live updates on the site, cache synchronization and more uses a background thread to listen on a Redis channel. Our short lived “defer” queue powers extremely short lived non-critical tasks that can run between requests and hijacked controller actions that tend to wait long times on IO (a single unicorn worker can sometimes serve 10s or even 100s of web requests in our setup). Our background scheduler handles recurring jobs.

An example would be here.

Overall, this category is often extremely difficult to debug. In some cases we simply disable components in test mode to ensure consistency, the defer queue runs inline. We also evict threaded component out of our big monolith. I find it significantly simpler to work through and repair a concurrent test suite for a gem that takes 5 seconds to run vs repairing a sub-section in a giant monolith that has a significantly longer run time.

Other tricks I have used is simulating an event loop, pulsing it in tests simulating multiple threads in a single thread. Joining threads that do work and waiting for them to terminate and lots of puts debugging.

Resource leaks

Our JavaScript test suite integration tests have been amongst the most difficult tests to stabilise. They cover large amounts of code in the application and require Chrome web driver to run. If you forget to properly clean up a few event handlers, over thousands of tests this can lead to leaks that make fast tests gradually become very slow or even break inconsistently.

To work through these issues we look at using v8 heap dumps after tests, monitoring memory usage of chrome after the test suite runs.

It is important to note that often these kind of problems can lead to a confusing state where tests consistently work on production CI yet consistently fail on resource strained Travis CI environment.

Mitigation patterns

Over the years we have learned quite a few strategies you can adopt to help grapple with this problem. Some involve coding, others involve discussion. Arguably the most important first step is admitting you have a problem, and as a team, deciding how to confront it.

Start an honest discussion with your team

How should you deal with flaky tests? You could keep running them until they pass. You could delete them. You could quarantine and fix them. You could ignore this is happening.

At Discourse we opted to quarantine and fix. Though to be completely honest, at some points we ignored and we considered just deleting.

I am not sure there is a perfect solution here.

“Deleting and forgetting” can save money at the expense of losing a bit of test coverage and potential app bug fixes. If your test suite gets incredibly erratic, this kind of approach could get you back to happy state. As developers we are often quick to judge and say “delete and forget” is a terrible approach, it sure is drastic and some would judge this to be lazy and dangerous. However, if budgets are super tight this may be the only option you have. I think there is a very strong argument to say a test suite of 100 tests that passes 100% of the time when you rerun it against the same code base is better than a test suite of 200 tests where passing depends on a coin toss.

“Run until it passes” is another approach. It is an attempt to have the cake and eat it at the same time. You get to keep your build “green” without needing to fix flaky tests. Again, it can be considered somewhat “lazy”. The downside is that this approach may leave broken application code in place and make the test suite slower due to repeat test runs. Also, in some cases, “run until it passes” may fail on CI consistently and work on local consistently. How many retries do you go for? 2? 10?

“Do nothing” which sounds shocking to many, is actually surprisingly common. It is super hard to let go of tests you spent time carefully writing. Loss aversion is natural and means for many the idea of losing a test may just be too much to cope with. Many just say “the build is a flake, it sometimes fails” and kick it off again. I have done this in the past. Fixing flaky tests can be very very hard. In some cases where there is enormous amounts of environment at play and huge amounts of surface area, like large scale full application integration tests hunting for the culprit is like searching for a needle in a haystack.

“Quarantine and fix” is my favourite general approach. You “skip” the test and have the test suite keep reminding you that a test was skipped. You lose coverage temporarily until you get around to fixing the test.

There is no, one size fits all. Even at Discourse we sometimes live between the worlds of “Do nothing” and “Quarantine and fix”.

That said, having an internal discussion about what you plan to do with flaky tests is critical. It is possible you are doing something now you don’t even want to be doing, it could be behaviour that evolved.

Talking about the problem gives you a fighting chance.

If the build is not green nothing gets deployed

At Discourse we adopted continuous deployment many years ago. This is our final shield. Without this shield our test suite could have gotten so infected it would likely be useless now.

Always run tests in random order

From the very early days of Discourse we opted to run our tests in random order, this exposes order dependent flaky tests. By logging the random seed used to randomise the tests you can always reproduce a failed test suite that is order dependent.

Sadly rspec bisect has been of limited value

One assumption that is easy to make when presented with flaky tests, is that they are all order dependent. Order dependent flaky tests are pretty straightforward to reproduce. You do a binary search reducing the amount of tests you run but maintain order until you find a minimal reproduction. Say test #1200 fails with seed 7, after a bit of automated magic you can figure out that the sequence #22,#100,#1200 leads to this failure. In theory this works great but there are 2 big pitfalls to watch out for.

1. You may have not unrooted all your flaky tests, if the binary search triggers a different non-order dependent test failure, the whole process can fail with very confusing results.

2. From our experience with our code base the majority of our flaky tests are not order dependent. So this is usually an expensive wild goose chase.

Continuously hunt for flaky tests

Recently Roman Rizzi introduced a new system to hunt for flaky tests at Discourse. We run our test suite in a tight loop, over and over again on a cloud server. Each time tests fail we flag them and at the end of a week of continuous running we mark flaky specs as “skipped” pending repair.

This mechanism increased test suite stability. Some flaky specs may only show up 1 is 1000 runs. At snail pace, when running tests once per commit, it can take a very long time to find these rare flakes.

Quarantine flaky tests

This brings us to one of the most critical tools at your disposal. “Skipping” a flaky spec is a completely reasonable approach. There are though a few questions you should explore:

• Is the environment flaky and not the test? Maybe you have a memory leak and the test that failed just hit a threshold?

• Can you decide with confidence using some automated decision metric that a test is indeed flaky

There is a bit of “art” here and much depends on your team and your comfort zone. My advice here though would be to be more aggressive about quarantine. There are quite a few tests over the years I wish we quarantined earlier, which cause repeat failures.

Run flaky tests in a tight loop randomizing order to debug

One big issue with flaky tests is that quite often they are very hard to reproduce. To accelerate a repro I tend to try running a flaky test in a loop.

100.times do
   it "should not be a flake" do
      yet_it_is_flaky
   end
end

This simple technique can help immensely finding all sorts of flaky tests. Sometimes it makes sense to have multiple tests in this tight loop, sometimes it makes sense to drop the database and Redis and start from scratch prior to running the tight loop.

Invest in a fast test suite

For years at Discourse we have invested in speeding up to our test suite. There is a balancing act though, on one hand the best tests you have are integration tests that cover large amounts of application code. You do not want the quest for speed to compromise the quality of your test suite. That said there is often large amount of pointless repeat work that can be eliminated.

A fast test suite means

• It is faster for you to find flaky tests
• It is faster for you to debug flaky tests
• Developers are more likely to run the full test suite while building pieces triggering flaky tests

At the moment Discourse has 11,000 or so Ruby tests it takes them 5m40s to run single threaded on my PC and 1m15s or so to run tests concurrently.

Getting to this speed involves a regular amount of “speed maintenance”. Some very interesting recent things we have done:

• Daniel Waterworth introduced test-prof into our test suite and refined a large amount of tests to use: the let_it_be helper it provides (which we call fab! cause it is awesome and it fabricates). Prefabrication can provide many of the speed benefits you get from fixtures without inheriting the many of the limitations fixtures prescript.

• David Taylor introduced the parallel tests gem which we use to run our test suite concurrently saving me 4 minutes or so each time I run the full test suite. Built-in parallel testing is coming to Rails 6 thanks to work by Eileen M. Uchitelle and the Rails core team.

On top of this the entire team have committed numerous improvements to the test suite with the purpose of speeding it up. It remains a priority.

Add purpose built diagnostic code to debug flaky tests you can not reproduce

A final trick I tend to use when debugging flaky tests is adding debug code.

An example is here.

Sometimes, I have no luck reproducing locally no matter how hard I try. Diagnostic code means that if the flaky test gets triggered again I may have a fighting chance figuring out what state caused it.

def test_something
   make_happy(user)
   if !user.happy
      STDERR.puts "#{user.inspect}"
   end
    assert(user.happy)
end

Let’s keep the conversation going!

Do you have any interesting flaky test stories? What is your team’s approach for dealing with the problem? I would love to hear more so please join the discussion on this blog post.

Extra reading

• Eradicating Non-Determinism in Tests by Martin Fowler

• Google: Where do our flaky tests come from?

• An Empirical Analysis of Flaky Tests (pdf) - Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, Darko Marinov

• Microsoft: Eliminating Flaky tests

• Flaky tests at GitLab

• iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests by Wing Lam, Reed Oei, August Shi, Darko Marinov, Tao Xie

In 2015 I wrote about some of the tooling Ruby provides for diagnosing managed memory leaks. The article mostly focused on the easy managed leaks.

This article covers tools and tricks you can use to attack leaks that you can not easily introspect in Ruby. In particular I will discuss mwrap, heaptrack, iseq_collector and chap.

image.png1787×742 230 KB

An unmanaged memory leak

This little program leaks memory by calling malloc directly. It starts off consuming 16MB and finishes off consuming 118MB of RSS. The code allocates 100k blocks of 1024 bytes and de-allocates 50 thousand of them.

require 'fiddle'
require 'objspace'

def usage
  rss = `ps -p #{Process.pid} -o rss -h`.strip.to_i * 1024
  puts "RSS: #{rss / 1024} ObjectSpace size #{ObjectSpace.memsize_of_all / 1024}"
end

def leak_memory
  pointers = []
  100_000.times do
    i = Fiddle.malloc(1024)
    pointers << i
  end

  50_000.times do
    Fiddle.free(pointers.pop)
  end
end

usage
# RSS: 16044 ObjectSpace size 2817

leak_memory

usage
# RSS: 118296 ObjectSpace size 3374

Even though our RSS is 118MB, our Ruby object space is only aware of 3MB, introspection wise we have very little visibility of this very large memory leak.

A real world example of such a leak is documented by Oleg Dashevskii, it is an excellent article worth reading.

Enter Mwrap

Mwrap is a memory profiler for Ruby that keeps track of all allocations by intercepting malloc and family calls. It does so by intercepting the real calls that allocate and free memory using LD_PRELOAD. It uses liburcu for bookkeeping and is able to keep track of allocation and de-allocation counts per call-site for both C code and Ruby. It is reasonably lightweight and will approximately double the RSS for the program being profiled and approximately halve the speed.

It differs from many other libraries in that it is very lightweight and Ruby aware. It track locations in Ruby files and is not limited to C level backtrackes valgrind+masif and similar profilers show. This makes isolating actual sources of an issue much simpler.

Usage involves running an application via the mwrap wrapper, it inject the LD_PRELOAD environment and execs the Ruby binary.

Let’s append mwrap to our above script:

require 'mwrap'

def report_leaks
  results = []
  Mwrap.each do |location, total, allocations, frees, age_total, max_lifespan|
    results << [location, ((total / allocations.to_f) * (allocations - frees)), allocations, frees]
  end
  results.sort! do |(_, growth_a), (_, growth_b)|
    growth_b <=> growth_a
  end

  results[0..20].each do |location, growth, allocations, frees|
    next if growth == 0
    puts "#{location} growth: #{growth.to_i} allocs/frees (#{allocations}/#{frees})"
  end
end

GC.start
Mwrap.clear

leak_memory

GC.start

# Don't track allocations for this block
Mwrap.quiet do
  report_leaks
end

Next we will launch our script with the mwrap wrapper

% gem install mwrap
% mwrap ruby leak.rb
leak.rb:12 growth: 51200000 allocs/frees (100000/50000)
leak.rb:51 growth: 4008 allocs/frees (1/0)

Mwrap correctly detected the leak in the above script (50,000 * 1024). Not only it detected it, it isolated the actual line in the script ( i = Fiddle.malloc(1024) ) which caused the leak. It correctly accounted for the Fiddle.free calls.

It is important to note we are dealing with estimates here, mwrap keeps track of total memory allocated at the call-site and then keeps track of de-allocations. However, if you have a single call-site that is allocating memory blocks of different sizes the results can be skewed, we have access to the estimate: ((total / allocations) * (allocations - frees))

Additionally, to make tracking down leaks easier mwrap keeps track of age_total which is the sum of the lifespans of every object that was freed, and max_lifespan which is the lifespan of the oldest object in the call-site. If age_total / frees is high, it means the memory growth survives many garbage collections.

Mwrap has a few helpers that can help you reduce noise. Mwrap.clear will clear all the internal storage. Mwrap.quiet {} will suppress Mwrap tracking for a block of code.

Another neat feature Mwrap has is that it keeps track of total allocated bytes and total freed bytes. If we remove the clear from our script and run:

usage
puts "Tracked size: #{(Mwrap.total_bytes_allocated - Mwrap.total_bytes_freed) / 1024}"

# RSS: 130804 ObjectSpace size 3032
# Tracked size: 91691

This is very interesting cause even though our RSS is 130MB, Mwrap is only seeing 91MB, this demonstrates we have bloated our process. Running without mwrap shows that the process would normally be 118MB so in this simple case accounting is a mere 12MB, the pattern of allocation / deallocation caused fragmentation. Knowing about fragmentation can be quite powerful, in some cases with untuned glibc malloc processes can fragment so much that a very large amount memory consumed in RSS is actually free.

Could Mwrap isolate the old redcarpet leak?

In Oleg’s article he discussed a very thorough way he isolated a very subtle leak in redcarpet. There is lots of detail there. It is critical that you have instrumentation. If you are not graphing process RSS you have very little chance at attacking any memory leak.

Let’s step into a time machine and demonstrate how much easier it can be to use Mwrap for such leaks.

def red_carpet_leak
  100_000.times do

    markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML, extensions = {})
    markdown.render("hi")
  end
end

GC.start
Mwrap.clear

red_carpet_leak

GC.start

# Don't track allocations for this block
Mwrap.quiet do
  report_leaks
end

Redcarpet version 3.3.2

redcarpet.rb:51 growth: 22724224 allocs/frees (500048/400028)
redcarpet.rb:62 growth: 4008 allocs/frees (1/0)
redcarpet.rb:52 growth: 634 allocs/frees (600007/600000)

Redcarpet version 3.5.0

redcarpet.rb:51 growth: 4433 allocs/frees (600045/600022)
redcarpet.rb:52 growth: 453 allocs/frees (600005/600000)

Provided you can afford for a process to run at half speed simply re-launching it in production with Mwrap and logging Mwrap output once in a while to a file can identify a broad spectrum of memory leaks.

A mysterious memory leak

Recently we upgraded Rails to version 6 at Discourse. Overall the experience was extremely positive, performance remained more or less the same, Rails 6 includes some very nice features we get to use (like Zeitwerk).

Rails amended the way templates are rendered which required a few backwards compatible changes.

Fast forward a few days after our upgrade and we noticed RSS for our Sidekiq job runner was climbing.

Mwrap kept on reporting a sharp incline in memory due to memory being allocated at:

We initially found this very confusing and kept thinking to ourselves, why is Mwrap complaining? Could it be broken?

During the period where memory was climbing the Ruby heaps were not growing in size in a significant manner.

image.png1031×858 66.9 KB

2 million slots in the heap are a meager 78MB (40 bytes per slot), strings and arrays can take up more space, but this simply did not explain the enormous memory usage we were seeing. This was confirmed when I ran rbtrace -p SIDEKIQ_PID -e ObjectSpace.memsize_of_all.

Where did all the memory go?

Heaptrack

Heaptrack is a memory heap profiler for Linux.

Milian Wolff does a great job explaining what it is and how it came to be on his blog. He also has several talks about it (1, 2, 3)

In essence it is an incredibly efficient native heap profiler that gathers backtraces from a profiled applications using libunwind.

It is significantly faster than Valgrind/Massif and has a feature that makes is much more suitable for temporary production profiling.

It can attach to an already running process!

As with most heap profilers, when every single malloc family function is called it needs to do some accounting. This accounting certainly slows down the process a bit.

The design, in my mind, is the best possible design for this type of program. It intercepts using an LD_PRELOAD trick or a GDB trick to load up the profiler. It ships the data out of the profiled process as quickly as possibly using a FIFO special file. The wrapper heaptrack is a simple shell script, something that makes troubleshooting simple. A second process runs to read from the FIFO and compress the tracking data on the fly. Since heaptrack operates in “chunks” you can start looking at the profiled information seconds after you start profiling, mid way through a profiling session. Simply copy the profile file to another location and run the heaptrack gui.

This ticket at GitLab alerted me to the possibility of running heaptrack. Since they were able to run it, I knew it was a possibility for me.

We run our application in a container, I needed to relaunch our container with --cap-add=SYS_PTRACE which allows GDB to use ptrace which we needed so heaptrack can inject itself. Additionally, I needed a small hack on the shell file to allow root to profile a non root process (we run our Discourse application under a restricted account in the container).

Once this was done it was as simple as running heaptrack -p PID and waiting for results to stream in.

The UX of heaptrack is fantastic and extremely rich, it was very easy to follow what was happening with my memory leak.

image.png1555×911 356 KB

At a top level I could see two jumps, one was due to cppjieba and the other was originating from Ruby objspace_xmalloc0

I knew about cppjieba, segmenting Chinese is expensive, large dictionaries are needed, it was not leaking.

But why was ruby allocating memory and further more, not telling me about it?

image.png1846×871 333 KB

The largest increase was coming from iseq_set_sequence in compile.c. So it follows that we were leaking instruction sequences.

This made the leak Mwrap detected make sense. mod.module_eval(source, identifier, 0) was causing a leak cause it was creating instruction sequences that were never being removed.

In retrospect if I carefully analyzed a heap dump from Ruby I should have seen all these IMEMOs, cause they are included in heap dumps, just invisible from in-process introspection.

From here on debugging was pretty simple, I tracked down all calls to the module eval and dumped out what it was evaluating. I discovered we kept on appending methods over and over to a big class.

Simplified, this is the bug we were seeing:

require 'securerandom'
module BigModule; end

def leak_methods
  10_000.times do
    method = "def _#{SecureRandom.hex}; #{"sleep;" * 100}; end"
    BigModule.module_eval(method)
  end
end

usage
# RSS: 16164 ObjectSpace size 2869

leak_methods

usage
# RSS: 123096 ObjectSpace size 5583

Ruby has a class to contain instruction sequences called: RubyVM::InstructionSequence. However, Ruby is lazy about creating these wrapping objects, cause it is inefficient to have them around unless needed.

Interestingly Koichi Sasada created the iseq_collector gem. If we add this snippet we can now find our hidden memory:

require 'iseq_collector'
puts "#{ObjectSpace.memsize_of_all_iseq / 1024}"
# 98747

ObjectSpace.memsize_of_all_iseq will materialize every instruction sequence, which can introduce slight process memory growth and slightly more GC work.

If we, for example, count the number of ISEQs before and after running the collector we will notice that after running ObjectSpace.memsize_of_all_iseq our RubyVM::InstructionSequence class count grows from 0 to 11128 in the example above:

def count_iseqs
  ObjectSpace.each_object(RubyVM::InstructionSequence).count
end

These wrappers will stay around for the life of a method and need to be visited when a full GC runs.

For those curious, our fix to our issue was reusing the class responsible for rendering email templates. (fix 1, fix 2)

chap

During my debugging I came across a very interesting tool.

Tim Boddy, extracted an internal tool used at VMWare for analysis of memory leaks and open sourced it a few years ago. The only video I can find about it is here. Unlike most tools out there this tool has zero impact on a running process. It can simply run against core dump files, as long as the allocator being used is glibc (no support for jemalloc/tcmalloc etc)

The initial leak I had can be very easily detected using chap. Not many distros include a binary for chap, but you can easily build it from source. It is very actively maintained.

# 444098 is the `Process.pid` of the leaking process I had
sudo gcore -p 444098

chap core.444098
chap> summarize leaked
Unsigned allocations have 49974 instances taking 0x312f1b0(51,573,168) bytes.
   Unsigned allocations of size 0x408 have 49974 instances taking 0x312f1b0(51,573,168) bytes.
49974 allocations use 0x312f1b0 (51,573,168) bytes.

chap> list leaked
...
Used allocation at 562ca267cdb0 of size 408
Used allocation at 562ca267d1c0 of size 408
Used allocation at 562ca267d5d0 of size 408
...


chap> summarize anchored 
....
Signature 7fbe5caa0500 has 1 instances taking 0xc8(200) bytes.
23916 allocations use 0x2ad7500 (44,922,112) bytes.

Chap can use signatures to find where various memory is allocated and can complement GDB. When it comes to debugging Ruby it can do a great job helping you finding out what the actual memory is in use for a process. summarize used gives the actual memory, sometimes glibc malloc can fragment so much that the used number is enormously different to the actual RSS. See: Feature #14759: [PATCH] set M_ARENA_MAX for glibc malloc - Ruby master - Ruby Issue Tracking System for more discussion. Chap can correctly account for all memory usage and provide deep analysis around memory allocation behaviors.

Additionally chap can be integrated into build pipelines to automatically detect leaks and flag builds that are leaking.

Future work

This round of debugging did prompt me to raise a few issues with our supporting tool-sets:

• I would love to see heaptrack support grabbing call-stack frames from Ruby. Milian is interested in this problem as well: 412929 – Can we grab a frame from Ruby land?

• I would love Ruby to support richer introspection: https://bugs.ruby-lang.org/issues/16245

• I would love to see Mwrap be a little bit easier to use in production. Tracked here.

Summary

Our existing tooling for debugging very complex memory leaks in 2019 is vastly superior to what we had 4 years ago! Mwrap, heaptrack and chap provide us with very powerful tools for attacking memory related issues both in development and production.

If you are hunting a simple memory leak in Ruby, I recommend my earlier article from 2015, most of it still holds.

I hope that next time you are stuck debugging a complex native memory leak you have an easier time!

If you have any interesting battle stories or tools I have forgotten to mention you would like to share, please post a comment!

Disclaimers and technical details

If you are looking for comprehensive benchmarks, head over the Anthropic announcement blog post, or to the fantastic LMSYS leaderboard.

My goal with this blog post is to provide anecdote only. Here are a limited number of tests and examples pitting Claude 3 Opus against the current (as of 08-03-2024) flagship model by Open AI, GPT-4 Turbo version 0125-preview.

I hope to provide entertainment and a glimpse into the capabilities of both of these impressive models.

Persona editor in Discourse AI1108×1270 125 KB

I performed all my testing using Discourse AI which provides consistent tool (the ability to invoke functions) and persona support across a wide variety of large language models.

The words here are my own. I did not use an LLM to author this blog post. I did use some of the trivial proofreading tools.

Proof reading the above sentence784×311 30 KB

Opening moves, grok mode

Grok, X’s language model is infamous for having a “fun mode” where many of the filters are removed, so I decided to start this journey by attempting to simulate it…

Bypassing Opus’s safety guards is a huge challenge, it takes a fair amount of careful prompt engineering. It is clear that Anthropic have invested a huge amount of time on safety, so much that just getting it to be a “bit edgy” requires jailbreaking.

To me, Opus does a bit better in the example here, it is more concise and the jokes are actually paced a lot better. “Beatles mop-top. Hey Sam, the 60s called” and "Dario’s fashion sense is very “Silicon Valley chic”, both are better and far more punchy than what GPT-4 had to offer here.

The final countdown

Claude 3 Opus is a stickler when it comes to copyright.

This is not a surprise given:

These days all LLM manufacturers are struggling with fair use, coupled with not properly understanding the world and dates this can lead to somewhat amusing interactions.

Not only does Claude refuse incorrectly, later on it can be easily coerced to agree incorrectly, “A Farewell to Arms” is still in copyright for a few more years. That said the entire refusal here was wrong anyway.

GPT-4 on the other hand aces this:

Who tells better jokes?

Is any of this funny? I am not sure, jokes are hard. Opus though is far better at delivery and GPT-4 tends to feel quite tame and business like compared to Opus.

Discourse Setting Explorer

We ship with a persona that injects source code context by searching through our repository, it allows us to look up information regarding settings in Discourse. For example:

Overall in this particular interaction, I preferred the response from Claude. It had more nuance, and it was able to complete the task faster than GPT-4.

SQL Support

One of the most popular internal uses of LLMs at Discourse has been SQL authoring. We have it integrated into a persona that can retrieve schema from the database, giving you accurate SQL generation. (Given persona support and the enormous 200k/120k context window of these models, you could use this for your own database as well by including the full schema in your system prompt)

Let’s look at what the Sql Helper persona can do:

Both are very interesting journeys with twists and turns. I picked a pretty complex example to highlight the behaviors of the models better.

Claude was off to a phenomenal start, but then found itself in a deep rabbit hole which I had to dig it out of. GPT-4 totally missed on the user_visits table on first go and needed extra care to send it down the right path.

GPT-4 missed that to_char(lw.day, 'Day') produces a day name and instead implemented it by hand.

Both models generated queries that return errors and both recovered with simple guidance, I found the GPT-4 recovery a bit more enjoyable.

The subtle error in Claude was concerning, it missed a bunch of activity.

Overall both are great, however if you are building an extremely complex query you are going to need to be prepared to get involved.

End result of the query generation session on both models712×411 25.8 KB

Let’s draw some pictures

I am very impressed with Claude 3s prompt expansion prowess. My favorite in the series is:

image1024×1024 122 KB

LLMs are spectacular at writing prompts for image generation models. Even simpler models like GPT-3.5 can do a pretty great job. However I find that these frontier models outdo the simpler ones and Claude here did phenomenally well.

Let’s review some source code

Integrating LLMs into GitHub is truly magical.

We just added a GitHub Helper persona that can perform searches, read code and read PRs via tool calls.

This means we can do stuff like this:

Both are good reviews, but I feel Opus did a bit better here. The suggestions for tests were more targeted, commit message is a bit more comprehensive.

It is important to note though from many experiments that this is not a mechanism for removing the human from the loop, if you treat this as a brainstorming and exploration session you can get the maximum amount of benefit.

A coding assistant

Being able to talk to a Github repo (search, read files) unlocks quite a lot of power on both models:

Both offered an interesting exploration, both found the place where code needed changing. Neither provided a zero intervention solution.

I find GPT-4 more “to the point” and Claude a bit more “creative” that said both do a good job and can be helpful while coding as long as you you treat these models as “helpers” that sometimes make mistakes vs an end-to-end solver of all problems.

A front end for Google

One of our personas, the researcher, uses Google for Retrieval-Augmented-Generation:

I love the superpower of being able to search Google in any language I want.

I love how eager Claude is to please, but still feel GPT-4 has a slight upper hand here.

Implementation notes

Implementing tools on language models without a clear tool API is complicated, fragile, and tricky.

GPT-4 is significantly easier to integrate into complex workflows due to its robust tool framework. Claude is “workable,” but many refinements are still needed.

Claude’s streaming API wins over Open AI. You can get token counts after streaming, something that is absent from Open AI’s API.

Claude Opus is significantly slower than GPT-4 Turbo, something you feel quite a lot when testing it. It is also significantly more expensive at present.

That said, Opus is an amazing and highly available language model that can sometimes do better than GPT-4. It is an impressive achievement by Anthropic!

Token counts

The elephant in the room is API costs especially on the next generation 1-2 million token language models such as Claude 3 (which is artificially limited to 200k tokens) and Gemini 1.5 pro.

The pricing model is going to have to change.

At the moment APIs ship with no memory. You can not manage context independently of conversation.

A new breed of language model APIs is going to have to evolve this year:

• Load context API (which allows you to load up all the context information, Eg: full GitHub repos, books, etc…)
• Conversation API - which let’s you query the LLM with a pre-loaded context.

Absent of this, it is going to be very easy to reach situations with Claude 3 Opus where every exchange costs $2, admittedly it could be providing this value, but the cost quickly can become prohibitive.

Other thoughts and conclusion

I am trying to rush out this blog post, usually I wait a bit longer when posting, but Claude is “hot” at the moment. Many are very curious. Hopefully you find the little examples here interesting, feel free to leave a note here if you want to talk about any of this!

My first impressions are that Claude 3 Opus is a pretty amazing model which is highly capable. The overcautious approach to copyright and lack of native tool support are my two biggest gripes. Nonetheless it is an incredibly fun model to interact with, it “gets” what you are asking it to do and consistently does a good job.

If you are looking for a way to run Claude 3 / GPT-4 and many other language models with tool support, check out Discourse AI, I used it for all the experiments and presentation here.

Regarding the conflict between hardcoded ids and database sequences, I would propose a better solution. The trick is easy, you could just set a minimal value for every sequence before starting tests.

For example, you can set those values to 1000000, and then enjoy using any id below this number explicitly — it won’t conflict with an id autogenerated by the database.

That’s how the trick works on Ruby on Rails / Postgresl

We went on a similar journey with flakey tests a while back. Most of them were database state issues, or race conditions in feature specs. Keith posted some of the tools we added to help identify and fix these flakey failures and maybe they’d be useful to you, too! Although I suspect you do most of these in Discourse anyway.