<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title></title>
    <description></description>
    <link>https://slcz.github.io/</link>
    <atom:link href="https://slcz.github.io/feed.xml" rel="self" type="application/rss+xml"/>
    <pubDate>Thu, 04 Feb 2016 07:10:45 +0000</pubDate>
    <lastBuildDate>Thu, 04 Feb 2016 07:10:45 +0000</lastBuildDate>
    <generator>Jekyll v3.0.2</generator>
    
      <item>
        <title>Write Amplification, Randomness and Math</title>
        <description>&lt;p&gt;Imagine sitting inside a sushi restaurant, where shshi plates are placed on a slowly moving conveyor bar. Customer picks up full plates and leaves empty ones on the bar. Suppose a waiter is standing at the end of conveyor and moving empty plates to the kitchen, meanwhile keeps adding new plates into the loop. How fast should the waiter moves the plates in order to keep the bar full without leaving empty plates on the bar for too long?&lt;/p&gt;

&lt;p&gt;&lt;img alt=&quot;&quot; width=&quot;400&quot; src=&quot;/assets/sushi.jpg&quot; /&gt;&lt;/p&gt;

&lt;p&gt;From time to time, while working on flash controller and SSDs, I encountered the same question. In a somewhat similar situation as the sushi bar metaphor, a flash device is written sequentially regardless of host writes’ address pattern. Considering a simplified case, suppose an external host (applications) write to the device consistently and the workload is randomly distributed across the entire address space. How fast should the SSD move fragmented blocks around in order to keep up with the data ingest rate?&lt;/p&gt;

&lt;p&gt;We know a SSD exposes smaller capacity to the applications than its raw capacity. Overtime, a younger write with the same host address may overwrite one or more older ones (to the same address), creating space fragmentation. Therefore, a background garbage collector is running and packs data and defragments the holes. For a given workload, the more the over-provisioning space, the less work SSD’s garbage collector has to do to re-pack the working dataset. Obviously, the trade-off of having more space reservation is the wasteful of flash capacity.&lt;/p&gt;

&lt;p&gt;Modern flash controller inside SSD may employ sophisticated garbage collection algorithm. Some of them exploit spatial locality, e.g., they may try to pack infrequently accessed data together in order to reduce the write amplification (defined as back-end writes to host writes ratio). For instance, a multiple streams GC algorithm writes cold data into a separate stream.&lt;/p&gt;

&lt;p&gt;&lt;img alt=&quot;multiple stream GC&quot; width=&quot;500&quot; src=&quot;/assets/gc.png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;While it is understandable that an intelligent algorithm may take advantage of workload patterns, sometimes we are interested in the case where the entire write workload (both in terms of address and data pattern) is randomly distributed. This provides a fair ground when comparing the benchmarks of the I/O throughput. The same applies to data patterns, too. Random pattern is used heavily in micro-benchmarking tools such as &lt;a href=&quot;http://freecode.com/projects/fio&quot;&gt;&lt;em&gt;fio&lt;/em&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Another point that may not be so obvious is that in terms of distribution of the fragmentation, a randomly generated address pattern is different from a uniformly strided (stride » 1) address pattern. For write addresses that are generated by a true random generator, an old aged block has better probability of being overwritten by younger ones, while the younger blocks are less likely. A uniformly strided address pattern causes all the blocks on the device to have the same probability of being “staled”.&lt;/p&gt;

&lt;p&gt;&lt;img alt=&quot;fragmentation distribution&quot; width=&quot;500&quot; src=&quot;/assets/holes.png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;In a randomly distributed workload where there is no obvious distinction of address pattern locality, multiple-queue GC algorithm won’t work better than a naive single queued, round-robin GC algorithm. Therefore, I choose the simpler GC to model random workload write amplification. Also, I choose to model the GC algorithm as being lazy, i.e., GC only happens when there is no free space left. In another words, gc-threshold above is set close to zero.&lt;/p&gt;

&lt;p&gt;&lt;img alt=&quot;multiple stream GC&quot; width=&quot;500&quot; src=&quot;/assets/naive_gc.png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;While it is possible to run a simulator and get write amplification distribution on space over-provisioning, wouldn’t it be interesting to try to derive the model analytically? We know for any block that carries data, we could calculate the probability that it is still valid (not overwritten). As below,&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            P_{valid}=(1-\frac{1}{U})^W
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (1)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;
&lt;p&gt;Where,&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;script type=&quot;math/tex&quot;&gt;P_{valid}&lt;/script&gt; is the probability that the block is valid.&lt;/li&gt;
  &lt;li&gt;&lt;script type=&quot;math/tex&quot;&gt;U&lt;/script&gt; is the host addressable space (number of blocks).&lt;/li&gt;
  &lt;li&gt;&lt;script type=&quot;math/tex&quot;&gt;W&lt;/script&gt; is the total number of of younger writes from host&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Due to lazy GC policy, &lt;script type=&quot;math/tex&quot;&gt;U, W&lt;/script&gt; are both very large numbers that are in the same magnitude as the total physical capacity in the device (millions to billions of 4KB blocks). Further, we introduce 2 variables,
&lt;script type=&quot;math/tex&quot;&gt;f&lt;/script&gt; and &lt;script type=&quot;math/tex&quot;&gt;\lambda&lt;/script&gt;. &lt;script type=&quot;math/tex&quot;&gt;f&lt;/script&gt;
 is the ratio of usable capacity to physical capacity. &lt;script type=&quot;math/tex&quot;&gt;\lambda&lt;/script&gt;,
defined as the ratio of host writes to total device writes, is the
inverse of write amplification, which we denoted as &lt;script type=&quot;math/tex&quot;&gt;\omega&lt;/script&gt;.&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            U=fS
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (2)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            W=\lambda S=\frac{S}{\omega}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (3)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;From &lt;em&gt;Eq.&lt;/em&gt; (1), (2) and (3), we get,&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            P_{valid}=(1-\frac{1}{U})^W=
            (1-\frac{\lambda}{f}\frac{1}{\lambda S})^{W}=
            (1-\frac{\lambda}{f}\frac{1}{W})^W
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (4)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;In the case where garbage collection is lazy, &lt;script type=&quot;math/tex&quot;&gt;W&lt;/script&gt; is large. As &lt;script type=&quot;math/tex&quot;&gt;W&lt;/script&gt;
approaches infinity, the page valid probability becomes,&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            \Psi=\lim_{W \to \infty} P_{valid}=\lim_{W \to \infty}(1-\frac{\lambda}{f}\frac{1}{W})^W=e^{-\frac{\lambda}{f}}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (5)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;&lt;script type=&quot;math/tex&quot;&gt;\Psi&lt;/script&gt; approximates the valid probability of the earliest block written in the system. &lt;em&gt;Eq.&lt;/em&gt; (5) is due to the fact that
&lt;script type=&quot;math/tex&quot;&gt; \lim_{n \to \infty} (1+\frac{k}{n})^n = e^k&lt;/script&gt;.
Also, write amplification has the following relationship to the block valid probability,&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            \omega=\frac{1}{\lambda}=\frac{1}{1-\Psi}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (6)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            \Psi=1-\lambda
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (7)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;With &lt;em&gt;Eq.&lt;/em&gt; (5) and (7),&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            1-\lambda=e^{-\frac{\lambda}{f}}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (8)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            1-\frac{1}{\omega}=e^{-\frac{1}{f\omega}}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (9)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;Where,&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;script type=&quot;math/tex&quot;&gt;\omega&lt;/script&gt; is the write amplification,&lt;/li&gt;
  &lt;li&gt;&lt;script type=&quot;math/tex&quot;&gt;f&lt;/script&gt; is the ratio of physical capacity to usable capacity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While &lt;em&gt;Eq.&lt;/em&gt; (9) does not have elementary solution, it can be solved by
using &lt;a href=&quot;https://en.wikipedia.org/wiki/Lambert_W_function&quot;&gt;lambertW function&lt;/a&gt;.&lt;/p&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            \omega=\frac{\zeta}{\zeta-W(\zeta e^\zeta)}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (10)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;table class=&quot;numbered-equation&quot;&gt;
    &lt;tr&gt;
        &lt;td class=&quot;eq&quot;&gt;
            &lt;script type=&quot;math/tex; mode=display&quot;&gt;
            \zeta=-\frac{1}{f}
            &lt;/script&gt;
        &lt;/td&gt;
        &lt;td class=&quot;equation-number&quot;&gt;
            (11)
        &lt;/td&gt;
    &lt;/tr&gt;
&lt;/table&gt;

&lt;p&gt;&lt;em&gt;Eq.&lt;/em&gt; (10) and (11) solves write amplification as a function of usable
ratio. With a good &lt;a href=&quot;http://keithbriggs.info/software/LambertW.c&quot;&gt;LambertW function approximator&lt;/a&gt;, I
plug in the numbers and get the following table,&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th&gt;UsableRatio&lt;/th&gt;
      &lt;th&gt;WriteAmplification&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;95%&lt;/td&gt;
      &lt;td&gt;10.17&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;90%&lt;/td&gt;
      &lt;td&gt;5.18&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;85%&lt;/td&gt;
      &lt;td&gt;3.52&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;80%&lt;/td&gt;
      &lt;td&gt;2.69&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;75%&lt;/td&gt;
      &lt;td&gt;2.20&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;70%&lt;/td&gt;
      &lt;td&gt;1.88&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;65%&lt;/td&gt;
      &lt;td&gt;1.65&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;60%&lt;/td&gt;
      &lt;td&gt;1.48&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;55%&lt;/td&gt;
      &lt;td&gt;1.35&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;50%&lt;/td&gt;
      &lt;td&gt;1.26&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;p&gt;Also plotted,&lt;/p&gt;

&lt;p&gt;&lt;img alt=&quot;write amp to usable ratio&quot; width=&quot;800&quot; src=&quot;/assets/wa.png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;This matches with the simulation very well. In reality, most of the
consumer grade SSDs are thinly over-provisioned and leave less than
10% of capacity reservation. This would slow down the performance
by 5X to 10X under stress writes. You could even tell how much space does a
particular SSD preserve by observing their steady state
throughput degradation ratio.
Just don’t forget to pre-condition the SSD by filling the SSD
before any stress testing. It may take very long for the performance to stabilize.&lt;/p&gt;

&lt;p&gt;The above analytical exercise may also help space and performance planing.
For example, in an anticipated write heavy environment, one could get
predictable throughput by intentionally creating the file system at lower
than advertised capacity.
Next time, format the drive at 70% of the advertised capacity if you want
to ensure the worst case throughput is better than 50% of the peak!&lt;/p&gt;

</description>
        <pubDate>Tue, 02 Feb 2016 21:15:31 +0000</pubDate>
        <link>https://slcz.github.io/blog/about-ssd-write-amplification/</link>
        <guid isPermaLink="true">https://slcz.github.io/blog/about-ssd-write-amplification/</guid>
        
        <category>SSD</category>
        
        <category>Flash</category>
        
        <category>Write Amplification</category>
        
        
      </item>
    
  </channel>
</rss>
