Why Your Ceph Cluster Needs SSDs With Power Loss Protection

A consumer SSD took 1.73 ms per sync write. An enterprise SSD with power loss protection took 0.46 ms.

That gap is what every VM on a Ceph cluster feels. I benchmarked both drives with fio and explain where the difference comes from, why the datasheet hides it, and what to buy instead.

Tested in September 2026 on a Proxmox VE 9.2.11 node in my homelab, fio 3.39, one SATA SSD of each kind.

The two drives

On paper, the consumer drive is the faster one.

SanDisk SDSSDH3500G 500GB (consumer)

3D TLC NAND (consumer-grade)
No power loss protection (PLP)
Sequential read/write up to 560/530 MB/s, but only inside the nCache 2.0 SLC cache burst window
Random IOPS up to 95K read / 84K write, same caveat: burst only, not sustained
200 TBW endurance, no DWPD spec (not rated for sustained enterprise write load)

Micron 5100 MTFDDAK480TCB 480GB (enterprise)

3D eTLC NAND (enterprise-grade, higher endurance per cell than consumer TLC)
Full power loss protection
Sequential read/write 540/410 MB/s (sustained, no cache-dependent peak)
Random IOPS 93K read / 43K write (steady-state rating)
1.5 DWPD / 1300 TBW endurance, large overprovisioning (spare area) for consistent latency under sustained load

The SanDisk wins almost every headline number. The line that decides the result is the second one: power loss protection.

Why sync writes matter for Ceph

Ceph does not acknowledge a client write until it is durable. BlueStore commits each write through its RocksDB write-ahead log and waits for the device to confirm the flush, and the primary OSD answers the client only after every replica has done the same. With size=3, each client write waits for the slowest of three disks.

So the number that matters is not peak throughput but how long a single flushed write takes. A 4k sync write at iodepth=1 is the closest single-disk proxy for that. It is also exactly the case the datasheet IOPS figures leave out: those are measured at high queue depth, without a flush after every write.

fio benchmarks

I ran two tests on each drive:

4k random writes with --sync=1 at iodepth=1. Every write has to be committed before the next one starts, so this measures commit latency.
4M mixed sequential reads and writes, 50/50, at iodepth=16, to see how each drive behaves when it is saturated.

One caveat on the setup: the SanDisk is the node’s LVM root disk. Its test file sat on the root filesystem, went through device mapper, and shared the drive with whatever the host was writing at the time. The Micron was a dedicated, otherwise idle disk mounted at /mnt/ssd-test. That favors the Micron, so read the factors below as indicative rather than exact.

Sync write latency

Consumer disk:

fio --filename=/root/fiotest --size=20G --direct=1 --sync=1 --rw=randwrite --bs=4k \
  --numjobs=1 --iodepth=1 --runtime=60 --time_based \
  --group_reporting --name=4k-randwrite-plp --ioengine=libaio

4k-randwrite-plp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.39
Starting 1 process
4k-randwrite-plp: Laying out IO file (1 file / 20480MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=2210KiB/s][w=552 IOPS][eta 00m:00s]
4k-randwrite-plp: (groupid=0, jobs=1): err= 0: pid=3753158: Sun Sep 20 17:43:36 2026
  write: IOPS=558, BW=2235KiB/s (2288kB/s)(131MiB/60001msec); 0 zone resets
    slat (usec): min=13, max=1668, avg=51.37, stdev=23.92
    clat (usec): min=3, max=27873, avg=1730.74, stdev=697.18
     lat (usec): min=1025, max=27899, avg=1782.11, stdev=701.01
    clat percentiles (usec):
     |  1.00th=[ 1188],  5.00th=[ 1287], 10.00th=[ 1352], 20.00th=[ 1467],
     | 30.00th=[ 1598], 40.00th=[ 1663], 50.00th=[ 1713], 60.00th=[ 1762],
     | 70.00th=[ 1811], 80.00th=[ 1860], 90.00th=[ 1926], 95.00th=[ 1975],
     | 99.00th=[ 3851], 99.50th=[ 4817], 99.90th=[12518], 99.95th=[16188],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1424, max= 2872, per=100.00%, avg=2235.23, stdev=252.01, samples=119
   iops        : min=  356, max=  718, avg=558.81, stdev=63.00, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 1000=0.01%
  lat (msec)   : 2=96.38%, 4=2.68%, 10=0.79%, 20=0.10%, 50=0.03%
  cpu          : usr=0.98%, sys=3.63%, ctx=33647, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,33521,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2235KiB/s (2288kB/s), 2235KiB/s-2235KiB/s (2288kB/s-2288kB/s), io=131MiB (137MB), run=60001-60001msec

Disk stats (read/write):
    dm-1: ios=115/172357, sectors=3192/1388656, merge=0/0, ticks=67/57917, in_queue=57984, util=81.94%, aggrios=112/101475, aggsectors=9336/1391728, aggrmerge=27/71267, aggrticks=72/50376, aggrin_queue=87571, aggrutil=82.26%
  sdb: ios=112/101475, sectors=9336/1391728, merge=27/71267, ticks=72/50376, in_queue=87571, util=82.26%

Enterprise disk:

fio --filename=/mnt/ssd-test/fiotest --size=20G --direct=1 --sync=1 --rw=randwrite --bs=4k \
  --numjobs=1 --iodepth=1 --runtime=60 --time_based \
  --group_reporting --name=4k-randwrite-plp --ioengine=libaio

4k-randwrite-plp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.39
Starting 1 process
4k-randwrite-plp: Laying out IO file (1 file / 20480MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=6728KiB/s][w=1682 IOPS][eta 00m:00s]
4k-randwrite-plp: (groupid=0, jobs=1): err= 0: pid=3765636: Sun Sep 20 18:03:17 2026
  write: IOPS=2063, BW=8253KiB/s (8452kB/s)(484MiB/60001msec); 0 zone resets
    slat (usec): min=8, max=4925, avg=21.82, stdev=17.98
    clat (usec): min=2, max=68116, avg=458.77, stdev=562.47
     lat (usec): min=219, max=68193, avg=480.59, stdev=564.04
    clat percentiles (usec):
     |  1.00th=[  219],  5.00th=[  227], 10.00th=[  241], 20.00th=[  277],
     | 30.00th=[  306], 40.00th=[  334], 50.00th=[  371], 60.00th=[  437],
     | 70.00th=[  502], 80.00th=[  562], 90.00th=[  627], 95.00th=[  676],
     | 99.00th=[ 3326], 99.50th=[ 4752], 99.90th=[ 5604], 99.95th=[ 5735],
     | 99.99th=[ 6456]
   bw (  KiB/s): min= 6056, max=13448, per=100.00%, avg=8256.54, stdev=1685.84, samples=119
   iops        : min= 1514, max= 3362, avg=2064.13, stdev=421.46, samples=119
  lat (usec)   : 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%, 250=12.94%
  lat (usec)   : 500=56.66%, 750=28.60%, 1000=0.51%
  lat (msec)   : 2=0.13%, 4=0.17%, 10=0.96%, 50=0.01%, 100=0.01%
  cpu          : usr=2.05%, sys=5.90%, ctx=124046, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,123804,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=8253KiB/s (8452kB/s), 8253KiB/s-8253KiB/s (8452kB/s-8452kB/s), io=484MiB (507MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/375759, sectors=0/10330368, merge=0/255651, ticks=0/52568, in_queue=68630, util=66.67%

	Consumer	Enterprise	Factor
IOPS	558	2063	3.7x
BW	2235 KiB/s	8253 KiB/s	3.7x
avg clat	1.73 ms	459 µs	3.8x
p99	3.85 ms	3.33 ms	~
p99.9	12.5 ms	5.6 ms	2.2x

The Micron commits a sync write 3.8x faster on average: 0.46 ms against 1.73 ms. That is clat, the completion latency, the time until the device reports the write as done. IOPS and bandwidth follow at 3.7x, which is what you expect at iodepth=1, where throughput is the inverse of latency.

At p99 the two drives are close, 3.85 ms against 3.33 ms, and the gap reopens at p99.9. The difference is in the typical write, and for Ceph that is the expensive kind, because every write pays it.

The reason is PLP. Without it, the drive has to flush its volatile DRAM cache to NAND before it can acknowledge a sync write, because anything still in cache would be lost on power failure. With PLP, capacitors hold enough energy to flush the cache after power is cut, so the drive acknowledges the write as soon as it is in DRAM.

This is also why the SanDisk’s 84K write IOPS rating means nothing here. With a flush after every write, it managed 558.

Under saturation

Consumer disk:

fio --filename=/root/fiotest --size=20G --direct=1 --rw=rw --rwmixread=50 --bs=4M \
  --numjobs=1 --iodepth=16 --runtime=60 --time_based \
  --group_reporting --name=4m-rw --ioengine=libaio

4m-seq-rw: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=96.1MiB/s,w=160MiB/s][r=24,w=40 IOPS][eta 00m:00s]
4m-seq-rw: (groupid=0, jobs=1): err= 0: pid=3757672: Sun Sep 20 17:50:42 2026
  read: IOPS=32, BW=129MiB/s (135MB/s)(7756MiB/60183msec)
    slat (usec): min=161, max=225459, avg=15288.49, stdev=14897.15
    clat (msec): min=26, max=841, avg=256.36, stdev=119.03
     lat (msec): min=26, max=858, avg=271.65, stdev=123.81
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   81], 10.00th=[  114], 20.00th=[  148],
     | 30.00th=[  199], 40.00th=[  230], 50.00th=[  251], 60.00th=[  275],
     | 70.00th=[  305], 80.00th=[  338], 90.00th=[  393], 95.00th=[  451],
     | 99.00th=[  684], 99.50th=[  743], 99.90th=[  827], 99.95th=[  844],
     | 99.99th=[  844]
   bw (  KiB/s): min=32768, max=466944, per=99.89%, avg=131822.93, stdev=59397.00, samples=120
   iops        : min=    8, max=  114, avg=32.18, stdev=14.50, samples=120
  write: IOPS=33, BW=136MiB/s (142MB/s)(8156MiB/60183msec); 0 zone resets
    slat (usec): min=228, max=207650, avg=14833.30, stdev=14247.80
    clat (msec): min=30, max=895, avg=198.94, stdev=81.73
     lat (msec): min=34, max=905, avg=213.77, stdev=85.98
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   78], 10.00th=[  114], 20.00th=[  134],
     | 30.00th=[  153], 40.00th=[  171], 50.00th=[  188], 60.00th=[  209],
     | 70.00th=[  228], 80.00th=[  262], 90.00th=[  305], 95.00th=[  347],
     | 99.00th=[  430], 99.50th=[  472], 99.90th=[  659], 99.95th=[  776],
     | 99.99th=[  894]
   bw (  KiB/s): min=16384, max=450560, per=99.96%, avg=138717.87, stdev=62023.37, samples=120
   iops        : min=    4, max=  110, avg=33.87, stdev=15.14, samples=120
  lat (msec)   : 50=0.98%, 100=6.36%, 250=56.71%, 500=34.14%, 750=1.53%
  lat (msec)   : 1000=0.28%
  cpu          : usr=0.64%, sys=1.83%, ctx=4085, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1939,2039,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=7756MiB (8133MB), run=60183-60183msec
  WRITE: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=8156MiB (8552MB), run=60183-60183msec

Disk stats (read/write):
    dm-1: ios=8122/31862, sectors=14755480/16688024, merge=0/0, ticks=1118815/3201704, in_queue=4320519, util=99.88%, aggrios=15362/16749, aggsectors=14780152/16725088, aggrmerge=5/20712, aggrticks=2064436/1556255, aggrin_queue=3623047, aggrutil=97.44%
  sdb: ios=15362/16749, sectors=14780152/16725088, merge=5/20712, ticks=2064436/1556255, in_queue=3623047, util=97.44%

Enterprise disk:

fio --filename=/mnt/ssd-test/fiotest --size=20G --direct=1 --rw=rw --rwmixread=50 --bs=4M \
  --numjobs=1 --iodepth=16 --runtime=60 --time_based \
  --group_reporting --name=4m-rw --ioengine=libaio

4m-seq-rw: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=260MiB/s,w=176MiB/s][r=65,w=44 IOPS][eta 00m:00s]
4m-seq-rw: (groupid=0, jobs=1): err= 0: pid=3767619: Sun Sep 20 18:06:36 2026
  read: IOPS=53, BW=216MiB/s (226MB/s)(12.7GiB/60066msec)
    slat (usec): min=122, max=85828, avg=9511.92, stdev=4122.90
    clat (usec): min=1859, max=197633, avg=116818.14, stdev=15146.99
     lat (msec): min=20, max=206, avg=126.33, stdev=15.21
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   96], 10.00th=[  103], 20.00th=[  109],
     | 30.00th=[  112], 40.00th=[  115], 50.00th=[  117], 60.00th=[  121],
     | 70.00th=[  124], 80.00th=[  127], 90.00th=[  132], 95.00th=[  138],
     | 99.00th=[  153], 99.50th=[  163], 99.90th=[  188], 99.95th=[  194],
     | 99.99th=[  199]
   bw (  KiB/s): min=147456, max=409600, per=99.86%, avg=220838.30, stdev=36805.83, samples=120
   iops        : min=   36, max=  100, avg=53.91, stdev= 8.98, samples=120
  write: IOPS=55, BW=223MiB/s (234MB/s)(13.1GiB/60066msec); 0 zone resets
    slat (usec): min=227, max=77246, avg=8658.46, stdev=3884.91
    clat (msec): min=38, max=298, avg=155.62, stdev=22.66
     lat (msec): min=41, max=308, avg=164.28, stdev=23.50
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  118], 10.00th=[  133], 20.00th=[  142],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  157], 60.00th=[  161],
     | 70.00th=[  167], 80.00th=[  174], 90.00th=[  182], 95.00th=[  190],
     | 99.00th=[  209], 99.50th=[  220], 99.90th=[  236], 99.95th=[  247],
     | 99.99th=[  300]
   bw (  KiB/s): min=172032, max=352256, per=99.90%, avg=228483.63, stdev=30563.50, samples=120
   iops        : min=   42, max=   86, avg=55.77, stdev= 7.45, samples=120
  lat (msec)   : 2=0.02%, 4=0.03%, 20=0.03%, 50=0.21%, 100=4.73%
  lat (msec)   : 250=94.97%, 500=0.02%
  cpu          : usr=1.15%, sys=2.75%, ctx=6567, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3243,3354,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=216MiB/s (226MB/s), 216MiB/s-216MiB/s (226MB/s-226MB/s), io=12.7GiB (13.6GB), run=60066-60066msec
  WRITE: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=13.1GiB (14.1GB), run=60066-60066msec

Disk stats (read/write):
  sda: ios=27589/25263, sectors=25581872/27388376, merge=20/137330, ticks=1445594/2147154, in_queue=3593204, util=99.86%

	Consumer	Enterprise	Factor
Read BW	129 MiB/s	216 MiB/s	1.7x
Write BW	136 MiB/s	223 MiB/s	1.6x
Read clat stdev	119 ms	15 ms	7.9x
Read clat p99.9	827 ms	188 ms	4.4x
Read clat max	841 ms	198 ms	4.3x
Write clat stdev	82 ms	23 ms	3.6x
Write clat p99.9	659 ms	236 ms	2.8x

Both drives ran at their limit: util is 97.44% for the SanDisk and 99.86% for the Micron.

This test does not flush, so PLP is not what separates the drives here. The Micron is rated for steady-state load, the SanDisk only for its SLC cache burst window, and the results fit that. The Micron delivers 1.7x the read and 1.6x the write throughput, but the more important number is the spread. Its read latency stdev is 15 ms against 119 ms, its worst read 198 ms against 841 ms. On an OSD, a tail like the SanDisk’s turns into VMs that are fine most of the time and occasionally stall.

What to buy

For Ceph clusters that host VMs, for example a hyperconverged Proxmox VE cluster, I would pick write-intensive NVMe drives with PLP. Ceph writes a lot, and every write waits for the slowest replica.

I used SATA SSDs because that is what my homelab has. In an enterprise environment, there is little reason left to choose SAS/SATA SSDs over NVMe, because the price difference is no longer that big. The consumer-versus- enterprise gap is the same on NVMe: a consumer NVMe drive without PLP has the same flush problem, from a faster baseline.

The Ceph hardware recommendations make the same point about PLP and cover the rest of drive selection.

My takeaway

If you run Ceph in production, there is no way around enterprise drives with PLP. Your workloads have to keep running during I/O spikes and during backfill after a drive failure, and that is exactly when a consumer drive’s latency hits every client.

In a small homelab, consumer drives work, and depending on the workload you may not notice the difference. Just don’t let the datasheet decide. Check for power loss protection before you look at a single IOPS number.