• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

Dedupe and Checksumming in VSAN beta, what about Compression? Yup!

Duncan Epping · Oct 16, 2015 ·

I was talking to our engineering team a couple of weeks ago with regards to our upcoming beta. During a demo I noticed that in our UI it said “Space Efficiency” instead of deduplication. After some discussions it became clear to me why that is… It isn’t “just” deduplication which will be in the upcoming beta for Virtual SAN, but the engineering team managed to also get compression in there. Huge accomplishment if you ask me and I hope this will make the official next release.

So where does compression fit? Compression happens after deduplication and we expect to store 2KB for every unique 4KB block. Combine this with “RAID5/6” over the network, aka “erasure coding” and with deduplication and thin provisioning and I bet you will see significant space savings! Great new functionality (if and when released), which will especially make the all-flash configuration a no-brainer if you ask me. Especially with flash prices still dropping and the price being almost on the same level as SAS drives these days it will make this a very compelling solution.

Related

Server 6.x, beta, virtual san, vsan

Reader Interactions

Comments

  1. Fletcher Cocquyt says

    16 October, 2015 at 13:14

    Is compression worth the CPU load ?
    Maybe for a very finite workload set?

    • Joseph Hoot says

      16 October, 2015 at 14:04

      We have used compression in our ZFS appliance over dedupe because the LZJB algorithm we use is achieving a 1.65x compression ratio and doesn’t tax the CPU like a GZIP or BJZIP algorithm. So I would counter that by suggesting that dedupe may be more CPU intensive than the compression, if the proper algorithms are leveraged.

      • Duncan Epping says

        16 October, 2015 at 15:03

        Exactly Joseph, dedupe is definitely more expensive then compression in this case. And we used a non-aggressive algorithm for a reason indeed, nice savings against little to no cost.

        • Rick Boyett says

          16 October, 2015 at 15:32

          Perhaps DeDupe would be most worth if you can specify what you want to dedupe. For instance, a desktop OS in a persistent VDI deployment. Then use compression for all other data. Would something like that be doable, or make any sense?

          • John Nicholson (@Lost_Signal) says

            16 October, 2015 at 21:23

            Rick,

            You can deliver “persistent desktops” while still using Linked clones application layering, and profile virtualization to keep the desktops “lean”. The Horizon suite getting UEM, and App Volumes really reduced the need for true full clone desktops.

  2. Nonce says

    16 October, 2015 at 15:00

    Terribly written if you ask me.

    • Duncan Epping says

      16 October, 2015 at 15:03

      Thanks for your valuable feedback.

    • Paul R. says

      19 October, 2015 at 18:30

      Terribly Commented if you ask me. I found this information valuable myself.

  3. Dave says

    16 October, 2015 at 22:01

    At what level are these settings configurable? Are things like dedupe or compression things that can be applied at storage policy level per VM or is at the disk group level per host?

    • Duncan says

      16 October, 2015 at 23:03

      Configured at a cluster level and on/off per cluster. Dedupe within diskgroup.

  4. Sylvain says

    17 October, 2015 at 14:45

    Hi Duncan,
    One question about the functionalities that this beta will offer: is it possible to cumulate “erasure coding” with Stretched Cluster?
    What I would like to know is if during an outage on one site, we could still benefit from some redundancy on the remaining one (if we have enough hosts of course). Like we would have with traditional arrays being replicated but still providing RAID of some sort on each local array.
    Thanks
    Sylvain

  5. Chris says

    18 October, 2015 at 00:23

    So will the new features be available for the base VSAN or will it require all flash?

    • Joseph Hoot says

      18 October, 2015 at 13:47

      I hope they will allow HYB configurations. Many storage vendors are pushing all flash, which is good to help bring down flash costs and make them more affordable for all of us. However, for us, the IOPS requirements that we have just aren’t anywhere near what flash provides. With a proper cache layer for read prefetching + write buffer with destaging, we are perfectly happy to enjoy the benefits of increased capacity available with the HYB configurations.

  6. nephdotinfo says

    30 October, 2015 at 01:21

    When we spoke about dedup and compression at Vmworld with Christian, I asked if it will be possible to use a GPU card to offload those workload using CUDA. Terrible idea on a second thought, but, what about using SmartNic such as Mellanox/Tillera for inline dedup/compression or Cavium for encryption ? Thoses network card are economically viable _ compared to an Intel X710, a Cavium card is hardly 10% more expensive _ and we will use them for a lot of other usage (DPI, advanced networking, inline snort etc…).
    A Cavium card is typically shipped with 2GB RAM, so sufficent for a small to medium dedup base.

  7. Daniel Boring says

    8 December, 2015 at 20:39

    Very excited for this!!!! C’mon beta pick me 🙂

    • Duncan Epping says

      11 December, 2015 at 14:56

      Do you have an all-flash cluster Daniel?

  8. Vince says

    6 January, 2016 at 19:05

    What if customer already has VSAN implemented with HDD/SSD combo whats the roadmap to take advantage of new compression/de-dupe functionality once the upgrade to new code?

Primary Sidebar

About the author

Duncan Epping is a Chief Technologist in the Office of CTO of the Cloud Platform BU at VMware. He is a VCDX (# 007), the author of the "vSAN Deep Dive", the “vSphere Clustering Technical Deep Dive” series, and the host of the "Unexplored Territory" podcast.

Upcoming Events

May 24th – VMUG Poland
Aug 21st – VMware Explore
Sept – VMUG Slovenia (virtual)
Oct – VMUG Sweden
Nov 6th – VMware Explore
Nov 23rd – UK VMUG
Dec 7th – Swiss German VMUG

Recommended Reads

Sponsors

Want to support Yellow-Bricks? Buy an advert!

Advertisements

Copyright Yellow-Bricks.com © 2023 · Log in