Tag Archives: backups

Using Azure Cool Blob Storage with Veeam 9.5u3

“Hey, AKSysAdmin. I want to push all our backups to cheap Azure storage. Can you do a proof-of-concept for me and a quick cost write up?”

We are all eagerly awaiting the implementation of Veeam’s Scale-Out Backup Repository Archive Tier functionality in v10. The Archive Tier functionality will allow Veeam customers to leverage to “cheap” cloud storage like AWS’s S3 and Glacier and Azure’s rather hilariously named Cool Blob Storage. In the meantime if you wanted to use Azure Blob Storage right now what are your options?

  • A “middleware” appliance like NetApp’s AltaVault, Microsoft’s StorSimple or a VTL
  • Roll your own IaaS solution in Azure

The first option is pretty straight-forward. You buy an appliance that provides a storage target for your on-prem Veeam Backup and Replication server and send your Backup Copy jobs to that Backup Repository. Once your backups are located there, “magic” happens that handles the hot/warm/cold tier-ing of the data out to Azure as well as the conversion from structured data to unstructured data.

The second option is a little more complicated. You’ll need to spin up an Azure IaaS VM, attach blob storage to it and make it usable to your on-prem Veeam infrastructure.


Before we go too much further we should probably talk about the different blob storage types.

Block Blobs

These are pretty much what they sound like, block based storage of large contiguous files. They work great for things are not accessed via randomized read and writes. The individual blocks stored in each blob are referenced by a BlockID and can be uploaded/modified/downloaded simultaneously, assembled and then committed with a single operation . You can see how well this type of storage lends itself to streaming services where large files are split into smaller pieces and uploaded or downloaded sequentially.  The maximum size (as of writing) for a block blob is about 4.75TBs.

Page Blobs

Page blobs are composed of 512-byte pages optimized for random read and write operations. Changes to the pages require immediate commits unlike block blobs. Page blobs work great for things like virtual disks where some other mechanism is organizing the data inside the blob. Page blobs are used for the underlying storage for Azure IaaS data disks. The maximum size (as of writing) for a page blob is 8TBs.

Azure Blob Storage Tiers: Hot, Cool and Archive

Azure Storage Accounts allow you group all your various pieces of blob storage together for the purposes of management and billing. With Blob and General Purpose v2 Storage Accounts you can elect to use storage tiers. Cool Blob Storage has lower storage costs (and higher access costs) and is intended for things like short-term backup and disaster recovery data. Archive storage has even lower storage costs (and even higher access costs) and is designed for data that can tolerate hours of potential retrieval time. Archive storage is intended for long-term backups, secondary backup storage or data that has archival requirements. In order to read the data in an archive storage the blob needs to be rehyrdated which can take up to 15 hours. Blob size is also a factor in rehyrdation time.

I should mention that the option to have your blobs stored in locally redundant storage (LRS) or globally redundant storage (GRS) exists for all of these flavors.


This is all great but how do I use it?

Well if you went with the first option you break out your wallet for a capital purchase and follow Veeam’s Deployment Guide for AltaVault or vendor equivalent.

The second option is a little more involved. You need to deploy an instance of Veeam’s Cloud Connect for the Enterprise, add some data disks to the resulting Azure Iaas VM, configure them in Windows, setup a Backup Repository using them and finally add the resulting repository to your on-prem install as a Cloud Backup Repository. For the price of the IaaS VM and the underlying storage you now have a cloud-based backup repository using Azure blob storage.

Here’s why you probably don’t want to do this.

Veeam will support Azure Cool Blob storage fairly soon so you have to ask yourself if it makes sense to buy a purpose built “middleware” appliance to bridge the gap. A few years ago it would of been a no-brainer but with more and more backup vendors supporting cloud storage natively it seems like market for these devices will shrink.

The second option has some issues as well. Your freshly created Cloud Backup Repository is backed by Azure IaaS data disks which sit on top of page blob storage. Guess what page blobs don’t support? Storage tiers. If you create a storage account in the cool tier you’ll notice the only container option you have is for block blobs. If you try and add a data disk to your IaaS VM using a blob storage account you get this error:

Not going to work.

What if you setup a Azure File Storage container and utilized it instead of a data disk? Same problem. Only block blob storage supports archiving tiers at this point in time.

What if you just provisioned extra data disks for your VM, and use Storage Spaces and ReFS to get your storage? Well that will sort of work but there many limitations:

  • Data disks are limited to 4TBs
  • Most IaaS VMs only support 15 data disks
  • If you need more than 15 data disks your IaaS VM is going to get really expensive
  • You have to correctly manage and configure a VM with 15 disks using Storage Spaces
  • All your disks are running on page blob storage which is not really that cheap

The “roll-your-own-IaaS” solution will be performance and capacity limited right out of the gate. It will be complicated and potentially brittle and it doesn’t take advantage of the pricing of storage tiers making it rather pointless in my opinion.

Why you still may want to do this

If the backup dataset that you want to archive is fairly small this might still make sense but if that’s the case I would forgo the entire exercise of trying to cram a round peg into a square hole and look very seriously at a DRaaS provider like Iland where you will get so much more than just cloud storage for your backups for what will likely be a competitive price.

Why even if you still want to do this it’s probably not a good idea

Everything is elastic in the cloud except the bill and unless you have an accurate picture of what you really need you might be surprised once you get that bill. There is a bunch of things that are not really accounted for in your traditional on-premise billing structure: IP addresses, data transfer between virtual networks, IOPS limited performance tiers and so on. In short, there is a lot more to doing the cost analysis than just comparing the cost of storage.

Speaking of – let’s take a look at the current storage prices and see if they really are “cheap”. These prices are based on the Azure Storage Overview pricing and are located in the WestUS2 region of Azure ComCloud.

Standard Page Blobs (Unmanaged Disks)

$0.045 per GB N/A per GB $0.06 per GB $0.075 per GB

This also comes with a $0.0005 per 10,000 transactions charge when Standard Page Blobs are attached to a VM as an Unmanaged Disk.


Block Blob Pricing

Hot Cool Archive
First 50 terabyte (TB) / month $0.0184 per GB $0.01 per GB $0.002 per GB
Next 450 TB / Month $0.0177 per GB $0.01 per GB $0.002 per GB
Over 500 TB / Month $0.017 per GB $0.01 per GB $0.002 per GB

There are also some operational charges and data transfer costs

Write Operations* (per 10,000) $0.05 $0.10 $0.10
List and Create Container Operations (per 10,000) $0.05 $0.05 $0.05
Read Operations** (per 10,000) $0.004 $0.01 $5
All other Operations (per 10,000), except Delete, which is free $0.004 $0.004 $0.004
Data Retrieval (per GB) Free $0.01 $0.02
Data Write (per GB) Free $0.0025 Free


To replace our rather small GFS tape set we’d need somewhere north of 100TBs. The first problem is with the limitation requiring us to use page blob backed data disks, we won’t even be able to meet our capacity requirements (4TBs per data disk, 15 data disks per IaaS VM = 60 TBs).

If we put aside the capacity issue, let’s look at a notional cost just for comparison’s sake: 100TBs * 1024 = 102,400 GBs * $0.045 = $4,608 per month. This doesn’t include the cost of the IaaS VM and associated infrastructure you may need (IP addresses, Virtual Networks, Site-to-Site VPN, etc.) nor any of the associated transaction charges.

The storage charge is more than expected since we’re not really using the technology as intended. Block blob storage in the archive tier gets us a much more respectable number: 100TBs * 1024 = 102,400 GBs * $0.002 = $204.8 per month. BUT we need to factor in the cost of some kind of “middleware” appliance to utilize this storage so tack on an extra $40-$60k (it’s hard to pin this cost down since it will come via a VAR so I could be totally off). If we “op-ex” that cost over three years it’s an additional $1388.00 a month bringing your total to $1593.68 per month for “cheap” storage.

OK. Looks like our “cheap” cloud storage may not be as cheap as we thought. Let’s take a look at our on-premise options.

LTO data tapes… personally I loath them but they have their place. Particularly for archiving GFS data sets that are small. A 24 slot LTO-6 tape library like Dell’s TL2000 is around $20k and 40 LTO-6 tapes with a raw capacity of 100TBs (not including compression) comes to about $602 per month over three years.

What about on-premise storage? A Dell MD1400 with 12 10TB 7.2K RPM NLSAS drives is somewhere in the $15-$20k range and brings 80TBs of usable storage in RAID-60 configuration. Allocated out over three years this comes to roughly $555 per month.


Technology choices are rarely simple and no matter how much executive and sales folks push “cloud-first” like it’s some kind of magic bullet, cloud services are a technology like any other with distinct pros and cons, use cases and pitfalls. Getting an accurate picture of how much it will cost to shift a previously capital expense based on-premise service to cloud services is actually a fairly difficult task. There are a tremendous amount of things that you get “included” in your on-premise capital purchases that you have to pay for every month once that service is in the cloud and unless you have a good grasp on them you will get a much bigger bill than you expected. I really recommend SysAdmin1138’s post about the challenges of moving an organization to this new cost model if you are considering any significant cloud infrastructure.

If you want to use Azure Blob Storage right now for Veeam the answer is: You can but it’s not going to work the way you want, it’s probably going to cost more than you think and you’re not really using the technology the way it was intended to be used which is asking for trouble. You could buy some middleware appliance but with Scale-Out Backup Repository Archive Tier functionality on the immediate horizon this sounds like a substantial infrastructure investment that you’re only going to get limited return of business value on. It might make sense to wait.

Finally a little bit of disclaimer. I tried to pull the pricing numbers from old quotes that I have (hence the LTO-6 and LTO-8 tapes) to try and keep the math grounded in something like reality. Your prices may vary wildly and I highly encourage you to compare all the different cost options and spend some time to try to capture all of the potential costs of cloud services that may be hidden (i.e., it’s not just paying for the storage). Cloud services and their pricing are constantly changing too so it’s worth checking with Microsoft to get these numbers from the source.

Until next time, stay frosty.

Scheduling Backups with Veeam Free and PowerShell

Veeam Free Edition is an amazing product. For the low price of absolutely zero you get a whole laundry list of enterprise-grade features: VeeamZip (Full Backups), granular and application-aware restore of items, native tape library support and direct access to NFS-based VM storage using Veeam’s NFS client. One thing that Veeam Free doesn’t include however is a scheduling mechanism. We can fix that with a little bit of PowerShell that we run as a scheduled task.

I have two scripts. The first one loads the Veeam PowerShell Snap-In, connects to the Veeam server, gets a list of virtual machines and then backs them up to a specified destination.


I had Veeam setup on a virtual machine running on the now defunct HumbleLab. One of the disadvantages of this configuration is I don’t have separate storage to move the resulting backup files onto. You could solve this by simply using an external hard drive but I wanted something a little more… cloud-y. I setup Azure Files so I could connect to cheap, redundant and most importantly off-site, off-line storage via SMB3 to store a copy of my lab backups. The biggest downside to this is security. Azure Files is really not designed to be a full featured replacement for a traditional Windows file server. It’s really more of SMB-as-a-Service offering designed to be programmatically accessed by Azure VMs. SMB3 provides transit encryption but you would still probably be better off using a Site-to-Site VPN between your on-prem Veeam server and a Windows file server running as VM in Azure or by using Veeam’s Cloud Connect functionality. There’s also no functionality replacing or replicating NTFS permissions. The entire “security” of your Azure Files SMB share rests in the storage key. This is OK for a lab but probably not OK for production.

Here’s the script that fires off once a week and copies the backups out to Azure Files. For something like my lab it’s a perfect solution.


Until next time, stay frosty!

Veeam, vSphere Tags and PowerCLI – A Practical Use-case

vSphere tags have been around for a while but up until quite recently I had yet find a particularly compelling reason to use them. We are a small enough shop that it just didn’t make sense to use them for categorization purposes with only 200 virtual machines.

This year we hopped on the Veeam bandwagon and while I was designing our backup jobs I found the perfect use-case for vSphere tags. I wanted a mechanism that provided us a degree of flexibility so I could protect different “classes” of virtual machines to the degree appropriate to their criticality but also was automatic since manual processes have a way of not getting done. I also did not want an errant admin to place a virtual machine in the wrong folder or cluster and then have the virtual machine be silently excluded from any backup jobs. The solution was to use vSphere tags and PowerShell with VMware’s PowerCLI module.

If we look at the different methods you can use for targeting source objects in Veeam Backup Jobs, tags seem to offer the most flexibility. Targeting based on clusters works fine as long as you want to include everything on a particular cluster. Targeting based on hosts works as long you don’t expect DRS to move a virtual machine to another host that is not included in your Backup Job. Datastores initially seemed to make the most sense since our different virtual machine “classes” roughly correspond to what datastore they are using (and hence what storage they are on) but not every VM falls cleanly into each datastore “class”. Folders would functionally work the same way here as tags but tags are just a bit cleaner. If I create a folder structure for a new line of business application, I don’t have to go back into each of my Veeam jobs and update their targeting to add the new folders, I just tag the folders appropriately and away I go.

Tags also work great for exclusions. We run a separate vSphere cluster for our SQL workloads primarily for licensing reasons (SQL Datacenter licensing for the win!). I just setup a tag for the whole cluster, use it for targeting VMs for my application-aware Veeam Backup Jobs and also to exclude those virtual machines from the standard Backup Jobs (why back it up twice? Especially when one backup is not transaction-consistent?).

How about checking to see if there are any virtual machines that missed a tag assignment and therefore are not being backed up?

PowerShell to the rescue once again!


Another simple little snippet that will send you an email if you managed to inadvertently place a virtual machine into a location where it is not getting a tag assignment and therefore not getting automatically included in a backup job.

Until next time!

Use PowerCLI to get a list of old snapshots

VMware’s snapshot implementation is pretty handy. A quick click and you have a rollback point for whatever awesomeness your Evil Knievel rockstar developer is about to rollout. There are, as expected, some downsides.

Snapshots are essentially implemented as differencing disks and at the point-in-time you make a snapshot a child .vmdk file is created that tracks any disk changes from that point forward. When you delete the snapshot the changed blocks are merged back into the parent .vmdk. If you decide to rollback all the changes you just revert back to original parent disk which remained unchanged.

There are some performance implications:

  • Disk I/O is impacted for the VM as ESXi has to work a bit harder to write the new changes to child .vmdk file but reference the parent .vmdk file for unchanged blocks.
  • The child .vmdk files grow over time as more and more changes accumulate. The bigger it gets the more the disk I/O impacted. As you can imagine multiple snapshots can certainly impact disk performance.
  • When the snapshot is deleted, the process of merging the changes in the child .vmdk back into the parent .vmdk can be very slow depending on the size of the child .vmdk. Ask my DBA how long it took to delete a snapshot after he restored a 4TB database…

They also don’t make a great backup mechanism in of themselves:

  • The parent and child .vmdk are located on the same storage and if that storage disappears so does the production data (the parent .vmdk) and the “backup” (the child .vmdk) with it.
  • They are not transaction consistent. The hypervisor temporarily freezes I/O in order to take the snapshot without doing some kind of application-aware quiescing (although VMware Tools can quiesce the guest file system and memory).


For all these reasons I like to keep track of what snapshots are out there and how long they have been hanging around. Luckily PowerCLI makes this pretty easy:

This little snippet will connect to your vCenter servers, go through all the VMs, locate any snapshots older than seven days and then send you a nicely formatted email:


If you haven’t invested a little time in learning PowerShell/PowerCLI, I highly recommend you start. It is such an amazing toolset and force multiplier that I cannot imagine working without these days.

Until next time, stay frosty.

Extra credit reading:


World Backup Recovery Testing Day?

Yesterday was apparently the widely celebrated World Backup Day. Just like reality, the party ends some time unless you happen to be Andrew W.K and now you have woken up with a splitting headache, a vague sadness and an insatiable desire for eggs benedict. If installing and configuring a new backup system is an event that brings you joy and revelry like a good party, the monotony of testing the recovery of your backups is your hangover that stretches beyond a good greasy breakfast. I propose that today should thus be World Backup Recovery Testing Day.

There is much guidance out there for anyone who does cursory research on how to design a robust backup system so I think I will save you from my “contributions” to that discussion. As much as I would like to relay my personal experience with backups; I do not think it would be wise to air my dirty laundry this publically. In my general experience, backup systems seems to get done wrong all the time. Why?


Backups? We don’t need those. We have snapshots.

AHAHAHAHAHAHA. Oh. Have fun with that.

I am not sure what it is about backup systems but they never seem to make the radar of leadership. Maybe because they are secondary systems so they do not seem as necessary in the day-to-day operations of the business as production systems. Maybe because they are actually more complicated than they may seem. Maybe because the risk to cost ratio does not seem like a good buy from a business perspective, especially if the person making the business decision does not fully understand the risk.

This really just boils down to the same thing: Technical staff, not communicating the true nature of the problem domain to leadership and/or leadership not adequately listening to the technical staff. Notice the and/or. Communication: it goes both ways. If you are constantly bemoaning the fact that management never listens to you, perhaps you should change the way you are communicating with your management? I am not a manager so I have no idea what the corollary to this is (ed. feel free to comment managers!).

Think about it. If you are not technical, the difference between snapshots and a true backup seem superfluous. Why would you pay more money for a duplicate system? If you do not have an accurate grasp of the risk and the potential consequences why would you authorize additional expenditures?


I am in IT. I work with computers not people.

You do not work with people, you say? Sure you do. Who uses computers? People. Generally people that have some silly business mission related to making money. You best talk to them and figure out what is important to them and not you. The two are not always the same. I see this time and time again. Technical staff implements a great backup system but fails to backup the stuff that is critical to the actual business.

Again. Communication. As a technical person, one database looks more or less identical to another one. I need to talk to the people that actually use that application and get some context, otherwise how would I know which one needs a 15 minute Recovery Time Objective and which one is a legacy application that would be fine with a 72 hour Recovery Time Objective. If it was up to me, I would backup everything, with infinite granularity and infinite retention but despite the delusion that many sysadmin’s labour under they are not god and do not have those powers. Your backup system will have limitations and the business context should inform your decision on how you accommodate those limitations. If you have enough storage to retain all your backups for six weeks or half your backups for 4 weeks and half for 4 months and you just make a choice, maybe you will get lucky and get it right. However, the real world is much more complicated than this scenario it is highly likely you will get it wrong and retain the wrong data for too long at the expensive of the right data. These kind of things can be Resume Generating Events.

My favorite version of this is the dreaded Undocumented Legacy Application that is living on some aging workstation tucked away in a forgotten corner. Maybe it is running the company’s timesheet system (people get pissed if they cannot get paid), maybe it is running the HVAC control software (people get pissed if the building is a nice and frosty 48 degrees Fahrenheit), maybe it is something like SCADA control software (engineers get pissed with water/oil/gas does not flow down the right pipes at the right time, also people may get hurt). How is technical staff going to have backup and recovery plans for things like this if they do not even know they exist in the first place?

It is hard to know if you have done it wrong

In some ways, the difficulty of getting backup systems right is that you only know if you have got it right once the shit hits the fan. Think about the failure mechanism for production systems: You screwed up your storage design – stuff runs slow. You screwed up your firewall ACLs – network traffic is blocked. You screwed up your webserver – the website does not work any more. If there is a technical failure you generally know about it rather quickly. Yes, there are whole sets of integration errors that lie in wait in infrastructure and only rear their ugly head when you hit a corner case but whatever, you cannot test everything. #YOLO #DEVOPS

There is no imminent failure mechanism constantly pushing your backup system towards a better and more robust design since you only really test if you need it. Without this Darwinian IT version of natural selection you generally end up with a substandard design and/or implementation. Furthermore, for some reason backups up here are associated with tapes, and junior positions are associated with tape rotation. This cultural prejudice has slowly morphed into junior positions being placed in charge of the backup system; arguably not the right skillset to be wholly responsible for such a critically important piece of infrastructure.

Sooooo . . . we do a lot of things wrong and it seems the best we can do is a simulated recovery test. That’s why I nominate April 1st as World Backup Recovery Testing Day!


Until next time,

Stay Frosty