Those days, in one of my company mail threads, a colleague stated: "We have tested the restores, when the oldest backup is an incremental and prior full backups have been deleted. It works!"
This is a pretty funny scenario. So I've decided to do some further investigations to understand what really happens in the background:
- Are the prior Fulls Backups really deleted when there are some Incremental's created after it?
- Do we pay for them?
Let's see..
First the Definitions
There are two backup types available in the Block Volume service:
- Incremental: This backup type includes only the changes since the last backup.
- Full: This backup type includes all changes since the volume was created.
"You can restore a volume from any of your incremental or full volume backups. Both backup types enable you to restore the full volume contents to the point-in-time snapshot of the volume when the backup was taken. You don't need to keep the initial full backup or subsequent incremental backups in the backup chain and restore them in sequence, you only need to keep the backups taken for the times you care about."
"For incremental backups, they are a record of all the changes since the last backup. If the first backup on a volume is created as incremental, it is effectively a full backup. For full backups, they are a record of all the changes since the volume was created."
Source: https://docs.cloud.oracle.com/en-us/iaas/Content/Block/Concepts/blockvolumebackups.htm#backuptype
Oracle also gives an example:
"For example, in a scenario where you create a 16 TB block volume, modify 40 GB on the volume, and then launch a full backup of the volume, upon completion, the volume backup size is 40 GB. If you then modify an additional 4 GB and create an incremental backup, the unique size of the incremental backup will be 4 GB. If the full backup is deleted, the incremental backup will retain the full 44 GB necessary to restore the volume contents."
So what it's saying is:
- Task 1 > Create A Volume of 16 TB
- Task 2 > Change 40GB
- Task 3 > Take a Full Backup > It will be a 40GB backup.
- Taks 4 > Modify 4GB.
- Task 5 > Take a Incr Backup > It will be a 4GB backup.
- Taks 6 > Full is Deleted. > Incremental of Task 5 will be updated to 44G!
There is also another scenario:
"If there was a third incremental backup of non-overlapping blocks, with a size of 1 GB, created after the second incremental backup, and then the full backup is deleted, the third backup would stay at a 1 GB size, and the second incremental backup size would be updated to 44 GB. The blocks are accounted for in the earliest backup that references them."
- Task 1 > Create A Volume of 16 TB
- Task 2 > Change 40GB
- Task 3 > Take a Full Backup > It will be a 40GB backup.
- Taks 4 > Modify 4GB.
- Task 5 > Take a Incr Backup > It will be a 4GB backup.
- Taks 6 > Modify 1GB (non-overlapping).
- Task 7 > Take a Incr Backup > It will be a 1GB backup.
- Task 8 > Full is Deleted. > Incremental of Task 5 will be updated to 44G! Incremental of Task 7 will not change.
But how do I get charged while all those merges are happening?
Trying on my lab
Lets test it.. On this demo, I will create a 50GB volume and create/modify some non-overlapping files on it. Let's play..
First create the mounting points..
[root@demoserv ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdc 8:32 0 50G 0 disk sda 8:0 0 46.6G 0 disk ├─sda2 8:2 0 8G 0 part [SWAP] ├─sda3 8:3 0 38.4G 0 part / └─sda1 8:1 0 200M 0 part /boot/efi [root@demoserv ~]# pvcreate /dev/sdc Physical volume "/dev/sdc" successfully created. [root@demoserv ~]# vgcreate vg_test /dev/sdc Volume group "vg_test" successfully created [root@demoserv ~]# lvcreate -n lv_1 -L 10G vg_test Logical volume "lv_1" created. [root@demoserv ~]# lvcreate -n lv_2 -L 10G vg_test Logical volume "lv_2" created. [root@demoserv ~]# lvcreate -n lv_3 -L 10G vg_test Logical volume "lv_3" created. [root@demoserv ~]# mkfs.ext4 /dev/vg_test/lv_1 [root@demoserv ~]# mkfs.ext4 /dev/vg_test/lv_2 [root@demoserv ~]# mkfs.ext4 /dev/vg_test/lv_3 [root@demoserv ~]# mkdir /test1 /test2 /test3 [root@demoserv ~]# mount /dev/vg_test/lv_1 /test1 [root@demoserv ~]# mount /dev/vg_test/lv_2 /test2 [root@demoserv ~]# mount /dev/vg_test/lv_3 /test3 [root@demoserv ~]#
Note I've created the disk and allocated 3 LVMs of 10GBs each, so I can control if the file that I will be generating is overlapping or not the disk blocks.
Creating a 10G file:
[root@demoserv ~]# head -c 10G </dev/urandom >/test1/sample.txt head: write error: No space left on device head: demoserv error [root@oci360comp ~]# ls -lah /test1/ total 9.7G drwxr-xr-x. 3 root root 4.0K Jun 26 17:02 . dr-xr-xr-x. 21 root root 4.0K Jun 26 15:53 .. drwx------. 2 root root 16K Jun 26 17:00 lost+found -rw-r--r--. 1 root root 9.7G Jun 26 17:33 sample.txt [root@demoserv ~]#
Now I will take a full backup.
Now I will duplicate the file to /test2/ (non-overlapping blocks) and take an incremental:
[root@demoserv ~]# cp -av /test1/sample.txt /test2/ ‘/test1/sample.txt’ -> ‘/test2/sample.txt’ [root@demoserv ~]#
Now I will duplicate once again the file to /test3/ (non-overlapping blocks) and take another incremental:
[root@demoserv ~]# cp -av /test1/sample.txt /test3/ ‘/test1/sample.txt’ -> ‘/test3/sample.txt’ [root@demoserv ~]#
IDs:
- test_vol - ocid1.volume.oc1.iad.abuwcljtdfnhnaiobjna4nm7zrf5fmni6xfnauhsgfisfcyqjhlj7bbdxnfq
- test_vol_full_1 - ocid1.volumebackup.oc1.iad.abuwcljtdoonvmsdjsvtmnj5fm7sm5jyy66vnyyuytq65gippkbpyou56k7a
- test_vol_incr_1 - ocid1.volumebackup.oc1.iad.abuwcljttpbbbieup3m2lvhvnt5ae45qxbgc3kqz7inzsn3v42qfvxofmnkq
- test_vol_incr_2 - ocid1.volumebackup.oc1.iad.abuwcljtjhwizzk4toole7rfombqamnvidtk4ymw42eatznq3okswvxychkq
The playground is ready.. now time to test.
Test 1 - Dropping the FULL
Everything was created on June 26th. 3 hours after I created the backups, I've dropped the first full.
And 3 days later.. the terminated volume was still there.. maybe the Full will never disappear while it's still needed (or locked) by the incrementals.
Test 2 - Dropping the INCR_1
What if I remove now the first incremental? On June 30th, I deleted it.
And after some time, it disappeared..
And after a real long time (9 days later)... test_vol_incr_2 was finally resized to 30G (but the previous full is still appearing):
Note that on July 5th to July 6th (9 days later) at midnight, the test_vol_incr_2 was finally merged with the 2 previous deleted backups. Check this chart from OCI360 tool:
The exactly moment where the transition happened:
Note that 34818000 (total billed) jumped to 71156000 (2x) and then to 10749300 (3x).
Am I paying for it?
If we check the test_vol_full_1 usage:
We were only charged during the period it existed until it got terminated. "Line2" column is the billed amount.
For test_vol_incr_1, we were also only charged for the 10G until it was terminated 3 days later..
And for test_vol_incr_2, as we've seen in the previous section, we were initially being charged for 10G and only for the 30G almost 9 days later.
Conclusion (for June 2020):
When the volumes are terminated, I stop paying them (even though they are needed for post incrementals recovery). I believe this is the expected behaviour even if it leaves charge gap.
My total backup was 30G (10G first full + 2x10G each incr).. however, as I dropped the FULL as soon as all my lab was created, I was not being charged for the 30G in the beginning.
So I payed for 20G for incr_1 and incr_2 from June 26 until June 30 (while both were not terminated). After incr_1 was removed, I was paying only for 10G while my true usage was 30G. Finally, on July 6th, when my backups got merged into the incr_2, I was charged again for the 30G.
This can be better checked on the OCI360 chart below:
