Time Machine - Every Backup is Large

[jdag]

I've noticed recently that my Time Machine backups are all very large (generally 350Mb to 500Mb), even if I do little to nothing in the hour between backup runs.

The program with this is that the time it takes to backup, then run the cleanup process, can be as long as 30 minutes. Meaning the next hourly run is only another 30 minutes away. So my backup process is running close to 50% of the time!

Any thoughts on why the system is determining such a large amount needs to be incrementally backed up each run?

 

[Sciuriware]

I've noticed recently that my Time Machine backups are all very large (generally 350Mb to 500Mb), even if I do little to nothing in the hour between backup runs.

The program with this is that the time it takes to backup, then run the cleanup process, can be as long as 30 minutes. Meaning the next hourly run is only another 30 minutes away. So my backup process is running close to 50% of the time!

Any thoughts on why the system is determining such a large amount needs to be incrementally backed up each run?

In 10.13.1 TM had several problems like ignoring excludes.
Upgrade to 10.13.2 and see if it changes.

I have the feeling that APPLE has been experimenting since 10.11 trying to make TM smarter.
Suddenly TM skipped backups or postponed them depending on the needs or human activity.
I fear that they turned some AI in TM down since 10.13.1
Good luck.
;JOOP!

 

[jdag]

Thanks for the input, but I have been on 10.13.2.

If TM is not properly ignoring my excludes, at least in terms of size determination even if it is not actually backing up those files, it would account for the excess processing. I have my 2 Lightroom Preview files being excluded, and they are huge (over 30gb each).

 

[Sciuriware]

Thanks for the input, but I have been on 10.13.2.

If TM is not properly ignoring my excludes, at least in terms of size determination even if it is not actually backing up those files, it would account for the excess processing. I have my 2 Lightroom Preview files being excluded, and they are huge (over 30gb each).

In that case you probably discovered a new bug.
I tested my backups and I found that, when nothing was done by me, the average backup size
is around 50 Mb, taking some 2 minutes (via UTP).
After installing some application of 1G4 the backup took only 5 minutes.
So I wonder via what medium and to what storage you are making backups.
;JOOP!

 

[jdag]

In that case you probably discovered a new bug.
I tested my backups and I found that, when nothing was done by me, the average backup size
is around 50 Mb, taking some 2 minutes (via UTP).
After installing some application of 1G4 the backup took only 5 minutes.
So I wonder via what medium and to what storage you are making backups.
;JOOP!

I am using a 2TB Time Capsule on a pretty fast network. For instance, right now, I am showing full speed wifi connection at 1300Mbps.

Another possibility I've considered is a corrupted backup file on the Time Capsule. Or, a corrupted file on my laptop itself.

I may just blow away the existing backup set and create a new one. I don't want to reformat the Time Capsule as I have 2 other computers that back up to it (and those are running as expected).

 

[Fishrrman]

OP:

TM is behaving like... well, like it behaves.
That's one of the primary reasons I've never used it.
The backups just grow and grow and grow and grow, backing up the same stuff over and over.

If you'd like to try a different "backup paradigm", I suggest you try either CarbonCopyCloner or SuperDuper. Both are FREE to download and try for 30 days.

A bootable cloned backup won't "grow" as you do incremental backups.
It will always remain the "same size" as the source volume, and always an exact copy of it (well, as "exact" as the last time you backed up incrementally).
And... you can boot from it.
And... you can mount it in the finder, and copy one file, many files, folders, or RE-clone the entire volume BACK TO the source if need be.

 

[Sciuriware]

I am using a 2TB Time Capsule on a pretty fast network. For instance, right now, I am showing full speed wifi connection at 1300Mbps.

Another possibility I've considered is a corrupted backup file on the Time Capsule. Or, a corrupted file on my laptop itself.

I may just blow away the existing backup set and create a new one. I don't want to reformat the Time Capsule as I have 2 other computers that back up to it (and those are running as expected).

DO NOT REFORMAT TC! If you want to get rid of a backup:
- switch off automatic backup in TM
- open the TC by Finder and delete only what is to be deleted.
- You can always test the backup by holding ALT while you open the TM menu right upper corner, choose "Verify"
;JOOP!
[doublepost=1514913729][/doublepost]

OP:

TM is behaving like... well, like it behaves.
That's one of the primary reasons I've never used it.
The backups just grow and grow and grow and grow, backing up the same stuff over and over.

If you'd like to try a different "backup paradigm", I suggest you try either CarbonCopyCloner or SuperDuper. Both are FREE to download and try for 30 days.

A bootable cloned backup won't "grow" as you do incremental backups.
It will always remain the "same size" as the source volume, and always an exact copy of it (well, as "exact" as the last time you backed up incrementally).
And... you can boot from it.
And... you can mount it in the finder, and copy one file, many files, folders, or RE-clone the entire volume BACK TO the source if need be.

Superduper! is a great addition, I use it, but it does not solve 'Jdag's problem.
;JOOP!

 

[jdag]

DO NOT REFORMAT TC! If you want to get rid of a backup:
- switch off automatic backup in TM
- open the TC by Finder and delete only what is to be deleted.
- You can always test the backup by holding ALT while you open the TM menu right upper corner, choose "Verify"
;JOOP!
[doublepost=1514913729][/doublepost]
Superduper! is a great addition, I use it, but it does not solve 'Jdag's problem.
;JOOP!

Thanks for the details.

But can I ask...why are you so against me re-formatting the TC and starting fresh? Yes, I realize the other 2 computers would also need to start fresh, but that's not a big issue to me. Is there something else????

 

[Sciuriware]

Thanks for the details.

But can I ask...why are you so against me re-formatting the TC and starting fresh? Yes, I realize the other 2 computers would also need to start fresh, but that's not a big issue to me. Is there something else????

The TC comes formatted and partitioned: do not touch it.
;JOOP!

 

[jdag]

The TC comes formatted and partitioned: do not touch it.
;JOOP!

I was actually referring to the "Erase Disk" option in the Airport Utility...my fault for using the term "format".

 

[jdag]

This article defines the right difference between all the backup types. and this will explain your concern about time consumption even though you are using incremental backup.
http://searchdatabackup.techtarget....l-differential-and-incremental-forever-backup

I appreciate the info. However, my concern is that the size of my incremental backups is very large, even when the computer is not touched in between 2 backup runs.

My wife's Mac, backing up to the same Time Capsule, will run a backup each hour and the size is showing as about 50-75mb. Mine runs and the size is 350-500mb.

 

[Sciuriware]

I appreciate the info. However, my concern is that the size of my incremental backups is very large, even when the computer is not touched in between 2 backup runs.

My wife's Mac, backing up to the same Time Capsule, will run a backup each hour and the size is showing as about 50-75mb. Mine runs and the size is 350-500mb.

The only thing I can think of: something is 'touching' those files.
You can see that on Terminal level: ls -lc (change time) which will give a different readout than 'ls -l' (modification times).
Even changing the protection of a file will make it a candidate for TM.
;JOOP!

 

[jdag]

That could be helpful...do you know of a way to show ls -lc across the entire drive, including hidden files?

 

[Sciuriware]

From the root down: all files changed in any way since 1 hour:

find / -type f -ctime -1h -print

will give some access errors to closed directories, unless you do it as root (sudo etc.), and ...

find / -type f -ctime -1h -exec ls -lc {} \; > /tmp/output

will give an extensive list in /tmp/output (even showing itself).

;JOOP!

 

[jdag]

From the root down: all files changed in any way since 1 hour:

find / -type f -ctime -1h -print

will give some access errors to closed directories, unless you do it as root (sudo etc.), and ...

find / -type f -ctime -1h -exec ls -lc {} \; > /tmp/output

will give an extensive list in /tmp/output (even showing itself).

;JOOP!

I cannot pretend to understand the output. However, I did notice a bunch of line items from my mailboxes flow by. I was able to scroll back to them, and saw entries for "changed files" that I have not touched. Mailbox folders and subfolders that I've not touched in months/years.

 

[jdag]

1st, thanks for the help Sciuriware and Wilfred jason. You got me sniffing around, and I think I've found something. The culprit does appear to be my mail files. While not scientific, I manually started the Time Machine backup 4 times:

1st run: using Mac normally, plugged in, normal apps running, etc., backup size ~350Mb
2nd run: closed email only, no other changes from 1st run, backup size ~100Mb
3rd run: kept email closed, no other changes from 1st run, backup size ~35Mb
4th run: kept email closed, no other changes from 1st run, backup size ~39Mb
5th run: opened email, no other changes from 1st run, backup size ~255Mb

Since all of my mail is cloud-based anyhow, I think I will delete my 3 mail accounts and recreate them.

 

[Sciuriware]

1st, thanks for the help Sciuriware and Wilfred jason. You got me sniffing around, and I think I've found something. The culprit does appear to be my mail files. While not scientific, I manually started the Time Machine backup 4 times:

1st run: using Mac normally, plugged in, normal apps running, etc., backup size ~350Mb
2nd run: closed email only, no other changes from 1st run, backup size ~100Mb
3rd run: kept email closed, no other changes from 1st run, backup size ~35Mb
4th run: kept email closed, no other changes from 1st run, backup size ~39Mb
5th run: opened email, no other changes from 1st run, backup size ~255Mb

Since all of my mail is cloud-based anyhow, I think I will delete my 3 mail accounts and recreate them.

Congratulations, some problems take a life time to be solved.
;JOOP!

 

[jdag]

Congratulations, some problems take a life time to be solved.
;JOOP!

Your comments were key...the thought of "something is 'touching' those files" got me thinking about what's always open/running (ie - my mail). And then when I did that list of files touched in the past hour and saw some old old old mail folders listed, that allowed me to focus there.

 

[anzio]

For future reference, after a backup, you can run “tmutil compare” in terminal. That will show you changes between the last two backups (or you can compare two other backups against each other using the steps below).

For me, the larger file sizes were because of some changes to iCloud Photo Library downloading original photos of ones I have recently opened on other devices. But the tmutil command should be able to help you verify it was Mail or find out if it was something else.

http://osxdaily.com/2012/01/21/compare-time-machine-backups-tmutil/

 

[jdag]

Excellent info...thanks

 

[jdag]

For future reference, after a backup, you can run “tmutil compare” in terminal. That will show you changes between the last two backups (or you can compare two other backups against each other using the steps below).

For me, the larger file sizes were because of some changes to iCloud Photo Library downloading original photos of ones I have recently opened on other devices. But the tmutil command should be able to help you verify it was Mail or find out if it was something else.

http://osxdaily.com/2012/01/21/compare-time-machine-backups-tmutil/

I am seeing a lot of files that are part of the backups that I would not imagine should be. Things from fonts to MS Excel application.

 

[Sciuriware]

For future reference, after a backup, you can run “tmutil compare” in terminal. That will show you changes between the last two backups (or you can compare two other backups against each other using the steps below).

For me, the larger file sizes were because of some changes to iCloud Photo Library downloading original photos of ones I have recently opened on other devices. But the tmutil command should be able to help you verify it was Mail or find out if it was something else.

http://osxdaily.com/2012/01/21/compare-time-machine-backups-tmutil/

Something happened on the way to the forum (pun intended). I wanted to write:
tmutil compare: this is a fruitful thread.
;JOOP!

 

[Elotloteo]

Did you manage to get your backups smaller at the end?

Mine is about 1.7GB on average, sometimes goes up to 3GB, even when I change very little in the files. Also my changes are mainly in small text files as I write code. When one backup TM finally manages to finish another one is already due, constantly eating-up my ageing MBPro resources.

My mailboxes are quite large and old with numerous accounts, have been bounced around from Mac to Mac via TM backup/restore since 2010.

I was wondering if it helped to delete the mailboxes and set them up from scratch, as it would be a big job in my case I'm afraid. I don't use any MS apps tho ;-)

1st, thanks for the help Sciuriware and Wilfred jason. You got me sniffing around, and I think I've found something. The culprit does appear to be my mail files. While not scientific, I manually started the Time Machine backup 4 times:

1st run: using Mac normally, plugged in, normal apps running, etc., backup size ~350Mb
2nd run: closed email only, no other changes from 1st run, backup size ~100Mb
3rd run: kept email closed, no other changes from 1st run, backup size ~35Mb
4th run: kept email closed, no other changes from 1st run, backup size ~39Mb
5th run: opened email, no other changes from 1st run, backup size ~255Mb

Since all of my mail is cloud-based anyhow, I think I will delete my 3 mail accounts and recreate them.

 

[jdag]

Did you manage to get your backups smaller at the end?

Mine is about 1.7GB on average, sometimes goes up to 3GB, even when I change very little in the files. Also my changes are mainly in small text files as I write code. When one backup TM finally manages to finish another one is already due, constantly eating-up my ageing MBPro resources.

My mailboxes are quite large and old with numerous accounts, have been bounced around from Mac to Mac via TM backup/restore since 2010.

I was wondering if it helped to delete the mailboxes and set them up from scratch, as it would be a big job in my case I'm afraid. I don't use any MS apps tho ;-)

My problem goes back ~18 months, and I have a new Mac since. But I would say my average TM backup is probably about 250mb now.

 

[jbarley]

When one backup TM finally manages to finish another one is already due, constantly eating-up my ageing MBPro resources

At least this part of your problem could be managed by using a TM scheduling app such as "TimeMachineEditor"
https://www.macupdate.com/app/mac/26704/timemachineeditor