macOS Modern API to async copy big files with throughput?

[dafi]

I'm maintaining a very old Obj-c app using the very useful FSCopyObjectAsync and FSMoveObjectAsync APIs to make async copy with progress info (including the throughput)
Now I must drop compatibility with 10.9 and I would to replace these APIs (consider they are no longer available with Swift), but Apple deprecated them without a replacement, any hints?
I found the C API copyfile, it doesn't return the throughput and the fixed read buffer size (64k) is ridicolous because copy is very slow compared to FSCopyObjectAsync.
I would avoid to fork copyfile...
Do you know some other more Apple-like API similar to FSCopyObjectAsync?

PS
Please apologize me for crossposting this question on ADF but there nobody replied to me, I hopw to be more lucky here

 

[Senor Cuete]

According to the developer documentation:​

File Manager​

Interact with files, folders, and volumes.
Deprecated
File Manager is deprecated in macOS 10.8 and later. The Foundation and Core Foundation frameworks provide APIs that you can use to replace File Manager functions in your app. See File System Programming Guide.
To get notifications when a folder changes, use the File System Events API. See File System Events Programming Guide.
To mount a local volume, use the Disk Arbitration API. You can unmount or eject a local volume by calling the unmountVolumeAtURLptions:completionHandler: method. See Disk Arbitration Programming Guide.
For low-level access, use POSIX and BSD APIs.
Framework

and:

Overview​

Warning
Using File Manager APIs on volumes formatted with APFS is strongly discouraged.
Some functions, like FSExchangeObjects, aren't supported in APFS. Other functions may have different in behaviors APFS and HFS+, such as the returned order of directory entries by FSGetCatalogInfoBulk, or how file modification times are calculated by FSGetCatalogInfo. Several File Manager APIs interact with the FSRef structure (also deprecated), which uses 32-bit inode numbers. This interaction can cause performance issues in APFS, which uses 64-bit inode numbers.
Whenever possible, use high-level APIs from Foundation and Core Foundation.

The NSFileManager class might have what you need.

 

[Senor Cuete]

In C the fastest way to read a file is this:

char *inputFileName
FILE *inputFile;
fpos_t fileSize;
char *fileBuffer, *nullTerminator;
size_t bytesRead;

inputFile = fopen(inputFileName, "r");
fseek(inputFile, 0, SEEK_END); //set to file End
fgetpos(inputFile, &fileSize); //filePosition = file size
fseek(inputFile, 0, SEEK_SET); //reset to file start for reading
fileBuffer = malloc(fileSize + 1));  //+1 for  NULL termination - if reading a C text file
bytesRead = fread(fileBuffer, sizeof(char), fileSize, inputFile);  //read whole file into memory
fclose(inputFile);

I know that this isn't what you need but interestingly, because disk drives can read data continuously at a high rate but have a surprisingly limited number of transactions per second, reading files from a disk in a series of operations is much slower that reading a whole file in one operation like this. If you do this you need to put each of these functions in an if() statement to check for errors.

 

[prime17569]

It appears that your best bet is to use NSFileManager in a background thread. The following links mention how you can check the progress of the copy/move operation by comparing the size of the destination file with the size of the source file:

Non-Deprecated Way to Copy Files A… | Apple Developer Forums

developer.apple.com

update progress bar during copy file with NSFileManager

I copy files then my application is finish launching from resource to caches directory. but I want to update my progress bar during copying the files. I copy the file with the code below: -(BOOL)

stackoverflow.com

 

[dafi]

It appears that your best bet is to use NSFileManager in a background thread. The following links mention how you can check the progress of the copy/move operation by comparing the size of the destination file with the size of the source file:

Non-Deprecated Way to Copy Files A… | Apple Developer Forums

developer.apple.com

update progress bar during copy file with NSFileManager

I copy files then my application is finish launching from resource to caches directory. but I want to update my progress bar during copying the files. I copy the file with the code below: -(BOOL)

stackoverflow.com

Thanks but this is a workaround for little or small apps not for real production applications
Looping to check size on external drives can dramatically impact performances, I prefer to fork copyfile

 

[prime17569]

Thanks but this is a workaround for little or small apps not for real production applications
Looping to check size on external drives can dramatically impact performances, I prefer to fork copyfile

Good point.

I wonder what API the Finder uses to check the progress of a copy or move operation...

 

[dafi]

Good point.

I wonder what API the Finder uses to check the progress of a copy or move operation...

Copyfile is easy to use but it internally uses a copy buffer size of 64k, too small to copy big files, the copy time is four or five times greater than FSCopyObjectAsync

 

[chown33]

Maybe find the source for copyfile, edit it to change the buffer size, and compile it directly, instead of using it as a library.

I don't know if copyfile is present in Apple's open sourced content, but it's probably worth looking for. It's listed as a BSD function in the linked man page, so if not found in Apple's code base, maybe search other BSD-based collections.

 

[dafi]

Maybe find the source for copyfile, edit it to change the buffer size, and compile it directly, instead of using it as a library.

I don't know if copyfile is present in Apple's open sourced content, but it's probably worth looking for. It's listed as a BSD function in the linked man page, so if not found in Apple's code base, maybe search other BSD-based collections.

Yes I have already found the source code and changed the buffer size but using a forked version isn't ideal but it works
I'm on mobile and don't find the the link to share with you

 

[chown33]

Good point.

I wonder what API the Finder uses to check the progress of a copy or move operation...

You should be able to do a symbol dump on the Finder's executable, and see what it uses. I wouldn't be surprised at all if it uses FSCopyObjectAsync.

It should also be pretty easy to get a library-dump of the Finder's executable. By "library" I mean both dyld libraries, such as reside in /usr/lib, and frameworks. The cmd to do this used to be otool, but I don't know what it is these days. You could try 'man otool' and see what it says, or just type 'otool' in a Terminal window and see what happens (default with no args is to emit usage text).

Personally, I doubt the Finder has been completely rewritten in Swift. That means it's probably still at least partly written in Objective-C, C, C++, and who knows what else.

 

[prime17569]

You should be able to do a symbol dump on the Finder's executable, and see what it uses. I wouldn't be surprised at all if it uses FSCopyObjectAsync.

It should also be pretty easy to get a library-dump of the Finder's executable. By "library" I mean both dyld libraries, such as reside in /usr/lib, and frameworks. The cmd to do this used to be otool, but I don't know what it is these days. You could try 'man otool' and see what it says, or just type 'otool' in a Terminal window and see what happens (default with no args is to emit usage text).

Personally, I doubt the Finder has been completely rewritten in Swift. That means it's probably still at least partly written in Objective-C, C, C++, and who knows what else.

Finder doesn't appear to use FSCopyObjectAsync, interestingly:

% nm -a /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder | grep "FSCopy"
                 U _FSCopyAliasInfo
                 U _FSCopyDADiskForVolume
                 U _FSCopyAliasInfo
                 U _FSCopyDADiskForVolume

Also, otool -L showed that the Finder seems to link against every framework under the sun, even Carbon:

% otool -L /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder | grep "Carbon"
    /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox (compatibility version 1.0.0, current version 1062.0.0)

 

[chown33]

As I recall, the older Finder used a setuid-root executable named Locum.

On Snow Leopard it resides at:
/System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/Resources/Locum

Finder would run this as a child process, which means it's not necessary for Finder to run an async copy in a local thread. The child process would run as root so it could move or copy things that an unprivileged async copy in the Finder process itself would be unable to accomplish.

I'd forgotten about this "Locum" process until your post reminded me.

 

[dafi]

Great work with otool!
But Finder probably uses private APIs based on FSCopyObjectAsync, I publish on MAS so I prefer to use only public API

The very annoying problem is FSCopyObjectAsync & CO can't be used by swift (they are unavailable) and you must write a Obj-C wrapper then call it from Swift code, really Apple?!??! Really??!?!?