68kMLA Classic Interface
This is a version of the 68kMLA forums for viewing on your favorite old mac. Visitors on modern platforms may prefer the main site.
| Click here to select a new forum. | |
| Netatalk 4.0 - Future-proofing Apple File Sharing | |
Posted by: slipperygrey on 2024-11-24 05:50:32Every now and then, I play around with Netatalk's source code, especially with my attempt to integrate userspace AppleTalk communication which worked with version 2, but requiring some hacks. These hacks were all related to Netatalks frequent use of fork() to generate child processes. While working around problems related to this, I got the impression that forks are somehow regarded as obsolete on macOS. Netatalk even needs some sort of hack to properly run on the latest versions of macOS (OBJC_DISABLE_INITIALIZE_FORK_SAFETY).I was thinking about this forking situation again recently. Pardon me if this discussed this in detail in a different thread and I forgot about it, but I would like to know more about how afpd's process forking is making your effort in writing a userspace AppleTalk stack more difficult. Using fork() has some benefits. The ability to spawn one afpd process for each connected user is arguably elegant, compared to juggling threads, mutex and so on in C code in a safe manner. If we have multi-thread C experts in the audience, please correct me. 🙂 If I don't misremember, you shared a private repo with me last year that had your modified netatalk code, but I cannot find it right now. | |
| Posted by: NJRoadfan on 2024-11-24 08:06:51 The use of fork() is not obsolete on macOS. The issue is Objective-C, which Netatalk doesn't use: https://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html | |
| Posted by: slipperygrey on 2024-11-24 08:26:56 The problem is that we link with several of Apple's ObjC libraries on macOS, which do have this policy restriction. | |
Posted by: robin-fo on 2024-11-24 13:35:40how afpd's process forking is making your effort in writing a userspace AppleTalk stack more difficultThe problem has to do with the AF_APPLETALK sockets which are used to communicate between e.g. afpd and the kernel. In my approach, I replace these with AF_UNIX sockets which works quite well and only requires minor changes to the Netatalk code. On the other end, a userspace program supplies the lower part of the AppleTalk stack (the functionality otherwise implemented in the kernel and atalkd).Now when the application forks, the AF_UNIX socket gets duplicated which is the cause of two problems: 1) This confuses my userspace stack and could lead to inconsistency with data transmission. 2) When you close one of the two sockets, the stack doesn't know if the socket has beed duplicated and therefore closes the corresponding DDP socket not knowing another process still needs it.Perhaps you could find a workaround for both scenarios, but there is still that objc problem mentioned above and the severe debug difficulties on the Mac related to forks. | |
| Posted by: slipperygrey on 2024-11-24 22:45:59 What tooling to you use for debugging on macOS? Even on Linux, following forking processes in gdb is hit and miss in my experience. There might be a trick that I'm missing. Are you not able to use the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" env variable workaround when developing? The article mentions other methods that we could attempt for a more persistent solution. FWIW, Apple's ObjC fork policy effectively breaks Python and Ruby as well. For instance the popular requests library in Python, or the entirety of Ruby on Rails. It's a widespread problem... | |
Posted by: robin-fo on 2024-11-24 23:54:26What tooling to you use for debugging on macOS? Even on Linux, following forking processes in gdb is hit and miss in my experience. There might be a trick that I'm missing.I use lldb on Apple Silicon. | |
Posted by: robin-fo on 2024-11-24 23:56:56Are you not able to use the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" env variable workaround when developing? The article mentions other methods that we could attempt for a more persistent solution.I can (need to) set this, but I think the objc issue is not the reason behind the debugging problems. | |
Posted by: slipperygrey on 2024-11-25 09:30:26I can (need to) set this, but I think the objc issue is not the reason behind the debugging problems.While I don't think this will make a big difference for your issues, this inspired me to apply the other workaround supplied by Apple: To instruct the linker to inject a section in the compiled binary with the contents "__DATA,__objc_fork_ok". I have implemented and tested it in the PR below. It seems to work correctly and removes the need to set the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY" env variable. The code paths behind the scenes is probably exactly the same though. For now I'm injecting the section into afpd, cnid_dbd, cnid_metad, and netatalk. In my testing it only seems like the two first are mandatory, but the latter two also contain forking logic. It might be the case that they don't presently link with any ObjC libraries, hence no assert in Apple code is triggered. On macOS, enable binary section workaround for ObjC fork policy by rdmark · Pull Request #1789 · Netatalk/netatalkRather than setting the OBJC_DISABLE_INITIALIZE_FORK_SAFETY env variable, use the linker to inject a section with __DATA,__objc_fork_ok into the resulting macOS binary. This allows for a more persi...
github.com
| |
| Posted by: NJRoadfan on 2024-11-25 14:29:52 Here is a response direct from DTS regarding the whole thing: https://developer.apple.com/forums/thread/737464?answerId=764686022#764686022 Basically, if you aren't using Cocoa and stick with POSIX calls, you should be fine. What libraries are being linked by afpd? With regards to @robin-fo's socket issue, afpd creates a listening socket in main.c, and then creates an ATP response socket in asp_getsess.c in asp_getsession() when it forks. | |
Posted by: Mk.558 on 2024-11-25 21:22:01@Mk.558 I had the inspiration to fiddle with the webmin module this morning. Can you please give me feedback on this approach to a tabbed interface?  I like it. I'd move Server Status to the front default page (idk...) global config & options to another tab, otherwise, I have no complaint. Someone else might have a better idea though. ...all of the Perl code living in a monolithic index.cgi... I'm looking nervously at that 633,556 characters in v.4.0 of CMN on just index.html...and it's going to get longer... | |
Posted by: slipperygrey on 2024-11-26 22:37:12Here is a response direct from DTS regarding the whole thing: https://developer.apple.com/forums/thread/737464?answerId=764686022#764686022According to otool, the macOS system libraries we link with are: libSystem, libiconv and LDAP.framework. otool -L libatalk.19.dylib
libatalk.19.dylib:
@rpath/libatalk.19.dylib (compatibility version 19.0.0, current version 19.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
/opt/homebrew/opt/mysql/lib/libmysqlclient.24.dylib (compatibility version 24.0.0, current version 24.0.0)
/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
No difference between Sonoma and Sequoia. And, no surprises when inspecting all the other daemon binaries. So, this got me thinking, can't we link with openldap instead of the Apple Framework? And well, we can: brew install openldap
meson setup build -Dwith-ldap-path=/opt/homebrew/opt/openldap
meson compile -C build
otool -L build/libatalk/libatalk.19.dylib
build/libatalk/libatalk.19.dylib:
@rpath/libatalk.19.dylib (compatibility version 19.0.0, current version 19.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
/opt/homebrew/opt/mysql/lib/libmysqlclient.24.dylib (compatibility version 24.0.0, current version 24.0.0)
/opt/homebrew/opt/openldap/lib/libldap.2.dylib (compatibility version 3.0.0, current version 3.200.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
| |
Posted by: slipperygrey on 2024-11-26 23:03:23I like it. I'd move Server Status to the front default page (idk...) global config & options to another tab, otherwise, I have no complaint. Someone else might have a better idea though.Thanks for your feedback! Do you mean moving the "Show Server Status" link to the default tab (File Sharing Services), and keeping the rest under Global Settings, correct? I can see how that would make sense, in that a user would want to check the AFP server's response on the same page that they control said server. The only drawback would be a loss of symmetry of having the four icon links in a single row. 🙂 | |
| Posted by: NJRoadfan on 2024-11-27 04:13:27 Linking OpenLDAP libraries appears to be the correct thing to do. Apple has "LDAP.framework" labelled as "do not use" and clearly that is the case. Folks are having App Store submissions rejected over this. Officially, LDAP appears to be handled by the Open Directory framework. Seems like a common bug in build environment detection of LDAP libraries on mac OS systems. The rest of the libraries should be C only. Maybe it was a good thing that macOS had that pesky fork message? | |
| Posted by: slipperygrey on 2024-11-27 08:42:07 Look at that, Apple’s docs corroborate this assertion. (Table A-1) Good thing we had this conversation! Let me tweak the build system to favor openldap on Darwin. | |
| Posted by: slipperygrey on 2024-11-27 11:27:56 This seems to do the trick. If the Homebrew openldap library is present, it will be used by default now. meson: Favor openldap when building on macOS by rdmark · Pull Request #1792 · Netatalk/netatalkOn macOS when doing find_library('ldap') Meson will find LDAP.Framework libraries which are deprecated by Apple. Ref. https://developer.apple.com/library/archive/documentation/MacOSX/Concep...
github.com
Eventually, I want to remove the "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" workaround from the netatalkd script, but not in the stable 4.0 release series. There may be older macOS versions where we still pull in other non-fork-safe Apple frameworks somehow... | |
| Posted by: slipperygrey on 2024-12-09 10:04:01 @robin-fo Version 4.0.8 is tagged now, that contains the build system fix for not linking with LDAP.framework, if Homebrew openldap is available. The recommended Homebrew packages are dynamically kept up to date in the manual: ...or alternatively, disable LDAP support altogether with "-Dwith-ldap=false" to bypass the issue. Regardless, you should be able to develop against netatalk freely on macOS without worrying about fork safety anymore. 🙂 | |
| Posted by: thecloud on 2024-12-11 20:09:26 I'm wondering if NetBSD still supports AppleTalk in the kernel "out of the box" as the docs suggest. In the past I've had success building netatalk and running afpd on current macOS systems (which provides AFP over TCP only), but couldn't get atalkd running because there was no kernel support for DDP. Today I set up NetBSD 10 in a QEMU 9.0.0 VM so I could try to run netatalk with that support, but it doesn't seem like the support is actually there: NetBSD 10.0 (GENERIC64) #0: Thu Mar 28 08:33:33 UTC 2024 Welcome to NetBSD! arm64# service atalkd onestart Starting atalkd. socket: Address family not supported by protocol family socket: Address family not supported by protocol family atalkd: can't get interfaces, exiting. Setting AppleTalk info with nbprgstr. arm64# pkg_info | grep netatalk netatalk22-2.3.2nb2 Netatalk appletalk file and print services arm64# Looking back through the threads, it seems like they all mention running netatalk on Debian, so that's what I'll be trying next. Did I miss some important configuration step on NetBSD? I did copy the init scripts into /etc/rc.d and edited /etc/services to add DDP mappings per the netatalk docs. | |
| Posted by: slipperygrey on 2024-12-12 01:47:24 @thecloud Looking at your terminal logs, I get the impression that this is an arm64 system. My immediate guess would be that the appletalk kernel module hasn't gotten ported to this architecture. I spun up my amd64 VM with NetBSD 10.0 here and could confirm that atalkd boots up happily (4.0.8 code.) @hauke Do you have any insights here? | |
Posted by: thecloud on 2024-12-12 15:35:55@thecloud Looking at your terminal logs, I get the impression that this is an arm64 system. My immediate guess would be that the appletalk kernel module hasn't gotten ported to this architecture.That's my guess as well. Using the same QEMU virtual machine parameters, a Debian arm64 installation successfully loads the appletalk module. Yay!! Yes, I am running arm64 (aarch64) VMs on Apple Silicon hardware, so QEMU can take advantage of HVF acceleration. Next step: build and run 4.0.8 instead of 2.x on this configuration, so I won't be off-topic. 🙂 | |
| Posted by: thecloud on 2024-12-12 16:34:47 Just to confirm, I pulled down cdn.netbsd.org/pub/NetBSD/NetBSD-10.0/source/sets/syssrc.tgz and had a look at /usr/src/sys/arch/evbarm/conf/GENERIC.common. Sure enough, the option for NETATALK is commented out. If I compare with /usr/src/sys/arch/amd64/conf/GENERIC, the NETATALK option is *not* commented out there. I suppose I can try building my own kernel to see if it works. Edit: built the new GENERIC64 kernel with NETATALK support. It was simple and took about 5 minutes on Apple Silicon. 🙂 atalkd seems to start up and work now. Not sure why it was commented out from that port. | |
| < 6 > |