Re: Patch: Add launchd Support

Hackers,

In Mac OS X 10.10 “Yosemite,” Apple removed SystemStarter, upon which our OS X start script has relied since 2007. So here is a patch that adds support for its replacement, launchd. It includes 7 day log rotation like the old script did. The install script still prefers the SystemStarter approach for older versions of the OS, for the sake of easier backward compatibility. We could change that if we wanted, since launchd has been part of the OS for around a decade.

"David E. Wheeler" <david(at)justatheory(dot)com> writes:
> In Mac OS X 10.10 Yosemite, Apple removed SystemStarter, upon which our OS X start script has relied since 2007. So here is a patch that adds support for its replacement, launchd. It includes 7 day log rotation like the old script did. The install script still prefers the SystemStarter approach for older versions of the OS, for the sake of easier backward compatibility. We could change that if we wanted, since launchd has been part of the OS for around a decade.

(1) I'd vote for just removing the SystemStarter stuff: it complicates
understanding what's happening, to no very good end. We can easily
check that the launchd way works back to whatever we think our oldest
supported OS X release is. (10.4.x according to the buildfarm, at least;
and I think SystemStarter was deprecated even then ...)

(2) AFAICS, this .plist file doesn't do anything about launchd's habit of
not waiting for the network to come up. See my comments in today's thread
in -general:
http://www.postgresql.org/message-id/1239.1413823243@sss.pgh.pa.us

(3) I don't think you want Disabled = true.

(4) I'm suspicious of all the -c arguments in the .plist file. In general
I'm not a fan of specifying GUCs on the postmaster command line; that
makes it impossible to override their values via "normal" methods like
postgresql.conf or ALTER SYSTEM.

(5) According to the launchd.plist man page, there are options for
redirecting stdout and stderr to someplace useful. It might be worth
exercising those ...

regards, tom lane

On 10/20/14, 5:59 PM, David E. Wheeler wrote:
> In Mac OS X 10.10 “Yosemite,” Apple removed SystemStarter, upon which our OS X start script has relied since 2007. So here is a patch that adds support for its replacement, launchd. It includes 7 day log rotation like the old script did. The install script still prefers the SystemStarter approach for older versions of the OS, for the sake of easier backward compatibility. We could change that if we wanted, since launchd has been part of the OS for around a decade.

You're enabling POSTGRESQL in /etc/hostconfig before any of the files are copied over... what happens if we puke before the files get copied? Would it be better to enable after the scripts are in place?

BTW, Mavericks has a comment that /etc/hostconfig is going away, but google isn't telling me what's replacing it...
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

On Oct 20, 2014, at 4:36 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> (1) I'd vote for just removing the SystemStarter stuff: it complicates
> understanding what's happening, to no very good end. We can easily
> check that the launchd way works back to whatever we think our oldest
> supported OS X release is. (10.4.x according to the buildfarm, at least;
> and I think SystemStarter was deprecated even then ...)

Okay. Might have to use OnDemand instead of KeepAlive on 10.4. The former was deprecated in 10.5, but I’m not sure when the former was added.

> (2) AFAICS, this .plist file doesn't do anything about launchd's habit of
> not waiting for the network to come up. See my comments in today's thread
> in -general:
> http://www.postgresql.org/message-id/1239.1413823243@sss.pgh.pa.us

Ha! How funny you posted a call for a patch today. I didn’t see that, just needed to get it working today myself.

Anyway, I knew there was a reason I didn’t bother with this years ago: launchd does not support dependencies. From the launchd.plist(5)

> DEPENDENCIES
> Unlike many bootstrapping daemons, launchd has no explicit dependency model.
> Interdependencies are expected to be solved through the use of IPC. It is
> therefore in the best interest of a job developer who expects dependents to
> define all of the sockets in the configuration file. This has the added ben-
> efit of making it possible to start the job based on demand instead of imme-
> diately. launchd will continue to place as many restrictions on jobs that
> do not conform to this model as possible.

This another reason not to use KeepAlive, I guess. OnDemand is supposed to fire up a job only when it’s needed. No idea what that means. We might be able to put something in LaunchEvents that gets it to fire when the network launches, but documentation is hella thin (and may only be supported on Yosemite, where there are a bunch of poorly-documented launchd changes).

> (3) I don't think you want Disabled = true.

It’s the default. When you run `launchctl load -w` it overrides it to false in its database. I’m fine to have it be less opaque, though.

> (4) I'm suspicious of all the -c arguments in the .plist file. In general
> I'm not a fan of specifying GUCs on the postmaster command line; that
> makes it impossible to override their values via "normal" methods like
> postgresql.conf or ALTER SYSTEM.

Yeah, I am okay with removing those; they weren’t in the SystemStarter script. Was the only way to replicate the log rotation stuff, but probably best not to do that in the start script, anyway.

> (5) According to the launchd.plist man page, there are options for
> redirecting stdout and stderr to someplace useful. It might be worth
> exercising those ...

Suggestions?

Best,

David

On Oct 20, 2014, at 4:58 PM, Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> wrote:

> You're enabling POSTGRESQL in /etc/hostconfig before any of the files are copied over... what happens if we puke before the files get copied? Would it be better to enable after the scripts are in place?

That code was there; I just indented it in an if/then block.

> BTW, Mavericks has a comment that /etc/hostconfig is going away, but google isn't telling me what's replacing it...

launchd.

Best,

David

"David E. Wheeler" <david(at)justatheory(dot)com> writes:
> On Oct 20, 2014, at 4:36 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> (1) I'd vote for just removing the SystemStarter stuff: it complicates
>> understanding what's happening, to no very good end. We can easily
>> check that the launchd way works back to whatever we think our oldest
>> supported OS X release is. (10.4.x according to the buildfarm, at least;
>> and I think SystemStarter was deprecated even then ...)

> Okay. Might have to use OnDemand instead of KeepAlive on 10.4. The former was deprecated in 10.5, but Im not sure when the former was added.

[ looks ... ] Yeah, there's no mention of KeepAlive in 10.4's
launchd.plist man page. It does have a convenient example
saying that OnDemand = false does what we want:

The following XML Property List simply keeps "exampled" running continu-
ously:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
http://www.apple.com/DTDs/PropertyList-1.0.dtd >
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.exampled</string>
<key>ProgramArguments</key>
<array>
<string>exampled</string>
</array>
<key>OnDemand</key>
<false/>
</dict>
</plist>

>> (5) According to the launchd.plist man page, there are options for
>> redirecting stdout and stderr to someplace useful. It might be worth
>> exercising those ...

> Suggestions?

I'd just drop them into files in the data directory; we're still going
to recommend that people use the logging_collector, so this is just a
stopgap to collect startup errors.

regards, tom lane

On Oct 20, 2014, at 5:17 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> [ looks ... ] Yeah, there's no mention of KeepAlive in 10.4's
> launchd.plist man page. It does have a convenient example
> saying that OnDemand = false does what we want:

Yeah, let’s see if we can cover both.

> I'd just drop them into files in the data directory; we're still going
> to recommend that people use the logging_collector, so this is just a
> stopgap to collect startup errors.

How about this?

<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>Label</key>
<string>org.postgresql.postgresql</string>
<key>UserName</key>
<string>postgres</string>
<key>GroupName</key>
<string>postgres</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/pgsql/bin/postgres</string>
<string>-D</string>
<string>/usr/local/pgsql/data</string>
</array>
<key>StandardOutPath</key>
<string>/usr/local/pgsql/data/launchd.log</string>
<key>StandardErrorPath</key>
<string>/usr/local/pgsql/data/launchd.log</string>
<key>OnDemand</key><!-- OS X 10.4 -->
<false/>
<key>KeepAlive</key><!-- OS X 10.5+ -->
<true/>
</dict>
</plist>

No fix for the networking issue, of course.

Best,

David

On Oct 20, 2014, at 5:03 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
> This another reason not to use KeepAlive, I guess. OnDemand is supposed to fire up a job only when it’s needed. No idea what that means.

I think the idea of OnDemand is for launchd items to act a bit like inetd does: launchd creates the listening socket (or mach port or file-change notification) on the port specified in the plist, and only starts the process when someone tries to connect to it. This might not be a terrible thing to support, but it would require more changes throughout postgres (postmaster would have to check in with launchd at start time to get the listen socket; ports and socket paths would no longer be specifiable in postgres’ config, etc) and I’m skeptical that it’d be worth the work without a more concrete motivation.

Apple has published their changes to Postgres (since they ship it in recent versions of OSX) here, fwiw, including the launchd plist they use: http://www.opensource.apple.com/source/PostgreSQL/

One thing I noticed is that Apple also used the label “org.postgres.postgres” for their launchd job. I don’t know if that will collide in some way with a second job with the same label. Launchctl load/unload takes a pathname, not a job label, so I don’t think it’d be a problem unless you actually do want to run both copies of postgres at the same time.

MacPorts also has a launchd job for their postgresql port, which invokes daemondo, which invokes a wrapper script, which invokes postgres. I’m not sure why they did it that way.

> 2) AFAICS, this .plist file doesn't do anything about launchd's habit of not waiting for the network to come up.

Have you experimented with this setting?:

<key>KeepAlive</key>
<dict><key>NetworkState</key><true/></dict>

The launchd.plist man page claims that if you set that key in the sub-dictionary:
> If true, the job will be kept alive as long as the network is up, where up is defined as at least one non-loopback interface being up and having IPv4 or IPv6 addresses assigned to them. If false, the job will be kept alive in the inverse condition.

On the other hand, it’s not unreasonable to have postgres running on a machine with only a loopback interface, depending on the use.

> We might be able to put something in LaunchEvents that gets it to fire when the network launches, but documentation is hella thin (and may only be supported on Yosemite, where there are a bunch of poorly-documented launchd changes).

If one were desperate enough... it’s possible to dig through the launchd sources to make up for the gaps in the documentation (also on opensource.apple.com; there used to be a community-ish site for it at macosforge.org as well). It’s rough going, though, IIRC.

>> (3) I don't think you want Disabled = true.
>
> It’s the default. When you run `launchctl load -w` it overrides it to false in its database. I’m fine to have it be less opaque, though.

Yes, AFAICT it’s conventional to specify Disabled=true in a launchd plist and use launchctl to enable the item.

> BTW, Mavericks has a comment that /etc/hostconfig is going away, but google isn't telling me what's replacing it…

I think that’s been “going away” for a decade now.

On Tue, Oct 21, 2014 at 3:53 AM, Wim Lewis <wiml(at)omnigroup(dot)com> wrote:
> I think the idea of OnDemand is for launchd items to act a bit like inetd
> does: launchd creates the listening socket (or mach port or file-change
> notification) on the port specified in the plist, and only starts the
> process when someone tries to connect to it. This might not be a terrible
> thing to support, but it would require more changes throughout postgres
> (postmaster would have to check in with launchd at start time to get the
> listen socket; ports and socket paths would no longer be specifiable in
> postgres’ config, etc) and I’m skeptical that it’d be worth the work
> without a more concrete motivation.

systemd socket activation on Linux works pretty much the same way. If
this will ever be implemented, it's most reasonable to tackle both
launchd and systemd together.

Regards,
Marti

On Oct21, 2014, at 02:53 , Wim Lewis <wiml(at)omnigroup(dot)com> wrote:
>
>> 2) AFAICS, this .plist file doesn't do anything about launchd's habit of
>> not waiting for the network to come up.

>> If true, the job will be kept alive as long as the network is up, where
>> up is defined as at least one non-loopback interface being up and having
>> IPv4 or IPv6 addresses assigned to them. If false, the job will be kept
>> alive in the inverse condition.
>
> On the other hand, it’s not unreasonable to have postgres running on a
> machine with only a loopback interface, depending on the use.

This, I think, shows the gist of the problem -- "Networking is up" is not
a clearly defined stated on modern desktop machines. Interfaces can come and
go all the time, and even things like ethernet may take a while to come up
during boot, or even depend on a user logging in if something like 802.11X
is used.

>> We might be able to put something in LaunchEvents that gets it to fire
>> when the network launches, but documentation is hella thin (and may only
>> be supported on Yosemite, where there are a bunch of poorly-documented
>> launchd changes).
>
> If one were desperate enough... it’s possible to dig through the launchd
> sources to make up for the gaps in the documentation (also on
> opensource.apple.com; there used to be a community-ish site for it at
> macosforge.org as well). It’s rough going, though, IIRC.

The correct way to deal with this would be to teach postgres to react to
connectivity changes dynamically. During postmaster launch, we'd ignore all
"listen" entries which mention non-bindable addresses. If the network
configuration changes, we'd rescan the "listen" entries, and attempt to
create any missing sockets.

OS X provides an API (via the SystemConfiguration framework IIRC) which allows
to subscribe to networking-changed notifications. I believe systemd provides
something similar via dbus or some such.

Whether or not doing this is worth the effort though, I don't know. I guess
it depends on whether postgres on OS X is actually used for productions --
my guess is that most installations are for development and testing purposes,
but maybe my view's biased there.

>>> (3) I don't think you want Disabled = true.
>>
>> It’s the default. When you run `launchctl load -w` it overrides it to false in
>> its database. I’m fine to have it be less opaque, though.
>
> Yes, AFAICT it’s conventional to specify Disabled=true in a launchd plist and
> use launchctl to enable the item.

Yup, macports also has Disabled=true in the launchd items they install.

best regards,
Florian Pflug

On 10/20/14 8:53 PM, Wim Lewis wrote:
> Apple has published their changes to Postgres (since they ship it in recent versions of OSX) here, fwiw, including the launchd plist they use: http://www.opensource.apple.com/source/PostgreSQL/
>
> One thing I noticed is that Apple also used the label “org.postgres.postgres” for their launchd job. I don’t know if that will collide in some way with a second job with the same label. Launchctl load/unload takes a pathname, not a job label, so I don’t think it’d be a problem unless you actually do want to run both copies of postgres at the same time.

I have a file /System/Library/LaunchAgents/com.apple.postgres.plist that
uses the label com.apple.postgres instead.

> MacPorts also has a launchd job for their postgresql port, which invokes daemondo, which invokes a wrapper script, which invokes postgres. I’m not sure why they did it that way.

Probably because it's apparently really difficult to do it in a
different way.

>> 2) AFAICS, this .plist file doesn't do anything about launchd's habit of not waiting for the network to come up.
>
> Have you experimented with this setting?:
>
> <key>KeepAlive</key>
> <dict><key>NetworkState</key><true/></dict>
>
> The launchd.plist man page claims that if you set that key in the sub-dictionary:
>> If true, the job will be kept alive as long as the network is up, where up is defined as at least one non-loopback interface being up and having IPv4 or IPv6 addresses assigned to them. If false, the job will be kept alive in the inverse condition.

My launchd.plist man page says:

NetworkState <boolean>
This key is no longer implemented as it never acted how most users
expected.

On Mon, Oct 20, 2014 at 03:59:01PM -0700, David E. Wheeler wrote:
> Hackers,
>
> In Mac OS X 10.10 “Yosemite,” Apple removed SystemStarter, upon
> which our OS X start script has relied since 2007. So here is a patch
> that adds support for its replacement, launchd. It includes 7 day log
> rotation like the old script did. The install script still prefers
> the SystemStarter approach for older versions of the OS, for the sake
> of easier backward compatibility. We could change that if we wanted,
> since launchd has been part of the OS for around a decade.

Where are we on this?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +

On Mar 19, 2015, at 8:12 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> Where are we on this?

I suggested this plist:

<dict>
<key>Disabled</key>
<false/>
<key>Label</key>
<string>org.postgresql.postgresql</string>
<key>UserName</key>
<string>postgres</string>
<key>GroupName</key>
<string>postgres</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/pgsql/bin/postgres</string>
<string>-D</string>
<string>/usr/local/pgsql/data</string>
</array>
<key>StandardOutPath</key>
<string>/usr/local/pgsql/data/launchd.log</string>
<key>StandardErrorPath</key>
<string>/usr/local/pgsql/data/launchd.log</string>
<key>OnDemand</key><!-- OS X 10.4 -->
<false/>
<key>KeepAlive</key><!-- OS X 10.5+ -->
<true/>
</dict>
</plist>

No one replied. Want a new patch with that?

David

On 3/20/15 6:11 PM, David E. Wheeler wrote:
> <key>ProgramArguments</key>
> <array>
> <string>/usr/local/pgsql/bin/postgres</string>
> <string>-D</string>
> <string>/usr/local/pgsql/data</string>
> </array>

Hrm, would /var/db/postgres be better? I'm not sure if the stuff Apple
does with /private/ would cause problems though. (In any case, I think
postgres is better than pgsql.)

> <key>StandardOutPath</key>
> <string>/usr/local/pgsql/data/launchd.log</string>
> <key>StandardErrorPath</key>
> <string>/usr/local/pgsql/data/launchd.log</string>

Wouldn't /var/log be better?
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com

On Mar 20, 2015, at 4:21 PM, Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> wrote:
>
> On 3/20/15 6:11 PM, David E. Wheeler wrote:
>> <key>ProgramArguments</key>
>> <array>
>> <string>/usr/local/pgsql/bin/postgres</string>
>> <string>-D</string>
>> <string>/usr/local/pgsql/data</string>
>> </array>
>
> Hrm, would /var/db/postgres be better? I'm not sure if the stuff Apple does with /private/ would cause problems though. (In any case, I think postgres is better than pgsql.)
>
>> <key>StandardOutPath</key>
>> <string>/usr/local/pgsql/data/launchd.log</string>
>> <key>StandardErrorPath</key>
>> <string>/usr/local/pgsql/data/launchd.log</string>
>
> Wouldn't /var/log be better?

/usr/local/pgsql has been the standard install location for the PostgreSQL core distribution for as long as I can remember, including on OS X. Our original OS X startup script refers to it. I figure it’s best to keep it consistent.

Best,

David

On Mar 20, 2015, at 4:11 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:

> No one replied. Want a new patch with that?

Here it is.

Best,

David