Re: PG_DUMP very slow because of STDOUT ??

Wait, now, here I see some correlation! Yes, it seems to be the memory! When I start my COPY-to-STDOUT experiment I had some 2000 MByte free (well ,the server has 24 GByte ... maybe other PostgreSQL processes used up the rest). Then, I could monitor via "ll -h" how the file nicely growed (obviously no congestion), and in parallel see, how "free -m" the "free" memory went down. Then, it reached a level below 192 MByte, and congestion began. Now it is going back and forth around 118-122-130 ... Obviously the STDOUT thing went out of some memory resources.
Now I "only" who and why is running out, and how I can prevent that. Could there be some extremely big STDOUT buffering in play ????

I will need to do some more investigation. And thanks for your very good pointers which help me to narrow the focus down!

Andras Fabian

-----Ursprüngliche Nachricht-----
Von: Craig Ringer [mailto:craig(at)postnewspapers(dot)com(dot)au]
Gesendet: Dienstag, 13. Juli 2010 12:17
An: Andras Fabian
Cc: pgsql-general(at)postgresql(dot)org
Betreff: Re: AW: AW: [GENERAL] PG_DUMP very slow because of STDOUT ??

On 13/07/10 17:18, Andras Fabian wrote:

> But still no definitive clue about the reasons. What is also quite interesting is, that when I start my COPY-to-STDOUT experiment, it is running quite fast in the beginning. Sometimes up to 400 Mbytes, sometimes up to 1.4 GBytes (I didn't find a real reason which I could be correlated to this) ... and then, suddenly it begins to stall. From there on, it only advances slowly with all the congestion_wait going on ... Hmm, maybe it has really something to do with the state of the memory ... (this would go well with the fact, that a freshly rebooted server is not having the problem in the beginning).

Is there any chance something is buffering the output of the COPY ... TO
STDOUT? Whether it's a pipe/socket buffer that grows to a certain limit,
then begins to be limited by the ability of the receiver to empty it, or
whether it's the receiver buffering the whole COPY in memory before
writing it out, I don't know.

Do you see anything growing in memory use on the system while the COPY
is in progress? Any of the memory use stats in "free -m" changing"?

--
Craig Ringer