[CHECKER] 4 memory leaks in Postgresql 7.4.2

Sorry if this is a repeat. I checked the mailing list archive and this
message didn't show up, so I am reposting it.

------------------------------------------------------------------------
----------------------------------------

Hi,

I'm from the Stanford Metacompilation research group where we use
static analysis to find bugs. I'm trying a new technique, so I would
appreciate feedback on these error reports.

This technique tries to infer API roles automatically from the code.
In particular, I am trying to check for ownership properties of
functions (i.e., which functions return ownership of a pointer, which
ones claim it, etc.). This allows us to potentially find memory leaks
that we didn't check for before because we didn't know what functions
allocated memory or acquired resources.

The tool is still in the early stages, so the bugs reported may be
false positives. I have applied it to Postgresql 7.4.2. Included
below are 4 error reports that I believe to be memory leaks.

Any feedback or confirmation of bugs would be appreciated.

Best,
Ted Kremenek

------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
File where bug occurred:
postgresql-7.4.2/src/backend/commands/vacuumlazy.c

Diagnosis:

(1) palloc0 returns memory at line 141 and address is stored in
vacrelstats
(2) vaccrelstats goes out of scope at the end of the function;
leaking the memory

131 BlockNumber possibly_freeable;
132
133 if (vacstmt->verbose)
134 elevel = INFO;
135 else
136 elevel = DEBUG2;
137
138 vacuum_set_xid_limits(vacstmt, onerel->rd_rel->relisshared,
139 &OldestXmin, &FreezeLimit);
140

---> vacrelstats is allocated

141 vacrelstats = (LVRelStats *) palloc0(sizeof(LVRelStats));
142
143 /* Set threshold for interesting free space = average request size
*/
144 /* XXX should we scale it up or down? Adjust vacuum.c too, if so
*/
145 vacrelstats->threshold = GetAvgFSMRequestSize(&onerel->rd_node);
146
147 /* Open all indexes of the relation */
148 vac_open_indexes(onerel, &nindexes, &Irel);
149 hasindex = (nindexes > 0);
150
151 /* Do the vacuuming */
152 lazy_scan_heap(onerel, vacrelstats, Irel, nindexes);
153
154 /* Done with indexes */
155 vac_close_indexes(nindexes, Irel);
156
157 /*
158 * Optionally truncate the relation.
159 *
160 * Don't even think about it unless we have a shot at releasing a
goodly
161 * number of pages. Otherwise, the time taken isn't worth it.
162 */
163 possibly_freeable = vacrelstats->rel_pages -
vacrelstats->nonempty_pages;
164 if (possibly_freeable >= REL_TRUNCATE_MINIMUM ||
165 possibly_freeable >= vacrelstats->rel_pages /
REL_TRUNCATE_FRACTION)
166 lazy_truncate_heap(onerel, vacrelstats);
167
168 /* Update shared free space map with final free space info */
169 lazy_update_fsm(onerel, vacrelstats);
170
171 /* Update statistics in pg_class */
172 vac_update_relstats(RelationGetRelid(onerel),
vacrelstats->rel_pages,
173 vacrelstats->rel_tuples, hasindex);

--> goes out of scope

174 }

------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak on error path (dtype != DTK_DELTA)
File where bug occurred:
postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

Diagnosis:

(1) memory is allocated on line 778 by calling pgtypes_alloc; stored
in result
(2) leaked on error path on line 785

File where bug occurred:
/home/kremenek/oses_src/postgresql-7.4.2/src/interfaces/ecpg/
pgtypeslib/interval.c

768 return NULL;
769 }
770
771 if ((ParseDateTime(str, lowstr, field, ftype, MAXDATEFIELDS, &nf,
ptr) != 0)
772 || (DecodeInterval(field, ftype, nf, &dtype, tm, &fsec) != 0))
773 {
774 errno = PGTYPES_INTVL_BAD_INTERVAL;
775 return NULL;
776 }
777

---> memory is allocated

778 result = (interval *) pgtypes_alloc(sizeof(interval));
779 if (!result)
780 return NULL;
781
782 if (dtype != DTK_DELTA)
783 {
784 errno = PGTYPES_INTVL_BAD_INTERVAL;
785 return NULL;

---> memory is leaked on error path

786 }
787
788 if (tm2interval(tm, fsec, result) != 0)

------------------------------------------------------------------------
------------------------------------------

[BUG] difficult to analyze, but it appears that subquery is memory
leaked
File where bug occurred:
postgresql-7.4.2/src/backend/optimizer/plan/subselect.c

Diagnosis:

(1) copyObject() appears to return newly allocated memory on line 261
(2) subquery is passed to subquery_planner on line 296, but within
that
function subquery itself never appears to be freed
(3) subquery has a field dereference on line 301, but afterwards
never
appears to be referenced
(4) pointer goes out of scope at end of function

This one I am less confident about, since it is difficult for me to
reason about the inter-procedural semantics.

File where bug occurred:
/home/kremenek/oses_src/postgresql-7.4.2/src/backend/optimizer/plan/
subselect.c

252 int paramid;
253 List *lst;
254 Node *result;
255
256 /*
257 * Copy the source Query node. This is a quick and dirty kluge to
258 * resolve the fact that the parser can generate trees with
multiple
259 * links to the same sub-Query node, but the planner wants to
scribble
260 * on the Query. Try to clean this up when we do querytree
redesign...
261 */
--->
262 subquery = (Query *) copyObject(subquery);
263
264 /*
265 * For an EXISTS subplan, tell lower-level planner to expect that
only
266 * the first tuple will be retrieved. For ALL and ANY subplans, we
267 * will be able to stop evaluating if the test condition fails, so
268 * very often not all the tuples will be retrieved; for lack of a
269 * better idea, specify 50% retrieval. For EXPR and MULTIEXPR
270 * subplans, use default behavior (we're only expecting one row
out,
271 * anyway).
272 *
273 * NOTE: if you change these numbers, also change
cost_qual_eval_walker()
274 * in path/costsize.c.
275 *
276 * XXX If an ALL/ANY subplan is uncorrelated, we may decide to
hash or
277 * materialize its result below. In that case it would've been
better
278 * to specify full retrieval. At present, however, we can only
detect
279 * correlation or lack of it after we've made the subplan :-(.
Perhaps
280 * detection of correlation should be done as a separate step.
281 * Meanwhile, we don't want to be too optimistic about the
percentage
282 * of tuples retrieved, for fear of selecting a plan that's bad for
283 * the materialization case.
284 */
285 if (slink->subLinkType == EXISTS_SUBLINK)
286 tuple_fraction = 1.0; /* just like a LIMIT 1 */
287 else if (slink->subLinkType == ALL_SUBLINK ||
288 slink->subLinkType == ANY_SUBLINK)
289 tuple_fraction = 0.5; /* 50% */
290 else
291 tuple_fraction = 0.0; /* default behavior */
292
293 /*
294 * Generate the plan for the subquery.
295 */
296 node->plan = plan = subquery_planner(subquery, tuple_fraction);
--->
297
298 node->plan_id = PlannerPlanId++; /* Assign unique ID to this
299 * SubPlan */
300
301 node->rtable = subquery->rtable;
302
303 /*
304 * Initialize other fields of the SubPlan node.
305 */
306 node->subLinkType = slink->subLinkType;

------------------------------------------------------------------------
------------------------------------------

[BUG] memory leak, pformats is allocated but never freed or stored
anywhere
File where bug occurred: postgresql-7.4.2/src/backend/tcop/postgres.c

Diagnosis:

(1) pformats is allocated on line 1261
(2) pformats is never stored anywhere, nor is it freed before going
out of scope

1251 MemoryContextSwitchTo(MessageContext);
1252
1253 /* Get the fixed part of the message */
1254 portal_name = pq_getmsgstring(input_message);
1255 stmt_name = pq_getmsgstring(input_message);
1256
1257 /* Get the parameter format codes */
1258 numPFormats = pq_getmsgint(input_message, 2);
1259 if (numPFormats > 0)
1260 {
--->
1261 pformats = (int16 *) palloc(numPFormats * sizeof(int16));
--->
1262 for (i = 0; i < numPFormats; i++)
1263 pformats[i] = pq_getmsgint(input_message, 2);
1264 }
1265
1266 /* Get the parameter value count */
1267 numParams = pq_getmsgint(input_message, 2);
1268
1269 if (numPFormats > 1 && numPFormats != numParams)
1270 ereport(ERROR,
1271 (errcode(ERRCODE_PROTOCOL_VIOLATION),

On 2-May-04, at 2:05 PM, Ted Kremenek wrote:
> I'm from the Stanford Metacompilation research group where we use
> static analysis to find bugs.

Neat. BTW, I saw a talk last summer from Madanlal Musuvathi on some
model checking work which I believe is being done by a related group at
Stanford; it was very interesting.

The problem with applying this kind of static analysis to PostgreSQL is
that palloc() is not like malloc(): if the return value goes out of
scope before it is freed, it is NOT necessarily the case that a memory
leak has occurred. Each palloc() allocation occurs within a "memory
context" (a.k.a an arena, if you're familiar with the usage in tcc).
Individual allocations can be released via pfree(), or the entire
memory context and all memory allocated within it can be released via
MemoryContextReset() or a similar function. Some areas of the code
bother doing a pfree() for each palloc(); some do not.

> [BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
> File where bug occurred:
> postgresql-7.4.2/src/backend/commands/vacuumlazy.c

I believe the CommitTransactionCommand() at vacuum.c:894 (which calls
CommitTransaction(), which calls AtCommit_Memory(), which performs a
MemoryContextDelete()) deallocates this memory reasonably soon after it
has been allocated, so this isn't a bug.

> [BUG] memory leak on error path (dtype != DTK_DELTA)
> File where bug occurred:
> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?

> [BUG] difficult to analyze, but it appears that subquery is memory
> leaked
> File where bug occurred:
> postgresql-7.4.2/src/backend/optimizer/plan/subselect.c

Not sure about this one -- I didn't bother tracking down exactly where
the memory context manipulation happens, but I think it's likely that
we release this memory fairly soon after it's allocated.

> [BUG] memory leak, pformats is allocated but never freed or stored
> anywhere

Doesn't look like a bug to me: pformats is allocated in the
MessageContext (e.g. tcop/postgres.c:1308), which is reset for every FE
command that is processed (e.g. postgres.c:2849).

-Neil

Thanks Neil. The information about palloc() is extremely useful, and
it will be interesting for us to see how our analysis can better deal
with this.

Thanks for the quick reply!

Best,
Ted

On May 2, 2004, at 12:23 PM, Neil Conway wrote:

> On 2-May-04, at 2:05 PM, Ted Kremenek wrote:
>> I'm from the Stanford Metacompilation research group where we use
>> static analysis to find bugs.
>
> Neat. BTW, I saw a talk last summer from Madanlal Musuvathi on some
> model checking work which I believe is being done by a related group
> at Stanford; it was very interesting.
>
> The problem with applying this kind of static analysis to PostgreSQL
> is that palloc() is not like malloc(): if the return value goes out of
> scope before it is freed, it is NOT necessarily the case that a memory
> leak has occurred. Each palloc() allocation occurs within a "memory
> context" (a.k.a an arena, if you're familiar with the usage in tcc).
> Individual allocations can be released via pfree(), or the entire
> memory context and all memory allocated within it can be released via
> MemoryContextReset() or a similar function. Some areas of the code
> bother doing a pfree() for each palloc(); some do not.
>
>> [BUG] memory leak of vacrelstats at end of function laxy_vaccum_rel()
>> File where bug occurred:
>> postgresql-7.4.2/src/backend/commands/vacuumlazy.c
>
> I believe the CommitTransactionCommand() at vacuum.c:894 (which calls
> CommitTransaction(), which calls AtCommit_Memory(), which performs a
> MemoryContextDelete()) deallocates this memory reasonably soon after
> it has been allocated, so this isn't a bug.
>
>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>> File where bug occurred:
>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
>> [BUG] difficult to analyze, but it appears that subquery is memory
>> leaked
>> File where bug occurred:
>> postgresql-7.4.2/src/backend/optimizer/plan/subselect.c
>
> Not sure about this one -- I didn't bother tracking down exactly where
> the memory context manipulation happens, but I think it's likely that
> we release this memory fairly soon after it's allocated.
>
>> [BUG] memory leak, pformats is allocated but never freed or stored
>> anywhere
>
> Doesn't look like a bug to me: pformats is allocated in the
> MessageContext (e.g. tcop/postgres.c:1308), which is reset for every
> FE command that is processed (e.g. postgres.c:2849).
>
> -Neil
>

Neil Conway <neilc(at)samurai(dot)com> writes:
> The problem with applying this kind of static analysis to PostgreSQL is
> that palloc() is not like malloc(): if the return value goes out of
> scope before it is freed, it is NOT necessarily the case that a memory
> leak has occurred.

I'm a bit surprised that a tool unaware of this fact would generate only
four complaints ... I'd have expected hundreds.

I concur with Neil's opinion that none of the backend cases represent
bugs. However:

>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>> File where bug occurred:
>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c

> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?

It's entirely likely that ecpg's derivative of the backend's datetime
modules contains lots and lots of memory leaks, since AFAIK the palloc
infrastructure is not there in the ecpg environment :-(.

regards, tom lane

Since the tool is in its nascent stages, I reported only a handful of
reports that I myself looked at felt that they were potentially bugs.

I appreciate everyone's feedback.

Best,
Ted

On May 2, 2004, at 4:50 PM, Tom Lane wrote:

> Neil Conway <neilc(at)samurai(dot)com> writes:
>> The problem with applying this kind of static analysis to PostgreSQL
>> is
>> that palloc() is not like malloc(): if the return value goes out of
>> scope before it is freed, it is NOT necessarily the case that a memory
>> leak has occurred.
>
> I'm a bit surprised that a tool unaware of this fact would generate
> only
> four complaints ... I'd have expected hundreds.
>
> I concur with Neil's opinion that none of the backend cases represent
> bugs. However:
>
>>> [BUG] memory leak on error path (dtype != DTK_DELTA)
>>> File where bug occurred:
>>> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
>> Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.
>
> regards, tom lane
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html

On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
> >> [BUG] memory leak on error path (dtype != DTK_DELTA)
> >> File where bug occurred:
> >> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> > Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.

This is true of course. But I'm hopeful we found most of the possible
leaks. This one certainly was not found.

Fix just committed.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes(at)jabber(dot)org
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!

On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:

> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.

I wonder why is this? Is there some limitation to using palloc outside
the backend itself? I ask because I have considered using it outside
Postgres several times (a consideration that has never materialized
yet), and I wonder if it needs something special to work.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
Dios hizo a Adán, pero fue Eva quien lo hizo hombre.

Alvaro Herrera wrote:
> On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
>
> > It's entirely likely that ecpg's derivative of the backend's datetime
> > modules contains lots and lots of memory leaks, since AFAIK the palloc
> > infrastructure is not there in the ecpg environment :-(.
>
> I wonder why is this? Is there some limitation to using palloc outside
> the backend itself? I ask because I have considered using it outside
> Postgres several times (a consideration that has never materialized
> yet), and I wonder if it needs something special to work.

The semantics of palloc is that most stuff is freed on statement
completion. In most cases, interfaces need different semantics so we
haven't seen much need for making something like palloc available to
clients. I can see ecpg using it in a few cases, and libpq too, but
probably not enough to make it worthwhile.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Mon, Jul 05, 2004 at 05:13:23PM -0400, Bruce Momjian wrote:
> Alvaro Herrera wrote:
> > On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
> >
> > > It's entirely likely that ecpg's derivative of the backend's datetime
> > > modules contains lots and lots of memory leaks, since AFAIK the palloc
> > > infrastructure is not there in the ecpg environment :-(.
> >
> > I wonder why is this? Is there some limitation to using palloc outside
> > the backend itself? I ask because I have considered using it outside
> > Postgres several times (a consideration that has never materialized
> > yet), and I wonder if it needs something special to work.
>
> The semantics of palloc is that most stuff is freed on statement
> completion. In most cases, interfaces need different semantics so we
> haven't seen much need for making something like palloc available to
> clients. I can see ecpg using it in a few cases, and libpq too, but
> probably not enough to make it worthwhile.

Yes, I understand that part -- what I was talking about was not using
the code in the Pg interfaces, but in another software project which
also consists of a daemon that has several well defined "durations" of
objects. In that (as of yet unwritten) code, palloc would fit very
well. But does palloc depend on some other part of the Postgres code?

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"El Maquinismo fue proscrito so pena de cosquilleo hasta la muerte"
(Ijon Tichy en Viajes, Stanislaw Lem)

Alvaro Herrera <alvherre(at)dcc(dot)uchile(dot)cl> writes:
> I wonder why is this? Is there some limitation to using palloc outside
> the backend itself? I ask because I have considered using it outside
> Postgres several times (a consideration that has never materialized
> yet), and I wonder if it needs something special to work.

It's useless without the rest of the backend infrastructure, including
elog error recovery and a cooperative main loop. You could certainly
reuse the code in a different application if you were willing to adopt
Postgres' memory management and error handling approaches
lock-stock-and-barrel, but I don't think you could be selective about
it. For a library that has to work inside a not-so-cooperative
application, the idea is a nonstarter.

regards, tom lane

Alvaro Herrera wrote:

> On Mon, Jul 05, 2004 at 05:13:23PM -0400, Bruce Momjian wrote:
>
>>Alvaro Herrera wrote:
>>
>>>On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
>>>
>>>
>>>>It's entirely likely that ecpg's derivative of the backend's datetime
>>>>modules contains lots and lots of memory leaks, since AFAIK the palloc
>>>>infrastructure is not there in the ecpg environment :-(.
>>>
>>>I wonder why is this? Is there some limitation to using palloc outside
>>>the backend itself? I ask because I have considered using it outside
>>>Postgres several times (a consideration that has never materialized
>>>yet), and I wonder if it needs something special to work.
>>
>>The semantics of palloc is that most stuff is freed on statement
>>completion. In most cases, interfaces need different semantics so we
>>haven't seen much need for making something like palloc available to
>>clients. I can see ecpg using it in a few cases, and libpq too, but
>>probably not enough to make it worthwhile.
>
>
> Yes, I understand that part -- what I was talking about was not using
> the code in the Pg interfaces, but in another software project which
> also consists of a daemon that has several well defined "durations" of
> objects. In that (as of yet unwritten) code, palloc would fit very
> well. But does palloc depend on some other part of the Postgres code?
>

If you don't mind you can write your application in C++ and use a
boost smartpointer:

http://www.boost.org/libs/smart_ptr/smart_ptr.htm

Regards
Gaetano Mendola

Thank you for the confirmation.

Best,
Ted

-----Original Message-----
From: Michael Meskes [mailto:meskes(at)postgresql(dot)org]
Sent: Monday, July 05, 2004 11:55 AM
To: Tom Lane
Cc: Neil Conway; Ted Kremenek; pgsql-bugs(at)postgresql(dot)org; Michael Meskes
Subject: Re: [BUGS] [CHECKER] 4 memory leaks in Postgresql 7.4.2

On Sun, May 02, 2004 at 07:50:46PM -0400, Tom Lane wrote:
> >> [BUG] memory leak on error path (dtype != DTK_DELTA)
> >> File where bug occurred:
> >> postgresql-7.4.2/src/interfaces/ecpg/pgtypeslib/interval.c
>
> > Looks suspicious to me, but ECPG is Michael Meskes' domain -- Michael?
>
> It's entirely likely that ecpg's derivative of the backend's datetime
> modules contains lots and lots of memory leaks, since AFAIK the palloc
> infrastructure is not there in the ecpg environment :-(.

This is true of course. But I'm hopeful we found most of the possible
leaks. This one certainly was not found.

Fix just committed.

Michael
--
Michael Meskes
Email: Michael at Fam-Meskes dot De
ICQ: 179140304, AIM/Yahoo: michaelmeskes, Jabber: meskes(at)jabber(dot)org
Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!