why is the LIMIT clause slowing down this SELECT?

On a 8.1.9 version database that has been recently vacuumed and
analyzed, I'm seeing some dramatic performance degradation if a limit
clause is included in the query. This seems counter-intuitive to me.

Here's the query and explain plan WITH the LIMIT clause:

SELECT *
FROM topic_feed
WHERE topic_id = 106947234
ORDER BY score DESC
LIMIT 25

Limit (cost=0.00..651.69 rows=25 width=29) (actual
time=72644.652..72655.029 rows=25 loops=1)
-> Index Scan Backward using topic_feed_score_index on topic_feed
(cost=0.00..21219.08 rows=814 width=29) (actual
time=72644.644..72654.855 rows=25 loops=1)
Filter: (topic_id = 106947234)
Total runtime: 72655.733 ms

==============

and now WITHOUT the LIMIT clause:

SELECT *
FROM topic_feed
WHERE topic_id = 106947234
ORDER BY score DESC

Sort (cost=1683.75..1685.78 rows=814 width=29) (actual
time=900.553..902.267 rows=492 loops=1)
Sort Key: score
-> Bitmap Heap Scan on topic_feed (cost=7.85..1644.40 rows=814
width=29) (actual time=307.900..897.993 rows=492 loops=1)
Recheck Cond: (topic_id = 106947234)
-> Bitmap Index Scan on
index_topic_feed_on_topic_id_and_feed_id (cost=0.00..7.85 rows=814
width=0) (actual time=213.205..213.205 rows=2460 loops=1)
Index Cond: (topic_id = 106947234)
Total runtime: 904.049 ms

-----------------------------------------

That's a pretty big delta (72.6 sec vs. 0.9 sec), and I would expect
the query with the limit to be faster.

Can anyone explain why this happens and what I can do about it?

thanks in advance,
Mason