Re: Some download statistics

(crossposting this to hackers, I'm sure there are interested people
there as well)

Since Dave set the site up for tracking clickthroughs, I hit the db with
a couple of queries to count our downloads. This is what I came up with:

I did some simple pivoting in Excel and split it into categories win32,
source, sig (MD5 or PGP signatures), RPMs, split (the split tarballs),
pgadmin and ODBC. Other stuff was so little that I cut it.

The numbers for just-past-release may be off a bit because of the move
of wwwmaster. Not sure what happened to the db.

Just before and about 24 hours after the release of 8.0, we had the
following:
win32 16480
source 5223
sig 2016
rpm 1908
split 1689
pgadmin 370
odbc 261
Grand Total 27947

Total tally since the tracking started (2005-01-11):
win32 106878 56,37%
source 39058 20,60%
rpm 11703 6,17%
split 9135 4,82%
sig 8931 4,71%
pgadmin 7281 3,84%
odbc 6601 3,48%
Total 189587

And the versions being downloaded (total):
8.0.1 85511
8.0.0 82517
notpg 14526 (pgadmin, odbc etc)
7.4.6 4297
7.4.7 2736
Grand Total 189587

This does *not* include bittorrent downloads.

Some conclusions that I draw from this:
1) The win32 distribution is very popular. Some of this is no doubt
because it's new. Also, about 3500 of those downloads are RC5 downloads.
RC5 downloads of the source were during this time so few that they
didn't show up after my cutoff.

2) Most of the people who use RPMs on their linux systems wait for their
distribution to ship the RPMs and don't get it. Or they are alraedy set
up to download directly from their mirror and not from the website (the
tracker only counts downloads that were initiated from the website)

3) There doesnt' seem to be much point to the distribution splits. A
total of less than 5% the *number* of downloads. And most people
probably get more than one file, so in reality that number shuold
proably be divided by 4 or 5.
I know several people who downloaded source *plus* the split ones,
because "hey, I need postgresql. And I certainly need base too. And I
need docs.". They don't realise it's included in the main tarball.
Frankly, I'd suggest dropping the splits. Thoughts?

4) People upgraded to 8.0.1. Not a lot of people have upgraded their 7.4
systems.

As always, it's statistics. You can say anything you want with it...
Someone who chews numbers for a living might be able to get more out of
it. If someone is interested in the excel pivot tables with this stuff
in it, let me know. But it's nothing exciting there :)

//Magnus

On Thu, 24 Feb 2005, Magnus Hagander wrote:

> 3) There doesnt' seem to be much point to the distribution splits. A
> total of less than 5% the *number* of downloads. And most people
> probably get more than one file, so in reality that number shuold
> proably be divided by 4 or 5.
> I know several people who downloaded source *plus* the split ones,
> because "hey, I need postgresql. And I certainly need base too. And I
> need docs.". They don't realise it's included in the main tarball.
> Frankly, I'd suggest dropping the splits. Thoughts?

this tracks only those going in through the web ... this doesn't track
those using an ftp client going in and downloading files ... specifically,
I know that all of the FreeBSD ports are based on the split distributions
... in the case of the postgresql80-server port, it only downloads base
and opt and ignores the rest ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Hi Magnus,

On Feb 24, 2005, at 11:35 AM, Magnus Hagander wrote:

> I did some simple pivoting in Excel and split it into categories win32,
> source, sig (MD5 or PGP signatures), RPMs, split (the split tarballs),
> pgadmin and ODBC. Other stuff was so little that I cut it.

Assuming this is from a HTTP log, I'm just curious if you took into
account that many Windows users have "download managers" that make
repeated requests to get a single file. You see this in the log with a
206 (partial content) HTTP response code. So you might have anywhere
from 10 to 20 hits in the log to download a single file. This makes it
a pain to get accurate download statistics.

John DeSoi, Ph.D.
http://pgedit.com/
Power Tools for PostgreSQL

On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:

> I know several people who downloaded source *plus* the split ones,
> because "hey, I need postgresql. And I certainly need base too. And I
> need docs.". They don't realise it's included in the main tarball.
> Frankly, I'd suggest dropping the splits. Thoughts?

I also found the split sources + a non-split sources version to be
confusing. As you, I think that splitting should be dropped.

--
Greetings from Troels Arvin, Copenhagen, Denmark

Troels Arvin <troels(at)arvin(dot)dk> writes:
> On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:
>> Frankly, I'd suggest dropping the splits. Thoughts?

> I also found the split sources + a non-split sources version to be
> confusing. As you, I think that splitting should be dropped.

Perhaps the confusion issue could be addressed by keeping the split
sources in a separate subdirectory:

pub/source/v.8.0.1/
postgresql-8.0.1.tar.bz2
postgresql-8.0.1.tar.bz2.md5
postgresql-8.0.1.tar.gz
postgresql-8.0.1.tar.gz.md5
split-tarballs/
postgresql-base-8.0.1.tar.bz2
postgresql-base-8.0.1.tar.bz2.md5
etc...

I suspect the demand for the split tarballs is mighty low nowaways,
but it's probably not zero yet.

Also: I notice that the README file that's supposed to tell people about
the split-tarball scheme is not present in any of the recent-version
subdirectories, so it's no wonder that they are confused.

regards, tom lane

On Thu, 24 Feb 2005, Tom Lane wrote:

> Troels Arvin <troels(at)arvin(dot)dk> writes:
>> On Thu, 24 Feb 2005 17:35:57 +0100, Magnus Hagander wrote:
>>> Frankly, I'd suggest dropping the splits. Thoughts?
>
>> I also found the split sources + a non-split sources version to be
>> confusing. As you, I think that splitting should be dropped.
>
> Perhaps the confusion issue could be addressed by keeping the split
> sources in a separate subdirectory:
>
> pub/source/v.8.0.1/
> postgresql-8.0.1.tar.bz2
> postgresql-8.0.1.tar.bz2.md5
> postgresql-8.0.1.tar.gz
> postgresql-8.0.1.tar.gz.md5
> split-tarballs/
> postgresql-base-8.0.1.tar.bz2
> postgresql-base-8.0.1.tar.bz2.md5
> etc...

That sounds like a reasonable compromise ... I could do that right away,
if nobody is in disagreement ... ?

> I suspect the demand for the split tarballs is mighty low nowaways,
> but it's probably not zero yet.

Considering that every new FreeBSD install uses them exclusively, I don't
expect it to drop to zero in the near future ..

> Also: I notice that the README file that's supposed to tell people about
> the split-tarball scheme is not present in any of the recent-version
> subdirectories, so it's no wonder that they are confused.

Actually, its never been ... just checked, its in the root directory ...
but, with that in mind, I've added putting a copy in there to the release
script ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

On Thursday 24 February 2005 21:43, Marc G. Fournier wrote:
[.......]
> >
> > pub/source/v.8.0.1/
> > postgresql-8.0.1.tar.bz2
> > postgresql-8.0.1.tar.bz2.md5
> > postgresql-8.0.1.tar.gz
> > postgresql-8.0.1.tar.gz.md5
> > split-tarballs/
> > postgresql-base-8.0.1.tar.bz2
> > postgresql-base-8.0.1.tar.bz2.md5
> > etc...
>
> That sounds like a reasonable compromise ... I could do that right away,
> if nobody is in disagreement ... ?

Hm, there is a distribution that uses the split tarballs: gentoo.

snipped from /usr/portage/dev-db/postgresql/postgresql-8.0.1-r1.ebuild :

SRC_URI="mirror://postgresql/source/v${PV}/${PN}-base-${MY_PV}.tar.bz2
mirror://postgresql/source/v${PV}/${PN}-opt-${MY_PV}.tar.bz2
doc? ( mirror://postgresql/source/v${PV}/${PN}-docs-${MY_PV}.tar.bz2 )"

Better is to drop a 'bug' to bugs.gentoo.org so that the ebuild will be
fixed;-)

<Earny>

"Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
> On Thu, 24 Feb 2005, Tom Lane wrote:
>> Also: I notice that the README file that's supposed to tell people about
>> the split-tarball scheme is not present in any of the recent-version
>> subdirectories, so it's no wonder that they are confused.

> Actually, its never been ... just checked, its in the root directory ...
> but, with that in mind, I've added putting a copy in there to the release
> script ...

It is present in the subdirectories for some older releases, eg v7.3.

regards, tom lane

On Thu, 24 Feb 2005, Tom Lane wrote:

> "Marc G. Fournier" <scrappy(at)postgresql(dot)org> writes:
>> On Thu, 24 Feb 2005, Tom Lane wrote:
>>> Also: I notice that the README file that's supposed to tell people about
>>> the split-tarball scheme is not present in any of the recent-version
>>> subdirectories, so it's no wonder that they are confused.
>
>> Actually, its never been ... just checked, its in the root directory ...
>> but, with that in mind, I've added putting a copy in there to the release
>> script ...
>
> It is present in the subdirectories for some older releases, eg v7.3.

'k, now I see it ... I was looking for a README file .. fixed ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

On Thu, Feb 24, 2005 at 10:27:03PM +0100, Ernst Herzberg wrote:
> On Thursday 24 February 2005 21:43, Marc G. Fournier wrote:
> [.......]
> > >
> > > pub/source/v.8.0.1/
> > > postgresql-8.0.1.tar.bz2
> > > postgresql-8.0.1.tar.bz2.md5
> > > postgresql-8.0.1.tar.gz
> > > postgresql-8.0.1.tar.gz.md5
> > > split-tarballs/
> > > postgresql-base-8.0.1.tar.bz2
> > > postgresql-base-8.0.1.tar.bz2.md5
> > > etc...
> >
> > That sounds like a reasonable compromise ... I could do that right away,
> > if nobody is in disagreement ... ?
>
> Hm, there is a distribution that uses the split tarballs: gentoo.
>
> snipped from /usr/portage/dev-db/postgresql/postgresql-8.0.1-r1.ebuild :
>
> SRC_URI="mirror://postgresql/source/v${PV}/${PN}-base-${MY_PV}.tar.bz2
> mirror://postgresql/source/v${PV}/${PN}-opt-${MY_PV}.tar.bz2
> doc? ( mirror://postgresql/source/v${PV}/${PN}-docs-${MY_PV}.tar.bz2 )"
>
> Better is to drop a 'bug' to bugs.gentoo.org so that the ebuild will be
> fixed;-)

How exactly is that a bug? Why should someone spend time downloading the
docs if they don't want to install them? And gentoo isn't the only OS
that uses them.

Unless generating the splits requires a non-trivial amount of work, I
see no reason not to have them.
--
Jim C. Nasby, Database Consultant decibel(at)decibel(dot)org
Give your computer some brain candy! www.distributed.net Team #1828

Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"

On Fri, 25 Feb 2005, Jim C. Nasby wrote:

> Unless generating the splits requires a non-trivial amount of work, I
> see no reason not to have them.

its all automated ... still wish I could figure out a way of splitting off
*just* libppq, since that would drop downloads of the full package
significantly ... in my case, I need postgresql "server" on maybe one
machine, but need just libpq on every other one that I run ..

I've gotta sit down and see if I can figure out how to do thatt .. would
really be interesting to see the stats if we had a seperate 'libpq'
download :)

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Marc,

> this tracks only those going in through the web ... this doesn't track
> those using an ftp client going in and downloading files ... specifically,

Can you give me stats for ftp.postgresql.org, for the 30 days beginning
February 17? These are extra numbers, plus would give me stats to compare
with last year, for which we only have ftp.postgresql.org.

For other download sites, in the month following release:
pgFoundry saw approximately 25,000 downloads of the pgInstaller.
SourceForge's statistics are broken, so we don't know how many downloads they
got.
I'm waiting for David Fetter's answer on Bittorrent.

It would be nice if Josh Drake could give us stats for Mammoth PostgreSQL 8.0.

--
Josh Berkus
Aglio Database Solutions
San Francisco

On Sun, 27 Feb 2005, Josh Berkus wrote:

> Can you give me stats for ftp.postgresql.org, for the 30 days beginning
> February 17?

Did you mean Jan 17th? Since toda yis only the r27th of Feb, 30 days
beginning Feb 17 coulbe difficult? :)

> pgFoundry saw approximately 25,000 downloads of the pgInstaller.

Does this take into consideratino something that was mentioned in a
seperate thread? Where Windows does 'multiple connects' to download the
same file?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Marc,

> Did you mean Jan 17th? Since toda yis only the r27th of Feb, 30 days
> beginning Feb 17 coulbe difficult? :)

Right, January 17.

>
> > pgFoundry saw approximately 25,000 downloads of the pgInstaller.
>
> Does this take into consideratino something that was mentioned in a
> seperate thread? Where Windows does 'multiple connects' to download the
> same file?

Not sure how pgInstaller counts downloads, so it's a firm I Don't Know.

--
Josh Berkus
Aglio Database Solutions
San Francisco

On Sun, 27 Feb 2005, Josh Berkus wrote:

> Marc,
>
>> Did you mean Jan 17th? Since toda yis only the r27th of Feb, 30 days
>> beginning Feb 17 coulbe difficult? :)
>
> Right, January 17.

20077 downloads, top 10 being (didn't think you wanted all 1720 lines):

2411 /var/spool/ftp/pub/odbc/versions/full/psqlodbc-07_03_0200.zip
2221 /var/spool/ftp/pub/odbc/versions/msi/psqlodbc-07_03_0200.zip
659 /var/spool/ftp/pub/binary/v8.0.1/win32/postgresql-8.0.1.zip
514 /var/spool/ftp/pub/source/v8.0.1/postgresql-8.0.1.tar.gz
483 /var/spool/ftp/pub/source/v8.0.0/postgresql-8.0.0.tar.gz
333 /var/spool/ftp/pub/source/v8.0.1/postgresql-8.0.1.tar.bz2
281 /var/spool/ftp/pub/source/v8.0.0/postgresql-8.0.0.tar.bz2
260 /var/spool/ftp/pub/binary/v8.0/win32/postgresql-8.0.zip
240 /var/spool/ftp/pub/odbc/versions/snapshots/psqlodbc-08_00_0004.zip
201 /var/spool/ftp/pub/odbc/versions/src/psqlodbc-07.03.0200.tar.gz

Broken down by top level directory:

8084 odbc
5075 source
5014 binary
593 stable_snapshot
540 pgadmin3
156 README
155 projects
136 dev
107 sync_timestamp
67 ls
54 README.dist-split
41 er.msg
32 README.cvsup
23 ct.msg

source directory broken down by version:

1631 v8.0.1
1553 v8.0.0
370 v7.4.6
321 v7.4.7
126 v7.4.5
123 v8.0.0beta
100 v7.3.9
84 v7.2.7
71 v7.4
55 v7.3.8
51 v7.3.4
50 v7.3.3
49 v7.1.3
46 v7.1
37 v7.2.2
35 v7.4.3
32 v7.2.3
31 v7.2.4
31 v7.2
28 v7.1.2
24 v7.2.5
23 v7.3.2
22 v7.4.1
19 v7.3
19 v7.0
18 v7.2.1
16 v7.4.2
16 v7.3.1
15 v7.0.3
12 v7.0.2
11 v7.1.1
10 v7.3.6
10 v7.0.1
8 v6.5
4 v7.2.6
4 v6.4
4 v1.09
4 v1.08
3 v6.2
2 v6.3
2 v6.1
2 v6.0
1 v7.4.4
1 v7.3.7
1 v7.3.5

and breakdown of binary:

1826 v8.0
1800 v8.0.1
316 v7.4.6
209 v7.0.3
124 v7.4
87 v7.4.5
76 v7.4.7
74 v7.1
71 v7.4.2
69 v7.1.2
54 v7.3.4
45 v7.3.1
41 v7.3.8
30 v7.3.9
24 v7.2
23 v8.0.0rc1
22 v7.3.3
20 v7.2.4
19 v7.3.2
16 v7.2.2
15 v7.4.1
12 v7.2.1
11 v7.3
9 v7.4.4
7 v7.1.1
7 v7.0
3 v7.4.3
1 v7.3.6
1 v7.2.7
1 v7.2.6
1 v7.1.3

help any?