Renaming conversion procs (was Re: Error on compile for Windows)

Hello, this is the first time I post on this list.

I have setup a build environment on Windows 7 32 bits, with Visual Studio
2005. Took me a while to get all the bits and parts working, but I finally
went through it. Only problem, I get 2 LINK errors:
fatal error LINK 1104: cannot open file
'.\Release\utf8_and_shift_jis_2004\utf8_and_shift_jis_2004.obj'
fatal error LINK 1104: cannot open file
'.\Release\euc_jis_2004_and_shift_jis_2004\
euc_jis_2004_and_shift_jis_2004.obj'

Anyone could help me with that? I have no idea what I could be missing.

Thanks so much for nay pointer, I really want to be able to see through that
project, as quite a few things are depending on it.

Bernard.

"Bernard Grosperrin" <bgrosperrin(at)laposte(dot)net> writes:
> I have setup a build environment on Windows 7 32 bits, with Visual Studio
> 2005. Took me a while to get all the bits and parts working, but I finally
> went through it. Only problem, I get 2 LINK errors:
> fatal error LINK 1104: cannot open file
> '.\Release\utf8_and_shift_jis_2004\utf8_and_shift_jis_2004.obj'
> fatal error LINK 1104: cannot open file
> '.\Release\euc_jis_2004_and_shift_jis_2004\
> euc_jis_2004_and_shift_jis_2004.obj'

Hm, do you have a file

src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c

and did anything get built from that?

We have seen reports of certain versions of "tar" failing to extract
this file from the source tarball, probably because its name is so
long. That could have happened to you --- if so, the recommendation
is to find another tool to un-tar with.

regards, tom lane

On 2/11/2009 11:40 AM, Tom Lane wrote:
> "Bernard Grosperrin" <bgrosperrin(at)laposte(dot)net> writes:
>> I have setup a build environment on Windows 7 32 bits, with Visual Studio
>> 2005. Took me a while to get all the bits and parts working, but I finally
>> went through it. Only problem, I get 2 LINK errors:
>> fatal error LINK 1104: cannot open file
>> '.\Release\utf8_and_shift_jis_2004\utf8_and_shift_jis_2004.obj'
>> fatal error LINK 1104: cannot open file
>> '.\Release\euc_jis_2004_and_shift_jis_2004\
>> euc_jis_2004_and_shift_jis_2004.obj'
>
> Hm, do you have a file
>
> src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c
>
> and did anything get built from that?
>
> We have seen reports of certain versions of "tar" failing to extract
> this file from the source tarball, probably because its name is so
> long. That could have happened to you --- if so, the recommendation
> is to find another tool to un-tar with.

I almost wonder if a `configure' test to detect this particular form of
source tree corruption would be helpful.

--
Craig Ringer

Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
> On 2/11/2009 11:40 AM, Tom Lane wrote:
>> src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c
>> We have seen reports of certain versions of "tar" failing to extract
>> this file from the source tarball, probably because its name is so
>> long. That could have happened to you --- if so, the recommendation
>> is to find another tool to un-tar with.

> I almost wonder if a `configure' test to detect this particular form of
> source tree corruption would be helpful.

If we were going to expend any effort on this problem, my vote would be
to rename the file to something less carpal-tunnel-promoting. There's
no real reason it couldn't be euc2004_and_sjis2004.c or some such.

The problem with doing anything is that we have only a couple of reports
and so no solid picture of what the restriction is.

regards, tom lane

Thanks Tom, took me a while to realize those files were missing. The build
script did not report the source of the error, but trying to compile just
those projects in VS showed the reason immediately. Took them of the source
documentation, put them back, and compiled without any error!

take care,
Bernard

-----Message d'origine-----
De : Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Envoyé : lundi 2 novembre 2009 4:40
À : Bernard Grosperrin
Cc : pgsql-general(at)postgresql(dot)org
Objet : Re: [GENERAL] Error on compile for Windows

"Bernard Grosperrin" <bgrosperrin(at)laposte(dot)net> writes:
> I have setup a build environment on Windows 7 32 bits, with Visual Studio
> 2005. Took me a while to get all the bits and parts working, but I finally
> went through it. Only problem, I get 2 LINK errors:
> fatal error LINK 1104: cannot open file
> '.\Release\utf8_and_shift_jis_2004\utf8_and_shift_jis_2004.obj'
> fatal error LINK 1104: cannot open file
> '.\Release\euc_jis_2004_and_shift_jis_2004\
> euc_jis_2004_and_shift_jis_2004.obj'

Hm, do you have a file

src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_ji
s_2004_and_shift_jis_2004.c

and did anything get built from that?

We have seen reports of certain versions of "tar" failing to extract
this file from the source tarball, probably because its name is so
long. That could have happened to you --- if so, the recommendation
is to find another tool to un-tar with.

regards, tom lane

>The problem with doing anything is that we have only a couple of reports
>and so no solid picture of what the restriction is.

If it help, I am using WinZip. And checking in the archive (now that I know
what to look for), I can see those 2 files are there! Hum... Cost me quite a
few hours.....

Regards,
Bernard

"Bernard Grosperrin" <bgrosperrin(at)laposte(dot)net> writes:
>> The problem with doing anything is that we have only a couple of reports
>> and so no solid picture of what the restriction is.

> If it help, I am using WinZip. And checking in the archive (now that I know
> what to look for), I can see those 2 files are there! Hum... Cost me quite a
> few hours.....

Anybody want to do a bit of experimentation and figure out exactly what
WinZip's filename length limit is? I'd be willing to rename the file
if we can be sure the new name would fix the problem.

regards, tom lane

On Nov 1, 2009, at 10:19 PM, Tom Lane wrote:

> Craig Ringer <craig(at)postnewspapers(dot)com(dot)au> writes:
>> On 2/11/2009 11:40 AM, Tom Lane wrote:
>>> src/backend/utils/mb/conversion_procs/
>>> euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c
>>> We have seen reports of certain versions of "tar" failing to extract
>>> this file from the source tarball, probably because its name is so
>>> long. That could have happened to you --- if so, the recommendation
>>> is to find another tool to un-tar with.
>
>> I almost wonder if a `configure' test to detect this particular
>> form of
>> source tree corruption would be helpful.
>
> If we were going to expend any effort on this problem, my vote would
> be
> to rename the file to something less carpal-tunnel-promoting. There's
> no real reason it couldn't be euc2004_and_sjis2004.c or some such.
>
> The problem with doing anything is that we have only a couple of
> reports
> and so no solid picture of what the restriction is.

I've seen this problem with tar on most versions of Solaris. While the
documented limits are 255 characters, with a limit of 100 characters
on the basename there are times when if the tarball is created with
gnutar the Solaris tar seems to fail to extract pathnames longer than
100
characters.

I've also seen it with winzip. Again, ISTR that the exact limits were
obscure
but that restricting the path to less than 100 characters avoided any
problems.

This case is 105 characters long, so wouldn't be hard to pull below
that threshold.

Cheers,
Steve

Steve Atkins <steve(at)blighty(dot)com> writes:
> I've also seen it with winzip. Again, ISTR that the exact limits were
> obscure but that restricting the path to less than 100 characters
> avoided any problems.

Hmm. It strikes me that the names seen by tar include "postgresql-x.y.z/".
The only file paths that approach 100 characters on that basis as of
8.4.1 are

postgresql-8.4.1/src/backend/utils/mb/conversion_procs/utf8_and_shift_jis_2004/utf8_and_shift_jis_2004.c
postgresql-8.4.1/src/backend/utils/mb/conversion_procs/utf8_and_euc_jis_2004/utf8_and_euc_jis_2004.c
postgresql-8.4.1/src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c

The first and third of these have in fact been reported as trouble
spots. AFAIR the second has not, but it's exactly 100 characters, which
would explain why it works ... or will work till we get to two digits in
the minor release number, anyway :-(. So that seems to validate your
theory.

If we want to set an upper limit of 100 characters, and allow for
release numbers up to 99.99.99, then the maximum length for
conversion_procs file names would be 19 characters (plus .c), and the
same for their directories. So we could rename these to, say,
utf8_and_sjis2004
utf8_and_euc2004
euc2004_sjis2004
This would be an easy change to make going forward (other than loss of
CVS history, but I'm not terribly worried about that for these files).
We could not so easily back-patch it because the .so filenames are
already embedded in installations' pg_proc tables. Personally I'd
be satisfied if it's fixed for 8.5 and beyond --- comments?

regards, tom lane

On Mon, Nov 2, 2009 at 18:54, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Steve Atkins <steve(at)blighty(dot)com> writes:
>> I've also seen it with winzip. Again, ISTR that the exact limits were
>> obscure but that restricting the path to less than 100 characters
>> avoided any problems.
>
> Hmm.  It strikes me that the names seen by tar include "postgresql-x.y.z/".
> The only file paths that approach 100 characters on that basis as of
> 8.4.1 are
>
> postgresql-8.4.1/src/backend/utils/mb/conversion_procs/utf8_and_shift_jis_2004/utf8_and_shift_jis_2004.c
> postgresql-8.4.1/src/backend/utils/mb/conversion_procs/utf8_and_euc_jis_2004/utf8_and_euc_jis_2004.c
> postgresql-8.4.1/src/backend/utils/mb/conversion_procs/euc_jis_2004_and_shift_jis_2004/euc_jis_2004_and_shift_jis_2004.c
>
> The first and third of these have in fact been reported as trouble
> spots.  AFAIR the second has not, but it's exactly 100 characters, which
> would explain why it works ... or will work till we get to two digits in
> the minor release number, anyway :-(.  So that seems to validate your
> theory.
>
> If we want to set an upper limit of 100 characters, and allow for
> release numbers up to 99.99.99, then the maximum length for
> conversion_procs file names would be 19 characters (plus .c), and the
> same for their directories.  So we could rename these to, say,
>        utf8_and_sjis2004
>        utf8_and_euc2004
>        euc2004_sjis2004
> This would be an easy change to make going forward (other than loss of
> CVS history, but I'm not terribly worried about that for these files).
> We could not so easily back-patch it because the .so filenames are
> already embedded in installations' pg_proc tables.  Personally I'd
> be satisfied if it's fixed for 8.5 and beyond --- comments?

Seems like this would be a major PITA for packagers and end-user. And
it would be an issue for the vast majority of our users - who use
binary packages on whatever platform they're on. And that only to help
those that have a broken (or severely limited) tar version, *and* try
to build from source.

Thus, +1 for doing this for 8.5 and beyond only.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Mon, Nov 2, 2009 at 18:54, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> [ rename some conversion libraries to shorten source path names ]

> Seems like this would be a major PITA for packagers and end-user.

If we actually wanted to back-patch it, I think the least painful way
would be to tweak the Makefiles to install the built .so's under the old
names in existing release branches. Then it would be a PITA only to the
developers ;-). I don't personally feel it's worth it, even so. There
are not that many people trying to build from source with weird tools.

regards, tom lane