OT: Canadian Tax Database

http://www.thestar.com/News/article/189175

"For instance, in some cases the field for the social insurance number
was instead filled in with a birth date."

Unbelievable. Sixty years of electronic computing, fifty years use in
business and the "professionals" who built the tax system for a wealthy
democratic country didn't use data types.

--
Richard Huxton
Archonet Ltd

Richard Huxton wrote:
> http://www.thestar.com/News/article/189175
>
> "For instance, in some cases the field for the social insurance number
> was instead filled in with a birth date."
>
> Unbelievable. Sixty years of electronic computing, fifty years use in
> business and the "professionals" who built the tax system for a
> wealthy democratic country didn't use data types.

This is Unbelievable? This is commonplace.

Joshua D. Drake

Joshua D. Drake wrote:
> Richard Huxton wrote:
>> http://www.thestar.com/News/article/189175
>>
>> "For instance, in some cases the field for the social insurance number
>> was instead filled in with a birth date."
>>
>> Unbelievable. Sixty years of electronic computing, fifty years use in
>> business and the "professionals" who built the tax system for a
>> wealthy democratic country didn't use data types.
>
> This is Unbelievable? This is commonplace.

I need to start adding 10% on my fees for things like "use of types".

I would put some foreign-key constraints in my current project, but I
don't think the client can afford it ;-)

--
Richard Huxton
Archonet Ltd

----- Original Message -----
From: "Joshua D. Drake" <jd(at)commandprompt(dot)com>
To: "Richard Huxton" <dev(at)archonet(dot)com>
Cc: <pgsql-general(at)postgresql(dot)org>
Sent: Thursday, March 08, 2007 8:00 AM
Subject: Re: [GENERAL] OT: Canadian Tax Database

> Richard Huxton wrote:
>> http://www.thestar.com/News/article/189175
>>
>> "For instance, in some cases the field for the social insurance number
>> was instead filled in with a birth date."
>>
>> Unbelievable. Sixty years of electronic computing, fifty years use in
>> business and the "professionals" who built the tax system for a wealthy
>> democratic country didn't use data types.
>
> This is Unbelievable? This is commonplace.
>
And due at least in part to government (and other institutions operated by
damned fools) opting for the least expensive provider rather than paying for
someone who actually knows what they're doing. Just as buying cheap junk
always comes back to get you, hiring incompetent fools that don't know their
ass from a hole in the ground will come back to get you too.

This time CRA is embarrassed, but they don't care because the people that
suffer are the taxpayers who ultimately paid for such shoddy work in the
first place. There's no consequences for the bureaucratic peons really
responsible for it. They probably even get paid obscene sums in overtime
for the time they spend fixing the problem. More annoying, for me, are the
scurrilous scoundrels that pass themselves off as competent software
consultants who take advantage of such incompetence in their clients' staff.
I couldn't begin to document all the cases I have seen where either the
wrong software was used (imagine a spreadsheet being used as an RDBMS) or
the right software was grossly abused (imagine forcing a data entry clerk to
enter the same data four times because the developer was too damned lazy or
incompetent to develop a simple form to collect the data once and then
submit it to the four externally owned databases that needed to be queried
using it, and then having to manually collate the results returned from the
queries). And then businesses operated by capable folk get burned by such
incompetent and unethical scoundrels and swear off custom software because
they'd rather have a COTS product that gives a 80% fit than try for a 100%
fit with a custom product that in the end doesn't work at all. I have been
told by some of these folk that they have found it virtually impossible to
find capable software developers. This is because these scoundrels I
mention outnumber capable developers by several orders of magnitude (and the
current state of the curricula at colleges 'training' programmers doesn't
help).

It is soooo easy to get cynical, and very discouraged, when I think about
this. :-( Maybe I should have myself lobotomized and become one of the
mindless grunts at Canada post.

Cheers

Ted

> And due at least in part to government (and other institutions operated by
> damned fools) opting for the least expensive provider rather than paying for
> someone who actually knows what they're doing. Just as buying cheap junk
> always comes back to get you, hiring incompetent fools that don't know their
> ass from a hole in the ground will come back to get you too.

What you describe is a hundred times better than the reality... most of
them actually get _expensive_ junk with some kick-back ;-)

Cheers,
Csaba.

On Thu, 2007-03-08 at 09:15 -0500, Ted Byers wrote:
> ----- Original Message -----
> From: "Joshua D. Drake" <jd(at)commandprompt(dot)com>
> To: "Richard Huxton" <dev(at)archonet(dot)com>
> Cc: <pgsql-general(at)postgresql(dot)org>
> Sent: Thursday, March 08, 2007 8:00 AM
> Subject: Re: [GENERAL] OT: Canadian Tax Database
>
>
> > Richard Huxton wrote:
> >> http://www.thestar.com/News/article/189175
> >>
> >> "For instance, in some cases the field for the social insurance number
> >> was instead filled in with a birth date."
> >>
> >> Unbelievable. Sixty years of electronic computing, fifty years use in
> >> business and the "professionals" who built the tax system for a wealthy
> >> democratic country didn't use data types.
> >
> > This is Unbelievable? This is commonplace.
> >
> And due at least in part to government (and other institutions operated by
> damned fools) opting for the least expensive provider rather than paying for
> someone who actually knows what they're doing. Just as buying cheap junk
> always comes back to get you, hiring incompetent fools that don't know their
> ass from a hole in the ground will come back to get you too.
>
Come on, they don't hire incompetent fools. The hire the people
they need to fill their quota regardless of how well trained
and experienced they are. I am not saying that non white males
are in any way less competent than white males, but by removing
them from the pool does not make things better. The biggest
problem with quotas is not hiring less qualified staff, it is
that less qualified staff know why they were hired and know that
they are very unlikely to be fired, so they have little incentive
to work hard or attempt to do their best, they can always fail
upwards.

...snip...

--
Guy Fraser
Network Administrator
The Internet Centre
1-888-450-6787
(780)450-6787

>> And due at least in part to government (and other institutions operated
>> by
>> damned fools) opting for the least expensive provider rather than paying
>> for
>> someone who actually knows what they're doing. Just as buying cheap junk
>> always comes back to get you, hiring incompetent fools that don't know
>> their
>> ass from a hole in the ground will come back to get you too.
>
> What you describe is a hundred times better than the reality... most of
> them actually get _expensive_ junk with some kick-back ;-)
>
I concede.

You're right.

I recall being told by one project manager I knew years ago who had an
opportunity to create a bid for an RFP issued by Transport Canada (long long
ago). He refuse, so his employer prepared the bid. He refused because the
RFP was a joke. There were absolutely no functional requirements, nor
non-functional requirements, identified in the RFP. His company didn't get
the contract, but being a bidder they did see the winning bid. It was just
as ludicrous! It, too, failed to identify any requirements, and so it did
not actually promise to deliver anything, working or not! They would have
satisfied the terms of their contract if, after a few years, and hundreds of
man-years, they walked away without delivering anything. That tragedy cost
Canada hundreds of millions, if not billions, of dollars (I don't know if
any final accounting was ever done - that would be opposed by the "civil
servants" responsible lest they should be criticised for their
incompetence), and ultimately nothing was delivered because the next elected
government cancelled the project, refusing to through more money into their
opposition's money pit - they prefer, of course, to through it into money
pits created by their political supporters. The decisions to create the
project, and to cancel it, were political, but the incompetence really
responsible for it was lower done within the ranks of the civil service.
The project could have delivered something good had the civil servants
involved been competent! Similar nonsense happened with the firearms
registry. For most of the early history of it, the software systems used
where so bad, and inappropriate, that the clerks that had to use it could
have been ten times more productive if they had the use of properly designed
and implemented software. I can not understand how it became so
outrageously expensive when the real needs for it were so simple (relative
to products I have worked on). I'll bet real, genuinely capable, software
engineers could have delivered a gold and platinum plated product for just a
few million dollars; nothing really relative to what it ended up costing us.

I know contractors that refuse to do business with the government because of
this sort of nonsense, and I know, from discussions with ex-civil servants,
that such incompetence is the norm in government. I know engineers who have
been burned by government by investing a fortune in new products or
services, and then educating relevant civil servants WRT to the new science
or technology involved, only to find these same civil servants give
contracts to provide the new product or service to incompetent bozos who
didn't know the first thing about it. They just happened to be cheaper.
Why waste time and money developing a product or service that is really
relevant only to government when the risk of such unethical conduct by
government is so high?

I don't support anyone out there can describe a project or three where
things were done right, to provide a cure for the depressing and
discouraging nature of what this thread has turned out to be?

Cheers

Ted

On Thursday 08 March 2007 08:15, "Ted Byers" <r(dot)ted(dot)byers(at)rogers(dot)com> wrote:
> They would have satisfied the terms of their contract
> if, after a few years, and hundreds of man-years, they walked away
> without delivering anything. That tragedy cost Canada hundreds of
> millions, if not billions, of dollars

It didn't happen to be a gun owners' registry, perhaps?

(fellow Canadians will understand :)

--
Opportunity is missed by most people because it is dressed in overalls and
looks like work. - Thomas Edison

----- Original Message -----
From: "Alan Hodgson" <ahodgson(at)simkin(dot)ca>
To: <pgsql-general(at)postgresql(dot)org>
Sent: Thursday, March 08, 2007 11:32 AM
Subject: Re: [GENERAL] OT: Canadian Tax Database

> On Thursday 08 March 2007 08:15, "Ted Byers" <r(dot)ted(dot)byers(at)rogers(dot)com>
> wrote:
>> They would have satisfied the terms of their contract
>> if, after a few years, and hundreds of man-years, they walked away
>> without delivering anything. That tragedy cost Canada hundreds of
>> millions, if not billions, of dollars
>
> It didn't happen to be a gun owners' registry, perhaps?
>
> (fellow Canadians will understand :)
>
No. This predated that fiasco by more than ten years. In fact, had it been
done right, it would have been a much much larger project than the registry.

Ted

>> > Richard Huxton wrote:
>> >> http://www.thestar.com/News/article/189175
>> >>
>> >> "For instance, in some cases the field for the social insurance number
>> >> was instead filled in with a birth date."
>> >>
>> >> Unbelievable. Sixty years of electronic computing, fifty years use in
>> >> business and the "professionals" who built the tax system for a
>> >> wealthy
>> >> democratic country didn't use data types.
>> >
>> > This is Unbelievable? This is commonplace.
>> >
>> And due at least in part to government (and other institutions operated
>> by
>> damned fools) opting for the least expensive provider rather than paying
>> for
>> someone who actually knows what they're doing. Just as buying cheap junk
>> always comes back to get you, hiring incompetent fools that don't know
>> their
>> ass from a hole in the ground will come back to get you too.
>>
> Come on, they don't hire incompetent fools. The hire the people

You CAN'T be serious! Have you ever dealt with them or with the
consequences of their incompetence?

> they need to fill their quota regardless of how well trained
> and experienced they are. I am not saying that non white males
> are in any way less competent than white males, but by removing
> them from the pool does not make things better. The biggest
> problem with quotas is not hiring less qualified staff, it is
> that less qualified staff know why they were hired and know that
> they are very unlikely to be fired, so they have little incentive
> to work hard or attempt to do their best, they can always fail
> upwards.
>
What does this have to do with anything? No one here, except you, has said
anything about the profile of the people involved WRT race, gender,
religion, &c. Nor has anyone said anything about "qualifications". The
only thing that has been said is that, based on what is seen in the "work",
the people responsible for that work must be incompetent. It is an
inference based on what is seen in what has been done and has nothing to do
with any of the prohibited grounds for discrimination used as excuses for
affirmative action. And yes, I have seen cases where less qualified, even
unqualified, people have been hired as a result of these affirmative action
initiatives (and I have been told, by HR personelle in government, that
certain favoured groups are deemed to be superior to white men, even if the
favoured party has no education nor experience and the latter have earned
doctorates and decades of experience), but no one has said anything about
such people being employed on the projects to which I referred. But this is
an aspect of our present society that is bound to degenerate into a flame
war, launched by the politically correct, so we ought to say little, or even
leave it alone. Those in power tend to be vicious, especially when there
are no effective checks on their conduct and no consequences for what they
do.

Cheers

Ted

Since this thread has already degraded, I'll offer my two cents. The
biggest screw ups in US history have been instigated by groups of
privileged White men. I know my name may sound otherwise, but I'm a
White American male, so I'm not pointing the finger at another group.
Let's see, Enron, Arthur Anderson, the entire Bush Administration and
its fiascos in Iraq, Katrina, foreign policy in general, etc. I've
worked for large, major IT providers and I can tell you that
incompetency shows no racial or ethnic boundaries. It tends to exist in
large, politically connected, no bid contractors, not low bid
contractors or ones who benefited from affirmative action.

Ted Byers wrote:
>>> > Richard Huxton wrote:
>>> >> http://www.thestar.com/News/article/189175
>>> >>
>>> >> "For instance, in some cases the field for the social insurance
>>> number
>>> >> was instead filled in with a birth date."
>>> >>
>>> >> Unbelievable. Sixty years of electronic computing, fifty years
>>> use in
>>> >> business and the "professionals" who built the tax system for a
>>> >> wealthy
>>> >> democratic country didn't use data types.
>>> >
>>> > This is Unbelievable? This is commonplace.
>>> >
>>> And due at least in part to government (and other institutions
>>> operated by
>>> damned fools) opting for the least expensive provider rather than
>>> paying for
>>> someone who actually knows what they're doing. Just as buying cheap
>>> junk
>>> always comes back to get you, hiring incompetent fools that don't
>>> know their
>>> ass from a hole in the ground will come back to get you too.
>>>
>> Come on, they don't hire incompetent fools. The hire the people
>
> You CAN'T be serious! Have you ever dealt with them or with the
> consequences of their incompetence?
>
>> they need to fill their quota regardless of how well trained
>> and experienced they are. I am not saying that non white males
>> are in any way less competent than white males, but by removing
>> them from the pool does not make things better. The biggest
>> problem with quotas is not hiring less qualified staff, it is
>> that less qualified staff know why they were hired and know that
>> they are very unlikely to be fired, so they have little incentive
>> to work hard or attempt to do their best, they can always fail
>> upwards.
>>
> What does this have to do with anything? No one here, except you, has
> said anything about the profile of the people involved WRT race,
> gender, religion, &c. Nor has anyone said anything about
> "qualifications". The only thing that has been said is that, based on
> what is seen in the "work", the people responsible for that work must
> be incompetent. It is an inference based on what is seen in what has
> been done and has nothing to do with any of the prohibited grounds for
> discrimination used as excuses for affirmative action. And yes, I
> have seen cases where less qualified, even unqualified, people have
> been hired as a result of these affirmative action initiatives (and I
> have been told, by HR personelle in government, that certain favoured
> groups are deemed to be superior to white men, even if the favoured
> party has no education nor experience and the latter have earned
> doctorates and decades of experience), but no one has said anything
> about such people being employed on the projects to which I referred.
> But this is an aspect of our present society that is bound to
> degenerate into a flame war, launched by the politically correct, so
> we ought to say little, or even leave it alone. Those in power tend
> to be vicious, especially when there are no effective checks on their
> conduct and no consequences for what they do.
>
> Cheers
>
> Ted
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org/

On Thu, 2007-03-08 at 10:15, Ted Byers wrote:

>
> I recall being told by one project manager I knew years ago who had an
> opportunity to create a bid for an RFP issued by Transport Canada (long long
> ago). He refuse, so his employer prepared the bid. He refused because the
> RFP was a joke. There were absolutely no functional requirements, nor

Your story reminded me of a dear friend who works for the department of
the interior here in the US who routinely was dressed down for writing
functional, reliable software quickly and with a minimum of bugs and
fuss. He made all the other people in his office feel bad.

sigh.

Thank God the DOI is inefficient. If they were good at what they do,
which is generally malicious, we'd all be in trouble.

> Your story reminded me of a dear friend who works for the department of
> the interior here in the US who routinely was dressed down for writing
> functional, reliable software quickly and with a minimum of bugs and
> fuss. He made all the other people in his office feel bad.
>
> sigh.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

Omar Eljumaily <omar2(at)omnicode(dot)com> writes:
> Thank God the DOI is inefficient. If they were good at what they do,
> which is generally malicious, we'd all be in trouble.

Guys, this was off-topic to start with ... if you'd like to argue
politics please take it to some other list ...

regards, tom lane

In article <45EFE705(dot)2030104(at)archonet(dot)com>,
Richard Huxton <dev(at)archonet(dot)com> wrote:
% http://www.thestar.com/News/article/189175
%
% "For instance, in some cases the field for the social insurance number
% was instead filled in with a birth date."
%
% Unbelievable. Sixty years of electronic computing, fifty years use in
% business and the "professionals" who built the tax system for a wealthy
% democratic country didn't use data types.

To be fair, this is not "the tax system". It's a staging database
used for electronic filing, and it's pretty common to use typeless
databases in the first stage of that sort of application.
--

Patrick TJ McPhee
North York Canada
ptjm(at)interlog(dot)com

Sorry everyone, my bad, but I should have expected it.

I was not denigrating anyone, if you actually read what I
said you can not conclude that I was. My entire point was
that the Government does not hire the best qualified hardest
working people regardless of their sex, culture, origin
or any other non work related issue. I personally hire
and work with very qualified people who are very diverse
in non work related factors, and I feel it is advantageous.

Like Ted I have also been told that due to my non work
related features, that I need not apply. I have also been
in a position to be asked to resolve issues created by the
person who was hired, while working elsewhere for a company
providing services to that department. Not even a government
employee gets paid the $120/Hr it cost to have me resolve
the problem, so wouldn't it be more efficient to just hire
qualified staff, not necessarily me, but someone who was
actually the best person for the job?

Government Hiring = Quotas = Suck

On Thu, 2007-03-08 at 11:06 -0800, Omar Eljumaily wrote:
> Since this thread has already degraded, I'll offer my two cents. The
> biggest screw ups in US history have been instigated by groups of
> privileged White men. I know my name may sound otherwise, but I'm a
> White American male, so I'm not pointing the finger at another group.
> Let's see, Enron, Arthur Anderson, the entire Bush Administration and
> its fiascos in Iraq, Katrina, foreign policy in general, etc. I've
> worked for large, major IT providers and I can tell you that
> incompetency shows no racial or ethnic boundaries. It tends to exist in
> large, politically connected, no bid contractors, not low bid
> contractors or ones who benefited from affirmative action.
>
>
> Ted Byers wrote:
> >>> > Richard Huxton wrote:
> >>> >> http://www.thestar.com/News/article/189175
> >>> >>
> >>> >> "For instance, in some cases the field for the social insurance
> >>> number
> >>> >> was instead filled in with a birth date."
> >>> >>
> >>> >> Unbelievable. Sixty years of electronic computing, fifty years
> >>> use in
> >>> >> business and the "professionals" who built the tax system for a
> >>> >> wealthy
> >>> >> democratic country didn't use data types.
> >>> >
> >>> > This is Unbelievable? This is commonplace.
> >>> >
> >>> And due at least in part to government (and other institutions
> >>> operated by
> >>> damned fools) opting for the least expensive provider rather than
> >>> paying for
> >>> someone who actually knows what they're doing. Just as buying cheap
> >>> junk
> >>> always comes back to get you, hiring incompetent fools that don't
> >>> know their
> >>> ass from a hole in the ground will come back to get you too.
> >>>
> >> Come on, they don't hire incompetent fools. The hire the people
> >
> > You CAN'T be serious! Have you ever dealt with them or with the
> > consequences of their incompetence?
> >
> >> they need to fill their quota regardless of how well trained
> >> and experienced they are. I am not saying that non white males
> >> are in any way less competent than white males, but by removing
> >> them from the pool does not make things better. The biggest
> >> problem with quotas is not hiring less qualified staff, it is
> >> that less qualified staff know why they were hired and know that
> >> they are very unlikely to be fired, so they have little incentive
> >> to work hard or attempt to do their best, they can always fail
> >> upwards.
> >>
> > What does this have to do with anything? No one here, except you, has
> > said anything about the profile of the people involved WRT race,
> > gender, religion, &c. Nor has anyone said anything about
> > "qualifications". The only thing that has been said is that, based on
> > what is seen in the "work", the people responsible for that work must
> > be incompetent. It is an inference based on what is seen in what has
> > been done and has nothing to do with any of the prohibited grounds for
> > discrimination used as excuses for affirmative action. And yes, I
> > have seen cases where less qualified, even unqualified, people have
> > been hired as a result of these affirmative action initiatives (and I
> > have been told, by HR personelle in government, that certain favoured
> > groups are deemed to be superior to white men, even if the favoured
> > party has no education nor experience and the latter have earned
> > doctorates and decades of experience), but no one has said anything
> > about such people being employed on the projects to which I referred.
> > But this is an aspect of our present society that is bound to
> > degenerate into a flame war, launched by the politically correct, so
> > we ought to say little, or even leave it alone. Those in power tend
> > to be vicious, especially when there are no effective checks on their
> > conduct and no consequences for what they do.
> >
> > Cheers
> >
> > Ted
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> > http://archives.postgresql.org/
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>
--
Guy Fraser
Network Administrator
The Internet Centre
1-888-450-6787
(780)450-6787

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/09/07 00:12, Patrick TJ McPhee wrote:
[snip]
>
> To be fair, this is not "the tax system". It's a staging database
> used for electronic filing, and it's pretty common to use typeless
> databases in the first stage of that sort of application.

Why? Why not filter out the obvious errors AEAP[*] in the data stream?

[*] As Early As Possible

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFF8h/MS9HxQb37XmcRAuF2AKCL7qA4NW6O5foRtQW3uDzHtg1FOQCgwnYh
BowoxdVktlw9VoqBSXON9MU=
=RNHI
-----END PGP SIGNATURE-----

Tom, I promise this isn't a political statement, even though it's on the
same thread.

I'm curious what people think about the following statement considering
the database typing talk being brought up here. My experience is that
more times than not I have to put data validation in my client code even
when it's available on the server, if for no other reason that users
don't understand what foreign key violation, etc messages mean. It begs
the question of whether it's really necessary on the server or not.
SQLite seems to take the position that it isn't since there is no
referential integrity and the following. To be honest, there's a lot of
power in the ability to view everything as a string, with of course
proper data validation.

http://www.sqlite.org/datatypes.html

>>>SQLite is "typeless". This means that you can store any kind of data
you want in any column of any table, regardless of the declared datatype
of that column. (See the one exception to this rule in section 2.0
below.) This behavior is a feature, not a bug. A database is suppose to
store and retrieve data and it should not matter to the database what
format that data is in. The strong typing system found in most other SQL
engines and codified in the SQL language spec is a misfeature - it is an
example of the implementation showing through into the interface. SQLite
seeks to overcome this misfeature by allowing you to store any kind of
data into any kind of column and by allowing flexibility in the
specification of datatypes.<<<

Patrick TJ McPhee wrote:
> To be fair, this is not "the tax system". It's a staging database
> used for electronic filing, and it's pretty common to use typeless
> databases in the first stage of that sort of application.
>

----- Original Message -----
From: "omar" <omar2(at)omnicode(dot)com>
To: <pgsql-general(at)postgresql(dot)org>
Sent: Friday, March 09, 2007 10:40 PM
Subject: Re: [GENERAL] OT: Canadian Tax Database

> Tom, I promise this isn't a political statement, even though it's on the
> same thread.
> I'm curious what people think about the following statement considering
> the database typing talk being brought up here. My experience is that
> more times than not I have to put data validation in my client code even
> when it's available on the server, if for no other reason that users don't
> understand what foreign key violation, etc messages mean. It begs the
> question of whether it's really necessary on the server or not. SQLite
> seems to take the position that it isn't since there is no referential
> integrity and the following. To be honest, there's a lot of power in the
> ability to view everything as a string, with of course proper data
> validation.

This risk of this is far too high. Treating everything as a string is,
IMHO, a very bad idea.

There are, especially for a web application, numerous forms of attack, so I
routinely provide code for client side validation, server side validation
(in a web app or in filters that process the data before providing it to
whatever is going to do something useful with the data. this includes
designing stored procedures to receive, and validate, data before the data
is stored in the database. On the client side, the main benefit is to
ensure the user doesn't miss anything that is necessary and that he enters
valid data. If the user is malicious, and wants to try a SQL injection
attack, nothing you do on the client side can prevent him from creating his
own version of your page bypassing all of your client side validation code.
And it is possible for a scoundrel to try a man in the middle attack
(intercepting a transaction mid stream and trying, e.g., a SQL injection
attack). So even with client side validation, server side validation is
absolutely essential. I like Perl for that, but it can be done in your
favourite programming language. And it can be done in .NET also, if you
prefer.

Maybe I am paranoid, but whether I am writing code to be run at the very
back end, or the very front end, or anywhere between the two, my preference
is to validate the data that specific object has received before I do
anything with it. That is key in secure application development. You
generally assume that your system, and any component therein, can been
compromised so you program on the assumption that it can be compromised
somewhere and write code that minimizes or eliminates the damage that can be
done if some component anywhere else in the system has been compromised.
Just 'coz I'm paranoid doesn't mean they're not out to get me. ;-) I
value really good system administrators who go the extra mile to make
intranets and systems as secure as humanly possible, but as an application
developer, I never assume they have not overlooked something. Instead, I
assume the opposite and that therefore, they got everything wrong and that
the intranet and every server in it either has been compromised or will soon
be compromised, and I therefore try to minimize the risk of damage or
violation of data confidentiality or security in a network or on a system
that has been compromised. I know perfection is not possible, but I hope we
can make it too expensive for a cyber criminal to get what he wants
illegally. If we make his cost greater than his potential return, he should
rationally move on to easier targets.

Cheers

Ted

omar <omar2(at)omnicode(dot)com> writes:

> I'm curious what people think about the following statement considering the
> database typing talk being brought up here. My experience is that more times
> than not I have to put data validation in my client code even when it's
> available on the server, if for no other reason that users don't understand
> what foreign key violation, etc messages mean. It begs the question of
> whether it's really necessary on the server or not. SQLite seems to take the
> position that it isn't since there is no referential integrity and the
> following. To be honest, there's a lot of power in the ability to view
> everything as a string, with of course proper data validation.

I believe that data validation is essential at the server side. The ideal
situation to me is something like data validation on server, errors /
exceptions being risen and then catched by the client code that will translate
them to a suitable message.

Inserting data validation on client side helps with simple input and eliminate
the average number of roundtrips needed for getting the data stored, but
shouldn't be the only validation done.

--
Jorge Godoy <jgodoy(at)gmail(dot)com>

On Sat, 2007-03-10 at 08:07 -0300, Jorge Godoy wrote:
> omar <omar2(at)omnicode(dot)com> writes:
>
> > I'm curious what people think about the following statement considering the
> > database typing talk being brought up here. My experience is that more times
> > than not I have to put data validation in my client code even when it's
> > available on the server, if for no other reason that users don't understand
> > what foreign key violation, etc messages mean. It begs the question of
> > whether it's really necessary on the server or not. SQLite seems to take the
> > position that it isn't since there is no referential integrity and the
> > following. To be honest, there's a lot of power in the ability to view
> > everything as a string, with of course proper data validation.
>
> I believe that data validation is essential at the server side. The ideal
> situation to me is something like data validation on server, errors /
> exceptions being risen and then catched by the client code that will translate
> them to a suitable message.
>
> Inserting data validation on client side helps with simple input and eliminate
> the average number of roundtrips needed for getting the data stored, but
> shouldn't be the only validation done.
>
I completely agree and would add that I also prefer to use server
side session cookies to validate the authenticity of the remote
user. Stopping man in the middle, client spoofing and SQL injection
are all good reasons to use multiple levels data and remote user
verification. One other good trick is to use table permissions to
only permit read only database access, and in many cases from a view
not the actual table.

Paranoia and systems administration/development go well together
in my humble opinion. Keeping the server and data safe is a big
part of our responsibility.

I give cheers to PostgreSQL developers, in developing the excellent
tools they have provided thus far and would like to encourage them
to keep up the excellent trend.

On Fri, 2007-03-09 at 19:40 -0800, omar wrote:
> below.) This behavior is a feature, not a bug. A database is suppose to
> store and retrieve data and it should not matter to the database what
> format that data is in. The strong typing system found in most other SQL

In my opinion, this is the most important -- and most misguided --
statement that you quoted.

Storing and retrieving arbitrary (string,binary) data is the most
ancient form of a database. Relational databases are much more.

Let me make a comparison --

In SQLite you:
* store variable A in database with key X.
* store variable B in database with key Y.
* ask for some data with key X, and put it in a variable C
* make assumptions about what the data is; i.e. part of a string, an
int, or another key
* do something useful with variable C (don't look in the database for
key C if C is a color and not a key) and put the result in variable D
* store variable D in database with key Z and keep track of key Z so
that you can find it when you need it later

In Assembly Language you:
* store register A in memory at address X.
* store register B in memory at address Y.
* ask for the data at address X, and put it in a register C
* make assumptions about what the data is; i.e. part of a string, an
int, or another key
* do something useful with register C (don't look in the memory at
address C if C is a part of a string and not a memory address) and put
the result in register D
* store register D in memory at address Z and keep track of address Z so
that you can find it when you need it later

See? They're both "simple." But at some point programmers decided they
wanted the "complexity" of data types, constraints, etc. And it's
actually more important with a database because the data is persistent,
and you will end up converting and rewriting the on-disk data to support
each new feature that requires a minor difference in data layout.

I could go on, but there is a lot of good info out there already. Look
for books by Chris Date, a.k.a. C.J. Date.

These links might help:

http://en.wikipedia.org/wiki/Relational_database
http://en.wikipedia.org/wiki/Relational_algebra
http://www.postgresql.org/docs/8.2/static/index.html

Regards,
Jeff Davis