Re: [PATCH] pgcrypto: pgp_encrypt (v2)

Finally, here is pgp_encrypt()/pgp_decrypt() - implementation
of password-based encryption from RFC2440 (OpenPGP).

The goal of this code is to be more featureful encryption solution
than current encrypt(), which only functionality is running cipher
over data.

Compared to encrypt(), pgp_encrypt() does following:

* It uses the equvialent of random Inital Vector to get cipher
into random state before it processes user data
* Stores SHA-1 of the data into result so any modification
will be detected.
* Remembers if data was text or binary - thus it can decrypt
to/from text data. This was a major nuisance for encrypt().
* Stores info about used algorithms with result, so user needs
not remember them - more user friendly!
* Uses String2Key algorithms (similar to crypt()) with random salt
to generate full-length binary key to be used for encrypting.
* Uses standard format for data - you can feed it to GnuPG, if needed.

Optional features (off by default):

* Can use separate session key - user data will be encrypted
with totally random key, which will be encrypted with S2K
generated key and attached to result.
* Data compression with zlib.
* Can convert between CRLF<->LF line-endings - to get fully
RFC2440-compliant behaviour. This is off by default as
pgcrypto does not know the line-endings of user data.

Interface is simple:

pgp_encrypt(data text, key text) returns bytea
pgp_decrypt(data text, key text) returns text
pgp_encrypt_bytea(data bytea, key text) returns bytea
pgp_decrypt_bytea(data bytea, key text) returns bytea

To change parameters (cipher, compression, mdc):

pgp_encrypt(data text, key text, parms text) returns bytea
pgp_decrypt(data text, key text, parms text) returns text
pgp_encrypt_bytea(data bytea, key text, parms text) returns bytea
pgp_decrypt_bytea(data bytea, key text, parms text) returns bytea

Parameter names I lifted from gpg:

pgp_encrypt('message', 'key', 'compress-algo=1,cipher-algo=aes256')

For text data, pgp_encrypt simply encrypts the PostgreSQL internal
data. This maps to RFC2440 data type 't' - 'extenally specified
encoding'. But this may cause problems if data is dumped and reloaded
into database which as different internal encoding. My next goal is
to implement data type 'u' - which means data is in UTF-8 encoding
by converting internal encoding to UTF-8 and back. And there wont
be any compatibility problems with current code, I think its ok to
submit this without UTF-8 support.

--
marko

PS. openssl/3des regression test fails, I'll send fix once
I understand why its happening.

On Mon, Jun 27, 2005 at 09:08:05PM -0400, Bruce Momjian wrote:
> Your patch has been added to the PostgreSQL unapplied patches list at:
>
> http://momjian.postgresql.org/cgi-bin/pgpatches
>
> It will be applied as soon as one of the PostgreSQL committers reviews
> and approves it.

Please use following updated patch instead.

It implements utf8 conversion, fixes couple of bugs and has
many code and comment cleanups.

--
marko

Marko Kreen wrote:
> Please use following updated patch instead.
>
> It implements utf8 conversion, fixes couple of bugs and has
> many code and comment cleanups.

The regression tests don't pass on my box. With the default Makefile,
there are a lot of errors WRT "no strong random source". After editing
the Makefile to make use the "random" device, I get the attached
regression.diffs.

While I understand the need to make sure people use a reasonably strong
crypto source, it would be nice if the regression tests passed out of
the box.

-Neil

Well, I needed to decide, whether it is realistic to implement
PGP public-key encryption someday, because then I better rename
the function 'pgp_encrypt' to something more specific immidiately.

But after hacking a while, I soon had working implementation,
so it seems silly to wait 8.2 for it. This is thanks to the
fact that for encryption I can ignore most of the OpenPGP
complex parts - signing and key-management. IMHO they don't
even make sense inside database, they need an end-user
application.

In short, the goal of this code is not to be full OpenPGP
implementation, but just provide more flexible encryption
than the usual symmetric encryption.

So here is updated pgp_encrypt patch, with support for both
symmetric and public-key encryption. Also, as Neil requested,
non-failing regression tests - unsupported parts are replaced
with empty test named '*-DISABLED'.

Note: it uses OpenSSL bignum code - so without OpenSSL
no public-key encryption. Look below for plans for
fixing this dependance.

New names for symmetric-key functions:

pgp_sym_encrypt(data, key)
pgp_sym_decrypt(data, key)

Public key functions:

pgp_pub_encrypt(data, key)
pgp_pub_decrypt(data, key)
pgp_pub_decrypt(data, key, psw)

Plus variants with 'arg' and for bytea.

Not mentioned in previous mails:

armor(bytea) -> text
dearmor(text) -> bytea

Apply and remove PGP Ascii Armor.

Implemented featureѕ:
* Elgamal-encryption keys - preferred public-key algo for OpenPGP.
* Password-protected secret keys.

Missing features (needs implementing ASAP):
* pgp_key_id() - Function to query the ID of a key, and
the key ID from encrypted packet - so user can implement
key rings on his own.

Maybe:
* As it does not parse sign packets, it does not support
recipient cipher preferences. It is not important for
planned use-case, but maybe it would be nice to have.

Does not support (and I see no need):
* RSA-encrypt and sign+encrypt keys - they are deprecated in
RFC, GnuPG does not even give option to generate them, so IMO
it's fine to ignore them.
* Picking a key from list of keys - accepting a keyring in
place of key.
* Several encryption subkeys under one key.
* Any form of signing - that also means no key integrity checks
(eg - does the subkey belong to master key).

Now, the code is significant in the following respect - the
module 'contrib/pgcrypto' is complete. As I said, I don't
think rest of the OpenPGP makes sense here, and there isn't
any other generally-useful crypto functionality that would be
good to have.

Still, there are various things to do:

1. Polishing of the new PGP code.
2. Rework documentation and add FAQ.
3. Try to extract OpenSSL and zlib settings from main
PostgreSQL config. Basically that means that ./configure
should put them to separate variables, not into main CFLAGS.
4. Include strong RNG. Yarrow, Fortuna, ?.
This is for both symmetricy and public-key encryption.
5. Include bignum code (only ops needed for Elgamal).
This is for public-key encrypt/decrypt.
6. Propose crypt(), gen_salt() into mainline for 8.2.
They are self-contained, except crypt-md5 depends on a MD5
implentation, which already exists in mainline.

Points 4 and 5 are suspicious - I'm not yet sure it can
be done in manageable way. But without those, the PGP
functions are unusable in default build. It may not be that
bad - Yarrow/Fortuna seem to be small pieces only requiring
external cipher and hash, which we have. And I could
drop parts of libtommath into separate directory. But
I don't think such things should be done in hurry.

In the end, I think the code can go into 8.1 in current state -
requiring OpenSSL. I'll look into dependencies in
8.2 timeframe.

--
marko

On Mon, Jul 04, 2005 at 04:59:58PM +1000, Neil Conway wrote:
> Marko Kreen wrote:
> >Please use following updated patch instead.
> >
> >It implements utf8 conversion, fixes couple of bugs and has
> >many code and comment cleanups.
>
> The regression tests don't pass on my box. With the default Makefile,
> there are a lot of errors WRT "no strong random source". After editing
> the Makefile to make use the "random" device, I get the attached
> regression.diffs.

I don't understand the regression of the dashes in armor test,
kinda seems you have different psql than in CVS.

But for the regression with 'random = dev', I'd say you did not
do 'make clean' after changing Makefile. It only changes flags
to random.c, so if you have existing random.o, it wont be
recompiled.

> While I understand the need to make sure people use a reasonably strong
> crypto source, it would be nice if the regression tests passed out of
> the box.

Look for the 'pgp_encrypt v3' mail I sent. There I use special
empty tests for disabled functionality, that succeed, but
hopefully show user through naming that the functionality
is missing.

--
marko

Your patch has been added to the PostgreSQL unapplied patches list at:

http://momjian.postgresql.org/cgi-bin/pgpatches

It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.

---------------------------------------------------------------------------

Marko Kreen wrote:
>
> Well, I needed to decide, whether it is realistic to implement
> PGP public-key encryption someday, because then I better rename
> the function 'pgp_encrypt' to something more specific immidiately.
>
> But after hacking a while, I soon had working implementation,
> so it seems silly to wait 8.2 for it. This is thanks to the
> fact that for encryption I can ignore most of the OpenPGP
> complex parts - signing and key-management. IMHO they don't
> even make sense inside database, they need an end-user
> application.
>
> In short, the goal of this code is not to be full OpenPGP
> implementation, but just provide more flexible encryption
> than the usual symmetric encryption.
>
> So here is updated pgp_encrypt patch, with support for both
> symmetric and public-key encryption. Also, as Neil requested,
> non-failing regression tests - unsupported parts are replaced
> with empty test named '*-DISABLED'.
>
> Note: it uses OpenSSL bignum code - so without OpenSSL
> no public-key encryption. Look below for plans for
> fixing this dependance.
>
> New names for symmetric-key functions:
>
> pgp_sym_encrypt(data, key)
> pgp_sym_decrypt(data, key)
>
> Public key functions:
>
> pgp_pub_encrypt(data, key)
> pgp_pub_decrypt(data, key)
> pgp_pub_decrypt(data, key, psw)
>
> Plus variants with 'arg' and for bytea.
>
> Not mentioned in previous mails:
>
> armor(bytea) -> text
> dearmor(text) -> bytea
>
> Apply and remove PGP Ascii Armor.
>
>
>
> Implemented feature?:
> * Elgamal-encryption keys - preferred public-key algo for OpenPGP.
> * Password-protected secret keys.
>
> Missing features (needs implementing ASAP):
> * pgp_key_id() - Function to query the ID of a key, and
> the key ID from encrypted packet - so user can implement
> key rings on his own.
>
> Maybe:
> * As it does not parse sign packets, it does not support
> recipient cipher preferences. It is not important for
> planned use-case, but maybe it would be nice to have.
>
> Does not support (and I see no need):
> * RSA-encrypt and sign+encrypt keys - they are deprecated in
> RFC, GnuPG does not even give option to generate them, so IMO
> it's fine to ignore them.
> * Picking a key from list of keys - accepting a keyring in
> place of key.
> * Several encryption subkeys under one key.
> * Any form of signing - that also means no key integrity checks
> (eg - does the subkey belong to master key).
>
>
> Now, the code is significant in the following respect - the
> module 'contrib/pgcrypto' is complete. As I said, I don't
> think rest of the OpenPGP makes sense here, and there isn't
> any other generally-useful crypto functionality that would be
> good to have.
>
>
> Still, there are various things to do:
>
> 1. Polishing of the new PGP code.
> 2. Rework documentation and add FAQ.
> 3. Try to extract OpenSSL and zlib settings from main
> PostgreSQL config. Basically that means that ./configure
> should put them to separate variables, not into main CFLAGS.
> 4. Include strong RNG. Yarrow, Fortuna, ?.
> This is for both symmetricy and public-key encryption.
> 5. Include bignum code (only ops needed for Elgamal).
> This is for public-key encrypt/decrypt.
> 6. Propose crypt(), gen_salt() into mainline for 8.2.
> They are self-contained, except crypt-md5 depends on a MD5
> implentation, which already exists in mainline.
>
>
> Points 4 and 5 are suspicious - I'm not yet sure it can
> be done in manageable way. But without those, the PGP
> functions are unusable in default build. It may not be that
> bad - Yarrow/Fortuna seem to be small pieces only requiring
> external cipher and hash, which we have. And I could
> drop parts of libtommath into separate directory. But
> I don't think such things should be done in hurry.
>
> In the end, I think the code can go into 8.1 in current state -
> requiring OpenSSL. I'll look into dependencies in
> 8.2 timeframe.
>
> --
> marko
>

[ Attachment, skipping... ]

>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:
> Your patch has been added to the PostgreSQL unapplied patches list

That is not the latest version of Marko's patch. But in any case, the
patch is not yet ready for application:

http://archives.postgresql.org/pgsql-patches/2005-07/msg00077.php

-Neil

On Tue, Jul 05, 2005 at 10:20:17AM +1000, Neil Conway wrote:
> Bruce Momjian wrote:
> >Your patch has been added to the PostgreSQL unapplied patches list
>
> That is not the latest version of Marko's patch.

Bruce got v3, thats indeed the latest.

Also, http://momjian.postgresql.org/cgi-bin/pgpatches shows v3.

> But in any case, the
> patch is not yet ready for application:
>
> http://archives.postgresql.org/pgsql-patches/2005-07/msg00077.php

Now I did fresh rebuild of CVS and played with it. Result
is that it is only partly my error. It just happens that I did
initdb with option '-E unicode'. Now, can anybody explain the
following difference:

========================================
$ psql -c '\l' template1; psql -c "select 'a\nxxxxxx'::text as
x;" template1
List of databases
Name | Owner | Encoding
--------------------+-------+-----------
contrib_regression | marko | SQL_ASCII
postgres | marko | SQL_ASCII
template0 | marko | SQL_ASCII
template1 | marko | SQL_ASCII
(4 rows)

x
----------
a
xxxxxx
(1 row)
========================================
$ psql -c '\l' template1; psql -c "select 'a\nxxxxxx'::text as
x;" template1
List of databases
Name | Owner | Encoding
--------------------+-------+----------
contrib_regression | marko | UTF8
postgres | marko | UTF8
template0 | marko | UTF8
template1 | marko | UTF8
(4 rows)

x
---
a
xxxxxx
(1 row)

=========================================

I can send new regression test for SQL_ASCII, but it still
would not work for all users.

--
marko