Re: [Gluster-devel] Change in glusterfs[master]: Transparent data encryption and metadata authentication in t...

[Edward Shishkin]

Hi all,

So, here is the "all-in-one-translator" version represented by the
Patch Set #2 at review.gluster.org/4667

Everything has been addressed except encryption in NFS mounts (see
next mail for details). That is:

. New design of EOF (end-of-file) handling;
. No oplock translator on the server side;
. All locks are acquired/released by the crypt translator;
. Now we can encrypt srtiped and(or) replicated volumes.

Common comments.

In the new design all files on the server are "padded", whereas the
real file size is stored as xattr. So we introduce a special layer
in the crypt translator, which performs file size translations: every
time when any callback returns struct iatt, we update its ia_size
with the real (non-padded) value.

The most unpleasant thing in this new design is FOP->readdirp_cbk():
in this case we need N translations, i.e. N calls to the server (N is
number of directory entries).

To perform translations we spawn N children. We need a valid list of
dirents after returning from FOP->readdirp_cbk() of previous
translator, but we don't want to create a copy of this list (which
can be large enough). For this reason we introduce a reference counter
in struct gf_dirent_t and allocate dynamic structures gf_dirent_t
(instead of on-stack ones), see respective changes in

./libglusterfs/src/gf-dirent.c
./libglusterfs/src/gf-dirent.h
./xlators/cluster/dht/src/dht-common.c
./xlators/protocol/client/src/client-rpc-fops.c

Thanks,
Edward.


On Mon, 14 Oct 2013 14:27:01 -0700
Anand Avati <address@hidden> wrote:

> Edward,
> It looks like this patch requires a higher version of openssl (I
> recall you have mentioned before that that dependency was on version
> 1.0.1c? I checked yum update on the build server and the latest
> available version is 1.0.0-27. Is there a "clean" way to get the
> right version of openssl to a RHEL/CENTOS-6.x server?
> 
> Also note that the previous submission of the patch was at 
> http://review.gluster.org/4667. The recent on 
> (http://review.gluster.org/6086) has a different Change-Id: in the 
> commit log. It will be good if you can re-submit the patch with the
> old Change-Id (and abandon #6086) so that we can maintain the history
> of resubmission and the old work on records.
> 
> Thanks!
> Avati
> 
> On 10/14/2013 07:26 AM, Edward Shishkin (Code Review) wrote:
> > Edward Shishkin has uploaded a new change for review.
> >
> >    http://review.gluster.org/6086
> >
> >
> > Change subject: Transparent data encryption and metadata
> > authentication in the systems with non-trusted server (take
> > II) ......................................................................
> >
> > Transparent data encryption and metadata authentication
> > in the systems with non-trusted server (take II)
> >
> > This new functionality can be useful in various cloud technologies.
> > It is implemented via a special encryption/crypt translator, which
> > works on the client side and performs encryption and authentication;
> >
> >                1. Class of supported algorithms
> >
> > The crypt translator can support any atomic symmetric block cipher
> > algorithms (which require to pad plain/cipher text before performing
> > encryption/decryption transform (see glossary in atom.c for
> > definitions). In particular, it can support algorithms with the EOF
> > issue (which require to pad the end of file by extra-data).
> >
> > Crypt translator performs translations
> > user -> (offset, size) -> (aligned-offset, padded-size) ->server
> > (and backward), and resolves individual FOPs (write(), truncate(),
> > etc) to read-modify-write sequences.
> >
> > A volume can contain files encrypted by different algorithms of the
> > mentioned class. To change some option value just reconfigure the
> > volume.
> >
> > Currently only one algorithm is supported: AES_XTS.
> >
> > Example of algorithms, which can not be supported by the crypt
> > translator:
> >
> > 1. Asymmetric block cipher algorithms, which inflate data, e.g. RSA;
> > 2. Symmetric block cipher algorithms with inline MACs for data
> >     authentication.
> >
> >                     2. Implementation notes.
> >
> > a) Atomic algorithms
> >
> > Since any process in a stackable file system manipulates with local
> > data (which can be obsoleted by local data of another process), any
> > atomic cipher algorithm without proper support can lead to non-POSIX
> > behavior. To resolve the "collisions" we introduce locks: before
> > performing FOP->read(), FOP->write(), etc. the process should first
> > lock the file.
> >
> > b) Algorithms with EOF issue
> >
> > Such algorithms require to pad the end of file with some extra-data.
> > Without proper support this will result in losing information about
> > real file size. Keeping a track of real file size is a
> > responsibility of the crypt translator. A special extended
> > attribute with the name "trusted.glusterfs.crypt.att.size" is used
> > for this purpose. All files contained in bricks of encrypted volume
> > do have "padded" sizes.
> >
> >                    3. Non-trusted servers and
> >                       Metadata authentication
> >
> > We assume that server, where user's data is stored on is
> > non-trusted. It means that the server can be subjected to various
> > attacks directed to reveal user's encrypted personal data. We
> > provide protection against such attacks.
> >
> > Every encrypted file has specific private attributes (cipher
> > algorithm id, atom size, etc), which are packed to a string
> > (so-called "format string") and stored as a special extended
> > attribute with the name "trusted.glusterfs.crypt.att.cfmt". We
> > protect the string from tampering. This protection is mandatory,
> > hardcoded and is always on. Without such protection various attacks
> > (based on extending the scope of per-file secret keys) are possible.
> >
> > Our authentication method has been developed in tight collaboration
> > with Red Hat security team and is implemented as "metadata loader of
> > version 1" (see file metadata.c). This method is NIST-compliant and
> > is based on checking 8-byte per-hardlink MACs created(updated) by
> > FOP->create(), FOP->link(), FOP->unlink(), FOP->rename() by the
> > following unique entities:
> >
> > . file (hardlink) name;
> > . verified file's object id (gfid).
> >
> > Every time, before manipulating with a file, we check it's MACs at
> > FOP->open() time. Some FOPs don't require a file to be opened (e.g.
> > FOP->truncate()). In such cases the crypt translator opens the file
> > mandatory.
> >
> >                          4. Generating keys
> >
> > Unique per-file keys are derived by NIST-compliant methods from the
> >
> > a) parent key;
> > b) unique verified object-id of the file (gfid);
> > Per-volume master key, provided by user at mount time is in the root
> > of this "tree of keys".
> >
> > Those keys are used to:
> >
> > 1) encrypt/decrypt file data;
> > 2) encrypt/decrypt file metadata;
> > 3) create per-file and per-link MACs for metadata authentication.
> >
> >                            5. Instructions
> >                   Getting started with crypt translator
> >
> > Example:
> >
> > 1) Create a volume "myvol" by specifying the option "encrypt":
> >
> >     # gluster volume create myvol encrypt pepelac:/vols/xvol
> >
> > 2) Set location (absolute pathname) of your master key:
> >
> >     # gluster volume set myvol encryption.master-key /home/me/mykey
> >
> > 3) Set other options to override default options, if needed.
> >
> > 4) On the client side make sure that the file /home/me/mykey exists
> >     and contains proper per-volume master key (that is 256-bit AES
> >     key). This key has to be in hex form, i.e. should be represented
> >     by 64 symbols from the set  {'0', ..., '9', 'a', ..., 'f'}.
> >     The key should start at the beginning of the file. All symbols
> > at offsets >= 64 are ignored.
> >
> > 5) Mount the volume "myvol" on the client side:
> >
> >     # glusterfs --volfile-server=pepelac --volfile-id=myvol /mnt
> >
> >     After successful mount the file which contains master key may be
> >     removed. NOTE: Keeping the master key between mount sessions is
> > in user's competence.
> >
> > **********************************************************************
> >
> > WARNING! Losing the master key will make content of all regular
> > files inaccessible. Mount with improper master key allows to access
> > content of directories: file names are not encrypted.
> >
> > **********************************************************************
> >
> >                 6. Options of crypt translator
> >
> > 1) "master-key": specifies location (absolute pathname) of the file
> >     which contains per-volume master key. There is no default
> > location for master key.
> >
> > 2) "data-key-size": specifies size of per-file key for data
> > encryption Possible values:
> >     . "256" default value
> >     . "512"
> >
> > 3) "block-size": specifies atom size. Possible values:
> >     . "512"
> >     . "1024"
> >     . "2048"
> >     . "4096" default value;
> >
> >                         7. Test cases
> >
> > Any workload, which involves the following file operations:
> >
> > ->create();
> > ->open();
> > ->readv();
> > ->writev();
> > ->truncate();
> > ->ftruncate();
> > ->link();
> > ->unlink();
> > ->rename();
> > ->readdirp().
> >
> >                          8. TODOs:
> >
> > 1) Currently size of IOs issued by crypt translator is restricted
> >     by block_size (4K by default). We can use larger IOs to improve
> >     performance.
> >
> > Change-Id: Idc3523a8752888d3dd10f4fe71aa38dfc53d9ded
> > Signed-off-by: Edward Shishkin <address@hidden>
> > ---
> > M cli/src/cli-cmd-parser.c
> > M configure.ac
> > M doc/gluster.8
> > M libglusterfs/src/gf-dirent.c
> > M libglusterfs/src/gf-dirent.h
> > M xlators/cluster/dht/src/dht-common.c
> > M xlators/encryption/Makefile.am
> > A xlators/encryption/crypt/Makefile.am
> > A xlators/encryption/crypt/src/Makefile.am
> > A xlators/encryption/crypt/src/atom.c
> > A xlators/encryption/crypt/src/crypt-common.h
> > A xlators/encryption/crypt/src/crypt-mem-types.h
> > A xlators/encryption/crypt/src/crypt.c
> > A xlators/encryption/crypt/src/crypt.h
> > A xlators/encryption/crypt/src/data.c
> > A xlators/encryption/crypt/src/keys.c
> > A xlators/encryption/crypt/src/metadata.c
> > A xlators/encryption/crypt/src/metadata.h
> > M xlators/mgmt/glusterd/src/glusterd-store.c
> > M xlators/mgmt/glusterd/src/glusterd-store.h
> > M xlators/mgmt/glusterd/src/glusterd-volgen.c
> > M xlators/mgmt/glusterd/src/glusterd-volume-ops.c
> > M xlators/mgmt/glusterd/src/glusterd-volume-set.c
> > M xlators/mgmt/glusterd/src/glusterd.h
> > M xlators/protocol/client/src/client-rpc-fops.c
> > 25 files changed, 8,733 insertions(+), 28 deletions(-)
> >
> >
> >    git pull ssh://git.gluster.org/glusterfs refs/changes/86/6086/1
> >
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/gluster-devel