keyformat.txt               emacs, please switch to -*- org -*- mode
-------------


Some notes on the format of the secret keys used with gpg-agent.

* Location of keys

The secret keys[1] are stored on a per file basis in a directory below
the ~/.gnupg home directory.  This directory is named

   private-keys-v1.d

and should have permissions 700.

The secret keys are stored in files with a name matching the
hexadecimal representation of the keygrip[2] and suffixed with ".key".

* Extended Private Key Format

** Overview
GnuPG 2.3+ uses a new format to store private keys that is both
more flexible and easier to read and edit by human beings.  The new
format stores name,value-pairs using the common mail and http header
convention.  Example (here indented with two spaces):

  Description: Key to sign all GnuPG released tarballs.
    The key is actually stored on a smart card.
  Use-for-ssh: yes
  OpenSSH-cert: long base64 encoded string wrapped so that this
    key file can be easily edited with a standard editor.
  Token: D2760001240102000005000011730000 OPENPGP.1 -
  Token: FF020001008A77C1 PIV.9C -
  Key: (shadowed-private-key
    (rsa
    (n #00AA1AD2A55FD8C8FDE9E1941772D9CC903FA43B268CB1B5A1BAFDC900
    2961D8AEA153424DC851EF13B83AC64FBE365C59DC1BD3E83017C90D4365B4
    83E02859FC13DB5842A00E969480DB96CE6F7D1C03600392B8E08EF0C01FC7
    19F9F9086B25AD39B4F1C2A2DF3E2BE317110CFFF21D4A11455508FE407997
    601260816C8422297C0637BB291C3A079B9CB38A92CE9E551F80AA0EBF4F0E
    72C3F250461E4D31F23A7087857FC8438324A013634563D34EFDDCBF2EA80D
    F9662C9CCD4BEF2522D8BDFED24CEF78DC6B309317407EAC576D889F88ADA0
    8C4FFB480981FB68C5C6CA27503381D41018E6CDC52AAAE46B166BDC10637A
    E186A02BA2497FDC5D1221#)
    (e #00010001#)
    (shadowed t1-v1
     (#D2760001240102000005000011730000# OPENPGP.1)
    )))

GnuPG 2.2 is also able to read and write keys using the new format
However, it only makes use of some of the values.

Keys in the extended format can be recognized by looking at the first
byte of the file.  If it starts with a '(' it is a naked S-expression,
otherwise it is a key in extended format.

*** Names
A name must start with a letter and end with a colon.  Valid
characters are all ASCII letters, numbers and the hyphen.  Comparison
of names is done case insensitively.  Names may be used several times
to represent an array of values.  Note that the name "Key" is special
in that it is mandatory and must occur only once.

*** Values
Values are UTF-8 encoded strings.  Values can be wrapped at any point,
and continued in the next line indicated by leading whitespace.  A
continuation line with one leading space does not introduce a blank so
that the lines can be effectively concatenated.  A blank line as part
of a continuation line encodes a newline.

*** Comments
Lines containing only whitespace, and lines starting with whitespace
followed by '#' are considered to be comments and are ignored.

** Well known names
*** Description
This is a human readable string describing the key.

*** Key
The name "Key" is special in that it is mandatory and must occur only
once.  The associated value holds the actual S-expression with the
cryptographic key.  The S-expression is formatted using the 'Advanced
Format' (GCRYSEXP_FMT_ADVANCED) that avoids non-printable characters
so that the file can be easily inspected and edited.  See section
'Private Key Format' below for details.

*** Created
The UTC time the key was created in ISO compressed format
(yyyymmddThhmmss).  This information can be used to re-create an
OpenPGP key.

*** Label
This is a short human readable description for the key which can be
used by the software to describe the key in a user interface.  For
example as part of the description in a prompt for a PIN or
passphrase.  It is often used instead of a comment element as present
in the S-expression of the "Key" item.

*** OpenSSH-cert
This takes a base64 encoded string wrapped so that this
key file can be easily edited with a standard editor.  Several of such
items can be used.

*** Token
If such an item exists it overrides the info given by the "shadow"
parameter in the S-expression.  Using this item makes it possible to
describe a key which is stored on several tokens and also makes it
easy to update this info using a standard editor.  The syntax is
similar to the "shadow" parameter:

- Serialnumber of the token.
- Key reference from the token in full format (e.g. "OpenPGP.2").
- An optional fixed length of the PIN or "-".
- The human readable serial number of a card.  This is usually what is
  printed on the actual card.  This value is taken directly from the
  card but when asking to insert a card it is useful to have this
  value available.  GnuPG takes care of creating and possibly updating
  this entry.  This is percent-plus-escaped.


*** Use-for-ssh
If given and the value is "yes" or "1" the key is allowed for use by
gpg-agent's ssh-agent implementation.  This is thus the same as
putting the keygrip into the 'sshcontrol' file.  Only one such item
should exist.  If another non-zero value between 1 and 99999 is used,
this is taken to establish the order in which the keys are returned to
ssh; lower numbers are returned first.  If a negative value is used
this overrides currently active (inserted) cards and thus allows to
prefer on-disk keys over inserted cards.  A value of -1 has the
highest priority; values are capped at -999 and have a lower priority
but still above the positive values, inserted cards or the order in
sshcontrol.


*** Use-for-p11
If given and the value is "yes" or "1" the key is allowed for use by
GnuPG's PKCS#11 interface (Scute).  Note that Scute needs to be
configured to use this optimization.

*** Remote-list
Allow to list the key with the KEYINFO command from a remote machine
via the extra socket.  A boolean value is expected; the default is
"no".  Note that KEYINFO will anyway provide information if the
keygrip is specified.

*** Confirm
If given and the value is "yes", a user will be asked confirmation by
a dialog window when the key is about to be used for
PKSIGN/PKAUTH/PKDECRYPT operation.  If the value is "restricted", it
is only asked for the access through extra/browser socket.

*** Prompt
This field is for card key.  If given and the value is "yes"
(default), a user will be prompted about insertion of the card by a
dialog window when card is not available.  When the value is "no", a
card operation is refused with GPG_ERR_UNUSABLE_SECKEY error.

*** Backup-info
This gives information for a backup of the key.  The following fields
are space delimited:

- Hexified keygrip (uppercase) to make it easy to identify the
  filename.  When restoring software should make sure that the keygrip
  matches the one derived from the "Key" field.
- Backup time in as ISO string.
- Name of the backup software.
- Arbitrary information.

* Private Key Format
** Unprotected Private Key Format

The content of the file is an S-Expression like the ones used with
Libgcrypt.  Here is an example of an unprotected file:

(private-key
 (rsa
  (n #00e0ce9..[some bytes not shown]..51#)
  (e #010001#)
  (d #046129F..[some bytes not shown]..81#)
  (p #00e861b..[some bytes not shown]..f1#)
  (q #00f7a7c..[some bytes not shown]..61#)
  (u #304559a..[some bytes not shown]..9b#)
 )
 (created-at timestamp)
 (uri http://foo.bar x-foo:whatever_you_want)
 (comment whatever)
)

"comment", "created-at" and "uri" are optional.  "comment" is
currently used to keep track of ssh key comments. "created-at" is used
to keep track of the creation time stamp used with OpenPGP keys; it is
optional but required for some operations to calculate the fingerprint
of the key.  This timestamp should be a string with the number of
seconds since Epoch or an ISO time string (yyyymmddThhmmss).

** Protected Private Key Format

A protected key is like this:

(protected-private-key
   (rsa
    (n #00e0ce9..[some bytes not shown]..51#)
    (e #010001#)
    (protected mode (parms) encrypted_octet_string)
    (protected-at <isotimestamp>)
   )
   (uri http://foo.bar x-foo:whatever_you_want)
   (comment whatever)
)


In this scheme the encrypted_octet_string is encrypted according to
the algorithm described after the keyword protected; most protection
algorithms need some parameters, which are given in a list before the
encrypted_octet_string.  The result of the decryption process is a
list of the secret key parameters.  The protected-at expression is
optional; the isotimestamp is 15 bytes long (e.g. "19610711T172000").

The currently defined protection modes are:

*** openpgp-s2k3-sha1-aes-cbc

  This describes an algorithm using AES in CBC mode for
  encryption, SHA-1 for integrity protection and the String to Key
  algorithm 3 from OpenPGP (rfc4880).

  Example:

  (protected openpgp-s2k3-sha1-aes-cbc
    ((sha1 16byte_salt no_of_iterations) 16byte_iv)
    encrypted_octet_string
  )

  The encrypted_octet string should yield this S-Exp (in canonical
  representation) after decryption:

  (
   (
    (d #046129F..[some bytes not shown]..81#)
    (p #00e861b..[some bytes not shown]..f1#)
    (q #00f7a7c..[some bytes not shown]..61#)
    (u #304559a..[some bytes not shown]..9b#)
   )
   (hash sha1 #...[hashvalue]...#)
  )

  For padding reasons, random bytes are appended to this list - they can
  easily be stripped by looking for the end of the list.

  The hash is calculated on the concatenation of the public key and
  secret key parameter lists: i.e. it is required to hash the
  concatenation of these 6 canonical encoded lists for RSA, including
  the parenthesis, the algorithm keyword and (if used) the protected-at
  list.

  (rsa
   (n #00e0ce9..[some bytes not shown]..51#)
   (e #010001#)
   (d #046129F..[some bytes not shown]..81#)
   (p #00e861b..[some bytes not shown]..f1#)
   (q #00f7a7c..[some bytes not shown]..61#)
   (u #304559a..[some bytes not shown]..9b#)
   (protected-at "18950523T000000")
  )

  After decryption the hash must be recalculated and compared against
  the stored one - If they don't match the integrity of the key is not
  given.

*** openpgp-s2k3-ocb-aes

  This describes an algorithm using AES-128 in OCB mode, a nonce
  of 96 bit, a taglen of 128 bit, and the String to Key algorithm 3
  from OpenPGP (rfc4880).

  Example:

  (protected openpgp-s2k3-ocb-aes
    ((sha1 16byte_salt no_of_iterations) 12byte_nonce)
    encrypted_octet_string
  )

  The encrypted_octet string should yield this S-Exp (in canonical
  representation) after decryption:

  (
   (
    (d #046129F..[some bytes not shown]..81#)
    (p #00e861b..[some bytes not shown]..f1#)
    (q #00f7a7c..[some bytes not shown]..61#)
    (u #304559a..[some bytes not shown]..9b#)
   )
  )

  For padding reasons, random bytes may be appended to this list -
  they can easily be stripped by looking for the end of the list.

  The associated data required for this protection mode is the list
  forming the public key parameters.  For the above example this is
  is this canonical encoded S-expression:

  (rsa
   (n #00e0ce9..[some bytes not shown]..51#)
   (e #010001#)
   (protected-at "18950523T000000")
  )

*** openpgp-native

  This is a wrapper around the OpenPGP Private Key Transport format
  which resembles the standard OpenPGP format and allows the use of an
  existing key without re-encrypting to the default protection format.

  Example:

  (protected openpgp-native
    (openpgp-private-key
     (version V)
     (algo PUBKEYALGO)
     (skey _ P1 _ P2 _ P3 ... e PN)
     (csum n)
     (protection PROTTYPE PROTALGO IV S2KMODE S2KHASH S2KSALT S2KCOUNT)))

  Note that the public key parameters in SKEY are duplicated and
  should be identical to their copies in the standard parameter
  elements.  Here is an example of an entire protected private key
  using this format:

  (protected-private-key
   (rsa
    (n #00e0ce9..[some bytes not shown]..51#)
    (e #010001#)
    (protected openpgp-native
     (openpgp-private-key
      (version 4)
      (algo rsa)
      (skey _ #00e0ce9..[some bytes not shown]..51#
            _ #010001#
            e #.........................#)
      (protection sha1 aes #aabbccddeeff00112233445566778899#
                  3 sha1 #2596f93e85f41e53# 3:190))))
   (uri http://foo.bar x-foo:whatever_you_want)
   (comment whatever))

** Shadowed Private Key Format

To keep track of keys stored on IC cards we use a third format for
private keys which are called shadow keys as they are only a reference
to keys stored on a token:

(shadowed-private-key
   (rsa
    (n #00e0ce9..[some bytes not shown]..51#)
    (e #010001#)
    (shadowed protocol (info))
   )
   (uri http://foo.bar x-foo:whatever_you_want)
   (comment whatever)
)

The currently used protocols are "t1-v1" (token info version 1) and
"tpm2-v1" (TPM format key information).  The second list with the
information has this layout for "t1-v1":

(card_serial_number id_string_of_key fixed_pin_length)

FIXED_PIN_LENGTH is optional.  It can be used to store the length of
the PIN; a value of 0 indicates that this information is not
available.  The rationale for this field is that some pinpad equipped
readers don't allow passing a variable length PIN.

This is the (info) layout for "tpm2-v1":

(parent tpm_private_string tpm_public_string)

Although this precise format is encapsulated inside the tpm2daemon
itself and nothing in gpg ever uses this.

More items may be added to the list.

** OpenPGP Private Key Transfer Format

This format is used to transfer keys between gpg and gpg-agent.

(openpgp-private-key
  (version V)
  (algo PUBKEYALGO)
  (curve CURVENAME)
  (skey _ P1 _ P2 _ P3 ... e PN)
  (csum n)
  (protection PROTTYPE PROTALGO IV S2KMODE S2KHASH S2KSALT S2KCOUNT))


 * V is the packet version number (3 or 4).
 * PUBKEYALGO is a Libgcrypt algo name
 * CURVENAME is the name of the curve - only used with ECC.
 * P1 .. PN are the parameters; the public parameters are never encrypted
   the secret key parameters are encrypted if the "protection" list is
   given.  To make this more explicit each parameter is preceded by a
   flag "_" for cleartext or "e" for encrypted text.
 * CSUM is the deprecated 16 bit checksum as defined by OpenPGP.  This
   is an optional element.
 * If PROTTYPE is "sha1" the new style SHA1 checksum is used if it is "sum"
   the old 16 bit checksum (above) is used and if it is "none" no
   protection at all is used.
 * PROTALGO is a Libgcrypt style cipher algorithm name
 * IV is the initialization vector.
 * S2KMODE is the value from RFC-4880.
 * S2KHASH is a libgcrypt style hash algorithm identifier.
 * S2KSALT is the 8 byte salt
 * S2KCOUNT is the count value from RFC-4880.

** Persistent Passphrase Format

Note: That this has not yet been implemented.

To allow persistent storage of cached passphrases we use a scheme
similar to the private-key storage format.  This is a master
passphrase format where each file may protect several secrets under
one master passphrase.  It is possible to have several of those files
each protected by a dedicated master passphrase.  Clear text keywords
allow listing the available protected passphrases.

The name of the files with these protected secrets have this form:
pw-<string>.dat.  STRING may be an arbitrary string, as a default name
for the passphrase storage the name "pw-default.dat" is suggested.


(protected-shared-secret
   ((desc descriptive_text)
    (key [key_1] (keyword_1 keyword_2 keyword_n))
    (key [key_2] (keyword_21 keyword_22 keyword_2n))
    (key [key_n] (keyword_n1 keyword_n2 keyword_nn))
    (protected mode (parms) encrypted_octet_string)
    (protected-at <isotimestamp>)
   )
)

After decryption the encrypted_octet_string yields this S-expression:

(
 (
  (value key_1 value_1)
  (value key_2 value_2)
  (value key_n value_n)
 )
 (hash sha1 #...[hashvalue]...#)
)

The "descriptive_text" is displayed with the prompt to enter the
unprotection passphrase.

KEY_1 to KEY_N are unique identifiers for the shared secret, for
example an URI.  In case this information should be kept confidential
as well, they may not appear in the unprotected part; however they are
mandatory in the encrypted_octet_string.  The list of keywords is
optional.  The order of the "key" lists and the order of the "value"
lists must match, that is the first "key"-list is associated with the
first "value" list in the encrypted_octet_string.

The protection mode etc. is identical to the protection mode as
described for the private key format.

list of the secret key parameters.  The protected-at expression is
optional; the isotimestamp is 15 bytes long (e.g. "19610711T172000").

The "hash" in the encrypted_octet_string is calculated on the
concatenation of the key list and value lists: i.e it is required to
hash the concatenation of all these lists, including the
parenthesis and (if used) the protected-at list.

Example:

(protected-shared-secret
   ((desc "List of system passphrases")
    (key "uid-1002" ("Knuth" "Donald Ervin Knuth"))
    (key "uid-1001" ("Dijkstra" "Edsger Wybe Dijkstra"))
    (key)
    (protected mode (parms) encrypted_octet_string)
    (protected-at "20100915T111722")
   )
)

with "encrypted_octet_string" decoding to:

(
 (
  (value 4:1002 "signal flags at the lock")
  (value 4:1001 "taocp")
  (value 1:0    "premature optimization is the root of all evil")
 )
 (hash sha1 #0102030405060708091011121314151617181920#)
)

To compute the hash this S-expression (in canonical format) was
hashed:

   ((desc "List of system passphrases")
    (key "uid-1002" ("Knuth" "Donald Ervin Knuth"))
    (key "uid-1001" ("Dijkstra" "Edsger Wybe Dijkstra"))
    (key)
    (value 4:1002 "signal flags at the lock")
    (value 4:1001 "taocp")
    (value 1:0    "premature optimization is the root of all evil")
    (protected-at "20100915T111722")
   )

* Notes

[1] I usually use the terms private and secret key exchangeable but prefer the
term secret key because it can be visually be better distinguished
from the term public key.

[2] The keygrip is a unique identifier for a key pair, it is
independent of any protocol, so that the same key can be used with
different protocols.  PKCS-15 calls this a subjectKeyHash; it can be
calculated using Libgcrypt's gcry_pk_get_keygrip ().

[3] Even when canonical representation are required we will show the
S-expression here in a more readable representation.