Fast HTTP parser

Last update: Jan 07, 2023

Related tags

Overview

httptools is a Python binding for the nodejs HTTP parser.

The package is available on PyPI: pip install httptools.

APIs

httptools contains two classes httptools.HttpRequestParser, httptools.HttpResponseParser and a function for parsing URLs httptools.parse_url. See unittests for examples.

class HttpRequestParser:

    def __init__(self, protocol):
        """HttpRequestParser

        protocol -- a Python object with the following methods
        (all optional):

          - on_message_begin()
          - on_url(url: bytes)
          - on_header(name: bytes, value: bytes)
          - on_headers_complete()
          - on_body(body: bytes)
          - on_message_complete()
          - on_chunk_header()
          - on_chunk_complete()
          - on_status(status: bytes)
        """

    def get_http_version(self) -> str:
        """Return an HTTP protocol version."""

    def should_keep_alive(self) -> bool:
        """Return ``True`` if keep-alive mode is preferred."""

    def should_upgrade(self) -> bool:
        """Return ``True`` if the parsed request is a valid Upgrade request.
	The method exposes a flag set just before on_headers_complete.
	Calling this method earlier will only yield `False`.
	"""

    def feed_data(self, data: bytes):
        """Feed data to the parser.

        Will eventually trigger callbacks on the ``protocol``
        object.

        On HTTP upgrade, this method will raise an
        ``HttpParserUpgrade`` exception, with its sole argument
        set to the offset of the non-HTTP data in ``data``.
        """

    def get_method(self) -> bytes:
        """Return HTTP request method (GET, HEAD, etc)"""


class HttpResponseParser:

    """Has all methods except ``get_method()`` that
    HttpRequestParser has."""

    def get_status_code(self) -> int:
        """Return the status code of the HTTP response"""


def parse_url(url: bytes):
    """Parse URL strings into a structured Python object.

    Returns an instance of ``httptools.URL`` class with the
    following attributes:

      - schema: bytes
      - host: bytes
      - port: int
      - path: bytes
      - query: bytes
      - fragment: bytes
      - userinfo: bytes
    """

Development

Clone this repository with git clone --recursive [email protected]:MagicStack/httptools.git
Create a virtual environment with Python 3.5: python3.5 -m venv envname
Activate the environment with source envname/bin/activate
Install Cython with pip install cython
Run make and make test.

License

MIT.

Comments

Swap http-parse to llhttp
Hi!

First of all, I'm pretty new to Cython, so I'm sorry if I made any dumb mistakes here (pretty sure I might have messed up setup.py).

Some observations:

~~The tests are failing due to some import error, and I'm pretty sure it is because of the changes I made to setup.py~~ Tests are now passing

I changed the upgrade behavior and don't know if it is the right approach, would love some feedback

The upgrade tests used offset that I couldn't reproduce with llhttp so I might have missed something

@elprans I would love some feedback here to get this going!

Also tagging @indutny in case you have any 2cents or some warnings about something we might miss on this migration, or that we need extra care :D
opened by victoraugustolls 23

Failed building wheel for httptools

Any idea on possible workaround:

After pip3 install httptools:

Collecting httptools
  Using cached https://files.pythonhosted.org/packages/1b/03/215969db11abe8741e9c266a4cbe803a372bd86dd35fa0084c4df6d4bd00/httptools-0.0.13.tar.gz
Building wheels for collected packages: httptools
  Building wheel for httptools (setup.py) ... error
  Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-lq_cgux2 --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/httptools
  copying httptools/__init__.py -> build/lib.linux-x86_64-3.6/httptools
  creating build/lib.linux-x86_64-3.6/httptools/parser
  copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.6/httptools/parser
  copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.6/httptools/parser
  running egg_info
  writing httptools.egg-info/PKG-INFO
  writing dependency_links to httptools.egg-info/dependency_links.txt
  writing top-level names to httptools.egg-info/top_level.txt
  reading manifest file 'httptools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'httptools.egg-info/SOURCES.txt'
  copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.6/httptools/parser
  running build_ext
  building 'httptools.parser.parser' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/httptools
  creating build/temp.linux-x86_64-3.6/httptools/parser
  creating build/temp.linux-x86_64-3.6/vendor
  creating build/temp.linux-x86_64-3.6/vendor/http-parser
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.6/httptools/parser/parser.o -O2
  httptools/parser/parser.c:4:20: fatal error: Python.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for httptools
  Running setup.py clean for httptools
Failed to build httptools
Installing collected packages: httptools
  Running setup.py install for httptools ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-xaxh1zu8/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/httptools
    copying httptools/__init__.py -> build/lib.linux-x86_64-3.6/httptools
    creating build/lib.linux-x86_64-3.6/httptools/parser
    copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.6/httptools/parser
    copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.6/httptools/parser
    running egg_info
    writing httptools.egg-info/PKG-INFO
    writing dependency_links to httptools.egg-info/dependency_links.txt
    writing top-level names to httptools.egg-info/top_level.txt
    reading manifest file 'httptools.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'httptools.egg-info/SOURCES.txt'
    copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.6/httptools/parser
    running build_ext
    building 'httptools.parser.parser' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/httptools
    creating build/temp.linux-x86_64-3.6/httptools/parser
    creating build/temp.linux-x86_64-3.6/vendor
    creating build/temp.linux-x86_64-3.6/vendor/http-parser
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.6/httptools/parser/parser.o -O2
    httptools/parser/parser.c:4:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-ffzualnz/httptools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-xaxh1zu8/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-ffzualnz/httptools/

Ubuntu 16.04 LTS

opened by dejanbatanjac 21

Example with aiohttp

In your uvloop docs you talk about running aiohttp with this parser instead of the built in one. Do you have any documentation on how that would work?

opened by nhumrich 13
Chunked request
i am working on httptools integration with aiohttp, some tests are failing with chunking and compression. if i modify httptools tests, it does not pass as well.

CHUNKED_REQUEST1_3 = b'''POST /test.php?a=b+c HTTP/1.2 User-Agent: Fooo Host: bar Transfer-Encoding: chunked b\r\n+\xce\xcfM\xb5MI,I\x04\x00\r\n0\r\n\r\n'''
opened by fafhrd91 10
provide wheel for python3.9

Would it be possible to add wheels for python3.9 with a new patch release? :) Currently it is only possible to compile the package on the fly in a virtualized environment on python3.9.

opened by dvzrv 9

Upgrade response

here test code. also i am not sure how to get data after end of message back. i am about \x89\x04data


import httptools

import unittest
from unittest import mock

UPGRADE_RESPONSE = b'''HTTP/1.1 101 Switching Protocols
UPGRADE: websocket
SEC-WEBSOCKET-ACCEPT: rVg+XakFNFOxk3ZH0lzrZBmg0aU=
TRANSFER-ENCODING: chunked
CONNECTION: upgrade
DATE: Sat, 07 May 2016 23:44:32 GMT
SERVER: Python/3.4 aiohttp/1.0.3
\r\n\x89\x04data'''


class TestResponseParser(unittest.TestCase):

    def test_parser_upgrade_response(self):
        m = mock.Mock()

        headers = {}
        m.on_header.side_effect = headers.__setitem__

        p = httptools.HttpResponseParser(m)
        try:
            p.feed_data(UPGRADE_RESPONSE)
        except:
            pass

        self.assertEqual(p.get_http_version(), '1.1')
        self.assertEqual(p.get_status_code(), 101)

        m.on_status.assert_called_once_with(b'Switching Protocols')

        m.on_headers_complete.assert_called_once_with()
        self.assertEqual(m.on_header.call_count, 6)
        self.assertEqual(len(headers), 6)

        m.on_message_complete.assert_called_once_with()

opened by fafhrd91 9

HttpParserUpgrade raised after event hooks are called.

The HttpParserUpgrade exception appears to get called after the event hooks have been called into (at least with small requests that fit into a single data_received call)

This makes it not terribly useful, as we may have already called into a handler function / issued a response / etc... by the time the upgrade is raised.

Sketch of protocol class:

def __init__(self, ...):
    self.request_parser = httptools.HttpRequestParser(self)
    ...

    # The asyncio.Protocol hooks...
    ...

    def data_received(self, data):
        print('data_received()')
        try:
            self.request_parser.feed_data(data)
        except httptools.HttpParserUpgrade:
            print('upgrade exception')

    # Event hooks called back into by HttpRequestParser...
    ...

    def on_headers_complete(self):
        print('on_headers_complete()')

    def on_message_complete(self):
        print('on_message_complete()')

Output:

data_received()
on_headers_complete()
on_message_complete()
upgrade exception

On first sight this looks awkward to resolve, so it could just be considered a constraint of the implementation, and deal with it in python-land instead (which would be acceptable from my POV)

opened by tomchristie 5

Python 2 support
Would you accept a Python 2 support PR?

It looks like not a lot of code changes are required.

I played around a little, here are my findings:

the Cython code just compiles and seems to work on Python 2 unchanged

the __all__ in the __init__.py fails, because I think __all__ in Cython is a unicode object, something Python 2 doesn't like. I'd advocate changing the __init__.py to explicitly import everything needed instead of doing a from .. import *. This way you are in full control of what's exposed in the public API.

the test suite relies on some features from Python 3:

in particular unitest.mock, which is available as mock on PyPI so that's easy

assertRaisesRegex which is missing. It'd need to be backported, or the test rewritten as the regexes used aren't complicated.

subTest, either backported or the test rewritten.

I'm a fan of pytest myself so alternatively we can port the tests over over to that; it's compatible with Python 2 and Python 3. I'd be happier doing that instead.

I recommend a tool like tox so we can easily run the tests on both Pythons.

the Makefile is geared towards Python 3, but doesn't do all that much. I just called the build_ext command myself with Python 2. We could cruftify the makefile to support Python 2. Alternatively we could also just simply document the few standard Python commands needed; tox should care of building stuff and running stuff anyway and that's a simple command.
opened by faassen 5
llhttp critical CVE's
Hi! (Again),

Thanks for fixing the previous CVE's which i reported here last year. Today i noticed 3 CVE's (9.1 score) which are a bit obscurely reported by the nodeJS release notes, and which have been fixed in llhttp 6.0.7. [https://www.opencve.io/cve?vendor=llhttp]

This morning I tried to prepare a merge-request to bump llhttp to 6.0.9, but I came to the conclusion it's best to get some advice. It seems there's some controversie about llhttp dropping support for LF delimiting headers and requiring CR delimiters.

I'm not sure how httptools and other projects using httptools potentially could get impacted by such a change, and if there are better ways to get arround this.. or if it's not an issue.

So i stopped the process of refactoring the test-cases, to get them to just pass, and ask for help.. (I'd hoped to be of use, but this is a too big of a question for me / I'll likely hold up the process of getting this fixed properly and timely)

notes:

Add the release branch to .gitmodules: [submodule "vendor/llhttp"] path = vendor/llhttp url = https://github.com/nodejs/llhttp.git branch = release

Use a recent git version to run: git submodule update --remote (or use some other methods if you prefer not to update your git client).

test_parser.py contains 'HTTP/1.2' in some of the validations causing tests to fail with an exception about invalid version .. change the version to HTTP/1.1 everywhere

then all the remaining exceptions are about the CR 's that are required File "httptools\parser\parser.pyx", line 212, in httptools.parser.parser.HttpParser.feed_data httptools.parser.errors.HttpParserError: Missing expected CR after header value

Thanks again!
opened by nlsj1985 4

Doesn't build against Python 3.11

        running build_ext
        building 'httptools.parser.parser' extension
        creating build/temp.linux-x86_64-3.11
        creating build/temp.linux-x86_64-3.11/httptools
        creating build/temp.linux-x86_64-3.11/httptools/parser
        creating build/temp.linux-x86_64-3.11/vendor
        creating build/temp.linux-x86_64-3.11/vendor/llhttp
        creating build/temp.linux-x86_64-3.11/vendor/llhttp/src
        x86_64-linux-gnu-gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/tmp/pip-req-build-cyh4gi6y/vendor/llhttp/include -I/tmp/pip-req-build-cyh4gi6y/vendor/llhttp/src -I/home/dima/.cache/pypoetry/virtualenvs/silly-chat-9D3wOO4G-py3.11/include -I/usr/include/python3.11 -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.11/httptools/parser/parser.o -O2
        httptools/parser/parser.c: In function ‘__pyx_pf_9httptools_6parser_6parser_10HttpParser_10feed_data’:
        httptools/parser/parser.c:3452:23: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         3452 |       __pyx_v_err_pos = llhttp_get_error_pos(__pyx_v_self->_cparser);
              |                       ^
        httptools/parser/parser.c: In function ‘__Pyx_AddTraceback’:
        httptools/parser/parser.c:454:62: error: dereferencing pointer to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
          454 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
              |                                                              ^~
        httptools/parser/parser.c:9421:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
         9421 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
              |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
        error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
        [end of output]

opened by dimaqq 3

Deprecation warnings in Python 3.10

I am not sure if its due to the deprecation warnings but there are failures while installing it on Python 3.10. Error log as below :

Probably related to https://github.com/python/cpython/pull/20878

    Running setup.py install for httptools ... error
    ERROR: Command errored out with exit status 1:
     command: /root/dev-denv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ldf_awgo/install-record.txt --single-version-externally-managed --compile --install-headers /root/dev-denv/include/site/python3.10/httptools
         cwd: /tmp/pip-install-3qm507hv/httptools/
    Complete output (155 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.10
    creating build/lib.linux-x86_64-3.10/httptools
    copying httptools/_version.py -> build/lib.linux-x86_64-3.10/httptools
    copying httptools/__init__.py -> build/lib.linux-x86_64-3.10/httptools
    creating build/lib.linux-x86_64-3.10/httptools/parser
    copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.10/httptools/parser
    copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.10/httptools/parser
    running egg_info
    writing httptools.egg-info/PKG-INFO
    writing dependency_links to httptools.egg-info/dependency_links.txt
    writing requirements to httptools.egg-info/requires.txt
    writing top-level names to httptools.egg-info/top_level.txt
    reading manifest file 'httptools.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'httptools.egg-info/SOURCES.txt'
    copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.10/httptools/parser
    running build_ext
    building 'httptools.parser.parser' extension
    creating build/temp.linux-x86_64-3.10
    creating build/temp.linux-x86_64-3.10/httptools
    creating build/temp.linux-x86_64-3.10/httptools/parser
    creating build/temp.linux-x86_64-3.10/vendor
    creating build/temp.linux-x86_64-3.10/vendor/http-parser
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/root/dev-denv/include -I/root/cpython/Include -I/root/cpython -I/tmp/pip-install-3qm507hv/httptools/vendor/http-parser -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.10/httptools/parser/parser.o -O2
    httptools/parser/parser.c: In function ‘__pyx_tp_dealloc_9httptools_6parser_6parser_HttpParser’:
    httptools/parser/parser.c:8187:5: error: lvalue required as increment operand
         ++Py_REFCNT(o);
         ^~
    httptools/parser/parser.c:8189:5: error: lvalue required as decrement operand
         --Py_REFCNT(o);
         ^~
    httptools/parser/parser.c: In function ‘__Pyx_ParseOptionalKeywords’:
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10178:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                         (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 :
                         ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:580:45: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                 ^~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c:10194:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                             (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                             ^
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:446:26: note: declared here
     static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~
    httptools/parser/parser.c: In function ‘__Pyx_decode_c_bytes’:
    httptools/parser/parser.c:10379:9: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
             return PyUnicode_FromUnicode(NULL, 0);
             ^~~~~~
    In file included from /root/cpython/Include/unicodeobject.h:1026:0,
                     from /root/cpython/Include/Python.h:97,
                     from httptools/parser/parser.c:20:
    /root/cpython/Include/cpython/unicodeobject.h:551:42: note: declared here
     Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
                                              ^~~~~~~~~~~~~~~~~~~~~
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/dev-denv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3qm507hv/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ldf_awgo/install-record.txt --single-version-externally-managed --compile --install-headers /root/dev-denv/include/site/python3.10/httptools Check the logs for full command output.

opened by tirkarthi 3

How to do an early stop?

Hi there :wave:

I'm trying to implement a flag --limit-request-header-count on uvicorn, which I implemented changing the value of an attribute too_many_headers and raising an exception from on_header callback, see https://github.com/encode/uvicorn/pull/1683/files.

My question is: is my implementation as intended? Should the HttpParserCallbackError send more information when an exception happens from one of the callbacks?

opened by Kludex 2
bump to llhttp v6.0.10

Please bump httptools version. It seems that there was some parsing issue remaining with regard to the CVE's llhttp v6.0.10 changelog: http: disable chunked encoding when OBS fold is used

Fixes: https://hackerone.com/reports/1630336 Fixes: https://hackerone.com/reports/1665156 Fixes: https://hackerone.com/reports/1675191

opened by nlsj1985 4
PROXY protocol v1 / v2 support
Behind a reverse-proxy or a load balancer, you often need to get the original client IP. A lot of load balancer provide these information with the PROXY protocol - HAproxy for exemple. I have a load balancer that provide only this method to get the client info, and I would not be the only one.

Here a doc of the protocol: http://www.haproxy.org/download/2.4/doc/proxy-protocol.txt

In a nutshell, it's one line added on top of the body, here for the v1 protocol:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n GET / HTTP/1.1\r\n Host: 192.168.0.11\r\n \r\n

This protocol is implemented in Gunicorn, and can be activated with --proxy-protocol: https://docs.gunicorn.org/en/stable/settings.html#proxy-protocol

edit: A NodeJS parser: https://github.com/racker/node-proxy-protocol/blob/master/index.js
opened by elrik75 0
Response parser requires support for HEAD responses without body

According to the specification of HTTP/1.1 ...

Responses to the HEAD request method never include a message body because the associated response header fields (e.g., Transfer-Encoding, Content-Length, etc.), if present, indicate only what their values would have been if the request method had been GET.

Source: RFC 7230 (Section 3.3)

... responses to HEAD requests do not have a body. Even if e.g. the Content-Length header is present. However, the response parser cannot know based on the data it receives whether the corresponding request was using the HEAD or another method and whether to expect a body or to just ignore any header flags about the content.

Suggestion:

Add an additional boolean flag to HttpResponseParser.feed_data(self, data: bytes) to tell the parser that it should not expect a body (or even throw a parser exception if there is one present).

Because HTTP/1.1 also allows pipelining, this additional parameter should also rather be a list of "body presence" flags. Here's why: If you send multiple requests to a server without waiting for the responses (= pipelining), you are going to receive back a bunch of responses together. Without them being parsed yet, one cannot identify the message boundaries and therefore they cannot be fed to the parser one by one. So the parser has to support that the whole stack of responses is fed to it and it needs a list of the "expected body presence". Example: If the pipelined requests were something like {HEAD, GET, HEAD, GET}, the "body presence" flags need to alternate as well. For convenience, it may be good to have a parameter that can be set using both types, a single flag and a list of flags.

opened by mxscho 1

Releases(v0.5.0)

v0.5.0(Sep 13, 2022)
Changes

Bump bundled llhttp to 6.0.9 fixes CVE-2022-32213, CVE-2022-32214, CVE-2022-32215 (by @nlsj1985 in 56d6a163 for #83)

Test and build against Python 3.11 (by @elprans in 509cd149 for #84)

Source code(tar.gz)
Source code(zip)
v0.4.0(Feb 22, 2022)
Changes

Bump bundled http-parser to 2.9.4 and llhttp to 6.0.6 fixes CVE-2021-22959 & CVE-2021-22960 (by @elprans in 4d5dddd3 for #77)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 10, 2021)
This release has no functional changes, only packaging: Python 3.5 is EOL, so wheels are no longer built, and Python 3.10 has been added to the roster along with aarch64 wheels on Linux and universal2 wheels on macOS.

Changes:

Use cibuildwheel to build release wheels (by @elprans in 2f57b6b7)

Source code(tar.gz)
Source code(zip)
v0.1.2(Apr 26, 2021)
Bug Fixes

Fix httptools.__all__ (by @elprans in 9340d321 for #52)

Build

Add Python 3.9 in the build/test matrix (by @b0g3r in e2d1a464 for #62)

Source code(tar.gz)
Source code(zip)
v0.2.0(Apr 23, 2021)
New Features

Swap http-parse to llhttp (by @victoraugustolls and @fantix in 63b5de2b for #56)

Bug Fixes

Fix httptools.__all__ (by @elprans in 9340d321 for #52)

Build

Add Python 3.9 in the build/test matrix (by @b0g3r in e2d1a464 for #62)

Source code(tar.gz)
Source code(zip)
v0.1.1(Apr 26, 2021)

Restore Python 3.5 support (#48).
Source code(tar.gz)
Source code(zip)
v0.1.0(Feb 5, 2020)

No functional changes from 0.0.13, except the new release flow and binary wheels.
Source code(tar.gz)
Source code(zip)

Owner

magicstack

Any sufficiently advanced software is indistinguishable from magic.

GitHub Repository

suite de mocks http em json

Ritchie Formula Repo Documentation Contribute to the Ritchie community This repository contains rit formulas which can be executed by the ritchie-cli.

1 Nov 01, 2021

Asynchronous HTTP client/server framework for asyncio and Python

Async http client/server framework Key Features Supports both client and server side of HTTP protocol. Supports both client and server Web-Sockets out

13.1k Jan 01, 2023

Python requests like API built on top of Twisted's HTTP client.

treq: High-level Twisted HTTP Client API treq is an HTTP library inspired by requests but written on top of Twisted's Agents. It provides a simple, hi

553 Dec 18, 2022

Requests + Gevent = <3

GRequests: Asynchronous Requests GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. Note: You should probably

4.2k Dec 30, 2022

Script to automate PUT HTTP method exploitation to get shell.

116 Nov 10, 2022

hackhttp2 make everything easier

hackhttp2 intro This repo is inspired by hackhttp, but it's out of date already. so, I create this repo to make simulation and Network request easier.

5 Jun 15, 2022

Some example code for using a raspberry pi to draw text (including emojis) and twitch emotes to a HUB75 RGB matrix via an HTTP post endpoint.

7 Nov 05, 2022

r - a small subset of Python Requests

r a small subset of Python Requests a few years ago, when I was first learning Python and looking for http functionality, i found the batteries-includ

4 Dec 15, 2022

HTTP/2 for Python.

Hyper: HTTP/2 Client for Python This project is no longer maintained! Please use an alternative, such as HTTPX or others. We will not publish further

1k Dec 23, 2022

Pretty fast mass-dmer with multiple tokens support made with python requests

mass-dm-requests - Little preview of the Logger and the Spammer Features Logging User IDS Sending DMs (Embeds are supported) to the logged IDs Includi

14 Nov 18, 2022

🔄 🌐 Handle thousands of HTTP requests, disk writes, and other I/O-bound tasks simultaneously with Python's quintessential async libraries.

15 Dec 12, 2022

PycURL - Python interface to libcurl

PycURL -- A Python Interface To The cURL library PycURL is a Python interface to libcurl, the multiprotocol file transfer library. Similarly to the ur

933 Jan 09, 2023

Small, fast HTTP client library for Python. Features persistent connections, cache, and Google App Engine support. Originally written by Joe Gregorio, now supported by community.

Introduction httplib2 is a comprehensive HTTP client library, httplib2.py supports many features left out of other HTTP libraries. HTTP and HTTPS HTTP

457 Dec 10, 2022

A toolbelt of useful classes and functions to be used with python-requests

The Requests Toolbelt This is just a collection of utilities for python-requests, but don't really belong in requests proper. The minimum tested reque

892 Jan 06, 2023

Python Simple SOAP Library

PySimpleSOAP / soap2py Python simple and lightweight SOAP library for client and server webservices interfaces, aimed to be as small and easy as possi

369 Jan 02, 2023

Bot que responde automáticamente as perguntas do giga unitel

Gigabot+ Bot que responde automáticamente as perguntas do giga unitel LINK DOWNLOAD: Gigabot.exe O script pode apresentar alguns erros, pois não tive

20 Jul 16, 2021

As easy as /aitch-tee-tee-pie/ 🥧 Modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more. https://twitter.com/httpie

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

25.4k Jan 01, 2023

Fast HTTP parser

Related tags

Overview

APIs

Development

License

Comments

Releases(v0.5.0)

v0.5.0(Sep 13, 2022)

Changes

v0.4.0(Feb 22, 2022)

Changes

v0.3.0(Aug 10, 2021)

v0.1.2(Apr 26, 2021)

Bug Fixes

Build

v0.2.0(Apr 23, 2021)

New Features