Simple HTTP Server for CircuitPython

Added separate class for HTTP headers, that allows accessing headers using case-insensitive names.

Also, changed some docstrings the were unclear and one that was misleading as it was inconsistent with the code.

Changed the order of Docs References, so that the most "important" ones are on the top, with enums and dataclass-like classes below.

Updated library version to 1.1.0 in sphinx configuration file.

Fixes #26 Fixes #28

This PR, other that separating adafruit_httpserver into multiple files #8 adds following functionality:

• Access to additional attributes of HTTPRequest:
  • http_version extracted from Start line, e.g. "HTTP/1.1"
  • query_params extracted from path, e.g. /path?foo=bar is parsed to {"foo": "bar"}
  • headers as a dict, e.g. {'user-agent': '...', 'host': 'esp32-s2-tft.local', 'connection': 'close'}
  • body as bytes, so both text content and images can be processed #16
• Added new parameter to HTTPResponse which allows to set it's headers #24

Other than that, this PR also includes:

• Refactor or addition of "enums" like HTTPMethod and MIMEType
• ~~Removed blocking socket~~ Implemented socket_timeout property, which fixes #21
• Changed the way server receives data, which fixes #2
• Smaller refactor here and there
• More typing in functions/classes etc. #6

Despite major refactor, the usage of library stays nearly unchanged, so it won't be neccessary to rewrite existing projects from scratch, only minor changes might be required. It surely is not perfect, but I belive it is a step in the right direction.

Closes #2 Closes #6 Closes #8 Closes #16 Closes #17 Closes #21 Closes #24

Trying to use adafruit_requests to connect either the 5100S-based WIZnet Pico EVB running Adafruit CircuitPython 7.2.5 on 2022-04-06; Raspberry Pi Pico with rp2040 or the Adafruit Feather RP2040 with Adafruit Ethernet FeatherWing running Adafruit CircuitPython 7.2.5 on 2022-04-06; Adafruit Feather RP2040 with rp2040 to an Adafruit Feather ESP32-S2 TFT running HTTPServer on an IPv4 (this repo simpletest example; no mDNS/hostname/FQDN involved) results in one of the following exception traces in all cases:

Either (less often):

Traceback (most recent call last):
  File "code.py", line 211, in <module>
  File "adafruit_requests.py", line 815, in get
  File "adafruit_requests.py", line 685, in request
OutOfRetries: Repeated socket failures

Or (more often):

Traceback (most recent call last):
  File "code.py", line 211, in <module>
  File "adafruit_requests.py", line 815, in get
  File "adafruit_requests.py", line 661, in request
  File "adafruit_requests.py", line 529, in _get_socket
  File "adafruit_wiznet5k/adafruit_wiznet5k_socket.py", line 251, in connect
  File "adafruit_wiznet5k/adafruit_wiznet5k.py", line 574, in socket_connect
RuntimeError: Failed to establish connection.

I was initially going to file this issue in WIZnet, but a sanity check of trying to connect to the HTTPServer from ESP32-S2 (e.g., Adafruit Feather ESP32-S2 TFT) also gets an exception every time, after about a minute. That surprised me, I may be doing something wrong. Maybe it's a Requests issue.

ESP32-S2 Client Code:

import traceback
import wifi
import socketpool
import ssl
import adafruit_requests
from adafruit_httpserver import HTTPServer, HTTPResponse
from secrets import secrets

wifi.radio.connect(secrets['ssid'], secrets['password'])
pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

URLS = [
    "http://wifitest.adafruit.com/testwifi/index.html",
    "http://192.168.5.32",   # LAN Apache server
    "http://192.168.6.164",  # LAN ESP32-S2 with adafruit_httpserver
]

for url in URLS:
    try:
        print(url)
        with requests.get(url) as response:
            print(response.status_code, response.reason)
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)

Output:

code.py output:
http://wifitest.adafruit.com/testwifi/index.html
200 bytearray(b'OK')
http://192.168.5.32
200 bytearray(b'OK')
http://192.168.6.164
Traceback (most recent call last):
  File "code.py", line 22, in <module>
  File "adafruit_requests.py", line 720, in get
  File "adafruit_requests.py", line 661, in request
  File "adafruit_requests.py", line 512, in _get_socket
RuntimeError: Sending request failed

Both Espressif client and Espressif server are running: Adafruit CircuitPython 7.2.5 on 2022-04-06; Adafruit Feather ESP32-S2 TFT with ESP32S2

Connecting to the HTTPServer from a browser or curl works fine.

Connecting to local Apache server at an IPv4 from any of these clients works fine.

Yes, I've seen issue #17 and others. But I'm not the one to do major rewrites from scratch, and I liked how this server was small, and could both server files and cgi-ish content. The files being served are mainly css and js and some small graphics. These also benefit from having the ability to have cache control. The forms and table based content became to large to send as a single string body so I added the ability to send data with chunked transfer encoding.

I looked at ampule, but TBH, I had already figured this out here, and I didn't need the extra dependencies either. This is a collection of a few added capabilities. All of the additions should not break current api implementations, just extends the capabilities. Feel free to accept, make comments or just throw away.

This also solves the issue #21 that I was having.

The next limit we ran into was and 1kb buffer for reading the request. Not knowing how much memory different devices have, it seemed best to expose the buffer size rather than just increase it.

Co-authored-by: Shae Erisson [email protected]

The Python documentation states that otherwise, the blocking status of the newly accepted socket is implementation-defined:

if the listening socket is in non-blocking mode, whether the socket returned by accept() is in blocking or non-blocking mode is operating system-dependent. If you want to ensure cross-platform behaviour, it is recommended you manually override this setting. https://docs.python.org/3/library/socket.html#socket-timeouts

When the connected socket is non-blocking (as it is on picow), the http library works erratically, depending whether the request has already arrived by the time recvfrom_into call occurs.

Closes: adafruit/circuitpython#7086

I am building a tiny web server using adafruit_httpserver.server and circuitpython 8.0.0-beta.5 on ESP32-S2. When multibyte characters (Japanese) are included in response body, the content-length header value received by a web client looks shorter, and some tailing characters in the body are missing.

page_html = """
<html>
  <head><title>test</title></head>
  <body>
   some text in multibyte language here..
  </body>
</html>
"""

@server.route("/test")
def base(request):
  return HTTPResponse(content_type='text/html;charset=UTF-8',body=page_html)

$ curl -v http://192.168.xx.xx/test
 :
< HTTP/1.1 200 OK
< Content-Length: 117
< Content-Type: text/html;charset=UTF-8
< Connection: close
<
* Excess found in a read: excess = 10, size = 117, maxdownload = 117, bytecount = 0
(html response are shown, but the last several characters in body are missing)

Looking into adafruit_httpserver/response.py, content-length is calculated as len(body), as in, response_headers.setdefault("Content-Length", content_length or len(body)) and the whole response are sent after converted into bytes. If I replace it with len(body.encode("utf-8")), the above trouble disappears, but I'm not sure this modification is right.

response.py#L78

Fix for issue #26 which was caused by the passed in headers dict getting modified for the response. Then the calling program reused the headers in subsequent calls. Thanks to @dhalbert and @anecdata for assistance on Discord.

Hello,

I started using this library today and followed the documentation describing how to set the status code when responding to a request. The triple-quoted docstring for HTTPResponse.__init__ describes status as:

    :param tuple status: The HTTP status code to return, as a tuple of (int, "message").
     Common statuses are available in `HTTPStatus`.

while the type hint for the constructor argument defines it as status: tuple = HTTPStatus.OK. The name tuple is recognized by the documentation generator, which creates a link to the Python.org documentation for the tuple type.

Passing a tuple does not actually work here, and causes the response to be badly formatted. The value given to status is saved in self.status, then passed to self._send_response, which in turn calls self._send_bytes with the value produced by self._HEADERS_FORMAT.format: https://github.com/adafruit/Adafruit_CircuitPython_HTTPServer/blob/67c3ac3b76dd419a90558460b727a07b2169351f/adafruit_httpserver.py#L232-L236

This format method is str.format, since self._HEADERS_FORMAT is just a string: https://github.com/adafruit/Adafruit_CircuitPython_HTTPServer/blob/67c3ac3b76dd419a90558460b727a07b2169351f/adafruit_httpserver.py#L179-L185

Passing a tuple as documented – e.g. (200, 'OK') causes it to be rendered at the position of the first {} in the format string, making the response look like this:

HTTP/1.1 (200, 'OK')
Content-Type: text/plain
Content-Length: 5
Connection: close

hello

This is not a valid HTTP response, so when curl receives it it fails with

* Unsupported HTTP version in response
* Closing connection 0

The docs do suggest to use the common values pre-defined as static fields in HTTPStatus, which aren't tuples but HTTPStatus instances. They are only provided for status code 200, 404, and 500, so it's likely that users would want to provide other values and try to use tuples like (403, 'Forbidden') for example. HTTPStatus instances are rendered correctly in the format string because the class specifically defines its string representation with a __str__ method: https://github.com/adafruit/Adafruit_CircuitPython_HTTPServer/blob/67c3ac3b76dd419a90558460b727a07b2169351f/adafruit_httpserver.py#L49-L50

I'm not sure whether this should be best handled as purely a documentation change – no longer suggesting tuple – or as a code change, actually supporting tuple and rendering it correctly, maybe even raising an exception if a tuple is provided that is not made of an int and a string. I'm far from an authority on the matter, but it seems more Pythonic to me to actually support tuples rather than require an HTTPStatus object or a protocol-formatted string to directly inject into the response.

Thanks for this library!

The regular expression fails with a stack overflow for paths of more than 135 characters. The split also appears to be much faster.

Failing parse:

>>> import re
>>> _REQUEST_RE = re.compile(r"(\S+)\s+(\S+)\s")
>>> _REQUEST_RE.match("GET /" + ("a" * 135) + " whatev")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded

Co-authored-by: Shae Erisson [email protected]

Given the following object:

headers = {
    "Access-Control-Allow-Headers": "*",
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Methods": "GET,POST,DELETE"
}

And the following server function:

@server.route("/api/v1/info", HTTPMethod.GET)
def info(request):
    """Return info"""
    try:
        obj = {"version": Api.api_version, "name": Api.api_name}
        response = json.dumps(obj)
        return HTTPResponse(status=CommonHTTPStatus.OK_200, body=response, content_type=MIMEType.TYPE_JSON, headers=Api.headers)
    except:
        return HTTPResponse(status=CommonHTTPStatus.BAD_REQUEST_400, headers=Api.headers)

The content-type gets turned to octet-stream and the content-length gets set to 1024.

doing this in the response.py code in the _construct_response_bytes function works fine:

headers.setdefault("Access-Control-Allow-Headers", "*")
headers.setdefault("Access-Control-Allow-Origin", "*")
headers.setdefault("Access-Control-Allow-Methods", "*")

this is a temporary solution, but not very preferred. I have looked around in the code and I have no idea what causes this issue

There seems to be a problem handling requests from Chrome. Chrome locks the http server for a minute, blocking any other requests from being handled - other than requests from the same Chrome tab. A minute after the last request from Chrome, the server will start handling other requests again.

If you start a server, it will handle requests correctly from clients like node-red, firefox etc. (anything other than Chrome!). If you are looping with server.poll() the loop will run correctly, but will freeze for 60 seconds after the Chrome request. An example from an ESP32-S3 I am using:

pool = socketpool.SocketPool(wifi.radio)
server = HTTPServer(pool)
server.request_buffer_size = 2048
server.socket_timeout = 1
server.start(str(wifi.radio.ipv4_address))

@server.route("/")
def base(request: HTTPRequest):
    with HTTPResponse(request, content_type=MIMEType.TYPE_TXT, chunked=False) as response:
        try:
            print("REQUEST: /")
            response.send("HELLO")
        except OSError:
            response.send("ERROR")

while True:
    try:
        led.value = True
        time.sleep(0.05)
        led.value = False
        time.sleep(0.1)
        server.poll()
        
    except OSError as error:
        print(error)
        continue

As soon as you hit the server with Chrome, the server responds but stops the loop and stops handling requests from other clients. If you make another request from chrome in the same tab, it will be handled, and seemingly at that time pending requests from other clients can sometimes be handled but the loop will freeze again right after.

Making a request to an undefined path from Chrome freezes the loop also - you get the 404 in chrome and then server loop freezes - whatever code you put in your handler does not fix this problem.

A minute after the last Chrome request, the loop restarts and the server functions normally.

If you quit Chrome while the loop is locked, the loop restarts immediately.

I've tried a bunch of things to fix this, including closing the connection in the handler, sending various keep-alive responses to drop the connection - none work.

I think this needs an SDK level fix!