Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Overview

Ethereum ETL

Build Status Join the chat at https://gitter.im/ethereum-eth Telegram Discord

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

     > docker build -t ethereum-etl:latest .
     > docker image ls
    
  3. Run a container out of the image

     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
    
  4. Run streaming to console or Pub/Sub

     > docker build -t ethereum-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects//topics/crypto_ethereum
    

Projects using Ethereum ETL

  • Google - Public BigQuery Ethereum datasets
  • Nansen by D5 - Analytics platform for Ethereum
Comments
  • Add support for internal transactions

    Add support for internal transactions

    Parity provides APIs for retrieving internal transactions. trace_filter API allows retrieving traces for a range of blocks. This method is already supported by web3.py parity module.

    The scope of this task is to

    1. Design the model (the field names from this PR can be used as a basis: https://github.com/blockchain-etl/ethereum-etl/pull/72).
    2. Implement the export job (the job implementation will mostly resemble the flow in ExportTokenTransfersJob.
    3. Write unit tests (an example can be found here)
    4. Add documentation to README (an example can be found here) (5. Add instructions for how to start parity/geth i.e. what options to use).

    =======================

    Remarks:

    • There will be another task to add the BigQuery schema here
    • Geth also provides support for tracing transactions. There will be another task for implementing internal transactions exporter job for Geth. https://github.com/ethereum/go-ethereum/pull/15516, https://www.linkedin.com/pulse/geth-18-iceberg-michael-hunt-jr-
    opened by medvedev1088 21
  • Add Dockerfile

    Add Dockerfile

    The scope for this task is:

    1. Add Dockerfile (this can be used as a basis). Both ipc and rpc should be supported.
    2. Test it.
    3. Update documentation in README with instructions on how to use it.
    help wanted 
    opened by medvedev1088 15
  • Installing Ethereum-ETL

    Installing Ethereum-ETL

    Dear all,

    unfortunately I can't install ethereum-etl based on the instructions in the readme. Running "pip3 install ethereum-etl" as well as cloning the repo and running "pip3 install -e ." always results in:

    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b79bd0750>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/ethereum-etl/

    Could not find a version that satisfies the requirement ethereum-etl (from versions: ) No matching distribution found for ethereum-etl

    Does anybody has an idea what the problem is?

    Thanks and greets Peter

    opened by PeHo89 13
  • Add support for geth traces

    Add support for geth traces

    The scope of this task:

    1. Design the model for geth traces (trace.py can be used as a basis).
    2. Implement the export and extract jobs. 2.a The export job should use traceBlockByNumber (https://github.com/ethereum/go-ethereum/pull/15516) and output raw JSON responses. It should use callTracer, an example JS API call: debug.traceBlockByNumber('0x250252', {'tracer': 'callTracer'}); the same is available via JSON RPC. The job should use BatchWeb3Provider similar to export_blocks_job.py. 2.b The extract job should take the file with raw JSON responses, parses them, map to our domain entity and output to CSV file.
    3. Write unit tests (real mainnet responses should be used in test cases, so the same tests can be run against geth node).
    4. Add documentation to README (an example can be found here). Include instructions for how to start geth i.e. what options to use (e.g. geth --gcmode archive)

    ======================= Notes:

    • Parity traces are already supported https://github.com/blockchain-etl/ethereum-etl/pull/104.
    opened by medvedev1088 13
  • web3 upgrade

    web3 upgrade

    Here are the changes to upgrade web3.py for issue #308.

    • One major change was dropping the support for python 3.6 on ethereum etl since web3.py dropped 3.6 support
    • You may want to take a look at these notes on upgrading from web3.py v4 to v5 to see if anything jumps out at you.
    • I did still have the infura test falling locally but I think I needed to just dig in a little more and see why my setup wasn't working. If I have time tomorrow I will look at that.
    • I had to add a few of those mock response files. I copied ones you already had in there and modified them a little.
    • Also, I did not test the docker file but noticed it was using python3.6 so just changed the image it was pulling from to python3.7 hope that works.

    Let me know what you think, happy to tweak this some more if needed.

    opened by dbfreem 11
  • Windows - SyntaxError: invalid syntax

    Windows - SyntaxError: invalid syntax

    C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master>python export_blocks_and_transactions.py --start-block 0 --end-block 500000 \ --ipc-path ~/Library/Ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv Traceback (most recent call last): File "export_blocks_and_transactions.py", line 4, in from ethereumetl.ipc import IPCWrapper File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in from web3.providers.ipc import get_default_ipc_path, PersistantSocket File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3_init_.py", line 8, in from web3.main import Web3 # noqa: E402 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\main.py", line 26, in from web3.providers.eth_tester import ( File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers_init_.py", line 8, in from .websocket import WebsocketProvider # noqa: F401 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers\websocket.py", line 9, in import websockets File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets_init_.py", line 3, in from .client import * File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\client.py", line 20, in from .protocol import WebSocketCommonProtocol File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\protocol.py", line 18, in from .compatibility import asyncio_ensure_future File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\compatibility.py", line 15 asyncio_ensure_future = asyncio.async # Python < 3.5 ^ SyntaxError: invalid syntax

    bug duplicate 
    opened by RaianeQC 11
  • adds support for non-mainnet in etl stream

    adds support for non-mainnet in etl stream

    relates #178

    DEMO on ronin network, an eth sidechain. Notice that latest block can now be read, thanks to middleware injection.

    2021-08-12 19:24:34,321 - ProgressLogger [INFO] - Started work. Items to process: 1.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.054600.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000169.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000143.
    
    opened by numonedad 10
  • Ethereum data export error

    Ethereum data export error

    When i use “ethereumetl export_all -s 0 -e 5999999 -b 100000 -p file://$HOME/Library/Ethereum/geth.ipc -o output”,The following error occurred:“ValueError: Unknown uri scheme /datalake/blockdata/geth.ipc”,Ask what is the reason

    opened by sznariOsmosis 10
  • Add support for extracting Origin Protocol data

    Add support for extracting Origin Protocol data

    This implements a prototype that extracts data generated by Origin Protocol's open source platform for building decentralized marketplaces on Ethereum and IPFS.

    Fixes https://github.com/blockchain-etl/ethereum-etl/issues/215

    opened by franckc 10
  • ValueError: result is None in response (export_all)

    ValueError: result is None in response (export_all)

    I run it on ubuntu

    ethereumetl export_all -s 4900000 -e 4999999 -b 100000 -p https://mainnet.infura.io -o output

    but after n time it has error:

    File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export total_items=self.end_block - self.start_block + 1 │ total_items=self.end_block - self.start_block + 1 File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) │ self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() │ self._check_completed_futures() File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_compl│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_comp eted_futures │leted_futures future.result() │ future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() │ return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception │ raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run │ File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) │ result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe execute │_execute work_handler(batch) │ work_handler(batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results yield rpc_response_to_result(response_item) │ yield rpc_response_to_result(response_item) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result raise ValueError(error_message) │ raise ValueError(error_message) ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request proc│ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request pro essing failed'}}.

    opened by minhtrih 10
  • Error in exporting traces using Parity

    Error in exporting traces using Parity

    @medvedev1088 We used Parity to export traces. It works perfectly for almost every block. BUT error happens in some block ranges, such as 2,280,000~2,400,000 and 3800000~4,000,000. Error:web3.utils.threads.Timeout: 60 seconds image

    Our server has far more than 8 GB memory. Maybe we could change default time. But we could not figure out the correct command

    opened by cxd1995 10
  • Add decoding log functions here

    Add decoding log functions here

    https://github.com/blockchain-etl/ethereum-etl/blob/6710e6b894789d83ce3c28db79b4ec5bf2ec3845/ethereumetl/jobs/export_receipts_job.py#L78

    Add feature to decode logs in the exportation of receipts

    opened by pauline-thrackle 0
  • [Errno 101]

    [Errno 101]

    Command: ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000
    --blocks-output blocks.csv --transactions-output transactions.csv
    --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c Error: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd63910>: Failed to establish a new connection: [Errno 101] Network is unreachable')) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd621d0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) how to solve this problem?

    opened by Facebeside 0
  • M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    System:

    • M1 Mac
    • Python 3.9.15
    • pip 22.1.2
    • Poetry (version 1.2.2)

    pyproject.toml

    [tool.poetry]
    name = "asd"
    version = "0.1.0"
    description = ""
    authors = ["Your Name <[email protected]>"]
    readme = "README.md"
    packages = [{include = "asd"}]
    
    [tool.poetry.dependencies]
    python = "^3.9"
    ethereum-etl = "^2.1.1"
    pyetherchain = "^0.3.2"
    blake2b-py = { git = "https://github.com/ethereum/blake2b-py.git", tag="v0.1.4" }
    mythril = {extras = ["mythril"], version = "^0.23.12"}
    
    
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    reprod script

    from ethereumetl.service.eth_contract_service import EthContractService
    
    bytecode_as_str = '608060405234801561001057600080fd5b506040516111893803806111898339818101604052602081101561003357600080fd5b810190808051906020019092919050505060016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000208190555080600260006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff1602179055506b033b2e3c9fd0803ce8000000600381905550600160048190555050611095806100f46000396000f3fe608060405234801561001057600080fd5b50600436106100b45760003560e01c80636924500911610071578063692450091461020d578063957aa58c146102175780639c52a7f114610235578063bf353dbb14610279578063d9638d36146102d1578063ebecb39d14610346576100b4565b80631504460f146100b95780631a0b287e146100e757806329ae81141461012957806336569e7714610161578063495d32cb146101ab57806365fae35e146101c9575b600080fd5b6100e5600480360360208110156100cf57600080fd5b810190808035906020019092919050505061039e565b005b610127600480360360608110156100fd57600080fd5b810190808035906020019092919080359060200190929190803590602001909291905050506105bd565b005b61015f6004803603604081101561013f57600080fd5b8101908080359060200190929190803590602001909291905050506107d7565b005b6101696109dc565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6101b3610a02565b6040518082815260200191505060405180910390f35b61020b600480360360208110156101df57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610a08565b005b610215610b36565b005b61021f610c27565b6040518082815260200191505060405180910390f35b6102776004803603602081101561024b57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c2d565b005b6102bb6004803603602081101561028f57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610d5b565b6040518082815260200191505060405180910390f35b6102fd600480360360208110156102e757600080fd5b8101908080359060200190929190505050610d73565b604051808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018281526020019250505060405180910390f35b61039c6004803603606081101561035c57600080fd5b810190808035906020019092919080359060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610db7565b005b6000806001600084815260200190815260200160002060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff166359e02dd76040518163ffffffff1660e01b81526004016040805180830381600087803b15801561041e57600080fd5b505af1158015610432573d6000803e3d6000fd5b505050506040513d604081101561044857600080fd5b810190808051906020019092919080519060200190929190505050915091506000816104755760006104b1565b6104b061049461048c8560001c633b9aca0061100b565b600354611037565b6001600087815260200190815260200160002060010154611037565b5b9050600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16631a0b287e85836040518363ffffffff1660e01b815260040180838152602001807f73706f740000000000000000000000000000000000000000000000000000000081525060200182815260200192505050600060405180830381600087803b15801561055857600080fd5b505af115801561056c573d6000803e3d6000fd5b505050507fdfd7467e425a8107cfd368d159957692c25085aacbcf5228ce08f10f2146486e84848360405180848152602001838152602001828152602001935050505060405180910390a150505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610671576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b6001600454146106e9576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f6d617400000000000000000000000000000000000000000000000000000000008214156107315780600160008581526020019081526020016000206001018190555061079f565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541461088b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610903576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f706172000000000000000000000000000000000000000000000000000000000082141561093757806003819055506109a5565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a4505050565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b60035481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610abc576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60016000808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610bea576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60006004819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450565b60045481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610ce1576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60008060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60006020528060005260406000206000915090505481565b60016020528060005260406000206000915090508060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16908060010154905082565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610e6b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610ee3576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f7069700000000000000000000000000000000000000000000000000000000000821415610f6557806001600085815260200190815260200160002060000160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550610fd3565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b600080821480611028575082828385029250828161102557fe5b04145b61103157600080fd5b92915050565b600081611050846b033b2e3c9fd0803ce800000061100b565b8161105757fe5b0490509291505056fea265627a7a72315820b20103aad12e7c2a10372688d31c070bc7aa268459f8bc3a58d4298c7f6bdf0964736f6c634300050c003200000000000000000000000035d1b3f3d7966a1dfe207aa4514c12a259a0492b'
    
    def main():
        contract_service = EthContractService()
        func_sigs = contract_service.get_function_sighashes(bytecode_as_str)
        print(func_sigs)
    
    
    if __name__ == '__main__':
        main()
    
    opened by 007vasy 0
  • stream command failed to extract contract information

    stream command failed to extract contract information

    Command : ethereumetl stream --start-block 15789281 -e block,transaction,log,token_transfer,contract --log-file log.txt --provider-uri https://mainnet.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161

    Error : 2022-11-30 16:07:55,003 - root [ERROR] - An exception occurred while syncing block data. Traceback (most recent call last): File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 77, in _do_stream synced_blocks = self._sync_cycle() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 98, in _sync_cycle self.blockchain_streamer_adapter.export_all(self.last_synced_block + 1, target_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 62, in export_all traces = self._export_traces(start_block, end_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 159, in _export_traces job.run() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/jobs/base_job.py", line 30, in run self._end() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 106, in _end self.batch_work_executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown self.executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result return self.__get_result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute work_handler(batch) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 90, in _export_batch json_traces = self.web3.parity.traceBlock(block_number) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/module.py", line 57, in caller result = w3.manager.request_blocking(method_str, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 198, in request_blocking return self.formatted_response(response, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 171, in formatted_response raise ValueError(response["error"]) ValueError: {'code': -32601, 'message': 'the method trace_block does not exist/is not available'}

    opened by infocodiste 1
  • can't support proxy contract erc20

    can't support proxy contract erc20

    I looked at the source code and found that bytecode is used to judge whether it is erc20, but many contracts currently use the proxy contract upgrade solution to deploy the contract, which makes it impossible to judge whether it is erc20

    opened by SuperEdison 2
Releases(v2.1.2)
Owner
Blockchain ETL
Facilitating data science on blockchain data. Available in Google BigQuery https://goo.gl/oY5BCQ
Blockchain ETL
Tutela: an Ethereum and Tornado Cash Anonymity Tool

Tutela: an Ethereum and Tornado Cash Anonymity Tool The repo contains open-source code for Tutela, an anonymity tool for Ethereum and Tornado Cash use

TutelaLabs 96 Dec 05, 2022
Modern(-ish) password hashing for your software and your servers

bcrypt Good password hashing for your software and your servers Installation To install bcrypt, simply: $ pip install bcrypt Note that bcrypt should b

Python Cryptographic Authority 947 Dec 28, 2022
A simple web application with tools of cryptography, made with Flask and Cryptography.

Crypto Tools A web application made with Flask that allows the use of some cryptography tools like message digest, RSA key pair generation and a decip

Felipe Valentin 0 Jan 20, 2022
Learn Blockchains by Building One, A simple Blockchain in Python using Flask as a micro web framework.

Blockchain ✨ Learn Blockchains by Building One Yourself Installation Make sure Python 3.6+ is installed. Install Flask Web Framework. Clone this repos

Vaibhaw 46 Jan 05, 2023
bitcoin-ticker is a E-ink ticker that shows usefull information about bitcoin

bitcoin-ticker bitcoin-ticker is a E-ink ticker that shows usefull information about bitcoin. Due to the limited refresh lifetime, new information is

32 Nov 09, 2022
An automated Risk Management Monitor Bot for ByBit cryptocurrencies exchange.

An automated Risk Management Monitor Bot for ByBit cryptocurrencies exchange that forces all open positions to adhere to a specific risk ratio, defined per asset. It supports USDT Perpetual, Inverse

Hadi Aladdin 25 Nov 27, 2022
Simple BRender SDK KeyGen.

Super Simple BRender KeyGen Simple BRender SDK KeyGen. Info This is just a super simple keygen for the BRender SDK installer coded in python. All test

1 Jan 15, 2022
Crypto Portfolio Clustering with and without optimization techniques (elbow method, PCA).

Crypto Portfolio Clustering Crypto Portfolio Clustering with and without optimization techniques (elbow method, PCA). Analysis This is an anlysis of c

David L 0 Feb 18, 2022
SimpleWallet - Simple wallet for Bitcoin

Simple Wallet This is a basic python starter package to be used as a template fo

Mystic 1 Jan 08, 2022
Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse.

Tink A multi-language, cross-platform library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse. Ubuntu

Google 12.9k Jan 05, 2023
Gridlock - Encryption and decryption python project

Gridlock Encryption Encryption and decryption of plain text messages inspired by

Matthew 2 Mar 23, 2022
Small utility to encrypt and decrypt messages

Safe Safe is a small utility to encrypt and decrypt messages using a pair of public and private keys. Installation You need to have GPG installed in y

Gustavo Eguez 2 Dec 21, 2021
cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.

pyca/cryptography cryptography is a package which provides cryptographic recipes and primitives to Python developers. Our goal is for it to be your "c

Python Cryptographic Authority 5.2k Dec 30, 2022
Get the SHA256 hash of any file with this Python Script

Hashfile-SHA256 A SHA256 hash verifying script, written in python. Report Bug Table of Contents About The Project Built With Getting Started Prerequis

Ethan Gallucci 1 Nov 01, 2021
A community effort to bring back Duino-Coin

Duino-Coin-Revived A community effort to bring back Duino-Coin! Along with reviving the cryptocurrency, we will add many improvements to it, including

1 Dec 22, 2021
Freqtrade is a free and open source crypto trading bot written in Python

Freqtrade is a free and open source crypto trading bot written in Python. It is designed to support all major exchanges and be controlled via Telegram. It contains backtesting, plotting and money man

20.2k Jan 07, 2023
A simple script useful to switch from Dashlane to Bitwarden by converting the password file to the right format.

A simple script useful to switch from Dashlane to Bitwarden by converting the password file to the right format.

3 May 06, 2022
Deribit_Algo_Project_Python - Deribit algo project written in python trading crypto futures

This is a Algo/script trading for Deribit. You need an account with deribit, to

24 Jan 09, 2023
SysWhispers integrated shellcode loader w/ ETW patching & anti-sandboxing

TymSpecial Shellcode Loader Description This project was made as a way for myself to learn C++ and gain insight into how EDR products work. TymSpecial

Nick Frischkorn 145 Dec 20, 2022
Um sistema de Criptografia RSA feito totalmente em Python

Um sistema de Criptografia RSA feito totalmente em Python

Luis Müdder 3 Nov 23, 2021