Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Overview

Ethereum ETL

Build Status Join the chat at https://gitter.im/ethereum-eth Telegram Discord

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

     > docker build -t ethereum-etl:latest .
     > docker image ls
    
  3. Run a container out of the image

     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
    
  4. Run streaming to console or Pub/Sub

     > docker build -t ethereum-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects//topics/crypto_ethereum
    

Projects using Ethereum ETL

  • Google - Public BigQuery Ethereum datasets
  • Nansen by D5 - Analytics platform for Ethereum
Comments
  • Add support for internal transactions

    Add support for internal transactions

    Parity provides APIs for retrieving internal transactions. trace_filter API allows retrieving traces for a range of blocks. This method is already supported by web3.py parity module.

    The scope of this task is to

    1. Design the model (the field names from this PR can be used as a basis: https://github.com/blockchain-etl/ethereum-etl/pull/72).
    2. Implement the export job (the job implementation will mostly resemble the flow in ExportTokenTransfersJob.
    3. Write unit tests (an example can be found here)
    4. Add documentation to README (an example can be found here) (5. Add instructions for how to start parity/geth i.e. what options to use).

    =======================

    Remarks:

    • There will be another task to add the BigQuery schema here
    • Geth also provides support for tracing transactions. There will be another task for implementing internal transactions exporter job for Geth. https://github.com/ethereum/go-ethereum/pull/15516, https://www.linkedin.com/pulse/geth-18-iceberg-michael-hunt-jr-
    opened by medvedev1088 21
  • Add Dockerfile

    Add Dockerfile

    The scope for this task is:

    1. Add Dockerfile (this can be used as a basis). Both ipc and rpc should be supported.
    2. Test it.
    3. Update documentation in README with instructions on how to use it.
    help wanted 
    opened by medvedev1088 15
  • Installing Ethereum-ETL

    Installing Ethereum-ETL

    Dear all,

    unfortunately I can't install ethereum-etl based on the instructions in the readme. Running "pip3 install ethereum-etl" as well as cloning the repo and running "pip3 install -e ." always results in:

    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b79bd0750>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/ethereum-etl/

    Could not find a version that satisfies the requirement ethereum-etl (from versions: ) No matching distribution found for ethereum-etl

    Does anybody has an idea what the problem is?

    Thanks and greets Peter

    opened by PeHo89 13
  • Add support for geth traces

    Add support for geth traces

    The scope of this task:

    1. Design the model for geth traces (trace.py can be used as a basis).
    2. Implement the export and extract jobs. 2.a The export job should use traceBlockByNumber (https://github.com/ethereum/go-ethereum/pull/15516) and output raw JSON responses. It should use callTracer, an example JS API call: debug.traceBlockByNumber('0x250252', {'tracer': 'callTracer'}); the same is available via JSON RPC. The job should use BatchWeb3Provider similar to export_blocks_job.py. 2.b The extract job should take the file with raw JSON responses, parses them, map to our domain entity and output to CSV file.
    3. Write unit tests (real mainnet responses should be used in test cases, so the same tests can be run against geth node).
    4. Add documentation to README (an example can be found here). Include instructions for how to start geth i.e. what options to use (e.g. geth --gcmode archive)

    ======================= Notes:

    • Parity traces are already supported https://github.com/blockchain-etl/ethereum-etl/pull/104.
    opened by medvedev1088 13
  • web3 upgrade

    web3 upgrade

    Here are the changes to upgrade web3.py for issue #308.

    • One major change was dropping the support for python 3.6 on ethereum etl since web3.py dropped 3.6 support
    • You may want to take a look at these notes on upgrading from web3.py v4 to v5 to see if anything jumps out at you.
    • I did still have the infura test falling locally but I think I needed to just dig in a little more and see why my setup wasn't working. If I have time tomorrow I will look at that.
    • I had to add a few of those mock response files. I copied ones you already had in there and modified them a little.
    • Also, I did not test the docker file but noticed it was using python3.6 so just changed the image it was pulling from to python3.7 hope that works.

    Let me know what you think, happy to tweak this some more if needed.

    opened by dbfreem 11
  • Windows - SyntaxError: invalid syntax

    Windows - SyntaxError: invalid syntax

    C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master>python export_blocks_and_transactions.py --start-block 0 --end-block 500000 \ --ipc-path ~/Library/Ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv Traceback (most recent call last): File "export_blocks_and_transactions.py", line 4, in from ethereumetl.ipc import IPCWrapper File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in from web3.providers.ipc import get_default_ipc_path, PersistantSocket File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3_init_.py", line 8, in from web3.main import Web3 # noqa: E402 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\main.py", line 26, in from web3.providers.eth_tester import ( File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers_init_.py", line 8, in from .websocket import WebsocketProvider # noqa: F401 File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers\websocket.py", line 9, in import websockets File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets_init_.py", line 3, in from .client import * File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\client.py", line 20, in from .protocol import WebSocketCommonProtocol File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\protocol.py", line 18, in from .compatibility import asyncio_ensure_future File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\compatibility.py", line 15 asyncio_ensure_future = asyncio.async # Python < 3.5 ^ SyntaxError: invalid syntax

    bug duplicate 
    opened by RaianeQC 11
  • adds support for non-mainnet in etl stream

    adds support for non-mainnet in etl stream

    relates #178

    DEMO on ronin network, an eth sidechain. Notice that latest block can now be read, thanks to middleware injection.

    2021-08-12 19:24:34,321 - ProgressLogger [INFO] - Started work. Items to process: 1.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - 1 items processed. Progress is 100%.
    2021-08-12 19:24:34,375 - ProgressLogger [INFO] - Finished work. Total items processed: 1. Took 0:00:00.054600.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000169.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Started work.
    2021-08-12 19:24:34,376 - ProgressLogger [INFO] - Finished work. Total items processed: 0. Took 0:00:00.000143.
    
    opened by numonedad 10
  • Ethereum data export error

    Ethereum data export error

    When i use “ethereumetl export_all -s 0 -e 5999999 -b 100000 -p file://$HOME/Library/Ethereum/geth.ipc -o output”,The following error occurred:“ValueError: Unknown uri scheme /datalake/blockdata/geth.ipc”,Ask what is the reason

    opened by sznariOsmosis 10
  • Add support for extracting Origin Protocol data

    Add support for extracting Origin Protocol data

    This implements a prototype that extracts data generated by Origin Protocol's open source platform for building decentralized marketplaces on Ethereum and IPFS.

    Fixes https://github.com/blockchain-etl/ethereum-etl/issues/215

    opened by franckc 10
  • ValueError: result is None in response (export_all)

    ValueError: result is None in response (export_all)

    I run it on ubuntu

    ethereumetl export_all -s 4900000 -e 4999999 -b 100000 -p https://mainnet.infura.io -o output

    but after n time it has error:

    File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 70, in _export total_items=self.end_block - self.start_block + 1 │ total_items=self.end_block - self.start_block + 1 File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) │ self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() │ self._check_completed_futures() File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_compl│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_comp eted_futures │leted_futures future.result() │ future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() │ return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result │ File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception │ raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run │ File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) │ result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_│ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe execute │_execute work_handler(batch) │ work_handler(batch) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in _export_batch blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/jobs/export_blocks_job.py", line 77, in <listcomp> blocks = [self.block_mapper.json_dict_to_block(result) for result in results] │ blocks = [self.block_mapper.json_dict_to_block(result) for result in results] File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 58, in rpc_response_batch_to_results yield rpc_response_to_result(response_item) │ yield rpc_response_to_result(response_item) File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result │ File "/usr/local/lib/python3.6/dist-packages/ethereumetl/utils.py", line 67, in rpc_response_to_result raise ValueError(error_message) │ raise ValueError(error_message) ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request proc│ValueError: result is None in response {'jsonrpc': '2.0', 'id': 0, 'error': {'code': -32603, 'message': 'request pro essing failed'}}.

    opened by minhtrih 10
  • Error in exporting traces using Parity

    Error in exporting traces using Parity

    @medvedev1088 We used Parity to export traces. It works perfectly for almost every block. BUT error happens in some block ranges, such as 2,280,000~2,400,000 and 3800000~4,000,000. Error:web3.utils.threads.Timeout: 60 seconds image

    Our server has far more than 8 GB memory. Maybe we could change default time. But we could not figure out the correct command

    opened by cxd1995 10
  • Add decoding log functions here

    Add decoding log functions here

    https://github.com/blockchain-etl/ethereum-etl/blob/6710e6b894789d83ce3c28db79b4ec5bf2ec3845/ethereumetl/jobs/export_receipts_job.py#L78

    Add feature to decode logs in the exportation of receipts

    opened by pauline-thrackle 0
  • [Errno 101]

    [Errno 101]

    Command: ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000
    --blocks-output blocks.csv --transactions-output transactions.csv
    --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c Error: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd63910>: Failed to establish a new connection: [Errno 101] Network is unreachable')) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mainnet.infura.io', port=443): Max retries exceeded with url: /v3/7aef3f0cd1f64408b163814b22cc643c (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4b8fd621d0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) how to solve this problem?

    opened by Facebeside 0
  • M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    M1 mac issue Symbolic Execution not available: No module named 'mythril.ether'

    System:

    • M1 Mac
    • Python 3.9.15
    • pip 22.1.2
    • Poetry (version 1.2.2)

    pyproject.toml

    [tool.poetry]
    name = "asd"
    version = "0.1.0"
    description = ""
    authors = ["Your Name <[email protected]>"]
    readme = "README.md"
    packages = [{include = "asd"}]
    
    [tool.poetry.dependencies]
    python = "^3.9"
    ethereum-etl = "^2.1.1"
    pyetherchain = "^0.3.2"
    blake2b-py = { git = "https://github.com/ethereum/blake2b-py.git", tag="v0.1.4" }
    mythril = {extras = ["mythril"], version = "^0.23.12"}
    
    
    
    [build-system]
    requires = ["poetry-core"]
    build-backend = "poetry.core.masonry.api"
    

    reprod script

    from ethereumetl.service.eth_contract_service import EthContractService
    
    bytecode_as_str = '608060405234801561001057600080fd5b506040516111893803806111898339818101604052602081101561003357600080fd5b810190808051906020019092919050505060016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000208190555080600260006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff1602179055506b033b2e3c9fd0803ce8000000600381905550600160048190555050611095806100f46000396000f3fe608060405234801561001057600080fd5b50600436106100b45760003560e01c80636924500911610071578063692450091461020d578063957aa58c146102175780639c52a7f114610235578063bf353dbb14610279578063d9638d36146102d1578063ebecb39d14610346576100b4565b80631504460f146100b95780631a0b287e146100e757806329ae81141461012957806336569e7714610161578063495d32cb146101ab57806365fae35e146101c9575b600080fd5b6100e5600480360360208110156100cf57600080fd5b810190808035906020019092919050505061039e565b005b610127600480360360608110156100fd57600080fd5b810190808035906020019092919080359060200190929190803590602001909291905050506105bd565b005b61015f6004803603604081101561013f57600080fd5b8101908080359060200190929190803590602001909291905050506107d7565b005b6101696109dc565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6101b3610a02565b6040518082815260200191505060405180910390f35b61020b600480360360208110156101df57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610a08565b005b610215610b36565b005b61021f610c27565b6040518082815260200191505060405180910390f35b6102776004803603602081101561024b57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610c2d565b005b6102bb6004803603602081101561028f57600080fd5b81019080803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610d5b565b6040518082815260200191505060405180910390f35b6102fd600480360360208110156102e757600080fd5b8101908080359060200190929190505050610d73565b604051808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018281526020019250505060405180910390f35b61039c6004803603606081101561035c57600080fd5b810190808035906020019092919080359060200190929190803573ffffffffffffffffffffffffffffffffffffffff169060200190929190505050610db7565b005b6000806001600084815260200190815260200160002060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff166359e02dd76040518163ffffffff1660e01b81526004016040805180830381600087803b15801561041e57600080fd5b505af1158015610432573d6000803e3d6000fd5b505050506040513d604081101561044857600080fd5b810190808051906020019092919080519060200190929190505050915091506000816104755760006104b1565b6104b061049461048c8560001c633b9aca0061100b565b600354611037565b6001600087815260200190815260200160002060010154611037565b5b9050600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16631a0b287e85836040518363ffffffff1660e01b815260040180838152602001807f73706f740000000000000000000000000000000000000000000000000000000081525060200182815260200192505050600060405180830381600087803b15801561055857600080fd5b505af115801561056c573d6000803e3d6000fd5b505050507fdfd7467e425a8107cfd368d159957692c25085aacbcf5228ce08f10f2146486e84848360405180848152602001838152602001828152602001935050505060405180910390a150505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610671576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b6001600454146106e9576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f6d617400000000000000000000000000000000000000000000000000000000008214156107315780600160008581526020019081526020016000206001018190555061079f565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020541461088b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610903576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f706172000000000000000000000000000000000000000000000000000000000082141561093757806003819055506109a5565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a4505050565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b60035481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610abc576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60016000808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610bea576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60006004819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450565b60045481565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610ce1576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b60008060008373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001908152602001600020819055505961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a45050565b60006020528060005260406000206000915090505481565b60016020528060005260406000206000915090508060000160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16908060010154905082565b60016000803373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020019081526020016000205414610e6b576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260168152602001807f53706f747465722f6e6f742d617574686f72697a65640000000000000000000081525060200191505060405180910390fd5b600160045414610ee3576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260108152602001807f53706f747465722f6e6f742d6c6976650000000000000000000000000000000081525060200191505060405180910390fd5b7f7069700000000000000000000000000000000000000000000000000000000000821415610f6557806001600085815260200190815260200160002060000160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550610fd3565b6040517f08c379a000000000000000000000000000000000000000000000000000000000815260040180806020018281038252601f8152602001807f53706f747465722f66696c652d756e7265636f676e697a65642d706172616d0081525060200191505060405180910390fd5b5961012081016040526020815260e0602082015260e0600060408301376024356004353360003560e01c60e01b61012085a450505050565b600080821480611028575082828385029250828161102557fe5b04145b61103157600080fd5b92915050565b600081611050846b033b2e3c9fd0803ce800000061100b565b8161105757fe5b0490509291505056fea265627a7a72315820b20103aad12e7c2a10372688d31c070bc7aa268459f8bc3a58d4298c7f6bdf0964736f6c634300050c003200000000000000000000000035d1b3f3d7966a1dfe207aa4514c12a259a0492b'
    
    def main():
        contract_service = EthContractService()
        func_sigs = contract_service.get_function_sighashes(bytecode_as_str)
        print(func_sigs)
    
    
    if __name__ == '__main__':
        main()
    
    opened by 007vasy 0
  • stream command failed to extract contract information

    stream command failed to extract contract information

    Command : ethereumetl stream --start-block 15789281 -e block,transaction,log,token_transfer,contract --log-file log.txt --provider-uri https://mainnet.infura.io/v3/9aa3d95b3bc440fa88ea12eaa4456161

    Error : 2022-11-30 16:07:55,003 - root [ERROR] - An exception occurred while syncing block data. Traceback (most recent call last): File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 77, in _do_stream synced_blocks = self._sync_cycle() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/streaming/streamer.py", line 98, in _sync_cycle self.blockchain_streamer_adapter.export_all(self.last_synced_block + 1, target_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 62, in export_all traces = self._export_traces(start_block, end_block) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/streaming/eth_streamer_adapter.py", line 159, in _export_traces job.run() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/blockchainetl/jobs/base_job.py", line 30, in run self._end() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 106, in _end self.batch_work_executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 97, in shutdown self.executor.shutdown() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 437, in result return self.__get_result() File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/executors/batch_work_executor.py", line 63, in _fail_safe_execute work_handler(batch) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/ethereumetl/jobs/export_traces_job.py", line 90, in _export_batch json_traces = self.web3.parity.traceBlock(block_number) File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/module.py", line 57, in caller result = w3.manager.request_blocking(method_str, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 198, in request_blocking return self.formatted_response(response, File "/Users/chintanpatel/Library/Python/3.8/lib/python/site-packages/web3/manager.py", line 171, in formatted_response raise ValueError(response["error"]) ValueError: {'code': -32601, 'message': 'the method trace_block does not exist/is not available'}

    opened by infocodiste 1
  • can't support proxy contract erc20

    can't support proxy contract erc20

    I looked at the source code and found that bytecode is used to judge whether it is erc20, but many contracts currently use the proxy contract upgrade solution to deploy the contract, which makes it impossible to judge whether it is erc20

    opened by SuperEdison 2
Releases(v2.1.2)
Owner
Blockchain ETL
Facilitating data science on blockchain data. Available in Google BigQuery https://goo.gl/oY5BCQ
Blockchain ETL
Simple BRender SDK KeyGen.

Super Simple BRender KeyGen Simple BRender SDK KeyGen. Info This is just a super simple keygen for the BRender SDK installer coded in python. All test

1 Jan 15, 2022
Vhost password decrypt for python

vhost_password_decrypt Where is symkey.dat Windows:C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\ssl\symkey.dat Linux:/etc/vmware-vpx/ssl/symkey.

Jing Ling 152 Dec 22, 2022
BlockVis - Create beautiful visualizations of Bitcoin Blockheaders

BlockVis Create beautiful visualizations of Bitcoin Blockheaders How to run To r

Egge 2 Jan 05, 2022
💰 An Alfred Workflow that provides current price of cryptocurrency

Coin Ticker for Alfred Workflow An Alfred Workflow that provides current price and status about cryptocurrency from cryptocompare.com. Supports Alfred

Bumsoo Kim (Ian) 14 Nov 17, 2022
Simple encryption/decryption utility using Pycryptodome module. Working with AES and RSA algorithms.

EncypherUtil Simple encryption/decryption utility using PyCryptodome module. Working with AES and RSA algorithms. THIS UTILITY IS NOT LICENSED AS CRYP

Egor Yakubovich 0 Jun 14, 2022
EncryptAGit - Encrypt Your Git Repos

EncryptAGit - Encrypt Your Git Repos

midnite_runr 25 Oct 06, 2022
Django-based Crypto Portfolio Tracker – keep an eye on Shiba Inu and other Crypto

Crypto Tracker 🐍 📈 – Central Portfolio Tracking Easy asset tracking – at a glance 🚀 Dashboard to centrally monitor current crypto portfolio develop

65 Jan 08, 2023
A Docker image for plotting and farming the Chia™ cryptocurrency on one computer or across many.

An easy-to-use WebUI for crypto plotting and farming. Offers Plotman, MadMax, Chiadog, Bladebit, Farmr, and Forktools in a Docker container. Supports Chia, Cactus, Chives, Flax, Flora, HDDCoin, Maize

Guy Davis 328 Jan 01, 2023
Alpkunt 9 Sep 09, 2022
The Qis|krypt⟩ is a software suite of protocols of quantum cryptography and quantum communications

The Qis|krypt⟩ is a software suite of protocols of quantum cryptography and quantum communications, as well, other protocols and algorithms, built using IBM’s open-source Software Development Kit for

Qiskrypt 14 Oct 31, 2022
Use this script to track the gains of cryptocurrencies using historical data and display it on a super-imposed chart in order to find the highest performing cryptocurrencies historically

crypto-performance-tracker Use this script to track the gains of cryptocurrencies using historical data and display it on a super-imposed chart in ord

Andrei 25 Aug 31, 2022
Buckley 2 Jul 24, 2022
SDU experiment of introduction to the cryptography

Lab 01 (2 hrs): Programming Basics Program 1: Type Hint, String, Bytes, Hex, Base64 Lab 02 (4 hrs): Classical Cryptography Part 1 (3 hrs): Program 1:

1 Jan 03, 2022
J. Brandon Walker 1 May 13, 2022
A Trading strategy for the Freqtrade crypto bot.

Important Thing to notice 1) Do not use this strategy on live. It is still undergoing dry-run. 2) The Hyperopt is highly optimized towards "shitcoin"

160 Dec 26, 2022
Scrambler - Useful File/Directory Encryption Program

This is a program that is used to scramble/encrypt files on your computer. Do not use this program to do malicious things with. I am not responsible for any damage that you do with this software.

0 Oct 01, 2021
dashboard to track crypto prices and change via the coinmarketcap APIs

crypto-dashboard Dashboard to track crypto prices and change via the coinmarketcap APIs. Uses chart.js and ag-grid. Requirements: python 3 (was writte

4 Nov 09, 2021
Python implementation of EIP 1577 content hash

ContentHash for Python Python implementation of EIP 1577 content hash. Description This is a simple package made for encoding and decoding content has

Filip Š 11 Jul 19, 2022
SVSHI - Secure and Verified Smart Home Infrastructure

The SVSHI (Secure and Verified Smart Home Infrastructure) (pronounced like "sushi") project is a platform/runtime/toolchain for developing and running formally verified smart infrastructures, such as

Dependable Systems Laboratory 3 Oct 28, 2022
offline half-random brute force script for Ethereum private keys

eth200swinger offline half-random brute force script for Ethereum private keys, goes from the beginning to end of range and vice versa, saves any foun

2 Oct 06, 2022