Overall: - Added ip logger detection for 3 services namely Grabify.link, Linkify.me, IPLogger.org

Config: - Added config option to rename traceroute map html file - Added configurable ip logger detection with 3 possible values to flag on, which are (DOMAIN, DOMAIN_IP, SSL_CERT_SUBJECT_DOMAIN)

Requester & Utils: - Fixed a bug where Odyssey would not get the correct part of the url where the port is

ResponseParser: - Fixed a bug where Odyssey would try to resolve an ip & port combo after using urlparse.netloc as the domain to be resolve - Made the regex for meta tag redirects more lenient - Fixed some grammar errors with variable names - Added slightly more descriptive errors - Fixed a bug where meta tag redirects were handled incorrectly without any extra thought on how they actually work within browsers - Added a few more javascript redirect detections - Odyssey will now exit properly if domain from location header does not resolve

Overall:

• Updated error message format so its more clear what happened

  • Format: [ERROR] (<MODULE>) <TYPE> MESSAGE

• Reorganised imports

• Removed the following modules and included them in left over modules:

  • get_server
  • get_certs
  • get_cookies
  • get_ip_data

Bootstrap:

• Added default values if the ip metadata doesn't supply a value

Config:

• Added hardcoded default configuration values if none are set in the configuration file

  • User Agent: "Mozilla/5.0"
  • Segment Buffer: 4096
  • Tracking Cookies: []
  • Clear Cookies: []

• Changed configuration name from config.json to odyssey.json

• Fixed a weird bug where the config.json would not be detected properly

• Added a new configuration setting DISPLAY_TRACKERS which should be used if you want to display the tracking cookies in the results Odyssey outputs, defaults to displaying tracking cookies, if set to False it will display all cookies set by the websites visited in the route

Requester:

• Renamed the raw socket requests module from requestor.py => requester.py

Parser:

• Made the parser a class called ResponseParser instead a huge function
• Reject parsing anything other than text/html content type
• Renamed content_parser.py => response_parser.py
• Cleaned up code that handles Refresh header redirects
• Cleaned up code that handles Location header redirects
• Future proofed the location header host ip check to use the ipaddress module, and used ipaddress' is_global field to check if an IP is routeable

Utils:

• Renamed dictionarize function => _parse_headers
• Moved _parse_headers function and get_value function into a class called HeaderUtils
• Changed regular expression to be more inclusive for the find_urls function
• Moved the get_cookies function into utils module
• Moved schemes dictionary from various files into one file for easier use

• Moved most the code that was in the trace_path module to the requestor module
• Merged http_request & https_request modules into one combined module called requestor which now handles both http & https urls and does all the parsing that trace_path did prior to sending requests to the next url
• Added a double check for the content mimetype, since we cannot trust any server on the internet (Ref: https://stackoverflow.com/a/43049834)
• Made it so only content-types that are deemed to be "text/html" can be passed to BeautifulSoup parser to be parsed to find HTML-based redirects
• Set default encoding cp437 to be used when decoding arbitrary html content, otherwise it will try to locate and use the encoding specified in the content-type header
• Refractored the get_certs module to allow any port to be used in the SSL communication to get the certificate of the web server just like the code in the requestor module which was previously located in the trace_path module

• Updated ip data util func docustring
• Updated content type check so we dont run it weird unicode errors
• Updated cookie retrieval to retain cookie case sensitivity
• Added a fix for urls with ports in them not working, from now it will take the port from the url to be the requesting port in the request unless the url has no port then it will default to the port of the url scheme
• Fixed a typo on one of the tracking cookies
• Fixed a bug where meta redirect urls would lose their case sensitivity after being extracted from the meta tags, same for the refresh header extracted URL

• Added tracker boolean to each point's tooltip on the map
• Added odyssey configuration file, to allow people to change certain settings the software uses (ex. useragent & tracking cookies presets)
• Added configuration value loader
• Made the method to get cookie prefixes and data not case sensitive as it used to be
• General code fixes to follow PEPs & Codacy scan results

• Project renamed to Odyssey
• New repository logo
• Added requirements.txt for ease of installation
• Changed from using format() string function to f-strings, in order to clean up the code a bit, because it was a bit bulky
• Completely reorganized the codebase files, and correctly segregated them into their respective folders
• Changed the get_data() method a lot, renamed it to get_ip_data() so that the name is meaningful and also made it just get the JSON response from IP-API instead of sending a new request to the API every time a data point was used in the map and CLI response
• Added a check for 'Refresh' http header seen being used by Instagram to redirect out of their website
• Added docustrings for each function the project
• Fixed the bug where decoding the raw content received from the web server threw a UnicodeDecode error
• Added a pre-decode content type check to see if the raw content is actually text which also fixes some issues when decoding the raw response using 'unicode-escape' (the UnicodeDecode error)
• Made the new method to get the headers of the raw response independent of the raw response content

• Fixed a case where the next URL was seen to be a local ip / resolved to a local ip in the location header of the previous URL
• Added an offset setting in the code for URLs that are at the same geo coordinates, so that they can be distinct
• Changed font family for tool-tip on map
• Added distinctions color-wise for the markers, green being the start, orange being the bounces and red being the end
• Changed icon of markers

• Added map generation to the URL redirect traceroute, so it can be visualized where the request goes around the world.
• With this update, I had to add the time module to sleep the code, as the IP-API has a rate-limit and sometimes with many requests, the script can through an error from IP-API. I will be working on finding a similar IP information API to use in the future, to avoid this problem.
• Added the folium module, to help with the map generation seen in this update

ContentParser:

• Added continuation statement to skip over javascript that does not help us find new redirects
• Added new method to clear HTML tags from strings in order to get more accurate redirects from the javascript code of a website
• Updated raw content decoding method from utf-8 to unciode-escape to handle python unicode characters (ex. \u0003), this still does not handle other languages yet

Start:

• Added SSL certificate retrieval for results

Utils:

• Added new get_ssl_cert() method in the utils.py module to get the ssl certificate from the url passed as the parameter
• Added some more tracking cookie prefixes
• Updated raw content decoding method from utf-8 to unciode-escape to handle python unicode characters (ex. \u0003), this still does not handle other languages yet

• Acknowledgement of .decode() method not being appropriate for the purpose
• Adjusted result message format
• Added tracking cookies to the result message
• Added cookie detection method = Added tracking cookie detection function to the cookie detection method (will be adding more tracking cookies to the code as it goes along)

• Added more code to detect different types of javascript-based redirects [contentparser module]
• Added the type of server running on the URL in the result traceroute, defaults to None if nothing is detected [start module]
• Added code to detect local redirects as they weren't being picked up [start module]
• Fixed an issue with links that included fragments but no queries, which would lead the request path being the wrong one, leading to inaccurate traceroutes [start module]
• Added code to detect singular line javascript code that redirects users [contentparser module]
• Quick code patch to certain cases of meta tags not being picked up by the contentparser module [contentparser module]
• Added check to see if next_url variable is set if last redirect was local-based (for some reason it has issues) [start module]
• Added new get_server() function to the utils module to get the type of server running on the URL [utils module]

Just a quick update to fix my docustrings & naming conventions, also added some javascript analysis code in the contentparser.py module to analyse javascript-based redirects

Added the zip file with all the modules (as seen in the main repo branch) required to run the project