These scripts look for non-printable unicode characters in all text files in a source tree

Overview

find-unicode-control

These scripts look for non-printable unicode characters in all text files in a source tree. find_unicode_control.py should work with python2 as well as python3. It uses python-magic if available to determine file type, or simply spawns the file --mime-type command. They should be functionally the same and find_unicode_control.py could eventually get disposed.

usage: find_unicode_control.py [-h] [-p {all,bidi}] [-v] [-c CONFIG] path [path ...]

Look for Unicode control characters

positional arguments:
  path                  Sources to analyze

optional arguments:
  -h, --help            show this help message and exit
  -p {all,bidi}, --nonprint {all,bidi}
                        Look for either all non-printable unicode characters or bidirectional control characters.
  -v, --verbose         Verbose mode.
  -d, --detailed        Print line numbers where characters occur.
  -t, --notests         Exclude tests (basically test.* as a component of path).
  -c CONFIG, --config CONFIG
                        Configuration file to read settings from.

If unicode BIDI control characters or non-printable characters are found in a file, it will print output as follows:

$ python3 find_unicode_control.py -p bidi *.c
commenting-out.c: bidirectional control characters: {'\u202e', '\u2066', '\u2069'}
early-return.c: bidirectional control characters: {'\u2067'}
stretched-string.c: bidirectional control characters: {'\u202e', '\u2066', '\u2069'}

Using the -d flag, the output is more detailed, showing line numbers in files, but this mode is also slower:

find_unicode_control.py -p bidi -d .
./commenting-out.c:4 bidirectional control characters: ['\u202e', '\u2066', '\u2069', '\u2066']
./commenting-out.c:6 bidirectional control characters: ['\u202e', '\u2066']
./early-return.c:4 bidirectional control characters: ['\u2067']
./stretched-string.c:6 bidirectional control characters: ['\u202e', '\u2066', '\u2069', '\u2066']

The optimal workflow would be to do a quick scan through a source tree and if any issues are found, do a detailed scan on only those files.

Configuration file

If files need to be excluded from the scan, make a configuration file and define a scan_exclude variable to a list of regular expressions that match the files or paths to exclude. Alternatively, add a scan_exclude_mime list with the list of mime types to ignore; this can again be a regular expression. Here is an example configuration that glibc uses:

scan_exclude = [
        # Iconv test data
        r'/iconvdata/testdata/',
        # Test case data
        r'libio/tst-widetext.input$',
        # Test script.  This is to silence the warning:
        # 'utf-8' codec can't decode byte 0xe9 in position 2118: invalid continuation byte
        # since the script tests mixed encoding characters.
        r'localedata/tst-langinfo.sh$']

Notes

This script was quickly hacked together to scan repositories with mostly LTR, unicode content. If you have RTL content (either in comments, literals or even identifiers in code), it will give false warnings that you need to weed out. For now you need to exclude such RTL code using scan_exclude but a long term wish list (if this remains relevant, hopefully more sophisticated RTL diagnostics will make it obsolete!) is to handle RTL a bit more intelligently.

Owner
Siddhesh Poyarekar
Toolchain hacker and all round nice guy. My openhub profile will probably tell you more about my work: https://www.openhub.net/accounts/siddhesh
Siddhesh Poyarekar
Casefy (/keษชsfaษช/) is a lightweight Python package to convert the casing of strings

Casefy (/keษชsfaษช/) is a lightweight Python package to convert the casing of strings. It has no third-party dependencies and supports Unicode.

Diego Miguel Lozano 12 Jan 08, 2023
Script to decrypt / import chromium (edge/chrome) cookies

Cloonie Script to decrypt / import chromium (edge/chrome) cookies. Requirements Install the python dependencies via pip: pip install -r requirements.t

Lorenzo Bernardi 5 Sep 13, 2022
A library from RCTI+ to handle RabbitMQ tasks (connect, send, receive, etc) in Python.

Introduction A library from RCTI+ to handle RabbitMQ tasks (connect, send, receive, etc) in Python. Requirements Python =3.7.3 Pika ==1.2.0 Aio-pika

Dali Kewara 1 Feb 05, 2022
Various importers for cointracker

cointracker_importers Various importers for cointracker To convert nexo .csv format to cointracker .csv format: Download nexo csv file. run python Nex

Stefanos Anastasiou 9 Oct 24, 2022
Simple profile athena generator for Fortnite Private Servers.

Profile-Athena-Generator A simple profile athena generator for Fortnite Private Servers. This profile athena generrator features: Item variants Get al

Fevers 10 Aug 27, 2022
Find dependent python scripts of a python script in a project directory.

Find dependent python scripts of a python script in a project directory.

2 Dec 05, 2021
A python lib for generate random string and digits and special characters or A combination of them

A python lib for generate random string and digits and special characters or A combination of them

Torham 4 Nov 15, 2022
Two fast AUC calculation implementations for python

fastauc Two fast AUC calculation implementations for python: python-based is approximately 5X faster than the default sklearn.metrics.roc_auc_score()

Vsevolod Kompantsev 26 Dec 11, 2022
Conveniently measures the time of your loops, contexts and functions.

Conveniently measures the time of your loops, contexts and functions.

Maciej J Mikulski 79 Nov 15, 2022
Animation retargeting tool for Autodesk Maya. Retargets mocap to a custom rig with a few clicks.

Animation Retargeting Tool for Maya A tool for transferring animation data and mocap from a skeleton to a custom rig in Autodesk Maya. Installation: A

Joaen 63 Jan 06, 2023
Software to help automate collecting crowdsourced annotations using Mechanical Turk.

Video Crowdsourcing Software to help automate collecting crowdsourced annotations using Mechanical Turk. The goal of this project is to enable crowdso

Mike Peven 1 Oct 25, 2021
Retrying library for Python

Tenacity Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just

Julien Danjou 4.3k Jan 05, 2023
Small project to interact with python, C, HTML, JavaScript, PHP.

Micro Hidroponic Small project to interact with python, C, HTML, JavaScript, PHP. Table of Contents General Info Technologies Used Screenshots Usage P

Filipe Martins 1 Nov 10, 2021
A Python script that transcript Arcaea chart file (.aff file) into AutoJS touchscreen script which automatically plays the Arcaea chart

ArcaeaAutoplay (AutoJS Version) A Python script that transcript Arcaea chart file (.aff file) into AutoJS touchscreen script which automatically plays

7 Dec 03, 2021
๐Ÿ’‰ ์ฝ”๋กœ๋‚˜ ์ž”์—ฌ๋ฐฑ์‹  ์˜ˆ์•ฝ ๋งคํฌ๋กœ ์ปค์Šคํ…€ ๋นŒ๋“œ (์†๋„ ํ–ฅ์ƒ ๋ฒ„์ „)

Korea-Covid-19-Vaccine-Reservation ์ฝ”๋กœ๋‚˜ ์ž”์—ฌ ๋ฐฑ์‹  ์˜ˆ์•ฝ ๋งคํฌ๋กœ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•œ ์ปค์Šคํ…€ ๋นŒ๋“œ์ž…๋‹ˆ๋‹ค. ๋” ๋น ๋ฅธ ๋ฐฑ์‹  ์˜ˆ์•ฝ์„ ๋ชฉํ‘œ๋กœ ํ•˜๋ฉฐ, ์†๋„๋ฅผ ์šฐ์„ ํ•˜๊ธฐ ๋•Œ๋ฌธ์— ์‚ฌ์šฉ์ž๋Š” ์ด์— ๋Œ€์ฒ˜๊ฐ€ ๊ฐ€๋Šฅํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ง€์ •ํ•œ ์ขŒํ‘œ ๋‚ด ๋Œ€๊ธฐ์ค‘์ธ ๋ณ‘์›์—์„œ ์ž”์—ฌ ๋ฐฑ์‹ 

Queue.ri 21 Aug 15, 2022
API Rate Limit Decorator

ratelimit APIs are a very common way to interact with web services. As the need to consume data grows, so does the number of API calls necessary to re

Tomas Basham 575 Jan 05, 2023
A simple language and reference decompiler/compiler for MHW THK Files

Leviathon A simple language and reference decompiler/compiler for MHW THK Files. Project Goals The project aims to define a language specification for

11 Jan 07, 2023
A string to hashtags module

A string to hashtags module

Fayas Noushad 4 Dec 01, 2021
produces PCA on genotypes from fasta files (popPhyl's ID format)

popPhyl_PCA Performs PCA of genotypes. Works in two steps. 1. Input file A single fasta file containing different loci, in different populations/speci

camille roux 2 Oct 08, 2021
Fraud Multiplication Table Detection in python

Fraud-Multiplication-Table-Detection-in-python In this program, I have detected fraud multiplication table using python without class. Here, I have co

Sachin Vinayak Dabhade 4 Sep 24, 2021