Build capture utility for Linux

Last update: Nov 03, 2022

Related tags

General Utilities cxbuild

Overview

CX-BUILD

Compilation Database alternative

Build

Prerequisite

the CXBUILD uses linux system call trace utility called strace which was customized. So If you want use the cxbuild, you have to build the custom strace(It prints whole path to recognize build commands, https://github.com/damho1104/strace.git)

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
make -j8

When after you build 'cstrace', please copy cstrace binary into '/usr/bin' or '/bin' directory(or add $PATH environement variable) to make the cxbuild freely invokes.

Export single portable binary

Sometimes python and python module requirements is annoying. the pyinstaller can generate everything in single binary. It is really helpful in many cases.

To do this, Required Python 3.x(we tested 3.6.9) and above version. when you ready to build the cxbuild binary, at the source root directory, type below:

python3 -mvenv python
source python/bin/activate
pip install wheel
pip install -r requirements.txt

pip install pyinstaller
pyinstaller build.spec
chmod +x dist/cxbuild
cp dist/cxbuild /usr/local/bin

or Just use cxbuild with python interpreter like this:

source env/bin/activate
pip install -r requirements.txt
python cxbuild.py capture make -j 8

Usage

Capture command

When you ready build your repository, you can use 'capture' command, to make compilation database

cxbuild capture [build command here]

If succeed, "[pwd]/.xdb/compile-commands.json" file will be generated(some intermediate file exist).

Captured command

You can run post process only after build capture completed, It only required when debug something.

The 'captured' command assumes that there exists trace files(full_trace.log) in .xdb directory. and This command run only post process phase, It will generate compile_commands.json file from full_trace.log.

cxbuild captured

Try it

Assumes that 'cstrace' and 'cxbuild' exist in $PATH environment

/bin/cstrace
/bin/cxbuild

gzip

Prepare source build (in Ubuntu OS, apt source command will serve source files to build):

$ apt source gzip
$ls -al
total 1084
drwxr-xr-x  3 minhyuk minhyuk    4096 Nov  1 10:51 .
drwxr-xr-x 12 minhyuk minhyuk    4096 Nov  1 10:51 ..
drwxr-xr-x 11 minhyuk minhyuk    4096 Nov  1 10:52 gzip-1.6
-rw-r--r--  1 minhyuk minhyuk   15604 Jul  8 01:53 gzip_1.6-5ubuntu1.1.debian.tar.xz
-rw-r--r--  1 minhyuk minhyuk    2060 Jul  8 01:53 gzip_1.6-5ubuntu1.1.dsc
-rw-r--r--  1 minhyuk minhyuk 1074924 Aug 21  2013 gzip_1.6.orig.tar.gz

then prepare build tools

$ cd gzip-1.6
$ ./configure

actual build will with cxbuild just type like this:

Making all in tests make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/tests' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6/tests' make[1]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6' -- [Analyzing build Activities] /home/minhyuk/cx/tests/gzip-1.6/.xdb/xtrace_tree.json > Finished(00:00:09) ">

$ cxbuild capture make -j 8
> eXtension of Compilation Database
-- [Build]
make -j 8
cstrace -s 65535 -e trace=open,openat,execve -v -y -f -o "/home/minhyuk/cx/tests/gzip-1.6/.xdb/full_cstrace.log" -- make -j 8
  GEN      version.c
  GEN      version.h
make  all-recursive
make[1]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6'
Making all in lib
make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/lib'
  GEN      alloca.h
  GEN      configmake.h
  GEN      c++defs.h
  GEN      arg-nonnull.h
  GEN      warn-on-use.h
  GEN      unused-parameter.h
<...skipped...>
Making all in tests
make[2]: Entering directory '/home/minhyuk/cx/tests/gzip-1.6/tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6/tests'
make[1]: Leaving directory '/home/minhyuk/cx/tests/gzip-1.6'
-- [Analyzing build Activities]
/home/minhyuk/cx/tests/gzip-1.6/.xdb/xtrace_tree.json
> Finished(00:00:09)

Check compile_commands.json:

] ">

$ cat .xdb/compile_commands.json
[
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.o -MD -MP -MF .deps/c-strcasecmp.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strcasecmp.c",
       "file": "c-strcasecmp.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.o -MD -MP -MF .deps/c-ctype.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-ctype.c",
       "file": "c-ctype.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.o -MD -MP -MF .deps/cloexec.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.o /home/minhyuk/cx/tests/gzip-1.6/lib/cloexec.c",
       "file": "cloexec.c"
   },
   {
       "directory": "/home/minhyuk/cx/tests/gzip-1.6/lib",
       "command": "gcc -D HAVE_CONFIG_H -I /home/minhyuk/cx/tests/gzip-1.6/lib -g -O2 -MT /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.o -MD -MP -MF .deps/c-strncasecmp.Tpo -c -o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.o /home/minhyuk/cx/tests/gzip-1.6/lib/c-strncasecmp.c",
       "file": "c-strncasecmp.c"
   },
   <...skipped...>
]

MongoDB utility to inflate the contents of small collection to a new larger collection

MongoDB Data Inflater ("data-inflater") The data-inflater tool is a MongoDB utility to automate the creation of a new large database collection using

3 Nov 28, 2021

A utility tool to create .env files

A utility tool to create .env files dump-env takes an .env.template file and some optional environmental variables to create a new .env file from thes

89 Dec 8, 2022

Yet another retry utility in Python

Yet another retry utility in Python, avereno being the Malagasy word for retry.

Haute École d'Informatique de Madagascar

4 Nov 2, 2021

The git for the Python Story Utility Package library.

SUP The git for the Python Story Utility Package library. Installation: Install SUP by simply running pip install psup in your terminal. Check out our

6 Nov 27, 2022

A utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.

Mixed development source packages on top of stable constraints using pip mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects

6 Jun 8, 2022

A collection of utility functions to prototype geometry processing research in python

gpytoolbox This repo is a work in progress and contains general utility functions I have needed to code while trying to work on geometry process resea

73 Jan 6, 2023

A morse code encoder and decoder utility.

morsedecode A morse code encoder and decoder utility. Installation Install it via pip: pip install morsedecode Alternatively, you can use pipx to run

2 Dec 25, 2021

Utility to add/remove licenses to/from source files

Utility to add/remove licenses to/from source files. Supports processing any combination of globs, files, and directories (recurse). Pruning options allow skipping non-licensing files.

2 Jan 29, 2022

EVE-NG tools, A Utility to make operations with EVE-NG more friendly.

EVE-NG tools, A Utility to make operations with EVE-NG more friendly. Also it support different snapshot operations with same style as Libvirt/KVM

8 Jan 5, 2023

Comments

__test_dummy_source__.o is generated at cwd
When you capture a program, __test_dummy_source__.o is generated at cwd.

$ cxbuild capture gcc -o test test.c $ ls __test_dummy_source__.o test test.c

This is not intended behavior. This should be fixed.
bug
opened by HansolChoe 0

bug: compile_commands.json may have duplicate data

compile_commands.json may have duplicate data(not entirely duplicated because command is slightly different, but compilation target file is same) like below:

[
    {
        "directory": "/home/workspace/src",
        "command": "gcc -D HAVE_CONFIG_H -I /home/workspace/src -g -O2 -std=gnu99 -Wall -Wextra -Wdeclaration-after-statement -Wpointer-arith -funsigned-char -D _FORTIFY_SOURCE=2 -Wcast-align -Wcast-qual -Wshadow -Wbad-function-cast -Wwrite-strings -Wundef -Wuninitialized -Winit-self -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Waggregate-return -pipe -MT /home/workspace/src/g72x.lo -MD -MP -MF .deps/g72x.Tpo -c -fPIC -D PIC -o /home/workspace/src/.libs/g72x.o -I /usr/lib/gcc/x86_64-linux-gnu/7/include -I /usr/local/include -I /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I /usr/include/x86_64-linux-gnu -I /usr/include /home/workspace/src/g72x.c",
        "file": "/home/workspace/src/g72x.c"
    },
   ...
    {
        "directory": "/home/workspace/src",
        "command": "gcc -D HAVE_CONFIG_H -I /home/workspace/src -g -O2 -std=gnu99 -Wall -Wextra -Wdeclaration-after-statement -Wpointer-arith -funsigned-char -D _FORTIFY_SOURCE=2 -Wcast-align -Wcast-qual -Wshadow -Wbad-function-cast -Wwrite-strings -Wundef -Wuninitialized -Winit-self -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Waggregate-return -pipe -MT /home/workspace/src/g72x.lo -MD -MP -MF .deps/g72x.Tpo -c -o /home/workspace/src/g72x.o -I /usr/lib/gcc/x86_64-linux-gnu/7/include -I /usr/local/include -I /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I /usr/include/x86_64-linux-gnu -I /usr/include /home/workspace/src/g72x.c",
        "file": "/home/workspace/src/g72x.c"
    },
  ...
]

The command below does not have -fPIC -D PIC but other data are same. This is confusing.

This symptom can be reproduced by cxbuild captured command at root of this file(fixed.1.zip). It is normal that .xdb/artifacts.zip is empty.

bug

opened by HansolChoe 0

Some paths are not captured properly (not absoulte path)

In some cases, paths in projects.json are not recorded in absolute path.

If you capture the build information of gzip-1.11, you may see logs like below:

$ ./configure && cxbuild capture make -j10
...
> eXtension of Compilation Database
-- [Analyzing build Activities]
-- [Build trace preprocessed]
-- [project.json written [/home/vagrant/workspace/gzip-1.11/.xdb/project.json]]
-- [compile_commands.json written [/home/vagrant/workspace/gzip-1.11/.xdb/compile_commands.json]]
  > All dependency should start with '//'. Ignore 'threadlib.c'.
  > All dependency should start with '//'. Ignore 'lock.c'.
-- [artifacts.zip written [/home/vagrant/workspace/gzip-1.11/.xdb/artifacts.zip]]

threadlib.c and lock.c are not absolute path. In this case, cxbuild is not killed, but it prints warning messages and ignores the two files.

These files may be okay to ignore, but we need to know when these things happen.

bug

opened by HansolChoe 1

Releases(dpp)

dpp(Nov 3, 2022)
Commits

clang-11 no such compiler error (vulcan-action)

artifact.zip except for library(.so etc) file (vulcan-action)

6c466f1: Fix get_system_include_list and get_predefined_macro (HansolChoe)

48bf810: Fix _ArtifactBuilder to use the function in cslib (HansolChoe)

f7ef04f: Fix _gnu_compiler_tool's compiler setting and pytest (HansolChoe)

b975946: Remove is_debug_mode call csutil (HansolChoe)

1e0b592: Fixed _ArtifactBuilder to collect source or header file (Yeongcheol Kim)

51a75a5: Revised to not remove artifacts directory (Yeongcheol Kim)

1eb568b: Revised to check CXBUILD_OUTPUT_DIR environment variable and set as working_dir (Yeongcheol Kim)

afd9e0b: cc를 compiler로 인식하도록 패턴 추가 (HansolChoe)

72f8b8e: cc를 command로 인식하도록 추가 (HansolChoe)

Source code(tar.gz)
Source code(zip)
Release-cxbuild-Ubuntu-18.04.tar.gz(10.29 MB)

Owner

GLaDOS (G? L? Automatic Debug Operation System)

GitHub Repository

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

3.5k Jan 05, 2023

Dependency Injector is a dependency injection framework for Python.

What is Dependency Injector? Dependency Injector is a dependency injection framework for Python. It helps implementing the dependency injection princi

2.6k Jan 04, 2023

a simple function that randomly generates and applies console text colors

ChangeConsoleTextColour a simple function that randomly generates and applies console text colors This repository corresponds to my Python Functions f

6 Sep 20, 2022

A tiny Python library for generating public IDs from integers

pids Create short public identifiers based on integer IDs. Installation pip install pids Usage from pids import pid public_id = pid.from_int(1234) #

7 Nov 11, 2021

Python implementation of Gorilla time series compression

Gorilla Time Series Compression This is an implementation (with some adaptations) of the compression algorithm described in section 4.1 (Time series c

19 Jan 01, 2023

Software to help automate collecting crowdsourced annotations using Mechanical Turk.

Video Crowdsourcing Software to help automate collecting crowdsourced annotations using Mechanical Turk. The goal of this project is to enable crowdso

1 Oct 25, 2021

A Python class for checking the status of an enabled Minecraft server

mcstatus provides an easy way to query Minecraft servers for any information they can expose. It provides three modes of access (query, status and ping), the differences of which are listed below in

1.1k Jan 06, 2023

A BlackJack simulator in Python to simulate thousands or millions of hands using different strategies.

BlackJack Simulator (in Python) A BlackJack simulator to play any number of hands using different strategies The Rules To keep the code relatively sim

4 Jun 24, 2022

A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urb

12 Sep 01, 2022

A repository containing several general purpose Python scripts to automate daily and common tasks.

General Purpose Scripts Introduction This repository holds a curated list of Python scripts which aim to help us automate daily and common tasks. You

46 Dec 25, 2022

JeNot - A tool to notify you when Jenkins builds are done.

JeNot - Jenkins Notifications NOTE: under construction, buggy, and not production-ready What A tool to notify you when Jenkins builds are done. Why Je

1 Jun 24, 2022

Napari plugin for loading Bitplane Imaris files .ims

napari-imaris-loader Napari plugin for loading Bitplane Imaris files '.ims'. Notes: For this plugin to work "File/Preferences/Experimental/Render Imag

4 Dec 01, 2022

A tool written in python to generate basic repo files from github

7 Dec 02, 2021

Creates a C array from a hex-string or a stream of binary data.

hex2array-c Creates a C array from a hex-string. Usage Usage: python3 hex2array_c.py HEX_STRING [-h|--help] Use '-' to read the hex string from STDIN.

3 Nov 24, 2022

Format Norminette Output!

7 Apr 19, 2022

JavaScript-style async programming for Python.

promisio JavaScript-style async programming for Python. Examples Create a promise-based async function using the promisify decorator. It works on both

191 Dec 30, 2022

Writing Alfred copy/paste scripts in Python

Writing Alfred copy/paste scripts in Python This repository shows how to create Alfred scripts in Python. It assumes that you using pyenv for Python v

2 Oct 26, 2021

Python tool to check a web applications compliance with OWASP HTTP response headers best practices

Check Your Head A quick and easy way to check a web applications response headers!

6 Nov 09, 2021

Hide new MacBook Pro notch with black wallpaper.

1 Oct 27, 2021

A dictionary that can be flattened and re-inflated

deflatable-dict A dictionary that can be flattened and re-inflated. Particularly useful if you're interacting with yaml, for example. Installation wit

2 Oct 18, 2021