SikkieNL/NeoDash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v. 1.0.0.0

Browse Source
This commit is contained in:
NeoDashh
2017-12-08 09:40:03 +07:00
parent 1d588bf976
commit 3a6bb4b0ce
1539 changed files with 340734 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

116
contrib/devtools/README.md Normal file
View File

@@ -0,0 +1,116 @@
Contents
========
This directory contains tools for developers working on this repository.
clang-format.py
===============
A script to format cpp source code according to [.clang-format](../../src/.clang-format). This should only be applied to new files or files which are currently not actively developed on. Also, git subtrees are not subject to formatting.
fix-copyright-headers.py
========================
Every year newly updated files need to have its copyright headers updated to reflect the current year.
If you run this script from the root folder it will automatically update the year on the copyright header for all
source files if these have a git commit from the current year.
For example a file changed in 2015 (with 2015 being the current year):
```// Copyright (c) 2009-2013 The Bitcoin Core developers```
would be changed to:
```// Copyright (c) 2009-2015 The Bitcoin Core developers```
git-subtree-check.sh
====================
Run this script from the root of the repository to verify that a subtree matches the contents of
the commit it claims to have been updated to.
To use, make sure that you have fetched the upstream repository branch in which the subtree is
maintained:
* for `src/secp256k1`: https://github.com/bitcoin/secp256k1.git (branch master)
* for `src/leveldb`: https://github.com/bitcoin/leveldb.git (branch bitcoin-fork)
* for `src/univalue`: https://github.com/bitcoin/univalue.git (branch master)
Usage: `git-subtree-check.sh DIR COMMIT`
`COMMIT` may be omitted, in which case `HEAD` is used.
github-merge.sh
===============
A small script to automate merging pull-requests securely and sign them with GPG.
For example:
./github-merge.sh bitcoin/bitcoin 3077
(in any git repository) will help you merge pull request #3077 for the
bitcoin/bitcoin repository.
What it does:
* Fetch master and the pull request.
* Locally construct a merge commit.
* Show the diff that merge results in.
* Ask you to verify the resulting source tree (so you can do a make
check or whatever).
* Ask you whether to GPG sign the merge commit.
* Ask you whether to push the result upstream.
This means that there are no potential race conditions (where a
pullreq gets updated while you're reviewing it, but before you click
merge), and when using GPG signatures, that even a compromised github
couldn't mess with the sources.
Setup
---------
Configuring the github-merge tool for the bitcoin repository is done in the following way:
git config githubmerge.repository bitcoin/bitcoin
git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
git config --global user.signingkey mykeyid (if you want to GPG sign)
optimize-pngs.py
================
A script to optimize png files in the bitcoin
repository (requires pngcrush).
security-check.py and test-security-check.py
============================================
Perform basic ELF security checks on a series of executables.
symbol-check.py
===============
A script to check that the (Linux) executables produced by gitian only contain
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
still compatible with the minimum supported Linux distribution versions.
Example usage after a gitian build:
find ../gitian-builder/build -type f -executable | xargs python contrib/devtools/symbol-check.py
If only supported symbols are used the return value will be 0 and the output will be empty.
If there are 'unsupported' symbols, the return value will be 1 a list like this will be printed:
.../64/test_neodash: symbol memcpy from unsupported version GLIBC_2.14
.../64/test_neodash: symbol __fdelt_chk from unsupported version GLIBC_2.15
.../64/test_neodash: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
.../64/test_neodash: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
update-translations.py
======================
Run this script from the root of the repository to update all translations from transifex.
It will do the following automatically:
- fetch all translations
- post-process them into valid and committable format
- add missing translations to the build system (TODO)
See doc/translation-process.md for more information.

62
contrib/devtools/clang-format.py Normal file
View File

@@ -0,0 +1,62 @@
#!/usr/bin/env python
'''
Wrapper script for clang-format
Copyright (c) 2015 MarcoFalke
Copyright (c) 2015 The Bitcoin Core developers
Distributed under the MIT software license, see the accompanying
file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
import os
import sys
import subprocess
tested_versions = ['3.6.0', '3.6.1', '3.6.2'] # A set of versions known to produce the same output
accepted_file_extensions = ('.h', '.cpp') # Files to format
def check_clang_format_version(clang_format_exe):
try:
output = subprocess.check_output([clang_format_exe, '-version'])
for ver in tested_versions:
if ver in output:
print "Detected clang-format version " + ver
return
raise RuntimeError("Untested version: " + output)
except Exception as e:
print 'Could not verify version of ' + clang_format_exe + '.'
raise e
def check_command_line_args(argv):
required_args = ['{clang-format-exe}', '{files}']
example_args = ['clang-format-3.x', 'src/main.cpp', 'src/wallet/*']
if(len(argv) < len(required_args) + 1):
for word in (['Usage:', argv[0]] + required_args):
print word,
print ''
for word in (['E.g:', argv[0]] + example_args):
print word,
print ''
sys.exit(1)
def run_clang_format(clang_format_exe, files):
for target in files:
if os.path.isdir(target):
for path, dirs, files in os.walk(target):
run_clang_format(clang_format_exe, (os.path.join(path, f) for f in files))
elif target.endswith(accepted_file_extensions):
print "Format " + target
subprocess.check_call([clang_format_exe, '-i', '-style=file', target], stdout=open(os.devnull, 'wb'), stderr=subprocess.STDOUT)
else:
print "Skip " + target
def main(argv):
check_command_line_args(argv)
clang_format_exe = argv[1]
files = argv[2:]
check_clang_format_version(clang_format_exe)
run_clang_format(clang_format_exe, files)
if __name__ == "__main__":
main(sys.argv)

46
contrib/devtools/fix-copyright-headers.py Normal file
View File

@@ -0,0 +1,46 @@
#!/usr/bin/env python
'''
Run this script to update all the copyright headers of files
that were changed this year.
For example:
// Copyright (c) 2009-2012 The Bitcoin Core developers
it will change it to
// Copyright (c) 2009-2015 The Bitcoin Core developers
'''
import os
import time
import re
year = time.gmtime()[0]
CMD_GIT_DATE = 'git log --format=@%%at -1 %s | date +"%%Y" -u -f -'
CMD_REGEX= "perl -pi -e 's/(20\d\d)(?:-20\d\d)? The Bitcoin/$1-%s The Bitcoin/' %s"
REGEX_CURRENT= re.compile("%s The Bitcoin" % year)
CMD_LIST_FILES= "find %s | grep %s"
FOLDERS = ["./qa", "./src"]
EXTENSIONS = [".cpp",".h", ".py"]
def get_git_date(file_path):
r = os.popen(CMD_GIT_DATE % file_path)
for l in r:
# Result is one line, so just return
return l.replace("\n","")
return ""
n=1
for folder in FOLDERS:
for extension in EXTENSIONS:
for file_path in os.popen(CMD_LIST_FILES % (folder, extension)):
file_path = os.getcwd() + file_path[1:-1]
if file_path.endswith(extension):
git_date = get_git_date(file_path)
if str(year) == git_date:
# Only update if current year is not found
if REGEX_CURRENT.search(open(file_path, "r").read()) is None:
print n,"Last git edit", git_date, "-", file_path
os.popen(CMD_REGEX % (year,file_path))
n = n + 1

74
contrib/devtools/git-subtree-check.sh Normal file
View File

@@ -0,0 +1,74 @@
#!/bin/sh
DIR="$1"
COMMIT="$2"
if [ -z "$COMMIT" ]; then
COMMIT=HEAD
fi
# Taken from git-subtree (Copyright (C) 2009 Avery Pennarun <apenwarr@gmail.com>)
find_latest_squash()
{
dir="$1"
sq=
main=
sub=
git log --grep="^git-subtree-dir: $dir/*\$" \
--pretty=format:'START %H%n%s%n%n%b%nEND%n' "$COMMIT" |
while read a b junk; do
case "$a" in
START) sq="$b" ;;
git-subtree-mainline:) main="$b" ;;
git-subtree-split:) sub="$b" ;;
END)
if [ -n "$sub" ]; then
if [ -n "$main" ]; then
# a rejoin commit?
# Pretend its sub was a squash.
sq="$sub"
fi
echo "$sq" "$sub"
break
fi
sq=
main=
sub=
;;
esac
done
}
latest_squash="$(find_latest_squash "$DIR")"
if [ -z "$latest_squash" ]; then
echo "ERROR: $DIR is not a subtree" >&2
exit 2
fi
set $latest_squash
old=$1
rev=$2
if [ "d$(git cat-file -t $rev 2>/dev/null)" != dcommit ]; then
echo "ERROR: subtree commit $rev unavailable. Fetch/update the subtree repository" >&2
exit 2
fi
tree_subtree=$(git show -s --format="%T" $rev)
echo "$DIR in $COMMIT was last updated to upstream commit $rev (tree $tree_subtree)"
tree_actual=$(git ls-tree -d "$COMMIT" "$DIR" | head -n 1)
if [ -z "$tree_actual" ]; then
echo "FAIL: subtree directory $DIR not found in $COMMIT" >&2
exit 1
fi
set $tree_actual
tree_actual_type=$2
tree_actual_tree=$3
echo "$DIR in $COMMIT currently refers to $tree_actual_type $tree_actual_tree"
if [ "d$tree_actual_type" != "dtree" ]; then
echo "FAIL: subtree directory $DIR is not a tree in $COMMIT" >&2
exit 1
fi
if [ "$tree_actual_tree" != "$tree_subtree" ]; then
git diff-tree $tree_actual_tree $tree_subtree >&2
echo "FAIL: subtree directory tree doesn't match subtree commit tree" >&2
exit 1
fi
echo "GOOD"

185
contrib/devtools/github-merge.sh Normal file
View File

@@ -0,0 +1,185 @@
#!/bin/bash
# This script will locally construct a merge commit for a pull request on a
# github repository, inspect it, sign it and optionally push it.
# The following temporary branches are created/overwritten and deleted:
# * pull/$PULL/base (the current master we're merging onto)
# * pull/$PULL/head (the current state of the remote pull request)
# * pull/$PULL/merge (github's merge)
# * pull/$PULL/local-merge (our merge)
# In case of a clean merge that is accepted by the user, the local branch with
# name $BRANCH is overwritten with the merged result, and optionally pushed.
REPO="$(git config --get githubmerge.repository)"
if [[ "d$REPO" == "d" ]]; then
echo "ERROR: No repository configured. Use this command to set:" >&2
echo "git config githubmerge.repository <owner>/<repo>" >&2
echo "In addition, you can set the following variables:" >&2
echo "- githubmerge.host (default git@github.com)" >&2
echo "- githubmerge.branch (default master)" >&2
echo "- githubmerge.testcmd (default none)" >&2
exit 1
fi
HOST="$(git config --get githubmerge.host)"
if [[ "d$HOST" == "d" ]]; then
HOST="git@github.com"
fi
BRANCH="$(git config --get githubmerge.branch)"
if [[ "d$BRANCH" == "d" ]]; then
BRANCH="master"
fi
TESTCMD="$(git config --get githubmerge.testcmd)"
PULL="$1"
if [[ "d$PULL" == "d" ]]; then
echo "Usage: $0 pullnumber [branch]" >&2
exit 2
fi
if [[ "d$2" != "d" ]]; then
BRANCH="$2"
fi
# Initialize source branches.
git checkout -q "$BRANCH"
if git fetch -q "$HOST":"$REPO" "+refs/pull/$PULL/*:refs/heads/pull/$PULL/*"; then
if ! git log -q -1 "refs/heads/pull/$PULL/head" >/dev/null 2>&1; then
echo "ERROR: Cannot find head of pull request #$PULL on $HOST:$REPO." >&2
exit 3
fi
if ! git log -q -1 "refs/heads/pull/$PULL/merge" >/dev/null 2>&1; then
echo "ERROR: Cannot find merge of pull request #$PULL on $HOST:$REPO." >&2
exit 3
fi
else
echo "ERROR: Cannot find pull request #$PULL on $HOST:$REPO." >&2
exit 3
fi
if git fetch -q "$HOST":"$REPO" +refs/heads/"$BRANCH":refs/heads/pull/"$PULL"/base; then
true
else
echo "ERROR: Cannot find branch $BRANCH on $HOST:$REPO." >&2
exit 3
fi
git checkout -q pull/"$PULL"/base
git branch -q -D pull/"$PULL"/local-merge 2>/dev/null
git checkout -q -b pull/"$PULL"/local-merge
TMPDIR="$(mktemp -d -t ghmXXXXX)"
function cleanup() {
git checkout -q "$BRANCH"
git branch -q -D pull/"$PULL"/head 2>/dev/null
git branch -q -D pull/"$PULL"/base 2>/dev/null
git branch -q -D pull/"$PULL"/merge 2>/dev/null
git branch -q -D pull/"$PULL"/local-merge 2>/dev/null
rm -rf "$TMPDIR"
}
# Create unsigned merge commit.
(
echo "Merge pull request #$PULL"
echo ""
git log --no-merges --topo-order --pretty='format:%h %s (%an)' pull/"$PULL"/base..pull/"$PULL"/head
)>"$TMPDIR/message"
if git merge -q --commit --no-edit --no-ff -m "$(<"$TMPDIR/message")" pull/"$PULL"/head; then
if [ "d$(git log --pretty='format:%s' -n 1)" != "dMerge pull request #$PULL" ]; then
echo "ERROR: Creating merge failed (already merged?)." >&2
cleanup
exit 4
fi
else
echo "ERROR: Cannot be merged cleanly." >&2
git merge --abort
cleanup
exit 4
fi
# Run test command if configured.
if [[ "d$TESTCMD" != "d" ]]; then
# Go up to the repository's root.
while [ ! -d .git ]; do cd ..; done
if ! $TESTCMD; then
echo "ERROR: Running $TESTCMD failed." >&2
cleanup
exit 5
fi
# Show the created merge.
git diff pull/"$PULL"/merge..pull/"$PULL"/local-merge >"$TMPDIR"/diff
git diff pull/"$PULL"/base..pull/"$PULL"/local-merge
if [[ "$(<"$TMPDIR"/diff)" != "" ]]; then
echo "WARNING: merge differs from github!" >&2
read -p "Type 'ignore' to continue. " -r >&2
if [[ "d$REPLY" =~ ^d[iI][gG][nN][oO][rR][eE]$ ]]; then
echo "Difference with github ignored." >&2
else
cleanup
exit 6
fi
fi
read -p "Press 'd' to accept the diff. " -n 1 -r >&2
echo
if [[ "d$REPLY" =~ ^d[dD]$ ]]; then
echo "Diff accepted." >&2
else
echo "ERROR: Diff rejected." >&2
cleanup
exit 6
fi
else
# Verify the result.
echo "Dropping you on a shell so you can try building/testing the merged source." >&2
echo "Run 'git diff HEAD~' to show the changes being merged." >&2
echo "Type 'exit' when done." >&2
if [[ -f /etc/debian_version ]]; then # Show pull number in prompt on Debian default prompt
export debian_chroot="$PULL"
fi
bash -i
read -p "Press 'm' to accept the merge. " -n 1 -r >&2
echo
if [[ "d$REPLY" =~ ^d[Mm]$ ]]; then
echo "Merge accepted." >&2
else
echo "ERROR: Merge rejected." >&2
cleanup
exit 7
fi
fi
# Sign the merge commit.
read -p "Press 's' to sign off on the merge. " -n 1 -r >&2
echo
if [[ "d$REPLY" =~ ^d[Ss]$ ]]; then
if [[ "$(git config --get user.signingkey)" == "" ]]; then
echo "ERROR: No GPG signing key set, not signing. Set one using:" >&2
echo "git config --global user.signingkey <key>" >&2
cleanup
exit 1
else
if ! git commit -q --gpg-sign --amend --no-edit; then
echo "Error signing, exiting."
cleanup
exit 1
fi
fi
else
echo "Not signing off on merge, exiting."
cleanup
exit 1
fi
# Clean up temporary branches, and put the result in $BRANCH.
git checkout -q "$BRANCH"
git reset -q --hard pull/"$PULL"/local-merge
cleanup
# Push the result.
read -p "Type 'push' to push the result to $HOST:$REPO, branch $BRANCH. " -r >&2
if [[ "d$REPLY" =~ ^d[Pp][Uu][Ss][Hh]$ ]]; then
git push "$HOST":"$REPO" refs/heads/"$BRANCH"
fi

75
contrib/devtools/optimize-pngs.py Normal file
View File

@@ -0,0 +1,75 @@
#!/usr/bin/env python
'''
Run this script every time you change one of the png files. Using pngcrush, it will optimize the png files, remove various color profiles, remove ancillary chunks (alla) and text chunks (text).
#pngcrush -brute -ow -rem gAMA -rem cHRM -rem iCCP -rem sRGB -rem alla -rem text
'''
import os
import sys
import subprocess
import hashlib
from PIL import Image
def file_hash(filename):
'''Return hash of raw file contents'''
with open(filename, 'rb') as f:
return hashlib.sha256(f.read()).hexdigest()
def content_hash(filename):
'''Return hash of RGBA contents of image'''
i = Image.open(filename)
i = i.convert('RGBA')
data = i.tobytes()
return hashlib.sha256(data).hexdigest()
pngcrush = 'pngcrush'
git = 'git'
folders = ["src/qt/res/movies", "src/qt/res/icons", "share/pixmaps"]
basePath = subprocess.check_output([git, 'rev-parse', '--show-toplevel']).rstrip('\n')
totalSaveBytes = 0
noHashChange = True
outputArray = []
for folder in folders:
absFolder=os.path.join(basePath, folder)
for file in os.listdir(absFolder):
extension = os.path.splitext(file)[1]
if extension.lower() == '.png':
print("optimizing "+file+"..."),
file_path = os.path.join(absFolder, file)
fileMetaMap = {'file' : file, 'osize': os.path.getsize(file_path), 'sha256Old' : file_hash(file_path)};
fileMetaMap['contentHashPre'] = content_hash(file_path)
pngCrushOutput = ""
try:
pngCrushOutput = subprocess.check_output(
[pngcrush, "-brute", "-ow", "-rem", "gAMA", "-rem", "cHRM", "-rem", "iCCP", "-rem", "sRGB", "-rem", "alla", "-rem", "text", file_path],
stderr=subprocess.STDOUT).rstrip('\n')
except:
print "pngcrush is not installed, aborting..."
sys.exit(0)
#verify
if "Not a PNG file" in subprocess.check_output([pngcrush, "-n", "-v", file_path], stderr=subprocess.STDOUT):
print "PNG file "+file+" is corrupted after crushing, check out pngcursh version"
sys.exit(1)
fileMetaMap['sha256New'] = file_hash(file_path)
fileMetaMap['contentHashPost'] = content_hash(file_path)
if fileMetaMap['contentHashPre'] != fileMetaMap['contentHashPost']:
print "Image contents of PNG file "+file+" before and after crushing don't match"
sys.exit(1)
fileMetaMap['psize'] = os.path.getsize(file_path)
outputArray.append(fileMetaMap)
print("done\n"),
print "summary:\n+++++++++++++++++"
for fileDict in outputArray:
oldHash = fileDict['sha256Old']
newHash = fileDict['sha256New']
totalSaveBytes += fileDict['osize'] - fileDict['psize']
noHashChange = noHashChange and (oldHash == newHash)
print fileDict['file']+"\n size diff from: "+str(fileDict['osize'])+" to: "+str(fileDict['psize'])+"\n old sha256: "+oldHash+"\n new sha256: "+newHash+"\n"
print "completed. Checksum stable: "+str(noHashChange)+". Total reduction: "+str(totalSaveBytes)+" bytes"

181
contrib/devtools/security-check.py Normal file
View File

@@ -0,0 +1,181 @@
#!/usr/bin/python2
'''
Perform basic ELF security checks on a series of executables.
Exit status will be 0 if succesful, and the program will be silent.
Otherwise the exit status will be 1 and it will log which executables failed which checks.
Needs `readelf` (for ELF) and `objdump` (for PE).
'''
from __future__ import division,print_function
import subprocess
import sys
import os
READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
OBJDUMP_CMD = os.getenv('OBJDUMP', '/usr/bin/objdump')
def check_ELF_PIE(executable):
'''
Check for position independent executable (PIE), allowing for address space randomization.
'''
p = subprocess.Popen([READELF_CMD, '-h', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
ok = False
for line in stdout.split('\n'):
line = line.split()
if len(line)>=2 and line[0] == 'Type:' and line[1] == 'DYN':
ok = True
return ok
def get_ELF_program_headers(executable):
'''Return type and flags for ELF program headers'''
p = subprocess.Popen([READELF_CMD, '-l', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
in_headers = False
count = 0
headers = []
for line in stdout.split('\n'):
if line.startswith('Program Headers:'):
in_headers = True
if line == '':
in_headers = False
if in_headers:
if count == 1: # header line
ofs_typ = line.find('Type')
ofs_offset = line.find('Offset')
ofs_flags = line.find('Flg')
ofs_align = line.find('Align')
if ofs_typ == -1 or ofs_offset == -1 or ofs_flags == -1 or ofs_align == -1:
raise ValueError('Cannot parse elfread -lW output')
elif count > 1:
typ = line[ofs_typ:ofs_offset].rstrip()
flags = line[ofs_flags:ofs_align].rstrip()
headers.append((typ, flags))
count += 1
return headers
def check_ELF_NX(executable):
'''
Check that no sections are writable and executable (including the stack)
'''
have_wx = False
have_gnu_stack = False
for (typ, flags) in get_ELF_program_headers(executable):
if typ == 'GNU_STACK':
have_gnu_stack = True
if 'W' in flags and 'E' in flags: # section is both writable and executable
have_wx = True
return have_gnu_stack and not have_wx
def check_ELF_RELRO(executable):
'''
Check for read-only relocations.
GNU_RELRO program header must exist
Dynamic section must have BIND_NOW flag
'''
have_gnu_relro = False
for (typ, flags) in get_ELF_program_headers(executable):
# Note: not checking flags == 'R': here as linkers set the permission differently
# This does not affect security: the permission flags of the GNU_RELRO program header are ignored, the PT_LOAD header determines the effective permissions.
# However, the dynamic linker need to write to this area so these are RW.
# Glibc itself takes care of mprotecting this area R after relocations are finished.
# See also http://permalink.gmane.org/gmane.comp.gnu.binutils/71347
if typ == 'GNU_RELRO':
have_gnu_relro = True
have_bindnow = False
p = subprocess.Popen([READELF_CMD, '-d', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
for line in stdout.split('\n'):
tokens = line.split()
if len(tokens)>1 and tokens[1] == '(BIND_NOW)' or (len(tokens)>2 and tokens[1] == '(FLAGS)' and 'BIND_NOW' in tokens[2]):
have_bindnow = True
return have_gnu_relro and have_bindnow
def check_ELF_Canary(executable):
'''
Check for use of stack canary
'''
p = subprocess.Popen([READELF_CMD, '--dyn-syms', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
ok = False
for line in stdout.split('\n'):
if '__stack_chk_fail' in line:
ok = True
return ok
def get_PE_dll_characteristics(executable):
'''
Get PE DllCharacteristics bits
'''
p = subprocess.Popen([OBJDUMP_CMD, '-x', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
for line in stdout.split('\n'):
tokens = line.split()
if len(tokens)>=2 and tokens[0] == 'DllCharacteristics':
return int(tokens[1],16)
return 0
def check_PE_PIE(executable):
'''PIE: DllCharacteristics bit 0x40 signifies dynamicbase (ASLR)'''
return bool(get_PE_dll_characteristics(executable) & 0x40)
def check_PE_NX(executable):
'''NX: DllCharacteristics bit 0x100 signifies nxcompat (DEP)'''
return bool(get_PE_dll_characteristics(executable) & 0x100)
CHECKS = {
'ELF': [
('PIE', check_ELF_PIE),
('NX', check_ELF_NX),
('RELRO', check_ELF_RELRO),
('Canary', check_ELF_Canary)
],
'PE': [
('PIE', check_PE_PIE),
('NX', check_PE_NX)
]
}
def identify_executable(executable):
with open(filename, 'rb') as f:
magic = f.read(4)
if magic.startswith(b'MZ'):
return 'PE'
elif magic.startswith(b'\x7fELF'):
return 'ELF'
return None
if __name__ == '__main__':
retval = 0
for filename in sys.argv[1:]:
try:
etype = identify_executable(filename)
if etype is None:
print('%s: unknown format' % filename)
retval = 1
continue
failed = []
for (name, func) in CHECKS[etype]:
if not func(filename):
failed.append(name)
if failed:
print('%s: failed %s' % (filename, ' '.join(failed)))
retval = 1
except IOError:
print('%s: cannot open' % filename)
retval = 1
exit(retval)

163
contrib/devtools/symbol-check.py Normal file
View File

@@ -0,0 +1,163 @@
#!/usr/bin/python2
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
A script to check that the (Linux) executables produced by gitian only contain
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
still compatible with the minimum supported Linux distribution versions.
Example usage:
find ../gitian-builder/build -type f -executable | xargs python contrib/devtools/symbol-check.py
'''
from __future__ import division, print_function
import subprocess
import re
import sys
import os
# Debian 6.0.9 (Squeeze) has:
#
# - g++ version 4.4.5 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=g%2B%2B)
# - libc version 2.11.3 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=libc6)
# - libstdc++ version 4.4.5 (https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=libstdc%2B%2B6)
#
# Ubuntu 10.04.4 (Lucid Lynx) has:
#
# - g++ version 4.4.3 (http://packages.ubuntu.com/search?keywords=g%2B%2B&searchon=names&suite=lucid&section=all)
# - libc version 2.11.1 (http://packages.ubuntu.com/search?keywords=libc6&searchon=names&suite=lucid&section=all)
# - libstdc++ version 4.4.3 (http://packages.ubuntu.com/search?suite=lucid&section=all&arch=any&keywords=libstdc%2B%2B&searchon=names)
#
# Taking the minimum of these as our target.
#
# According to GNU ABI document (http://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html) this corresponds to:
# GCC 4.4.0: GCC_4.4.0
# GCC 4.4.2: GLIBCXX_3.4.13, CXXABI_1.3.3
# (glibc) GLIBC_2_11
#
MAX_VERSIONS = {
'GCC': (4,4,0),
'CXXABI': (1,3,3),
'GLIBCXX': (3,4,13),
'GLIBC': (2,11)
}
# See here for a description of _IO_stdin_used:
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
# Ignore symbols that are exported as part of every executable
IGNORE_EXPORTS = {
'_edata', '_end', '_init', '__bss_start', '_fini', '_IO_stdin_used'
}
READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
CPPFILT_CMD = os.getenv('CPPFILT', '/usr/bin/c++filt')
# Allowed NEEDED libraries
ALLOWED_LIBRARIES = {
# bitcoind and bitcoin-qt
'libgcc_s.so.1', # GCC base support
'libc.so.6', # C library
'libpthread.so.0', # threading
'libanl.so.1', # DNS resolve
'libm.so.6', # math library
'librt.so.1', # real-time (clock)
'ld-linux-x86-64.so.2', # 64-bit dynamic linker
'ld-linux.so.2', # 32-bit dynamic linker
# bitcoin-qt only
'libX11-xcb.so.1', # part of X11
'libX11.so.6', # part of X11
'libxcb.so.1', # part of X11
'libfontconfig.so.1', # font support
'libfreetype.so.6', # font parsing
'libdl.so.2' # programming interface to dynamic linker
}
class CPPFilt(object):
'''
Demangle C++ symbol names.
Use a pipe to the 'c++filt' command.
'''
def __init__(self):
self.proc = subprocess.Popen(CPPFILT_CMD, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def __call__(self, mangled):
self.proc.stdin.write(mangled + '\n')
return self.proc.stdout.readline().rstrip()
def close(self):
self.proc.stdin.close()
self.proc.stdout.close()
self.proc.wait()
def read_symbols(executable, imports=True):
'''
Parse an ELF executable and return a list of (symbol,version) tuples
for dynamic, imported symbols.
'''
p = subprocess.Popen([READELF_CMD, '--dyn-syms', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Could not read symbols for %s: %s' % (executable, stderr.strip()))
syms = []
for line in stdout.split('\n'):
line = line.split()
if len(line)>7 and re.match('[0-9]+:$', line[0]):
(sym, _, version) = line[7].partition('@')
is_import = line[6] == 'UND'
if version.startswith('@'):
version = version[1:]
if is_import == imports:
syms.append((sym, version))
return syms
def check_version(max_versions, version):
if '_' in version:
(lib, _, ver) = version.rpartition('_')
else:
lib = version
ver = '0'
ver = tuple([int(x) for x in ver.split('.')])
if not lib in max_versions:
return False
return ver <= max_versions[lib]
def read_libraries(filename):
p = subprocess.Popen([READELF_CMD, '-d', '-W', filename], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
if p.returncode:
raise IOError('Error opening file')
libraries = []
for line in stdout.split('\n'):
tokens = line.split()
if len(tokens)>2 and tokens[1] == '(NEEDED)':
match = re.match('^Shared library: \[(.*)\]$', ' '.join(tokens[2:]))
if match:
libraries.append(match.group(1))
else:
raise ValueError('Unparseable (NEEDED) specification')
return libraries
if __name__ == '__main__':
cppfilt = CPPFilt()
retval = 0
for filename in sys.argv[1:]:
# Check imported symbols
for sym,version in read_symbols(filename, True):
if version and not check_version(MAX_VERSIONS, version):
print('%s: symbol %s from unsupported version %s' % (filename, cppfilt(sym), version))
retval = 1
# Check exported symbols
for sym,version in read_symbols(filename, False):
if sym in IGNORE_EXPORTS:
continue
print('%s: export of symbol %s not allowed' % (filename, cppfilt(sym)))
retval = 1
# Check dependency libraries
for library_name in read_libraries(filename):
if library_name not in ALLOWED_LIBRARIES:
print('%s: NEEDED library %s is not allowed' % (filename, library_name))
retval = 1
exit(retval)

60
contrib/devtools/test-security-check.py Normal file
View File

@@ -0,0 +1,60 @@
#!/usr/bin/python2
'''
Test script for security-check.py
'''
from __future__ import division,print_function
import subprocess
import sys
import unittest
def write_testcode(filename):
with open(filename, 'w') as f:
f.write('''
#include <stdio.h>
int main()
{
printf("the quick brown fox jumps over the lazy god\\n");
return 0;
}
''')
def call_security_check(cc, source, executable, options):
subprocess.check_call([cc,source,'-o',executable] + options)
p = subprocess.Popen(['./security-check.py',executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
(stdout, stderr) = p.communicate()
return (p.returncode, stdout.rstrip())
class TestSecurityChecks(unittest.TestCase):
def test_ELF(self):
source = 'test1.c'
executable = 'test1'
cc = 'gcc'
write_testcode(source)
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-zexecstack','-fno-stack-protector','-Wl,-znorelro']),
(1, executable+': failed PIE NX RELRO Canary'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fno-stack-protector','-Wl,-znorelro']),
(1, executable+': failed PIE RELRO Canary'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro']),
(1, executable+': failed PIE RELRO'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-pie','-fPIE']),
(1, executable+': failed RELRO'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE']),
(0, ''))
def test_PE(self):
source = 'test1.c'
executable = 'test1.exe'
cc = 'i686-w64-mingw32-gcc'
write_testcode(source)
self.assertEqual(call_security_check(cc, source, executable, []),
(1, executable+': failed PIE NX'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat']),
(1, executable+': failed PIE'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--dynamicbase']),
(0, ''))
if __name__ == '__main__':
unittest.main()

203
contrib/devtools/update-translations.py Normal file
View File

@@ -0,0 +1,203 @@
#!/usr/bin/python
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Run this script from the root of the repository to update all translations from
transifex.
It will do the following automatically:
- fetch all translations using the tx tool
- post-process them into valid and committable format
- remove invalid control characters
- remove location tags (makes diffs less noisy)
TODO:
- auto-add new translations to the build system according to the translation process
'''
from __future__ import division, print_function
import subprocess
import re
import sys
import os
import io
import xml.etree.ElementTree as ET
# Name of transifex tool
TX = 'tx'
# Name of source language file
SOURCE_LANG = 'neodash_en.ts'
# Directory with locale files
LOCALE_DIR = 'src/qt/locale'
# Minimum number of messages for translation to be considered at all
MIN_NUM_MESSAGES = 10
def check_at_repository_root():
if not os.path.exists('.git'):
print('No .git directory found')
print('Execute this script at the root of the repository', file=sys.stderr)
exit(1)
def fetch_all_translations():
if subprocess.call([TX, 'pull', '-f', '-a']):
print('Error while fetching translations', file=sys.stderr)
exit(1)
def find_format_specifiers(s):
'''Find all format specifiers in a string.'''
pos = 0
specifiers = []
while True:
percent = s.find('%', pos)
if percent < 0:
break
try:
specifiers.append(s[percent+1])
except:
print('Failed to get specifier')
pos = percent+2
return specifiers
def split_format_specifiers(specifiers):
'''Split format specifiers between numeric (Qt) and others (strprintf)'''
numeric = []
other = []
for s in specifiers:
if s in {'1','2','3','4','5','6','7','8','9'}:
numeric.append(s)
else:
other.append(s)
# numeric (Qt) can be present in any order, others (strprintf) must be in specified order
return set(numeric),other
def sanitize_string(s):
'''Sanitize string for printing'''
return s.replace('\n',' ')
def check_format_specifiers(source, translation, errors, numerus):
source_f = split_format_specifiers(find_format_specifiers(source))
# assert that no source messages contain both Qt and strprintf format specifiers
# if this fails, go change the source as this is hacky and confusing!
#assert(not(source_f[0] and source_f[1]))
try:
translation_f = split_format_specifiers(find_format_specifiers(translation))
except IndexError:
errors.append("Parse error in translation for '%s': '%s'" % (sanitize_string(source), sanitize_string(translation)))
return False
else:
if source_f != translation_f:
if numerus and source_f == (set(), ['n']) and translation_f == (set(), []) and translation.find('%') == -1:
# Allow numerus translations to omit %n specifier (usually when it only has one possible value)
return True
errors.append("Mismatch between '%s' and '%s'" % (sanitize_string(source), sanitize_string(translation)))
return False
return True
def all_ts_files(suffix=''):
for filename in os.listdir(LOCALE_DIR):
# process only language files, and do not process source language
if not filename.endswith('.ts'+suffix) or filename == SOURCE_LANG+suffix:
continue
if suffix: # remove provided suffix
filename = filename[0:-len(suffix)]
filepath = os.path.join(LOCALE_DIR, filename)
yield(filename, filepath)
FIX_RE = re.compile(b'[\x00-\x09\x0b\x0c\x0e-\x1f]')
def remove_invalid_characters(s):
'''Remove invalid characters from translation string'''
return FIX_RE.sub(b'', s)
# Override cdata escape function to make our output match Qt's (optional, just for cleaner diffs for
# comparison, disable by default)
_orig_escape_cdata = None
def escape_cdata(text):
text = _orig_escape_cdata(text)
text = text.replace("'", '&apos;')
text = text.replace('"', '&quot;')
return text
def postprocess_translations(reduce_diff_hacks=False):
print('Checking and postprocessing...')
if reduce_diff_hacks:
global _orig_escape_cdata
_orig_escape_cdata = ET._escape_cdata
ET._escape_cdata = escape_cdata
for (filename,filepath) in all_ts_files():
os.rename(filepath, filepath+'.orig')
have_errors = False
for (filename,filepath) in all_ts_files('.orig'):
# pre-fixups to cope with transifex output
parser = ET.XMLParser(encoding='utf-8') # need to override encoding because 'utf8' is not understood only 'utf-8'
with open(filepath + '.orig', 'rb') as f:
data = f.read()
# remove control characters; this must be done over the entire file otherwise the XML parser will fail
data = remove_invalid_characters(data)
tree = ET.parse(io.BytesIO(data), parser=parser)
# iterate over all messages in file
root = tree.getroot()
for context in root.findall('context'):
for message in context.findall('message'):
numerus = message.get('numerus') == 'yes'
source = message.find('source').text
translation_node = message.find('translation')
# pick all numerusforms
if numerus:
translations = [i.text for i in translation_node.findall('numerusform')]
else:
translations = [translation_node.text]
for translation in translations:
if translation is None:
continue
errors = []
valid = check_format_specifiers(source, translation, errors, numerus)
for error in errors:
print('%s: %s' % (filename, error))
if not valid: # set type to unfinished and clear string if invalid
translation_node.clear()
translation_node.set('type', 'unfinished')
have_errors = True
# Remove location tags
for location in message.findall('location'):
message.remove(location)
# Remove entire message if it is an unfinished translation
if translation_node.get('type') == 'unfinished':
context.remove(message)
# check if document is (virtually) empty, and remove it if so
num_messages = 0
for context in root.findall('context'):
for message in context.findall('message'):
num_messages += 1
if num_messages < MIN_NUM_MESSAGES:
print('Removing %s, as it contains only %i messages' % (filepath, num_messages))
continue
# write fixed-up tree
# if diff reduction requested, replace some XML to 'sanitize' to qt formatting
if reduce_diff_hacks:
out = io.BytesIO()
tree.write(out, encoding='utf-8')
out = out.getvalue()
out = out.replace(b' />', b'/>')
with open(filepath, 'wb') as f:
f.write(out)
else:
tree.write(filepath, encoding='utf-8')
return have_errors
if __name__ == '__main__':
check_at_repository_root()
# fetch_all_translations()
postprocess_translations()