码厩技术博客

#215 Apache 访问 /tmp 目录下的文件失败

项目中有一个下载日志文件的功能，大致流程是 WEB 后端调用底层方法收集并压缩一些 .log 文件生成一个 zip 压缩包，放在 /tmp 目录下，前端访问指定路径下载。
之前系统环境用的 CentOS 6.5，现在升级到了 CentOS 7，结果测试时发现下载文件下载失败。

#214 Linux 文本处理：awk

awk 的文档写出来可能有一本很厚的书，里面甚至有一种内嵌的解释性编程语言在里面。但是我们普通人就把他当一个小工具，了解一下基础用法就好了，不用深入研究。它的基本功能是将字符串切割之后按照 $1 ... $n 来处理，$0 表示整个字符串（整行）。

#213 Linux 文本处理：grep

• egrep = grep -E
• fgrep = grep -F

# -G, --basic-regexp    基本正则, 默认
# -E, --extended-regexp 拓展正则
# -P, --perl-regexp     Perl 正则
# -w 完全匹配字词
# -x 完全匹配整行
grep markjour /var/log/auth.log
grep -E markjour /var/log/auth.log

# -F, --fixed-strings
grep -F markjour /var/log/auth.log

tail -1000 /var/log/auth.log | grep -Ev 'gnome-keyring-daemon|CRON'

• -r 目录
• -R 目录，处理软链
• -v 排除
• -i 忽略大小写 (ignore-case)
• -m 控制匹配次数
• -a 包含二进制内容的文件当作纯文本处理

• -I 包含二进制内容的文件跳过

• -b 输出命中内容的偏移量

• -n 输出行号
• -o 仅输出匹配部分
• -h 不输出文件名（匹配多个文件时默认输出文件名 -H）
• -L, --files-without-match 仅输出没有匹配的文件名
• -l, --files-with-matches 仅输出匹配文件名

• -c, --count 仅输出文件名和匹配行数

• --include=GLOB 只查找匹配 GLOB（文件模式）的文件

• --exclude=GLOB 跳过匹配 GLOB 的文件
• --exclude-from=FILE 跳过所有匹配给定文件内容中任意模式的文件

• --exclude-dir=GLOB 跳过所有匹配 GLOB 的目录

• -B, --before-context=NUM 打印文本及其前面NUM 行

• -A, --after-context=NUM 打印文本及其后面NUM 行
• -C, --context=NUM 打印NUM 行输出文本
• -NUM 等同于 --context=NUM

#212 Subprocess Popen

如果命令不存在就会报：

FileNotFoundError: [Errno 2] No such file or directory: 'pythonjit'

示例

运行命令

import subprocess
subprocess.run('ls')

import subprocess
from subprocess import PIPE

result = subprocess.run(['touch', '/tmp/abc'])
print(result.stdout)

result = subprocess.run(['ls'], stdout=PIPE, text=True)
print(result.stdout)

cmd = ["python", "-c", "import time; time.sleep(3); print('hello')"]
result = subprocess.run(cmd, capture_output=True, text=True)
# capture_output=True => stdout = stderr = PIPE
print(result.stdout)

直接使用 Popen，灵活一些，也就复杂一些：

import sys
import subprocess
cmd = ["python", "-c", "import time; time.sleep(3); print('hello')"]
index = 0
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# print(p.stdout.read())  # 这一句会阻塞，直到程序执行完成
while True:
    index += 1
    try:
        output, errors = p.communicate(timeout=0.1)
        print(output)
        break
    except subprocess.TimeoutExpired:
        print(str(index), end=' ')
        sys.stdout.flush()

只是一个演示，如果需要异步执行的话，Python 3 支持基于 asyncio 的 subprocess。
参见：AsyncIO 异步执行命令

获取输出

import subprocess
output = subprocess.check_output(['ls'], text=True)
print(output)

获取返回代码

import subprocess
from subprocess import DEVNULL
try:
    subprocess.check_call(['ls', '/etc/abc'], stdout=DEVNULL, stderr=DEVNULL)
except subprocess.CalledProcessError as e:
    print(f'命令返回代码: {e.returncode}')

stdin/stdout/stderr

import subprocess
from subprocess import PIPE, STDOUT
process = subprocess.Popen(['grep', 'o'], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
process.stdin.write(b'Hello, World.\nWhats up?\nI am fine.\n3Q!\nAnd you?\n')
process.stdin.close()
return_code = process.wait()
print(process.stdout.read())
# b'Hello, World.\nAnd you?\n'

import subprocess
from subprocess import PIPE

result = subprocess.run(['ls', '-xyz'], stderr=PIPE)
print(repr(result.stderr.decode('utf-8')))

result = subprocess.run(['ls', '-xyz'], stderr=PIPE, text=True)
print(repr(result.stderr))
# 'ls: 不适用的选项 -- y\n请尝试执行 "ls --help" 来获取更多信息。\n'

管道

import subprocess
cmd = [
    ['tail', '-n10000', '/var/log/server.log'],
    ['grep', 'upload file'],
    ['awk', '{print $3}'],
]
stdin = None
for _cmd in cmd:
    proc = subprocess.Popen(_cmd, stdin=stdin, stdout=subprocess.PIPE, text=True)
    stdin = proc.stdout
output, error = proc.communicate()
print(output.strip())
print(error)

属性和方法

• PIPE -1，管道，表示捕获标准输入，或标准输出，或标准错误
  默认 stdin，stdout，stderr 是 None，也就是说没有配置，使用正常的标准输入，标准输出，标准错误。
• STDOUT -2，为标准错误准备，表示捕获标准错误，将其和标准输出混在一起，相当于 2>&1

• DEVNULL -3，空设备，表示丢弃标准输入，或标准输出，或标准错误（重定向到 /dev/null）

• Popen 进程的 Python 封装

• CompletedProcess 程序执行结果，包含命令，参数，返回吗，标准输出，标准错误

• SubprocessError

• CalledProcessError 如果得到错误状态码（非零），并且执行参数中 check=True

• TimeoutExpired 等待子进程超时

• list2cmdline 将命令列表组合一个完整命令

其实标准库中有另一个库可以做这个事情：

import shlex
cmd = 'grep -F "hello world" /tmp/abc.log'
args = shlex.split(cmd)
print(args)
# ['grep', '-F', 'hello world', '/tmp/abc.log']
print(shlex.join(args))
# grep -F 'hello world' /tmp/abc.log

• call 极简版本

def call(*popenargs, timeout=None, **kwargs):
    with Popen(*popenargs, **kwargs) as p:
        try:
            return p.wait(timeout=timeout)
        except:
            p.kill()
            raise

• check_call call 的封装，如果得到错误状态，就抛出 CalledProcessError

def check_call(*popenargs, **kwargs):
    retcode = call(*popenargs, **kwargs)
    if retcode:
        cmd = kwargs.get("args")
        if cmd is None:
            cmd = popenargs[0]
        raise CalledProcessError(retcode, cmd)
    return 0

• check_output 运行程序，获取标准输出

def check_output(*popenargs, timeout=None, **kwargs):
    for kw in ('stdout', 'check'):
        if kw in kwargs:
            raise ValueError(f'{kw} argument not allowed, it will be overridden.')
    if 'input' in kwargs and kwargs['input'] is None:
        if kwargs.get('universal_newlines') or kwargs.get('text') or kwargs.get('encoding') or kwargs.get('errors'):
            empty = ''
        else:
            empty = b''
        kwargs['input'] = empty
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, **kwargs).stdout

• getoutput 运行程序，获取输出

def getoutput(cmd, *, encoding=None, errors=None):
    return getstatusoutput(cmd, encoding=encoding, errors=errors)[1]

• getstatusoutput 运行程序，获取状态码和输出（stdout + stderr）

def getstatusoutput(cmd, *, encoding=None, errors=None):
    try:
        data = check_output(cmd, shell=True, text=True, stderr=STDOUT,
                            encoding=encoding, errors=errors)
        exitcode = 0
    except CalledProcessError as ex:
        data = ex.output
        exitcode = ex.returncode
    if data[-1:] == '\n':
        data = data[:-1]
    return exitcode, data

• run 运行程序

def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs):
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE
    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE
    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                exc.stdout, exc.stderr = process.communicate()
            else:
                process.wait()
            raise
        except:
            process.kill()
            raise
        retcode = process.poll()
        if check and retcode:
            raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
    return CompletedProcess(process.args, retcode, stdout, stderr)

封装关系：

Popen -> call -> check_call
Popen -> run -> check_output -> getstatusoutput -> getoutput

Popen

https://github.com/python/cpython/blob/master/Lib/subprocess.py

class Popen:
    def __init__(self, args, bufsize=-1, executable=None,
                 stdin=None, stdout=None, stderr=None,
                 preexec_fn=None, close_fds=True,
                 shell=False, cwd=None, env=None, universal_newlines=None,
                 startupinfo=None, creationflags=0,
                 restore_signals=True, start_new_session=False,
                 pass_fds=(), *, user=None, group=None, extra_groups=None,
                 encoding=None, errors=None, text=None, umask=-1, pipesize=-1,
                 process_group=None): pass
        # 创建子进程，执行系统命令
        # args          命令和参数
        # bufsize
        # executable    替代
        # stdin
        # stdout
        # stderr
        # preexec_fn
        # close_fds
        # shell
        # cwd
        # env
        # universal_newlines
        # startupinfo
        # creationflags
        # restore_signals
        # start_new_session
        # pass_fds
        # user
        # group
        # extra_groups
        # encoding
        # errors
        # text
        #   如果指定不一样的 text 和 universal_newlines 值，则抛出 SubprocessError
        #   self.text_mode = encoding or errors or text or universal_newlines
        # umask
        # pipesize
        # process_group

    def __repr__(self): pass

    @property
    def universal_newlines(self):
        return self.text_mode

    @universal_newlines.setter
    def universal_newlines(self, universal_newlines):
        self.text_mode = bool(universal_newlines)

    def __enter__(self):
        return self
    def __exit__(self, exc_type, value, traceback): pass
    def __del__(self, _maxsize=sys.maxsize, _warn=warnings.warn): pass

    def communicate(self, input=None, timeout=None) -> (stdout, stderr): pass
    def poll(self) -> None|returncode: pass  # 检查进程是否结束，返回状态码或 None（没有结束）
    def wait(self, timeout=None) -> returncode: pass  # 等待进程结束，返回状态码
    def send_signal(self, sig):
        self.poll()
        if self.returncode is not None:  # 程序已经执行结束
            return
        try:  # 防止并发杀进程
            os.kill(self.pid, sig)
        except ProcessLookupError:
            pass
    def terminate(self):  # 结束进程 -15
        self.send_signal(signal.SIGTERM)
    def kill(self):  # 强杀进程 -9
        self.send_signal(signal.SIGKILL)

#211 Python 执行系统命令的方法

这里是指在本地执行命令的方法，通过 SSH 的方式这里不做讨论。

#210 MySQL Boolean

我们通常使用以下几种方式表示布尔值：

1. tinyint(1)
2. bit(1)
3. enum('Y','N')

在 MySQL 中 bool, boolean 是 tinyint(1) 的别名。

如果只是一个 True OR False 的布尔型，没有比 bit(1) 更合适的了。
但是也有些时候，我们有好几个 bool 型用一个字段表示，最好用 bit(m)，我也用过 int 型。

附：不同值的长度

PS: blob 如果指定长度 m，MySQL 会选择足够存储数据的最小长度。
PS: MySQL set 类型

参考资料与拓展阅读

• MySQL 文档 (v8.0), 11.9 Using Data Types from Other Database Engines
• MySQL 文档 (v5.7), 11.9 Using Data Types from Other Database Engines
• MySQL 文档 (v5.6), 11.8 Using Data Types from Other Database Engines

#209 数据库：应该怎么存储 IP 地址

IP 地址的存储方式就两种：字符串，整型数。

表示方法

ipv4 VARCHAR(15) -- xx.xx.xx.xx
ipv6 VARCHAR(39) -- xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

ipv4 INT UNSIGNED
ipv6 BINARY(16)

-- 如果同时需要表达 IPv4 和 IPv6
ip VARCHAR(39)
ip BINARY(16)

PS：我还在有些地方看到过这种采用 12 位整数表示 IPv4 的方法：'192.168.0.1' => 192168000001, 如果存到 MySQL 的话，只能采用 BIGINT 或 VARCHAR(12) 了。
如果省略中间的分号，ipv6 只需要 VARCHAR(32) 就行了。

整型和字符串的转换

• INET_ATON / INET_NTOA
• INET6_ATON / INET6_NTOA

#208 Python 单元测试框架

• doctest 标准库
• unitest 标准库
• pytest，基于 unitest
  pytest-dev/pytest
• nose/nose2，基于 unitest
  nose-devs/nose
  nose-devs/nose2

其他项目：

1. openatx/uiautomator2
   Android Uiautomator2 Python Wrapper
2. pytest-dev/pytest
   The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
3. pytest-dev/pytest-testinfra
   Testinfra test your infrastructures
4. jeffzh3ng/fuxi
   Penetration Testing Platform
5. locustio/locust
   Scalable user load testing tool written in Python
6. svanoort/pyresttest
   Python Rest Testing
7. SECFORCE/sparta
   Network Infrastructure Penetration Testing Tool
8. sivel/speedtest-cli
   Command line interface for testing internet bandwidth using speedtest.net
9. infobyte/faraday
   Collaborative Penetration Test and Vulnerability Management Platform
10. robotframework/robotframework
    Generic automation framework for acceptance testing and RPA
11. Manisso/fsociety
    fsociety Hacking Tools Pack – A Penetration Testing Framework
12. cocotb/cocotb
    cocotb, a coroutine based cosimulation library for writing VHDL and Verilog testbenches in Python
13. wolever/parameterized
    Parameterized testing with any Python test framework
14. cujanovic/SSRF-Testing
    SSRF (Server Side Request Forgery) testing resources
15. liwanlei/FXTest
    接口自动化测试平台——python+flask版，支持http协议,java 版本开发完毕https://github.com/liwanlei/plan
16. kmmbvnr/django-jenkins
    Plug and play continuous integration with django and jenkins
17. xiaocong/uiautomator
    Python wrapper of Android uiautomator test tool.
18. cobrateam/splinter
    splinter - python test framework for web applications
19. FSecureLABS/needle
    The iOS Security Testing Framework
20. robotframework/SeleniumLibrary
    Web testing library for Robot Framework
21. githublitao/api_automation_test
    接口自动化测试平台，已停止维护，看心情改改
22. ndb796/python-for-coding-test
    [한빛미디어] "이것이 취업을 위한 코딩 테스트다 with 파이썬" 전체 소스코드 저장소입니다.
23. FactoryBoy/factory_boy
    A test fixtures replacement for Python
24. Blazemeter/taurus
    Automation-friendly framework for Continuous Testing by
25. jazzband/django-nose
    Django test runner using nose
26. flipkart-incubator/Astra
    Automated Security Testing For REST API's
27. nose-devs/nose
    nose is nicer testing for python
28. SecuraBV/CVE-2020-1472
    Test tool for CVE-2020-1472
29. localstack/localstack
    💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline!
30. spulec/freezegun
    Let your Python tests travel through time
31. Tencent/FAutoTest
    A UI automated testing framework for H5 and applets
32. robotframework/RIDE
    Test data editor for Robot Framework
33. microsoft/playwright-python
    Python version of the Playwright testing and automation library.
34. autotest/autotest
    Autotest - Fully automated tests on Linux
35. ansible-community/molecule
    Molecule aids in the development and testing of Ansible roles
36. pytest-dev/pytest-xdist
    pytest plugin for distributed testing and loop-on-failures testing modes.
37. powerfulseal/powerfulseal
    A powerful testing tool for Kubernetes clusters.
38. google/nogotofail
    An on-path blackbox network traffic security testing tool
39. wntrblm/nox
    Flexible test automation for Python
40. mjhea0/flaskr-tdd
    Flaskr: Intro to Flask, Test-Driven Development (TDD), and JavaScript
41. aws-ia/taskcat
    Test all the CloudFormation things! (with TaskCat)
42. facebookarchive/huxley
    A testing system for catching visual regressions in Web applications.
43. pytest-dev/pytest-bdd
    BDD library for the py.test runner
44. dagrz/aws_pwn
    A collection of AWS penetration testing junk
45. boxed/mutmut
    Mutation testing system
46. vishnubob/wait-for-it
    Pure bash script to test and wait on the availability of a TCP host and port
47. ethereum/casper
    Casper contract, and related software and tests
48. jhlau/doc2vec
    Python scripts for training/testing paragraph vectors
49. fsociety-team/fsociety
    A Modular Penetration Testing Framework
50. 0xInfection/TIDoS-Framework
    The Offensive Manual Web Application Penetration Testing Framework.
51. praetorian-inc/pentestly
    Python and Powershell internal penetration testing framework
52. hizhangp/yolo_tensorflow
    Tensorflow implementation of YOLO, including training and test phase.
53. zhuifengshen/xmind2testcase
    XMind2TestCase基于python实现，提供了一个高效测试用例设计的解决方案！
54. AnasAboureada/Penetration-Testing-Study-Notes
    Penetration Testing notes, resources and scripts
55. jseidl/GoldenEye
    GoldenEye Layer 7 (KeepAlive+NoCache) DoS Test Tool
56. testcontainers/testcontainers-python
    Testcontainers is a Python library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.
57. spulec/moto
    A library that allows you to easily mock out tests based on AWS infrastructure.
58. buildbot/buildbot
    Python-based continuous integration testing framework; your pull requests are more than welcome!
59. fossasia/open-event-server
    The Open Event Organizer Server to Manage Events https://test-api.eventyay.com
60. AirtestProject/Poco
    A cross-engine test automation framework based on UI inspection
61. Yijunmaverick/CartoonGAN-Test-Pytorch-Torch
    Pytorch and Torch testing code of CartoonGAN [Chen et al., CVPR18]
62. ionelmc/pytest-benchmark
    py.test fixture for benchmarking code
63. aws/aws-sam-cli
    CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
64. crossbario/autobahn-testsuite
    Autobahn WebSocket protocol testsuite
65. sixpack/sixpack
    Sixpack is a language-agnostic a/b-testing framework
66. SofianeHamlaoui/Lockdoor-Framework
    🔐 Lockdoor Framework : A Penetration Testing framework with Cyber Security Resources
67. mongomock/mongomock
    Small library for mocking pymongo collection objects for testing purposes
68. OWASP/owasp-mstg
    The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security testing and reverse engineering. It describes the technical processes for verifying the controls listed in the OWASP Mobile Application Security Verification Standard (MASVS).
69. python-needle/needle
    Automated tests for your CSS.
70. gyoisamurai/GyoiThon
    GyoiThon is a growing penetration test tool using Machine Learning.
71. SeldomQA/seldom
    WebUI/HTTP automation testing framework based on unittest
72. joestump/python-oauth2
    A fully tested, abstract interface to creating OAuth clients and servers.
73. maltize/sublime-text-2-ruby-tests
    Sublime Text 2 plugin for running ruby tests! (Unit, RSpec, Cucumber)
74. emirozer/fake2db
    create custom test databases that are populated with fake data
75. owid/covid-19-data
    Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data
76. Teemu/pytest-sugar
    a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)
77. GeorgeSeif/Semantic-Segmentation-Suite
    Semantic Segmentation Suite in TensorFlow. Implement, train, and test new Semantic Segmentation models easily!
78. jamesls/fakeredis
    Fake implementation of redis API (redis-py) for testing purposes
79. HypothesisWorks/hypothesis
    Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
80. se2p/pynguin
    The PYthoN General UnIt Test geNerator is a test-generation tool for Python
81. luogu-dev/cyaron
    CYaRon: Yet Another Random Olympic-iNformatics test data generator
82. knownsec/pocsuite3
    pocsuite3 is an open-sourced remote vulnerability testing framework developed by the Knownsec 404 Team.
83. seleniumbase/SeleniumBase
    A Python framework that inspires developers to become better test automation engineers. 🧠💡
84. terraform-compliance/cli
    a lightweight, security focused, BDD test framework against terraform.
85. coqui-ai/TTS
    🐸💬 - a deep learning toolkit for Text-to-Speech, battle-tested in research and production
86. facebookresearch/DomainBed
    DomainBed is a suite to test domain generalization algorithms
87. qubvel/ttach
    Image Test Time Augmentation with PyTorch!
88. RhinoSecurityLabs/pacu
    The AWS exploitation framework, designed for testing the security of Amazon Web Services environments.
89. doyensec/inql
    InQL - A Burp Extension for GraphQL Security Testing
90. revsys/django-test-plus
    Useful additions to Django's default TestCase
91. online-judge-tools/oj
    Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
92. ticarpi/jwt_tool
    :snake: A toolkit for testing, tweaking and cracking JSON Web Tokens
93. yashaka/selene
    User-oriented Web UI browser tests in Python
94. davidtvs/pytorch-lr-finder
    A learning rate range test implementation in PyTorch
95. kevinburke/hamms
    Malformed servers to test your HTTP client
96. tarpas/pytest-testmon
    Selects tests affected by changed files. Continous test runner when used with pytest-watch.
97. pluwen/awesome-testflight-link
    Collection of Testflight public app link（iOS/iPad OS/macOS）。
98. facebookarchive/bootstrapped
    Generate bootstrapped confidence intervals for A/B testing in Python.
99. devpi/devpi
    Python PyPi staging server and packaging, testing, release tool
100. eth-brownie/brownie
     A Python-based development and testing framework for smart contracts targeting the Ethereum Virtual Machine.
101. scanapi/scanapi
     Automated Integration Testing and Live Documentation for your API
102. kevin1024/vcrpy
     Automatically mock your HTTP interactions to simplify and speed up testing
103. sodadata/soda-core
     Data profiling, testing, and monitoring for SQL accessible data.
104. CleanCut/green
     Green is a clean, colorful, fast python test runner.
105. trailofbits/deepstate
     A unit test-like interface for fuzzing and symbolic execution
106. ashishb/adb-enhanced
     🔪Swiss-army knife for Android testing and development 🔪 ⛺
107. D4Vinci/One-Lin3r
     Gives you one-liners that aids in penetration testing operations, privilege escalation and more
108. newsapps/beeswithmachineguns
     A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications).
109. nilboy/tensorflow-yolo
     tensorflow implementation of 'YOLO : Real-Time Object Detection'(train and test)
110. NVlabs/FUNIT
     Translate images to unseen domains in the test time with few example images.
111. kyclark/tiny_python_projects
     Code for Tiny Python Projects (Manning, 2020, ISBN 1617297518). Learning Python through test-driven development of games and puzzles.
112. XifengGuo/CapsNet-Keras
     A Keras implementation of CapsNet in NIPS2017 paper "Dynamic Routing Between Capsules". Now test error ＝ 0.34%.
113. defparam/smuggler
     Smuggler - An HTTP Request Smuggling / Desync testing tool written in Python 3
114. joeyespo/pytest-watch
     Local continuous test runner with pytest and watchdog.
115. s0md3v/Blazy
     Blazy is a modern login bruteforcer which also tests for CSRF, Clickjacking, Cloudflare and WAF .
116. chakki-works/sumeval
     Well tested & Multi-language evaluation framework for text summarization.
117. kiwitcms/Kiwi
     The leading open source test management system with over 1 million downloads!
118. chenjj/espoofer
     An email spoofing testing tool that aims to bypass SPF/DKIM/DMARC and forge DKIM signatures.🍻
119. MisterBianco/BoopSuite
     A Suite of Tools written in Python for wireless auditing and security testing.
120. cszn/KAIR
     Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR
121. rueckstiess/mtools
     A collection of scripts to set up MongoDB test environments and parse and visualize MongoDB log files.
122. MozillaSecurity/funfuzz
     A collection of fuzzers in a harness for testing the SpiderMonkey JavaScript engine.
123. schemathesis/schemathesis
     A modern API testing tool for web applications built with Open API and GraphQL specifications.
124. worstcase/blockade
     Docker-based utility for testing network failures and partitions in distributed applications
125. kvalle/diy-lang
     A hands-on, test driven guide to implementing a simple programming language
126. nidhaloff/igel
     a delightful machine learning tool that allows you to train, test, and use models without writing code
127. minimaxir/gpt-3-experiments
     Test prompts for OpenAI's GPT-3 API and the resulting AI-generated texts.
128. tonglei100/sweetest
     小而美的自动化测试解决方案，支持 Web UI 测试，Http 接口测试，DB 操作测试，App 测试，小程序测试，Windows GUI 测试，文件操作
129. sdispater/cleo
     Cleo allows you to create beautiful and testable command-line interfaces.
130. joedicastro/vps-comparison
     A comparison between some VPS providers. It uses Ansible to perform a series of automated benchmark tests over the VPS servers that you specify. It allows the reproducibility of those tests by anyone that wanted to compare these results to their own. All the tests results are available in order to provide independence and transparency.
131. ryansmcgee/seirsplus
     Models of SEIRS epidemic dynamics with extensions, including network-structured populations, testing, contact tracing, and social distancing.
132. karec/cookiecutter-flask-restful
     Flask cookiecutter template for builing APIs with flask-restful, including JWT auth, cli, tests, swagger, docker and more
133. facebookresearch/FixRes
     This repository reproduces the results of the paper: "Fixing the train-test resolution discrepancy" https://arxiv.org/abs/1906.06423
134. honeynet/droidbot
     A lightweight test input generator for Android. Similar to Monkey, but with more intelligence and cool features!
135. darrenburns/ward
     Ward is a modern test framework for Python with a focus on productivity and readability.
136. httpie/http-prompt
     An interactive command-line HTTP and API testing client built on top of HTTPie featuring autocomplete, syntax highlighting, and more. https://twitter.com/httpie
137. codingo/Reconnoitre
     A security tool for multithreaded information gathering and service enumeration whilst building directory structures to store results, along with writing out recommendations for further testing.
138. deepmind/pycolab
     A highly-customisable gridworld game engine with some batteries included. Make your own gridworld games to test reinforcement learning agents!
139. mjpost/sacrebleu
     Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons
140. pschanely/CrossHair
     An analysis tool for Python that blurs the line between testing and type systems.
141. microsoft/restler-fuzzer
     RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
142. RussBaz/enforce
     Python 3.5+ runtime type checking for integration testing and data validation
143. pyvisa/pyvisa
     A Python package with bindings to the "Virtual Instrument Software Architecture" VISA library, in order to control measurement devices and test equipment via GPIB, RS232, or USB.
144. garethdmm/gryphon
     Powerful, proven, and extensible framework for building trading strategies at any frequency, with a focus on crypto currencies. Battle-tested with billions traded.
145. taverntesting/tavern
     A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax
146. dephell/dephell
     :package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.
147. hash3liZer/WiFiBroot
     A Wireless (WPA/WPA2) Pentest/Cracking tool. Captures & Crack 4-way handshake and PMKID key. Also, supports a deauthentication/jammer mode for stress testing
148. GoVanguard/legion
     Legion is an open source, easy-to-use, super-extensible and semi-automated network penetration testing tool that aids in discovery, reconnaissance and exploitation of information systems.
149. EnableSecurity/sipvicious
     SIPVicious OSS has been around since 2007 and is actively updated to help security teams, QA and developers test SIP-based VoIP systems and applications.
150. deepchecks/deepchecks
     Test Suites for Validating ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.
151. microsoft/dowhy
     DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
152. YudeWang/deeplabv3plus-pytorch
     Here is a pytorch implementation of deeplabv3+ supporting ResNet(79.155%) and Xception(79.945%). Multi-scale & flip test and COCO dataset interface has been finished.
153. james-atkinson/speedcomplainer
     A python app that will test your internet connection and then complain to your service provider (and log to a data store if you'd like)
154. flyteorg/flyte
     Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.
155. galkan/crowbar
     Crowbar is brute forcing tool that can be used during penetration tests. It is developed to support protocols that are not currently supported by thc-hydra and other popular brute forcing tools.
156. bslatkin/dpxdt
     Make continuous deployment safe by comparing before and after webpage screenshots for each release. Depicted shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.
157. xtiankisutsa/MARA_Framework
     MARA is a Mobile Application Reverse engineering and Analysis Framework. It is a toolkit that puts together commonly used mobile application reverse engineering and analysis tools to assist in testing mobile applications against the OWASP mobile security threats.
158. Quitten/Autorize
     Automatic authorization enforcement detection extension for burp suite written in Jython developed by Barak Tawily in order to ease application security people work and allow them perform an automatic authorization tests
159. HXSecurity/DongTai
     DongTai is an interactive application security testing(IAST) product that supports the detection of OWASP WEB TOP 10 vulnerabilities, multi-request related vulnerabilities (including logic vulnerabilities, unauthorized access vulnerabilities, etc.), third-party component vulnerabilities, etc.
160. bhdresh/CVE-2017-0199
     Exploit toolkit CVE-2017-0199 - v4.0 is a handy python script which provides pentesters and security researchers a quick and effective way to test Microsoft Office RCE. It could generate a malicious RTF/PPSX file and deliver metasploit / meterpreter / other payload to victim without any complex configuration.
161. screetsec/BruteSploit
     BruteSploit is a collection of method for automated Generate, Bruteforce and Manipulation wordlist with interactive shell. That can be used during a penetration test to enumerate and maybe can be used in CTF for manipulation,combine,transform and permutation some words or file text :p
162. cirosantilli/linux-kernel-module-cheat
     The perfect emulation setup to study and develop the Linux kernel v5.4.3, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 19.10 host.

#207 将文件当作磁盘

dd if=/dev/zero of=cdata.img bs=1G count=5
mkfs ext4 -F cdata.img
LANG=en fdisk -lu cdata.img
sudo mount -o loop cdata.img /mnt/iso

sudo umount /mnt/iso

#206 EUI-64 地址

相关文章：

• MAC 地址

一、概念

不知道为了什么，IEEE 设计了一个 64 位的地址方案，称之为 EUI-64 (Extended Unique Identifier)。
MAC 地址这种又被称之为 EUI-48，确切的说，应该是 MAC 地址是 EUI-48 格式。

目前已知的区别：

1. MAC 地址的概念只存在二层计算机网络。
   而 EUI 不止被用于 IEEE 802 协议（局域网和城域网），可以用于其他类型设备，比如蓝牙。

TODO: 需要一些调查研究。

MAC To EUI64

就是在 OUI 和设备号之间插入 FFFE 即可。

xx:xx:xx:yy:yy:yy
xx:xx:xx:ff:fe:yy:yy:yy

二、场景

根据维基百科，

采用 EUI-48 格式的场景

• IEEE 802 网络
• 以太网
• 802.11 无线网络（Wi-Fi)
• 蓝牙
• IEEE 802.5 令牌环
• 光纤分布式数据接口（FDDI）
• 异步传输模式（ATM），仅作为NSAP 地址的一部分交换虚拟连接
• 光纤通道（FC）和串行连接的 SCSI（SAS）
• ITU-T G.hn 标准
  使用现有家庭布线（电力线、电话线和同轴电缆）创建千兆局域网。

采用 EUI-64 格式的场景

• IEEE 1394 （FireWire），苹果公司和索尼、松下等公司开发的一种串行总线技术
• InfiniBand，一种高性能网络标准（高吞吐，低延迟）
• IPv6
• ZigBee / 802.15.4 / 6LoWPAN
  PS: 6LoWPAN: IPv6 over Low-Power Wireless Personal Area Networks, 低功耗无线个人局域网 IPv6
• IEEE 11073-20601

三、IPv6 中的应用

Update @ 2021-12-30

在 IPv6 的文档中频繁看到这个词。

无状态自动配置机制中：

1. MAC 转 EUI-64
2. U/L 位（首字节低二位）设位 1, 表示本地地址