[笔记] CodeQL for Python 文档学习记录
官方文档：https://help.semmle.com/QL/learn-ql/python/pointsto-type-infer.html
Python-CodeQL中抽象语法树相关的类
Abstract syntax tree
- AstNode > - Module – A Python module > - Class – The body of a class definition > - Function – The body of a function definition > - Stmt – A statement >> - Assert – An assert statement >> - Assign – An assignment >>> - AssignStmt – An assignment statement, x = y >>> - ClassDef – A class definition statement >>> - FunctionDef – A function definition statement >> - AugAssign – An augmented assignment, x += y >> - Break – A break statement >> - Continue – A continue statement >> - Delete – A del statement >> - ExceptStmt – The except part of a try statement >> - Exec – An exec statement >> - For – A for statement >> - If – An if statement >> - Pass – A pass statement >> - Print – A print statement (Python 2 only) >> - Raise – A raise statement >> - Return – A return statement >> - Try – A try statement >> - While – A while statement >> - With – A with statement > - Expr – An expression >> - Attribute – An attribute, obj.attr >> - Call – A function call, f(arg) >> - IfExp – A conditional expression, x if cond else y >> - Lambda – A lambda expression >> - Yield – A yield expression >> - Bytes – A bytes literal, b"x" or (in Python 2) "x" >> - Unicode – A unicode literal, u"x" or (in Python 3) "x" >> - Num – A numeric literal, 3 or 4.2 >>> - IntegerLiteral >>> - FloatLiteral >>> - ImaginaryLiteral >> - Dict – A dictionary literal, {'a': 2} >> - Set – A set literal, {'a', 'b'} >> - List – A list literal, ['a', 'b'] >> - Tuple – A tuple literal, ('a', 'b') >> - DictComp – A dictionary comprehension, {k: v for ...} >> - SetComp – A set comprehension, {x for ...} >> - ListComp – A list comprehension, [x for ...] >> - GenExpr – A generator expression, (x for ...) >> - Subscript – A subscript operation, seq[index] >> - Name – A reference to a variable, var >> - UnaryExpr – A unary operation, -x >> - BinaryExpr – A binary operation, x+y >> - Compare – A comparison operation, 0 < x < 10 >> - BoolExpr – Short circuit logical operations, x and y, x or y
Analyzing control flow
同一个函数作用域中查找互斥块
import python
from BasicBlock b1, BasicBlock b2
where b1 != b2 and not b1.strictlyReaches(b2) and not b2.strictlyReaches(b1) and
exists(Function shared, BasicBlock entry |
entry.contains(shared.getEntryNode()) and
entry.strictlyReaches(b1) and entry.strictlyReaches(b2)
)
select b1, b2
This typically gives a very large number of results, because it is a common occurrence in normal control flow. It is, however, an example of the sort of control-flow analysis that is possible. Control-flow analyses such as this are an important aid to data flow analysis. For more information, see Analyzing data flow and tracking tainted data in Python.
搜索结果特别大，可用性不高，可以优化一下
Pointer analysis & type inference
指针分析： https://zhuanlan.zhihu.com/p/79804033 （算法较为复杂，尝试丢给求解器计算） 类型推断： https://zhuanlan.zhihu.com/p/97441100
The predicate ControlFlowNode.pointsTo(...) shows which object a control flow node may 'point to' at runtime.
ControlFlowNode.pointsTo(...) has three variants:
predicate pointsTo(Value object)
predicate pointsTo(Value object, ControlFlowNode origin)
predicate pointsTo(Context context, Value object, ControlFlowNode origin)
For complex data flow analyses, involving multiple stages, the ControlFlowNode version is more precise, but for simple use cases the Expr based version is easier to use. For convenience, the Expr class also has the same three predicates. Expr.pointsTo(...) also has three variants:
predicate pointsTo(Value object)
predicate pointsTo(Value object, AstNode origin)
predicate pointsTo(Context context, Value object, AstNode origin)
查询无法被执行的异常处理块
当一个Try语句中，先后有多个Handler，但是前面所捕捉的错误类型是后面的超类，就会导致后面的错误捕捉永远无法被执行
该查询中利用i<j来表示Handler的顺序，利用getType和pointsTo确定类关系。
猜测 | 符号在此处的作用类似与管道，把输入定向到后面的子句。注意在第二个exists中由于调用了谓词，所以进行了两次定向。
getASuperType()可以获得超类。
import python
from Try t, ExceptStmt ex1, ExceptStmt ex2
where
exists(int i, int j |
ex1 = t.getHandler(i) and ex2 = t.getHandler(j) and i < j
)
and
exists(ClassValue cls1, ClassValue cls2 |
ex1.getType().pointsTo(cls1) and
ex2.getType().pointsTo(cls2) |
not cls1 = cls2 and
cls1 = cls2.getASuperType()
)
select t, ex1, ex2
寻找可能将不可迭代对象作为循环迭代的位置
不溯源
通过loopup可以查看一些类属性
import python
from For loop, Value iter, ClassValue cls
where loop.getIter().getAFlowNode().pointsTo(iter)
and cls = iter.getClass()
and not exists(cls.lookup("__iter__"))
select loop, cls
溯源（结合AstNode）
使用 predicate pointsTo(Value object, AstNode origin) 这个谓词，并输出origin
import python
from For loop, Value iter, ClassValue cls, AstNode origin
where loop.getIter().pointsTo(iter, origin) and
cls = iter.getClass() and
not cls.hasAttribute("__iter__")
select loop, cls, origin
Finding calls using call-graph analysis
直接通过函数名查找函数调用
import python
from Call call, Name name
where call.getFunc() = name and name.getId() = "eval"
select call, "call to 'eval'."
问题：
- 可能误认为某些对自定义方法名为eval的方法的调用
- 默认了调用的函数名为eval，可能漏掉一些情况
改良版
利用Value::named()和getACall取得对eval正确调用，然后在控制流图上检索出来
同时结合控制流图检索可以有效防止跑到python内置模块中搜索一大堆无关结果，提高了结果的相关性
（技巧：合理利用CFG）
import python
from ControlFlowNode call, Value eval
where eval = Value::named("eval") and
call = eval.getACall()
select call, "call to 'eval'."
Analyzing data flow & tracking tainted data —— 数据流、污点追踪
Tracking user-controlled, or tainted, data is a key technique for security researchers.
数据流分析和污点追踪的主要用途在于分析用户可控的输入是否可能作为污点数据进行恶意行为，或者一些敏感数据会不会被泄露。
CodeQL的CFG分析模块设计思路
主要要解决标准库无法跟踪、变量间赋值导致的混淆以及需要大量计算时间的问题...
- Local data flow, concerning the data flow within a single function. When reasoning about local data flow, you only consider edges between data flow nodes belonging to the same function. It is generally sufficiently fast, efficient and precise for many queries, and it is usually possible to compute the local data flow for all functions in a CodeQL database.
- Global data flow, effectively considers the data flow within an entire program, by calculating data flow between functions and through object properties. Computing global data flow is typically more time and energy intensive than local data flow, therefore queries should be refined to look for more specific sources and sinks.
Taint Tracking 的组成
- One or more sources of potentially insecure or unsafe data, represented by the TaintTracking::Source class. 多个追踪源
- One or more sinks, to where the data or taint may flow, represented by the TaintTracking::Sink class. 多个薄弱位置
- Zero or more sanitizers, represented by the Sanitizer class. 0或多个过滤器
Taint Tracking 查询的基本形式
类继承之后要按照实际情况重写几个主要的谓词，以便具体化源和目标位置的特征。
主要负责完成追踪的是hasFlow方法。
/**
* @name ...
* @description ...
* @kind problem
*/
import semmle.python.security.TaintTracking
class MyConfiguration extends TaintTracking::Configuration {
MyConfiguration() { this = "My example configuration" }
override predicate isSource(TaintTracking::Source src) { ... }
override predicate isSink(TaintTracking::Sink sink) { ... }
/* optionally */
override predicate isExtension(Extension extension) { ... }
}
from MyConfiguration config, TaintTracking::Source src, TaintTracking::Sink sink
where config.hasFlow(src, sink)
select sink, "Alert message, including reference to $@.", src, "string describing the source"
从HTPP请求到Unsafe函数的Track
在定义Sink类型的时候需要继承自TainiTracking::Sink，并写一个类似“构造函数”的东西说明Sink的特征，最后重写sinks方法说明Sink的类型。
/* Import the string taint kind needed by our custom sink */
import semmle.python.security.strings.Untrusted
/* Sources */
import semmle.python.web.HttpRequest
/* Sink */
/** A class representing any argument in a call to a function called "unsafe" */
class UnsafeSink extends TaintTracking::Sink {
UnsafeSink() {
exists(FunctionValue unsafe |
unsafe.getName() = "unsafe" and
unsafe.getACall().(CallNode).getAnArg() = this
)
}
override predicate sinks(TaintKind kind) {
kind instanceof StringKind
}
}
class HttpToUnsafeConfiguration extends TaintTracking::Configuration {
HttpToUnsafeConfiguration() {
this = "Example config finding flow from http request to 'unsafe' function"
}
override predicate isSource(TaintTracking::Source src) { src instanceof HttpRequestTaintSource }
override predicate isSink(TaintTracking::Sink sink) { sink instanceof UnsafeSink }
}
from HttpToUnsafeConfiguration config, TaintTracking::Source src, TaintTracking::Sink sink
where config.hasFlow(src, sink)
select sink, "This argument to 'unsafe' depends on $@.", src, "a user-provided value"
样例中其实还有很多东西没说明白，其实关键在于弄清楚以下几个东西的写法：
- Source (TaintTracking::Source) > 表示起始点
- Sink (TaintTracking::Sink) > 表示利用点 (Source和Sink都封装了很多常用的类)
- Configuration (TaintTracking::Configuration) > 负责串联Source和Sink
- TaintKind (TaintKind) > 可以使用内置的或自定的 文档写的并不是很深入、详细，可能需要翻看封装好的库借鉴一下
TaintTracking::Sink和TaintTracking::Source中用于确定污点类型的谓词分别如下：
abstract class Source {
abstract predicate isSourceOf(TaintKind kind);
...
}
abstract class Sink {
abstract predicate sinks(TaintKind taint);
...
}
案例测试
测试用例（其一）
Source： input()
Sink： eval()
module: calc.py
import os, sys, re
class calc:
def checkExpr(self, expr):
a = re.split(pattern=r"[+|-|x|/]", string = expr)
print(a)
try:
int(a[0])
int(a[1])
except:
return 0
return 1
def getResult(self, expr):
if(self.checkExpr(expr)):
full_expr = "print(%s)"%(expr)
print(full_expr)
eval(full_expr) #sink2
return 1
else:
print("hacker!")
return 0
module: test1.py
import os, sys, re
import calc as ca
class calc:
def checkExpr(self, expr):
a = re.split(pattern=r"[+|-|x|/]", string = expr)
print(a)
try:
int(a[0])
int(a[1])
except:
return 0
return 1
def getResult(self, expr):
if(self.checkExpr(expr)):
full_expr = "print(%s)"%(expr)
print(full_expr)
eval(full_expr) #sink2
return 1
else:
print("hacker!")
return 0
def checkExpr(expr):
a = re.split(pattern=r"[+|-|x|/]", string = expr)
print(a)
try:
int(a[0])
int(a[1])
except:
return 0
return 1
def getResult(expr):
if(checkExpr(expr)):
full_expr = "print(%s)"%(expr)
print(full_expr)
eval(full_expr) #sink2
return 1
else:
print("hacker!")
return 0
expr = input("expr> ")
test = calc()
test.getResult(expr)
getResult(expr)
module: test2
import os, sys
black_list = ["cmd", "cmd.exe"]
cmd = input("cmd> ")
for item in black_list:
if item in cmd:
print("hacker!")
break
else:
print(os.popen(cmd).read()) #sink1
我的发现
CodeQL似乎对python的跨模块DataFlow支持不太好（也有一定可能是我写的查询不够完善）。
估计这是个普遍性问题，NAVEX的作者设计思路也是在一开始单独分析每个模块中的sink，只不过她在不同的研究中采取了不同的方式完成从Source到Sink的串联。
由此可以确定，跨模块是一大难点，针对多模块python应用如何解决模块间的溯源是一个可以进行创新的角度。
控制变量分析：
- 当Sink在不同模块的类内部的成员方法时，无法完整溯源
- 当Sink在同一个模块的类内部的成员方法时，可以溯源到Source
- 当Sink在同一个模块单独作为函数声明时，可以溯源到Source
- 当Sink在不同模块单独作为函数声明时，无法完整溯源
查询脚本
//import semmle.python.security.strings.Untrusted
import python
/* Sources */
//import semmle.python.web.HttpRequest
class EvalSink extends TaintSink {
EvalSink() {
exists(FunctionValue eval |
eval.getName() = "eval" and
eval.getACall().(CallNode).getAnArg() = this
)
}
override predicate sinks(TaintKind kind) {
kind instanceof StringKind
}
}
class InputSrc extends TaintSource{
InputSrc() {
exists(FunctionValue input|
input.getName() = "input" and
this = input.getACall()
)
}
override predicate isSourceOf(TaintKind kind) {
kind instanceof StringKind
}
}
class InputToEvalConfiguration extends TaintTracking::Configuration {
InputToEvalConfiguration() {
this = "Example config finding flow from http request to 'unsafe' function"
}
override predicate isSource(TaintSource src) { src instanceof InputSrc }
override predicate isSink(TaintSink sink) { sink instanceof EvalSink }
}
from InputToEvalConfiguration config, TaintSource src, TaintSink sink
where config.hasFlow(src, sink)
select sink, "This argument to 'eval' depends on $@.", src, "a user-provided value"