Vulnerability Scanning on Third Party Deps (#36506)
* initial flatten deps scan
* move 3rd party scan to separate action
* allow fork to run
* install requests
* use packages
* pip install
* rename
* conditional vuln report
* trailing whitespace
* trailing whitespace
* detailed print
* add testing file
* add upload test sarif
* results sarif
* move sarif
* upload modified sarif
* test flow
* test with results.sarif
* formatting
* test naming convention
* description with text in artifactLocation
* don't use locations
* use template sarif
* just use template
* add one field mod
* add another field mod
* use actual osvReport
* add field
* add field
* test
* no information uri
* no information uri
* add name
* template NA data for results
* back to minimal template
* dynamic rules
* template update
* no results
* only use template
* test
* new test
* new test
* add back locations
* descriptive fields
* test
* use package name
* variable commit hash
* add chromium accessibility readme support
* use batch query test
* clean up
* use variables for sarif template
* initial automating ancestor commit
* allow for workflow on testing
* install gitpython in workflow
* wrap in try
* expand try
* check commit is not none
* quiet clone
* fix commit newline
* proper print for failed deps
* remove gitpython
* remove import
* fix origin source
* remove .dart from dep names
* update dep
* typo
* update
* clone into controlled name repo now
* fix github upstream clone url
* test CVE finding
* use templated rule and result
* typo
* remove test CVE
* add link straight to OSV DB
* comments
* use os mkdir
* check time of pinned commit
* quiet git
* print osv api query results if vulns found
* move upstream mapping into DEPS file
* add testing for DEPS file
* add khronos exception
* add basic ancestor commit test
* no vulns message
* do not produce empty sarif
* add yaml
* remove unused python dep
* no change?
* no more print, causing recipe issues
* string test
* string test
* no more fstrings
* convert to .format
* syntax
* remove unused dep
* test
* switch test script
* no encoding
* add back test
* typo
* remove scan flat deps tests again
* update
* fix tests
* typo
* newline
* use checkout dir
* prefix
* update to use prefix
* lint
* runhook attempt
* lint
* lint
* lint
* lint
* no license blurb
* cleanup
* enable for main
* do not raise error
* run on branch
* data indentation
* check file existence
* workflow updates
* add push for testing
* syntax
* workflow test
* test github action
* syntax
* allow empty report
* update cron
* pin hash
* newline
* sort by key with prefix omitted
* alphabetize, copyright header
* pylint tests
* lint
* lint
* trailing whitespace?
* lint
* update
* get error types
* allow test
* use output
* only main branch
* licenses check
* results.sarif
* revert
* license updates
* add upstream
* replace Requests library with urllib, remove pylint wrapper
* lint
* undo license
* clone test nit
* isinstance
* DEPS formatting
Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
* use subprocess.check_output
* lint
* lint
* review syntax from comments
* remove line
* more description in error
* lint
* fix checkout path
* remove duplicate eval
* lint
* lint
* lint
* clone-test mkdir and cleanup
* use shutil.rmtree for non-empty dir
* lint
* linting
* linting
* var name
* Update ci/deps_parser_tests.py
Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
* Update ci/deps_parser_tests.py
Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
* more description
* lint
* refactor deps file parsing
* early return
* lint
Co-authored-by: Zachary Anderson <zanderso@users.noreply.github.com>
diff --git a/.github/workflows/third_party_scan.yml b/.github/workflows/third_party_scan.yml
new file mode 100644
index 0000000..3e4a6b2
--- /dev/null
+++ b/.github/workflows/third_party_scan.yml
@@ -0,0 +1,56 @@
+name: Third party dependency scan
+on:
+ # Only the default branch is supported.
+ branch_protection_rule:
+ branches: [ main ]
+ schedule:
+ - cron: "0 8 * * *" # runs daily at 08:00
+
+
+# Declare default permissions as read only.
+permissions: read-all
+
+jobs:
+ analysis:
+ name: Third party dependency scan
+ runs-on: ubuntu-latest
+ permissions:
+ # Needed to upload the results to code-scanning dashboard.
+ security-events: write
+ actions: read
+ contents: read
+
+ steps:
+ - name: "Checkout code"
+ uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b
+ with:
+ persist-credentials: false
+
+ - name: setup python
+ uses: actions/setup-python@98f2ad02fd48d057ee3b4d4f66525b231c3e52b6
+ with:
+ python-version: '3.7.7' # install the python version needed
+
+ - name: install dependency
+ run: pip install git+https://github.com/psf/requests.git@4d394574f5555a8ddcc38f707e0c9f57f55d9a3b
+
+ - name: execute py script
+ run: python ci/deps_parser.py
+
+ - name: parse deps_parser output.txt
+ run: python ci/scan_flattened_deps.py
+
+ # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
+ # format to the repository Actions tab.
+ - name: "Upload artifact"
+ uses: actions/upload-artifact@6673cd052c4cd6fcf4b4e6e60ea986c889389535
+ with:
+ name: SARIF file
+ path: osvReport.sarif
+ retention-days: 5
+
+ # Upload the results to GitHub's code scanning dashboard.
+ - name: "Upload to code-scanning"
+ uses: github/codeql-action/upload-sarif@a3a6c128d771b6b9bdebb1c9d0583ebd2728a108
+ with:
+ sarif_file: osvReport.sarif
diff --git a/DEPS b/DEPS
index 14b8300..0664cda 100644
--- a/DEPS
+++ b/DEPS
@@ -97,6 +97,122 @@
# Setup Git hooks by default.
"setup_githooks": True,
+
+ # Upstream URLs for third party dependencies, used in
+ # determining common ancestor commit for vulnerability scanning
+ # prefixed with 'upstream_' in order to be identified by parsing tool.
+ # The vulnerabiity database being used in this scan can be browsed
+ # using this UI https://osv.dev/list
+ # If a new dependency needs to be added, the upstream (non-mirrored)
+ # git URL for that dependency should be added to this list
+ # with the key-value pair being:
+ # 'upstream_[dep name from last slash and before .git in URL]':'[git URL]'
+ # example:
+ "upstream_abseil-cpp": "https://github.com/abseil/abseil-cpp.git",
+ "upstream_angle": "https://github.com/google/angle.git",
+ "upstream_archive": "https://github.com/brendan-duncan/archive.git",
+ "upstream_args": "https://github.com/dart-lang/args.git",
+ "upstream_async": "https://github.com/dart-lang/async.git",
+ "upstream_bazel_worker": "https://github.com/dart-lang/bazel_worker.git",
+ "upstream_benchmark": "https://github.com/google/benchmark.git",
+ "upstream_boolean_selector": "https://github.com/dart-lang/boolean_selector.git",
+ "upstream_boringssl_gen": "https://github.com/dart-lang/boringssl_gen.git",
+ "upstream_boringssl": "https://github.com/openssl/openssl.git",
+ "upstream_browser_launcher": "https://github.com/dart-lang/browser_launcher.git",
+ "upstream_buildroot": "https://github.com/flutter/buildroot.git",
+ "upstream_cli_util": "https://github.com/dart-lang/cli_util.git",
+ "upstream_clock": "https://github.com/dart-lang/clock.git",
+ "upstream_collection": "https://github.com/dart-lang/collection.git",
+ "upstream_colorama": "https://github.com/tartley/colorama.git",
+ "upstream_convert": "https://github.com/dart-lang/convert.git",
+ "upstream_crypto": "https://github.com/dart-lang/crypto.git",
+ "upstream_csslib": "https://github.com/dart-lang/csslib.git",
+ "upstream_dart_style": "https://github.com/dart-lang/dart_style.git",
+ "upstream_dartdoc": "https://github.com/dart-lang/dartdoc.git",
+ "upstream_equatable": "https://github.com/felangel/equatable.git",
+ "upstream_ffi": "https://github.com/dart-lang/ffi.git",
+ "upstream_file": "https://github.com/google/file.dart.git",
+ "upstream_fixnum": "https://github.com/dart-lang/fixnum.git",
+ "upstream_flatbuffers": "https://github.com/google/flatbuffers.git",
+ "upstream_fontconfig": "https://gitlab.freedesktop.org/fontconfig/fontconfig.git",
+ "upstream_freetype2": "https://gitlab.freedesktop.org/freetype/freetype.git",
+ "upstream_gcloud": "https://github.com/dart-lang/gcloud.git",
+ "upstream_glfw": "https://github.com/glfw/glfw.git",
+ "upstream_glob": "https://github.com/dart-lang/glob.git",
+ "upstream_googleapis": "https://github.com/google/googleapis.dart.git",
+ "upstream_googletest": "https://github.com/google/googletest.git",
+ "upstream_gtest-parallel": "https://github.com/google/gtest-parallel.git",
+ "upstream_harfbuzz": "https://github.com/harfbuzz/harfbuzz.git",
+ "upstream_html": "https://github.com/dart-lang/html.git",
+ "upstream_http_multi_server": "https://github.com/dart-lang/http_multi_server.git",
+ "upstream_http_parser": "https://github.com/dart-lang/http_parser.git",
+ "upstream_http": "https://github.com/dart-lang/http.git",
+ "upstream_icu": "https://github.com/unicode-org/icu.git",
+ "upstream_imgui": "https://github.com/ocornut/imgui.git",
+ "upstream_inja": "https://github.com/pantor/inja.git",
+ "upstream_json": "https://github.com/nlohmann/json.git",
+ "upstream_json_rpc_2": "https://github.com/dart-lang/json_rpc_2.git",
+ "upstream_libcxx": "https://github.com/llvm-mirror/libcxx.git",
+ "upstream_libcxxabi": "https://github.com/llvm-mirror/libcxxabi.git",
+ "upstream_libexpat": "https://github.com/libexpat/libexpat.git",
+ "upstream_libjpeg-turbo": "https://github.com/libjpeg-turbo/libjpeg-turbo.git",
+ "upstream_libpng": "https://github.com/glennrp/libpng.git",
+ "upstream_libtess2": "https://github.com/memononen/libtess2.git",
+ "upstream_libwebp": "https://chromium.googlesource.com/webm/libwebp.git",
+ "upstream_libxml": "https://gitlab.gnome.org/GNOME/libxml2.git",
+ "upstream_linter": "https://github.com/dart-lang/linter.git",
+ "upstream_logging": "https://github.com/dart-lang/logging.git",
+ "upstream_markdown": "https://github.com/dart-lang/markdown.git",
+ "upstream_matcher": "https://github.com/dart-lang/matcher.git",
+ "upstream_mime": "https://github.com/dart-lang/mime.git",
+ "upstream_mockito": "https://github.com/dart-lang/mockito.git",
+ "upstream_oauth2": "https://github.com/dart-lang/oauth2.git",
+ "upstream_ocmock": "https://github.com/erikdoe/ocmock.git",
+ "upstream_package_config": "https://github.com/dart-lang/package_config.git",
+ "upstream_packages": "https://github.com/flutter/packages.git",
+ "upstream_path": "https://github.com/dart-lang/path.git",
+ "upstream_platform": "https://github.com/google/platform.dart.git",
+ "upstream_pool": "https://github.com/dart-lang/pool.git",
+ "upstream_process_runner": "https://github.com/google/process_runner.git",
+ "upstream_process": "https://github.com/google/process.dart.git",
+ "upstream_protobuf": "https://github.com/google/protobuf.dart.git",
+ "upstream_pub_semver": "https://github.com/dart-lang/pub_semver.git",
+ "upstream_pub": "https://github.com/dart-lang/pub.git",
+ "upstream_pyyaml": "https://github.com/yaml/pyyaml.git",
+ "upstream_quiver-dart": "https://github.com/google/quiver-dart.git",
+ "upstream_rapidjson": "https://github.com/Tencent/rapidjson.git",
+ "upstream_root_certificates": "https://github.com/dart-lang/root_certificates.git",
+ "upstream_sdk": "https://github.com/dart-lang/sdk.git",
+ "upstream_shaderc": "https://github.com/google/shaderc.git",
+ "upstream_shelf": "https://github.com/dart-lang/shelf.git",
+ "upstream_skia": "https://skia.googlesource.com/skia.git",
+ "upstream_source_map_stack_trace": "https://github.com/dart-lang/source_map_stack_trace.git",
+ "upstream_source_maps": "https://github.com/dart-lang/source_maps.git",
+ "upstream_source_span": "https://github.com/dart-lang/source_span.git",
+ "upstream_sqlite": "https://github.com/sqlite/sqlite.git",
+ "upstream_sse": "https://github.com/dart-lang/sse.git",
+ "upstream_stack_trace": "https://github.com/dart-lang/stack_trace.git",
+ "upstream_stream_channel": "https://github.com/dart-lang/stream_channel.git",
+ "upstream_string_scanner": "https://github.com/dart-lang/string_scanner.git",
+ "upstream_SwiftShader": "https://swiftshader.googlesource.com/SwiftShader.git",
+ "upstream_term_glyph": "https://github.com/dart-lang/term_glyph.git",
+ "upstream_test_reflective_loader": "https://github.com/dart-lang/test_reflective_loader.git",
+ "upstream_test": "https://github.com/dart-lang/test.git",
+ "upstream_tinygltf": "https://github.com/syoyo/tinygltf.git",
+ "upstream_typed_data": "https://github.com/dart-lang/typed_data.git",
+ "upstream_usage": "https://github.com/dart-lang/usage.git",
+ "upstream_vector_math": "https://github.com/google/vector_math.dart.git",
+ "upstream_Vulkan-Headers": "https://github.com/KhronosGroup/Vulkan-Headers.git",
+ "upstream_VulkanMemoryAllocator": "https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git",
+ "upstream_watcher": "https://github.com/dart-lang/watcher.git",
+ "upstream_web_socket_channel": "https://github.com/dart-lang/web_socket_channel.git",
+ "upstream_webdev": "https://github.com/dart-lang/webdev.git",
+ "upstream_webkit_inspection_protocol": "https://github.com/google/webkit_inspection_protocol.dart.git",
+ "upstream_wuffs-mirror-release-c": "https://github.com/google/wuffs-mirror-release-c.git",
+ "upstream_yaml_edit": "https://github.com/dart-lang/yaml_edit.git",
+ "upstream_yaml": "https://github.com/dart-lang/yaml.git",
+ "upstream_yapf": "https://github.com/google/yapf.git",
+ "upstream_zlib": "https://github.com/madler/zlib.git",
}
gclient_gn_args_file = 'src/third_party/dart/build/config/gclient_args.gni'
diff --git a/ci/deps_parser.py b/ci/deps_parser.py
index 4e46c8d..bf4f6d8 100644
--- a/ci/deps_parser.py
+++ b/ci/deps_parser.py
@@ -3,7 +3,7 @@
# Copyright 2013 The Flutter Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
-
+#
# Usage: deps_parser.py --deps <DEPS file> --output <flattened deps>
#
# This script parses the DEPS file, extracts the fully qualified dependencies
@@ -12,11 +12,16 @@
import argparse
import os
+import re
import sys
SCRIPT_DIR = os.path.dirname(sys.argv[0])
CHECKOUT_ROOT = os.path.realpath(os.path.join(SCRIPT_DIR, '..'))
+CHROMIUM_README_FILE = 'third_party/accessibility/README.md'
+CHROMIUM_README_COMMIT_LINE = 4 # the fifth line will always contain the commit hash
+CHROMIUM = 'https://chromium.googlesource.com/chromium/src'
+
# Used in parsing the DEPS file.
class VarImpl:
@@ -55,15 +60,30 @@
# Extract the deps and filter.
deps = local_scope.get('deps', {})
filtered_deps = []
- for val in deps.values():
+ for _, dep in deps.items():
# We currently do not support packages or cipd which are represented
# as dictionaries.
- if isinstance(val, str):
- filtered_deps.append(val)
-
+ if isinstance(dep, str):
+ filtered_deps.append(dep)
return filtered_deps
+def parse_readme(deps):
+ """
+ Opens the Flutter Accessibility Library README and uses the commit hash
+ found in the README to check for viulnerabilities.
+ The commit hash in this README will always be in the same format
+ """
+ file_path = os.path.join(CHECKOUT_ROOT, CHROMIUM_README_FILE)
+ with open(file_path) as file:
+ # read the content of the file opened
+ content = file.readlines()
+ commit_line = content[CHROMIUM_README_COMMIT_LINE]
+ commit = re.search(r'(?<=\[).*(?=\])', commit_line)
+ deps.append(CHROMIUM + '@' + commit.group())
+ return deps
+
+
def write_manifest(deps, manifest_file):
print('\n'.join(sorted(deps)))
with open(manifest_file, 'w') as manifest:
@@ -97,6 +117,7 @@
def main(argv):
args = parse_args(argv)
deps = parse_deps_file(args.deps)
+ deps = parse_readme(deps)
write_manifest(deps, args.output)
return 0
diff --git a/ci/deps_parser_tests.py b/ci/deps_parser_tests.py
new file mode 100644
index 0000000..dd96d16
--- /dev/null
+++ b/ci/deps_parser_tests.py
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+#
+# Copyright 2013 The Flutter Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import os
+import sys
+import unittest
+from deps_parser import VarImpl
+
+SCRIPT_DIR = os.path.dirname(sys.argv[0])
+CHECKOUT_ROOT = os.path.realpath(os.path.join(SCRIPT_DIR, '..'))
+DEPS = os.path.join(CHECKOUT_ROOT, 'DEPS')
+UPSTREAM_PREFIX = 'upstream_'
+
+
+class TestDepsParserMethods(unittest.TestCase):
+ # Extract both mirrored dep names and URLs &
+ # upstream names and URLs from DEPs file.
+ def setUp(self): # lower-camel-case for the python unittest framework
+ # Read the content.
+ with open(DEPS, 'r') as file:
+ deps_content = file.read()
+
+ local_scope_mirror = {}
+ var = VarImpl(local_scope_mirror)
+ global_scope_mirror = {
+ 'Var': var.lookup,
+ 'deps_os': {},
+ }
+
+ # Eval the content.
+ exec(deps_content, global_scope_mirror, local_scope_mirror)
+
+ # Extract the upstream URLs
+ # vars contains more than just upstream URLs
+ # however the upstream URLs are prefixed with 'upstream_'
+ upstream = local_scope_mirror.get('vars')
+ self.upstream_urls = upstream
+
+ # Extract the deps and filter.
+ deps = local_scope_mirror.get('deps', {})
+ filtered_deps = []
+ for _, dep in deps.items():
+ # We currently do not support packages or cipd which are represented
+ # as dictionaries.
+ if isinstance(dep, str):
+ filtered_deps.append(dep)
+ self.deps = filtered_deps
+
+ def test_each_dep_has_upstream_url(self):
+ # For each DEP in the deps file, check for an associated upstream URL in deps file.
+ for dep in self.deps:
+ dep_repo = dep.split('@')[0]
+ dep_name = dep_repo.split('/')[-1].split('.')[0]
+ # vulkan-deps and khronos do not have one upstream URL
+ # all other deps should have an associated upstream URL for vuln scanning purposes
+ if dep_name not in ('vulkan-deps', 'khronos'):
+ # Add the prefix on the dep name when searching for the upstream entry.
+ self.assertTrue(
+ UPSTREAM_PREFIX + dep_name in self.upstream_urls,
+ msg=dep_name + ' not found in upstream URL list. ' +
+ 'Each dep in the "deps" section of DEPS file must have associated upstream URL'
+ )
+
+ def test_each_upstream_url_has_dep(self):
+ # Parse DEPS into dependency names.
+ deps_names = []
+ for dep in self.deps:
+ dep_repo = dep.split('@')[0]
+ dep_name = dep_repo.split('/')[-1].split('.')[0]
+ deps_names.append(dep_name)
+
+ # For each upstream URL dep, check it exists as in DEPS.
+ for upsream_dep in self.upstream_urls:
+ # Only test on upstream deps in vars section which start with the upstream prefix
+ if upsream_dep.startswith(UPSTREAM_PREFIX):
+ # Strip the prefix to check that it has a corresponding dependency in the DEPS file
+ self.assertTrue(
+ upsream_dep[len(UPSTREAM_PREFIX):] in deps_names,
+ msg=upsream_dep + ' from upstream list not found in DEPS. ' +
+ 'Each upstream URL in DEPS file must have an associated dep in the "deps" section'
+ )
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/ci/scan_flattened_deps.py b/ci/scan_flattened_deps.py
new file mode 100644
index 0000000..ece1154
--- /dev/null
+++ b/ci/scan_flattened_deps.py
@@ -0,0 +1,314 @@
+#!/usr/bin/env python3
+#
+# Copyright 2013 The Flutter Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+#
+# Usage: scan_flattened_deps.py --flat-deps <flat DEPS file> --output <vulnerability report>
+#
+# This script parses the flattened, fully qualified dependencies,
+# and uses the OSV API to check for known vulnerabilities
+# for the given hash of the dependency
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+from urllib import request
+
+SCRIPT_DIR = os.path.dirname(sys.argv[0])
+CHECKOUT_ROOT = os.path.realpath(os.path.join(SCRIPT_DIR, '..'))
+DEP_CLONE_DIR = CHECKOUT_ROOT + '/clone-test'
+DEPS = os.path.join(CHECKOUT_ROOT, 'DEPS')
+HELP_STR = 'To find complete information on this vulnerability, navigate to '
+OSV_VULN_DB_URL = 'https://osv.dev/vulnerability/'
+SECONDS_PER_YEAR = 31556952
+UPSTREAM_PREFIX = 'upstream_'
+
+failed_deps = [] # deps which fail to be be cloned or git-merge based
+
+sarif_log = {
+ '$schema':
+ 'https://json.schemastore.org/sarif-2.1.0.json', 'version':
+ '2.1.0', 'runs': [{
+ 'tool': {'driver': {'name': 'OSV Scan', 'rules': []}},
+ 'results': []
+ }]
+}
+
+
+def sarif_result():
+ """
+ Returns the template for a result entry in the Sarif log,
+ which is populated with CVE findings from OSV API
+ """
+ return {
+ 'ruleId':
+ 'N/A', 'message': {'text': 'OSV Scan Finding'}, 'locations': [{
+ 'physicalLocation': {
+ 'artifactLocation': {
+ 'uri': 'No location associated with this finding'
+ },
+ 'region': {'startLine': 1, 'startColumn': 1, 'endColumn': 1}
+ }
+ }]
+ }
+
+
+def sarif_rule():
+ """
+ Returns the template for a rule entry in the Sarif log,
+ which is populated with CVE findings from OSV API
+ """
+ return {
+ 'id': 'OSV Scan', 'name': 'OSV Scan Finding',
+ 'shortDescription': {'text': 'Insert OSV id'}, 'fullDescription': {
+ 'text': 'Vulnerability found by scanning against the OSV API'
+ }, 'help': {
+ 'text':
+ 'More details in the OSV DB at: https://osv.dev/vulnerability/'
+ }, 'defaultConfiguration': {'level': 'error'},
+ 'properties': {'tags': ['supply-chain', 'dependency']}
+ }
+
+
+def parse_deps_file(deps_flat_file):
+ """
+ Takes input of fully qualified dependencies,
+ for each dep find the common ancestor commit SHA
+ from the upstream and query OSV API using that SHA
+
+ If the commit cannot be found or the dep cannot be
+ compared to an upstream, prints list of those deps
+ """
+ deps_list = []
+ with open(DEPS, 'r') as file:
+ local_scope = {}
+ global_scope = {'Var': lambda x: x} # dummy lambda
+ # Read the content.
+ deps_content = file.read()
+
+ # Eval the content.
+ exec(deps_content, global_scope, local_scope)
+
+ # Extract the deps and filter.
+ deps_list = local_scope.get('vars')
+ queries = [] # list of queries to submit in bulk request to OSV API
+ with open(deps_flat_file, 'r') as file:
+ lines = file.readlines()
+
+ headers = {
+ 'Content-Type': 'application/json',
+ }
+ osv_url = 'https://api.osv.dev/v1/querybatch'
+
+ if not os.path.exists(DEP_CLONE_DIR):
+ os.mkdir(DEP_CLONE_DIR) #clone deps with upstream into temporary dir
+
+ # Extract commit hash, save in dictionary
+ for line in lines:
+ dep = line.strip().split(
+ '@'
+ ) # separate fully qualified dep into name + pinned hash
+
+ common_commit = get_common_ancestor_commit(dep, deps_list)
+ if isinstance(common_commit, str):
+ queries.append({'commit': common_commit})
+ else:
+ failed_deps.append(dep[0])
+
+ print(
+ 'Dependencies that could not be parsed for ancestor commits: ' +
+ ', '.join(failed_deps)
+ )
+ try:
+ # clean up cloned upstream dependency directory
+ shutil.rmtree(
+ DEP_CLONE_DIR
+ ) # use shutil.rmtree since dir could be non-empty
+ except OSError as clone_dir_error:
+ print(
+ 'Error cleaning up clone directory: %s : %s' %
+ (DEP_CLONE_DIR, clone_dir_error.strerror)
+ )
+ # Query OSV API using common ancestor commit for each dep
+ # return any vulnerabilities found.
+ data = json.dumps({'queries': queries}).encode('utf-8')
+ req = request.Request(osv_url, data, headers=headers)
+ with request.urlopen(req) as resp:
+ res_body = resp.read()
+ results_json = json.loads(res_body.decode('utf-8'))
+ if resp.status != 200:
+ print('Request error')
+ elif results_json['results'] == [{}]:
+ print('Found no vulnerabilities')
+ else:
+ results = results_json['results']
+ filtered_results = list(filter(lambda vuln: vuln != {}, results))
+ if len(filtered_results) > 0:
+ print(
+ 'Found vulnerability on {vuln_count} dependenc(y/ies), adding to report'
+ .format(vuln_count=str(len(filtered_results)))
+ )
+ print(*filtered_results)
+ return filtered_results
+ print('Found no vulnerabilities')
+ return {}
+
+
+def get_common_ancestor_commit(dep, deps_list):
+ """
+ Given an input of a mirrored dep,
+ compare to the mapping of deps to their upstream
+ in DEPS and find a common ancestor
+ commit SHA value.
+
+ This is done by first cloning the mirrored dep,
+ then a branch which tracks the upstream.
+ From there, git merge-base operates using the HEAD
+ commit SHA of the upstream branch and the pinned
+ SHA value of the mirrored branch
+ """
+ # dep[0] contains the mirror repo
+ # dep[1] contains the mirror's pinned SHA
+ # upstream is the origin repo
+ dep_name = dep[0].split('/')[-1].split('.')[0]
+ if UPSTREAM_PREFIX + dep_name not in deps_list:
+ print('did not find dep: ' + dep_name)
+ return {}
+ try:
+ # get the upstream URL from the mapping in DEPS file
+ upstream = deps_list.get(UPSTREAM_PREFIX + dep_name)
+ temp_dep_dir = DEP_CLONE_DIR + '/' + dep_name
+ # clone dependency from mirror
+ subprocess.check_output([
+ 'git', 'clone', '--quiet', '--', dep[0], temp_dep_dir
+ ])
+
+ # create branch that will track the upstream dep
+ print(
+ 'attempting to add upstream remote from: {upstream}'.format(
+ upstream=upstream
+ )
+ )
+ subprocess.check_output([
+ 'git', '--git-dir', temp_dep_dir + '/.git', 'remote', 'add', 'upstream',
+ upstream
+ ])
+ subprocess.check_output([
+ 'git', '--git-dir', temp_dep_dir + '/.git', 'fetch', '--quiet',
+ 'upstream'
+ ])
+ # get name of the default branch for upstream (e.g. main/master/etc.)
+ default_branch = subprocess.check_output(
+ 'git --git-dir ' + temp_dep_dir + '/.git remote show upstream ' +
+ "| sed -n \'/HEAD branch/s/.*: //p\'",
+ shell=True
+ ).decode().strip()
+ print(
+ 'default_branch found: {default_branch}'.format(
+ default_branch=default_branch
+ )
+ )
+ # make upstream branch track the upstream dep
+ subprocess.check_output([
+ 'git', '--git-dir', temp_dep_dir + '/.git', 'checkout', '-b',
+ 'upstream', '--track', 'upstream/' + default_branch
+ ])
+ # get the most recent commit from default branch of upstream
+ commit = subprocess.check_output(
+ 'git --git-dir ' + temp_dep_dir + '/.git for-each-ref ' +
+ "--format=\'%(objectname:short)\' refs/heads/upstream",
+ shell=True
+ )
+ commit = commit.decode().strip()
+
+ # perform merge-base on most recent default branch commit and pinned mirror commit
+ ancestor_commit = subprocess.check_output(
+ 'git --git-dir {temp_dep_dir}/.git merge-base {commit} {depUrl}'.format(
+ temp_dep_dir=temp_dep_dir, commit=commit, depUrl=dep[1]
+ ),
+ shell=True
+ )
+ ancestor_commit = ancestor_commit.decode().strip()
+ print('Ancestor commit: ' + ancestor_commit)
+ return ancestor_commit
+ except subprocess.CalledProcessError as error:
+ print("Subprocess error '{0}' occured.".format(error.output))
+ return {}
+
+
+def write_sarif(responses, manifest_file):
+ """
+ Creates a full SARIF report based on the OSV API response which
+ may contain several vulnerabilities
+
+ Combines a rule with a result in order to construct the report
+ """
+ data = sarif_log
+ for response in responses:
+ for vuln in response['vulns']:
+ new_rule = create_rule_entry(vuln)
+ data['runs'][0]['tool']['driver']['rules'].append(new_rule)
+ data['runs'][0]['results'].append(create_result_entry(vuln))
+ with open(manifest_file, 'w') as out:
+ json.dump(data, out)
+
+
+def create_rule_entry(vuln):
+ """
+ Creates a Sarif rule entry from an OSV finding.
+ Vuln object follows OSV Schema and is required to have 'id' and 'modified'
+ """
+ rule = sarif_rule()
+ rule['id'] = vuln['id']
+ rule['shortDescription']['text'] = vuln['id']
+ rule['help']['text'] += vuln['id']
+ return rule
+
+
+def create_result_entry(vuln):
+ """
+ Creates a Sarif res entry from an OSV entry.
+ Rule finding linked to the associated rule metadata via ruleId
+ """
+ result = sarif_result()
+ result['ruleId'] = vuln['id']
+ return result
+
+
+def parse_args(args):
+ args = args[1:]
+ parser = argparse.ArgumentParser(
+ description='A script to scan a flattened DEPS file using OSV API.'
+ )
+
+ parser.add_argument(
+ '--flat-deps',
+ '-d',
+ type=str,
+ help='Input flattened DEPS file.',
+ default=os.path.join(CHECKOUT_ROOT, 'deps_flatten.txt')
+ )
+ parser.add_argument(
+ '--output',
+ '-o',
+ type=str,
+ help='Output SARIF log of vulnerabilities found in OSV database.',
+ default=os.path.join(CHECKOUT_ROOT, 'osvReport.sarif')
+ )
+
+ return parser.parse_args(args)
+
+
+def main(argv):
+ args = parse_args(argv)
+ osv_scans = parse_deps_file(args.flat_deps)
+ write_sarif(osv_scans, args.output)
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main(sys.argv))
