Skip to content

Commit 0d5eff2

Browse files
pylapppylapp
authored
Version 2.3.0 (#36)
* [#19] [Feature] GitHub repositories backup (#26) * chore: #19 - prepare for feature Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * chore: project - clean up - Fixed typos Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * feat: #19 - dump of repositories of organisation Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * [#27] [Feature] Diver - Extract email addresses (#31) * chore: project - improve README and THIRD-PARTY Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * chore: project - rename bad command parameter Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * fix: Shell scripts - temp log file not found Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * feat: #27 - extract email adresses from Git history Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * doc: README - improve section for backup-all-repositories-from-org Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * Small improvements for doc and scripts (#33) * doc: proejct - improve README with keywords Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * feat: #27 - keep unique email address Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com> * feat: #20 - check for vulnerability alerts in organisation projects (#35) Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com>
1 parent f0feb36 commit 0d5eff2

File tree

5 files changed

+285
-11
lines changed

5 files changed

+285
-11
lines changed

‎CHANGELOG.md‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,27 @@
11
# CHANGELOG
22

3-
## Version 2.2.0
3+
## Version 2.3.0
44

5-
### Features (github)
5+
### Features
66

7-
- [#19](https://github.com/Orange-OpenSource/floss-toolbox/issues/19) Backup of repositories
7+
- [#20](https://github.com/Orange-OpenSource/floss-toolbox/issues/20) Find repositories with vulnerabilities (Dependabot)
88

9-
### Features (diver)
9+
## Version 2.2.0
1010

11+
### Features
12+
13+
- [#19](https://github.com/Orange-OpenSource/floss-toolbox/issues/19) Backup of repositories
1114
- [#27](https://github.com/Orange-OpenSource/floss-toolbox/issues/27) Extract email addresses
1215

1316
## Version 2.1.0
1417

15-
### Features (diver):
18+
### Features
1619

1720
- [#13](https://github.com/Orange-OpenSource/floss-toolbox/issues/13) List all contributors of a Git repository using Git history
1821

1922
## Version 2.0.0
2023

21-
### Features (github):
24+
### Features
2225

2326
- [#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4) Get all members of GitHub organization
2427
- [#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4) Get members who don't have 2FA enabled
@@ -34,7 +37,7 @@
3437

3538
## Version 1.0.0
3639

37-
### Features (diver):
40+
### Features
3841

3942
- Find contributors in files using a base of words and producing logs
4043
- Find contributors in git logs

‎README.md‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# floss-toolbox (version 2.2.0)
1+
# floss-toolbox (version 2.3.0)
22

33
Toolbox to help developers and open source referents to have cleaner projects in GitHub organizations.
44

@@ -373,4 +373,25 @@ _So imagine a python eating ruby gems in a shell. Gorgeous isn't it?_
373373
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME**.
374374
You have to also define the location to store clones at **REPOSITORIES_CLONE_LOCATION_PATH**
375375

376+
**You should also have your _git_ environment ready, i.e. add your SSH private key if you clone by SSH for example.**
377+
378+
### Check if there are vulnerabilities alerts in organisation repositories
379+
380+
_Keywords: #organisation #GitHub #repositories #Dependabot #vulnerabilities_
381+
382+
This feature allows to check in all repositories of the GitHub organisation if there are projects witch vulnerabilities alerts.
383+
384+
Run the following command:
385+
```shell
386+
bash GitHubWizard.sh vulnerabilities-alerts-for-all-repositories
387+
```
388+
389+
This script will trigger the _gh_ client which may ask you to athenticate to the GitHub API.
390+
Then the Shell script will pick configuration details from the Ruby configuration file; and triggers another Shell script for the data process. A Python code will be called too.
391+
392+
The Python code will process JSON data, the Sheel script will make a CURL request to to GraphQL API.
393+
394+
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME**.
395+
You have to also define the location to store clones at **REPOSITORIES_CLONE_LOCATION_PATH**
396+
376397
**You should also have your _git_ environment ready, i.e. add your SSH private key if you clone by SSH for example.**

‎toolbox/github/utils/GitHubFacade.rb‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
#!/usr/bin/env ruby
22
# Software Name: floss-toolbox
3-
# SPDX-FileCopyrightText: Copyright (c) 2021 Orange
3+
# SPDX-FileCopyrightText: Copyright (c) 2021-2022 Orange
44
# SPDX-License-Identifier: Apache-2.0
55
#
66
# This software is distributed under the Apache 2.0 license.
77
#
88
# Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
99

10-
# Version.............: 1.0.0
10+
# Version.............: 1.1.0
1111
# Since...............: 26/04/2021
1212
# Description.........: Received from arguments a feature to launch using GitHub API.
1313
# Loads configuration elements from configuration.rb, writes outputs using FileManager.rb and calls GitHubWrapper.rb for requests
@@ -51,7 +51,7 @@
5151
end
5252
feature_to_run=ARGV[0]
5353

54-
if feature_to_run != "get-members-2fa-disabled" && feature_to_run != "get-all-members" && feature_to_run != "get-members-without-company" && feature_to_run != "get-projects-without-team" && feature_to_run != "get-users-with-bad-email" && feature_to_run != "get-users-with-bad-fullname" && feature_to_run != "get-projects-conformity" && feature_to_run != "get-projects-without-licenses" && feature_to_run != "get-empty-projects" && feature_to_run != "set-users-permissions-to-push" && feature_to_run != "set-teams-permissions-to-push"
54+
if feature_to_run != "get-members-2fa-disabled" && feature_to_run != "get-all-members" && feature_to_run != "get-members-without-company" && feature_to_run != "get-projects-without-team" && feature_to_run != "get-users-with-bad-email" && feature_to_run != "get-users-with-bad-fullname" && feature_to_run != "get-projects-conformity" && feature_to_run != "get-projects-without-licenses" && feature_to_run != "get-empty-projects" && feature_to_run != "set-users-permissions-to-push" && feature_to_run != "set-teams-permissions-to-push"
5555
Log.error "Unknown feature. Exit now."
5656
exit $EXIT_UNKNOWN_FEATURE
5757
end

‎toolbox/github/utils/check-vulnerabilities-from-github.sh‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
#!/bin/bash
2+
# Software Name: floss-toolbox
3+
# SPDX-FileCopyrightText: Copyright (c) 2021-2022 Orange
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# This software is distributed under the Apache 2.0 license.
7+
#
8+
# Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
9+
10+
# Since...............: 25/02/2022
11+
# Description.........: Received from arguments a GitHub organisation name and a GitHub personal token path to dump check if vulnerabilities alert exists
12+
13+
#set -euxo pipefail
14+
VERSION="1.0.0"
15+
16+
# Config
17+
# ------
18+
19+
EXIT_OK=0
20+
EXIT_BAD_ARGUMENTS=1
21+
EXIT_BAD_SETUP=2
22+
23+
URL_EXTRACTER_FILE="./utils/extract-repos-field-from-json.py"
24+
VULNERABILITY_PARSER="./utils/count-vulnerabilities-nodes.py"
25+
GITHB_ORGANISATION_VULNERABILITIES_RESULTS="$$_github-organisation-repositories-vulnerabilities-count.csv"
26+
27+
# Functions
28+
# ---------
29+
30+
UsageAndExit(){
31+
echo "check-vulnerabilities-from-github.sh - Version $VERSION"
32+
echo "USAGE:"
33+
echo "bash check-vulnerabilities-from-github.sh ORGANISATION KEY TOKEN"
34+
echo "with ORGANISATION: GitHub organisation name"
35+
echo "with KEY: JSON key to sue use cloning URL"
36+
echo "with TOKEN: personal access token fir requests"
37+
echo "About exit codes:"
38+
echo -e "\t 0................: Normal exit"
39+
echo -e "\t 1................: Bad arguments given to the script"
40+
echo -e "\t 2................: File URL_EXTRACTER_FILE is not defined. Impossible to extract URL from API results."
41+
exit $EXIT_OK
42+
}
43+
44+
# Check setup
45+
# -----------
46+
47+
if [ "$#" -eq 0 ]; then
48+
UsageAndExit
49+
exit $EXIT_OK
50+
fi
51+
52+
if [ "$#" -ne 3 ]; then
53+
echo "ERROR: Bad arguments number. Exits now"
54+
UsageAndExit
55+
exit $EXIT_BAD_ARGUMENTS
56+
fi
57+
58+
if [ ! -f "$URL_EXTRACTER_FILE" ]; then
59+
echo "ERROR: Bad set up for URL extracter. Exits now"
60+
UsageAndExit
61+
exit $EXIT_BAD_SETUP
62+
fi
63+
64+
if [ -f "VULNERABILITY_PARSER" ]; then
65+
echo "ERROR: Bad set up vulnerability parser. Exits now"
66+
UsageAndExit
67+
exit $EXIT_BAD_SETUP
68+
fi
69+
70+
organisation_name=$1
71+
if [ -z "$organisation_name" ]; then
72+
echo "ERROR: No organisation name defined. Exits now."
73+
UsageAndExit
74+
exit $EXIT_BAD_ARGUMENTS
75+
fi
76+
77+
cloning_url_key=$2
78+
if [ -z "$cloning_url_key" ]; then
79+
echo "ERROR: No JSON key for URL. Exits now."
80+
UsageAndExit
81+
exit $EXIT_BAD_ARGUMENTS
82+
fi
83+
84+
personal_access_token=$3
85+
if [ -z "$personal_access_token" ]; then
86+
echo "ERROR: No personal access token defined. Exits now."
87+
UsageAndExit
88+
exit $EXIT_BAD_ARGUMENTS
89+
fi
90+
91+
# Run
92+
# ---
93+
94+
echo "-------------------------------------------------------"
95+
echo "check-vulnerabilities-from-github.sh - Version $VERSION"
96+
echo "-------------------------------------------------------"
97+
98+
# Step 1 - Authenticate to GitHub using gh
99+
100+
echo "Authenticating to GitHub using gh client..."
101+
gh auth login
102+
echo "Authentication done."
103+
104+
# Step 2 - Get repositories URL
105+
106+
echo "Get repositories URL..."
107+
repositories_list_raw_temp_file=".repos-raw.json"
108+
repositories_list_clean_temp_file=".repos-clean.json"
109+
gh api -X GET "orgs/$organisation_name/repos" -F per_page=500 --paginate > $repositories_list_raw_temp_file
110+
cat $repositories_list_raw_temp_file | sed -e "s/\]\[/,/g" > $repositories_list_clean_temp_file # Need to replace '][' by ',' because of pagination
111+
rm -f $repositories_list_raw_temp_file
112+
echo "All repositories URL got."
113+
114+
# Step 3 - Extract cloning URL
115+
116+
url_for_cloning=".url-for-cloning.txt"
117+
echo "Extract cloning from results (using '$cloning_url_key' as JSON key)..."
118+
python3 "$URL_EXTRACTER_FILE" --field $cloning_url_key --source $repositories_list_clean_temp_file > $url_for_cloning
119+
rm -f $repositories_list_clean_temp_file
120+
echo "Extraction done."
121+
122+
# Step 4 - Create workspace directory
123+
124+
dir_before_dump=`pwd`
125+
echo "Creating workspace directory..."
126+
directory_name=$(date '+%Y-%m-%d')
127+
cd "$repositories_location"
128+
if [ -d "$directory_name" ]; then
129+
echo "Removing old directory with the same name"
130+
yes | rm -rf $directory_name
131+
fi
132+
mkdir $directory_name
133+
cd $directory_name
134+
echo "Dump directory created with name '$directory_name' at location `pwd`."
135+
136+
# Step 5 - For each repository, check it
137+
138+
number_of_url=`cat "$dir_before_dump/$url_for_cloning" | wc | awk {'print $1 '}`
139+
cpt=1
140+
echo "Dumping of $number_of_url repositories..."
141+
github_request_result_file_raw=".vulnerabilities-github.txt"
142+
github_request_result_file_json=".vulnerabilities-github.json"
143+
144+
echo "status;project name;count of alerts" > $GITHB_ORGANISATION_VULNERABILITIES_RESULTS
145+
cpt_clean_repo=0
146+
cpt_dirty_repo=0
147+
while read url_line; do
148+
149+
# Step 5.1 - Get repo name
150+
# We assume URL are like "git@github.com:org/repo.js.git"
151+
repo_name=`echo "$url_line" | cut -d"/" -f 2 | awk -F".git" '{print $1}'`
152+
echo "Checking ($cpt / $number_of_url) '$repo_name'..."
153+
cpt=$((cpt+1))
154+
155+
# Step 5.2 - Request GitHub API
156+
graphqlScript='query { repository(name: \"'"$repo_name"'\", owner: \"'"$organisation_name"'\") {
157+
vulnerabilityAlerts(first: 100) {
158+
nodes {
159+
createdAt
160+
dismissedAt
161+
securityVulnerability {
162+
package {
163+
name
164+
}
165+
advisory {
166+
description
167+
}
168+
}
169+
}
170+
}
171+
}
172+
}'
173+
script="$(echo $graphqlScript)"
174+
175+
curl -s -i -H 'Content-Type: application/json' \
176+
-H "Authorization: bearer $personal_access_token" \
177+
-X POST -d "{ \"query\": \"$script\"}" https://api.github.com/graphql -o "$github_request_result_file_raw"
178+
179+
# Step 5.3 - Process result file
180+
181+
# We suppose in response paylaod we have the usefull line like
182+
# {"data":{"repository":{"vulnerabilityAlerts":{"nodes":[]}}}}
183+
cat $github_request_result_file_raw | grep vulnerabilityAlerts > "$github_request_result_file_json"
184+
count=`python3 "../$VULNERABILITY_PARSER" --file "$github_request_result_file_json"`
185+
186+
# Step 5.4 - Build final file
187+
if [ "$count" -eq "0" ]; then
188+
echo "😊;$repo_name;$count" >> $GITHB_ORGANISATION_VULNERABILITIES_RESULTS
189+
cpt_clean_repo=$((cpt_clean_repo+1))
190+
else
191+
echo "🚨;$repo_name;$count" >> $GITHB_ORGANISATION_VULNERABILITIES_RESULTS
192+
echo "🚨 WARNING: Project '$repo_name' has '$count' vulnerability alerts on GitHub!"
193+
cpt_dirty_repo=$((cpt_dirty_repo+1))
194+
fi
195+
196+
done < "$dir_before_dump/$url_for_cloning"
197+
198+
echo "GitHub organisation name.............: '$organisation_name'"
199+
echo "Total number of projects.............: '$number_of_url'"
200+
echo "Number of projects with alerts.......: '$cpt_dirty_repo'"
201+
echo "Number of projects without alerts....: '$cpt_clean_repo'"
202+
203+
rm "$github_request_result_file_raw"
204+
rm "$github_request_result_file_json"
205+
cd "$dir_before_dump"
206+
rm -f $url_for_cloning
207+
208+
echo "Check done!"

‎toolbox/github/utils/count-vulnerabilities-nodes.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/python3
2+
# Software Name: floss-toolbox
3+
# SPDX-FileCopyrightText: Copyright (c) 2022 Orange
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# This software is distributed under the Apache 2.0 license.
7+
#
8+
# Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
9+
10+
# Version.............: 1.0.0
11+
# Since...............: 25/02/2022
12+
# Description.........: In the given JSON file, produced by GitHub GraphQL API request file, count and return the number of vulnerabilities nodes
13+
#
14+
# Usage: python3 count-vulnerabilities-nodes.py --file file_name
15+
#
16+
17+
import argparse
18+
import json
19+
import sys
20+
21+
# Check arguments
22+
# ---------------
23+
24+
parser = argparse.ArgumentParser(description='In the given JSON file, produced by GitHub GraphQL API request file; count and return the number of vulnerabilities nodes.')
25+
required_args = parser.add_argument_group('Required arguments')
26+
required_args.add_argument('-f', '--file', help='The JSON file to process', required=True)
27+
args = parser.parse_args()
28+
29+
file_name = args.file
30+
31+
# Extraction
32+
# ----------
33+
34+
with open(file_name, 'r', encoding='utf-8') as f:
35+
entries = json.load(f)
36+
f.close()
37+
38+
vulnerabilitiy_nodes = entries["data"]["repository"]["vulnerabilityAlerts"]["nodes"]
39+
vulnerabilitiy_nodes_count=len(vulnerabilitiy_nodes)
40+
41+
print(vulnerabilitiy_nodes_count)
42+
sys.exit(0)

0 commit comments

Comments
 (0)