As we have been working through the process of developing our decentralized smart contract audit platform, we have determined that the final versions outlined here is a huge endeavor with many different part that can take years to fully implement. In order to start providing assurance over smart contracts sooner while building towards our finalized platform, we have decided to create a decentralized bug bounty program. We have made this decision as it can be built much sooner, provide assurance over smart contracts, and most of the features used in the bug bounty program will be used in our finalized audit program such as our token voting features as well as our decentralized treasury. We call it the Bug Exterminator.
Who Can be a Bug Exterminator?
Anyone can be a Bug Exterminator. There are no selection criteria, it is open to anyone who wants to spend time reviewing the code and receive payment for their findings. You simply need to fill out a registration form on our website that will create a username for yourself and an avatar or photo of yourself. You can remain anonymous for this role, we only need a username which you will utilize when submitting your bug reports for anything you find.
How is the Number of Bounties Determined?
The number of audits or bounties we can perform will always be a function of how much funds are in our treasury at any given time. We are still in the process of determining the total funding we will initially offer for each bounty (all bounties will have the same amount) and this decision will eventually be transitioned to the token holders. Determining the number of bounties that can be run each period will be a function of (total treasury funds – treasury reserves[1]) / bounty amount.
How Are Projects Selected for Bounties?
This will run exactly like the process for our smart contract audits. There will be a window where anyone who holds our AUDIT token can propose a project they would like us to have a bug bounty for. This can be members from the community who care about a particular project as well as dev teams who are interested in running a bug bounty but may not be able to fund it themselves, as well as increase the visibility of their project. The window will be long enough that members can discuss and debate the options in our discord as well as invite different projects to speak and tell us about themselves if they are seeking to participate.
After the proposal window has closed, the voting window will open where all token holders will vote yes or no to any project during the round (similar to Project Catalyst). Each AUDIT token held will equal one vote, and it will take the total sum of yes votes, minus the total sum of no votes, and put all projects in order from most net yes votes to least. We will do as many as we have the funds for, starting with the highest voted project and working our way down.
Note – Dev teams are allowed to submit their projects for voting by the community. They are permitted to offer incentives to the community to encourage people to vote for them. For example, a team could offer to airdrop 5% of their total supply to AUDIT token holders if they are selected for a bug bounty program. This may be appealing to teams that are poorly funded but have ample equity. It benefits the dev teams seeking assurance over their smart contracts and it benefits AUDIT token holders by getting tokens of various projects seeking assurance. Credit for this suggestion goes to Adam Rusch.
How Does the Bug Review Process Work?
- Bug Exterminators will have a window to review the code and submit a report for anything that they find.
- They are required to submit 1 report per bug (cannot include multiple bugs in the same report.
- Payouts will be on a first come first serve basis. So if the same bug is submitted by multiple people, whoever submitted first will receive the payment.
- Bug Exterminators are allowed and encouraged to work as a team. They would simply register the team during the registration process same as they would an individual.
- There will be a standard template that they can use when submitting the form to ensure that they have provided all the necessary information.
- They will need to include clear instructions so that a reviewer can easily understand and replicate the issue that was found.
- Each exterminator will be required to self-designate the risk rating when submitting the form.
- Payment amounts will already be pre-determined for each risk rating and they will receive a payment of ADA as well as AUDIT tokens to give them a voice in our organization.
Risk Rating
We will be using the OWASP risk rating criteria and an excel sheet will be provided as part of the template which utilizes simple drop downs to determine the risk rating. They will be required to justify each selection choice they made in determining the risk rating (examples will be provided in the template). The risk ratings that the exterminator will be able to choose from are High, Medium. Low, and Non-Critical. Any submission that does not apply to the smart contract itself will be designated as non-critical. An example of the OWASP risk rating criteria is shown below.
Leaderboard
There will be a leaderboard on the website that lists out each Bug Exterminator along with how many bugs they have found and how much money they have collected. This is to bring awareness to the great people making a difference in our community. It also makes it a bit of a competition to encourage participants to find the most and be the best. We plan to have various awards and prizes periodically for the people at the top of the leaderboard.
Bug Evaluation Process
This is where things get trickier. We needed to develop a way to review the bug reports submitted and confirm which risk rating the bugs should ultimately fall under. This is fairly simple to implement if it is a centralized entity such as our Dev team performing the review. But the goal is to implement a decentralized program which requires us to get a bit creative and develop new methodologies to what is currently being done. We decided to use a variation of our decentralized audit platform that will require having a pool of qualified candidates who are available to review the bug reports for compensation.
How Are Evaluators Selected & Onboarded?
We will utilize a new process called the decentralized interview. Since the goal is to move towards decentralized audits, we do not want to have the decisions to onboard a judge be in the hands of the development team. Instead, we would like to have the entire process in the hands of the DAO where the organization is deciding whom they would like to onboard. There will be an application form on the website where the candidate can provide their credentials and experience which demonstrates that they have the competency to perform this role.
Applicants can apply anytime, and the DAO will hold a vote once per month to admit new judges into the organization. The voting will work like any other vote where 1 token = 1 vote. Any Evaluator will need to obtain a supermajority (66% of the vote) to be admitted. They will need to KYC in order to verify the information that they provided but after they are admitted into the evaluator pool, they can utilize a pseudonym if they prefer so that their reviews are not linked to their identity.
We expect that members in the Evaluator pool will also be participating in the Bug Bounties which is OK and encouraged, but you cannot be an Evaluator on a project that you have submitted bugs for.
Bug Report Review
Once an Evaluator has been approved by the DAO, they are added to a pool of available Evaluators. For each bounty program that is running, the system will be configured to automatically select one Evaluator at random and assign them to a program. This evaluator will be responsible for going through all bug reports submitted by Exterminators and assessing them. They will be required to identify the bug or vulnerability that was found and certify that it is a valid bug/vulnerability. They will also be required to review the risk assessment that the Exterminator provided and determine if it is correct or not. The evaluator will be required to write a commentary on why they agree with the assessment that the exterminator gave or justify why they believe it should be different and indicate the rating that it has been given.
The Evaluator will do this for all reports that were submitted and when they are done, they will be required to compile a summary report that shows all bug reports submitted (both valid and invalid) along with their justification for why it is valid or invalid and why it gets the risk rating they gave it.
Certification
After the Evaluator has compiled the summary report with all justifications, they will submit this report for certification. The certification process works as follows: 3 new evaluators are chosen from the Evaluator pool at random who will be responsible to review the summary report and certify the results of the Evaluator (i.e. certify which reports should not be paid, which reports should be paid, and the risk rating for each). All three will need to be unanimous for the results to be finalized. If they are not unanimous, there is a message board on the website where they can debate and post comments about why they feel a certain issue Is valid, invalid, or why the risk rating is incorrect. They will have an additional 5 days to debate and then the majority opinion will be certified (this is still being thought through and refined).
The certification process was designed to limit the power of any single Evaluator, to provide better results by requiring multiple people to concur on the results, and to prevent an Evaluator from colluding with an Exterminator (e.g. Judge is selected, tells their friend to submit several reports that they will just approve and split the money with each other). Since we are designing this system to be decentralized, there needs to be a process that keeps everyone accountable, acting in good faith, and able to work independently of any centralized oversight.
Summary Report
Once the results are certified, the system will notify the Evaluator. They will then compile a final report that only contains the valid bugs certified (descriptions, risk ratings, payouts, Exterminator who found them. This report is published on the website for all to see. After it is published, payments are sent to everyone. The payouts for low, medium, and high are already pre-determined. As long as the evaluator can confirm the issue and confirm what the risk rating should be, they can be queued up to receive payment which will be processed by the DAOs treasury.
Benefits to Audit Program?
- Smart Contracts used for proposing and voting audit projects will also be utilized for proposing and selecting which bug bounty programs to run. This program will be a litmus test that allows us to refine our voting for when we get to our comprehensive audits
- The process of maintaining a pool of Evaluators and assigning them for review tasks will operate similarly to the audit program. Using this in the bug bounty program will give us the opportunity to view it in action in the real world and make any refinements as necessary.
- Recruiting talented individuals to audit on our behalf is crucial in performing decentralized smart contract audits. We expect the bug bounty program to attract qualified candidates who may wish to learn our audit methodology and audit perform audits on behalf of our DAO.
- The final phase of our decentralized audits include a crowd-sourced review where any additional bugs found would impact the auditor’s pay and standing. This bug bounty platform will be utilized for those purposes.
Open Infrastructure
We will be funding bounty programs that are selected by the community on a regular basis. We also plan to open up the platform for any project that would like to launch a bug bounty program and fund it themselves. We will do this for several reasons. First is to help ensure a steady stream of income to people who wish to participate as Bug Exterminators and Evaluators. Second is we believe bug bounty programs are an effective way to gain assurance over a smart contract and would like to encourage as many projects to do this as possible by opening up our infrastructure. Lastly, we will request a fee of 5% of the total bounty offered for using our services which will be sent to the DAO treasury and used to fund additional bounty programs.
- Once the results are certified, the evaluator will compile a completed report that only contains the valid bugs discovered and this is published on the website for all to see. After it is published, payments are sent to everyone.
- Evaluator work is expected to be 2 days of effort at the most, based on other bug bounty programs I have reviewed. They will be compensated 5K for their efforts with AUDIT token. The ones who certify the results have an easier job and will each be compensated 1K in AUDIT tokens for their effort.
[1] Reserves would be a level that the treasury plans to maintain for any unforeseen expenses that may arise