How To Be Openly Operated

The Process, Requirements, and Certification - v1.0


Diagram of the Openly Operated process

For a product to be Openly Operated, it must follow and pass the certification process: fulfill requirements, prove claims, and complete audits. To follow along with examples, open one of the Audits from Example Audits in a separate window. We're happy to answer questions about any of Openly Operated process — contact us at any time.

Note that reaping the user and business benefits of being Openly Operated is in direct conflict with trying to sprint to market first and using your initial users as guinea pigs.

Certification Process

  1. Fulfill Requirements - These are requirements for demonstrating a basic level of transparency, and they can be thought of as the "raw materials" for auditors, security and privacy experts, and the general public to view and analyze.
  2. Claims With Proof - Provide claims regarding the privacy, security, or other important aspects of the product, keeping them as simple as possible. Using content from the Requirements, provide proof of each individual claim. Proof should be reproducible and based on unforgeable evidence. Unsubstantiated assertions such as "Read our Privacy Policy" do not qualify as proof.
  3. Complete Audit - Submit the requirements and claims as an "Audit Kit" to Openly Operated, and we'll match you with auditors to complete the requested audits. The audit materials and results are then made public, allowing the community to ask questions, make requests, and get updates. Services should be re-audited frequently.

Step 1: Fulfill Requirements

These are the requirements for demonstrating a basic level of transparency, and they can be thought of as the "raw materials" for auditors, security and privacy experts, and the general public to analyze and work with.

Each requirement has a description describing the requirement, rationale or the reason behind the requirement, and verification which contains suggestions on how to verify completion of the requirement. Verification is a bare-minimum suggestion — auditors are free to and encouraged to go well beyond it to find issues and flaws.

Identify Operation
Description Clearly state what the product is, its purpose, and people who own and build it. Owners and operators of the service must be identified with at least their full name, recent photos, and reachable contact information, including business address. Pseudonyms, placeholders, and nicknames are not acceptable.
Rationale Identifying owners and operators not only creates meaningful accountability, but it also incentivizes more honest product operation and behavior. Millions of users give their personal information to a product in the course of using it — they have the right to know who is behind it.
Verification Manually verify each item (product, purpose, people, location) by appropriate means, whether that's search engine to find references, viewing social profiles, testing contact information, etc. Verifications of these items won't ever be 100% guaranteed, so it's sufficient to document how the operation's items seem reasonably legitimate and has no red flags.
Open Source
Description All server and client code must be open source and fully documented. All libraries and usage of third party or external APIs must also be noted, leaving no room for unknown behavior.
Rationale The source code determines the behavior of computers and servers, so seeing all the source code (not a partial release) is essential to knowing what is (and isn't) being done with user data, as well as revealing any malicious behavior.
Verification Completeness - Check that all source code is public and that there are no missing parts. Optionally, build the source code to create the product. No part of the source code should be precompiled binaries (closed source), and note any suspicious third party libraries.
User Data Handling - Find the points where user data enters the server, and follow them to their storage (encryption, security) and handling (third parties APIs and libraries). Be sure to check for logging, such as IPs and other Personally Identifiable Information.
Security - What measures are used to protect the server from both internal and external threats? Are they sufficient, up-to-date, and complete? Is it possible for employees to steal user data without leaving a trail?
Open Infrastructure
Description All configuration and infrastructure of the app must be fully documented and public. Any parts of the infrastructure that rely on third parties or opaque services should be noted. This should be detailed enough that an auditor could build a test environment for validation.
Rationale Configuration and infrastructure define the what, when, where, and how of deploying and executing the source code, as well as many crucial aspects relating to user data, such as storage and encryption. It is just as important as source code itself.
Verification Completeness - Check that all infrastructure is public and there are no missing pieces. Optionally, build the infrastructure to create a test environment.
Secrets and Keys - Are operators able to access secrets and keys such as database password, encryption keys, hashes? If so, how can they prove it isn't abused?
Security - What measures are used to protect the server from both internal and external threats? Are they sufficient, up-to-date, and complete? Is it possible for employees to steal user data without leaving a trail?
Audit Trail
Description Openly Operated products must have one or more trails of logs showing all operator actions from the moment the product is created. Audit trails must be impartial, authentic, and comprehensive.
Impartial - Audit trails should be generated by an impartial mechanism, which usually means they're automatically generated by the platform on which the product is built. If the platform's trail doesn't cover all operator actions, any custom audit trails should show proof of impartiality.
Authentic - Audit trails should be impossible or extremely difficult to forge, tamper with, or delete. Auditors and the general public should be able to personally verify authenticity of the audit trails.
Comprehensive - Audit trails should cover the entire time period from the creation of the production environment of the app until the current date. The trail should have a comprehensive record of all potentially dangerous operator actions that could compromise user data.
Rationale In order to see what code and infrastructure is really running in production and to see what the operator has done with user data, Openly Operated products need to present a reliable "source of truth" — unbiased, verifiable audit trails. Without audit trails, auditors and users are left to rely on blind trust of the operators, defeating the purpose of earning trust transparency.
Verification Verify the Impartial, Authentic, and Comprehensive traits described above. For the Comprehensive trait, pay special attention to these questions: Are all operator actions that could possibly compromise user data logged? Assuming the operator is a malicious actor (or bought out by a malicious actor), could user data be secretly exploited without this behavior being exposed?
Establish Provenance
Description Provenance means "the origin of something". Openly Operated products establish provenance by proving that the source code and infrastructure presented in the requirements (the "origin") matches the production environment. To do this, follow the code and infrastructure step-by-step, providing proof and explanation on each step for why tampering cannot occur.
Rationale The requirements of Open Source and Open Infrastructure aren't useful if there's no way to verify that the presented code and infrastructure presented are what's actually running in production. Establishing provenance ensures the code and infrastructure evidence that is cited in Proving Claims is reliable and accurate.
Verification Follow the code and infrastructure from the user to the server and back. Are there any gaps that could be compromise the integrity of the source code and infrastructure presented? Check that the servers that the public is connecting to are the servers being audited, not some other server, either by verifying that the DNS nameserver entries in the infrastructure, or by some other means.

Step 2: Claims With Proof

Provide claims regarding the privacy, security, or other important aspects of the product, keeping them as simple as possible, and by referring to content from the Requirements and other resources as necessary. Proof should be reproducible and based on unforgeable evidence. Unsubstantiated assertions such as "Read our Privacy Policy" do not qualify as proof.

Choosing Claims

Since Privacy Policies and Terms can be lengthy, choose the most relevant points to highlight and explain. This varies depending on the product, but should be what the end user is most likely interested in — for example, a photo sharing app should focus on the handling of user photos, instead of the algorithm used to compress photos.

Questions that may help in choosing privacy and security claims to prioritize:

  • What is the data in question? (e.g, User Email)
  • Who can access the data? (e.g, Customer Support Staff)
  • Why do they need access to this data? (e.g, Need to respond to customer inquiries)
  • When can they access the data? (e.g, Only when a customer emails in to our support email)
  • How can they access the data? (e.g, Support staff must log into the support dashboard)
  • How is the access audited? (e.g, The customer is sent an audit alert every time this data access occurs)
Prove Claims

Proof and explanations should be simple to understand and whenever possible, point to the exact files in the code, infrastructure or audit logs and results for reference. Break down proof into smaller steps when there is a sequence of logic. Auditors, end users, and the community can request clarifications and further proof in the final Complete Audit stage.

Step 3: Complete Audit

Submit the requirements and claims as an "Audit Kit" to Openly Operated, and we'll match you with auditors to complete the requested audits. During the audit, you'll address issues and questions brought up by the auditor. Finally, audit materials and results are then made public, allowing the community to ask questions, make requests, and get updates.

Assemble Audit Kit

Audit Kits contain the Requirements and Claims with Proof sections above, and they let auditors or anyone else verify the privacy, security, and other claims of an Openly Operated product. The format of an Audit Kit is described below, and you can view complete examples in the next section.

  • Title - Should be in the format "Audit - [Product Name]"
  • Certification Version - The Openly Operated certification version being used, for when the certification's standards and requirements are updated. Currently, v1.0 as stated at the top of this document.
  • Audit Scope - Scope what's being audited and what's not being audited. For example, if your product has a website, iOS app, and server, and you're only submitting an Audit Kit for the website, then be clear that the iOS app and server are not being audited.
  • Snapshot Date - The point in time that the Requirements and Proof of Claims are referencing — it's used to differentiate between the current audit and future audits of the same product. Consumers and users can use the Snapshot Date to know that "The product as of March 3rd, 2019 was audited, and here are the results for it."
  • Snapshot Version(s) - The Snapshot Version is the version of your product that's being audited, so consumers can more easily match the audits— e.g, "I'm using v1.2 of the app, I'll need to read the audit report for v1.2." If the products consists of different server, infrastructure, and client components, each of their version numbers should be included. Commit Hashes are also acceptable.
  • Revision Number, History - Should be formatted as "Revision #", with # starting at "1". The revision is increased by one every time changes to the Audit Kit are made during the audit process as a result of auditor feedback. Changes to the Audit Kit are saved to a "Revision History" section at the bottom of the Audit Kit.
  • Requirements, Claims With Proof - Provide the requirements and claims with proof sections above.
  • Next Audit Date - The date that you expect the next audit to be conducted. Since products are expected to change with new features and fixes, audits become stale and less useful over time — what is audited in this Audit Kit snapshot may not be anywhere close to what the product looks like in a year from now. This also helps users know when they can check back.
  • Read-Only Account (optional) - A read-only account of your operator console. This can go a long way in helping users have an idea of what's happening behind the scenes with their data, and how the everyday management of the product works — it's the closest thing to being an employee without actually being one. The read-only account should have extremely limited permissions, being careful to not allow access to secret keys or any user data. The read-only account can also help expedite verification of some requirements and claims. For example, it's much easier to find and verify the currently deployed code on production servers by just browsing to the operator console's deployment service.
  • Audit Bounty (optional) - Optionally, to further incentivize auditors to help find issues with your Openly Operated product, you can create a bounty reward for the Auditor for each relevant issue found. For example, you might create a cash bounty if an Auditor finds a gap in your proof of a certain privacy or security claim.
Example Audits

The following are example Audit Kits and reports that can be used as references when learning about Openly Operated and/or building your own Openly Operated products and Audit Kits.

  • OpenlyOperated.org Audit - A complete example of a simple Openly Operated service is this current website you're browsing, OpenlyOperated.org. This is a website and backend with a Content Management System and Email Newsletter functionality, all built from the ground up to be Openly Operated.
  • Confirmed VPN Audit - A consumer app with real users, live in the App Store and Google Play Store. Confirmed VPN is an Openly Operated VPN. Since VPNs increase security and privacy by routing a user's internet traffic through their servers, it's crucial that VPNs be totally transparent. Confirmed VPN was built to prove that Openly Operated works in real consumer products.
Submit For Audit

The Audit Kit should be submitted to Openly Operated using the email link below:

Submit Audit Kit

After submission, we'll get back to you with in three business days, and let you know if there's any material that's missing, incorrect, or insufficient.

Once everything looks good, we match you with an auditor with the appropriate skill set, and we'll produce an Audit Report Template for your auditor to use. The Audit Report Template contains your Audit Kit, plus additional fields for verification of each part of the Audit Kit, as well as sections for summary and information about the auditor. From then on, all correspondence with auditors should also CC audit@openlyoperated.org, so we can keep an accurate record of communications.

How does the final audit report get produced? Read the details of the auditing process and how a final audit report is produced on the Auditors page.

Community

After the Audit Kit and Audit Report is published, users of the product (and the public) can become part of helping improve the product by finding issues that the auditor may not have focused on. Anyone can contact the operator for further questions, clarifications, and feedback. If any changes are necessary, code/infrastructure is updated, or a new revision is released. Establishing a public community such as a Telegram channel or moderated subreddit is encouraged, and of course, users can create issues in the product's Git repository.

At the point, the operator can also choose to create a public bounty reward for finding issues or contributing to the product or its audit report and Audit Kit. The reward doesn't necessarily have to be cash — in fact, non-cash rewards such as a year's free subscription could build a stronger community around the product.

Change Log

V1.0
  • Initial release of Openly Operated certification process and requirements.