Snapshot On May 31st, 2019 - Revision 1
Openly Operated Certification v1.0
Auditors produce Audit Reports to help users and the public verify and find issues in the requirements and claims presented by an Openly Operated product. Transparency doesn't help gauge trustworthiness if very few people understand or have time to look through a product's public audit materials, so auditors help make sense of and verify them, saving everyone time and effort. Learn more about auditors and audit reports.
This Audit Kit contains end-to-end, verifiable proof of how OpenlyOperated.org works. It lets auditors and anyone to fully audit or personally verify the privacy and security of OpenlyOperated.org. Additionally, everyone is invited to report issues, questions, or concerns via the contact methods under the Identify Operation section.
For Auditors conducting an audit, download the following Audit Report Template (which contains all the materials in the Audit Kit) to perform verifications and produce an Audit Report:
This read-only account allows access to viewing the operator's AWS Console for OpenlyOperated.org — it allows you to see a limited version of what the operators and employees can see. The read-only account helps users feel more in control of what is happening behind the scenes with their data, but for a comprehensive look, users should read through the proof-backed points, references, and Audit Reports in the rest of this Audit Kit.
DANGER: Your IP will be logged and saved in the publicly viewable and permanent Infrastructure Audit Trail if you log in to the console with the read-only account below. It cannot be deleted, so if you're concerned about this, use a proxy.
Adminservers. Check that the code in the CodeCommit first step matches the code referenced in the Open Source section, and that the code is successfully deployed to the servers.
Adminservers. Check that none of the servers (or "instances") were launched with SSH keys — there should be nothing under the "Key Name" column.
The following are selected claims made by OpenlyOperated.org regarding user privacy and security. These claims are backed up by specific proof from source code, infrastructure, and other references. Learn about Claims With Proof, or for more technical details, view Requirements.
|The Claim||The email addresses collected for the Openly Operated newsletter are not viewable by anyone, including the operators and employees of OpenlyOperated.org.|
|The Proof||• Encrypted/Hashed Email Addresses - User email addresses collected for the newsletter are encrypted  using AES-256 , the same encryption the United States government uses for top secret information. The emails are also salted and hashed  using SHA-512  for fast lookup.|
|• No Access To Encryption Keys, Hash Salt - The encryption key and hash salt are randomly generated  on initialization and encrypted as a Secure String , so operators never see it and can't decrypt the email addresses.|
|• No Unaudited Access To Database - The database password is randomly generated  on initialization and encrypted as a Secure String , so operators never see it and can't access the database. Additionally, firewall rules 12] prevent the database from being accessed directly by operators or anyone external.|
|References||1. OpenlyOperated.org "Shared" source code repository, Newsletter Subscriber Model, Line 150|
|2. OpenlyOperated.org "Shared" source code repository, Security Utility, Lines 53-61|
|3. Wikipedia, "Advanced Encryption Standard", April 2019|
|4. OpenlyOperated.org "Shared" source code repository, Security Utility, Lines 12-16|
|5. OpenlyOperated.org "Shared" source code repository, Newsletter Subscriber Model, Line 149|
|6. Wikipedia, "Secure Hashing Algorithm 2", April 2019|
|7. OpenlyOperated.org infrastructure repository, "Shared" template, Lines 521-608|
|8. OpenlyOperated.org infrastructure repository, "Shared" template, Lines 950-960, 973-981|
|9. AWS Documentation, Systems Manager Parameter Store - Secure String|
|10. OpenlyOperated.org infrastructure repository, "Shared" template, Lines 1025-1033|
|11. OpenlyOperated.org infrastructure repository, "Shared" template, Lines 1428-1437|
|12. AWS Documentation, Security Groups For Linux Instances, April 2019|
|The Claim||OpenlyOperated.org does not do user tracking or use any third party analytics tools.|
|The Proof||• No Logging IPs Or Personally Identifiable Info - Only errors are logged for debugging issues , and even in those cases, no IP or Personally Identifiable Information is logged . Normal requests and successful requests, such as visits to the OpenlyOperated.org site, are not logged. |
|• No Third Party Analytics - Third party analytics tools are a privacy risk. Some tools are hosted by ad companies (like Google Analytics), and other tools save obscene amounts of user activity onto third party servers, like full screen recording and tracking . OpenlyOperated.org uses no third party analytics , minimizing user privacy risk.|
|• No Access To User Email Address - User tracking through the email newsletter is not possible as operators do not have direct access to the user email addresses — See the "Email Address Privacy" claim and proof above.|
|References||1. OpenlyOperated.org "Main" source code repository, App Logging, Lines 26-29|
|2. OpenlyOperated.org "Main" source code repository, App Logging, Line 23|
|3. "Investigation into at least 11 iOS apps sending sensitive data to Facebook, inc sexual activity", 9to5Mac|
|4. OpenlyOperated.org "Main" source code repository, Website Header Template|
The following are requirements for being Openly Operated, and they exist to ensure a product's full, verifiable transparency. Without them, claims about security or privacy could not be made because there could be gaps, loopholes, or backdoors in these claims. Learn about Requirements.
Corporate: 251 Little Falls Drive, Wilmington, DE 19808
San Francisco: 535 Mission St, 14th Floor, San Francisco, CA 94105
Johnny is responsible for technical decisions and building out Openly Operated's examples, specifications, and documentation. He was an engineer at Apple iCloud and has built several apps featured by the App Store. Johnny is an alumnus of Brown University.
Rahul builds tools for Openly Operated, and also handles legal, management, and directional decisions. He also created Duet Display, a second screen for your iPad, and has previously worked at Apple. Rahul is an alumnus of Stanford University and Georgia Tech.
OpenlyOperated.org is 100% open source. Architecture and code are detailed below.
Admin repositories depend on the
Shared repository. It is hosted on Amazon Web Services, with infrastructure brought up via CloudFormation. See Open Infrastructure for details.
This exists to reduce code duplication between the
Admin webapps, and contains common shared code like logging, email, database access, models, security, and other utilities. Updating
Shared requires a redeployment of
Admin for changes to take effect.
This is the public-facing OpenlyOperated.org website, which uses Handlebars.js templating to render pages generated by the Content Management System on
Admin. It also has an API for the email newsletter signup.
This is a private administrative dashboard for OpenlyOperated.org only accessible to whitelisted IPs. It uses Handlebars.js templating to render pages and has numerous documented APIs that make up its Content Management System and email newsletter features.
OpenlyOperated.org is 100% open infrastructure, with one public infrastructure repository. All configuration and usages of third-party APIs are described below.
OpenlyOperated.org runs on Amazon Web Services, using CloudFormation to bring up almost every resource, from servers (which run Ubuntu) to DNS configurations to deployment pipelines. All the CloudFormation templates are public on Github. OpenlyOperated.org is in the
us-east-1 Northern Virginia AWS region.
Adminservers, so even if the database credentials were stolen by someone, they still couldn't access the database.
Mainserver role cannot access the
Adminserver's database password. Database roles for each server type are also restricted to what is necessary for the role. Firewall rules via AWS Security Groups limit server communication to only the ports and the target groups that are necessary.
Since other companies that run third party APIs (such as analytics tools) are likely not Openly Operated nor provably transparent, usage of third party APIs should be minimized in order to reduce risks to user data. OpenlyOperated.org uses no third party APIs and and sends no user data to third parties. While it would have been faster and easier to use existing third party APIs for features like the newsletter, OpenlyOperated.org built those features from the ground up so that we can provide end-to-end proof of the privacy and security of user data.
Code is deployed using AWS CodePipeline, which consists of CodeCommit (a hosting service for code repositories), CodeBuild (building and testing code), and CodeDeploy (deploying code to servers) — see the relevant infrastructure code here.
To deploy code, code is
git pushed to the CodeCommit repository, where it is automatically built by CodeBuild and uploaded into an S3 "Artifact Bucket" (aka "Deployed Code" on the Infrastructure diagram). CodeDeploy then deploys the code in the artifact bucket to the servers. So, the code in the S3 "Artifact Bucket" is the same as the code in production.
Audit trails show all operator actions since the creation of the Openly Operated product. Their purpose is to provide an unbiased, verifiable source of truth. For OpenlyOperated.org, since its Infrastructure Audit Trail does not record the operator's manual database actions, there is an additional audit trail called the Database Audit Trail.
The infrastructure audit trail provides a log of all major operator actions in bringing up and maintaining the infrastructure. They are usually automatically generated by the platform that the product is built on, and should have a way to verify integrity. OpenlyOperated.org uses AWS's CloudTrail service for its infrastructure audit trail.
AWS provides a manual way to check the integrity of CloudTrail logs against their checksums, but since there are thousands of logs, Openly Operated built a free, open source Mac tool that automatically does this integrity check for CloudTrail logs in an AWS account, called Open Watch. Here's how to use it
Download the open source Open Watch tool and either use the precompiled build release, or follow the instructions to build and run it. Then, use the following settings:
Click "Audit" to begin. This may take up to 30 minutes or an hour, since Open Watch is downloading tens of thousands of audit logs and verifying each of them. If any integrity issues are found, they'll be displayed in the main whitespace area after verification is complete. Please note that it's better to keep the app in the foreground during auditing, because macOS may heavily throttle applications that are in the background and cause the process fail.
To fulfill the Comprehensive section of the Audit Trail requirement, it can be difficult to sift through the thousands of CloudTrail logs to find potentially dangerous operator actions. Fortunately, the Open Watch tool above also performs detection of several common types of dangerous operator actions, displayed as "Rules" in the Open Watch interface, with explanations of each rule.
To manually analyze the infrastructure audit trail, download it below.
OpenlyOperated.org Infrastructure Audit Logs - May 3rd, 2019 - Download
OpenlyOperated.org has a custom-built Administrator dashboard (the
Admin repository) that allows the operator to send manual database queries, in order to perform the necessary administrative task of database migrations. However, because these queries are arbitrary, that could be abused by the operator to retrieve or modify user records, putting user privacy at risk. These queries are not tracked or logged by the Infrastructure Audit Trail (CloudTrail), so OpenlyOperated.org tracks this with the Database Audit Trail.
It works like this: Right before a database query is executed, the
Admin server logs the query to two places: 1) A log viewable to the read-only account called
AdminAudit, accessible here and 2) A log file in a public S3 bucket , accessible here. This way, the public and auditors get to see all database queries that the operator executes (but not the responses), keeping the operator honest.
Adminserver, referenced above. The Infrastructure Audit Trail, along with the Establish Provenance section below, ensures that the referenced code is indeed the same code that's deployed in production.
|References||1. OpenlyOperated.org infrastructure code repository, Global file, Lines 407-408|
|2. OpenlyOperated.org infrastructure code repository, Shared file, Lines 764-775|
|3. OpenlyOperated.org "Admin" code repository, Database Query Controller, Line 50|
|4. OpenlyOperated.org "Shared" code repository, Audit Utility, Lines 15-39|
|5. OpenlyOperated.org infrastructure code repository, Shared file, Lines 195-211|
|6. OpenlyOperated.org "Admin" code repository, Database Query Controller, Line 52|
|7. OpenlyOperated.org "Shared" code repository, Audit Utility, Lines 41-51|
Use the read-only account to view the CloudWatch log stream
PostgresQueries in the
AdminAudit log group to see a complete list of all database queries manually executed by the operator. Alternatively, check the public S3 bucket
AdminAuditBucket for log files prefixed with
PostgresQueries. The queries listed, if any, should be for database migrations, not queries for extracting rows from the database. For example, a
CREATE TABLE query is fine, but a
SELECT * FROM USERS is not.
Provenance is proof that both the code and infrastructure presented match the code and infrastructure in the actual users' production environment.
The infrastructure presented above matches the infrastructure in production because it's brought up by CloudFormation, which is AWS's "code-as-infrastructure" service.
To verify this, use the Read-Only Account to log into the console and check the CloudFormation service in the
us-east-1 region. Under each stack, check the template tab and compare it with the infrastructure files in the CloudFormation public repository. Then, any differences between the template and the current infrastructure can be detected with the Detect Drift functionality.
Auditors should also verify that the domain that users are connecting to is indeed the one auditors are examining, by checking the Route 53, which is AWS's DNS service. Verify that the one of the nameservers in the openlyoperated.org Hosted Zone matches the nameserver returned by a host nameserver lookup from your local machine. On macOS, the command is
host -v openlyoperated.org.
For further proof of infrastructure provenance, download OpenlyOperated.org's full infrastructure audit logs above, under the "Infrastructure Audit Trail - Manual Analysis" section.
The source code presented above matches the source code in production because they're both
git remotes of the same repository, which is built and deployed without modification to production servers via CodePipeline.
Manual modification of source code in production is not possible, because direct access to the servers is disabled and blocked — see the "Open Infrastructure - Security & Encryption - No Direct Server Access" section above.
To verify provenance, use the Read-Only Account and examine the CodePipeline service. Check that the code in the CodeCommit matches the code referenced in the repository, and that the code is successfully deployed to the servers. CodePipeline and CodeDeploy also displays the history of all the previous code deployments, allowing verification of any previous version of source code in production.
Listening to and implementing community suggestions helps Openly Operated products be more trustworthy, secure, and transparent. Answering questions helps clarify confusion and misunderstandings about the product and how it works. OpenlyOperated.org is more than happy to take feedback and discuss any issues or concerns — see the Identify Operation section above for contact and community information.
After this current snapshot, OpenlyOperated.org will continue to be updated, adding new features, fixing bugs, and installing updates. These changes will require a new snapshot and re-audits. The next snapshot and audit date is scheduled for July 1st, 2019.
If you think full, provable transparency is important in the apps and services we all use, subscribe to the Openly Operated newsletter.
This site is written for everyone — companies, developers, everyday users. Make the future Openly Operated by sharing with friends.
User Benefits A deeper look into the many benefits for users, with examples and references.
For Companies See why companies and businesses also benefit from being Openly Operated.
How To The requirements for Openly Operated products, and how to get started.
About Us Read about the values, mission, origin, and creators of Openly Operated.
Reports See live examples of Openly Operated products and their audit reports.
Get Involved Discuss Openly Operated, transparency, the future of the web, and any related topics.