Add files via upload

Design Proposal/SDP Logger Storage Design outline.txt

@@ -0,0 +1,160 @@
+SDP Logger Storage Design Outline
+________________
+
+
+
+
+        This document outlines the following components of our Senior Design Project: 
+
+
+1. Logger Design and Deployment
+2. Monitor Outline and API Considerations
+3. Tutorials
+   1. Go installation
+   2. Local Logger repo initialization (WIP)
+   3. Local Trillian
+4. Roles 
+Logger
+Logger design
+1. Local trillian deployment
+* Source: trillian/examples/deployment/docker
+* Run with Docker Compose
+* Database client
+* Database server
+* Log_server
+* Log_signer
+(Following sourced from certificate-transparency-go/ManualDeployment.md)
+2. Tree provisioning
+* github.com/google/trillian/cmd/createtree
+3. Key Generation
+* OpenSSL command line 
+4. CA certificates
+* Ca-certificates package
+5. CTFE configuration
+1. Log_id
+2. Prefix of the path where the log will be served at
+3. MMD (maximum merge delay)
+4. Accepted root CA certs
+5. Private key for log instance
+6. Public key (not used in CTFE)
+Logger Deployment Steps:
+1. Run the trillian Database in a docker container
+2. Run the CTFE Personality Server
+Current Status: 
+* Trillian running perfect
+* Included Integration tests completely working + passing. Issues with running the personality manually halted ability to work fully
+* Fortunately The logger, once running, is merely an interacting entity. Doesn’t require much/any manual coding, as it is mainly an entity for 
+________________
+
+
+Monitor  
+Monitor Options:
+* CTClient.go (Function calls in above flowchart)
+   * Pros: Simple, clear functions for interaction with loggers
+   * Cons: Will require building monitor from scratch, which could be significantly more work that the other 2 options
+* CertSpotter:
+   * Pros: We have it running, least complex of the options
+   * Cons: No Gossip implemented, currently uses their own logger from online (not-local)
+* Monologue
+   * Pros: Most Fleshed out of the three options
+   * Cons: Seems like a Minimum Viable Product, and thus has the least documentation/explanations. TODO messages in the code
+
+
+We will likely be implementing a combination of these as needed. We will use CTClient.go as a baseline and add in functionality from other monitor implementations should we find any gaps. This will also involve creating our own monitor which performs the loop outlined in the monitor flowchart and interacts with the logger on our local machine. 
+
+
+We will be using Google’s logger found at https://ct.googleapis.com/pilot/. This follows the IETF CTv1 specifications outlined here https://www.rfc-editor.org/rfc/rfc6962. 
+
+
+Examples of HTTP GET queries for log information are:
+* STH: https://ct.googleapis.com/pilot/ct/v1/get-sth
+* Log Entries: https://ct.googleapis.com/pilot/ct/v1/get-entries?start=1000&end=1100
+
+
+
+
+
+
+
+
+Tutorials:
+Setting up the logger to run locally 
+A work in progress guide by Finn :-) 
+Requirements: 
+* Install Go
+   * If you’re on Windows get WSL up and running and install go there
+   * If you’re on Mac make sure you have git installed (msg me if not) 
+* Make sure the Go path (~/.go by default) is in your $PATH
+   * Running (echo $PATH | grep “go”) and looking for the directory will do this
+   * If it’s not, run (export PATH=$PATH:$(go env GOPATH)/bin )
+* Install Docker from https://docs.docker.com/engine/install/#server
+   * (TBH I was in the middle of writing about how to get trillian running and then stopped. But you’re gonna clone the trillian repo and then follow the instructions here: https://github.com/google/trillian/blob/master/examples/deployment/README.md)
+Preparing the repo
+1. Clone the entire repo (git clone https://github.com/google/certificate-transparency-go/)
+   1. Note: When we start making edits we can “fork” this repo and have our own copy in a personal github repo.
+2. Go into the folder (cd certificate-transparency-go) and run (go install)
+Running CTClient (this is a CLI for querying loggers (like Marcus’ command line tool)
+3. Go to the CTClient folder (cd certificate-transparency-go/client/ctclient)
+4. Run the file (go run .)
+   1. Wait. It takes a while and on the first run as it downloads many files.
+5. When it’s done it should output some different commands you can run.
+   1. To run these we can run go run . [flags] [cmd])
+   2. So E.X: (go run . -first 0 -last 10 getentries)
+________________
+      1. Setting up Trillian Locally By Jie
+1. Clone source
+git clone https://github.com/google/trillian.git
+cd trillian
+2. Download Docker and Docker Compose
+1. Install Docker Engine
+Install Docker Engine on Ubuntu | Docker Documentation
+2. Install Docker Compose
+sudo apt-get install docker-compose
+
+
+3. Post-Installation
+Post-installation steps for Linux | Docker Documentation
+4. Create Trillian container with Docker Compose
+# Set a random password
+export MYSQL_ROOT_PASSWORD="$(openssl rand -hex 16)"
+
+
+# Bring up services defined in this compose file.  This includes:
+# - local MySQL database
+# - container to initialize the database
+# - the trillian server
+docker-compose -f examples/deployment/docker-compose.yml up
+________________
+Roles: 
+
+
+Languages, Software, Packages, IDEs, and Additional coding Resources: 
+* Go
+   * The google certificate-transparency-go repository
+      * ctclient.go
+      * logclient.go
+      * Trillian
+   * VSCode
+* Docker for Logger DB deployment
+
+
+
+
+Group Members Operating Systems: 
+* MacOS
+* Windows using the Windows Subsystem for Linux as needed
+________________
+Authoring and Reviewing Notes:
+        TOPICS:                                                
+Logger
+	Design portion: Jie, Revisions by Finn
+Deployment portion + Current status: Finn
+	Monitor
+	Flowchart: Marcus, reviewed by Finn & Isaac 
+Api Pros/Cons: Finn, reviewed by Isaac
+Further Discussion: Marcus, reviewed by Finn
+	Tutorials
+	Finn
+Trillian portion: Jie - Reviewed by Finn
+	Roles and additional information
+	Millenia; reviewed by Marcus