Documentation

Get started within minutes.

Prerequisites

Supported Data Formats: Protocol Buffers and JSON (future)
RPC Framework: gRPC (recommended) or any implementation that supports HTTPS with SSL.
API Access Key: Requires RogueDB subscription. See Stripe Customer Portal for subscription details.
Subscription Identifier: Requires RogueDB subscription. See Stripe Customer Portal with subscription details.
RogueDB's Root CA Certificate: Available for download in GitHub repo.
RogueDB's Proto Files: Available for download in GitHub repo.

Assumptions

For the purpose of the code examples, all of RogueDB's proto files and Root CA Certificate are located in a folder called: roguedb. All of your proto files are located in a folder called: data_schemas.

‍

Supporting functions implementation denoted by a comment of "See Reference Functions..." are excluded for clarity and not required to use RogueDB. The implementations are included at the bottom for your convenience.

‍

Initial Connection

The following code establishes a connection to your RogueDB instance:

Python
C++
import grpc
from roguedb.roguedb_pb2_grpc import RogueDBStub

with open("roguedb/rogue_llc_ca.crt", "rb") as input_file:
    credentials = grpc.ssl_channel_credentials(input_file.read())
    channel = grpc.secure_channel(f"{roguedb.dev}:{443}", credentials)
    roguedb = RogueDBStub(channel=channel)
// #include <format>
// #include <grpcpp/grpcpp.h>
// #include "roguedb/roguedb.grpc.pb.h"

// See Reference Functions...
grpc::SslCredentialsOptions sslOptions{};
sslOptions.pem_root_certs = readFile("roguedb/rogue_llc_ca.crt");
    
std::unique_ptr<Rogue::Services::RogueDB::Stub> roguedb{
    Rogue::Services::RogueDB::NewStub(grpc::CreateChannel(
        std::format("{}:{}", "roguedb.dev", 443), 
        grpc::SslCredentials(sslOptions)))};

Authentication

RogueDB requires the Identifier and API Key (service account or user account). All queries have a field called "credentials" that needs to be initialized. See the following example of initializing the credentials:

1from roguedb.queries_pb2_grpc import Credentials
2
3credentials = Credentials(
4    api_key="ROGUEDB_API_KEY",
5    identifier="ROGUEDB_SUBSCRIPTION_ID")

1// #include "roguedb/queries.grpc.pb.h"
2
3Rogue::Services::Credentials credentials{};
4credentials.set_api_key("ROGUEDB_API_KEY");
5credentials.set_identifier("ROGUEDB_SUBSCRIPTION_ID");
6
7// All calls with gRPC require the following:
8grpc::ClientContext context{};
9Rogue::Services::Response response{};

Schema Registration

There are two primary steps for registering schema with a RogueDB database:

Identify the indices in the Protocol Buffer definition.
Provide the schemas to RogueDB.

Marking Indices

The first step of registering a Protocol Buffer is to mark the fields to be used for indexing with the pattern: // index-# (ex. // index-1). All schemas must have indices for use with RogueDB.

‍

Guidelines for these fields are the following:

Fields selected for an index should form a unique identifier per entry.
For maximal performance, order indices in the following manner:
- Uniqueness: Prefer stock symbols vs. timestamps for ordering as symbols are unique to a batch of entries.
- Granularity: Prefer stock symbols vs. aggregates as symbols allow for greater diversification and distribution of entries.
- Uniqueness: Timestamps as the first index cause high collision due to commonality across entries.

An example inspired by financial market data:

Protobuf
syntax = "proto3";

import "google/protobuf/timestamp.proto";

message Aggregate
{
    string symbol = 1; // index-1 Extra notes are not included after marking.
    google.protobuf.Timestamp timestamp = 2; // index-3z Non-numerical characters allowed, though discouraged.
    uint32 aggregate_period = 3; //index-2 No space between "//" and "index" is also valid.
    float open = 4;
    float close = 5;
    float low = 6;
    float high = 7;
    uint32 volume = 8;
}

Registering Schemas

The requirements to register your schema are the following:

Filename consists only of alphanumerical, underscore ("_"), and dash "-" characters.
Contents of the schema as a string.
Source type (ex. Protocol Buffers).

1from roguedb.queries_pb2 import Subscribe, Source
2
3subscribe = Subscribe(credentials=credentials)
4
5# See Reference Functions...
6proto_files = detect_files(directories=["data_schemas",])
7for proto_file in proto_files:
8    with open(proto_file, "r") as input_file:
9        schema = subscribe.schemas.add()
10        schema.contents = input_file.read()
11        schema.source = Source.PROTO
12        schema.filename = proto_file.name # Can be relative, absolute, or filename only.
13
14response = roguedb.subscribe(subscribe)

1// #include "roguedb/queries.grpc.pb.h"
2
3Rogue::Services::Subscribe subscribe{};
4*subscribe.mutable_credentials() = credentials;
5    
6// See Reference Functions for implementation details of detectFiles.
7const std::vector<std::string> directories{ "data_schemas/" };
8const std::vector<std::string> protoFiles{ detectFiles(directories) };
9for(const auto& file : protoFiles)
10{
11    Rogue::Services::Schema& schema{ *subscribe.add_schemas() };
12    // See Reference Functions...
13    schema.set_contents(readFile(file));
14    schema.set_source(Rogue::Services::Source::PROTO);
15    schema.set_filename(file.filename()); // Can be absolute, relative, or filename only.
16}
17
18roguedb.subscribe(&context, subscribe, &response);

CRUD Operations

RogueDB does not require a structured querying language (eg. SQL) to execute. The API works similar to standard data structures in popular programming languages. All calls require providing credentials for authentication.

Test schema utilized in the examples:

Protobuf
syntax = "proto3";

message Test
{
    int32 attribute1 = 1; // index-1
    bool attribute2 = 2;
}

Create

There are no differences in single versus batch create operations in terms of API.

1from roguedb.queries_pb2 import Insert
2from data_schemas.test_pb2 import Test
3
4insert = Insert(credentials=credentials)
5for index in range(10):
6    insert.messages.add().Pack(Test(attribute1=index, attribute2=False))
7
8response = roguedb.insert(insert)

1// #include "roguedb/queries.pb.h"
2// #include "data_schemas/test.pb.h"
3
4Rogue::Services::Insert insert{};
5*insert.mutable_credentials() = credentials;
6for(uint32_t index{0}; index < 10; index++)
7{
8    Test test{};
9    test.set_attribute1(index);
10    test.set_attribute2(false);
11    *insert.add_messages() = std::move(test);
12}
13
14roguedb.insert(&context, insert, &response);

Update

There are no differences in single versus batch update operations in terms of API.

1from roguedb.queries_pb2 import Update
2from data_schemas.test_pb2 import Test
3
4update = Update(credentials=credentials)
5for index in range(10):
6    update.messages.add().Pack(Test(attribute1=index, attribute2=False))
7
8response = roguedb.update(update)

1// #include "roguedb/queries.pb.h"
2// #include "data_schemas/test.pb.h"
3
4Rogue::Services::Update update{};
5*update.mutable_credentials() = credentials;
6for(uint32_t index{0}; index < 10; index++)
7{
8    Test test{};
9    test.set_attribute1(index);
10    *update.add_messages() = std::move(test);
11}
12
13roguedb.update(&context, update, &response) };

Delete

There are no differences in single versus batch delete operations in terms of API.

1from roguedb.queries_pb2 import Remove
2from data_schemas.test_pb2 import Test
3
4remove = Remove(credentials=credentials)
5for index in range(10):
6    remove.messages.add().Pack(Test(attribute1=index, attribute2=False))
7
8response = roguedb.remove(remove)

1// #include "roguedb/queries.pb.h"
2// #include "data_schemas/test.pb.h"
3
4Rogue::Services::Remove remove{};
5*remove.mutable_credentials() = credentials;
6for(uint32_t index{0}; index < 10; index++)
7{
8    Test test{};
9    test.set_attribute1(index);
10    *remove.add_messages() = std::move(test);
11}
12    
13roguedb.remove(&context, remove, &response);

Read

RogueDB's querying API allows programmatic composition by using logical operators (&&, ||) and comparison operators (==, !=, <, <=, >, >=) to replicate conditionals in your native progamming language. There are no joins, inner joins, outer joins, wheres, on, as, or any other special keywords required to be learned as a result.

‍

A query can use an entire message with its indices or specifying individual fields as the operands in a query. RogueDB supports basic queries that involve the same logical operator (eg. homogenous application) and a single schema. There must be a 1:1 (operand : conditional) or 1:1:1 (operand : conditional : operand field) for the query to be valid.

‍

Future support is coming for the following:

Complex queries that involve combining different logical operators for a single schema.
Support for multiple schemas in a single query.

The following example demonstrates using a schema's indices as the operand:

1from roguedb.queries_pb2 import BaseExpression, Search, LogicalOperator, ComparisonOperator
2from data_schemas.test_pb2 import Test
3
4# Conditional: Test.attribute1 >= 1 and Test.attribute1 <= 10
5search = Search(credentials=credentials)
6expression = search.queries.add().base_expression
7expression.logical_operator = LogicalOperator.AND
8
9expression.comparison_operators.append(ComparisonOperator.GREATER_EQUAL)
10expression.operands.add().Pack(Test(attribute1=1))
11
12expression.comparison_operators.append(ComparisonOperator.LESSER_EQUAL)
13expression.operands.add().Pack(Test(attribute1=10))
14
15response = roguedb.search(search)
16# See Reference Functions...
17messages = parse_messages(response)

1// #include <vector>
2// #include "roguedb/queries.pb.h"
3// #include "data_schemas/test.pb.h"
4
5Rogue::Services::Search search{};
6*search.mutable_credentials() = credentials;
7    
8// Conditional: Test.attribute1 >= 1 && Test.attribute1 <= 10
9Rogue::Services::BaseExpression& expression{ (*search.add_queries()).base_expression() };
10expression.set_logical_operator(Rogue::Services::LogicalOperator::AND);
11
12Test test{};
13expression.add_comparison_operators(Rogue::Services::ComparisonOperator::GREATER_EQUAL);
14test.set_attribute1(1);
15(*expression.add_operands()).PackFrom(test);
16    
17expression.add_comparison_operators(Rogue::Services::ComparisonOperator::LESSER_EQUAL);
18test.set_attribute1(10);
19(*expression.add_operands()).PackFrom(test);
20
21roguedb.search(&context, search, &response);
22// See Reference Functions...
23std::vector<Test> messages{ parseMessages<Test>(response) }:

The following example demonstrates using a schema's fields as the operand:

1from roguedb.queries_pb2 import Search, LogicalOperator, ComparisonOperator
2from data_schemas.test_pb2 import Test
3
4# Conditional: Test.attribute1 >= 1 and Test.attribute2 != False 
5search = Search(credentials=credentials)
6expression = search.queries.add().base_expression
7expression.logical_operator = LogicalOperator.AND
8
9expression.comparison_operators.append(ComparisonOperator.GREATER_EQUAL)
10expression.operands.add().Pack(Test(attribute1=1))
11expression.operand_fields.append(1)
12
13expression.comparison_operators.append(ComparisonOperator.NOT_EQUAL)
14expression.operands.add().Pack(Test(attribute2=False))
15expression.operand_fields.append(2)
16
17response = roguedb.search(search)
18# See Reference Functions...
19messages = parse_messages(response)

1// #include <vector>
2// #include "roguedb/queries.pb.h"
3// #include "data_schemas/test.pb.h"
4
5Rogue::Services::Search search{};
6*search.mutable_credentials() = credentials;
7
8// Conditional: Test.attribute1 >= 1 && Test.attribute2 != false
9Rogue::Services::BaseExpression& expression{ (*search.add_queries()).base_expression() };
10expression.set_logical_operator(Rogue::Services::LogicalOperator::AND);
11    
12Test test{};
13expression.add_comparison_operators(Rogue::Services::ComparisonOperator::GREATER_EQUAL);
14test.set_attribute1(1);
15(*expression.add_operands()).PackFrom(test);
16expression.add_operand_fields(1);
17    
18expression.add_comparison_operators(Rogue::Services::ComparisonOperator::NOT_EQUAL);
19test.set_attribute2(false);
20(*expression.add_operands()).PackFrom(test);
21expression.add_operand_fields(2);
22    
23roguedb.search(&context, search, &response);
24// See Reference Functions...
25std::vector<Test> messages{ parseMessages<Test>(response) }:

User Roles and Permissions

RogueDB supports the creation of users and assigning of permissions on a schema level basis. Your API access key associated with your subscription has all permissions by default and cannot be revoked. Users have no permissions by default. The following permissions can be assigned:

Read: The ability to perform the Read operation (aka search) for a schema.
Write: The ability to perform the Create, Update, and Delete operations (aka insert, update, and remove) for a schema.
Execute: The ability to modify a schema's definition through the Configuration service.
Admin: Grants all permissions and overrides any existing permission.

Usage

RogueDB uses a case insensitive mapping of schema to permissions. The following is an example of User creation:

1from roguedb.queries_pb2 import Insert
2from roguedb.user_pb2 import User, Permission
3
4insert = Insert(credentials=credentials)
5
6user = User(api_key="CUSTOMER_MANAGED_KEY", admin=False)
7user.permissions["test"] = Permission.READ_WRITE_EXECUTE
8insert.messages.add().Pack(user)
9
10response = roguedb.insert(insert)

1// #include "roguedb/queries.pb.h"
2// #include "roguedb/user.pb.h"
3
4Insert insert{};
5*insert.mutable_credentials() = credentials;
6
7Rogue::Services::User user{};
8user.set_api_key("CUSTOMER_MANAGED_KEY");
9user.set_admin(false);
10(*user.mutable_permissions())["user"] = Rogue::Services::Permission::READ;
11    
12*insert.add_messages() = std::move(user);
13roguedb.insert(&context, insert, &response);

Permission Propagation

Permissions flow downwards from the parent schema. For instance, consider the following example:

1syntax = "proto3";
2
3message MedicalRecords
4{
5    // Stores HIPPA related items of a customer.
6}
7
8message FinancialRecords
9{
10    // Stores financial related items of a customer.
11}
12
13message Customer
14{
15    MedicalRecords medical = 1;
16    FinancialRecords financial = 2;
17    // Stores additional personally identifiable information (PII).
18}

Now presume we have a user, Foo, with the following permissions:

MedicalRecords: No permissions by default.
FinancialRecords: Read permissions.
Customer: Write and Read permissions.

Due to the composition of Customer including MedicalRecords and FinancialRecords, Foo can do the following when using Customer as the record:

MedicalRecords: No permissions by default => Write and Read permissions.
FinancialRecords: Read permissions => Write and Read permissions.

Careful consideration needs to be made when designing schemas to ensure permissions do not propagate in unexpected ways. As such, the Execute permission and Admin role should be given sparingly, if at all, and under additional security mechanisms.
‍

Design Suggestion

Creation of a unique ID per customer with storage of MedicalRecords, FinancialRecords, and PII as separate schemas in the database prevents propagation of permissions. This further streamlines efficient storage and retrieval of data to only the required components versus the entirety of a record. UUIDs are an excellent mechanism for unique identifiers.

‍

Consider the following as a more secure alternative:

1syntax = "proto3";
2
3message MedicalRecords
4{
5	string customer_id = 1;
6    // Stores HIPPA related items of a customer.
7}
8
9message FinancialRecords
10{
11	string customer_id = 1;
12    // Stores financial related items of a customer.
13}
14
15message PII
16{
17    string customer_id = 1;
18    // Stores additional personally identifiable information (PII).
19}

Error Handling

RogueDB reports all error details in the Response message returned. The following example demonstrates parsing the errors returned:

1response = roguedb.insert(insert)
2
3if len(response.error_details) > 0:
4    for error in response.error_details:
5        print(f"Error: {error}")
6   raise RuntimeError("Error(s) encountered.")

1// #include <format>
2// #include "roguedb/queries.pb.h"
3
4roguedb.insert(&context, insert, &response);
5if(response.error_details().size() > 0)
6{
7    for(const auto& error : response.error_details())
8    {
9        std::cout << std::format("Error: {}", error) << std::endl;
10    }
11    throw std::runtime_error{"Error(s) encountered."};
12}

Reference Functions

Functions used in the examples that are not required. Below are the implementation details for your convenience to get an even quicker start to using RogueDB.

Python
C++
import os
from pathlib import Path

def detect_files(directories: list[str]) -> list[Path]:
    files = []
    for directory in directories:
        with os.scandir(directory) as scan:
            for item in scan:
                path = Path(item.path)
                if path.is_file() and path.suffix == ".proto":
                    files.append(path)
                elif path.is_dir():
                    for subitem in detect_files(directory=[item.path, ]):
                        files.append(subitem)
    return files
    
def parse_messages(response: Response) -> list:
    for message in response.messages:
	    test = Test()
        message.Unpack(test)
        messages.append(test)
#include <filesystem>
#include <fstream>
#include <string>
#include <vector>

template<typename Schema>
std::vector<Schema> parseMessages(const Rogue::Services::Response& response)
{
    std::vector<Schema> messages{};
	for(const auto& message : response.messages())
    {
        Schema message{};
        message.Unpack(message);
        messages.emplace_back(std::move(message));
    }
}

const std::string readFile(const std::string&& filename)
{
    std::ifstream inputFile{filename};
    if(!inputFile.is_open())
    {
        throw std::runtime_error{ std::format("Could not open: {}.", filename) };
    }
    std::stringstream buffer{};
    buffer << inputFile.rdbuf();
    inputFile.close();
    return buffer.str();
}

std::vector<std::filesystem::path> detectFiles(
    const std::vector<std::string>& directories)
{
    std::vector<std::filesystem::path> files{};
    for(const auto& directory : directories)
    {
        for(const auto& entry : std::filesystem::directory_iterator{directory})
        {
            if(!std::filesystem::is_directory(entry))
            {
                if(entry.path().extension() == ".proto")
                {
                    files.emplace_back(std::move(entry.path()));
                }
            }
            else
            {
                std::vector<std::string> temp{ entry.path() };
                for(auto& filename : detectFiles(entry.path()))
                {
                    files.emplace_back(std::move(filename));
                }
            }
        }
    }
    return files;
}