Documentation

Get started within minutes.

Prerequisites

Assumptions

All of RogueDB's Proto files and Root CA Certificate are located in a folder called: roguedb. All of your Protocol Buffer files are located in a folder called: data_schemas.

‍

This is not required. The examples will need to be altered in accordance to your directory structure and placement of files.

‍

Supporting functions implementation denoted by a comment of "See Reference Functions..." are excluded for clarity and not required to use RogueDB, but the implementations are included at the bottom for your convenience.

‍

Initial Connection

The following code establishes a connection to the configuration and interface services. We suggest either using our public libraries to reduce the few lines of boilerplate code or creating your own internal library.

  • ConfigurationStub provides access to manage schemas and state of a RogueDB instance.
  • InterfaceStub provides access to create, read, update, and delete operations (eg. CRUD operations) with a RogueDB instance.
Python
C++
1import grpc
2from roguedb.configure_pb2_grpc import ConfigurationStub
3from roguedb.interface_pb2_grpc import InterfaceStub
4
5configuration_port = 50051
6interface_port = 50052
7ip_address = "123.456.789.012"
8root_certificate = ""
9
10with open("roguedb/rogue_llc_ca.crt", "rb") as input_file:
11    root_certificate = input_file.read()
12    
13credentials = grpc.ssl_channel_credentials(root_certificate)
14
15configuration_channel = grpc.secure_channel(f"{ip_address}:{configuration_port}", credentials)
16configuration_stub = ConfigurationStub(channel=configuration_channel)
17
18interface_channel = grpc.secure_channel(f"{ip_address}:{interface_port}", credentials)
19interface_stub = InterfaceStub(channel=interface_channel)
1#include <format>
2
3#include <grpcpp/grpcpp.h>
4#include "roguedb/interface.grpc.pb.h"
5#include "roguedb/configure.grpc.pb.h"
6
7int main(int argc, char** argv)
8{
9    const uint32_t configurationPort{ 50051 };
10    const uint32_t interfacePort{ 50052 };
11    const std::string ipAddress{ "localhost" };
12    
13    grpc::SslCredentialsOptions sslOptions{};
14    
15    // See Reference Functions below for implementation of readFile.
16    sslOptions.pem_root_certs = readFile("roguedb/rogue_llc_ca.crt");
17    
18    std::unique_ptr<Rogue::Services::Configuration::Stub> configurationStub{
19        Rogue::Services::Configuration::NewStub(grpc::CreateChannel(
20            std::format("{}:{}", ipAddress, configurationPort), 
21            grpc::SslCredentials(sslOptions)))};
22
23    std::unique_ptr<Rogue::Services::Interface::Stub> interfaceStub{
24        Rogue::Services::Interface::NewStub(grpc::CreateChannel(
25            std::format("{}:{}", ipAddress, interfacePort), 
26            grpc::SslCredentials(sslOptions)))};
27}

Schema Registration

There are two primary steps for registering schema with a RogueDB database:

  • Identify the indices in the Protocol Buffer definition.
  • Provide the schemas to RogueDB.
Marking Indices

The first step of registering a Protocol Buffer is to mark the fields to be used for indexing with the pattern: // index-# (ex. // index-1). All schemas must have indices for use with RogueDB.

‍

Guidelines for these fields are the following:

  • Fields selected for an index should form a unique key (aka identifier)
  • Indices should be ordered in the way you think about data for most efficient look-up.

An example inspired by financial market data:

Protobuf
1syntax = "proto3";
2
3import "google/protobuf/timestamp.proto";
4
5message Aggregate
6{
7    string symbol = 1; // index-1 Extra notes are not included after marking.
8    google.protobuf.Timestamp timestamp = 2; // index-3z Non-numerical characters allowed, though discouraged.
9    uint32 aggregate_period = 3; //index-2 No space between "//" and "index" is also valid.
10    float open = 4;
11    float close = 5;
12    float low = 6;
13    float high = 7;
14    uint32 volume = 8;
15}

Observe that symbol, timestamp, and aggregate_period are selected for the indices. When thinking about market data, the symbol is often the first field to come to mind followed by the aggregate period (1 min., 5 min., etc.) and finally the time of recording (aka timestamp). These three fields prevent collision and form a unique idea for any instance of an Aggregate.

‍

Registering Schemas

The requirements to register your schema are the following:

  • Filename of the schema consisting only of alphanumerical and underscore ("_") characters.
  • Contents of the schema as a string.
  • Source type (ex. Protocol Buffers).
  • One message per file. Multiple messages supported in an upcoming release.
Python
C++
1from roguedb.queries_pb2 import Subscribe, Source
2
3# See Reference Functions for implementation of detect_files
4proto_files = detect_files(directories=["data_schemas",])
5subscribe = Subscribe()
6subscribe.api_key = "ROGUEDB_API_KEY"
7for proto_file in proto_files:
8    with open(proto_file, "r") as input_file:
9        schema = subscribe.schemas.add()
10        schema.contents = input_file.read()
11        schema.source = Source.PROTO
12        schema.filename = proto_file.name
13
14# See Initial Connection for configuration_stub initialization 
15response = configuration_stub.subscribe(subscribe)
1#include <string>
2#include <vector>
3
4#include "roguedb/queries.pb.h"
5
6int main(int argc, char** argv)
7{
8    const std::vector<std::string> directories{ "data_schemas/" };
9    
10    // See Reference Functions for implementation details of detectFiles.
11    const std::vector<std::string> protoFiles{ detectFiles(directories) };
12    Rogue::Services::Subscribe subscribe{};
13    subscribe.set_api_key("ROGUEDB_API_KEY");
14    
15    for(const auto& file : protoFiles)
16    {
17        Rogue::Services::Schema& schema{ *subscribe.add_schemas() };
18        // See Reference Functions for implementation details of readFile
19        schema.set_contents(readFile(file));
20        schema.set_source(Rogue::Services::Source::PROTO);
21        schema.set_filename(file.filename());
22    }
23    
24    // See Initial Connection for how to initialize configurationStub.
25    const Rogue::Services::Response response{ configurationStub.subscribe(subscribe) };
26}

CRUD Operations

All operations do not require a special structured querying language (eg. SQL) to execute. The API has been defined to mimic common standard data structures such as dictionaries, set, vectors, lists, and other data structures. A Payload must consist of only one schema, but all Payloads do not have to have the same schema. Failure to adhere to this requirement will result in the query not executing.

‍

Below is the Test schema used in the examples below:

Protobuf
1syntax = "proto3";
2
3message Test
4{
5    int32 attribute1 = 1; // index-1
6    bool attribute2 = 2;
7}
Create

There are no differences in single versus batch create operations in terms of API.

Python
C++
1from roguedb.queries_pb2 import Insert
2from data_schemas.test_pb2 import Test
3
4insert = Insert()
5insert.api_key = "ROGUEDB_API_KEY"
6payload = insert.data.add()
7for index in range(10):
8    payload.messages.add().Pack(Test(attribute1=index, attribute2=False))
9
10# See Initial Connection for initialization of interface_stub    
11response = interface_stub.insert(insert)
1#include "roguedb/queries.pb.h"
2#include "data_schemas/test.pb.h"
3
4int main(int argc, char** argv)
5{
6    Insert insert{};
7    insert.set_api_key("ROGUEDB_API_KEY");
8    Rogue::Services::Payload& payload{ *insert.add_payloads() };
9    for(uint32_t index{0}; index < 10; index++)
10    {
11        Test test{};
12        test.set_attribute1(index);
13        test.set_attribute2(false);
14        *payload.add_messages() = std::move(test);
15    }
16    
17    // See Initial Connection for how to initialize interfaceStub.
18    const Rogue::Services::Response response{ interfaceStub.insert(insert) };
19}
Update

There are no differences in single versus batch update operations in terms of API.

Python
C++
1from roguedb.queries_pb2 import Update
2from data_schemas.test_pb2 import Test
3
4update = Update()
5update.api_key = "ROGUEDB_API_KEY"
6payload = update.data.add()
7for index in range(10):
8    payload.messages.add().Pack(Test(attribute1=index, attribute2=False))
9
10# See Initial Connection for initialization of interface_stub    
11response = interface_stub.update(update)
1#include "roguedb/queries.pb.h"
2#include "data_schemas/test.pb.h"
3
4int main(int argc, char** argv)
5{
6    Update update{};
7    insert.set_api_key("ROGUEDB_API_KEY");
8    Rogue::Services::Payload& payload{ *update.add_payloads() };
9    for(uint32_t index{0}; index < 10; index++)
10    {
11        Test test{};
12        test.set_attribute1(index);
13        test.set_attribute2(false);
14        *payload.add_messages() = std::move(test);
15    }
16    
17    // See Initial Connection for how to initialize interfaceStub.
18    const Rogue::Services::Response response{ interfaceStub.update(update) };
19}
Delete

There are no differences in single versus batch delete operations in terms of API.

Python
C++
1from roguedb.queries_pb2 import Remove
2from data_schemas.test_pb2 import Test
3
4remove = Remove()
5remove.api_key = "ROGUEDB_API_KEY"
6payload = remove.data.add()
7for index in range(10):
8    payload.messages.add().Pack(Test(attribute1=index, attribute2=False))
9
10# See Initial Connection for initialization of interface_stub
11response = interface_stub.remove(remove)
1#include "roguedb/queries.pb.h"
2#include "data_schemas/test.pb.h"
3
4int main(int argc, char** argv)
5{
6    Remove remove{};
7    remove.set_api_key("ROGUEDB_API_KEY");
8    Rogue::Services::Payload& payload{ *remove.add_payloads() };
9    for(uint32_t index{0}; index < 10; index++)
10    {
11        Test test{};
12        test.set_attribute1(index);
13        test.set_attribute2(false);
14        *payload.add_messages() = std::move(test);
15    }
16    
17    // See Initial Connection for how to initialize interfaceStub.
18    const Rogue::Services::Response response{ interfaceStub.remove(remove) };
19}
Read

RogueDB's querying API allows programmatic composition by using logical operators (&&, ||) and comparison operators (==, !=, <, <=, >, >=) to replicate conditionals in your native progamming language. There are no joins, inner joins, outer joins, wheres, on, as, or any other special keywords required to be learned as a result.

‍

A query can use an entire message with its indices or specifying individual fields as the operands in a query. RogueDB supports basic queries that involve the same logical operator (eg. homogenous application) and a single schema. There must be a 1:1 or 1:1:1 of operand and conditional or operand, conditional, and operand field for the query to be valid.

‍

Future support is coming for complex queries that involve combining different logical operators for a single schema and support for  multiple schemas in a single query.

‍

The following example demonstrates using a schema's indices as the operand:

Python
C++
1from roguedb.queries_pb2 import BaseExpression, Search, LogicalOperator, ComparisonOperator
2from data_schemas.test_pb2 import Test
3
4# Conditional: Test.attribute1 >= 1 and Test.attribute1 <= 10 
5search = Search()
6search.api_key = "ROGUEDB_API_KEY"
7expression = search.queries.add().base_expression
8expression.logical_operator = LogicalOperator.AND
9
10expression.comparison_operators.append(ComparisonOperators.GREATER_EQUAL)
11expression.operands.add().Pack(Test(attribute1=1))
12
13expression.comparison_operators.append(ComparisonOperators.LESSER_EQUAL)
14expression.operands.add().Pack(Test(attribute1=10))
15
16# See Initial Connection for initialization of interface_stub
17response = interface_stub.search(search)
18messages = list()
19for payload in response.payloads:
20    for message in payload.messages:
21        test = Test()
22        message.Unpack(test)
23        messages.append(test)
1#include <vector>
2#include "roguedb/queries.pb.h"
3#include "data_schemas/test.pb.h"
4
5int main(int argc, char** argv)
6{
7    // Conditional: Test.attribute1 >= 1 && Test.attribute1 <= 10 
8    Rogue::Services::Search search{};
9    search.set_api_key("ROGUEDB_API_KEY");
10    Rogue::Services::BaseExpression& expression{ (*search.add_queries()).base_expression() };
11    expression.set_logical_operator(Rogue::Services::LogicalOperator::AND);
12    
13    expression.add_comparison_operators(Rogue::Services::ComparisonOperator::GREATER_EQUAL);
14    Test test{};
15    test.set_attribute1(1);
16    (*expression.add_operands()).PackFrom(test);
17    
18    expression.add_comparison_operators(Rogue::Services::ComparisonOperator::LESSER_EQUAL);
19    test.set_attribute1(10);
20    (*expression.add_operands()).PackFrom(test);
21    
22    const Rogue::Services::Response response{ interfaceStub.search(search) };
23    std::vector<Test> messages{}:
24    for(const auto& payload : response.payloads())
25    {
26        for(const auto& message : payload.messages())
27        {
28            Test test{};
29            message.Unpack(test);
30            messages.emplace_back(std::move(test));
31        }
32    }
33}

The following example demonstrates using a schema's fields as the operand:

Python
C++
1from roguedb.queries_pb2 import BaseExpression, Search, LogicalOperator, ComparisonOperator
2from data_schemas.test_pb2 import Test
3
4# Conditional: Test.attribute1 >= 1 and Test.attribute2 != False 
5search = Search()
6search.api_key = "ROGUEDB_API_KEY"
7expression = search.queries.add().base_expression
8expression.logical_operator = LogicalOperator.AND
9
10expression.comparison_operators.append(ComparisonOperators.GREATER_EQUAL)
11expression.operands.add().Pack(Test(attribute1=1))
12expression.operand_fields.append(1)
13
14expression.comparison_operators.append(ComparisonOperators.NOT_EQUAL)
15expression.operands.add().Pack(Test(attribute2=False))
16expression.operand_fields.append(2)
17
18# See Initial Connection for initialization of interface_stub
19response = interface_stub.search(search)
20messages = list()
21for payload in response.payloads:
22    for message in payload.messages:
23        test = Test()
24        message.Unpack(test)
25        messages.append(test)
1#include <vector>
2#include "roguedb/queries.pb.h"
3#include "data_schemas/test.pb.h"
4
5int main(int argc, char** argv)
6{
7    // Conditional: Test.attribute1 >= 1 && Test.attribute2 != false 
8    Rogue::Services::Search search{};
9    search.set_api_key("ROGUEDB_API_KEY");
10    Rogue::Services::BaseExpression& expression{ (*search.add_queries()).base_expression() };
11    expression.set_logical_operator(Rogue::Services::LogicalOperator::AND);
12    
13    expression.add_comparison_operators(Rogue::Services::ComparisonOperator::GREATER_EQUAL);
14    Test test{};
15    test.set_attribute1(1);
16    (*expression.add_operands()).PackFrom(test);
17    expression.add_operand_fields(1);
18    
19    expression.add_comparison_operators(Rogue::Services::ComparisonOperator::NOT_EQUAL);
20    test.set_attribute2(false);
21    (*expression.add_operands()).PackFrom(test);
22    expression.add_operand_fields(2);
23    
24    const Rogue::Services::Response response{ interfaceStub.search(search) };
25    std::vector<Test> messages{}:
26    for(const auto& payload : response.payloads())
27    {
28        for(const auto& message : payload.messages())
29        {
30            Test test{};
31            message.Unpack(test);
32            messages.emplace_back(std::move(test));
33        }
34    }
35}

User Roles and Permissions

RogueDB supports the creation of users and assigning of permissions on a schema level basis. Your API access key associated with your subscription has all permissions by default and cannot be revoked. Users have no permissions by default. The following permissions can be assigned:

  • Read: The ability to perform the Read operation (aka search) for a schema.
  • Write: The ability to perform the Create, Update, and Delete operations (aka insert, update, and remove) for a schema.
  • Execute: The ability to modify a schema's definition through the Configuration service.
  • Admin: Grants all permissions and overrides any existing permission.

Permissions flow downwards from the parent schema. For instance, consider the following example:

Protobuf
1syntax = "proto3";
2
3message MedicalRecords
4{
5    // Stores HIPPA related items of a customer.
6}
7
8message FinancialRecords
9{
10    // Stores financial related items of a customer.
11}
12
13message Customer
14{
15    MedicalRecords medical = 1;
16    FinancialRecords financial = 2;
17    // Stores additional personally identifiable information (PII).
18}

Now presume we have a user, Foo, with the following permissions:

  • MedicalRecords: No permissions by default.
  • FinancialRecords: Read permissions.
  • Customer: Write and Read permissions.

Due to the composition of Customer including MedicalRecords and FinancialRecords, Foo can do the following when using Customer as the record:

  • MedicalRecords: No permissions by default => Write and Read permissions.
  • FinancialRecords: Read permissions => Write and Read permissions.

Careful consideration needs to be made when designing schemas to ensure permissions do not propagate in unexpected ways. As such, the Execute permission and Admin role should be given sparingly, if at all, and under additional security mechanisms.
‍

Design Suggestion

Creation of a unique ID per customer with storage of MedicalRecords, FinancialRecords, and PII as separate schemas in the database prevents propagation of permissions. This further streamlines efficient storage and retrieval of data to only the required components versus the entirety of a record. UUIDs are an excellent mechanism for unique identifiers.

‍

Consider the following as a more secure alternative:

Protobuf
1syntax = "proto3";
2
3message MedicalRecords
4{
5	string customer_id = 1;
6    // Stores HIPPA related items of a customer.
7}
8
9message FinancialRecords
10{
11	string customer_id = 1;
12    // Stores financial related items of a customer.
13}
14
15message PII
16{
17    string customer_id = 1;
18    // Stores additional personally identifiable information (PII).
19}
Usage

RogueDB uses a case insensitive mapping of schema to permissions. The following is an example of User creation:

Python
C++
1from roguedb.queries_pb2 import Insert
2from roguedb.user_pb2 import User, Permission
3from data_schemas.test_pb2 import Test
4
5insert = Insert()
6insert.api_key = "ROGUEDB_API_KEY"
7payload = insert.data.add()
8
9user = User(api_key="CUSTOMER_MANAGED_KEY", admin=False)
10user.permissions["test"] = Permission.READ_WRITE_EXECUTE
11payload.messages.add().Pack(user)
12
13# See Initial Connection for initialization of interface_stub    
14response = interface_stub.insert(insert)
1#include "roguedb/queries.pb.h"
2#include "roguedb/user.pb.h"
3#include "data_schemas/test.pb.h"
4
5int main(int argc, char** argv)
6{
7    Insert insert{};
8    insert.set_api_key("ROGUEDB_API_KEY");
9    Rogue::Services::Payload& payload{ *insert.add_payloads() };
10    
11    Rogue::Services::User user{};
12    user.set_api_key("CUSTOMER_MANAGED_KEY");
13    user.set_admin(false);
14    (*user.mutable_permissions())["user"] = Rogue::Services::Permission::READ;
15    
16    *payload.add_messages() = std::move(user);
17    
18    // See Initial Connection for how to initialize interfaceStub.
19    const Rogue::Services::Response response{ interfaceStub.insert(insert) };
20}

Error Handling

RogueDB reports all error details in the Response message returned. The following example demonstrates parsing the errors returned:

Python
C++
1# Code leading up to this point...
2response = interface_stub.insert(insert)
3
4if len(response.error_details) > 0:
5    for error in response.error_details:
6        print(f"Error: {error}")
7   raise RuntimeError("Error encountered.")
1#include <format>
2#include "roguedb/queries.pb.h"
3
4int main(int argc, char** argv)
5{
6    // Code leading up to this point...
7    const Rogue::Services::Response response{ interfaceStub.insert(insert) };
8    if(response.error_details().size() > 0)
9    {
10        for(const auto& error : response.error_details())
11        {
12        	std::cout << std::format("Error: {}", error) << std::endl;
13        }
14        throw std::runtime_error{"Error encountered."};
15    }
16}

Reference Functions

Functions used in the examples that are not required. Below are the implementation details for your convenience to get an even quicker start to using RogueDB.

Python
C++
1import os
2from pathlib import Path
3
4def detect_files(directories: list[str]) -> list[Path]:
5    files = []
6    for directory in directories:
7        with os.scandir(directory) as scan:
8            for item in scan:
9                path = Path(item.path)
10                if path.is_file() and path.suffix == ".proto":
11                    files.append(path)
12                elif path.is_dir():
13                    for subitem in detect_files(directory=[item.path, ]):
14                        files.append(subitem)
15    return files
1#include <filesystem>
2#include <fstream>
3#include <string>
4#include <vector>
5
6const std::string readFile(const std::string&& filename)
7{
8    std::ifstream inputFile{filename};
9    if(!inputFile.is_open())
10    {
11        throw std::runtime_error{ std::format("Could not open: {}.", filename) };
12    }
13    std::stringstream buffer{};
14    buffer << inputFile.rdbuf();
15    inputFile.close();
16    return buffer.str();
17}
18
19std::vector<std::filesystem::path> detectFiles(
20    const std::vector<std::string>& directories)
21{
22    std::vector<std::filesystem::path> files{};
23    for(const auto& directory : directories)
24    {
25        for(const auto& entry : std::filesystem::directory_iterator{directory})
26        {
27            if(!std::filesystem::is_directory(entry))
28            {
29                if(entry.path().extension() == ".proto")
30                {
31                    files.emplace_back(std::move(entry.path()));
32                }
33            }
34            else
35            {
36                std::vector<std::string> temp{ entry.path() };
37                for(auto& filename : detectFiles(entry.path()))
38                {
39                    files.emplace_back(std::move(filename));
40                }
41            }
42        }
43    }
44    return files;
45}

Still have questions?

Please reach out for additional clarification.

Contact

First Mover Advantage
 Limited Availability Discounts

0-1,000 Users: 50% Discount. Promo: FIRST1K
1,001-5,000 Users: 25% Discount. Promo: FIRST5K
5,001-10,000 Users: 10% Discount. Promo: FIRST10K

Get Started